6 fileIndex

indexFTP recursively lists all the files on an FTP-server (using RCurl::getURL).
From those paths, createIndex generates fileIndex and gridIndex. It also downloads all (ca 84) description files with metadata and creates metaIndex and geoIndex.
All indexes are updated irregularly with the internal function updateIndexes.

selectDWD helps to query the fileIndex.

The DWD often (but irregularly) updates or expands datasets, at which point the filenames in historical folders change.

If you find the index to be outdated (“download.file errors: […] cannot open URL”), check in the latest commits if the github development version already has an updated index (install with rdwd::updateRdwd()).
If not, please let me know and I will update it.

Meanwhile, either use current=TRUE in selectDWD:

# all files at a given path, with current file index (RCurl required):
links <- selectDWD(res="monthly", var="more_precip", per="hist", current=TRUE)

or, for repetitive usage, create your own file index (for a certain subfolder):

# recursively list files on the FTP-server:
files <- indexFTP("hourly/sun") # use dir="some_path" to save the output elsewhere
berryFunctions::headtail(files, 5, na=TRUE)
# create and use a personal file index:
cursun <- createIndex(files)
head(cursun)
sunlink <- selectDWD("Potsdam", res="hourly", var="sun", per="r", findex=cursun)


# with other FTP servers, this should also work...
funet <- indexFTP(base="ftp.funet.fi/pub/standards/w3/TR/xhtml11/", folder="")
p <- RCurl::getURL(    "ftp.funet.fi/pub/standards/w3/TR/xhtml11/",
                       verbose=T, ftp.use.epsv=TRUE, dirlistonly=TRUE)

6.1 metaIndex

selectDWD also uses a complete data.frame with meta information, metaIndex (derived from the “Beschreibung” files in fileIndex).

# All metadata at all folders:
data(metaIndex)
str(metaIndex, vec.len=2)
## 'data.frame':    102812 obs. of  12 variables:
##  $ Stations_id  : int  1 1 1 1 1 ...
##  $ von_datum    : Date, format: "1891-01-01" "1891-01-01" ...
##  $ bis_datum    : Date, format: "1985-12-31" "1985-12-31" ...
##  $ Stationshoehe: int  478 478 478 478 478 ...
##  $ geoBreite    : num  47.8 47.8 ...
##  $ geoLaenge    : num  8.85 8.85 ...
##  $ Stationsname : chr  "Aach" "Aach" ...
##  $ Bundesland   : chr  "Baden-Wuerttemberg" "Baden-Wuerttemberg" ...
##  $ res          : chr  "annual" "annual" ...
##  $ var          : chr  "more_precip" "more_precip" ...
##  $ per          : chr  "historical" "recent" ...
##  $ hasfile      : logi  TRUE FALSE TRUE ...
View(data.frame(sort(unique(rdwd:::metaIndex$Stationsname)))) # ca 6k entries

readDWD can correctly read such a data.frame from any folder on the FTP server:

# file with station metadata for a given path:
m_link <- selectDWD(res="monthly", var="more_precip", per="hist", meta=TRUE)
print_short(m_link) # (Monatswerte = monthly values, Beschreibung = description)
## [1] "---/monthly/more_precip/historical/RR_Monatswerte_Beschreibung_Stationen.txt"
meta_monthly_rain <- dataDWD(m_link) # not executed in website creation
str(meta_monthly_rain)

Meta files may list stations for which there are actually no files. These refer to nonpublic datasets (The DWD cannot publish all datasets because of copyright restrictions). To request those, please contact or .