Chapter 5 Research Project
5.1 Abstract
The Australasian Region of is highly important in terms of impacts on the weather and climate of the Earth. It is considered the most important energy source region in the entire global circulation system owing to a number of coincident factors, the most significant being geographic location and topography, both of which contribute to the development of the warmest large area of ocean on Earth, the tropical warm pool. This is a region of intensive ocean/atmosphere interaction with widespread convection and environment favorable for atmosphere phenomenon in the tropical central pacific which is another significant climate driver of the planet which causes extreme weather.
5.2 Introduction
Recognising the high importance of accurate monitoring of precipitation extremes, the World Meteorological Organization (WMO) established the Space-based Weather and Climate Extremes Monitoring (SWCEM) Demonstration Project (SEMDP) with focus on monitoring heavy precipitations and drought in countries of the South-East Asia and the Pacific, including Australia. For many decades, accurate measurements of precipitations using rain gauges have been conducted at meteorological observation stations of the Australian Bureau of Meteorology (BoM). Surface-based observations form a foundation for deriving precipitation climatology (long-term station averages over basic period, typically 30 years, e.g. 1961-1990) and determining precipitation extremes (i.e. departure from climatology which could be expressed as deciles, percentiles, highest/lowest on record etc.). While surface-based measurements provide accurate precipitation records at places of observations, interpolation of meteorological station data to produce maps is a challenge as rainfall is highly variable - spatially and temporally. Non-uniform spatial distribution of meteorological stations (often separated by large distances, particularly in low density population areas) and local topography (e.g. mountains) create additional challenges in producing accurate maps.
Over the past few decades, space-based observations demonstrated their usefulness for continuous monitoring of precipitations. Satellite remote sensing provides potentially global coverage; thus maps of precipitation climatology derived using space-based data could be of immense importance for regions with limited surface-based observations. In addition, having long-term records of space-based precipitation data obtained by various satellite instruments allows not only produce climatology but also examine occurrence of precipitation extremes. As an important contribution to SEMDP, “Spatial and temporal variability in extreme precipitations in Australia” research project will be focused on analyses of extreme precipitations and their relations to floods in Australia.
First, validation of precipitations derived from space-based observations with in situ data will be performed using rain gauge data from archives of BoM, to examine accuracy of satellite remote sensing data over Australia. Specifically, performance of satellite precipitation estimates in quantifying the extreme rainfall events measured over different spatial scales (e.g. 0.25deg, 0.5deg, 1.0deg etc.) will be examined. Following on the relations between heavy precipitation and floods will be examined.
The mathematical methods which are proposed to be employed for achieving these research aims could be:
- : It is very important to understand if trend-cyclic component in these data exist, is any cyclic periodicity included in these series and what is the trend component in these data.\
- The flood forecasting model should be developed, where the model parameters, such trend, trend damping coefficients, seasonality parameters and other will be defined using the optimization techniques. The methods of non-linear squared error minimization or maximum likelihood can be utilized on this stage of the project.
The resulting improved knowledge of spatial and temporal variability in precipitation extremes and their relations to floods in Australia will advance scientific understanding of physical processes leading to climate extremes such as prolonged time periods of excessive rainfall and rainfall deficit, which in turn could lead to flooding and droughts. Hence, outcomes of the proposed MS research potentially have the ability to save lives and reduce economic losses, demonstrating a clear benefit to society.
5.3 Automating download of tables
Automated downloading of Bureau of Meteorology data for my research by weather station
Looked at one example in how to do this and wrote the script below which searches all the weather stations I need, zips them and download them
#daily code = 136
#monthy code = 139
### Below Code to get a list of BOM Weather Stations in NT
library(RCurl)
library(dplyr)
# Where our files will be downloaded to
setwd('Q:/data')
x = getURL("https://raw.githubusercontent.com/Phayder/Datasets/main/stations.csv")
station = read.csv(text = x)
#renaming our column and subsetting for only station number
names(station)[1]="stn_num"
station=station[c(1)]
bomdata= function(station,code){
for(i in 1: length(station)){
p.url=paste("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_stn_num=",station[i],"&p_display_type=availableYears&p_nccObsCode=",code,sep ="")
download.file(p.url,"test.txt")
filelist = list.files(pattern = ".txt")
foo= file(filelist,"r")
text= suppressWarnings(readLines(foo))
close(foo)
l= regexpr(":",text[1])
m= unlist(gregexpr(",", text[1], perl = TRUE))
pc= substr(text[1],l[[1]]+1,l[[1]]+(m[2]-(l[[1]]+1)))
url=paste("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile&p_stn_num=",station[i],"&p_c=",pc,"&p_nccObsCode=",code,"&p_startYear=2013", sep ="")
suppressWarnings(download.file(url,paste(station[i],".zip",sep= ""), mode = "wb"))
unlink("test.txt")
}
}This will loop through all the stations in the list (stations)
## I used 136 as I am interested in daily data, however I can change that to 139 to do monthly
for(i in station){
bomdata(i,136)
}After unzipping my .zip files I am left with a lot of .csv files. Now we merge all of them into a csv with only one header for all the weather stations.
setwd('Q:/Bom_CSV')
lib.loc=.libPaths()[-1]
out.file=""
path = "Q:/Bom_CSV"
file.names = dir(path, pattern =".csv")
for(i in 1:length(file.names)){
file = read.table(file.names[i],header=TRUE, sep=",", stringsAsFactors=FALSE)
out.file = rbind(out.file, file)
}
# Here the output csv is written to a new empty folder, this is important if you run this regularly or else running this again will include the merged data and the new data, leading to duplication
write.table(out.file, file = "Q:/Final_BoM_Data/Final_Data.csv",sep=",",
row.names = FALSE, qmethod = "double",fileEncoding="windows-1252")