UPDATE: Remember when I said there are some missing data? Yeah. Only for the case I want to analyze. The exact 16 hours I most wanted. Funny that the operational data that is saved all over the place has more observations than the archived ASOS data at NCDC. Redundancy is an important part of data archival.
I am analyzing a particular case and have been looking at the unique observations that the Oklahoma mesonet collects. I want to add to this huge data source. Therein lies the issue. Merging data sets is quite a task. Of course the fancier one gets, the more trouble there is.
Recently the NWS added many stations to its list of archived, 1 and 5 minute ASOS data. It is a decent dataset even if it is spatially sparse. The issue is the format. Now regular hourly and special observations get transmitted in METAR format. There is a nice decoder written for GEMPAK which processes this data. I would say it is awesome but it is suitable. What makes it better is that it retains the whole METAR data string for subsequent data mining. This ensures that some of the metadata (+TSGRFC, PRSFR, etc) are not lost.
However, the potential research quality dataset currently being archived at NCDC undergoes no quality control and the files can have transmission problems. It also suffers because if the data are not relayed in time, the data is lost. Thus one must process the data and visually inspect for issues. I did this many years ago as part of my PhD training for the BAMEX field project and it was a nightmare to write code to process the 1 minute data. I ended up having multiple Fortran codes to deal with some of the transmission problems, formatting problems or missing data.
The five minute data are stored in METAR format, but not exactly METAR form. I would think someone could process the minute data into a quality 5 minute data set with a decent, readable format and provide some measure of quality control. (I will share my code which reads the data. ) This could be an interesting data set. As it is now it is difficult to work with, but not impossible. I heard that the community was trying to organize a network of networks for surface data. I hope they succeed and I hope they model whatever they do with the successful components of the Oklahoma Mesonet, MesoWest, and the Iowa Environmental Mesonet.
Showing posts with label data. Show all posts
Showing posts with label data. Show all posts
Tuesday, February 8, 2011
Thursday, January 20, 2011
Ice is nice
http://www.mesonet.org/index.php/weather/meteogram/
The Norman time series of weather data, specifically wind, was rather strange today. Just after 2:20am the wind began to slow down dropping to 3-4 mph. Funny thing is I remember it being windy at the site around 9am. This is the effect of freezing rain on an anemometer. The propeller didn't freeze altogether as I have seen in Iowa data but the extra mass made it harder to spin. The entire assembly didn't freeze either as the wind direction did change albeit slightly.
The airport in Norman first reported unknown precipitation at 2:11 am at a temperature of 30 and it dropped from there, while the winds stayed in the 15-20mph range throughout the night.
As far as good data goes, I think there is a sonic anemometer but it is made of metal and I wonder how good the wind data would be with a coat of ice on it. Therefore it would be difficult to correct. Human quality control could handle this but I doubt with station to station comparison this time series could be flagged as being questionable. Realize that the rain gauge at the site is not a typical heated tipping bucket so there was no precipitation recorded at the station.
This is not a criticism of the data in any way; rather it highlights the difficulty of making all-weather observations year round even on well funded networks. This even happens in hurricanes when the power goes out or when water tops the instruments or penetrates the wiring.
Its important to maintain the observations that are good for future work either in weather or climate and hopefully this provides a gut-check. Some observations will sneak in that are bad and therefore it is important to realize that no observation system is perfect or error-free. Sometimes this is why "tricks" or manipulation must be performed because real data has issues and they must be sorted out, flagged, approved or removed.
The Norman time series of weather data, specifically wind, was rather strange today. Just after 2:20am the wind began to slow down dropping to 3-4 mph. Funny thing is I remember it being windy at the site around 9am. This is the effect of freezing rain on an anemometer. The propeller didn't freeze altogether as I have seen in Iowa data but the extra mass made it harder to spin. The entire assembly didn't freeze either as the wind direction did change albeit slightly.
The airport in Norman first reported unknown precipitation at 2:11 am at a temperature of 30 and it dropped from there, while the winds stayed in the 15-20mph range throughout the night.
As far as good data goes, I think there is a sonic anemometer but it is made of metal and I wonder how good the wind data would be with a coat of ice on it. Therefore it would be difficult to correct. Human quality control could handle this but I doubt with station to station comparison this time series could be flagged as being questionable. Realize that the rain gauge at the site is not a typical heated tipping bucket so there was no precipitation recorded at the station.
This is not a criticism of the data in any way; rather it highlights the difficulty of making all-weather observations year round even on well funded networks. This even happens in hurricanes when the power goes out or when water tops the instruments or penetrates the wiring.
Its important to maintain the observations that are good for future work either in weather or climate and hopefully this provides a gut-check. Some observations will sneak in that are bad and therefore it is important to realize that no observation system is perfect or error-free. Sometimes this is why "tricks" or manipulation must be performed because real data has issues and they must be sorted out, flagged, approved or removed.
Subscribe to:
Posts (Atom)