How to Programmatically Retrieve Weather Data

English: Sample image of the National Weather ...

Image via Wikipedia

Background:
I was working on a project for work a few weeks ago where I needed to get a good idea of the ambient temperatures near the office for a couple of months. I’m running a set of experiments that test how hot our units get when transmitting different amounts of data. I wanted to know how well our products wick-away heat under various temperature conditions. At the time that I started the test, the temperature chamber was occupied, so I decided to go ahead using our outside testing environment.

Getting the Data
At first I didn’t have much data to work with, so I retrieved the data I needed manually from the National Weather Service’s Hourly Weather Data page. After doing this for a while, especially as more data came in became very tiring and inefficient.

Next I tried to write some code in python (since my whole analysis suite was written in python) to retrieve the data I needed from that site. That didn’t work so well since the data wasn’t coded in any XML formats… I resorted to going down the path of scraping data, but this also turned to be a very arduous process.

I then decided to look for other repositories of weather data that exposed a web services interface. After a brief search I came across the Weather Underground API site. This was perfect… and it was free, with a limitation – the free developer plan supported 500 calls per day, 10 per minute.  I figured that the constraints were just fine.

So I developed the following code to  extract a single day’s worth of data, place hourly data into a dictionary containing 24 bins.

import urllib2
import simplejson as json

def get_historical_temps( datestr ):

url_to_open = ‘http://api.wunderground.com/api//history_’ + datestr + ‘/q/MO/Saint_Louis.json’
f = urllib2.urlopen(url_to_open)

# Read json data

json_string = f.read()

f.close()
parsed_json = json.loads(json_string)

# Define dictionary

hour_temps_dict = {}

for h in range(0,24):
try:
hour_temps_dict[h] = parsed_json[‘history’][‘observations’][h][‘tempi’]
except IndexError:
hour_temps_dict[h] = -99

return hour_temps_dict

Conclusion

It was with this hourly data that I was able to import the data into the rest of my data flow which eventually inserted the data into a MySQL database.

When I started looking for ways to do this, even though there were many sites showing how to get the data, none of them showed a really simple method for getting the exact type of data I wanted.   So, I pulled up my bootstraps and went about figuring it out on my own.

As you can see, it’s pretty easy to get weather data.  One last thing, to help with figuring out which data fields you need, I recommend using a JSON viewer such as JSONView for Firefox.

Good luck… let me know if you need any help.

Advertisements

Leave a comment

Filed under Programming

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s