Raspberry Pi – Fetching live meteorologic data from the web!

In version 1.0 of the paludarium, I had fetched an entire year of data, and I put that into the controlling software. That software in turn simulated that year of weather over and over again. But wouldn’t it be the coolest thing if I managed to pull the meteorologic data live from the web? With the Raspberry Pi I figured it should be possible. Todays goal was to fetch live meteorologic data from the La Selva biological station in Costa Rica. After some fighting with basic Python, I managed to get it going!

Where the data is

The data is actually freely available here: http://www.ots.ac.cr/meteoro/default.php?pestacion=2.

La Selva Biological Station in Costa Rica – Home of the meteo station who’s data I’ll be using!

I found out you can build a URL that will fetch half-hour data for one entire day. As an example:

http://www.ots.ac.cr/meteoro/detailed_view.php \
?dataPageSize=48&pfmin=20130813&pfmax=20130813&est=2

The variables in bold mention the number of lines to show (48 for 24 hours of half-hourly data), begin and end date and the number “2”, which is the La Selva station (other numbers will show data from other stations in Costa Rica).

Now I got stoked. Looking at the source of the web page, I could clearly see the tables just sitting there with all the meteo data inside! Time to boot the Raspberry Pi and play with some Python.


Fetching the meteo data using Python

I wrote a simple Python script to fetch the data and print it into a file. I plan to run this script once a day to update the measurements:

import urllib
import urllib2
import datetime
from bs4 import BeautifulSoup
from string import replace
import re
import time

#Determine the date I want to fetch. For now fetch data 2 days back to make sure I get 24 hours
jungledate = datetime.datetime.now()-datetime.timedelta(days=2)

location = 2  # La Selva Station is location 2

#Build the URL we need
url = 'http://www.ots.ac.cr/meteoro/detailed_view.php?dataPageSize=48& \
pfmin='+str(jungledate.year)+str(jungledate.month).zfill(2)+str(jungledate.day).zfill(2)+'& \
pfmax='+str(jungledate.year)+str(jungledate.month).zfill(2)+str(jungledate.day).zfill(2)+ \
'&est='+str(location)

# Try to open the web page. On failure, wait for a minute then retry.
opened = 0
while opened == 0:
        opened=1
        try:
                response = urllib2.urlopen(url)
        except urllib2.URLError:
                print "opening site failed.Waiting 10 seconds"
                time.sleep(60)
                opened = 0

html = response.read()

#Parse the HTML page by using Beautiful Soup
soup = BeautifulSoup(html)
#get all rows that are the start of a table that has the class of either Row or AltRow
rows = soup.findAll('tr',{ 'class' : ["Row" , "AltRow"]})

#open up a local file to store one day of data in
f1=open('./meteodata', 'w+')

#grab all rows and print them into the file (and on screen)
for tr in rows:
        cols = tr.findAll('td')
        for td in cols:
                text = td.find(text=True)
                text = re.sub('\xa0', '', text)
                print >>f1, text,
                print text,
        print >>f1
        print

f1.close()

This results in a file containing data like this:

08/14/2013 21:00 24.83 24.90 24.79 92.90 0.000 0.317 0.308 107.80 7.89 1006.524 [..] 
08/14/2013 20:30 25.03 25.15 24.84 92.60 0.000 0.446 0.434 99.20 8.07 1006.063 [..]
08/14/2013 20:00 25.26 25.45 25.12 91.90 0.000 0.369 0.318 113.30 21.50 1005.797 [..]
08/14/2013 19:30 25.49 25.60 25.40 92.20 0.000 0.937 0.905 105.30 14.34 1005.397 [..]
08/14/2013 19:00 25.76 25.93 25.58 91.90 0.000 0.692 0.651 106.60 18.67 1004.868 [..]
08/14/2013 18:30 26.04 26.26 25.88 91.10 0.000 0.306 0.286 86.30 10.45 1004.469 [..]
(and so on)

You see the magic appearing?? I can use these numbers directly to control the paludarium!


Reading the data back from the file whenever I need it

To avoid having to read the website every 30 minutes for new data, I decided to fetch an entire day and store that day into a file. Using a very simple Python script I can read the data I want out of the file. For now, it’s a simple script that jsut reads one line of data and puts it into variables:

import re
import string

needhour = 15
needminutes = 30
needstring= str(needhour).zfill(2)+":"+str(needminutes).zfill(2)

data = open('./meteodata')
with data as meteo:
    for line in meteo:
        line = string.split(line)
        if line[1] == needstring:
                AirTemp = float(line[2])
                Humidity = float(line[5])
                Rain = float(line[6])
                WindSpeed = float(line[7])
                Pressure = float(line[11])
                Light = float(line[12])
data.close()

print "Airtemp   ",AirTemp
print "Humidity  ",Humidity
print "Rain      ",Rain
print "WindSpeed ",WindSpeed
print "Pressure  ",Pressure
print "Light     ",Light

As you can see, I specify the hours (0-23) and minutes I need (0 or 30), and the script fishes the data out of the meteo file and puts it into variables. In time this will become a function to request the data where ever I need it in the code.


What it will do

What it will do? What WON’T it do!!

I WANT THIS TOO!!!

With this, I can simulate the weather inside the paludarium matches to the exact same weather in Costa Rica!