In version 1.0 of the paludarium, I had fetched an entire year of data, and I put that into the controlling software. That software in turn simulated that year of weather over and over again. But wouldn’t it be the coolest thing if I managed to pull the meteorologic data live from the web? With the Raspberry Pi I figured it should be possible. Todays goal was to fetch live meteorologic data from the La Selva biological station in Costa Rica. After some fighting with basic Python, I managed to get it going!<\/p><\/blockquote>\n
The data is actually freely available here: http:\/\/www.ots.ac.cr\/meteoro\/default.php?pestacion=2<\/a>. <\/p>\n La Selva Biological Station in Costa Rica – Home of the meteo station who’s data I’ll be using!<\/p><\/div>\n I found out you can build a URL that will fetch half-hour data for one entire day. As an example:<\/p>\n http:\/\/www.ots.ac.cr\/meteoro\/detailed_view.php \\ The variables in bold mention the number of lines to show (48 for 24 hours of half-hourly data), begin and end date and the number “2”, which is the La Selva station (other numbers will show data from other stations in Costa Rica).<\/p>\n Now I got stoked. Looking at the source of the web page, I could clearly see the tables just sitting there with all the meteo data inside! Time to boot the Raspberry Pi and play with some Python. I wrote a simple Python script to fetch the data and print it into a file. I plan to run this script once a day to update the measurements:<\/p>\n This results in a file containing data like this:<\/p>\n You see the magic appearing?? I can use these numbers directly to control the paludarium! To avoid having to read the website every 30 minutes for new data, I decided to fetch an entire day and store that day into a file. Using a very simple Python script I can read the data I want out of the file. For now, it’s a simple script that jsut reads one line of data and puts it into variables:<\/p>\n As you can see, I specify the hours (0-23) and minutes I need (0 or 30), and the script fishes the data out of the meteo file and puts it into variables. In time this will become a function to request the data where ever I need it in the code. What it will do? What WON’T it do!! <\/p>\n I WANT THIS TOO!!!<\/p><\/div>\n With this, I can simulate the weather inside the paludarium matches to the exact same weather in Costa Rica!<\/p>\n","protected":false},"excerpt":{"rendered":" In version 1.0 of the paludarium, I had fetched an entire year of data, and I put that into the controlling software. That software in turn simulated that year of weather over and over again. But wouldn’t it be the … Read more
\nWhere the data is<\/strong><\/p>\n![\"La](\"http:\/\/paluweb.nl\/wp-content\/uploads\/2013\/08\/la-selva.jpg\")
<\/a>
\n?dataPageSize=48<\/strong>&pfmin=20130813<\/strong>&pfmax=20130813<\/strong>&est=2<\/strong><\/p>\n
\n
\nFetching the meteo data using Python<\/strong> <\/p>\n\r\nimport urllib\r\nimport urllib2\r\nimport datetime\r\nfrom bs4 import BeautifulSoup\r\nfrom string import replace\r\nimport re\r\nimport time\r\n\r\n#Determine the date I want to fetch. For now fetch data 2 days back to make sure I get 24 hours\r\njungledate = datetime.datetime.now()-datetime.timedelta(days=2)\r\n\r\nlocation = 2 # La Selva Station is location 2\r\n\r\n#Build the URL we need\r\nurl = 'http:\/\/www.ots.ac.cr\/meteoro\/detailed_view.php?dataPageSize=48& \\\r\npfmin='+str(jungledate.year)+str(jungledate.month).zfill(2)+str(jungledate.day).zfill(2)+'& \\\r\npfmax='+str(jungledate.year)+str(jungledate.month).zfill(2)+str(jungledate.day).zfill(2)+ \\\r\n'&est='+str(location)\r\n\r\n# Try to open the web page. On failure, wait for a minute then retry.\r\nopened = 0\r\nwhile opened == 0:\r\n opened=1\r\n try:\r\n response = urllib2.urlopen(url)\r\n except urllib2.URLError:\r\n print \"opening site failed.Waiting 10 seconds\"\r\n time.sleep(60)\r\n opened = 0\r\n\r\nhtml = response.read()\r\n\r\n#Parse the HTML page by using Beautiful Soup\r\nsoup = BeautifulSoup(html)\r\n#get all rows that are the start of a table that has the class of either Row or AltRow\r\nrows = soup.findAll('tr',{ 'class' : [\"Row\" , \"AltRow\"]})\r\n\r\n#open up a local file to store one day of data in\r\nf1=open('.\/meteodata', 'w+')\r\n\r\n#grab all rows and print them into the file (and on screen)\r\nfor tr in rows:\r\n cols = tr.findAll('td')\r\n for td in cols:\r\n text = td.find(text=True)\r\n text = re.sub('\\xa0', '', text)\r\n print >>f1, text,\r\n print text,\r\n print >>f1\r\n print\r\n\r\nf1.close()\r\n<\/pre>\n\r\n08\/14\/2013 21:00 24.83 24.90 24.79 92.90 0.000 0.317 0.308 107.80 7.89 1006.524 [..] \r\n08\/14\/2013 20:30 25.03 25.15 24.84 92.60 0.000 0.446 0.434 99.20 8.07 1006.063 [..]\r\n08\/14\/2013 20:00 25.26 25.45 25.12 91.90 0.000 0.369 0.318 113.30 21.50 1005.797 [..]\r\n08\/14\/2013 19:30 25.49 25.60 25.40 92.20 0.000 0.937 0.905 105.30 14.34 1005.397 [..]\r\n08\/14\/2013 19:00 25.76 25.93 25.58 91.90 0.000 0.692 0.651 106.60 18.67 1004.868 [..]\r\n08\/14\/2013 18:30 26.04 26.26 25.88 91.10 0.000 0.306 0.286 86.30 10.45 1004.469 [..]\r\n(and so on)\r\n<\/pre>\n
\n
\nReading the data back from the file whenever I need it<\/strong><\/p>\n\r\nimport re\r\nimport string\r\n\r\nneedhour = 15\r\nneedminutes = 30\r\nneedstring= str(needhour).zfill(2)+\":\"+str(needminutes).zfill(2)\r\n\r\ndata = open('.\/meteodata')\r\nwith data as meteo:\r\n for line in meteo:\r\n line = string.split(line)\r\n if line[1] == needstring:\r\n AirTemp = float(line[2])\r\n Humidity = float(line[5])\r\n Rain = float(line[6])\r\n WindSpeed = float(line[7])\r\n Pressure = float(line[11])\r\n Light = float(line[12])\r\ndata.close()\r\n\r\nprint \"Airtemp \",AirTemp\r\nprint \"Humidity \",Humidity\r\nprint \"Rain \",Rain\r\nprint \"WindSpeed \",WindSpeed\r\nprint \"Pressure \",Pressure\r\nprint \"Light \",Light\r\n<\/pre>\n
\n
\nWhat it will do<\/strong><\/p>\n![\"I](\"http:\/\/paluweb.nl\/wp-content\/uploads\/2013\/08\/tempchange.png\")
<\/a>