Article View: chi.internet
Article #121Re: Parse wunderground.com
From: Mark Anderson
Date: Mon, 26 Nov 2012 03:18
Date: Mon, 26 Nov 2012 03:18
33 lines
3261 bytes
3261 bytes
In article presence@MUNGEpanix.com says... > wow. > > Why not just add 10 characters of awk to my version of whatever the hell > you're trying to do up there. > > Anybody that uses perl isn't a programmer. > > wget -qO- "http://rss.wunderground.com/auto/rss_full/IL/Chicago.xml?units=english" | grep Temp That's interesting. I couldn't find an xml or rss version like this for my neighborhood. I hit a specific weather station that's linked to my zip code which is some cgi script on their side. For what I'm using this data for I do need as close as local as possible so even Ohare or Midway data would be off -- sometimes by a lot. Remember the so called 20 inches of snow? Normally I prefer to awk but html with all its backslashes and >< and general fuckery makes choosing a field separator rather challenging. Then you have to clean with substr which can be a PITA. The HTML::TreeBuilder breaks up html code into teeny little objects that can be sliced and diced in a civilized manner. I started out to just extract current temp, timestamp it, and log it. Then a what the heck moment came because the Bears were kicking ass and it became trivial to get a bunch of other data too. Wunderground hid key attributes in both div and span statements so the script had to slice up html on both tags. They also do funny characters like   and ° which has to be stripped in order not to corrupt a text file with non ASCII crap. The resulting data which is CSV will probably get piped through awk one day.
Technical Details
Message-ID:
<MPG.2b1ca9a129fca2e7989a66@news.speakeasy.net>Path:
arc1-ca.pugleaf.net!s9ni11425pbb.0!nntp.google.com!npeer03.iad.highwinds-media.com!news.highwinds-media.com!feed-me.highwinds-media.com!nx02.iad01.newshosting.com!newshosting.com!216.196.98.146.MISMATCH!border3.nntp.dca.giganews.com!Xl.tags.giganews.com!border1.nntp.dca.giganews.com!nntp.giganews.com!local2.nntp.dca.giganews.com!nntp.megapath.net!news.megapath.net.POSTED!not-for-mailReferences:
<MPG.2b1c803350640b6b989a65@news.speakeasy.net> <k8uens$84b$1@reader1.panix.com>Show Full Headers
Complete Article Headers
X-Google-Thread: 10d5fa,f007932fe2b14462
X-Google-NewGroupId: yes
X-Google-Attributes: gid837b0c45a0,domainid0,public,usenet
X-Google-Language: ENGLISH,ASCII-7-bit
Received: by 10.66.79.40 with SMTP id g8mr3115443pax.45.1353899933511;
Sun, 25 Nov 2012 19:18:53 -0800 (PST)
Path: arc1-ca.pugleaf.net!s9ni11425pbb.0!nntp.google.com!npeer03.iad.highwinds-media.com!news.highwinds-media.com!feed-me.highwinds-media.com!nx02.iad01.newshosting.com!newshosting.com!216.196.98.146.MISMATCH!border3.nntp.dca.giganews.com!Xl.tags.giganews.com!border1.nntp.dca.giganews.com!nntp.giganews.com!local2.nntp.dca.giganews.com!nntp.megapath.net!news.megapath.net.POSTED!not-for-mail
NNTP-Posting-Date: Sun, 25 Nov 2012 21:18:53 -0600
From: Mark Anderson <mea@nospambradnylion.com>
Newsgroups: chi.internet
Subject: Re: Parse wunderground.com
Date: Sun, 25 Nov 2012 22:18:50 -0500
Message-ID: <MPG.2b1ca9a129fca2e7989a66@news.speakeasy.net>
References: <MPG.2b1c803350640b6b989a65@news.speakeasy.net> <k8uens$84b$1@reader1.panix.com>
MIME-Version: 1.0
User-Agent: MicroPlanet-Gravity/3.0.4
Lines: 36
X-Usenet-Provider: http://www.giganews.com
NNTP-Posting-Host: 66.92.134.90
X-Trace: sv3-ql0cJB8sHzbKgTVAiqfdxTtIyWWYhspzwL1tJvL9KAUDw6Y+wSAVExOOQDLEmCi84IUHh5KCj6ymJEc!XTG1X7eQshrl+LZmgJbuCPnI+UsXFzC8DmrNvBe6yHoz50Q5GcnBk0UNH5Y1FlhPQW4KB2riDmNc!tqk3mJYhMa1+jjc2g6VoL8YwI8uZ
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
X-Postfilter: 1.3.40
Bytes: 2696
X-Original-Bytes: 2635
X-Received-Bytes: 2843
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit