🚀 go-pugleaf

Peter Caffin wrote:
> 
> Sharkie <sharkie@outlook.net> wrote:
> > I'm looking for a way, or a Unix utility to check
> > URL links. I have a text file with lots of links
> > and want to check if any of the links are broken.
> 
> Actually, this is a bit of a "me too" posting. Hopefully, whatever there
> is will be able to communicate with proxy servers.
> 
> > What's a good way of doing this?
> > Are there any programs/utilities that would do it?
> 
> If there's nothing about, I could probably start writing something, but,
> it'd be silly to reinvent the wheel if there's something already out
> there..
> 
> [x-posted to aus.computers.linux]
> 
> --:     _           _    _ _
>  _oo__ |_|_ |__  _ |  _ |_|_o _  pc at it dot net dot a u |
> //`'\_ | (/_|(/_|  |_(_|| | || |            it.net.au/~pc |
> /

Look at 'lynx', the text based www browser.  You can drive this from a
script with the '-dump' option (I think that's what it is) along with
your URL.  It will dump the HTML, which you can redirect to a file.

  i.e. lynx -dump http://www.linux.org/ > test_url.txt

You may want to use the '-source' instead of '-dump'.

  i.e. lynx -source http://www.linux.org/ > test_url.html

Lynx can use an environment varable 'http_proxy' set to point to a proxy
too.

The look at this file and check for error messages like "page not
found",
or "no server found".

It's been a long time since I did this, and the '-dump' may be wrong, 
but it's something to get you started.

One thing I remember; the script I ran would NOT run from cron!.  The
problem
was lynx needed to find the terminal type, and in a cron job there was
no
terminal type.

Brian
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Brian Miller                                 Telstra
CDN Product Group                            30/242 Exhibition Street
ITG Communication Network Platforms          Melbourne, VIC 3000
bmiller@telstra.com.au                       Australia
Tel: +61-3-9632-3883                         FAX: +61-3-9632-3884
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=