🚀 go-pugleaf

On Thursday, in article <dFMTwNj030n@sktb.demon.co.uk>
     pla@sktb.demon.co.uk "Paul L. Allen" wrote:

> In article <876390153snz@isis.demon.co.uk>
>     ian@isis.demon.co.uk (Ian Smith) writes:
>  
> > Can anybody enlighten me about the command format mainstream 
> > browsers use to call a proxy server ?
> 
> The details of HTTP 1.1 are in RFC 2068 and HTTP 1.0 in RFC 1945.  I
> don't think HTTP 0.9 ever made it out of an IETF draft although it was
> used for a long time.  Servers tend to follow what the specs say more
> closely than browsers do, though servers also accepted verious pieces
> of crap sent by broken browsers (like Netscape and IE).  If you do what
> the spec says, a server will accept it.  Well, with one exception.  Most
> servers still do not accept HTTP 0.9 requests although both 1.0 and 1.1
> say this is a mandatory requirement.

I did look at 1.1, but it scared the life out of me. 1.0 looked much 
cruder and so a lot more my style :) 1.1 is for next year...

> > I'm (reluctantly) putting a proxy server option into John Washington's 
> > KA9Q HTTP client and the command format I'm testing is:
> > 
> > GET proxy_server:port full_URL HTTP/1.0
> 
> Ummmm, you're not sending that, are you?  It should be GET full_URL HTTP/1.0.

No, you're right. I've awarded myself a smack 'ead !  I'd done the code 
a while ago and just returned to it. I should have reread what I'd 
actually written and not gone from memory :(

What actually happens is that I log into proxy_server:port and send

      GET full_URL HTTP/1.0

> > which appears to work and so far as I can see is all that's required 
> > with HTTP/1.0 (the HTTP/1.0 RFC is a bit spare with the proxy info),
> 
> Well, it's all that is required but you should strongly consider sending
> 
>   GET full_URL HTTP/1.0
>   Host: fqdn_extracted_from_URL
>   <CR>
> 
> Note that you need a blank line after HTTP 1.0 or greater requests to
> indicate that all headers that you want to supply have been supplied (the
> 1.0 spec didn't make that clear enough, can't remember if the 1.1 spec
> does).  Otherwise it's going to break when Demon switch the homepages
> service to relying on the Host header being present rather than using a
> load of static IP addresses.

I've been sending the Host: line for a while since it's one of the 
things required these days if you not to be refused with a snotty 
message that your browser isn't up to it, which is probably what you're 
alluding to. I've also got a user agent line (which can be spoofed) as 
another line of attack on the refusal thing although the UA tends to 
change some of what you get rather than fix a refusal. However, it also 
registers as 0.00% browser usage on Browserwatch :)

> You also need to support the POST method for transmitting long form data.

Hmmm, I have looked at that, but it looks messy and I've yet to find a 
need great enough for the effort. Almost the only things I ever send are 
search terms to search engines and they don't need POST. I think 
if-modified-since will come before POST.

> > but  when I did a trace on what Netscape 2.0 sent it didn't appear to
> > include the proxy name
> 
> Good.  Where did you get the idea that it should?

From the murky and unreliable depths of my memory, I'm afraid :(

Ian.
-- 

  |  Ian Smith             |  "The Moving Finger writes;
  |  ian@isis.demon.co.uk  |   and, having writ, Moves on."

On Friday, in article <df3uEJj030n@sktb.demon.co.uk>
     pla@sktb.demon.co.uk "Paul L. Allen" wrote:

> In article <876506961snz@isis.demon.co.uk>
>     ian@isis.demon.co.uk (Ian Smith) writes:

> It's not that much worse than 1.0.  A load of new things, but no radical
> changes to the philosophy of sending and accepting requests.

You're obviously less easily intimidated than I am :) I found the 1.0  
RFC somewhat impenetrable and 1.1 is more than twice as long. I got the 
impression that under 1.1 things are a lot more transactional. With 1.0 
you pretty much say what you want and that's it, but 1.1 looked as if it 
had a lot more to-ing and fro-ing. Maybe I'm wrong; I just skimmed it, 
shuddered and filed it for future reference.

[...]

> Yeah.  At the moment all the homepages sites have their own IP address but
> they all map to the same server (or cluster of servers).  They can figure
> out which site you actually wanted by the IP address the request is sent
> to.  Soon that will stop.  It will be the same IP address for all, and
> you need the host header to work out which site you were after.

I don't look at much on Demon (apart from testing :), but it's nice to 
know that I won't even notice the transition.
 
> However, thinking about it, the proxy should manage to work that out for
> itself - you're sending it the full URL.

Hmmm, I wonder why it doesn't.

[...]

> Yeah, I hate that.  I was digging around the clara.net pages so I could
> advise a friend on how to do something on the clara webspace.  After
> battling through loads of frames, each of which URLs with no text but
> just an image to tell you what it was about), I hit their web central which
> said I didn't have a good enough browser so I could sod off.  Nothing like
> making important pages accessible to all, eh?  Even those using Lynx.

Er, I don't see images or frames :) Online, if I view pages at all, it's 
only to follow a link and the display is cut down to the minimum which 
will facilitate that. Crude, huh ? The way I see it, online is strictly 
for getting the stuff, offline is for looking at it and, mostly, 
discarding it. I've only once come across one form I felt like filling 
in and there's always more than one way to skin a cat...

I still do a lot of page/file collection with a script and it's a real 
bummer to find a collection of rejections waiting for you instead of the 
expected goodies :(

> That's nice.  If you dig around <URL:http://www.delorie.com/web> for his
> web stuff you'll find a gigantic text file showing what browsers have hit
> his site.  Only 3 of the 6 for my platform show up there, and in a limited
> number of versions at that.  The file is *huge*.  There's a lot of browsers
> out there, even if idiots like clara.net decide everyone is running the
> latest Netscape or IE.

I'm not sure how many DOS browsers there are, but I doubt that it's as 
many as six and I don't class what I'm using online as a browser. There 
are several browsers with configurable UA tags, so just because you see 
a different tag may not mean a different browser or even, as in my case, 
a browser at all. For a while I claimed to be using Netscape 6, but, 
much to my surprise, I got mail begging to know where I'd got it :)

[ good POST griff]

Hmmm, it still looks messy to me, but maybe my mess threshold is lower 
than yours. However, the way you describe it is a lot more straight 
forward than I remember the RFC being so I'll keep your info and if/when 
I go back to look at POST I'll reread it. Thanks.

> > Almost the only things I ever send are search terms to search engines and
> > they don't need POST.
> 
> Some do because of the limitations on length of GET query strings.

Ah, well, when the need arises...                        
I haven't hit any limitation yet and I've used some pretty long strings.

> > I think if-modified-since will come before POST.
> 
> HEAD is also useful.

Yup, that's already in.

Ian.
-- 

  |  Ian Smith             |  "The Moving Finger writes;
  |  ian@isis.demon.co.uk  |   and, having writ, Moves on."

In article <876559733snz@isis.demon.co.uk>
    ian@isis.demon.co.uk (Ian Smith) writes:

> On Friday, in article <df3uEJj030n@sktb.demon.co.uk>
>      pla@sktb.demon.co.uk "Paul L. Allen" wrote:
> 
> > In article <876506961snz@isis.demon.co.uk>
> >     ian@isis.demon.co.uk (Ian Smith) writes:
> 
> > It's not that much worse than 1.0.  A load of new things, but no radical
> > changes to the philosophy of sending and accepting requests.
> 
> You're obviously less easily intimidated than I am :)

I didn't say I understood it all. :-)

> [...]
> 
> > Yeah.  At the moment all the homepages sites have their own IP address but
> > they all map to the same server (or cluster of servers).  They can figure
> > out which site you actually wanted by the IP address the request is sent
> > to.  Soon that will stop.  It will be the same IP address for all, and
> > you need the host header to work out which site you were after.
> 
> I don't look at much on Demon (apart from testing :), but it's nice to 
> know that I won't even notice the transition.

You won't as long as you send a host header.

> > However, thinking about it, the proxy should manage to work that out for
> > itself - you're sending it the full URL.
> 
> Hmmm, I wonder why it doesn't.

Ummm, I just tried it and it does.  Started a chat window on port 80,
went to the proxy, told it GET http://sktb.demon.co.uk/ HTTP/1.0 and
the host header appeared.

> > Yeah, I hate that.  I was digging around the clara.net pages so I could
> > advise a friend on how to do something on the clara webspace.  After
> > battling through loads of frames, each of which URLs with no text but
> > just an image to tell you what it was about), I hit their web central which
> > said I didn't have a good enough browser so I could sod off.  Nothing like
> > making important pages accessible to all, eh?  Even those using Lynx.
> 
> Er, I don't see images or frames :)

Neither do I, most times.  Telnet and spool to a file is how I look at
most pages these days.  If I can't spot what I'm looking for and how to
get it easily I figure it's another fluff site which concentrates on
appearances rather than content.  The times I've soldiered on have not
thrown up any exceptions to this rule.

> I've only once come across one form I felt like filling in and there's
> always more than one way to skin a cat...

Yeah, but it's a pain.

> I still do a lot of page/file collection with a script and it's a real 
> bummer to find a collection of rejections waiting for you instead of the 
> expected goodies :(

Nope.  Because if the page isn't accessible easily it's unlikely to have
any goodies on it.

> > That's nice.  If you dig around <URL:http://www.delorie.com/web> for his
> > web stuff you'll find a gigantic text file showing what browsers have hit
> > his site.  Only 3 of the 6 for my platform show up there, and in a limited
> > number of versions at that.  The file is *huge*.  There's a lot of browsers
> > out there, even if idiots like clara.net decide everyone is running the
> > latest Netscape or IE.
> 
> I'm not sure how many DOS browsers there are, but I doubt that it's as 
> many as six and I don't class what I'm using online as a browser.

If it lets you view web pages on-line it's a browser.  Browsers are a
sub-class of user agents.

> There  are several browsers with configurable UA tags, so just because you
> see a different tag may not mean a different browser or even, as in my case, 
> a browser at all.

Yeah, a lot of hits for Netscape will be from browsers pretending to be
Netscape.  But Netscape users don't configure their browser to pretend to
be something else.

> For a while I claimed to be using Netscape 6, but, much to my surprise, I
> got mail begging to know where I'd got it :)

:-)  Whilst it makes sense to pretend to be Netscape for those broken pages
that insist on it, pretending to be Foo version bar does not.  That list
of browsers and user agents may have fake Netscape entries and even fake IE
entries, but the rest are real.

> > > Almost the only things I ever send are search terms to search engines and
> > > they don't need POST.
> > 
> > Some do because of the limitations on length of GET query strings.
> 
> Ah, well, when the need arises...                        
> I haven't hit any limitation yet and I've used some pretty long strings.

Servers are often limited to 1024 characters in the entire GET request.
Some browsers may have even lower limits on the length of a URL they can
send.

--Paul