Re: HTTP directory listing

Article View: borland.public.cppbuilder.nativeapi

Article #12927

Back to borland.public.cppbuilder.nativeapi Previous Next

#12927

From: Remy Lebeau (TeamB)
Date: Thu, 06 Nov 2003 18:51

14 lines
1756 bytes

"Fernando" <frr@easyjob.net> wrote in message
news:ae7kqvk22psl3sbr25qqg28kftob0aie09@4ax.com...

> Then how does google or other robots fetch the files to index?

Search engines use "spiders" or "bots", essentially automated programs that
download a given page, parse out the links, download each page linked to,
parse the links, etc. on and on, cataloging each page that is downloaded and
then re-downloading them periodically checking for updates and new/outdated
links.


Gambit

Technical Details

Message-ID: <3faa9772$1@newsgroups.borland.com>

Path:

arc1-ca.pugleaf.net!archiver1.google.com!news2.google.com!fu-berlin.de!in.100proofnews.com!in.100proofnews.com!nntp-relay.ihug.net!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mail

References:

<3fa2a041@newsgroups.borland.com> <3fa2b389@newsgroups.borland.com> <3fa2b5e1$3@newsgroups.borland.com> <3fa2eb96$2@newsgroups.borland.com> <3fa2f69b$1@newsgroups.borland.com> <3fa2fd5c@newsgroups.borland.com> <3fa36b7d$2@newsgroups.borland.com> <3fa3a088@newsgroups.borland.com> <3faa1444$1@newsgroups.borland.com> <ae7kqvk22psl3sbr25qqg28kftob0aie09@4ax.com>

Show Full Headers

Complete Article Headers

X-Google-Language: ENGLISH,ASCII-7-bit
X-Google-Thread: 10cc67,af8b744a5697d84d
X-Google-Attributes: gid10cc67,public
X-Google-ArrivalTime: 2003-11-06 10:50:15 PST
From: "Remy Lebeau \(TeamB\)" <gambit47.no.spam@no.spam.yahoo.com>
Newsgroups: borland.public.cppbuilder.nativeapi
References: <3fa2a041@newsgroups.borland.com> <3fa2b389@newsgroups.borland.com> <3fa2b5e1$3@newsgroups.borland.com> <3fa2eb96$2@newsgroups.borland.com> <3fa2f69b$1@newsgroups.borland.com> <3fa2fd5c@newsgroups.borland.com> <3fa36b7d$2@newsgroups.borland.com> <3fa3a088@newsgroups.borland.com> <3faa1444$1@newsgroups.borland.com> <ae7kqvk22psl3sbr25qqg28kftob0aie09@4ax.com>
Subject: Re: HTTP directory listing
Date: Thu, 6 Nov 2003 10:51:26 -0800
Lines: 16
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2800.1158
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
NNTP-Posting-Host: 67.22.137.149
Message-ID: <3faa9772$1@newsgroups.borland.com>
X-Trace: newsgroups.borland.com 1068144498 67.22.137.149 (6 Nov 2003 10:48:18 -0700)
Path: arc1-ca.pugleaf.net!archiver1.google.com!news2.google.com!fu-berlin.de!in.100proofnews.com!in.100proofnews.com!nntp-relay.ihug.net!newsfeeds.ihug.co.nz!ihug.co.nz!newsgroups.borland.com!not-for-mail

Reply to Parent Message