Re: IP address list or domain name list for new search engine

Article View: comp.protocols.tcp-ip

Article #15577

Back to comp.protocols.tcp-ip Previous Next

#15577

From: Thor Kottelin
Date: Tue, 28 Dec 1999 00:00

26 lines
1557 bytes


Doug Taylor wrote:
> 
> I would like to develop a new search engine but am somewhat daunted by
> the thought of using the 'brute-force' method of checking all 4 billion
> possible IP addresses for web servers from which to collect data.

This wouldn't even be enough (keywords: Host header).

> Is
> there any way of getting a complete list of IP addresses or registered
> domain names to start my search? I have made the request of Network
> Solutions and am awaiting their response.

Network Solutions registers domain names in a few TLDs; they do not
register web sites.

Why not simply spider instead?

Thor

-- 
My PGP public key is available on key servers - please use it!

Powerful, automated web hosting: <http://www.virtualis.com/vr/tkotteli/>

Technical Details

Message-ID: <386912DD.5A96A1AB@anta.net>

Path: arc1-ca.pugleaf.net!unknown!not-for-mail

References: <3868E3B2.B54A0687@home.com>

Show Full Headers

Complete Article Headers

X-Google-Language: ENGLISH,ASCII-7-bit
X-Google-Thread: fe4ef,769aaf0d373fd10,start
X-Google-Attributes: gidfe4ef,public
X-Google-Thread: 10bb05,769aaf0d373fd10,start
X-Google-Attributes: gid10bb05,public
X-Google-Thread: f9325,769aaf0d373fd10,start
X-Google-Attributes: gidf9325,public
From: Thor Kottelin <thor@anta.net>
Subject: Re: IP address list or domain name list for new search engine
Date: 1999/12/28
Message-ID: <386912DD.5A96A1AB@anta.net>
X-Deja-AN: 565758354
Content-Transfer-Encoding: 7bit
References: <3868E3B2.B54A0687@home.com>
X-Accept-Language: en,sv,fi
Content-Type: text/plain; charset=us-ascii
Organization:  
Mime-Version: 1.0
Newsgroups: comp.protocols.tcp-ip,comp.protocols.tcp-ip.domains,demon.ip.developers

Reply to Parent Message