From my files

Thread View: torfree.general

1 messages

[T] Back to Threads [V] Tree View [G] Group Articles

1 total messages Started by Nabil Mohamed El-Khodari Mon, 12 May 1997 00:00

#465

Author: Nabil Mohamed El-Khodari
Date: Mon, 12 May 1997 00:00

292 lines
13671 bytes

This is an article I have wrote more than a week ago. I sent it to a selected group of TFN users then. Some new information became known since the writing of this. Several changes happened. These are mostly published in other posts (Guess what?). I am pub
lishing this (as was intended) now, as it contains basically the not.torfree.net logs information. The logs has since been removed, as well as ISOLDA :-). Needless to say a member of my list that this has been sent to, has passed the information.

Enjoy.

Nabil

This is a very serious issue. I am posting it here for the whole board and TFN members to decide it. We all know that there is highly objectionable material on the not.torfree.net whether it is something that some may consider child pronography still unde
r http://not.torfree.net/~sasha/isolda/ - which is not my main concern as actually Dmytri has posted it to the torfree newsgroups before - and that there (was/is) business pages on that site.

Beofore January 9th - when I was removed from the board - the sites under not.torfree.net were accessible by 'excite' search - among other search engines - Now they are not.

Does that mean they do not exist any longer? Absolutely not. If you tried to search for not.torfree.net you will see some articles on other sites that contain reference to not.torfree, but not to any site that EXISTS on not.torfree. So what happenned?

A file called robots.txt was most probably added to the root directory. This simple text file actually locks search engine robots from searching either the whole server or some directories as the programmer wishes. For more info about this, please refer
to BYTE, Volume 22 Number 5, May 1997, page 63-64.

For those who are not technically inclined, robots are programs that go to every site on the WWW and catalogue the pages and files there. It does not need an invitation to do that, though there are certain techniques (mostly related to something called me
ta tags in HTML programming) to make your page be listed in say the top 10 sites on a search for a specific keyword. There are also other meta tags that can be inserted within individual pages that would disallow these robots from listing them or any URLs
that may be included in that page (an example is present at the BYTE issue referenced above).

Is that information beyond our sysadm? Not really, another peculiar site on Bruce Becker's server (under the GTS domain) also will not show in search engines. The site is http://hotlips.gts.net/ - what a peculiar choice of subdomain name ;^) - But actual
ly if you type that URL you will at least get into the index.html and see a page. So that technique is obviously known and used by our system administrator.

Now the big question is what is on not.torfree.net? Obviously we do not know. But my trials - as Bob's trials before me - show the usage of the not.torfree.net, which I shall give an abbreviated example here for the usage during last month.

Actually, I have discovered a very small page under http://not.torfree.net/~cmt/

Here what it says:

----------------------- Begin page --------------------------

The Target Communication Homepage

-------------------------------------------------------------------
Welcome to the Target Communications homepage! Taget Communications offers technical writing, documents, communications and consulting at all levels. Whatever your needs... Some text to be edited out by myself.

� Colette Marie Theriault 1996
------------------------ END PAGE ---------------------------------

If this is not a 'business' page, what is? Who the hell is Colette Marie anyway? I believe they will say she is a TFN volunteer ;^)

In the following, please note that notes between brackets [] are mine and not the program's :)

P.S. I have advised Ed. Carriere to ask the lady to remove this page. Hope she did.

--------------------- START NOT.TORFREE.NET SERVER STATISTICS -----

Web Server Statistics for TFN Office Website

______________________________________________________________

Program started at Fri-02-May-1997 17:13 local time.
Analysed requests from Thu-20-Mar-1997 02:14 to Thu-01-May-1997 13:54
(42.5 days).

Total successful requests: 4 295 (366)
Average successful requests per day: 101 (52)
Total successful requests for pages: 1 691 (141)
<snip>
______________________________________________________________

Monthly Report

(Go To: Top: Daily summary: Hourly summary: Domain report: Host
report: Directory report: Request report)

Each unit ( + ) represents 20 requests for pages, or part thereof.
month: pages:
-------- -----
Mar 1997: 775: +++++++++++++++++++++++++++++++++++++++
Apr 1997: 911: ++++++++++++++++++++++++++++++++++++++++++++++
May 1997: 5: +

[note that the usage for April is more than March]
______________________________________________________________

Daily Summary

<snip>
______________________________________________________________

Hourly Summary
<snip>
______________________________________________________________

Domain Report

(Go To: Top: Monthly report: Daily summary: Hourly summary: Host
report: Directory report: Request report)

Printing all domains, sorted by amount of traffic.
#reqs: %bytes: domain
----- ------ ------
1049: 28.99%: .com (Commercial, mainly USA)
1155: 21.30%: .net (Network)
932: 21.25%: [unresolved numerical addresses]
698: 16.63%: .ca (Canada)
<snip>

______________________________________________________________

Host Report
<snip>

______________________________________________________________

Directory Report

(Go To: Top: Monthly report: Daily summary: Hourly summary: Domain
report: Host report: Request report)

Printing all directories, sorted by number of page requests.
Printing directories to depth 1.
#reqs: %bytes: directory
----- ------ ---------
1526: 16.28%: /~dchan/
1598: 65.34%: /~sasha/
429: 7.72%: /stats/
412: 1.50%: [root directory]
78: 0.14%: /volunteer/
47: 8.69%: /~ken/
29: 0.03%: /~cmt/
14: 0.02%: /~ai422/
100: 0.12%: /java/
41: 0.01%: /images/
6: 0.12%: /cgi-bin/
15: 0.02%: /e-bin/

[This clearly shows that pages under the directory ~sasha is receiving most of the hits followed by ~dchan. These subdirectories of course can contain any number of pages under same or different subdirectories.]
______________________________________________________________

Request Report

(Go To: Top: Monthly report: Daily summary: Hourly summary: Domain
report: Host report: Directory report)

Printing all requested files with at least 20 requests, sorted by
number of requests.
#reqs: %bytes: file
----- ------ ----
224: 0.41%: /stats/
152: 0.47%: /
142: 0.62%: /~dchan/
129: 0.36%: /stats/nov.html
116: 1.49%: /~sasha/sasha/
111: 5.08%: /~dchan/country-code.html
94: 1.04%: /~dchan/flowers.html
78: 0.14%: /volunteer/
69: 0.93%: /~dchan/talker.html
68: 0.05%: /~sasha/
62: 1.15%: /~sasha/isolda/
48: 1.37%: /~dchan/search.html
46: 8.69%: /~ken/
30: 0.75%: /~sasha/transforum/
30: 0.18%: /~sasha/sasha/pic.html
29: 0.03%: /~cmt/
26: 0.29%: /~sasha/sasha/art.html
24: 0.07%: /stats/jul.html
23: 0.03%: /gifwizard.html
21: 0.15%: /~dchan/aboutme.html
20: 1.11%: /~sasha/sasha/losev.html

[Here is shown actually what was referred to in the comments on the above report, i.e. the existence of many subdirectories under both sasha and dchan directories]

______________________________________________________________

This analysis was produced by analog1.91beta4/Unix.
Running time: 5 seconds.

---------------------- END NOT.TORFREE.NET usage report -----------

Hopefully that program will not be removed from the server ;( (well, few days later, the program is still there, but the log is gone!!!!)

So what is the point you may ask? The point is that it seems that sysadm. has things to hide on public-owned property. Can you think of one good reason [I said good] for somone to block the robots from catalogueing the files and pages that exist on certai
n servers?

Of course, it will be claimed by some that my logic is not right, as obviously honest users [HTML programmers] will not like that, as they want their pages to be listed with search engines. Of course I have to agree that this is partly true, but what if:

- These pages are URL's to other pages that are adequately advertised.
- These pages are mirrored by domain names owned by the authros [or their customers]

We have to understand that 'commercial sites' will charge great somes of money for 'business' pages, for allowing the programmer to use CGI-BIN, and according to the 'volume' of traffic. Again, it may be that the free bandwidth (T1) as comapared to the ty
pes of connections these author's may have that are the main factors for using such 'free' service. They care about the 'responsiveness' of the site to requests if getting multiple hits simultaneously. This is particularly true in the case of Sasha who is
fond of using highly graphical elements (including some peculiar animated GIF's ;-) in his pages.

In another development, I hate to convey to you that a further change has occurred to the torfree domain registration. This time by adding another server to the registry called torfr-hst?

Following is a copy of that host as registered on the internic:

------------------------------
Whois Search Result

------------------------------------------------------------------------
Query: whois -h rs.internic.net SUM torfree.net

------------------------------------------------------------------------

[No name] (TORFR-HST) TORFREE.NET 199.71.188.18
Toronto Free-Net (TORFREE-DOM) TORFREE.NET

To single out one record, look it up with "!xxx", where xxx is the
handle, shown in parenthesis following the name, which comes first.

The InterNIC Registration Services Host contains ONLY Internet Information
(Networks, ASN's, Domains, and POC's).
Please use the whois server at nic.ddn.mil for MILNET Information.
------------------------------
Whois Search Result

------------------------------------------------------------------------
Query: whois -h rs.internic.net HA F TORFR-HST

------------------------------------------------------------------------

[No name] (TORFR-HST)

Hostname: TORFREE.NET
Address: 199.71.188.18
System: ? running ?

Coordinator:
Hostmaster (HOS342-ORG) hostmaster@TORFREE.NET
+1 416 699 1868

Record last updated on 20-Feb-97.
Database last updated on 3-May-97 06:14:38 EDT.
------------------------------

Whois Search Result

------------------------------------------------------------------------
Query: whois -h rs.internic.net HA F TORFREE-DOM

------------------------------------------------------------------------

Toronto Free-Net (TORFREE-DOM)
c/o Ryerson Polytechical University
350 Victoria St.
Toronto, Ontario, M5B 2K3
Canada

Domain Name: TORFREE.NET

Administrative Contact:
TFN Office (TO5-ORG) office@TORFREE.NET
+1 416 979 9224
Technical Contact, Zone Contact:
Hostmaster (HOS342-ORG) hostmaster@TORFREE.NET
+1 416 699 1868
Billing Contact:
TFN Office (TO5-ORG) office@TORFREE.NET
+1 416 979 9224

Record last updated on 27-Dec-96.
Record created on 27-Jul-94.
Database last updated on 30-Apr-97 06:02:55 EDT.

Domain servers in listed order:

NS.TORFREE.NET 199.71.188.17
NS2.TORFREE.NET 199.71.188.22
NS.GTS.NET 204.138.66.97

---------------------------- END OF OLD REGISTRATION -------------

No further cooments. Please read the more recent info. at trofree.general

--
Nabil M. El-Khdari | Community networking is not an aim in itself,
TFN Member | but a means to an end.. The end being community
TFN Information Provider | revitalization, empowerment and development.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Thread Navigation

This is a paginated view of messages in the thread with full content displayed inline.

Messages are displayed in chronological order, with the original post highlighted in green.

Use pagination controls to navigate through all messages in large threads.

Back to All Threads