Article View: demon.ip.developers
Article #77Re: Newshistory checking
From: Terrance Richard Boyes
Date: Fri, 16 Jul 1999 00:00
Date: Fri, 16 Jul 1999 00:00
29 lines
2688 bytes
2688 bytes
Jim Segrave <jes@nl.demon.net> wrote: [snip] > I did some experiments when this first went into ka9q and found (in > live testing) that using two smaller tables with two different hash > generators worked better than one larger table. You hash the ID two > different ways and only if both get a hit do you search the history > file. I seem to recall (though it's been a *long* time since I looked > at the code), that the Demon customer who added the hashing adopted my > suggestion. Yep, there's a HASH1 and HASH2 value, though there seem to be some problems if both of those overflow (collisions). I couldn't find my refs to how to understand the numbers behind a hashing algorithm. Any good books ? However with a bit of experimentation I upped both the HASH values from ~32K to ~256K, on 32K unique msgids this has dropped the collisions from ~5000 to ~700. It seems fairly efficient memory wise too (to be expected from a DOS prog. I guess ;). Oh and that 1000 checks/second included adding the new ids to the hashtable and writing them to the history file. -- <URL:http://www.pierrot.co.uk/> Team AMIGA Nothing is enough for the man to whom enough is too little: Epicurus
Technical Details
Message-ID:
<19990716.193917.0AA80202.XVaa@pierrot.co.uk>Path:
arc1-ca.pugleaf.net!unknown!not-for-mailReferences:
<19990713.200500.0AA80202.PEaa@pierrot.co.uk> <slrn7oo9bq.j5g.chris@ccserver.keris.org> <abbaryybaqryqrzbapbhx.fevjazn.pminews@sharra.llondel.demon.co.uk> <19990714.210035.0AA80202.TCaa@pierrot.co.uk> <E114k4m-0000SW-00@jes.noc.nl.demon.net>Show Full Headers
Complete Article Headers
X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 10bb05,75d5f9974c0fd91a X-Google-Attributes: gid10bb05,public From: tez@pierrot.co.uk (Terrance Richard Boyes) Subject: Re: Newshistory checking Date: 1999/07/16 Message-ID: <19990716.193917.0AA80202.XVaa@pierrot.co.uk> X-Deja-AN: 501889759 X-PGP-Fingerprint: 20 66 0F E8 85 64 9C 2C 6F 82 6F 19 6C 2F 6A 1D X-Cancel-Lock: TEZT:a7d4b822bea379712adac200f6715689 62f767f5a3b72c1f43b99b13ae3a9afb X-PGP-Keyfile: <URL:http://www.pierrot.co.uk/pgpauth/> References: <19990713.200500.0AA80202.PEaa@pierrot.co.uk> <slrn7oo9bq.j5g.chris@ccserver.keris.org> <abbaryybaqryqrzbapbhx.fevjazn.pminews@sharra.llondel.demon.co.uk> <19990714.210035.0AA80202.TCaa@pierrot.co.uk> <E114k4m-0000SW-00@jes.noc.nl.demon.net> X-PGP-Auth: pub 512/3A14C4B9 1999/04/22 Tez Boyes <tez@pierrot.co.uk> iQBVAwUBN4+YekXush46FMS5AQEZggIAp484asZB5Q2v2i3XmvJlMLDXgxI1CnIw T3YDJxs7AX9sv+ZbaKpbJXLPr8XVqi1qUZa4arQ7c21MB/ICwouZvg== =C2dA X-NaMLServ: $VER: v0.9b <tez@pierrot.co.uk> X-Complaints-To: abuse@demon.net X-Mail2News-Path: news.demon.net!tele-post-20.mail.demon.net!pierrot.demon.co.uk!pierrot.co.uk!tez@pierrot.co.uk X-Trace: mail2news.demon.co.uk 932161460 mail2news:15277 mail2news mail2news.demon.co.uk Organization: ECL.Net User-Agent: tin/pre-1.4-980618 (AMIGA) X-Lan-Trace: news.pierrot.lan 19990716.193917 news:XVaa IDENT tez@ami060.pierrot.lan Newsgroups: demon.ip.developers