🚀 go-pugleaf

>>>>> "Terrance" == Terrance Richard Boyes <tez@pierrot.co.uk> wrote:

   Terrance> I'm looking for an efficient way to check for duplicate
   Terrance> Message-IDs in a large NewsHistory file against the
   Terrance> output of NEWNEWS (or XOVER after extracting the
   Terrance> Message-ID).

[...]

   Terrance> Any thoughts on the best way to do it in C ?

I can tell you the way *I* did it in C, which is probably far from the
best way...

Essentially just set up an ordered list and insert message-id's in it
as they come out of NEWNEWS or XHDR. You could experiment with various
ways of quickly finding the right place to insert (I don't think I
bothered with much more than a binary chop) and if it's already there
drop it.

It doesn't seem that slow in practice (at least not compared with the
speed at which the message-id's come in anyway), although I haven't
done any benchmarking with large numbers of message-ids (only a few
hundred or so at a time). Some who actually knows about efficiency of
algorithms is welcome to tell me this sucks :-)

Of course another way that might be easier, depending on what you're
going to do with the articles, is not bother with deduping at all but
do an explicit CHECK on the message-id before downloading it, and then
download and throw it at the local newsserver with TAKETHIS if the
initial reply code was positive.

There's also the various database libraries that I guess would work
for this purpose.

   Terrance> NB a pointer to some source what I want will do just as
   Terrance> well as a detailed answer ;)

Presumably suck has to do this (at least it claims to explicitly
dedupe last time I used it). Probably worth a look anyway.

http://home.att.net/~bobyetman/index.html

Stefan