🚀 go-pugleaf

My apologies if you see this message twice ... so far I have not seen
the first posting once :(

In article <4if6q3$37g@monmouth.demon.co.uk>, Steven Clarke
<sclarke@iee.org> writes

>Which is buggy - the server or news reader?
>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>  The system demon use is within the bounds of the specification and
>  Free Agent is taking short cuts which may, by chance, have worked
>  on many previous News Servers but may not continue to do so in the
>  future.

There appear, at first sight, to be two distinct issues -- though they
are undoubtably linked.

First there is consumption of article number space. Several people have
reported (though hard evidence is regrettably rare) that the highest
available article goes up by leaps and bounds. The effect on a news
reading client is that it overestimates the number of articles in the
newsgroup. There are no other bad effects.

Second there is "backfill". In this scenario, articles become available
at a later time after they have been found to be unavailable earlier.
The effect on a news reading client is that if it merely records the
"high water mark" for a newsgroup, it will miss articles altogether.

I agree that the specification of NNTP is so sufficiently vague that
just about anything could be argued about it. The RFC comes from an age
where precision was not over-valued. Also NNTP has been defined a lot
more by its implementations than by the RFC -- I doubt there are many
newsreaders which assume raw RFC977 and nothing more at all.

>Why do the problems arise?
>~~~~~~~~~~~~~~~~~~~~~~~~~~
>The problems arise from the GROUP command which take a group name and
>returns the following:
>   211 n f l s group selected
>where n is an estimate of the of the number of articles in the group
>      f is the lowest article number in the group
>      l is the highest article number in the group
>      s is the group name
>
>There is no requirement that all numbers between f and l correspond
>to articles in the group and it is unreasonable to assume so.

This is correct so far as it goes.

It should be noted that there is a requirement in the RFC that "n" be
roughly correct. This is to deal with the first of the two problems
which I outlined above. As far as I can see the value of "n" from the
Demon machines is invariably l-f+1.

I would agree that it is unreasonable to assume that all the articles
exist. Some may be cancelled at a later time, or perhaps they are found
to be inadmissable into the newsbase after the number is assigned.

Since the RFC hints that n != l-f+1 any wise implementor would assume
that there are systems out there which do not use all possible article
numbers.

>Since the document was completed over 10 years ago, numerous servers
>and clients have been developed. In UNIX clients (which formed the
>vast majority in the early years), in became standard practice to
>keep a list of all the article that had already been seen to be
>listed in a file ~/.newsrc . This contained lists of articles in
>comma and hyphenated form eg.
>
>demon.service: 1-1277,1279-1281,1284-1290
>
>Thus indicating that no articles were found with article numbers 1278,
>1282 and 1283. The next time the client connected, it would check for
>these articles and others in the range 1291 to l.

I would suggest that this file structure exists for a number of other
reasons than for detecting backfill. However, if a news reader works as
described then backfill would not be a problem.

>Free Agent does not appear to check for these earlier article numbers
>and thus may miss these articles if the server later fills in these
>numbers.

This cuts down the command traffic - which is a good design decision for
an offline reader.... especially one which has noted that there is no
guarantee that all article numbers will be used. It also reduces the
necessity to store a large amount of state information. Although PC hard
disks are orders of magnitude bigger than they used to be, it is still a
reasonable decision to have made.

>Why do only demon have the problem?
>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The short answer is that they are running some custom software. I am not
aware that any other system suffers from "backfill". No other server
software I've looked at would eat article numbers on any other basis
than "num++". I would be interested to hear of any counterexamples.

>Article numbers are designated locally so the same article may have
>a different article number at different news servers - this will
>cause most newsreaders that rely on article numbers problems. 

This is true, but not relevant. You could equally observe that NEWNEWS
times are not transportable from one machine to another.

>Most ISPs are growing at a faster rate than the power of the
>computers they use as servers and thus reach the stage where they
>need to provide multiple machines to provide each service (demon
>currently have 4 customer facing mail stores, for example). Existing
>News Server software can not be scaled this easily as the article
>number to article pairings and the list of articles available on
>each server must remain indentical between news servers.

The ISP has three choices.

It can continue to run the news database on more and more powerful
computers (and make it more and more efficient). It can assign users to
particular news machines, so as to spread the load on a statistical
basis, OR it can do what Demon have done and distribute the database and
produce a design which keeps the article numbers in synch so that it
does not matter which machine you connect to.

[snip speculation on design of news system]

>How to avoid missing articles
>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>Either
>
>* Don't use Free Agent or similar faulty news readers
>or
>* Set the option "get all headers in selected groups"

OR ensure that backfill does not occur in the server.

There was a great deal of hard evidence posted of backfill several
months ago. It was then suggested that the effect had been banished, and
until recently it certainly seemed to have gone.

I have seen NO recent hard evidence posted for backfill. I _have_ seen
evidence posted of article number space being consumed rather fast, but
this evidence has been more anecdotal than anything else :(

>Using software which use the "NEWNEWS" method is preferable as it
>eliminates these problems and is not considered by Demon to be
>anti-social as it is by PIPEX. (PIPEX use a system wide index which
>makes search with this command much more inefficient).

There are a number of user-level features which are easier to provide in
an article number based system. The choice between that and NEWNEWS is
not just an issue of convenience to the server software.

Overall, the trend (and not just at Pipex) is very much away from the
use of NEWNEWS. I doubt that you will see many more newsreader clients
using NEWNEWS -- so if Demon's servers do not support article numbers
with the same semantics as everyone else, then this will restrict
Demon's users in what software they can choose from.

>This is email is based on a number of reples given last time this
>problem arrived and is slowing turning into a FAQ.

I look forward to further views.

-- 
richard                      Richard Clayton         T U R N P I K E  Ltd
                                                     tel: +44 1306 747747
"Assembly of Japanese bicycle require great peace of mind" quoted in ZAMM