🚀 go-pugleaf

In article <grahamybp19yqw.fsf@barnowl.demon.co.uk> you wrote:

: How else is it supposed to do it (using article numbers rather than
: NEWNEWS)? Taking it a bit further, if the previously highest article
: number read in a group is X, and on connecting the GROUP command
: returns that the highest article is now Y, I think it is reasonable to
: request the headers/bodies of all articles between X & Y.

If you want the full answer read the document "Request for
Comments: 977" by Brian Kantor and Phil Lapsley. It is the
specification document for the News Server Transfer Protocol
specification and is available at
  ftp://sunsite.doc.ic.ac.uk:/rfc/rfcrfc977.txt.gz
among many other sites.


Which is buggy - the server or news reader?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  The system demon use is within the bounds of the specification and
  Free Agent is taking short cuts which may, by chance, have worked
  on many previous News Servers but may not continue to do so in the
  future.


Why do the problems arise?
~~~~~~~~~~~~~~~~~~~~~~~~~~
The problems arise from the GROUP command which take a group name and
returns the following:
   211 n f l s group selected
where n is an estimate of the of the number of articles in the group
      f is the lowest article number in the group
      l is the highest article number in the group
      s is the group name

There is no requirement that all numbers between f and l correspond
to articles in the group and it is unreasonable to assume so.

Since the document was completed over 10 years ago, numerous servers
and clients have been developed. In UNIX clients (which formed the
vast majority in the early years), in became standard practice to
keep a list of all the article that had already been seen to be
listed in a file ~/.newsrc . This contained lists of articles in
comma and hyphenated form eg.

demon.service: 1-1277,1279-1281,1284-1290

Thus indicating that no articles were found with article numbers 1278,
1282 and 1283. The next time the client connected, it would check for
these articles and others in the range 1291 to l.

Free Agent does not appear to check for these earlier article numbers
and thus may miss these articles if the server later fills in these
numbers.


Why do only demon have the problem?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Article numbers are designated locally so the same article may have
a different article number at different news servers - this will
cause most newsreaders that rely on article numbers problems. 

Most ISPs are growing at a faster rate than the power of the
computers they use as servers and thus reach the stage where they
need to provide multiple machines to provide each service (demon
currently have 4 customer facing mail stores, for example). Existing
News Server software can not be scaled this easily as the article
number to article pairings and the list of articles available on
each server must remain indentical between news servers.

Thus demon have written their own software to ensure that multiple
servers (currently up to two) remain synchronised. For commercial
reasons, they are keeping the feeding method secret but from the
hints Malcolm Muir has dropped and by checking the missing article
numbers against the articles when they later arrive, the following
model can be deduced:

An internal news server (subsequently called dispatch), receives
articles from a number of newsfeeds and allocates an article
number for each valid article.

When articles are fed to customers facing news server(s), the
article number is specified in the transfer.

Occassionally (presumably when the load is high), a backlog builds
up. When an interruption in the feed occurs, this backlog is
temporarily overlooked while newer news is fed (this news may be
fed later when the load is reduced).

This leave holes in the article numbering on the customer facing
machines which are later filled but faulty news readers fail to
read these "late" articles.

I expect that this is a vast simplification but seems to be an
appropriate model which would explain some the problems. It
explains why the problems have been much worse while the single
remaining server has been so heavily loaded and where the missing
articles come from.

I welcome Malcolm's comments - perhaps a general comment on
accuracy as I don't expect Demon to correct the mistakes :-)


How to avoid missing articles
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Either

* Don't use Free Agent or similar faulty news readers
or
* Set the option "get all headers in selected groups"

Using software which use the "NEWNEWS" method is preferable as it
eliminates these problems and is not considered by Demon to be
anti-social as it is by PIPEX. (PIPEX use a system wide index which
makes search with this command much more inefficient).

This is email is based on a number of reples given last time this
problem arrived and is slowing turning into a FAQ.

All comments welcome,

Steve Clarke