🚀 go-pugleaf

ted@nowtsfree.freeserve.co.uk (Ted) writes:

[snipppppety]

> My news-reader (xrn) thinks the last article seen was no. 69232, as
> recorded in my ~/.newsrc, so its shown no new articles to me for the last
> day or two.

> I presume leafnode skips a news-group because freeserve offers it crazy
> article numbers. 

Not quite. Leafnode generally puts up a warning about "upstream servers
switched?" when it detects the maximum article # in the parent is suddenly
less than its own, but as it works by Message-ID instead, it seems to work
anyway. (It just drops down to "last initial_fetch combined with max_fetch"
and gets, in my case, no more'n 250 or 150 articles.)

> Why does this happen?  Is it my setup, or freeserve's news-servers?

Probably your end of things. I think you should've had nodesc=1 as a normal
rule - leafnode isn't particularly happy with group descriptions and they
just slow the all-newsgroup download down. Something I snipped above
pointed at nodesc, anyway.

> Is there a fix I should know about?

nodesc=1 and fetchnews -fvvvv once more for good measure, should fix
it. Oh, and remove the ranges in ~/.newsrc, so it just looks like
	uk.comp.os.linux:
with no numbers - you'll have to do a selective catch-up in xrn but that
way you can avoid reading too mcuh, anyway.

~Tim
-- 
| Geek Code: GCS dpu s-:+ a-- C++++ UBLUAVHSC++++ P+++ L++ E--- W+++(--) N++ 
| w--- O- M-- V-- PS PGP++ t--- X+(-) b D+ G e++(*) h++(*) r--- y-	     
| The sun is melting over the hills,         | http://www.glutinous.custard.org
| All our roads are waiting / To be revealed | piglet@glutinous.custard.org

[posted & mailed, since this concerns issues with news collection...]

In article <2000Mar4.032757@marimba>, Ted 
<ted@nowtsfree.freeserve.co.uk> wrote:

>I'm having problems with article numbering in news-groups served
>up by freeserve to leafnode-1.9.10. Things started to go wrong on
>Jan 24th 2000, when freeserve's news-servers seemed to be down.

I use Freeserve's news only as a fill-in these days so can't comment on
what happened at that time: was there any announcement about the problem
or any side-effects which might have occurred?

>When I eventually got hold of a news-server, leafnode didn't
>like anything on offer and skipped all news-groups. Later, it
>downloaded some news, but restarted the article numbers from 1.

For the avoidance of doubt: was this Freeserve's news service or had you
been trying elsewhere in the meantime?

>Since then there's been a few more hiccoughs and articles have
>sequenced into two separate series within the same news-groups.

There are some issues with any news fetching mechanism which relies on
a server's article numbers to identify "new" news.

Currently in uk.comp.os.linux the machine which announced itself as
newsg1.svr.pol.co.uk offers articles between 67685 and 71837.  All the
other machines which currently answer to the name "news.freeserve.net"
appear to be synchronised.  Whether this was the case at the time you
had problems I obviously can't tell.

>I've just done a fetchnews -f and found the same problems with
>article numbering in /var/spool/news/leaf.node/groupinfo.

>For example:
>/var/spool/news/leaf.node/news.freeserve.co.uk dated 03:23 04 Mar 2000
>has the entry
>    uk.comp.os.linux 71815

That's consistent with what I'm finding.

>But:
>/var/spool/news/leaf.node/groupinfo downloaded at 03:23 04 Mar 2000
>has the entry
>    uk.comp.os.linux 1709 1 0 Discussion of the Linux operating system

This looks like the article numbers had been reset at some point, or
things were perhaps in a state of flux: the three numbers should be
number of articles, minimum and maximum.  Saying there are 1709 articles
with a maximum of zero points to something strange happening.

>Here are some of the logged messages from leafnode for the ucol
>news-group: (I've added extra newlines to this)

>Jan 20 02:22:35 marimba fetchnews[22177]:
>	uk.comp.os.linux: 145 articles fetched (to 65075), 0 killed

That looks reasonably consistent with the article numbers today.

>Jan 24 04:03:22 marimba fetchnews[32722]:
>	skip uk.comp.os.linux (948686599): 607953, 1490065

This *appears* from the leafnode sources to be a symptom of unexpected
results for the group in question.

>Jan 24 04:33:54 marimba fetchnews[14]:
>	uk.comp.os.linux: 473 articles fetched (to 474), 0 killed

And this one seems to suggest that the article counts at Freeserve had
been reset.

>Jan 26 02:10:07 marimba fetchnews[4248]:
>	uk.comp.os.linux: 285 articles fetched (to 65360), 0 killed

but then restored to the previous sequence.

>... and again ...

Yes, it certainly looks like the same sort of thing happened towards the
end of February.  My suspicion is that one of the servers wasn't
synchronised with the rest for some reason.  That it seems to be
happening at an obscure time of day makes me wonder if it's a side
effect of the expire process within INN - Freeserve apparently uses
v1.7.2 so it may be this is a known issue.  I'll check elsewhere.

>My news-reader (xrn) thinks the last article seen was no. 69232, as
>recorded in my ~/.newsrc, so its shown no new articles to me for
>the last day or two.

I can't see that it has any way to cope with article counts which get
recycled (short of you editing the .newsrc to say something like
uk.comp.os.linux: 4000-69232 and I'm not sure whether this would work).
However, I think you'll find that the article numbers used within your
system aren't synchronised with those at Freeserve anyway so you
probably aren't missing any articles.  I'll cc this by mail though...

>I presume leafnode skips a news-group because freeserve offers
>it crazy article numbers. Why does this happen?

This remains to be seen, but yes leafnode/fetch makes assumptions that
the article numbers will continue to increment.  The -f option gets
round this so it's not a problem.

>Is it my setup, or freeserve's news-servers?

Taken at face value it's a problem with news.freeserve.net BUT it's
actually unwise to rely on article numbers being unique.  As it's not
the only thing which leafnode uses to identify new/duplicate articles it
probably doesn't matter that much.

>Is there a fix I should know about?

About the only thing I can suggest - since many news fetching mechanisms
use the article counts within a group to a greater or lesser extent - is
to switch to NEWNEWS.  This would, however, mean abandonning leafnode in
favour of a "full size" news server and installing additional things to
do the collection.  It's also just possible that the same sort of
oddness would mess up the operation of NEWNEWS.

I think an examination of INN-1.7.2 is the next stage, unless any
comment from Freeserve is forthcoming.

Dan

-- 
Dan Glover (dan@dangl.demon.co.uk)
Today's Excuse:
  SCSI Chain overterminated

In article <FqwHH7.A4s@dangl.demon.co.uk>,
Dan Glover  <dan@dangl.demon.co.uk> wrote:
>I think an examination of INN-1.7.2 is the next stage, unless any
>comment from Freeserve is forthcoming.

It may be helpful to mention that I'm running INN 1.7.4 with Slurp +
y2k patches, taking my feed from Freeserve, and I experienced no problems
or missing messages in any of the groups I take.  NEWNEWS is fast on
Freeserve too.

Nick
-- 
http://www.leverton.org/                      ... So express yourself ...

In article <89s3rf$fvr$1@linux.warren.demon.co.uk>, Nick Leverton 
<leveret@warren.demon.co.uk> wrote:
>In article <FqwHH7.A4s@dangl.demon.co.uk>,
>Dan Glover  <dan@dangl.demon.co.uk> wrote:

>>I think an examination of INN-1.7.2 is the next stage, unless any
>>comment from Freeserve is forthcoming.

My first pass at this hasn't borne fruit - I was hoping for something
which said: "while articles are being expired the response to a GROUP
command might be inconsistent".  If it *is* possible then I'd expect it
to be a transient condition, hopefully only affecting an individual
group for a very short time.  I don't know my way around INN well enough
to spot this in the source without spending a long time.

Re-reading the logs I see one of the cases of "low article number
syndrome" occurred during the evening so I'm more inclined towards the
theory that there's another news machine which occasionally pops up and
is out of sync with the others.  Obviously some comment by POL staff
about the architecture would help in establishing whether this is
actually a possibility.

>It may be helpful to mention that I'm running INN 1.7.4 with Slurp +
>y2k patches, taking my feed from Freeserve, and I experienced no problems
>or missing messages in any of the groups I take.  NEWNEWS is fast on
>Freeserve too.

Indeed. I note that there doesn't seem to be any suggestion to remove
NEWNEWS from the draft NNTP son-of-rfc977.  If it ever becomes a
standard we can presumably berate all the news admins who have chosen
not to support it...

I see that you've been doing business with that nice Mr Gradwell: I'd
better sort something soon as well.

Dan

-- 
Dan Glover (dan@dangl.demon.co.uk)
Today's Excuse:
  Dew on the telephone lines.

Dan Glover <dan@dangl.demon.co.uk> writes:
> <ted@nowtsfree.freeserve.co.uk> wrote:
>
> >I'm having problems with article numbering in news-groups served
> >up by freeserve to leafnode-1.9.10. Things started to go wrong on
> >Jan 24th 2000, when freeserve's news-servers seemed to be down.
>
> I use Freeserve's news only as a fill-in these days so can't comment on
> what happened at that time: was there any announcement about the problem
> or any side-effects which might have occurred?

There was an announcement about expansion of email storage capacity on
Fri 21st Jan. I couldn't reach their mail servers early am Jan 24th either.
I guess they were having problems that night.

> >When I eventually got hold of a news-server, leafnode didn't
> >like anything on offer and skipped all news-groups. Later, it
> >downloaded some news, but restarted the article numbers from 1.
>
> For the avoidance of doubt: was this Freeserve's news service or had you
> been trying elsewhere in the meantime?

No, just Freeserve.

> [snip]
>
> >I presume leafnode skips a news-group because freeserve offers
> >it crazy article numbers. Why does this happen?

> This remains to be seen, but yes leafnode/fetch makes assumptions that
> the article numbers will continue to increment.  The -f option gets
> round this so it's not a problem.

A look at the source for leafnode, shows I misunderstood fetchnews's
'skip' messages. They refer to a group being unread for too long.
I use a script to touch the groups I want before running fetchnews.
This may have altered the st_ctime values of files in leafnode's
interesting.groups directory which may have contributed to my problems.

Ted.
-- 
Ted <ted@nowtsfree.freeserve.co.uk>
   http://www.nowtsfree.freeserve.co.uk

 "Tim Haynes" <piglet@glutinous.custard.org> writes:
> ted@nowtsfree.freeserve.co.uk (Ted) writes:
> 
> [snipppppety] [snip]
> 
> > I presume leafnode skips a news-group because freeserve offers it crazy
> > article numbers. 
> 
> Not quite. Leafnode generally puts up a warning about "upstream servers
> switched?" when it detects the maximum article # in the parent is suddenly
> less than its own, but as it works by Message-ID instead, it seems to work
> anyway. (It just drops down to "last initial_fetch combined with max_fetch"
> and gets, in my case, no more'n 250 or 150 articles.)

I've just looked closer at the source while replying to Dan's post.
I may have caused the 'skip' messages from fetchnews, because I sometimes
run a script that touches groups I want to read before running fetchnews.
Maybe I forgot for a while. The 'skip' messages refer to groups that are
unread for too long. I was misunderstanding them.

> [snip]
> > Is there a fix I should know about?
> 
> nodesc=1 and fetchnews -fvvvv once more for good measure, should fix
> it. Oh, and remove the ranges in ~/.newsrc, so it just looks like
> 	uk.comp.os.linux:
> with no numbers - you'll have to do a selective catch-up in xrn but that
> way you can avoid reading too mcuh, anyway.

Yes, I'm trying that now. Thanks.

Ted.
-- 
Ted <ted@nowtsfree.freeserve.co.uk>
   http://www.nowtsfree.freeserve.co.uk