🚀 go-pugleaf

It is not that hard to write a news collector. I would recommend Java 
as the ideal programming language if you wanted to write one from 
scratch.

Alternatively, I believe there are some shareware ActiveX controls 
which wrap up socket protocols like NNTP in an easy-to-use wrapper.

Writing a news reader is a big task, even in the best Rapid 
Application Development system.

Aside from the knowledge you would gain by writing a news collector, 
I can think of no useful benefit, when there are excellent mail and 
news readers available for Windows, including Agent and Virtual 
Access.

[I must declare an interest - I do some contracting work on Virtual 
Access.]

-- 
Nikki Locke, Trumphurst Ltd.      PC & Unix consultancy & programming
nikki@trmphrst.demon.co.uk            http://www.trumphurst.com/

Anthony Naggs <URL:mailto:amn@ubik.demon.co.uk> wrote:
> Finally my tolerance of Turnpike's quirks has expired.   (It trashed my
> 'usernews' file, it's record of which articles I've read, etc...) 

Well any program (or user) may do that anyway... Tin handles it
resaonably well in that it at least backs up the list of read
articles when you run it. So _usually_ you only lose the last
session...

> As a little detour whilst looking at other packages, I want to see how
> difficult it would be to write my own news reader/collector.  Obviously
> I have RFC977 (& 850!), but I'm sure there must be supplementary

RFC1036 and SORFC1036. And RFC0822 (so you can handle email replies
from your newsreader).

> material, (eg descriptions of XOVER & XHDR).  I'd appreciate suggestions
> of what to read & where to find it.  :-)

Well as far as a news collector goes, and depending on how you
want to deal with killfiling and such like isn't too difficult.
At least the NNTP side isn't. There are a small number of commands
with simple syntax and fairly generic return codes. Of course getting
that code to talk to anything else is another matter, and I don't
have a clue how easy (or not) that is under Windows.

As for XOVER and the like, I'd be interesting in finding any
definitive documentation as well![1] I've been looking at adding
more killfiling abilities to tin using overview files (if anyone's
interested I've got a working sol'n to filtering on Newsgroups:
 - even when you don't have the crossposted groups in your local
newsbase - Path: and Archive-Name:)

Another big difference is whether you use NEWNEWS (which gives
you a list of msgids to download) or retrieval by article number
(which means you have to store a local copy of which article
numbers have been downloaded for the machine - as well as 
individual user copies of what they've read).

NB that killfiling is better with XOVER, but NEWNEWS is easier
to implement, especially for use with multiple servers (since
you only need to store when you last accessed that server, all
dupes can be dealt with using a history file).

And remember some servers may support a limited set of commands.

You may find it useful to look at the source for INN, NNTPd, tin,
slrn and the like. And for a relatively simple view of how the
NNTP side works, NewNews.cmd which is in the one of the Demon OS/2
setups is a Rexx program and therefore relatively easy to read.
Also to cut testing costs down to a minimum it's handy to install a
local newsserver.

[1] though with a little experimentation most of the syntax for
a given server isn't too hard to work out.

Writing a newsreader is a different matter... Though there's an
obvious relationship. The RFCs tell you about how to collect the
news, but not what to do with it. Turnpike et al. tend to store
all the news in a single file, many UNIX-originated programs
use a file per article. There are various advantages to both
methods.

-- 
<URL:http://www.pierrot.co.uk/>                                  Team AMIGA
It's not the size of the ship, it's the size of the waves.  - Little Richard

On Fri, 24 Jul 1998 01:18:44 +0100, Anthony Naggs <amn@ubik.demon.co.uk>
wrote:

>As a little detour whilst looking at other packages, I want to see how
>difficult it would be to write my own news reader/collector.  Obviously
>I have RFC977 (& 850!), but I'm sure there must be supplementary
>material, (eg descriptions of XOVER & XHDR).  I'd appreciate suggestions
>of what to read & where to find it.  :-)

You should read the Usenet Seal of Approval (I don't have a local copy
here.. but a quick search of DejaNews/AltaVista should turn up a copy). 

I'm pretty sure that the advice given in that document to people
thinking of writing a newsreader is, basically, "don't", which may sound
disheartening but there are a number of fairly convincing arguments put
forward to dissuade potential authors of 'from scratch' readers.

Personally, I think Forte Agent is great - for both mail and news. My
only gripes are that the editor has no "Undo" facility, and the
Message-ID generation is broken by default (this is rectified with an
INI file modification, but /that/ leaves you without knowing the
Message-IDs of the articles you post until they actually appear on a
news server :( ).

Still, if you *do* decide to take the plunge and have a go at writing
one, I'd be delighted to be considered as an Alpha/Beta tester ;)
-- 
Chris Cowley

After much consideration Terrance Richard Boyes decided to share these
wise words:
>Anthony Naggs <URL:mailto:amn@ubik.demon.co.uk> wrote:
>> Finally my tolerance of Turnpike's quirks has expired.   (It trashed my
>> 'usernews' file, it's record of which articles I've read, etc...) 
>
>Well any program (or user) may do that anyway... Tin handles it
>resaonably well in that it at least backs up the list of read
>articles when you run it. So _usually_ you only lose the last
>session...

One of the few things I liked with Tin.  TP apparently doesn't backup
the file, and the 'mark as read' is rather limited.  Eg to get my read
list in a moderate state I would be happy for most newsgroups to mark as
read all articles over a week old.  


>> As a little detour whilst looking at other packages, I want to see how
>> difficult it would be to write my own news reader/collector.  Obviously
>> I have RFC977 (& 850!), but I'm sure there must be supplementary
>
>RFC1036 and SORFC1036. And RFC0822 (so you can handle email replies
>from your newsreader).

Yep, pretty obvious after a peek at an index of RFCs.  Though I couldn't
find SORFC1036 in my brief venture out, (which is presumably different
from draft-ietf-drums-msg-fmt-04.txt which updates RFC822).

However I did find some useful Internet-Drafts:
        draft-ietf-nntpext-base-04.txt
        draft-ietf-nntpext-imp-02.txt

>Well as far as a news collector goes, and depending on how you
>want to deal with killfiling and such like isn't too difficult.
>At least the NNTP side isn't. There are a small number of commands
>with simple syntax and fairly generic return codes. Of course getting
>that code to talk to anything else is another matter, and I don't
>have a clue how easy (or not) that is under Windows.

It shouldn't be too hard, Winsock supports both synchronous &
asynchronous use of sockets.  And Win 95/NT is properly multi-threaded
so blocking a thread whilst waiting for an NNTP response makes
programming fairly straihtforward.


>As for XOVER and the like, I'd be interesting in finding any
>definitive documentation as well![1] 

Of the two documents I mentioned above the first replaces RFC977, the
second describes a few extensions including XOVER.

[snip]
>NB that killfiling is better with XOVER, but NEWNEWS is easier
>to implement, especially for use with multiple servers (since
>you only need to store when you last accessed that server, all
>dupes can be dealt with using a history file).

Agreed, but ...

>And remember some servers may support a limited set of commands.

... the scary thing is that draft-ietf-nntpext-imp-02.txt observes:
::    NEWNEWS is one of the commands frequently disabled.


>You may find it useful to look at the source for INN, NNTPd, tin,
>slrn and the like. And for a relatively simple view of how the
>NNTP side works, NewNews.cmd which is in the one of the Demon OS/2
>setups is a Rexx program and therefore relatively easy to read.
>Also to cut testing costs down to a minimum it's handy to install a
>local newsserver.

Suggestions noted, thanks.

As a first pass writing a local news server could be the way to go.  It
would allow some interesting things, such as; processing of Cancels or
NoCeM, using a score file during expiry...  And talking NNTP to a local
server is ideal for generating spam cancels, eg for retromoderation.


>Writing a newsreader is a different matter... 

Indeed, probably I wont write one.  But now is the time to read up and
work out how hard it would actually be.  Of all the newsreaders I've
used on Unix, DOS & Windows down the years I've found all of them
awkward and cumbersome in some way.  Eg I want to:
+   Present articles in different orders for different newsgroups.
+   Forward copies of articles that include the headers, to report
    spam.  Doubly so if I want the complaint to come from a different
    userid, such as newsmaster@ubik.dcu.


Thanks for the reply.  ;-)


Tony
-- 
  There's another train there always is
  maybe the next one is yours
  get up and climb aboard another train

Anthony Naggs <URL:mailto:amn@ubik.demon.co.uk> wrote:
> After much consideration Terrance Richard Boyes decided to share these
> wise words:
>>
>>RFC1036 and SORFC1036. And RFC0822 (so you can handle email replies
>>from your newsreader).

> Yep, pretty obvious after a peek at an index of RFCs.  Though I couldn't
> find SORFC1036 in my brief venture out, (which is presumably different
> from draft-ietf-drums-msg-fmt-04.txt which updates RFC822).

On Demon in /pub/news/docs, there's source for inn, tin and so
on in there as well.

Thanks for the ietf file names, now I can find them :)

-- 
<URL:http://www.pierrot.co.uk/>                                  Team AMIGA
The gentlemen looked one another over with microscopic carelessness.

Terrance Richard Boyes <tez@pierrot.co.uk> writes:

> On Demon in /pub/news/docs, there's source for inn,

ftp.isc.org for the canonical copy.

-- 
`It is inelegant and it was hacked on. But it was hacked on with
 careful consideration.' - PLA on perl and OO

In article <902505530.28303.2.nnrp-01.d4e48b20@news.demon.co.uk>,
Alisdair Wren <alisdair@nospam.demon.co.uk> writes

>I am current in the process of writing a newsreader:  The biggest 
>problem I have come across is sorting the list of newsgroups in a 
>reasonable time:  I've got it down to 11.87 seconds so far.  The second 
>biggest problem is searching the newsgroup list in a reasonable time 
>;-).  The actual programming involved isn't too difficult, it just 
>getting all the algorithms to run in a reasonable time.

try Knuth Volume 3 (the second edition is now out!)  or indeed any
textbook on algorithms

>While I'm about it:  Does anyone know if newsgroup names are case 
>sensitive??

they are case sensitive but you must not use upper case. In practice at
the moment there are newsgroups which clash if considered casesensitive
but the lower case version is getting all the traffic

>Obviously, Binary comparisons take much less time than Text 
>comparisons, so if I could just stick everything in lower case life 
>would be much easier.  The only reason I ask is that Intel have defined 
>both the Intel.* and the intel.* heirarchies.

indeed

-- 
richard                      richard.clayton    @    T U R N P I K E .com
 http://www.demon.net/news/features/crypto/  for Demon's views on crypto
"Assembly of Japanese bicycle require great peace of mind" quoted in ZAMM

In article <902567237.22987.2.nnrp-01.d4e48b20@news.demon.co.uk>,
Alisdair Wren <alisdair@nospam.demon.co.uk> writes

>A little while ago, Richard Clayton <richard@turnpike.com> digitally said:
>>>While I'm about it:  Does anyone know if newsgroup names are case 
>>>sensitive??
>>
>>they are case sensitive but you must not use upper case. In practice at
>>the moment there are newsgroups which clash if considered casesensitive
>>but the lower case version is getting all the traffic
>
>So what would you suggest:  Going for double the speed and therefore 
>nobbling anyone who wants to use upper-case newsgroups (which shouldn't 
>exist and exist a lower-case as well anyway), or being tolerant and 
>slow? :-)

we chose to downshift, but that's just what works well at the moment, it
may not be a longterm practical solution. If USEFOR ever produces
anything worthwhile we will follow that

it obviously failed to come across when I wrote before... if sorting is
taking any time you can actually measure then you're using the wrong
algorithm or implementing it poorly. You never use a BubbleSort for
anything (he says, provocatively)

-- 
richard                      richard.clayton    @    T U R N P I K E .com
 http://www.demon.net/news/features/crypto/  for Demon's views on crypto
"Assembly of Japanese bicycle require great peace of mind" quoted in ZAMM

In article <gUI7zNADfCz1EABV@turnpike.com>
           richard@turnpike.com "Richard Clayton" writes:

> it obviously failed to come across when I wrote before... if sorting is
> taking any time you can actually measure then you're using the wrong
> algorithm or implementing it poorly. You never use a BubbleSort for
> anything (he says, provocatively)

Yep, that's definitely true.  About 18 years ago I wrote a Z-80 assembler
in VAX Basic (VAX Basic was actually a *good* programming language, in
that it was a *very* structured variant of Basic; in the 18,000 lines of
source code, there was not ONE line number[1] nor any GOTOs)

After the first pass, I obviously had to sort the symbol table before the
code generating second pass.  My initial cut of the program used a bubble
sort.  This was OK on trivial examples, but when I got around to
compiling the real program for which I wrote the asssembler (the
firmware for a VDU that managed VT102 emulation as well as Tektronix
4010, and ran out at 32KB of code, with over 14,000 symbols) I found that
the sort was taking almost three HOURS (on a VAX-11/730).

Successive refinements firstly used a Shell sort, and later a variant of
Quicksort; the time was brought successively down firstly to 7 minutes,
and later to seven SECONDS.


[1] Actually, that's a lie: in order to preload some data structures, it
was necessary to use a command that takes info from a DATA statement; the
latter *have* to be numbered.  But otherwise, there were no line numbers.

-- 
Brian {Hamilton Kelly}                                         bhk@dsl.co.uk
      "You're one of the very rare few who can argue ad hominem, ad absurdum
       and ad nauseam simultaneously...worse still it looks like you want to
       carry on ad infinitum..."               WPSM on alt.config 1998/06/27

Richard Clayton <richard@turnpike.com> writes:

> it obviously failed to come across when I wrote before... if sorting is
> taking any time you can actually measure then you're using the wrong
> algorithm or implementing it poorly. You never use a BubbleSort for
> anything (he says, provocatively)

Although modified bubblesorts (variants on shellsorts &c) are quite
useful, especially when memory is a limited resource so you can't
afford to use partitioning/recursive schemes like quicksort[1].

[1] this is assuming you can't be bothered to implement an iterative
    variant of the quicksort, too.

-- 
`Thus may these Demon newsgroups be wrested from cabal domination
 and handed back to their humble subscribers.' - the par@noid goes
 paranoid. What do you *say* to people like this?

Nix <{$eri}$@esperi.demon.co.uk> wrote:
>
>Although modified bubblesorts (variants on shellsorts &c) are quite
>useful, especially when memory is a limited resource so you can't
>afford to use partitioning/recursive schemes like quicksort.

shellsort != bubblesort -- it's far more efficient, although (IIRC)
no-one has yet proven exactly how efficient it is...

Tony.
-- 
F.A.N.Finch       dot@dotat.at       fanf@demon.net       +44-7970-401-426
genes <=> memes    proteins <=> algorithms    amino acids <=> machine code

In article <gUI7zNADfCz1EABV@turnpike.com>
    Richard Clayton <richard@turnpike.com> writes:

> You never use a BubbleSort for anything (he says, provocatively)

Well, it's OK (and may even be the best choice) for *very* small data
sets but it scales very, very badly.  So badly, in fact, that for large
data sets you can give up hope of it ever finishing.

We know a song about that, don't we, children?

    I'm forever sorting bubbles...

--Paul

Tony Finch <fanf@demon.net> writes:

> Nix <{$eri}$@esperi.demon.co.uk> wrote:
> >
> >Although modified bubblesorts (variants on shellsorts &c) are quite
> >useful, especially when memory is a limited resource so you can't
> >afford to use partitioning/recursive schemes like quicksort.
> 
> shellsort != bubblesort

<wriggle> it uses bubblesorts, though </wriggle>

-- 
`Thus may these Demon newsgroups be wrested from cabal domination
 and handed back to their humble subscribers.' - the par@noid goes
 paranoid. What do you *say* to people like this?