🚀 go-pugleaf

[This is going to be rather long: apologies in advance.]

On Wednesday, in article
     <20120919.1103.12090207snz@i-m-t.demon.co.uk>
     v$af$ppint@i-m-t.demon.co.uk "ppint. at pplay" wrote:

>         - hi; in article, <20120917.1210.932816snz@twoplaces.co.uk>,
>          simon@twoplaces.co.uk "Simon Turner" commented & explained:
> >                          "ppint. at pplay" wrote:
> >>      (following article, <20120914.2213.12090153snz@i-m-t.demon.co.uk>,
> >>                            in which i wrote):
> >> []
> >> >    - since some time between wed sep 12 21:33:35 and thu sep 13
> >> >    20:07:59 attempting to collect mail from the original base a/c
> >> >    (user, wotsit) has repeatedly failed since 20:08:00 after noting
> >> >    that the first two emails (always the same two) are one mailkill
> >> >    rejected and one to unknown user address (in the scrolling info
> >> >    at the net> and in mail.log).
> >
> >I didn't comment on this before, but it sounds as though you've still
> >got "pop3 leave on" -- we need to get to the bottom of this at some
> >stage, as it should be off unless you've explicitly turned it on!
> 
>         - i <enter>ed "pop3 leave off" from the net> prompt after the
>         first time i dialled up after you commented it should default
>         to this state, and haven't knowingly changed matters in this
>         regard since: which suggests that the "leave off" doesn't get
>         carried out until a normal end to the mail-collecting process
>         occurs, as opposed to after each individual email is collected
>         (or rejected).  bic(oc)bw :-)

As noted in d.s, "pop3 leave" is a GremNOS setting: it tells GremNOS
whether (a) to delete messages once they have been received (or
rejected) or (b) leave them on the server.  GremNOS's "pop3 leave" state
is stored in a variable, which is initialised to 0 (i.e. off) at program
start-up: barring internal memory corruption, it can only be changed by
the "pop3 leave on|off" command.

Next time you connect (after reading this 8-), just before you do your
POP3 collection, run "pop3 leave": this will display the current state
of the setting, e.g.

  Leave downloaded mail on server: off

If it's off, that's OK; if not, something is turning it on!  You could
narrow it down by putting "pop3 leave" at the start and end of your
AUTOEXEC.NET file etc.

However, I think I've realised how you could be getting the symptoms of
it being on even though it's really off: see later for details.

BTW, if you're doing your POP3 collections by hand instead of putting
them in AUTOEXEC.NET, I heartily recommend putting them in a file (or
files) that you can later "source" -- that way, you only have to type
the commands in once.  I have an assortment of such 'script' files that
get sourced at regular intervals by the "at" command, and which can also
be run manually e.g. "source pop3dmn.scr".

> >The fact that you're getting a "TOP M 0 failed" error means (a) that the
> >POP3 collection system isn't in a "receiving data" state -- any data
> >from the previous message has (apparently) finished coming in -- and (b)
> >that it was a TOP command (which only returns the header, not the body)
> >that failed, rather than RETR which is used to grab the whole message.
> 
>         - hmmm; "interesting..." - especially as up to 6MB of the email
>         in question was collected, the last time i tried seeing what
>         happened if i just stayed online, in the wee smal hours - and
>         had to delete a lock file on the mailbox it was headed for, or
>         at least chose to, after i eventually gave up and disconnected.

That's very odd.  Did you get any POP3 error messages that time, or did
everything behave normally apart from the download not finishing before
you gave up?

> >You can find out whether the POP3-SMTP socket is still receiving data by
> >looking up its socket number ("socket"), then running "socket NNN" a
> >couple of times and comparing the rcvd totals.
> 
>         - ah; yes, i should've thought of that - i've only ever previously
>         checked ftp, hop check, ping and similar sessions.

It's very useful.  Back when I was using ISDN, I used to get bursts of
packet corruption that could stall an in-progress SMTP send: I got quite
used to using "socket 133" (as it always was) to inspect the socket's
TCP queue and see if it was stalled, waiting for a TCP acknowledgement
that would never come, and if so issuing kicks and/or TCP resets as
appropriate to restart timers or reset the TCP window management.

> >The GremNOS POP3 collection picks them up in message order (1, 2, ...),
> >which seems to be the order in which the messages arrived in the mailbox;
> >size of message doesn't come into it.  (Although it could: see below.)
> 
> >>      - cos if not, and someone (or something) sends a ginormous mail
> >>      to a live mailbox your domain, then it'll jam things up good'n'
> >>      proper, won't it?
> >
> >Yup, it will.  I hadn't given enough thought to that (having a nice fast
> >FTTC connection rather than dialup). 8-(
> 
>         - s/8-(/8-0/ ?      - [a]

8-D

> >Two improvements spring to mind: (a) modify the POP3 collection to allow
> >GremNOS to collect the messages in ascending size order rather than in
> >message number order; (b) implement Pete's ESMTP SIZE restrictions for
> >POP3 as well, which would allow you to set a size threshold above which
> >messages wouldn't be downloaded.
> 
>         - will the latter reject messages above the threshold, or leave
>         them uncollected?

It could (and, I think, should) do either: I can see it being useful for
quick downloads with a low threshold, just to see if that expected (but
short) mail has come in yet, and leave the main download (with a higher
threshold) for later; and the *really* big ones could be left on the
server to be collected at the weekend.

>         if the latter, this might trigger the strange
>         behaviour seen above, where messages rejected as killfiled/to an
>         unknown user-name are actually left on, despite "pop3 leave off"
>         being the most recent instruction.      - [b]

AHA!  The light has just gone on.  Why didn't I think of this before?

A possible (indeed, likely) reason why you kept getting the same
messages left on the server every time you had a problem is that the
POP3 connection was still active (downloading the huge message) when you
disconnected: the server will[*] only process pending deletions when the
client disconnects normally (by sending the QUIT command).  If the QUIT
doesn't appear, the server will eventually time out the connection, and
pending deletions will[*] not be performed -- so the same messages will
be present next time you try.

[*] Or at least should: this is an MS server, so all bets are off wrt
RFC compliance...

One thing I always used to do on dialup just before exiting GremNOS was
a quick "socket", to make sure there weren't any active sockets that
would be broken off mid-stream by the impending disconnection.

Note that noe of this explains the "TOP M 0 failed" errors you saw,
though: those can only have happened due to a real error somewhere (and
they sound very much like loss of synchronisation between server and
client, where GremNOS thinks the server has finished sending but it
actually hasn't, so the next "response" is actually part of the previous
data stream and makes no sense).

I'm going to improve the error messages, which should help work out
what's happening if the problems ever recur; I've checked the code
carefully and can't see how TOP (or a preceding command) could be
failing, so unless it happens again there's not a lot we can do.  (I've
been using POP3 daily for over five years without any problems, except
when Demon ineptly forged the postmaster e-mails and got the *ENV stuff
wrong, and even then I still got the message).

> >If you wish to record the session to a file, press F10 to get back to
> >the prompt, enter "record pop3.rec" (to record to file POP3.REC) then
> >press Enter to return to the session.  Recording will stop when the
> >session closes, or when you enter "record off" at the net> prompt.
> 
>         - *nods* - except i'll need to remove any & all finger (etc.)
>         lines from my autoexec.net when i wish to record pop3 sessions,

No, you shouldn't need to do anything special, since this is a telnet
session you'd be recording, with its own screen; the bit you were having
trouble with previously was trying to record the main command screen,
which is indeed impossible if you've already used F10 to escape back to
it from another session.  (All will become clear below, if you can wade
through that far!)

>         as something doesn't re-set when an smtp session ends

SMTP sending/receiving connections actually don't have their own
sessions -- their status messages get written to the main command
screen.  If you telnet into a server to talk some protocol to it (port
25 for SMTP, 110 for POP3, 119 for NNTP etc.), it's a telnet session.

>         - nor,
>         when an ftp session ends (which it used to, sort of - formerly,
>         it'd re-set to another (the next-highest numbered, iirc) session.

Are you sure KA9Q ever did this (resetting the "active session" marker
to the next-highest-numbered session when the currently active session
ended)?  The GremNOS code certainly doesn't behave that way, although
it's possible to persuade yourself that it does (I managed to persuade
myself thus earlier, before I'd understood how it really works).

The way it works (in plain GremNOS; TextWin versions are a different
matter altogether) is this: there is a concept of the "most recently
escaped from" session, which is the one with a '*' against its number in
the session list: this is the one you return to if you just press Enter,
and is the one recorded by a "record" command.

I'll call this the "last" session to save typing; but the description of
"most recently escaped from" is more accurate.  The last-session marker
is only ever set when you use F10 to escape back from a session to the
main command screen; if you then return to that session and end it,
thereby returning to the main command screen without pressing F10, the
marker -- which was set to that now-defunct session -- is reset to
nothing.  If you do a session list at this point, there is no '*' shown,
pressing Enter doesn't get you anywhere, and attempting to record
anything says "No current session".

Note that, if session 3 (say) has the last-session marker but you return
to session 2 (by entering "session 2") and end that session, thereby
getting back to the main command screen, the marker remains set to
session 3.  This is what I was seeing earlier when I persuaded myself
that it was being updated when a session ended; it wasn't, it was
staying exactly where it had previously been!

The only way to get one of the other sessions to have the last-session
marker is to return to it ("session 4", say) then press F10 to get back
to the main command screen; session 4 now has the last-session marker,
and will the one that gets recorded by the "record" command.

There's certainly an argument to be made that, when the session
currently marked as the "last" one ends, the last-session marker
shouldn't be reset to nothing, but should be reset to one of the other
active sessions (or, failing that, the main command screen -- which is
also technically a "session"); that would IMHO be better than the
current system, and would allow you to record the main command session
if you wanted to (currently you can do it right at the start, because
the last-session marker is initially set to the command session, but it
cannot be got back there if you've ever used F10).

>         [or is it properly "socket"?]

"Sockets" are Internet connections, e.g. between clients and servers.
"Sessions" in KA9Q are discrete 'screens', which can be switched between
by pressing F10 (to get back to the command screen) and entering
"session N".  There can be active sockets without associated sessions
(SMTP, NNTP, POP3-SMTP etc.) and sessions without associated sockets (a
"more" session viewing a local file, a finger that has already ended and
closed its socket etc.)  If a session has an active socket (e.g. an
active ftp or telnet session), the socket number is shown in the "S#"
column in the session list.

>         - ?  or possibly, type "s<enter>" to find which "s<0-9><enter>"
>         to type to get to the pop3 session - ? (note to self: try it.)

That's the one: get back to your POP3 (actually telnet) session, using
"s<enter>" to find out which one it is, then "s N<enter>" to get back to
it; then F10 to escape back to the command screen and update the "most
recently escaped from" marker; then "rec pop3.rec" to record it; <enter>
to return to it; and finally you can start talking to the server.

I told you it was going to be long!

-- 
Simon Turner                                    DoD #0461
simon@twoplaces.co.uk
    Trust me -- I know what I'm doing!          -- Sledge Hammer