Thread View: gnu.emacs.gnus
4 messages
4 total messages
Started by Rob Robertson
Fri, 22 Jan 1993 15:00
XOVER support for GNUS
Author: Rob Robertson
Date: Fri, 22 Jan 1993 15:00
Date: Fri, 22 Jan 1993 15:00
250 lines
9290 bytes
9290 bytes
I've cooked the the following hacks so that GNUS can support the XOVER
command that will be available in INN 1.3. It dramatically speeds up
transfer times. note these are just hacks, ideally nntp.el would probe
for XOVER and automagically use it if it found it.
To use it just set the following hook:
(setq nntp-server-hook
(function
(lambda ()
(require 'xover)
(progn
(fset 'gnus-retrieve-headers 'xover-retrieve-headers)
(setq gnus-Article-prepare-hook
(function
(lambda ()
(xover-get-xrefs-on-fly))))))))
and byte compile xover.el, and put it in yer lisp path.
note that since the Overview database doesn't have the Xref: line,
marking of Xrefs happens when you look at the article, so if you catch
up inside *Subject* mode or do a kill, the other cross references
won't be marked as read automagically...
rob
william robertson
rob@agate.berkeley.edu
"when i was your age, i walked 10 miles to school, unarmed."
;; hooks for GNUS to use the XOVER command to quickly download
;; headers.
;;
;; $Header: /home/agate_a/rob/src/elisp/RCS/xover.el,v 1.2 1993/01/17 05:13:14 rob Exp $
;;
;; by William Robertson, rob@agate.berkeley.edu
;; based on code from nntp.el in the GNUS distribution,
;; by Masanobu UMEDA
;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY. No author or distributor
;; accepts responsibility to anyone for the consequences of using it
;; or for whether it serves any particular purpose or works at all,
;; unless he says so in writing. Refer to the GNU Emacs General Public
;; License for full details.
;; Everyone is granted permission to copy, modify and redistribute
;; GNU Emacs, but only under the conditions described in the
;; GNU Emacs General Public License. A copy of this license is
;; supposed to have been given to you along with GNU Emacs so you
;; can know your rights and responsibilities. It should be in a
;; file named COPYING. Among other things, the copyright notice
;; and this notice must be preserved on all copies.
;; this file consists of two parts:
;; one a replacement for nntp-retrieve-headers that uses
;; XOVER to download the headers in the group
;; and since the overview database doesn't normally contain
;; the Xref data, we have to get it when we actually look at
;; articles. 'xover-get-xrefs-on-fly does that when called
;; from 'gnus-Article-prepare-hook.
;; this file wants to be byte-compiled
;; i use the following hook to load xover.elc in. in the future
;; i'd like to write sumpin that checks the server and sees if it
;; supports XOVER and does the right thing.
;;
;;(setq nntp-server-hook
;; (function
;; (lambda ()
;; (require 'xover)
;; (if (string-match "^agate" nntp-server-name)
;; (progn
;; (fset 'gnus-retrieve-headers 'xover-retrieve-headers)
;; (setq gnus-Article-prepare-hook
;; (function
;; (lambda ()
;; (xover-get-xrefs-on-fly)))))))))
(provide 'xover)
;;(defun xover-test-server ()
;; "Test if a server supports the XOVER command
;;To be called before we do a GROUP command"
;; (progn
;; (nntp-send-command "^[2345].*$" "XOVER" "1-1")
;; (string-match "^[4][0-9][0-9][ \t]" nntp-status-string)))
(defmacro nntp-set-header-xref (header xref)
"Set article xref of HEADER to xref."
(` (aset (, header) 3 (, xref))))
;; to be called by gnus-Article-prepare-hook so that it
;; adds the xrefs in to the header list of the articles
;; we've looked at, so that we won't get shown them again.
(defun xover-get-xrefs-on-fly ()
"Get Xrefs as we read the article, and place a reference in HEADER"
(save-restriction
(goto-char (point-min))
(search-forward "\n\n" nil 'move)
(narrow-to-region (point-min) (point))
(goto-char (point-min))
(if (re-search-forward "\nXref:[ \t]*\\(.*\\)\n" nil t)
(nntp-set-header-xref gnus-current-headers
(buffer-substring
(match-beginning 1) (match-end 1))))))
(defmacro xover-sequence-to-range (sequence)
"Convert a SEQUENCE to nntp ranges"
(` (let ((first (car sequence))
(last (car sequence)))
(while (and sequence (or (= last (car sequence)) ;Omit duplicated number
(= (1+ last) (car sequence)))) ;Still in sequence
(setq last (car sequence))
(setq sequence (cdr sequence)))
(if (= first last)
(int-to-string first)
(concat (int-to-string first) "-"
(int-to-string last))))))
;; the function 'xover-retrieve-headers is a replacement for
;; nntp-retrieve-headers, and uses the XOVER command to get the
;; headers. the syntax of the XOVER command is
;;
;; XOVER RANGE
;;
;; where
;;
;; RANGE : [0-9]+ | [0-9]+-[0-9]+
;;
;; it returns on line per available article:
;;
;; messno\tsubject\tfrom\tdate\tmsg-id\trefs\tlines\tbytes\n
;;
;; the reply is terminated by a .\r\n
(defun xover-retrieve-headers (sequence)
"Return list of article headers specified by SEQUENCE of article id.
The format of list is
`([NUMBER SUBJECT FROM XREF LINES DATE MESSAGE-ID REFERENCES] ...)'.
Reader macros for the vector are defined as `nntp-header-FIELD'.
Writer macros for the vector are defined as `nntp-set-header-FIELD'.
News group must be selected before calling me."
(save-excursion
(set-buffer nntp-server-buffer)
(erase-buffer)
(let ((last-point (point-min))
(number (length sequence))
(received 0)
(count 0)
(headers nil) ;Result list.
(article 0)
(subject nil)
(message-id)
(from nil)
(xref nil)
(lines 0)
(date nil)
(references nil)
(spin '(?- ?\ ?| ?/ ))
(index 0))
;; Send XOVER command.
(while sequence
(nntp-send-strings-to-server "XOVER" (xover-sequence-to-range sequence))
(setq count (1+ count))
;; Every 400 header requests we have to read stream in order
;; to avoid deadlock.
(if (or (null sequence) ;All requests have been sent.
(zerop (% count nntp-maximum-request)))
(progn
(accept-process-output)
(while (progn
(goto-char last-point)
;; Count replies.
(while (re-search-forward "^\\.\r$" nil t)
(setq received (1+ received)))
(setq last-point (point))
(< received count))
;; If number of headers is greater than 100, give
;; informative messages.
(and (numberp nntp-large-newsgroup)
(> number nntp-large-newsgroup)
(message "NNTP: Transfering %c" (nth index spin))
(setq index (% (1+ index) (length spin))))
(nntp-accept-response))
))
)
;; Wait for text of last command.
(goto-char (point-max))
(re-search-backward "^[0-9]" nil t)
(if (looking-at "^[23]")
(while (progn
(goto-char (- (point-max) 3))
(not (looking-at "^\\.\r$")))
(nntp-accept-response)))
(and (numberp nntp-large-newsgroup)
(> number nntp-large-newsgroup)
(message "NNTP: 100%% of headers received."))
;; Now all of replies are received.
(setq received number)
(and (numberp nntp-large-newsgroup)
(> number nntp-large-newsgroup)
(message "NNTP: Parsing headers..."))
;; Then examines replies.
(goto-char (point-min))
(while (not (eobp))
(if (looking-at "\\([0-9]+\\)\t\\(.*\\)\t\\(.*\\)\t\\(.*\\)\t\\(.*\\)\t\\(.*\\)\t\\(.*\\)\t\\(.*\\)\r$")
(progn
(setq article
(string-to-int
(buffer-substring (match-beginning 1) (match-end 1))))
(setq subject
(buffer-substring (match-beginning 2) (match-end 2)))
(setq from
(buffer-substring (match-beginning 3) (match-end 3)))
(setq date
(buffer-substring (match-beginning 4) (match-end 4)))
(setq message-id
(buffer-substring (match-beginning 5) (match-end 5)))
(setq references
(buffer-substring (match-beginning 6) (match-end 6)))
(setq lines (string-to-int
(buffer-substring (match-beginning 8) (match-end 8))))
;; Set default value.
(setq xref nil)
;; Finished to parse one header.
(if (string= subject "")
(setq subject "(None)"))
(if (string= from "")
(setq from "(Unknown User)"))
(setq headers
(cons (vector article subject from
xref lines date
message-id references) headers))
(setq received (1- received))
(and (numberp nntp-large-newsgroup)
(> number nntp-large-newsgroup)
(zerop (% received 20))
(message "NNTP: Parsing headers... %d%%"
(/ (* received 100) number)))))
(forward-line 1))
(and (numberp nntp-large-newsgroup)
(> number nntp-large-newsgroup)
(message "NNTP: Parsing headers... done"))
(nreverse headers))))
Re: regarding `XOVER support for GNUS', etc.
Author: Rob Robertson
Date: Sat, 23 Jan 1993 02:14
Date: Sat, 23 Jan 1993 02:14
76 lines
3555 bytes
3555 bytes
The Overview Database
first of all, nov, or the netnews overview database, the overview
database are all the same thing. it is a database of a couple of the
most popular headers. it was designed by Geoff Collyer as a standard
way of quickly getting header information, so that it could be used by
threading newsreaders. it is organized, one file per newsgroup, ala
/usr/spool/news/<group>/.overview
the file contains a one-line summary, in order, of every article in
that group. the fields are separated by tabs and any tabs or newlines
in the original headers are replaced by spaces. the fields in order
are article number (filename), subject, author, date, message-id,
references, byte count, line count. other optional locally arranged
headers maybe tacked on the end, but they should be proceded by the
header keyword.
this is how Geoff wrote this for Cnews, INN allows additional
flexibility by allowing the database to be in a different directory
hierarchy than /usr/spool/news, and you can call the .overview file
something else, like overview--without the dot.
ie
artno<T>Subject<T>From<T>Date<T>Mess-Id<T>References<T>Bytes<T>Lines
the format is very straight forward, and is not prethreaded, the
reason behind this being that threading can be done reasonably quickly
on the client, and rather than going for a particular kind of
threading, just make the data *quickly* available and let the client
decide how to order it.
The XOVER Command
the XOVER command is an extension to NNTP to allow access to the
overview database. the syntax of the command is basically:
XOVER range
range := artno | artno '-' artno
artno := [0-9]+
one has to have done a 'GROUP' command earlier to establish a current
group. if successful, it returns a code '224', the data [as described
above], then a '\r\n.\r\n' to let you know it's done.
Support
the overview database is supported in Cnews by Geoff's nov
distribution available for ftp at world.std.com
/src/news/nov.dist.tar.Z. in the future, it will probably be included
in future versions of Cnews. this package also contains some wiz bang
threading routines for clients, plus the official newsoverview.5 man
page!
there are patches to the reference implementaion of NNTP, to add the
XOVER command. hopefully stan barber will include them in his next
release of the reference implementaion.
InterNet News 1.3 (released real soon now) will support both the
overview database and the XOVER command.
as far as newsreaders go, new versions of trn and tin which are both
in beta test support the overview database and the XOVER command.
i've posted patches for GNUS, and Felix Lee promises to post even
better ones.
hope this explains it and if i got anything wrong, please let me know.
rob
--
william robertson
rob@agate.berkeley.edu
"when i was your age, i walked 10 miles to school, unarmed."
Re: XOVER support for GNUS
Author: Felix Lee
Date: Sat, 23 Jan 1993 04:02
Date: Sat, 23 Jan 1993 04:02
5 lines
648 bytes
648 bytes
Rob Robertson's XOVER patch does work, and it's quite zippy receiving headers (though a little verbose), but it's no faster at parsing the headers than the stock nntp.el. My nntp.el is substantially faster at parsing headers too. I should have it ready by Sunday. --
regarding `XOVER support for GNUS', etc.
Author: k p c
Date: Sat, 23 Jan 1993 07:42
Date: Sat, 23 Jan 1993 07:42
12 lines
1132 bytes
1132 bytes
i would be grateful if somebody would explain xover, inn, and nov in a nutshell. will they work with spool gnus as well as nntp gnus? how will they fit into the off-line sorting and threading idea? are they used for searching as well as retrieving? thanks. (in case others are also curious, please post replies.) -- i'm open to comparing research career notes with other cogsci/cogneuro people.
Thread Navigation
This is a paginated view of messages in the thread with full content displayed inline.
Messages are displayed in chronological order, with the original post highlighted in green.
Use pagination controls to navigate through all messages in large threads.
Back to All Threads