🚀 go-pugleaf

Nota Bene
          The following is probably old news to some of you. 
          Maybe similar concerns have been expressed before on
          the net.  Whatever the case may be, I feel now is a
          good time to think about it, again.  It was written a
          few months ago, but a conversation last night, at a
          UniForum meeting, rekindled my zeal.  So here it is:

Introduction

According to a document published on the net a few weeks back, a
full news feed taken at UUNET would fill a CD-ROM in 18 days. 
I'll spare you the mathemathical callistenics showing the impacts
on your respective communication, processing and storage
capacities; let's just say it has gotten out of hand, or at least
very nearly.

Usenet demands ever increasing amount of machine resources and of
system administration time.  Expiration cycles get shorter and
shorter.  Archiving any hierarchy is already utopic for most
sites.  There is a point of at which it becomes unpractical to
maintain a news feed.  The growth is unsustainable in the current
environment.  In the "store and forward" world, the situation has
become even more ludicrous; telephone lines are busy around the
clock, flooding the world with a lot of noise and a tiny bit of
actual information.  All of which gets rebroadcast down the line
to an ever increasing number of sites.  All this for a few
scattered readers, reading a few scattered items, in a few
scattered groups, sometimes.

Because it is where the problem first arises I will deal here
with the UUCP type of newsfeeds (store and forward); not all of
its problems are pertinent to the rest of the world, but some of
the solutions may have universal benefits.


The situation

The flooding algorithm meets the so-called tragedy of the
commons.  The volume of information that gets broadcast on the
net is bound to increase and so is the number of people who want
to access it, and contribute.  Any measure that would tend
towards restricting posting rights or placing quotas on volume is
sure to meet vociferous opposition.  Rightly so.

Unless we change something about the way we work it, we are about
to turn our machines off, out of boredom, frustration and
disgust.  But we have no Plan B.  If we don't want a net police
and if we don't want a slow net death from suffocation, we should
reconsider some of the basic premises under which we are all
working.

The first premise that needs reconsidering is that any decent
feed must be a full feed.  The second premise is that - for most
sites - the news must all come from one single place.


Full feed

A full feed is not readable by any one individual.  Too often,
sysadmins request complete feeds, just in case.  The only
justification for such a wall to wall coverage is found when one
redistributes the feed, whole or in part, and the aggregate
redistributed coverage includes the whole spectrum.  It is rarely
so, unless the sites downstreams also indulge in the "ALL, just
in case" policy.  Another reason for full-feeditis is that news
configuration gets real simple or, reverse side of the same coin,
micro-management of the sys file is a pain in the spooler.

I find it surprising that no system, of the public domain news
distribution products I have seen so far, permits dynamic
updating of ~news/sys files at the feeder sites, by the feedee
(!).  I heard that such mechanism was available in a commercial
product (Dynafeed for ClariNet?).  Isn't it time to give the
concept wider exposure?  Does anyone know how it is done and how
[un]practical it is?

Such an approach, given restricting guidelines regarding expected
volumes, site policies and copyright protections, would let
readers [un]subscribe "in flight" to any specific newsgroup,
without disturbing at least two system administrators.


One source

Another unwholesome habit, directly borne out of the full-feed
syndrome, is that of shopping at one single place (there are
sites which have two or more mother sites but, at least one is a
full feeder; the problem just gets compounded).  The dynamic
updating of sys files feature could include the notion of "head
of the class", whereby a site is chosen as the main distributor
for a hierarchy or for a class of traffic.  The use of wildcards
in sys files should be generalized so we could subscribe to
*.sources.* or *.next.* or such unrestricted use of regular
expressions.

One site could decide to specialize in sex, or politics.  Another
could choose to take over the responsability of the whole alt
hierarchy, or comp.unix, or both.  One site could cover all
that's not already covered by others.  The combinations are of
course endless.  Any suggestion on how to cut the cake?  What
about bidding in reply to requests for feeds?  Any idea about
automatic subscription expirations (unless specifically renewed
by client)?  Of course, this will require cooperation between
newsreaders and news distribution packages...

At elevia, on a small scale, I experimented with part of the
concept (manually).  I get a small number of groups from rcorco
and an even smaller number from unimtl; E-mail is routed
through unimtl (I do not redistribute any group but I would be
willing to do so in such an adaptive environment as I propose). 
It is a simple and effective arrangement which prevents hogging
the phone lines.  E-mail moves fairly quickly, news only slightly
less so.  I get a more or less constant flow of news, as opposed
to long periods of starvation followed by 48 solid hours of force
feeding, most of which to be discarded for lack of timeliness or
lack of reading time on my part.  My lines are more often
available for e-mail traffic or for my own interactive work, to
and from the office.


Conclusion

I realize that live [un]subscription is not a feature we will
have during the coming week, but we could at least talk about
spreading the news load thinner and broader.  This would multiply
the connections but reduce contiguous line occupation time. 
Smaller sites could take over some redistribution duties.  Larger
sites would not waste resources on (in relative terms) useless
data.  Browsing the groups will be easier and it will not become
a burden to the administrators.  Resource usage will be more
efficient.  Any ideas?



--
 Alain - Canadian Patent #1310575
 [ ... usual disclaimer... ]