🚀 go-pugleaf

What are some typical stats for news propagation?  I'm now getting
news from Netcom and here are the stats for the delay in HOURS from
posting date to arrival at my site:

13949 samples, mean 46.526327
Std. deviation: 23.344937
High=163 Low=1
Range of variance: 162

So the average delay from posting to delivery is 46.5 hours.

For articles originating on netcom, the numbers are:

478 samples, mean 33.420173
Std. deviation: 12.098320
High=110 Low=5
Range of variance: 105

This implies that it takes 33 hours on average for an article to get
queued and sent out once arriving at netcom, with worst case taking more
than 4 days!  I have a UUCP link to Netcom and I poll very frequently,
on a modem that is used for little to nothing else, so I am not adding
a delay of more than an hour on my end.

I'm curious to see what other services can do in terms of news
propagation delays.  Somebody must be able to beat what I'm getting with
Netcom for a reasonable price.  I'm trying to get a PageSat link working,
but I'm not having much success with that.  And people are telling me
that pageSat is several days behind too.

I've attached a couple of short programs below to calculate news delivery
times. These are quick hacks. They are shell scripts and all the forks
are pretty taxing on the system.  If anybody is willing, please run these
and send me the results.  Run them niced. It takes several hours to run
(especially if you have a large news spool).  The first script is
"newslag.sh".  It requires the "getdate" program from C-News. If you
don't have that, mail me and I can send you the sources. Change the
settings of history (the History file), spooldir (the top of your News
spool), and the GETDATE (the C News getdate pgm).

The "newslag.sh" script produces output lines with two fields, the
message-id and the delay between posting and arrival, in seconds.  I
then use the simple little generic awk program "sumstats.awk" attached
below to sum the totals and generate some basic stats.

Send the output from the "newslag.sh" run to the "sumstats.awk", e.g.
 
   nice sh newslag.sh > /tmp/newslag.out
   awk '{print $2/3600}' /tmp/newslag.out | awk -f sumstats.awk
 
You could hack the newslag.sh to only look at the first 1000 articles
(change "cat" at the top to "head -1000") to make it run faster and give
you a sample for just 1000 articles.  Once the newslag.sh runs to 
generate the delays, you can run various filters to generate numbers
for delays from specific domains/hosts etc. E.g.
 
   grep '@netcom' /tmp/newslag.out | awk '{print $2/3600}' | awk -f sumstats.awk
 
Please send me the results of your newsfeed.

Thanks.
 
--- CUT HERE ------- CUT HERE ------- CUT HERE ------- CUT HERE ----
#!/bin/sh
# This is a shell archive (produced by shar 3.49)
# To extract the files from this archive, save it to a file, remove
# everything above the "!/bin/sh" line above, and type "sh file_name".
#
# made 05/02/1994 20:11 UTC by david@bdt.com
# Source directory /home/david
#
# existing files will NOT be overwritten unless -c is specified
#
# This shar contains:
# length  mode       name
# ------ ---------- ------------------------------------------
#    625 -rwxr-xr-x newslag.sh
#    598 -rw-r----- sumstats.awk
#
# ============= newslag.sh ==============
if test -f 'newslag.sh' -a X"$1" != X"-c"; then
	echo 'x - skipping newslag.sh (File already exists)'
else
echo 'x - extracting newslag.sh (Text)'
sed 's/^X//' << 'SHAR_EOF' > 'newslag.sh' &&
#!/bin/sh
#
# Look at the history file and the article dates
# and print the delivery times for each article
# in seconds
#
history=/usr/lib/news/history
spooldir=/usr/spool/news
GETDATE=/usr/lib/newsbin/getdate
cat $history | awk '{ if ($NF > 2) print $1,$2,$3 }' | while read id etime art
do
X	etime=`echo $etime | cut -f1 -d'~'`
X	file=`echo $art | tr \. / | awk '{ printf("'"$spooldir"'/%s\n", $1) }'`
X	if [ -f $file ] ; then
X		atime="`grep '^Date: ' $file | head -1 | cut -c6-`"
X		atime=`$GETDATE "$atime"`
X		if [ $? -eq 0 -a "$atime" != "" ] ; then
X			delta=`expr $etime - $atime`
X			echo $id $delta
X		fi
X	fi
done
Xexit 0
SHAR_EOF
chmod 0755 newslag.sh ||
echo 'restore of newslag.sh failed'
Wc_c="`wc -c < 'newslag.sh'`"
test 625 -eq "$Wc_c" ||
	echo 'newslag.sh: original size 625, current size' "$Wc_c"
fi
# ============= sumstats.awk ==============
if test -f 'sumstats.awk' -a X"$1" != X"-c"; then
	echo 'x - skipping sumstats.awk (File already exists)'
else
echo 'x - extracting sumstats.awk (Text)'
sed 's/^X//' << 'SHAR_EOF' > 'sumstats.awk' &&
#
# Calculate mean average, low, high, std. deviation, range of variance
# For a set of input values (one per line)
X
BEGIN {
X	n = 0;
X	sum = 0;
}
{
X	sum += $1;
X	val[n] = $1;
X	if (n == 0) {
X		lo = $1;
X		hi = $1;
X	}
X	else {
X		if ($1 > hi)
X			hi = $1;
X		if ($1 < lo)
X			lo = $1
X	}
X	n++;
} END {
X	avg = sum / n;
X	printf "%d total, %d samples, mean %f\n", sum, n, avg;
X	dt2 = 0;
X	for (i = 0; i < n; i++) {
X		dt2 += (val[i] - avg) * (val[i] - avg);
X	}
X	stdev =  sqrt(dt2 / (n-1));
X	printf "Std. deviation: %f\n",  stdev;
X	printf("High=%d Low=%d\n", hi, lo);
X	printf("Range of variance: %d\n", hi - lo);
}
SHAR_EOF
chmod 0640 sumstats.awk ||
echo 'restore of sumstats.awk failed'
Wc_c="`wc -c < 'sumstats.awk'`"
test 598 -eq "$Wc_c" ||
	echo 'sumstats.awk: original size 598, current size' "$Wc_c"
fi
exit 0
-- 
David Beckemeyer (david@bdt.com)	| "Beer!  Now there's a temporary
Beckemeyer Development			|   solution."  -- Homer Simpson
P.O. Box 21575, Oakland, CA 94620	| WWW: <http://www.bdt.com/>