🚀 go-pugleaf

In article <Byv6Dr.2M9@mccc.edu> pjh@mccc.edu (Pete Holsberg) writes:
>
>Has anyone got a program/script that will indicate the number of news
>articles received today in each newgroup specified (and perhaps also the
>number of KB)??

After trying out several scripts/programs out there, I wrote one that
gives me a daily report like this:
-----
News statistics report for the last day:        current  (yesterday/average)
 
                 Total data traffic (in bytes): 48144644 (40593613/44112591)
                   Number of incoming articles: 17895    (15529/17373)
                   Number of active newsgroups: 1016     (996/972)
             Number of newsgroups read locally: 285      (335/314)
               Number of articles read locally: 3277     (3970/2443)
          Number of hosts reading news locally: 24       (30/19)
         Total number of connections to server: 252      (341/270)
 
Top ten active groups by data size: 
 
        (4230411)       alt.binaries.pictures.erotica.
        (2971875)       alt.binaries.pictures.misc.
        (888961)        alt.binaries.pictures.
        (871925)        news.lists.ps-maps.
        (754058)        alt.supermodels.
        (524354)        soc.culture.indian.
        (430942)        rec.sport.baseball.
        (402445)        soc.culture.china.
        (389716)        comp.sys.next.misc.
        (374807)        rec.arts.comics.misc.
 
Top ten active groups by number of articles: 
 
        (206)   rec.sport.baseball.
        (198)   soc.culture.indian.
        (187)   junk.
        (186)   control.
        (175)   rec.sport.football.pro.
        (154)   soc.motss.
        (151)   comp.sys.ibm.pc.games.
        (145)   rec.arts.tv.soaps.
        (139)   rec.sport.football.college.
        (134)   alt.games.video.classic.
-----
[...it also lists the top ten news-reading hosts by number of connections,
 and the ten most popular groups locally, but I'm not going to post those
 parts :-) ]

It should be fairly straight-forward, just set some vars at the top of
the script to adjust for local pathnames/preferences and look at the
"requirements". I've only tested it on my "leaf" node, a Sun Sparc
running 4.1.2, but I tried to only use standard Unix utils.

Feel free to hack it... I'd be interested in hearing of any improvements
made to it (I've already though of a few myself but have been too lazy 
to make them... :-) maybe if there's interest I'll keep updating it). Enjoy!
[and all you expert script-writers, forgive my crudity in advance 8^) 8^) ]

------------------- begin enclosed file

#!/bin/sh
#
# File: FX_stats (version 1.01)
#
# A news statistics report generator (slow and crude, but it works!).
# ...I never did get newsreport/statdaily to work right, so I borrowed
# some ideas from what it was supposed to do, got some other ideas from
# looking at the nntplog output, and made it generate reports a la
# David Letterman's "top ten" lists :-)
#                                                       - 11/92 FX
#
# This should run at nights as a cron job. The first time the program is 
# run, put seven zeroes in a single line on the file $statslog. You can
# delete this line after the first day.
#
# NOTE: this was developed on and "end" node. Might have to change a few
#	of the assumptions if this node were a feed for someone else.
#
# REQUIREMENTS:
#	- a "dummy" feed, like the one used by the newstat package:
#		(sys file entry):	statistics:all/all:f:.statistics
#	- copious logging turned on in nntp daemon (via syslog or equiv)

# set some vars:

newsdir="/var/spool/news"			# where are the news articles?
artlist="$newsdir/out.going/.statistics"	# the dummy feed file
artlist_work="/tmp/artlist.work"		# dummy feed file to work on
nntplog="/usr/lib/news/nntplog"			# the nntpd logfile
nntplog_work="/tmp/nntplog_work"		# nntp log file to work on
newsadmin="news"				# who to mail report to?
stats_report="/tmp/news_stats_report"		# output file to be mailed
statslog="/usr/lib/news/statslog"		# stats logfile
top_number=10					# show the top x of each list
top_text="ten"					# the top what? (in text)
days=`wc $statslog | awk '{print $1}'`		# how many days of stats data

# move the articles list file by sorting it and cd to news directory
# (want to move files to make sure we are working on static data)

sort $artlist > $artlist_work
rm -f $artlist
mv $nntplog $nntplog_work
touch $nntplog
cd $newsdir

# set all output to go to a file (report to be mailed)

exec > $stats_report 2>&1

### begin the report...

echo "News statistics report for the last day:        current  (yesterday/average)"
echo " "

### how much data came into our news server?

echo -n "                 Total data traffic (in bytes): "
x1=`awk '{s += $2} END {print s}' $artlist_work`
y1=`tail -1 $statslog | awk '{print $1}'`
a1=`awk '{s += $1} END {print s}' $statslog`
z1=`expr $a1 \/ $days`
echo "$x1 ($y1/$z1)"

### how many articles was that?

echo -n "                   Number of incoming articles: "
x2=`wc $artlist_work | awk '{print $1}'`
y2=`tail -1 $statslog | awk '{print $2}'`
a2=`awk '{s += $2} END {print s}' $statslog`
z2=`expr $a2 \/ $days`
echo "$x2	 ($y2/$z2)"

### how many active groups do we have? (groups with articles coming into them)

echo -n "                   Number of active newsgroups: "
x3=`awk '{print $1 "/" $2}' $artlist_work | awk -F/ '{ for (i=1;i<NF-1;i++) printf("%s.", $i)} {print " "}' | sort -u | wc | awk '{print $1}'`
y3=`tail -1 $statslog | awk '{print $3}'`
a3=`awk '{s += $3} END {print s}' $statslog`
z3=`expr $a3 \/ $days`
echo "$x3	 ($y3/$z3)"

### how many of those are actually read here?

echo -n "             Number of newsgroups read locally: "
x4=`grep group $nntplog_work | grep -v " exit " | awk '{print $8}' | sort -u | wc | awk '{print $1}'`
y4=`tail -1 $statslog | awk '{print $4}'`
a4=`awk '{s += $4} END {print s}' $statslog`
z4=`expr $a4 \/ $days`
echo "$x4	 ($y4/$z4)"

### and how many articles were read?

echo -n "               Number of articles read locally: "
x5=`grep " exit " $nntplog_work | grep -v "0 articles" | awk '{s += $8} END {print s}'`
y5=`tail -1 $statslog | awk '{print $5}'`
a5=`awk '{s += $5} END {print s}' $statslog`
z5=`expr $a5 \/ $days`
echo "$x5	 ($y5/$z5)"

### how many different machines are connecting here to read news?

echo -n "          Number of hosts reading news locally: "
x6=`grep group $nntplog_work | grep -v " exit " | awk '{print $6}' | sort -u | wc | awk '{print $1}'`
y6=`tail -1 $statslog | awk '{print $6}'`
a6=`awk '{s += $6} END {print s}' $statslog`
z6=`expr $a6 \/ $days`
echo "$x6	 ($y6/$z6)"

### total number of local connections to read news

echo -n "         Total number of connections to server: "
x7=`grep " connect" $nntplog_work | wc | awk '{print $1}'`
y7=`tail -1 $statslog | awk '{print $7}'`
a7=`awk '{s += $7} END {print s}' $statslog`
z7=`expr $a7 \/ $days`
echo "$x7	 ($y7/$z7)"

#
# Add the above numbers to our stats logfile so we can use them tomorrow
#

echo "$x1 $x2 $x3 $x4 $x5 $x6 $x7" >> $statslog

#
# Now for the "top ten" lists. Actually, they are sorted lists with only
# the top $top_number lines being shown (default=10, can be changed above)
#

### generate a per-group usage list (by data size)

echo " "
echo "Top $top_text active groups by data size: "
echo " "
awk '{print $1 "/" $2}' $artlist_work | awk -F/ '{ for (i=1;i<NF-1;i++) printf("%s.", $i)} {print " " $NF}' | sort | awk '{if ($1 != prev) {print s " " prev; s=$2; prev=$1} else {s += $2}}' | sort -rn | awk '{print "	(" $1 ")	" $2 }' | head -$top_number

### generate a per-group usage list (by number of articles)

echo " "
echo "Top $top_text active groups by number of articles: "
echo " "
awk '{print $1 "/" $2}' $artlist_work | awk -F/ '{ for (i=1;i<NF-1;i++) printf("%s.", $i)} {print " "}' | sort | uniq -c | sort -rn | awk '{print "	(" $1 ")	" $2 }' | head -$top_number

### generate a list of top newsreaders

echo " "
echo "Top $top_text news-reading hosts by number of connections: "
echo " "
grep connect $nntplog_work | awk '{print $6}' | sort | uniq -c | sort -rn | awk '{print "	(" $1 ")	" $2 }' | head -$top_number

### generate a list of most popular (by local readership) groups

echo " "
echo "The $top_text most popular groups here: "
echo " "
grep group $nntplog_work | grep -v " exit " | awk '{print $8}' | sort | uniq -c | sort -rn | awk '{print "	(" $1 ")	" $2 }' | head -$top_number

#
# clean up temp files and mail off report to news admin before exiting
#

mv -f $artlist_work $artlist_work.bak
mv -f $nntplog_work $nntplog_work.bak

Mail -s "NEWS - daily stats report" $newsadmin < $stats_report
rm -f $stats_report

exit

---------------- end included file
-- 
      Francisco X DeJesus  ----- S A I C -----  dejesus@chinalake.navy.mil
 ------------------------------------------------------------------------------
  * disclaimer: Opinions expressed here are mine. Typos and errors are yours *
 "Duck Season!" "Rabbit Season!" "...rabbit season." "It's Duck Season! SHOOT!"