🚀 go-pugleaf

Archive-name: newsstats.awk
Submitted-by: Greg A. Woods <woods@robohack.UUCP>
Version: #ident	"@(#)cnews.Local:newsstats	1.2	92/02/24 20:58:06 (woods)"

[ Grrr!  -- There's a major bug in the newsstats.awk I posted the
other day.  It doesn't print out the data for the "last" system in the
report.  I'm doing a complete re-post, along with a driver script.  ]

In article <137964@becker.UUCP> bdb@becker.UUCP (Bruce Becker) writes:
> In article <1992Feb23.045738.17803@robohack.UUCP> woods@robohack.UUCP (Greg A. Woods) writes:
> |This script has not been tested with B News.  In theory it should work
> |OK, since those fields of the log file which are looked at are
> |supposedly the same in the C New log file.
> 
> 	Uhhh... you've forgotten, haven't you  8^)...

I've forgotten far less than I want to forget about B News.... ;-)

> 	BTW, I have the feeling that "i" and "s" records
> 	are not really in the "accepted" category. As I
> 	recall, "i" records show the message-id of an
> 	article which is sent to a remote site in response
> 	to having accepted its "ihave" message; they're
> 	really an "i-want" record.  "s" records show the
> 	message-id of an article sent to a remote site as
> 	a result of having accepted its "sendme" message -
> 	more or less a "send-to".  Anyhow this may all be
> 	different when the new ihave/sendme stuff arrives
> 	on our doorsteps...

This makes sense...  But I didn't change the report headers, since
I couldn't come up with any more meaningful 4-char names for these
fields.  Any suggestions?

> 	You might want to have the patches by Larry Blair
> 	which give "c" & type for control messages, and "f"
> 	for failed cancels, etc. They make for a richer log
> 	format and include the path for duplicates. "Don't
> 	leave /usr/home without it"...

No thanks....  I don't that much can be gained by slowing down
relaynews to this stuff in the log.  Regardless, I'll leave log-file
format and content to Henry and Geoff.

As has also been pointed out to me, and, with my memory sufficiently
jogged, I in recall that I once knew this too, the "local" entries
with "remote" message-id's are "fake" articles used to hold place in
the history file for the articles who's cancels precede them (thus
causing the "canceled" article to be rejected as a "duplicate").

This is rather groty too, since it's nearly impossible to tell in a
portable way if the message-id's hostname part matches $4, thus if
it's a "faked" article or not.  It would be much simpler if the record
contained a different log flag ($5), such as 'c', to indicate the
special status of this record.

Ok, here's the new version, in a shar file!

------ cut here ------
#! /bin/sh
# This is a shell archive.  Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file".  To overwrite existing
# files, type "sh file -c".  You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g..  If this archive is complete, you
# will see the following message at the end:
#		"End of shell archive."
# Contents:  newsstats.sh stats.awk
# Wrapped by woods@robohack on Mon Feb 24 21:18:01 1992
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'newsstats.sh' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'newsstats.sh'\"
else
echo shar: Extracting \"'newsstats.sh'\" \(1192 characters\)
sed "s/^X//" >'newsstats.sh' <<'END_OF_FILE'
X#! /bin/sh
X: || exec /bin/sh $0 $argv:q
X# uncomment the following if you have trouble with older ksh's on USG systems
X#(alias) >/dev/null 2>&1 && exec /bin/sh "$0" ${1+"$@"}
X#
X#ident	"@(#)cnews.Local:newsstats.sh	1.2	92/02/24 20:58:06 (woods)"
X#
X#	newsstats - daily news statistics gathering
X#
X
X# =()<. ${NEWSCONFIG-@<NEWSCONFIG>@}>()=
X. ${NEWSCONFIG-/usr/lib/news/bin/config}
X
XPATH=$NEWSCTL/bin:$NEWSBIN/maint:$NEWSBIN:$NEWSPATH ; export PATH
Xumask $NEWSUMASK
X
X#
X# NOTE:  The newsstats.awk script is invariably quite complex.
X# It is hightly recommended that you put a link to the fastest version
X# of awk that you can your hands on into $NEWSBIN.  It will then be
X# picked up by any scripts in news that need it.
X#
X# In particular it is *highly* recommended that you get Mike Brennan's
X# <brennan@boeing.com> mawk from your neighbourhood comp.sources archive.
X# It's usually twice as fast as "old" awk, many times as fast as nawk,
X# and orders of magnitude faster than FSF's GNU version gawk.
X# Well maybe that's exaggerating, but you get the point!
X#
X
Xif [ $# -gt 1 ] ; then
X	awk -f $NEWSBIN/maint/newsstats.awk ${1+"$@"}
Xelse
X	awk -f $NEWSBIN/maint/newsstats.awk $NEWSCTL/log
Xfi
X
Xexit $?
END_OF_FILE
if test 1192 -ne `wc -c <'newsstats.sh'`; then
    echo shar: \"'newsstats.sh'\" unpacked with wrong size!
fi
# end of 'newsstats.sh'
fi
if test -f 'stats.awk' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'stats.awk'\"
else
echo shar: Extracting \"'stats.awk'\" \(6645 characters\)
sed "s/^X//" >'stats.awk' <<'END_OF_FILE'
X#!/bin/awk -f
X#
X#ident	"@(#)Local:stats.awk	1.1.1.2	92/02/24 20:58:09 (woods)"
X#
X#	newsstats.awk -- by Ross Ridge (ross@contact.uucp) Public Domain
X#
X# woods@robohack.UUCP - Mon Feb 24 20:13:51 EST 1992
X#	fixed boo-boo in silly loop that prints report (lost last entry)
X#	used some mnemonic variables for blank fields.
X#	(I was tempted to re-paragraph it too, but refrained.... :-)
X#
X# woods@robohack.UUCP - Sat Feb 22 21:38:51 EST 1992
X#	fixed for mawk-1.1 (print needed "" to print blank line in END)
X#	added %TAcpt and %Bad columns
X#
X# Ross Ridge:
X#	fixed for old awks and ihave/sendme
X#	added "Duplct" field and optimized things a bit
X#	fixed ihave/sendme again, added some Mark Moraes' fixes
X#	may not work with old awks anymore...  (yup, does!)
X#
X
X# for older awks...
XBEGIN	       {
X		why[""]=""
X		badngs[""]=""
X	       }
X#
X	       {
X		lines++
X	       }
X#
X# These are really i-want, rather than i-have?
X#
X$5 == "i"      {
X#
X# leap of faith: we assume the previous line in the log file
X# was the (accepted) article that generated this line
X#
X		if (lastsite == "" || accepted[lastsite] < 1)
X			lastsite = "unknown?!?"
X		else
X			accepted[lastsite]--
X		ihave[lastsite]++
X               }
X#
X# These are really sent-to, rather than send-me?
X#
X$5 == "s"      {
X		if (lastsite == "" || accepted[lastsite] < 1)
X			lastsite = "unknown?!?"
X		else
X			accepted[lastsite]--
X		sendme[lastsite]++
X	       }
X#
X$5 == "+" || $5 == "i" || $5 == "s" {
X		accepted[$4]++
X		name[$4] = $4
X		lastsite = $4
X#
X# this loop should merge duplicates before counting....
X#
X		for (i = 7; i <= NF; i++) {
X			sent[$i]++
X			name[$i] = $i
X		}
X		next
X               }
X#
X$5 == "-"      {
X		name[$4] = $4
X		if ($7 == "duplicate" && NF == 7) {
X			duplicate[$4]++
X		} else {
X			rejected[$4]++
X			if ($6 ~ /^<.*>$/)
X				i = 7
X			else
X				i = 6
X			s = $i
X			for(i++; i <= NF; i++)
X				s = s " " $i
X			why[s]++
X		}
X		next
X               }
X#
X$5 == "j"      {
X		junked[$4]++
X		name[$4] = $4
X		badngs[$NF]++
X		next
X	       }
X#
X               {
X		unknown++
X	       }
X#
XEND	       {
X		if (lines == unknown) {
X			if (lines == "")
X				print "empty log file"
X			else
X				print "no valid lines in input"
X			exit 1
X		}
X#
X# Do a simple O(n^2) sort on the site names.
X# I've always wondered how to sort in awk.
X#
X		namecount = 0
X		for (i in name) {
X			highest = name[i]
X			for (j in name)
X				if (name[j] > highest)
X					highest = j
X			names[namecount++] = highest
X			name[highest] = ""
X		}
X		atotal = dtotal = rtotal = jtotal = itotal = stotal = senttotal = 0
X		for (ctmp = namecount - 1; ctmp >= 0; ctmp--) {
X			s = names[ctmp]
X			arttotal[s] = badtotal[s] = 0
X			if (accepted[s] > 0) {
X				atotal += accepted[s]
X				arttotal[s] += accepted[s]
X			}
X			if (duplicate[s] > 0) {
X				dtotal += duplicate[s]
X				arttotal[s] += duplicate[s]
X				badtotal[s] += duplicate[s]
X			}
X			if (rejected[s] > 0) {
X				rtotal += rejected[s]
X				arttotal[s] += rejected[s]
X				badtotal[s] += rejected[s]	
X			}
X			if (junked[s] > 0) {
X				jtotal += junked[s]
X				arttotal[s] += junked[s]
X				badtotal[s] += junked[s]
X			}
X			if (ihave[s] > 0) {
X				itotal += ihave[s]
X				arttotal[s] += ihave[s]
X			}
X			if (sendme[s] > 0) {
X				stotal += sendme[s]
X				arttotal[s] += sendme[s]
X			}
X			if (sent[s] > 0)
X				senttotal += sent[s]
X		}
X#
X# I've trimed the site name down to 14 characters in order to fit new columns
X# Some sites seem to want to use their full domain name, especially if they
X# are using NNTP.  The best solution is to get a wider terminal (window) and
X# change all occurances of '14' to something larger....
X#
X#		print "123456789 123456789 123456789 123456789 123456789 123456789 1123456789 1234567890"
X		header = "Accept %TAcpt Duplct Reject Junked  % Bad I've S'me  Total   Xmit"
X		sevenblanks = "       "
X		fiveblanks = "     "
X		printf("%-14.14s %s\n", "Sitename", header)
X		print "--------------------------------------------------------------------------------"
X		thetotal = 0
X		for (ctmp = namecount - 1; ctmp >= 0; ctmp--) {
X			s = names[ctmp]
X			percent = 0
X			printf("%-14.14s ", s)
X			if (accepted[s] == 0)
X				printf(sevenblanks)
X			else {
X				printf("%6d ", accepted[s])
X				percent = (accepted[s] * 100.0) / atotal
X			}
X			if (percent > 0)
X				printf("%5.1f%% ", percent)
X			else
X				printf(sevenblanks)
X			if (duplicate[s] == 0)
X				printf(sevenblanks)
X			else
X				printf("%6d ", duplicate[s])
X			if (rejected[s] == 0)
X				printf(sevenblanks)
X			else
X				printf("%6d ", rejected[s])
X			if (junked[s] == 0)
X				printf(sevenblanks)
X			else
X				printf("%6d ", junked[s])
X			if (arttotal[s] == 0)
X				percent = 0
X			else
X				percent = (badtotal[s] * 100.0) / arttotal[s]
X			if (percent > 0)
X				printf("%5.1f%% ", percent)
X			else
X				printf(sevenblanks)
X			if (ihave[s] == 0)
X				printf(fiveblanks)
X			else
X				printf("%4d ", ihave[s])
X			if (sendme[s] == 0)
X				printf(fiveblanks)
X			else
X				printf("%4d ", sendme[s])
X			if (arttotal[s] == 0)
X				printf(sevenblanks)
X			else {
X				printf("%6d ", arttotal[s])
X				thetotal += arttotal[s]
X			}
X			if (sent[s] == 0) 
X				printf("\n")
X			else
X				printf("%6d\n", sent[s])
X		}
X		printf("\n%14s %s\n", "", header)
X		printf("%14s ", "Totals:")
X		if (atotal == 0)
X			printf(sevenblanks)
X		else
X			printf("%6d ", atotal)
X		printf(sevenblanks)
X		if (dtotal == 0)
X			printf(sevenblanks)
X		else
X			printf("%6d ", dtotal)
X		if (rtotal == 0)
X			printf(sevenblanks)
X		else
X			printf("%6d ", rtotal)
X		if (jtotal == 0)
X			printf(fiveblanks)
X		else
X			printf("%6d ", jtotal)
X		if (thetotal > 0)
X			percent = ((dtotal + rtotal + jtotal) * 100.0) / thetotal
X		if (percent > 0)
X			printf("%5.1f%% ", percent)
X		else
X			printf(sevenblanks)
X		if (itotal == 0)
X			printf(fiveblanks)
X		else
X			printf("%4d ", itotal)
X		if (stotal == 0)
X			printf(fiveblanks)
X		else
X			printf("%4d ", stotal)
X		if (thetotal == 0)
X			printf(sevenblanks)
X		else
X			printf("%6d ", thetotal)
X		if (senttotal == 0)
X			printf("\n")
X		else
X			printf("%6d\n", senttotal)
X#
X# You may want to change the following back to provide a more complete
X# report listing the names of the "bad" newsgroups.
X#
X		print ""
X## use this:
X#		print "Junked Newsgroups:"
X#		for (s in badngs)
X#			if (s != "")
X#				printf("\t%s: %d\n", s, badngs[s])
X## or this:
X		for (s in badngs)
X			if (s != "")
X				badngtotal += 1
X		print "Total # of Junked Newsgroups: " badngtotal
X##
X#
X# You also might want to remove this part if you get a lot of articles
X# rejected for reasons other than being duplicates.
X#
X		print ""
X		print "Why articles were rejected:"
X		for (s in why)
X			if (s != "")
X				printf("\t%s: %d\n", s, why[s])
X		print ""
X##
X		if (unknown != "")
X			printf("\nUnknown lines: %d\n", unknown)
X	       }
END_OF_FILE
if test 6645 -ne `wc -c <'stats.awk'`; then
    echo shar: \"'stats.awk'\" unpacked with wrong size!
fi
# end of 'stats.awk'
fi
echo shar: End of shell archive.
exit 0
-- 
						Greg A. Woods

woods@robohack.UUCP woods@Elegant.COM  VE3-TCP	UniForum Canada & Elegant Comm.
(416) 443-1734 [home]  (416) 595-5425 [work]	Toronto, Ontario; CANADA