🚀 go-pugleaf

This is probably the next best thing to a free space based expire.
It scans the news history file and compares it against the used and
free space to estimate the correct value to pass to expire.  It
correctly handles the case where you were down for a week and then got
it all in one day as well as compensating for the gradual increase in
volume and seasonal variations.  Output is to the tenth of a day.
Also included is a program that summarizes the ages of the articles.

Your history dates need to be in numeric form (B news 2.11.19 or C
news).  You will probably have to adjust the code that gets the free
space for your OS.  If people want to send me their versions of the
"dfree" code I will collect them for future releases.

Details in the README and Makefile.  I have been using it for several
weeks now and it correctly frees just enough space to last me to the
next expire run.


===CUT HERE===
echo Extracting README
sed 's/^X//' >README <<'EOF'
XThere are two programs here designed for use with usenet.  They both
Xparse the history file but produce different outputs.  The first was
X"newsdates" written by David Schmidt <davids@isc-br.ISC-BR.COM>.
X(Though I have modified it so bugs are probably mine.)
X
XNewsdates produces a histogram of the history file.  It expects the file
Xto be in recent format (B 2.11 or C).  Aside from other uses newsdates
Xis useful for figuring out how many days for expire will expire how many
Xarticles.  
X
XI (Jerry Aguirre <jerry@atc.olivetti.com>) modified newsdates, added a
Xspace check, and had it calculate the B news expire threshold
Xautomatically.  It writes this to stdout so it can be captured by the
Xscript that runs expire.  The new program is called "expiredays".
X
XThe routine "dfree" is separate in the expection that it is site
Xdependant and will have to be modifed for various OS that it will run
Xon.  The supplied version is for a Sun OS 4.1.1b environment.
X
XI use expiredays in my shell script in this way:
X
X    edays=`expiredays -f 1.4`
X    days=28
X    case "$edays" in
X	[12][0-9]) days=$edays ;;
X	[12][0-9].[0-9]) days=$edays ;;
X    esac
X    echo "expire -e $days"
X    expire -a `cat /usr/local/lib/news/archivelist` -e $days
X
XThe "case" is a sanity check just in case something ever goes wrong with
Xthe expiredays program.  It allows expire to range from 10 to 29 days.
XAdjust as needed.  Or you could just use:
X
X	expire -e `expiredays`
X
XThe expiredays program is now enhanced to output in 10ths of a day.
XFor example it could now output 12.9 instead of having to round down to
X12 days.  Modern B news expires can handle floating point expire times.
X
XArguments to expiredays are:
X	-a N	Number of days to average flow rate.  I.E. -a7 (the
X		default) will average the rate of the last 7 days.
X	
X	-e N	Max number of days to expire.  Even if there is enough
X		space to keep news longer.  (Default=28)
X	
X	-f N	Fudge factor.  I.E. -f 1.5 means free 1.5 days worth
X		of disks space.  A days disk space is from the average
X		calculated using the -a value.  The extra 0.5 is to
X		allow for today being a bussier than normal day.  If you
X		expire more than once a day you should change this.
X	
X	-m N	Minimum number of days to expire.  Even if there is not
X		enough space.  (Default=7)
X
XAll numeric arguments are handled as floating point numbers.  For
Xexample you may specify "-m 1.5" to keep a minimum of 1.5 days of news.
X	
XInternally expiredays calculates the number of articles/day from the
Xhistory file.  It then compares this to the space used in spool/news to
Xcalculate the average space used by each article.  Finally it calculates
Xhow many articles have to be expired based on the free space and the
Xhistogram of articles and translates this into a number of days.
X
XIf the disk space check returns number of inodes then that is also used
Xto calculate with the smaller limit of space or inodes being used.  My
Xspool/news is NFS mounted so I can not use that feature.
X
XThe biggest loophole is that only the average usage is calculated.  A
Xday that got a lot of large postings could throw off the calculation.
XBut there are usually enough articles to average out and my free space
Xruns down to a few Meg just before the next expire runs.
X
XCaution, freeing space by manually removing articles (find/rm) will
Xthrow off the calculations!  Expiredays will think the articles are
Xstill there and that removing them will free space.  If they have been
Xmanually removed expireing them will not free any space.
X
XThe program could stat each article to verify its existance and get an
Xexact size.  I consider that more overhead than is necessary at this
Xpoint.
X
XTo install first edit "Makefile" to define the path to history, spool,
Xand where you want it installed.  Then type "make" to build the
Xexecutables.  Typing "make test" will run a test program to check out
Xthe dfree routine and also run expiredays to verify its output.  Typing
X"make install" will install.   Edit your expire script/crontab entry
Xto make use of expiredays.
EOF
echo Extracting Makefile
sed 's/^X//' >Makefile <<'EOF'
XHISTORY="/usr/local/lib/news/history"
XSPOOL="/var/spool/news/."
XBIN=/usr/local/newsbin
X
Xall: expiredays newsdates
X
Xtest: dftest expiredays
X	@echo ''
X	@echo 'Testing dftest, verify output against df of spool/news'
X	@echo 'Note that df on your system may report in K instead of blocks'
X	df ${SPOOL}
X	./dftest
X	@echo ''
X	@echo 'Testing expiredays, verify that output is a'
X	@echo 'reasonable number of days to expire news in'
X	./expiredays
X
Xinstall: ${BIN}/newsdates ${BIN}/expiredays
X
Xexpiredays: expiredays.o dfree.o
X	cc -o expiredays expiredays.o dfree.o
X
Xexpiredays.o: expiredays.c Makefile
X	cc -c -O '-DHISTORY=${HISTORY}' '-DSPOOL=${SPOOL}' expiredays.c
X
Xedaynew: edaynew.o dfree.o
X	cc -o edaynew edaynew.o dfree.o
X
Xedaynew.o: edaynew.c
X	cc -c -O edaynew.c
X
Xdfree.o: dfree.c
X	cc -c -O dfree.c
X
Xdftest.o: dftest.c Makefile
X	cc -c -O '-DSPOOL=${SPOOL}' dftest.c
X
Xdftest: dftest.o dfree.o
X	cc -o dftest dftest.o dfree.o
X
Xnewsdates: newsdates.c Makefile
X	cc -o newsdates -O '-DHISTORY=${HISTORY}' newsdates.c
X
X${BIN}/newsdates: newsdates
X	cp -p newsdates $@
X
X${BIN}/expiredays: expiredays
X	cp -p expiredays $@
EOF
echo Extracting expiredays.c
sed 's/^X//' >expiredays.c <<'EOF'
X#include <stdio.h>
X#include <sys/types.h>
X#include <ctype.h>
X
X/* expiredays: calculate number of days for B news expire.
X *
X * version 1.1 20Feb92 Jerry Aguirre <jerry@atc.olivetti.com>
X *
X *	Reads the history file to figure out the number of articles
X * due to expire on each day and compares this to the used and free
X * disk space to estimate the expire value required to free enough
X * space.  Can be run as:
X *		expire -e `expiredays`
X */
X
X#undef  DEBUG
X#undef  DEBUG2
X
X/* default parameters, can be overridden from command line */
X#define EFREE   1.5	/* days worth of news to keep free */
X#define EMAXDAYS 28	/* max expire period (days) returned */
X#define EMINDAYS  7	/* min expire period (days) returned */
X#define AVGDAYS   7	/* period to average news usage across */
X
X#ifndef HISTORY
X#define HISTORY "/usr/lib/news/history"
X#endif
X
X#ifndef SPOOL
X#define SPOOL "/usr/spool/news/."
X#endif /* SPOOL */
X
X
X#define ONEDAY (60*60*24)
X#define GRACE (60*60)
X#define MAXDAY 90
X#define BUFSIZE 1024
X/* currently we measure in tenths of a day */
X#define UNITDAY 10
X#define ONEUNIT (ONEDAY / UNITDAY)
X#define MAXUNIT (MAXDAY * UNITDAY)
X
Xextern char *index();
X
Xmain(argc, argv)
X  int	argc;
X  char	*argv[];
X{
X    int arg, i, maxdays, mindays, avgdays;
X    double freedays, f, atof();
X    time_t recvtime, exptime, today;
X    char buffer[1024], *dates, *edate, *articles;
X    int ndays, expday;
X    int nunits, expunit;
X#ifdef DEBUG
X    int ttyo;
X#endif /* DEBUG */
X    long total, bogus, unit[MAXUNIT][4];
X#define HISTONLY 0
X#define ARTICLES 1
X#define EXPIRES  2
X#define CANCELLED 3
X    FILE *fp;
X
X    maxdays = EMAXDAYS;
X    mindays = EMINDAYS;
X    freedays = EFREE;
X    avgdays = AVGDAYS;
X    for (arg = 1; arg < argc; arg++) {
X	if (argv[arg][0] == '-') {
X	    switch (argv[arg][1]) {
X	    case 'a':	/* number of days to average flow rate across */
X		if (isdigit(argv[arg][2]))
X		    i = atoi(&argv[arg][2]);
X		else if (((arg + 1) < argc) && isdigit(argv[arg+1][0]))
X		    i = atoi(&argv[++arg][0]);
X		else
X		    i = -1;
X		if (i > 0) avgdays = i;
X		break;
X
X	    case 'e':	/* max days to expire at */
X		if (isdigit(argv[arg][2]))
X		    i = atoi(&argv[arg][2]);
X		else if (((arg + 1) < argc) && isdigit(argv[arg+1][0]))
X		    i = atoi(&argv[++arg][0]);
X		else
X		    i = -1;
X		if (i > 0) maxdays = i;
X		break;
X
X	    case 'f':	/* Number of days worth of news to free */
X		if (isdigit(argv[arg][2]))
X		    f = atof(&argv[arg][2]);
X		else if (((arg + 1) < argc) && isdigit(argv[arg+1][0]))
X		    f = atof(&argv[++arg][0]);
X		else
X		    f = -1;
X		if (f > 0.1) freedays = f;
X		break;
X
X	    case 'm':	/* minumum number of days to expire */
X		if (isdigit(argv[arg][2]))
X		    i = atoi(&argv[arg][2]);
X		else if (((arg + 1) < argc) && isdigit(argv[arg+1][0]))
X		    i = atoi(&argv[++arg][0]);
X		else
X		    i = -1;
X		if (i > 0) mindays = i;
X		break;
X
X	    default:
X		fprintf(stderr, "%s: unknown option: %s\n",
X				argv[0], argv[arg]);
X		break;
X	    }
X	} else {
X	    fprintf(stderr, "%s: unknown option: %s\n",
X			    argv[0], argv[arg]);
X	}
X    }
X
X    if ((fp=fopen(HISTORY, "r")) == NULL) {
X	fprintf(stderr, "%s: error opening history file.\n", argv[0]);
X	exit(1);
X    }
X
X#ifdef DEBUG
X    ttyo = isatty(2);
X#endif /* DEBUG */
X    bogus = 0;
X    total = 0;
X    for (nunits = 0; nunits < MAXUNIT; nunits++) {
X	unit[nunits][HISTONLY] = 0;
X	unit[nunits][ARTICLES] = 0;
X	unit[nunits][EXPIRES] = 0;
X	unit[nunits][CANCELLED] = 0;
X    }
X    time(&today);
X    while (fgets(buffer, BUFSIZE, fp) != NULL) {
X	articles = dates = 0;
X	expday = -1;
X	expunit = -1;
X	dates = index(buffer, '\t');
X	if ((buffer[0] == '<') && dates &&
X	  isascii(*(dates+1)) && isdigit(*(dates+1))) {
X	    dates++;
X	    recvtime = atol(dates);
X	    edate = index(dates, '~');
X	    if (edate && isascii(*(edate+1)) && isdigit(*(edate+1))) {
X		edate++;
X		exptime = atol(edate);
X		expday = ((today + GRACE) - exptime) / ONEDAY;
X		expunit = ((today + GRACE) - exptime) / ONEUNIT;
X		if (expday >= MAXDAY)
X		    expday = MAXDAY - 1;
X		else if (expday < 0)
X		    expday = 0;
X		if (expunit >= MAXUNIT)
X		    expunit = MAXUNIT - 1;
X		else if (expunit < 0)
X		    expunit = 0;
X	    }
X	    ndays = ((today + GRACE) - recvtime) / ONEDAY;
X	    if (ndays >= MAXDAY)
X		ndays = MAXDAY - 1;
X	    else if (ndays < 0)
X		ndays = 0;
X	    nunits = ((today + GRACE) - recvtime) / ONEUNIT;
X	    if (nunits >= MAXUNIT)
X		nunits = MAXUNIT - 1;
X	    else if (nunits < 0)
X		nunits = 0;
X	    articles = index(dates, '\t');
X	    if (articles) {
X		articles++;
X		if ((*articles != '\n') && (*articles != '\0')) {
X		    if (0 == strcmp("cancelled\n", articles))
X			unit[nunits][CANCELLED]++;
X		    else {
X			/* has articles */
X			if (expday >= 0) unit[nunits][EXPIRES]++;
X			else unit[nunits][ARTICLES]++;
X			total++;
X		    }
X		} else
X		    unit[nunits][HISTONLY]++;
X	    } else
X		unit[nunits][HISTONLY]++;	
X	} else {
X	    bogus++;
X#ifdef DEBUG
X	    printf("*%s*\n", buffer);
X#endif /* DEBUG */
X	}
X    }
X
X    fclose(fp);
X
X#ifdef DEBUG
X    printf("\n");
X    if (bogus) printf("Bad history lines: %ld\n", bogus);
X    printf("Total articles = %ld\n", total);
X#ifdef DEBUG2
X    printf("Days history articles expires cancelled\n");
X    for (nunits = 0; nunits < MAXUNIT; nunits++) {
X	if (unit[nunits][HISTONLY] || unit[nunits][ARTICLES] ||
X	    unit[nunits][EXPIRES]  || unit[nunits][CANCELLED])
X	    printf("%4.1f %7ld %8ld %7ld %9ld\n",
X		((float)nunits / UNITDAY), unit[nunits][HISTONLY],
X		unit[nunits][ARTICLES], unit[nunits][EXPIRES],
X		unit[nunits][CANCELLED]);
X    }
X#endif /* DEBUG2 */
X#endif /* DEBUG */
X    {
X	long used, free, iused, ifree;
X	double art7, apd, spd, freeart, exp, nart;
X	double apu, spu;
X
X	if (dfree(SPOOL, &used, &free, &iused, &ifree) < 0) {
X		perror(argv[0]);
X		return 1;
X	}
X#ifdef DEBUG
X	printf("blocks %ld, free %ld, ifree %ld\n", used, free, ifree);
X	printf("avgdays = %d, Maxdays = %d, mindays = %d, freedays = %4.2f\n",
X		avgdays,      maxdays,      mindays,      freedays);
X#endif /* DEBUG */
X	art7 = 0;
X	for (nunits = 0; nunits < (avgdays * UNITDAY); nunits++)
X	    art7 += unit[nunits][ARTICLES] + unit[nunits][EXPIRES];
X	apd = art7 / avgdays;	/* average number of articles per day */
X	spd = (apd * used) / total; /* average blocks per day */
X	apu = art7 / (avgdays * UNITDAY); /* average number of articles per unit */
X	spu = (apu * used) / total; /* average blocks per unit */
X#ifdef DEBUG
X	printf("articles per day = %.0f, blocks per day = %.0f\n", apd, spd);
X	printf("articles per unit = %.0f, blocks per unit = %.0f\n", apu, spu);
X#endif /* DEBUG */
X	freeart = (free * apu) / spu;
X	if ((ifree >= 0) && (ifree < freeart)) freeart = ifree; /* inode limit */
X	nart = (apu * UNITDAY * freedays) - freeart;
X#ifdef DEBUG
X	printf("free art estimate %.0f, ", freeart);
X	if ((apu * UNITDAY * freedays) < freeart) 
X	    printf("Have room for %.0f more articles\n", -nart);
X	else
X	    printf("Need to expire %.0f articles\n", nart);
X#endif /* DEBUG */
X	exp = 0;
X	for (nunits = MAXUNIT - 1; nunits > 0; nunits--) {
X		exp += unit[nunits][ARTICLES];
X		if (exp >= nart) break;
X	}
X	if (nunits > (maxdays * UNITDAY)) nunits = (maxdays * UNITDAY);
X	if (nunits < (mindays * UNITDAY)) nunits = (mindays * UNITDAY);
X	printf("%4.1f\n", ((float)nunits / UNITDAY));
X    }
X    return 0;
X}
EOF
echo Extracting dfree.c
sed 's/^X//' >dfree.c <<'EOF'
X#include <sys/types.h>
X#include <sys/vfs.h>
X
X/* path is file system to test: ie "/var/spool/news/."
X * used and free are disk blocks.  Units not important as long as they
X * are consistant.  free does not include the BSD4.2 reserved space.
X * iused and ifree are inodes set to -1 and ignored if not available.
X */
Xdfree(path, used, free, iused, ifree)
X    char *path;
X    long *used, *free, *iused, *ifree;
X{
X	struct statfs buf;
X
X	/* this is for a sun OS, some require reading the superblock  */
X	if (statfs(path, &buf) < 0) return -1;
X	*used = buf.f_blocks - buf.f_bfree;
X	*free = buf.f_bavail;
X	if ((buf.f_files > 0) && (buf.f_ffree >= 0))
X		*iused = buf.f_files - buf.f_ffree;
X	else
X		*iused = -1;
X	*ifree = buf.f_ffree;
X	return 0;
X}
EOF
echo Extracting dftest.c
sed 's/^X//' >dftest.c <<'EOF'
X/* a stand alone program for testing the dfree module */
X
Xmain()
X{
X	int i;
X	long a, b, c, d;
X
X	i = dfree(SPOOL, &a, &b, &c, &d);
X
X	printf("%d=dfree(\"%s\", blocks-used=%ld, blocks-free=%ld,\n",
X	    i, SPOOL, a, b);
X	printf("         inodes-used=%ld, inodes-free=%ld)\n",
X	    c, d);
X
X	return i;
X}
EOF
echo Extracting newsdates.c
sed 's/^X//' >newsdates.c <<'EOF'
X#include <stdio.h>
X#include <sys/types.h>
X#include <ctype.h>
X
X/* newsdates: print a histogram of the history file listing number
X *		of articles sorted by age.
X */
X
X#ifndef HISTORY
X#define HISTORY "/usr/lib/news/history"
X#endif
X
X#undef DEBUG
X
X#define ONEDAY (60*60*24)
X#define GRACE (60*60)
X#define MAXDAY 365
X#define BUFSIZE 1024
X
Xextern char *index();
X
Xmain(argc, argv)
X  int	argc;
X  char	*argv[];
X{
X    time_t recvtime, exptime, today;
X    char buffer[1024], *dates, *edate, *articles;
X    int ndays, expday;
X#ifdef DEBUG
X    int last=(MAXDAY+1);
X    int ttyo;
X#endif /* DEBUG */
X    long bogus, day[MAXDAY][4];
X#define HISTONLY 0
X#define ARTICLES 1
X#define EXPIRES  2
X#define CANCELLED 3
X    FILE *fp;
X
X    if ((fp=fopen(HISTORY, "r")) == NULL) {
X	fprintf(stderr, "%s: error opening history file.\n", argv[0]);
X	exit(1);
X    }
X
X#ifdef DEBUG
X    ttyo = isatty(2);
X#endif /* DEBUG */
X    bogus = 0;
X    time(&today);
X    while (fgets(buffer, BUFSIZE, fp) != NULL) {
X	articles = dates = 0;
X	expday = -1;
X	dates = index(buffer, '\t');
X	if ((buffer[0] == '<') && dates &&
X	  isascii(*(dates+1)) && isdigit(*(dates+1))) {
X	    dates++;
X	    recvtime = atol(dates);
X	    edate = index(dates, '~');
X	    if (edate && isascii(*(edate+1)) && isdigit(*(edate+1))) {
X		edate++;
X		exptime = atol(edate);
X		expday = ((today + GRACE) - exptime) / ONEDAY;
X		if (expday >= MAXDAY)
X		    expday = MAXDAY - 1;
X		else if (expday < 0)
X		    expday = 0;
X	    }
X	    ndays = ((today + GRACE) - recvtime) / ONEDAY;
X	    if (ndays >= MAXDAY)
X		ndays = MAXDAY - 1;
X	    else if (ndays < 0)
X		ndays = 0;
X#ifdef DEBUG
X	    if (ttyo && (ndays != last)) {
X		fprintf(stderr, "\r%d  \r", ndays);
X		last = ndays;
X	    }
X#endif /* DEBUG */
X	    articles = index(dates, '\t');
X	    if (articles) {
X		articles++;
X		if ((*articles != '\n') && (*articles != '\0')) {
X		    if (0 == strcmp("cancelled\n", articles))
X			day[ndays][CANCELLED]++;
X		    else {
X			/* has articles */
X			if (expday >= 0) day[ndays][EXPIRES]++;
X			else day[ndays][ARTICLES]++;
X		    }
X		} else
X		    day[ndays][HISTONLY]++;
X	    } else
X		day[ndays][HISTONLY]++;	
X	} else {
X	    bogus++;
X	    printf("*%s*\n", buffer);
X	}
X    }
X
X    fclose(fp);
X
X#ifdef DEBUG
X    printf("\n");
X#endif /* DEBUG */
X    if (bogus) printf("Bad history lines: %ld\n", bogus);
X    printf("Days history articles expires cancelled\n");
X    for (ndays = 0; ndays < MAXDAY; ndays++) {
X	if (day[ndays][HISTONLY] || day[ndays][ARTICLES] ||
X	    day[ndays][EXPIRES]  || day[ndays][CANCELLED])
X	    printf("%4d %7ld %8ld %7ld %9ld\n",
X		ndays, day[ndays][HISTONLY], day[ndays][ARTICLES],
X		day[ndays][EXPIRES], day[ndays][CANCELLED]);
X    }
X}
EOF
exit 0