🚀 go-pugleaf

In article <2rfr8p$41t@netnews.upenn.edu> kirsten@ssc.sas.upenn.edu (Kirsten Starcher) writes:
>We have a large number of IBM mainframe 9-track tapes, both labelled
>and unlabelled, that we would like to transcribe to our RS/6000s.  Does
>anyone know of a script or program that we could use to make a dumb
>copy of the tapes to online storage? (Preserving record length and
>block size.)
>
>Failing that, would it be feasible to write something that looks at each
>block, determines its length, and writes the block length to disk along
>with the block itself?  What would be the best approach to this?
>
>Kirsten Starcher
>kirsten@ssc.sas.upenn.edu           kirsten@morgan.ucs.mun.ca

I am not following you here - "transcribe" to what?  The basic
"otherscsi" magnetic tape driver on AIX 3.2.5 should have no trouble
reading from a generic SCSI 9-track tape drive.  There is software
around that does labeled tapes.  The AT&T Unix Toolchest has something,
for example.  If you have a lot of tapes, you probably don't want to
rewrite them all.  The EBCDIC conversion can be done on the fly.  Or
maybe you want to convert everything into tar format?

Regards,

Lester

--
A. Lester Buck
Cornell Theory Center
lbuck@tc.cornell.edu

In article <2rfr8p$41t@netnews.upenn.edu>,
Kirsten Starcher <kirsten@ssc.sas.upenn.edu> wrote:
>We have a large number of IBM mainframe 9-track tapes, both labelled
>and unlabelled, that we would like to transcribe to our RS/6000s.  Does
>anyone know of a script or program that we could use to make a dumb
>copy of the tapes to online storage? (Preserving record length and
>block size.)

Attached is a program that transcribes a tape to a disk file
(tape2disk.c) and a program that does the reverse (disk2tape.c).

These programs were written to use the async I/O capability of
ConvexOS in order to get higher performance by overlapping disk
and tape I/O.  You'll have to change a few lines to adapt to AIX.

Regards,
Ed Hamrick

/*
 * tape2disk - Utility to copy a tape to a disk file (or pipe)
 *
 * This program copies an entire mag tape to a disk file (or pipe)
 * in a way that the tape can be completely reconstructed from the
 * disk file.
 *
 * The disk format is relatively simple.  For each tape record, a 4-byte
 * record length is stored, followed by the data.  For each end of file
 * mark, a zero is stored.  For each error reading the tape, a negative
 * one is stored.
 *
 * The companion program disk2tape rewrites a tape from the disk file.
 *
 * At the front of each disk file is a single 80-byte record containing
 * a comment for self-identification of the file.
 *
 * Usage:
 *
 * tape2disk tape disk comment
 *
 * if disk is "-", then output will be written to stdout
 *
 * Limitations:
 *
 * The maximum record length handled by this program is 256 KBytes.  This
 * is the maximum length supported by the 3490 and Metrum tape drives.
 */

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <fcntl.h>
#include <errno.h>

/* Maximum data block size */

#define BUFFER_SIZE (256*1024)

/* Length of the initial label */

#define LABEL_SIZE 80

/* Number of sequential EOF's or errors that will terminate reading */

#define EOF_THRESHOLD 2

int       act;			/* Currently active I/O buffer */
int       eof;			/* Number of sequential EOF's or errors */
int       len;			/* Length of disk record */

int       fil;			/* File number (starts at 0) */
int       rec;			/* Record number (starts at 0) */
long long byt;			/* Total number of bytes */

struct {
  int  ret;			/* Value returned from read() */
  char buf[BUFFER_SIZE];	/* Data returned from read() */
} tap[2];

char hdr[LABEL_SIZE];		/* 80 byte file label */

long long itod() {
	struct timeval tp;
	gettimeofday(&tp,0);
	return (((long long)(tp.tv_sec) % 86400) * 1000 + 
		             tp.tv_usec / 1000);
}

main (int argc, char **argv) {

  int tape_fd;		/* Tape file descriptor */
  int disk_fd;		/* Disk file descriptor */

  long long t1,t2;	/* Start and end times */

  int first;		/* Flag for first time through loop */

  if (argc != 4) {
    fprintf(stderr,"Usage: tape2disk tape disk comment\n");
    exit(1);
  }

  memset(hdr, ' ', LABEL_SIZE);
  memcpy(hdr, argv[3], strlen(argv[3]));

  tape_fd = open(argv[1],O_RDONLY);

  if (!strcmp(argv[2],"-")) {
    disk_fd = fileno(stdout);
  }
  else {
    disk_fd = open(argv[2],O_RDWR|O_CREAT, 0755);
  }

  fcntl(tape_fd,F_SETFL,_FASIO);
  fcntl(disk_fd,F_SETFL,O_LARGEFILE);

  if (write(disk_fd, hdr, LABEL_SIZE) != LABEL_SIZE) {
    fprintf(stderr,"Error writing disk file\n");
    exit(1);
  }

  act = 0;
  eof = 0;

  fil = 0;
  rec = 0;
  byt = 0;

  first = 1;

  t1 = itod();
  do {
    if (read(tape_fd,tap[act].buf,BUFFER_SIZE) != BUFFER_SIZE) {
      fprintf(stderr,"Error starting tape I/O on file %d record %d errno %d\n",
              fil, rec, errno);
      exit(1);
    }
    if (!first) {				/* If not first time */
      if (tap[act^1].ret < 0) {			/*   if it got an error */
        len = 4;				/*     only store ret code */
        fprintf(stderr,"Error reading file %d record %d errno %d\n",
                fil, rec, errno);
      }
      else {					/*   if no error */
        len = 4 + tap[act^1].ret;		/*     store len + data */
        if (len == 4) {				/*   if an EOF mark */
          fil++;				/*     update file no */
          rec = 0;				/*     reset record no */
        }
        else {					/*   if a normal record */
          rec++;				/*     update record no */
          byt += len-4;				/*     update byte count */
        }
      }
      if (len == 4) {				/*   if not a data record */
        eof++;					/*     update eof count */
      }
      else {					/*   if a data record */
        eof = 0;				/*     clear eof count */
      }
      if (write(disk_fd, &tap[act^1].ret, len) != len) { /* write the disk */
        fprintf(stderr,"Error writing disk on file %d record %d errno %d\n",
                fil, rec, errno);
        exit(1);
      }
      if (eof >= EOF_THRESHOLD) {		/* All done with copy */
        t2 = itod();
        fprintf(stderr,"Copied %d files %lld bytes at %d Bytes/sec\n",
                fil, byt, (int) (byt/(t2-t1)));
        exit(0);
      }
    }
    else {
      first = 0;
    }
    tap[act].ret = asiostat(tape_fd);		/* Wait for tape I/O complete */
    act ^= 1;					/* Update active buffer num */
  } while (1);
}


/*
 * disk2tape - Utility to copy a disk file (or pipe) to a tape
 *
 * This program copies a disk file (or pipe) to an entire tape.  It
 * restores the tape image that was produced by the tape2disk program.
 *
 * The disk format is relatively simple.  For each tape record, a 4-byte
 * record length is stored, followed by the data.  For each end of file
 * mark, a zero is stored.  For each error reading the tape, a negative
 * one is stored.
 *
 * At the front of each disk file is a single 80-byte record containing
 * a comment for self-identification of the file.
 *
 * Usage:
 *
 * disk2tape disk tape
 *
 * if disk is "-", then input will come from stdin
 *
 * Limitations:
 *
 * The maximum record length handled by this program is 256 KBytes.  This
 * is the maximum length supported by the 3490 and Metrum tape drives.
 */

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/ioctl.h>
#include <sys/mtio.h>
#include <fcntl.h>
#include <errno.h>

/* Maximum data block size */

#define BUFFER_SIZE (256*1024)

/* Length of the initial label */

#define LABEL_SIZE 80

int       act;			/* Currently active I/O buffer */
int       len;			/* Length of disk record */

int       fil;			/* File number (starts at 0) */
int       rec;			/* Record number (starts at 0) */
long long byt;			/* Total number of bytes */

struct {
  int  len;			/* Length of buffer to write */
  char buf[BUFFER_SIZE];	/* Data to write */
} tap[2];

char hdr[LABEL_SIZE];		/* 80 byte file label */

long long itod() {
	struct timeval tp;
	gettimeofday(&tp,0);
	return (((long long)(tp.tv_sec) % 86400) * 1000 + 
		             tp.tv_usec / 1000);
}
/*
 * Note that a special read routine is needed for reading from
 * a pipe because in general, a read can return successfully with
 * fewer bytes read than requested.  The read routine must be
 * synchronous for this reason.
 */

static int pipe_read(int pipe_fd, char *pipe_buf, int pipe_siz) {
  int tot = 0;
  int siz;
  while (tot < pipe_siz) {
    siz = read(pipe_fd, pipe_buf+tot, pipe_siz-tot);
    if (siz <  0) return siz;
    if (siz == 0) return tot;
    tot += siz;
  }
  return tot;
}


main (int argc, char **argv) {

  int tape_fd;		/* Tape file descriptor */
  int disk_fd;		/* Disk file descriptor */

  int i, j;

  long long t1,t2;	/* Start and end times */

  struct mtop weof = {MTWEOF, 1};

  if (argc != 3) {
    fprintf(stderr,"Usage: disk2tape disk tape\n");
    exit(1);
  }

  memset(hdr, ' ', LABEL_SIZE);

  if (!strcmp(argv[1],"-")) {
    disk_fd = fileno(stdin);
  }
  else {
    disk_fd = open(argv[1],O_RDONLY);
  }

  tape_fd = open(argv[2],O_RDWR);

  fcntl(tape_fd,F_SETFL,_FASIO);
  fcntl(disk_fd,F_SETFL,O_LARGEFILE);

  if (pipe_read(disk_fd, hdr, LABEL_SIZE) != LABEL_SIZE) {
    fprintf(stderr,"Error reading comment from disk file\n");
    exit(1);
  }
  for (i=79; i>=0; i--) {
    if (hdr[i] != ' ') break;
  }
  if (i > 0) {
    hdr[i+1] = 0;
    fprintf(stderr,"Comment - %s\n",hdr);
  }

  act = 0;

  fil = 0;
  rec = 0;
  byt = 0;

  tap[0].len = 0;
  tap[1].len = 0;

  t1 = itod();
  do {

    /* Read info about what should be written to tape */
    i = pipe_read(disk_fd, (char *) &tap[act].len, sizeof(int));
    if (i < 0) {
      fprintf(stderr,"Error reading disk at file %d record %d errno %d\n",
              fil, rec, errno);
      exit(1);
    }
    if (i == 0) {
      if (tap[act^1].len > 0) {
        if (asiostat(tape_fd) != tap[act].len) {
          fprintf(stderr,"Error writing tape at file %d record %d errno %d\n",
                  fil, rec, errno);
          exit(1);
        }
      }
      t2 = itod();
      fprintf(stderr,"Copied %d files %lld bytes at %d Bytes/sec\n",
              fil, byt, (int) (byt*1000/(t2-t1)));
      exit(0);
    }
    if (tap[act].len > 0) {
      if (pipe_read(disk_fd, tap[act].buf, tap[act].len) != tap[act].len) {
        fprintf(stderr,"Error reading disk at file %d record %d errno %d\n",
                fil, rec, errno);
        exit(1);
      }
    }

    /* Wait for completion of last write if necessary */
    if (tap[act^1].len > 0) {
      if (asiostat(tape_fd) != tap[act^1].len     ) {
        fprintf(stderr,"Error writing tape at file %d record %d errno %d\n",
                fil, rec, errno);
        exit(1);
      }
    }


    /* Write an eof or start the data write if necessary */
    if (tap[act].len < 0) {
        fprintf(stderr,"Unable to recreate error at file %d record %d\n",
                fil, rec);
    }
    else if (tap[act].len == 0) {
      if (ioctl(tape_fd, MTIOCTOP, &weof) < 0) {
        fprintf(stderr,"Error writing eof at file %d record %d errno %d\n",
                fil, rec, errno);
        exit(1);
      }
      fil++;
      rec = 0;
    }
    else {
      if (write(tape_fd,tap[act].buf,tap[act].len) != tap[act].len) {
        fprintf(stderr,"Error writing tape at file %d record %d errno %d\n",
                fil, rec, errno);
        exit(1);
      }
      rec++;
      byt += tap[act].len;
    }

    act ^= 1;					/* Update active buffer num */
  } while (1);
}

A. Lester Buck III (buck@trinity.tc.cornell.edu) wrote:
: In article <2rfr8p$41t@netnews.upenn.edu> kirsten@ssc.sas.upenn.edu (Kirsten Starcher) writes:
: >We have a large number of IBM mainframe 9-track tapes, both labelled
: >and unlabelled, that we would like to transcribe to our RS/6000s.  Does
: >anyone know of a script or program that we could use to make a dumb
: >copy of the tapes to online storage? (Preserving record length and
: >block size.)

: I am not following you here - "transcribe" to what?  The basic
: "otherscsi" magnetic tape driver on AIX 3.2.5 should have no trouble
: reading from a generic SCSI 9-track tape drive.  There is software
: around that does labeled tapes.  The AT&T Unix Toolchest has something,
: for example.  If you have a lot of tapes, you probably don't want to
: rewrite them all.  The EBCDIC conversion can be done on the fly.  Or
: maybe you want to convert everything into tar format?

Our problem is that we have hundreds of tapes, not all of them labelled. 
In addition, some files are in the beloved IBM RECFM U format (forcing us
to preserve block length information on our copies). Also, we are
ultimately going to have to peruse many of these tapes to determine what
to keep, and what to toss.  For a number of reasons, we intend to defer
that job to later, after they have been copied.  Right now, all we want to
do is copy the tapes to online storage (to be tar'ed or otherwise archived
later).  But it has to be done in such a way that we can recreate the
original tapes exactly.  (We don't want to do EBCDIC/ASCII translation on
a file that we wrongly assumed to be EBCDIC). 

We have decided, for the time being, to treat labelled tapes and 
unlabelled tapes identically, ... i.e. the header and trailer labels on 
the tapes will just be copied as individual files.  So what we think we'll 
need to do is this:

  for each physical file on the 9-track tape, make a file on our output
      medium of choice.

  for each block in each file
     read the block with no translation or truncation.

     measure the length of the block

     write an output record composed of the concatenation of
      - a 2 byte integer equal to the input block length plus 2
      - the untranslated input block

     (i.e. we're imitating how the IBM mainframe does variable length
      records).

Then later we can deal with data translation issues.

We were hoping somebody who reads this request might have already dealt 
with this problem.


kirsten@ssc.sas.upenn.edu

In article <2rg6f1$63v@netnews.upenn.edu>,
Kirsten Starcher <kirsten@econ.sas.upenn.edu> wrote:
>We have decided, for the time being, to treat labelled tapes and 
>unlabelled tapes identically, ... i.e. the header and trailer labels on 
>the tapes will just be copied as individual files.  So what we think we'll 
>need to do is this:
>
>  for each physical file on the 9-track tape, make a file on our output
>      medium of choice.
>
>  for each block in each file
>     read the block with no translation or truncation.
>
>     measure the length of the block
>
>     write an output record composed of the concatenation of
>      - a 2 byte integer equal to the input block length plus 2
>      - the untranslated input block
>
>     (i.e. we're imitating how the IBM mainframe does variable length
>      records).
>
>Then later we can deal with data translation issues.
>
>We were hoping somebody who reads this request might have already dealt 
>with this problem.
>
>
>kirsten@ssc.sas.upenn.edu
>

There are a number of people that have used exactly this code block to
duplicate tapes on unix systems with one tape and lots of disk.

I've written it several times ... a simple half page C program (if the
9trk driver is reasonable). You will find it valuable to make the first
segment of each "virtual tape" file a manual transcription of the physical
human readable label on the tape since even the IBM/ANSI labels give
little clue as to the real contents in many cases. Use more than 16bits
for length ... there are many tapes with blocks much longer. Most drive
and formatter manufacturers ignored the limited max block length in the
ANSI spec.

John