🚀 go-pugleaf

----------------------------------------------------------------------------
                                                        The Florida SunFlash

              Online: DiskSuite Questions & Answers

SunFLASH Vol 46 #19					        October 1992 
----------------------------------------------------------------------------

This article is based upon the answers to frequently asked questions
from the people who developed and support Sun's Online: DiskSuite
product. Although this FAQ is a few months old, it contains valuable 
technical information.

-johnj
----------------------------------------------------------------------------

Here are the answers to your most frequently asked questions about the
Solaris unbundled product Online: DiskSuite.  We hope you find this
information useful.

                             Topics

                         ** Mirroring **
                       ** Concatenation **
                         ** Striping **
                           ** RAID **
                       ** Optical Disks **
                       ** Compatibility **
                       ** IPI Dual Port **
                            ** SSM **
                     ** High Availability **
                      ** Data Integrity **
                   ** System Admin Support **
                      ** Client Pricing **
                         ** Known Bugs *

----------------------------------------------------------------------------
** Mirroring **
----------------------------------------------------------------------------

Q. Why should we care about mirroring?

A. Mirroring allows customers to maintain two or three copies of critical
   data on separate disks and controllers. If one disk or controller fails,
   the customer can still access their data, without having to wait hours
   for a restore from tape and/or for customer service to arrive.

   DiskSuite assures high data availability and reliability by supporting
   the mirroring of any file systems, including root, swap, and /usr.
   Mirroring of the system disk enhances system availability by preventing
   system failures due to crashing of the system disk.  Thus mirroring can
   save hours of downtime that may be caused by system and data disk failures,
   and prevent loss of valuable data.

Q. Is there a maximum number of supported mirrors?

A. DiskSuite supports 127 metadevices on a system, and it takes at least three
   metadevices to make a mirror: two submirrors and the metamirror.
   The maximum number of active mirrored pairs is therefore 42.  If you
   have "solitary mirrors" (things set up to be mirrors, but without a
   second disk initialized) you can have more.  If you use 3-way mirrors,
   you get less.

Q. Are reads balanced between the two disk copies, and what algorithm
   is used?

A. The default is to balance reads between the copies via a round-robin
   algorithm. There are two other options which can be defined in
   the metadisk configuration file /etc/md.conf or modified using the
   metaparam command, namely geometric reads and single disk reads.

   For random access reads, the default round-robin algorithm should provide
   better throughput as the load is balanced over different disk partitions
   and controllers.

   Geometric reads allow faster performance for sequential access by
   minimizing head movement.  In this case, the read operations are divided
   among the copies on the disks on the basis of logical disk block address
   ranges.  Becuase the read requests are sorted among the disks by block,
   the head is likely to have shorter seeks (at most, half-way across the
   platter). Reads from a given address range are performed by the
   assigned submirror.

   The third option is to direct all reads to a specified submirror, useful
   if one of the submirrors was on a faster device e.g. IPI vs SCSI.  IPI
   will usually read faster because it does read-ahead.  This option is
   also useful with the newer SCSI with read-ahead.

   As the I/O load i.e. the amount and sequentiality of the I/O is application
   and site specific, none of these algorithms will be right for every case.
   Algorithms such as geometric reads are defeated by heavy multi-user
   activity, especially multiple proceeses trying to do simultaneous I/O.
   The default round robin algorithm is the best choice for "server"
   (i.e., random, multiuser access).

Q. Does mirroring work with raw partitions?

A. Yes. DiskSuite provides all its mirroring/striping/concatenation
   on raw (character or block) partitions or the UFS file system.
   As far as we know, mirroring works with all available database
   management systems, file systems, and applications.

Q. A customer is interested in remote mirroring across the network.
   They want the following, where disk A's data is always the same as
   disk B's data. Is this possible?

    	     ----------                     ----------
   	     | Host A |-------- X25-------- | Host B |
   	     ----------			    ----------
    		|				|
   	      Disk A                          Disk B

A. No. We do not support remote mirroring across the network. There are
   performance and robustness issues with remote disk mirroring, over
   ethernet or X.25.  The performance hit could be as much as 50% to 90%
   for mirroring across the network.  The redundant read and write transactions
   over the network slows down the overall system throughput.  Unlike the
   DiskSuite mirroring which is transparent to applications, remote mirrroring
   will also need a lot of changes to the software, limiting the usefulness
   of such an approach.

----------------------------------------------------------------------------
** Concatenation **
----------------------------------------------------------------------------

(Remember that UNIX SVr4 uses a C signed long data type to hold
byte addresses for files (e.g. in lseek(2)), hence the 2Gb limit for
the size of a single file. -johnj )

Q. Can DiskSuite support file size greater than 2GB?

A. NO. Our Online DiskSuite implementation of the UNIX *file system* can grow
   indefinitely, but nobody's UNIX *files* can grow beyond 2 GB *unless*
   they violate the SVr4 interfaces and break application portability.

Q. Will filesystem concatenation likely be bundled (at some point) with
   Solaris 2.x?

A. Not bundled in Solaris 2.0 or 2.1.  Maybe thereafter.

Q. In Solaris 2.0 are files limited to 2 GB and are file systems limited to
   the size of a physical disk?

A. Yes, SVr4 puts a 2 GB limit on files.  The Solaris 2.0 bundled
   features allow only 2 GB file systems, and don't allow a file system to
   span partitions or disks.  DiskSuite solves this problem, and allows the
   file system to *grow* and use additional disks upto 1 Terabyte.  Actually,
   file system size up to 2**64 bytes are allowed.

Q. A customer is confused about the Terabyte filesystem vs. the apparent
   limitation of DiskSuite to 64 physical devices per stripe/concatenation.

A. This is really an issue of semantics.  The idea is that "the file system
   isn't going to get in the way" even if you had 1 Terabyte of disks attached.
   A single UFS mount-point could grow that large.

   There are a number of operational issues that get in the way long before
   you reach this limit (like, how do you connect that many disks, how do you
   manage that much data under a single mount point, etc.)

Q. It looks like the limitation of 64 physical devices per metapartition means
   that we can only build a filesystem up to 64 * 1.3 GB today, or 83 GB.

A. These limits go away with later releases of SVR4.

Q. It seems like creating a Terabyte filesystem, under the 64 device
   limitation, would require approx. 16 GB per drive.

A. Actually, the largest "standard" partition we allow is 2 GB (by SVID
   interface definition) ...  and we allow 8 physical partitions per disk ...
   so you actually could have 16 GB disks (when they become available!).

Q. What are the facilities that allow large filesystems (larger than 2 GB
   that could be used to hold large data files)?  Does it really work?
   For what devices and processor configurations is it practical?
   Is it accessible via NFS?

A. DiskSuite supports file systems up to 1024 GB.  It has been tested with
   SCSI and IPI disks, and the UNIX file system.  It works with all
   applications and is accessible via NFS.

   A new interface llseek () is the facility that allows areas of physical
   disk beyond 2 GB to be accessed.  e.g. mkfs and fsck use llseek () to
   create and check large file systems, dump uses it to dump them.

Q. Are there limits on the types of disks that can be used for disk
   concatenation/large file support?

A. Basically, any standard SCSI or IPI disk and device controller can be used.
   Any of Sun's disks will be guaranteed to work..  Third party devices...
   well, as we said in the training manual "should work, but make sure that
   the customer is using only first-rate products.  DiskSuite can produce
   very high I/O loads, and is likely to expose any weakness in controller
   design or implementation".

Q. Do partitions need to be the same size on each physical disk?

A. If you are mirroring, the partitions being mirrored need to be nearly
   the same size, in order to avoid wasting space on the larger submirror.
   The disks *can* be different geometries, but the size and the layout
   of partitions is important for performance.  We recommend that all
   disks in a concatenation or stripe have the same geometry.

Q. How are files spread across multiple disks? At a block level, or at the
   file level?  Is one disk filled first, then the others?  If the disks
   have different sized partitions, are they filled in a ratio?

A. The metadevice driver behaves exactly like a raw device driver.It knows
   nothing about files, only about logical blocks.  It maps logical blocks
   to physical blocks according to the metadevice configuration.

   If the customer is using the UNIX file system, files are spread all over
   the place in a basically unpredictable manner.  UFS will keep chunks of
   a file close together (e.g. maybe 100 KB contiguous), but any file > 16 MB
   is virtually guaranteed to be all over the partitions. Files within a
   directory are fairly likely to be sort of close together, but the
   closeness degrades over time.

   If the customer is using raw device access, then the placement
   and contiguity of files is completely under their control.  They can do
   whatever they want, but the most likely outcome is for their files to
   be contiguous and not spread across the partitions.  If you really want to
   guarantee that the files are spread across partitions, use disk striping.

Q. How different is the system startup that includes the commands
   to define the large filesystem?

A. DiskSuite replaces the standard version of UFS with a souped-up version
   for large filesystems.  The file system actually looks and behaves the
   same old way...the difference is in the metadevices that allow really
   big file systems.  The system initialization process had to be enhanced
   to bring the metadevices safely on-line before the file systems were
   started.  This is all done in the /etc/rc.* files, and there were only
   two or three lines added total.

Q. What are the DiskSuite performance gains/losses?

A. Customers should see no performance difference for concatenations.
   Mirrors should yield same read performance and a 0-10% degradation
   for writes.  Striping is variable in the estimated range -30% to +100%.

Q. Why will performance be better if when doing a concatenation  I select
   partitions that are far apart?

A. If you concatenate partitions on the same disk, that's a lose.  If you
   concatenate disks on different controllers and host adapters, that's a
   big win.   Having the partitions "far away" on the same disk may not
   have any performance impact.

----------------------------------------------------------------------------
** Striping **
----------------------------------------------------------------------------

Q. What can you tell us about DiskSuite striping performance?

A. The striping performance is very much dependent upon the application,
   specifically on the system configuration and the type of I/O.

	*  For single process I/O, striping helps a lot when the I/O is
	   large and sequential, but not when I/O is small and random.

	*  Striping will lose unless you have the striped drives
	   on separate controllers.  The advantage of striping is that
	   you can get several blocks of the same request going to the
	   drives in parallel.  When you stripe all drives on the same
	   controller, you lose this parallelism.  Concatenation would
	   be much better in this case.

    Since there are no standard benchmarks for striping or I/O, we haven't
    characterized DiskSuite striping performance.  We recommend that customers
    benchmark striping with their application to verify that striping does
    indeed speed up their application, before they commit to striping.
    In typical workstation/server environments, striping is not a big win,
    no matter whose product does the striping.

Q. Why is striping performance so unpredictable?

A. Striping consists of multiplexing a single large I/O stream across several
   drives.  This process can be thought of as "de-clustering"--breaking an
   I/O stream apart to write, and re-assembling them on reading.

   The first step in striping is to have several physical devices treated as a
   single set of blocks or addresses.  The OS or driver sends an I/O request
   to the striping entity (whether hardware or software), and then the striping
   algorithm dispatches as many I/Os as possible as rapidly as possible to the
   underlying devices.

   Striping involves some interesting side effects.  Although there are several
   independent disk arms and platters, they are all grouped as a logical unity.
   This means that all the arms move as one (and it's best for the platters
   to all be synchronized as well).  Random accesses are not only *not*
   improved, they are actually more time-consuming than they would be if
   all the disks were independent.  The standard striping algorithm is of
   little use with random access intensive activities such as NFS or
   relational databases.

   For striping to work well, the heads have to be on track nearly all
   the time, and the various steps in the "I/O chain" (CPU-SRAM-DRAM-
   VM-Driver-channel adaptor-disk controller-disk) have to be basically
   available and unconstrained all the time during I/O.

   Therefore striping is not particularly effective in the UNIX environment.
   There are too many things going on the system, competing for system
   resources, for striping to help much of the time.  There are multiple
   I/O-intensive processes, and most of those are using "random" access
   patterns a lot of the time.  The net effect is that competing I/O activities
   keep the I/O path busy and move the heads off track

Q. Any hints on how to get the most out of Online: DiskSuite striping?

A. A reasonable expectation for DiskSuite striping is:

         If you set everything up *just right* AND
         If there aren't any contention or bottleneck problems AND
         If you have large, contiguous files and little random access
         Then you can expect that a 3-way stripe
            MIGHT give you 2 disks worth of performance.  MAYBE.
            It's really hard to be sure.

   How do you satisfy the above conditions?  Well, here are some
   hints.  Obviously, you want to have the stripes set up for
   maximum throughput.  But this varies substantially depending on
   I/O load, number of users, and the load in the VM system.
   The rule of thumb for sequential access is "have your stripe
   interleave between 50% and 200% of your typical 'long I/O'".

   So, setting the interleave very low (less than 16b) is probably
   suboptimal (remember that "b" in this case is the disk sector,
   512 bytes).  Setting it between 16 kB-56 kB (32 and 127 b, in
   DiskSuite terms) is usually the right answer.

   Also make SURE that you have set up your file system the right way.
   The file system clustering implemented in SunOS 4.1.1 is turned OFF
   in DiskSuite by default. If you were previously running with it on,
   you'll be disappointed.  It's very simple to turn the clustering back
   ON again.  Refer the DiskSuite manual,  use tunefs to set maxcontig
   to 7, and rotdelay to 0 (for most modern disks).

   But what about general system setup?  It turns out, the best things for
   striping are the best things for I/O in general.

   	* You want to have as many independent spindles as possible.

 	* Don't stripe across spindles that have partitions in use
      	  by someone else.

        * Have as many separate I/O paths as possible, one for each
	  disk involved is best.  For SCSI, that means multiple host
	  adaptors.  For SMD and IPI, that means multiple controllers.

	* Try to have the system disk on as lightly-loaded an I/O channel
   	  as possible, so you aren't waiting around for paging.

	* Try to have as much RAM in the system as you can afford.

	* For best performance, make sure that all the disks in the stripe
	  are the same basic type and speed (e.g., avoid mixing 3 and 6 MB IPI,
 	  or sync and async SCSI drives).

      	* Use iostat -D 2 to evaluate your I/O load, making sure that
     	  it's well balanced.

Q. Does Online: DiskSuite support striping with parity? If not, when will
   it be supported?

A. No.  Online: DiskSuite provides the ability to mirror stripe sets to
   assure data availability. Parity striping is a disk array
   technology; disk arrays are available from third-party vendors.

   Striping spreads data across multiple disks to gain I/O throughput
   as reads and writes are spread across the disks and controllers.
   An interlace value for the stripes specifies how much data is to be
   placed on one disk before moving to the next disk on the stripe.
   The default value of the interlace is the cylinder size of the first
   disk in the stripe.  Striping increases I/O throughput when the
   I/O access is large (e.g. 100s of Kbytes, Megabytes) and sequential.

   Striping with parity is used in RAIDs (Redundant Arrays Of Inexpensive
   Disks) to provide a cheaper than mirroring redundancy feature.
   (See the section below on RAID for more detail).  DiskSuite provides
   mirroring for high data availability.  Parity striping yields poor
   I/O performance when done in the kernel. That's the reason we have
   chosen to *not* support it in DiskSuite.

   If the customer must have RAID, they'll need to buy an array controller.
   This is expensive and the benefits are dubious (see below under RAID).

----------------------------------------------------------------------------
** RAID **
----------------------------------------------------------------------------

Q. We have a large NFS file server deal that is in progress.  The customer
   is looking for support of up to 8 ethernets and using RAID
   technology for disk storage.  What are your general impressions
   about the RAID technology for this?

A. RAID is NOT good for high availability unless it's incredibly well
   implemented, which most of them aren't, with OK performance, but
   questionable reliability.  RAID-3 typically has poor write performance.

   Note that RAID is really pointless on an NFS server.  RAID has very poor
   syncronous write performance.  The read performance is great, but the
   network is the bottleneck for NFS even when there's only one disk.

Q. What are some RAID knock-offs?

A. The only advantage of RAID is its cheaper-than-mirroring redundancy
   features... but this advantage is dubious:

  * RAID has a lot of electronics, which introduce more failure points

  * the RAID controler and power supply are new single-points-of-failure

  * RAID is cheaper than mirroring ONLY in large configurations... it's
    actually quite expensive in small ones

  * once you've committed to a RAID box, you are "closing" the open system.
    RAID boxes are proprietary, internally... and once you've put the
    data in there you can't transparently get it back out again.  Said
    another way, with mirroring you can selectively mirror and unmirror
    partitions at will.  With RAID, you have to format the great big disk
    and restore the file system in the RAID format.

----------------------------------------------------------------------------
** Optical Disks **
----------------------------------------------------------------------------

Q. Can I use the mirroring utility of DiskSuite to mirror the data onto
   one of our disk drives and an optical disk or optical jukebox?

A. This is *not* supported by Sun because it is an extremely dangerous
   technique and customers can easily lose data if they try to do this.
   Removable media are really tricky for mirroring, as we designed for
   fixed media.  If the operator or admin does the wrong thing or misses
   a step while physically mounting or removing the optical media, they
   could destroy data on the magnetic disk as well as the optical disk.
   Another reason why this is not a good idea is that the write performance
   will drop to the level of the optical drive.

Q. I know Online: DiskSuite allows filesystems (SCSI & IPI) to reach 1024 GB
   but is there any limitions in our SCSI drivers that would prevent say
   large optical disks (20GB) to be used with DiskSuite on 670 or 690?

A. Our SCSI driver is almost certainly *not* able to support a partition
   larger than 2 GB. Since there are 8 partitions on a disk, the largest
   SCSI disk we could conceivably support would be 16 GB. But there would
   be no "c" (complete-disk) partition, and management of the disk might
   be a bit of a problem.

   It would be possible to have a 20 GB optical disk on our system, but
   only 16 GB at a time of its space could be used, and managing use of
   the disk and administering the system would be really hard. DiskSuite
   was designed with erasable media in mind. If the disk were a WORM,
   you would need special software (not available from Sun) just to make
   the disk usable.

Q. Can a customer use mirroring to 'clone' data to an optical disk,
   effectively doing an online back up of a relational database?

A. No. The "data cloning" technique for Online: DiskSuite is dangerous.
   Mirroring is *not* supported on removable media such as optical disks.
   In addition, databases have all kinds of referential integrity requirements,
   e.g. for managing pointers and lists. The bottom line is that Databases
   have to manage their own  online backups.  To back up a database, you
   either use their online backup system, or you shut the database down
   and do your back up off-line (with tapes, or mirroring + tapes, see below).

Q. A large customer needs to back up approx 30-50GB per server per night
   and convential tape strategies do not meet his requirements.  Can he
   solve the backup problem by mirroring IPI disks to optical disks and
   replacing the optical disks at night?

A. No.  This approach would make things more vulnerable to operator error.
   There are better alternatives, like Backup CoPilot, that actually does
   fully on-line, automatic backups.  Each server can have 4 tape drives
   (2 GB each), and you can use remote drives as well. Much safer and better.

   Optical disks are slower than tape, so this can make the backup problem
   worse.  There's another alternative that might be even more attractive
   for this customer's problem:  using mirroring with extra *winchester* disks.
   They could establish their mirrors, "break" the mirror, use Backup CoPilot
   to do a fast backup to tape of the mirrored disks, and then re-attach the
   mirror, starting an automatic resync.  This procedure could be automated
   safely, and gives lots of time to do the backup since it can be done
   online while the system is operating.

----------------------------------------------------------------------------
** Compatibility **
----------------------------------------------------------------------------

Q. What hardware and software configurations are supported by Disksuite
   (Sun-4, Sun-4c, SCSI)?

A. All SPARC CPUs, SCSI or IPI disks.  SMD has very limited support,
   although the configurations should work (SMD testing was limited to
   the Xy7053 and the 688 MB disk)

Q. What is DiskSuite impact on other OS bug fixes ?

A. Since DiskSuite is 4.1.1 and 4.1.2 only, there is a fairly limited
   "bug matrix".

   However, any bug-patches that come out later may scribble over our
   wonderful utilities.  You need to be really careful about conflicting
   bug-patches, no matter what the release.

Q. What is DiskSuite impact on NFS clients?

A. Well, basically none.  The clients benefit from the improved services
   possible with DiskSuite (e.g, file systems that can be expanded,
   file systems that are mirrored, servers that stay up more of the time),
   but there are no explicit impact on clients

Q. Can DiskSuite be used with PrestoServe?

A. Yes and No.  With striping and concatenation, PrstoServe is fully
   supported.  With mirroring, PrestoServe is a Bad Idea. Mirroring was not
   tested with PrestoServe, so this mode cannot be supported.

Q. So why no Presto with Mirrors?

A. Basically, it's that Presto and mirroring are working at cross-
   purposes.  Why do people want mirroring?  higher availability,
   and more failure resilience. What do people get when they buy Presto?
   A single point of failure that, if it fails, will corrupt their data
   and prevent access to the disks. Not exactly a smooth fit.

Q. Does Sybase work with Online: DiskSuite's raw partitions?

A. Yes.  But there may be a slight performance penalty for doing this.
   Sybase is a very heavy user of async I/O.  Their mirroring uses it to
   good effect.  Our mirroring supports async I/O, but it is not tuned
   for this.  However, in recent TPC-B measurements on a 4/90 running
   Sybase 4.8 DiskSuite mirroring got the same TPS rate as Sybase mirroring.

Q. If a customer wants to mirror Sybase disks, do we recommend Online:DiskSuite
   or Sybase mirroring? Has one method proved more reliable than the other?

A. Reliability and robustness is equal.  The issue is performance:
   Sybase's mirroring may yield better run-time performance and a tiny bit
   faster recovery than Online: DiskSuite.

   However, the counter-issue is coverage or what can be mirrored.
   Sybase's mirroring can only cover the data under Sybase control.
   They can't handle the system disk (boot, root, swap, /usr, and UFS files).

   If performance measurements indicate that Sybase mirroring has much better
   run-time performance for the application, customers could use both
   mirroring:  Sybase mirroring on the data and DiskSuite mirroring for
   the rest of the system data.

Q. For a database (Ingres) which opens file handles and runs everything
   from a server process, is it *really* necessary to completely shut
   down and restart the Ingres stuff before turning mirroring on/off?

A. Depends on whether you're using the file system or not.  Also depends
   on how cleverly configured you are.  If you've set your system up
   with "solitary mirrors" before you start up the database, you can
   turn mirroring on and off at will without mucking with the database.

   The rule of thumb is, if you are changing the device you are addressing
   at the file system or application level, you have to restart.  If you don't,
   then you don't.  The metadisk makes sure that the data is OK, but only the
   database can keep its pointers in sync.  If you use the solitary mirror
   for all your disks (e.g., /dev/md0d -m /dev/sd0d), then you can fiddle
   around even in the middle of a database update.

----------------------------------------------------------------------------
** IPI Dual Port **
----------------------------------------------------------------------------

Q. Is there a supported way to have a mirrored system disk and mirrored
   dual-ported IPI disks?

A. The answer is "it can be done" and "it's quite difficult to do".
   The only way this can really work is to get Consulting to write
   some special configuration management scripts.  This is NOT for
   the uninitiated!

Q. What is the proper way to mirror System and Data if the Data is to be
   dual-ported with another machine which also has mirrored System
   and possibly Data?

A. Our opinion is that the system disk should NOT be shared (dual-ported),
   although it should be mirrored.  That means that each system should
   have two "private" disks where the system lives. Why?  Because when one
   of the CPUs goes down, you'll want to be able to boot it and run diagnostics
   without any chance of mucking up the surviviing system or its data.
   You will need to be able to write to disks to do comprehensive diagnostics,
   so you'll want to have those "scratch" disks private to each system.

   No question about it, you must mirror the system disk.  It's the most
   likely to fail, and its failure is catastrophic without mirroring.
   Data disks must be mirrored and dual ported.

Q. How do we "fail over" mirrors from System A to System B?

    	     ----------     -------------         ----------
   	     | Host A |-----| Dual Port |-------- | Host B |
   	     ----------	    |   Disk	|	  ----------
    		            -------------

A. When you want to fail mirrors over, you have to completely configure
   the mirrored pair out of system A (including removing all copies of the
   db's resident on the disks in question, and making sure that they are
   meta-removed from the surviving db's on system A)...only then can you
   switch them over to system B and incorporate them into system B's
   mirroring.  This means several utilities must be fired off in
   just the right order...

   When you bring the mirrors up on system B, you are essentially "starting
   them from scratch".  If you *absolutely know* that the mirrored pair is
   in sync, you could conceivably bring them on line without resync.
   Unfortunately, it's almost impossible to have such absolute knowledge,
   so it is always best to resync before attempting to use the new mirrors.
   The good news is that once the resync starts, you can begin to read and
   write to the mirrors even though a full resync may not be complete for
   30 minutes or more.

Q. In "failing over" mirrors, how do I know which mirror I should use as the
   "master" to sync from?

A. Well, this is always the problem:  even in a single system, you have
   to worry about which is the good copy.  The general issue is "replication
   consistency", and DiskSuite does an extremely good job of making sure the
   right things happen.

   The DiskSuite state database maintains info on any disks that have failed.
   If a disk hasn't failed, either copy of the mirror is equally "good".

Q. What if the system trashed one submirror as it went down and I choose
   to resync the other (untrashed) submirror from it - oops,
   I just overwrote the good copy with the trashed copy?

A. Exactly the problem.  You have to make sure you get the good copy.
   The database tells you if either of the disks is "stale" (failed,
   data trashed).  If both the disks are ok, then the initialization
   sequence is to fsck one of the disks...if it passes, it is declared
   good and it is used to resync the  other one.  If it fails fsck, you try
   to fsck the other disk, and assumning it passes you resync from it.

Q. In "failing over" the mirrors from System A to System B, how do I make
   sure that I don't blow away the knowledge about what's a good disk
   and what isn't.

A. In an orderly failover (where nothing crashes), you do this by checking
   metastat to see if there are any bad disks, then unmount the file
   system to sync all the data, metaoffline the mirrored pair to flush
   all the kernel buffers and update the database, and THEN (and only then)
   you have a mirrored pair which actually is bitwise identical.

   At that point, you can move the pair of disks over to the other system,
   declare either one "the master copy", and metainit a new mirrored pair.
   If you do the wrong thing at the wrong time, you are really hosed and can
   destroy both copies of the data.

   But what about a disorderly shutdown, where the system crashed and
   can't be brought back to life?  Tougher situation.

   If you are running mirrors only on one of the two systems, it's not
   that hard:  you have one set of state databases, and it's easy to see
   how you can restart them safely.

   If you are running mirrors on BOTH systems, you have a problem:  you
   can't find out which of the mirrored disks (if any) were sick, or stale.
   It's quite possible that a stale copy of the data can pass fsck just
   as well as the more up to date copy...so that's not a conclusive test.

   You can try to restart your app on either copy, and see if oracle
   (or whatever) blows up on either one...or you can see which reflects
   the most up-to-date data.  That's a bit better, but time consuming
   and human-intensive.  Once you figure out which disk is better, you
   set up mirroring on the second system (editing files, firing off
   utilities) and off you go.

   Another issue is "what happens when system A comes back up?"
   It has had some or all of its mirrors removed from the system (because
   they have been switched out from under it by IDP).  You need to
   configure the non-existent disks out of the system...which can be
   difficult (sorry) if the disks registerred a failure *before* the
   crash.  We put all kinds of locks in the system so that you can't
   unintentionally mess up, but those locks can be annoying in the case
   where you're trying to reconfigure your disk farm.  The last resort
   is to "blow away" your state databases on system A, and to start
   from scratch.  You just need to make sure to run metastat and record
   the information in case there have been other disk failures (stale
   copies of the data that weren't switched over to the other system).

   The answer for now is to have each system running metastat every
   N seconds, and piping its output over to the other system.  That
   way, even though you can't read the databases, you have a reasonable
   assurance of getting the current configuration and state.

   The script would look (conceptually) like this:

   #!/bin/csh
   while (1)
	if ( -e /tmp/mdstat ) rm /tmp/mdstat
	if ( -e /usr/etc/metastat ) /usr/etc/metastat > /tmp/mdstat
	if ( `ping $HOSTB | grep alive | wc -w ' ) then
	        rsh $HOSTB rm /tmp/hostAmdstat
		rcp /tmp/mdstat /tmp/hostAmdstat
		date >> /usr/adm/mdlog
   #  or you could NFS mount this ...
	else
		echo "" >> /dev/console
   #  open console window
		echo "  HOST B appears to be down\!" > /dev/console
		echo -n "HOST B appears to be sick @"  >> /usr/adm/mdlog
		date >> /usr/adm/mdlog
	sleep 30
   end

   This way, you can reconstruct the crime after the crash.

----------------------------------------------------------------------------
** SSM **
----------------------------------------------------------------------------

Q. Is SPARCserver Manager 1.0 supported on SunOS 4.1.2? Can it be used to
   provide mirroring with IPI Dual Port?

A. No, SSM is not supported on SunOS 4.1.2. Use Online: DiskSuite for
   for 4.1.1 or 4.1.2, and for mirroring with IPI Dual Port.

Q. My customer bought SPARCserver Manager. Can he get an upgrade to DiskSuite?

A. If he had a valid service contract on the date of the DiskSuite announcement.
   Talk to your local CSS people (and escalate to David Macfawn if necessary).

Q. Customer has the understanding that only 8 mirror sets were possible
   with SSM. Has this number grown with DiskSuite?

A. Yes.  SSM supports 24 metadevices and it takes three metadevices to make
   a mirror: two submirrors and the metamirror.  DiskSuite supports 128
   metadevices and therefore 42 mirror sets.

Q. Does DiskSuite have better performance than SSM?

A. Yes.  DiskSuite can be 30% faster than SSM for mirrored writes. In other
   words, SSM suffers a 35% performance degradation when writing to mirrors,
   compared to a 5% performance degradation for DiskSuite.

----------------------------------------------------------------------------
** High Availability **
----------------------------------------------------------------------------

Q. Customer wants to know what 99.7% availability means?

A. 99.7% means that there will be 26 hours a year of down time or 1/2 hour
   per week. The 26 hours come from 8760 hours in a year * .003.

   This downtime does not include 'planned down time' for preventive
   maintenance, hardware or software upgrades, scheduled power outages etc.
   Degraded modes of operation where parts of the system fail while other
   parts are up are also not counted as down time.  Thus down time is defined
   as 'complete lack of service' due to unexpected system failure.

   To get to this level of high availability, the customer will have to
   do a lot of things e.g. use mirroring to quickly recover from system
   disk failures, mirror data disks to prevent data loss, use hot spares,
   do online backups, do quick fsck, use dual-porting, use manual or automatic
   failover scripts and procedures etc.  In most cases, the customer should
   use a consultant to do the right things for the customer's application
   environment.

Q. If customer utilizes SunIDP, Online DiskSuite mirroring, hot spare disks,
   and Backup CoPilot, does he have a 99.7% uptime system by default?

A. NO! If a customer uses Online: DiskSuite, Backup CoPilot and SunIDP
   they most certianly must *not* expect anything close to 99.7%.
   The only way to get beyond ~99.3% is to have manual or automatic failover
   scripts and procedures... these are site specific, and must be written by
   the customer or knowledgable consultant.

Q. Does Sun recommend planned down time with a high availability system?
   Or does the customer keep the applcation running until something goes
   wrong? Or in other  words do we recommend preventive maintenance?
   If yes, what kind of preventive maintenance do we do?

A. Sun has no recommendation on "planned downtime" or "preventive
   maintenance".  These are good things, and when properly scheduled and
   conducted these actually contribute to uptime.  Examples of these
   would be halting the system, running sundiag, cleaning tape heads,
   servicing building air conditioning systems, replacing whining or
   grinding fans, and rebooting.

Q. Do  you feel comfortable with a Unix file system that does not get
   fsck(ed) for upto 1 year?

A. Since QuickCheck is built into SunOS 4.1.2 and DiskSuite, fsck will be
   skipped if it's not needed.  So customers should run fsck periodically,
   for preventive maintenance.  If it's not needed it'll be automatically
   skipped by QuickCheck.

Q. Customer wants to know maximum time to recover, say for disk failure
   followed by controller failure ...

A. We haven't done any worst-case studies for mirror failover, but it would
   probably be on the order of tenths of a second on average, ranging from
   very few tenths of a second to a long time in the worst case.  The only
   difference  between disk and controller failure would be the time to
   recognize the failure.  In the case of a controller or disk "hang",
   we never recover. IPI controllers occasionally "panic", produce a core
   dump, then recover and that takes a few seconds.  Disk failures are
   usually faster - less than a second.  Essentially, we are gated by the
   hardware (or firmware) in failover times.

Q. A commercial customer wants to be able to boot a 690MP system within 15
   to 30 minutes.  The main problem is the potential for needing to do a
   full fsck on their 8-10 GB database.  Does a 690MP run fsck faster?

A. The 690MP will not do fsck any faster than a uniprocessor until Solaris 2.0
   with multi-threaded drivers is available.

   The keys for fast fsck today are:
    * 4.1.2 or Online: DiskSuite
    * Lots of separate I/O paths
    * as many "quiescent" file systems as possible.

   If the customer has set up their database tables as a few files spread
   on several spindles, fsck may be very short indeed.  The database tables
   are updated frequently, but they don't tend to change size (changing size
   is what makes a file system "dirty" or in need of an fsck).

   The bottom line is that they'll have to benchmark it.  They shouldn't
   have a big problem with the 15-30 minute reboot time.

----------------------------------------------------------------------------
** Data Integrity **
----------------------------------------------------------------------------

Q. Customer asked whether/how we could guarantee data for a completed
   transaction was on the disk in the event of a CPU failure.

A. We don't and can't.  We are *exactly* like physical disks in the
   semantics of reads and writes.  If the user has O_SYNC turned on,
   the completed writes are on the disk.  If not, then completed writes
   are most likely still in memory and will make it to disk sometime later.
   We're not in a position to change these semantics because, like physical
   devices, we are called by the vm system when it wants to write pages
   so that's the first time that we "see" the data.

Q. What is the largest possible amount of data that we can write and claim
   100% guaranteed recovery on a write operation to disk?  What are our
   data integrity features?

A. The "best case" you should know about is "as much data as they
   have time to write", with the following features:

  * unbuffered raw device, or file system synchronous writes.  This
    guarantees that the entire block(s) of data are written to disk
    before the write-complete signal is returned.  Of course, it's slow...
    but that's the awesome price of data integrity in any system.

  * read-after-write mode in our drivers
    This mode doesn't send the write-complete signal until the data
    has been verified as durably recorded on the platter.  Guess
    what: it's even slower.

  * disk mirroring (part of Online: diskSuite) creates two or even
    three copies of your data transparently, so that even if you lose a
    platter or controller you still have all your data.  the write
    signal doesn't complete until both copies of your data are
    written.  this can slow things down a bit, but not as much as the
    first two.

  * disk mirroring "serial-write mode" doesn't start writing
    the second copy of the data until the first has succeeded.
    that way, if your application is really picky, the write will be
    either "completely recorded" or not at all.

  * SCSI parity -- all disks have parity bits, and everybody uses
    them.  Almost all SCSI devices have parity on the SCSI bus as
    well.  But Sun is one of the few workstation vendors to actually
    use the SCSI-bus parity bits.  This is a bit of extra protection in
    the data path (so to speak).

  * Prestoserve checksum -- the original prestoserve didn't have parity
    in the NVRAM. We added a software checksum because we were worried
    about data integrity.  Other vendors haven't done this with the
    VME Prestoserve.

  * Relational DBMSs put all this together with transactions and
    logs that keep data integrity (and referential integrity) under
    normal and failure conditions.  Of course, all the protections
    cost CPU and I/O performance (130 TP-1's per second is really good,
    but it's only about 1/4 MB/s of user data).

----------------------------------------------------------------------------
** System Admin Support **
----------------------------------------------------------------------------

Q. When mirroring disks using DiskSuite, should one of the disks fail
   do you get a warning message?

A. Yes, there is an automatic indication, in the console window.  This
   is standard behavior for all failure or error messages.

   In addition, it's very simple to write a script that periodically
   checks the system to make sure all is well.  This script can be run
   from cron, or as an endless loop in a shell window that's rlogged-in.
   The system administrator can also write a very simple script fired off
   automatically on any failure using /etc/syslogd.conf. The script could
   write e-mail, or put an alert message up in the window system.

Q. My customer just installed Online DiskSuite and was asking with concern
   about what the nameless processes were that were running, that are
   related to DiskSuite?

A. The DiskSuite daemons are nameless because it's critical that nobody ever
   be able to kill them.  They spawn children as needed, and their
   function is critical to the data integrity features of DiskSuite.
   In the planned Solaris 2.x version of DiskSuite, these nameless daemons
   are replaced by threads.

----------------------------------------------------------------------------
** Client Pricing **
----------------------------------------------------------------------------

Q. Should I order separate DiskSuite products for Servers and Desktops?

A. Yes.  There are separate items on the end user and reseller price list for
   the larger Server/Workstation (ODS-1.0-4-4-21S) and for the smaller
   Desktop Workstation/Server (ODS-1.0-4-4-21D)>

Q. Do customers have to purchase Online: DiskSuite for each client in
   order to be able to run the updated commands (e.g. df)?

A. If you wish to *run* Online: DiskSuite on a client (e.g.. do disk
   mirroring for the data stored on the client's local disks), you need
   to purchase an RTU (list $1,000 in the U.S.).

   If you only wish to have access to the modified commands, then you may:
   1. NFS mount the commands from the server running Online: DiskSuite
      or
   2. Copy those commands to another machine such as the Sun file server

   In other words, you do not need to purchase an RTU to run the updated
   commands on a client.  Note that the only commands that are copyable are
   file-system-capacity related (df, du, etc.).  A complete list of the
   copyable commands (and the ones you'd need an RTU for) is given below.

   Here are the utilities that client systems CAN copy without an RTU:
    * df
    * du
    * ls
    * quota
    * tar
    * bar
    * find
    * libserver.a
    * libserver.so.1.0
    * libserver_p.a

   Here are the modified ones that they CAN'T copy for free:
    * fsck
    * growfs
    * lockfs
    * metaclear
    * metad
    * metadb
    * metadetach
    * metahs
    * metainit
    * metaoffline
    * metaonline
    * metaparam
    * metarc
    * metareplace
    * metascript
    * metastat
    * metasync
    * metattach

----------------------------------------------------------------------------
** Known Bugs **
----------------------------------------------------------------------------

** Conflicting OS Patches **

   Many of our products are released with some sort of patch fix
   that is needed in order for that particular product to run. When
   you install the product it automatically installs one or more
   patches that fixes something in the OS.

   Two such products are DiskSuite and Backup CoPilot. They both
   install three patches. The NFS Jumbo patch, the Lockd patch
   and a inactive performance patch.

   There was a problem with an older version of the NFS Jumbo patch
   overwriting the patch needed for DiskSuite.  This has been fixed
   and there is no patch conflict with the current version of the
   NFS Jumbo patch.  The most recent version of the patch (100483-02)
   ported for DiskSuite is equivalent to NFS Jumbo patch 100173-07.

   Users should specifically ask for the DiskSuite versions of the
   NFS Jumbo patch and the lockd patch.

** NC400 and DiskSuite/Backup Copilot **

   There are two conflicts:

	nfs_omnisrv.o / nfs_server.o
	omni_nfsd     / nfsd

    As you can see, these are not direct conflicts, since they co-install
    and work together, but the effect is that NFS clients connected through
    the NC400 interface go through the omni versions of these functions
    rather than the Backup CoPilot/DiskSuite versions.

    Now, what breaks?  The changes that were made to nfs_server.o and nfsd
    in Backup Copilot and Online: DiskSuite have the effect of throwing
    away NFS requests on locked file systems. This causes clients to timeout
    and either retry or return errors to the user.  Most of the time the
    client retries, which is what we want.  If we didn't drop the requests,
    the nfsd's would tend to "saturate" with waiting requests and further
    requests, even those for file systems which are not locked, would get
    dropped.  By dropping the requests, we keep the nfsd's operating so
    that requests on unlocked file systems can be processed.

    This is the functionality which is currently broken for NC400 users.
    There is no real impact to DiskSuite users, since the only time the
    file systems are locked is during a growfs operation, which is seldom
    done.  The impact is more severe for Copilot.  It means that the server
    will be essentially unavailable when file systems are locked during dumps.

    There is an NC400 patch available which fixes this.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
For information send mail to info-sunflash@sunvice.East.Sun.COM.
Subscription requests should be sent to sunflash-request@sunvice.East.Sun.COM.
Archives are on solar.nova.edu, paris.cs.miami.edu, uunet.uu.net,
src.doc.ic.ac.uk and ftp.adelaide.edu.au

All prices, availability, and other statements relating to Sun or third
party products are valid in the U.S. only. Please contact your local
Sales Representative for details of pricing and product availability in
your region. Descriptions of, or references to products or publications
within SunFlash does not imply an endorsement of that product or
publication by Sun Microsystems.

John McLaughlin, SunFlash editor, flash@sunvice.East.Sun.COM. (305) 776-7770.

TRACE: To: sunflash@housun.Central Errors-to: owner-sunflash@housun.Central.sun.com