🚀 go-pugleaf

  [Dr. David Kahaner is a numerical analyst on sabbatical to the 
   Office of Naval Research-Asia (ONR Asia) in Tokyo from NIST.  The 
   following is the professional opinion of David Kahaner and in no 
   way has the blessing of the US Government or any agency of it.  All 
   information is dated and of limited life time.  This disclaimer should 
   be noted on ANY attribution.]

  [Copies of previous reports written by Kahaner can be obtained using
   anonymous FTP from host cs.arizona.edu, directory japan/kahaner.reports.]

To: Distribution
From: 
 David K. Kahaner
 US Office of Naval Research Asia
 (From outside US):  23-17, 7-chome, Roppongi, Minato-ku, Tokyo 106 Japan
 (From within  US):  Unit 45002, APO AP 96337-0007
  Tel: +81 3 3401-8924, Fax: +81 3 3403-9670
  Email: kahaner@cs.titech.ac.jp
Re: Supercomputer packaging technologies compared.
29 Dec 1992
This file is named "package.sc"

ABSTRACT. Packaging technologies used in current supercomputers in the
US and Japan are compared. (Thorndyke)

During the period 4-14 March, 1992, I accompanied Mr. Lloyd M.
Thorndyke, (address below) on a series of visits to Japanese computer
companies in order to learn about the techniques currently in use in
packaging supercomputer multi-chip-modules (MCMs).
Packaging refers to the physical combination of various CPU, memory, and
other VLSIs on one or more "module". Thus, packaging addresses questions
of number of gates, pins, physical size, cooling, manufacturing, costs,
etc.  On the basis of those visits Thorndyke has produced a report, sections
of which are contained below. Since that visit Hitachi and Fujitsu have
introduced new supercomputer models. Some aspects of these were
presented to us, but other material was proprietary and not presented.
Thus the observations below need to be assessed in that light.  Mr.
Thorndyke's current address is the following.

	Mr Lloyd M. Thorndyke
	Chief Executive Officer, DataMax, Inc
	2800 East Old Shakopee Road
	Bloomington, Minnesota 55425
	 Tel: (612) 853-3041, Fax: (612) 853-4789

Mr. Thorndyke was also a Senior Vice President for Technology
Development, and executive manager of the CYBER 200 Supercomputer Family
operations, both at Control Data Corporation. He was a founder and Chief
Executive Officer of the ETA Systems, the supercomputer subsidary of
Control Data [now defunct].

              HIGH PERFORMANCE PACKAGING TECHNOLOGIES

                        Lloyd M. Thorndyke

INTRODUCTION
This report presents results of a visit made to Japan March 4-14, 1992
with the objective of assessing Japanese computer packaging technology.
Thanks are due to the Scientific Supercomputing Subcommittee of the IEEE
Computer Society which helped in making arrangements and contributed to
the success of the trip. it should be noted that the thoughts and
opinions expressed here are solely mine and do not represent views of
the Subcommittee, the Computer Society or the IEEE. Further, no material
in this document is restricted because of nondisclosure agreements.

SUMMARY
The trip was highlighted by visits to the three major Japanese
supercomputer companies; Fujitsu, Hitachi and Nippon Electric (NEC). As
a result of these visits, I came away impressed by their packaging
engineering and manufacturing capabilities. Originally, supercomputer
packaging was more art than science and even today retains much of that
aura. While considerable packaging engineering effort has been applied
in the US, I do not believe that we have achieved the same overall level
of competence I saw in Japan. While I have not had the chance to visit
an equivalent IBM facility, I would expect to find similar capabilities
there. The most important point is that one can find one advanced
US facility, but I found three in Japan!

SUPERCOMPUTER VERTICAL INTEGRATION
The three Japanese supercomputer companies are vertically integrated
enterprises, manufacturing and integrating all components from circuits
through systems. From the economic view, they are primarily merchant
semiconductor suppliers and, secondarily, suppliers to their
supercomputer divisions. Their dependence on the semiconductor business
forces them to buy the latest lithography and fabrication equipment or
face the loss of customers and market share.

The vertical Japanese structure is in sharp contrast with the US
supercomputer companies in which there has been only a small involvement
of the US companies in semiconductor design and fabrication. The
experiences from these involvements have, for the most part, been
negative. The costs of continually buying new equipment, in the face of
little or no outside revenue, prevents small companies from remaining at
the cutting edge of technology. Thus, such companies buying
semiconductor equipment quickly find that their equipment is becoming
obsolescent and falling behind in performance.  Those companies buying
from merchant suppliers find that they are generally a partial
generation behind the industry because of the delays in going from
research to production relationship with their vendors.

I believe that the vertically integrated Japanese companies will survive
in the supercomputer market. I also believe that the survival of the
small US traditional supercomputer companies and their counterparts
building massively parallel systems  depends upon the development of
vertical cooperation agreements with the advanced semiconductor
suppliers. Such agreements could make small companies more competitive
by providing early access to research devices as is the case with
vertically integrated companies. This action will require leadership,
vision and conviction that cooperation is the key to survival.

JAPANESE MEMORY/LOGIC CIRCUITS
Table 1 below is a tabulation of some characteristics of combined memory
and logic chips used by the Japanese in their supercomputers. Where
applicable, corresponding entries have been made for the Cray Research
C-90. In reviewing the chip performance, it is interesting to note that
the three Japanese chips have the same gate switch times of 70
picoseconds (ps).  Fujitsu had originally announced a speed of 80 ps,
but improved that to 70 ps when NEC announced 70 ps. It has been
suggested by some in the US that this is proof that the Japanese
cooperate with each other. I don't concur with that view, but rather
believe that because they all use the latest production technology and
lithography equipment, their devices will logically have the same
performance levels. The argument is reinforced by the 70 ps speed of the
Cray Research C-90.

--------------------------------------------------------------------------
              Table 1.  MEMORY/LOGIC CIRCUIT CHARACTERISTICS

    Mfr.                 Fujitsu       Hitachi       NEC     Cray   Res.
    Model                VP 2600  G-8 & C3800        SX-3        C-90

 Cache Memory
  Cap.(Kb)/Speed(Ns)    64/1.6        64/1.6      40/1.6
  Logic Gates  (K)         3.5         2.0          7.0

  Cap.(Kb)/Speed(Ns)        -         36/2.0        -
  Logic Gates  (K)                     4.0

 Main and/or             lMb/35Ns     512Kb/5.ONs  256Kb/2ONs
  Extended Memory
    (Cap./Speed)                        4Mb/80

 Spec.  Logic  Chips      15K/70ps      12K/70ps     1OK/70ps    1OK/70ps
    (Gates/Speed)         1.2K/60ps     25K/60ps     2OK/70ps

 Die Size (Cm)            13.Oxl3.0     8.3x8.3      13.3xl3.3    9.5x9.5

 Chip  Carrier  (CM)       17xl7         lOx1O       18.5xl8.5       ?

 Chip Power (Watts)           30            21           33          20

 Pins/Carrier                460           528          485         308

--------------------------------------------------------------------------

JAPANESE PACKAGING TECHNOLOGY CHARACTERISTICS

Table 2 below is a tabulation of the packaging characteristics of the
same four systems shown in Table 1.  All are based on the use of
multichip module (MCM) technology.

---------------------------------------------------------------------------
          Table 2.  CHARACTERISTICS OF MULTI-CHIP MODULES (MCM)

                       Fujitsu       Hitachi        NEC      Cray  Res.

 MCM Size (Cm)         24.5x24.5    10.6xlO.6      22.5x22.5    35x5O

 MCM Material           Ceramic       Mullite        Ceramic    Printed
                          Glass      (Ceramic)         and       Circuit
                        Composite                   Polyimide      Board

 Dielectric K               5.7            5.9           3.5        2.9

 Chips/MCM                144             36          100           60

 Pins/MCM              8,640          2,521        11,540       1,750 [1]

 MCM Impedance             65             56                        60

 Thermal Impedance          0.6            1.8           0.7        1.8
   (*C/Watt)

  K Gates/MCM           2,160          432/900        2,000         600

  Cooling           Conduction         Helium      Conduction    Conduction
   Path               to  Fluid       to   Fluid      to Fluid      to Fluid

  First Delivery       VP-2000         C-3800          SX-3          C-90
                        2Q89             4Q92          4Q88          4Q91

                Notes:  [1]  Plus  inter-MCM   connections

----------------------------------------------------------------------------

OBSERVATIONS ON PACKAGING APPROACH
As I weigh the different packaging approaches, I have come to the
conclusion that the Hitachi packaging is better optimized for chip
density growth. Since the Hitachi MCM is only one-fourth the size of
Fujitsu's and NEC'S, it has the largest growth potential in terms of
gates per chip before its capacity becomes too large. Also, I believe
this smaller and simpler MCM should be less costly to build. Viewed
another way when considering the next generation, large MCMs contain too
many gates for a product line building block. if such is the case, a new
MCM design and tooling will be required to down-size the MCM for
applications such as a modestly parallel system. In that event Hitachi
will be ready with the next generation package, while others will face a
new development.

In a broad view of the Japanese packaging approach, a strong commonality
exists. One of the Japanese executive remarked that all the Japanese
packaging was now similar. The Cray Research C-90 packaging does not
match the Japanese in several key respects:

 (1) Gate density per unit area,
 (2) Gates per MCM pin, and
 (3) Compactness. The Cray MCM is four times larger for the
      same number of gates.
 (4) Tooling costs. The printed circuit boards are relatively 
      inexpensive to tool.

For the Japanese to move to what I call practical parallelism (16-256)
processors, they need to develop significant packaging innovations to
allow a basic building block to emerge. Such aspects as cooling overhead
and "hot module replacement" (no downtime during maintenance) could
eliminate the current large NEC and Fujitsu MCMS.  This supports my
opinion that the small Hitachi MCM is an excellent candidate for the
next generation building block.

One offsetting factor that could encourage the continued development of
large MCMs would be the inclusion of memory with logic, rather than two
separate packages. However, unless memory densities are improved, this
approach does not seem to provide enough memory, even for a relatively
small cache.

In a meeting with Kyocera, I was told that the Japanese supercomputer
companies overtook US companies in supercomputer packaging five years
ago and that now the lead is insurmountable. I tend to agree, but for a
different reason. The newer technologies require increasingly greater
capital and tooling funding to enter production.  However, most US
supercomputer companies are relatively small, have low volumes and
limited capital money. These facts argue that the US companies can't
afford to retool technologies like the larger Japanese companies who can
spread the costs over a range of computers.  The US companies are in
danger of being driven from the market because of the high costs of a
broad product line and the multi-billion-dollar revenues required to
fund the R&D and tooling.  The only US companies that can compete in
such a broad market is IBM, and possibly Cray Research, although IBM is
not active in the traditional supercomputer market at this time, and
Cray Research does not currently have a broad product line containing
standard building block modules. The salvation for the rest of industry
is participation in vertical cooperation ventures with other computer
companies to share the prohibitive development and tooling costs.

One of the inescapable facts in supercomputer technology is that higher
performance circuits generate greater heat. In addition, supercomputers
demand larger amounts of faster memory, again at the expense of heat
generation. The ability to remove heat efficiently and supply power may
become the limiting factor as the gates per chip density increases. I
believe the Fujitsu and NEC modules are the best production modules in
terms of handling increased power per chip.  These modules can cool
about 30 watts per chip, but based on their thermal impedance, they
could handle chips in the 80 watt range.  Considerable growth is
available from their MCMS, but the large logical size may prevent
broader usage as noted previously.

In order to achieve a dense, high performance package, the ratio of
silicon to MCM area must be high. The chip leads canot be fanned out as
was done in the past, but must be brought under the chip. This is
accomplished by pin grid arrays (Fujitsu), flip chips (IBM) or flip TAB
(NEC). These designs shorten the interchip leads and may allow one to
live with slower MCM transmission times if the costs are reasonable.
However, any of these methods must address the impedance changes from
the driving gate to the receiving gate as the different transmission
media deteriorate the signal quality at the receiving end. The poorer
the mismatch, the longer the receiving end must wait for a quality
signal, affecting performance. A slower media with good impedance
matching from driver to receiver may be faster overall than a fast media
with poor impedance control. The C-90 MCM uses the TAB with conventional
fan-out and has a lower gate density per unit area. Although this design
will suffer somewhat in performance, the low dielectric of the board at
2.9 will provide some compensation.

The propagation delays of the PCBs and MCMs vary by a factor of about
2.5 from worst to best. The ETA Systems ETA-10 G-10 glass-epoxy PCB had
a measured delay of 168 ps per inch (PSPI), while coax cable is 96 PSPI
and the ceramic used in some MCMs is estimated around 210 PSPI. As the
size of the MCM is reduced, the total time delay, even with the slow
ceramic becomes tolerable, especially when the cost and commonality are
considered. Also, as Hitachi has done, critical signals can be
transmitted via coax within the MCM rather than using the ceramic MCM
wiring.

In reviewing the Japanese trends, I agree that they have converged on
similar technical approaches that are tooling-intensive and volume
sensitive. The NEC use of high performance polyimide signal layers on
the ceramic MCMs is required because of the size of the MCM, although I
believe this increases the MCM material and scrap costs.

----------------------------END OF REPORT--------------------------------