🚀 go-pugleaf

Here is a summary of responses to my request
on info about principal components analysis.

Many thanks to all who replied!

Steve Sullivan
sullivan@csn.org

=================================================

From: dtm1@Ra.MsState.Edu (David T. Morse)

Amick, D. J., & Walberg, H. J. (Eds.)  (1975).  Introductory
    multivariate analysis for educational, psychological,
    and social research.  Berkeley, CA: McCutchan.  Chapters
    5-7 cover exploratory, rotation schemes, and
    confirmatory factor analysis, respectively.

Cooley, W. W., & Lohnes, P. R. (1971).  Multivariate data
    analysis.  New York: Wiley.  Chapter 4 covers principal
    components; chapter 5 discusses factor analysis.
    FORTRAN programs are supplied for data analysis.

Cooper, J. C. B. (1983).  Factor analysis: An overview.
    American Statistician, 37, 141-147.

Forsythe, G. B., McGaghie, W. C., & Friedman, C. P. (1986).
    Construct validity of medical clinical competence
    measures: A multitrait-multimethod matrix study using
    confirmatory factor analysis.  American Educational
    Research Journal, 23, 315-336.

Gorsuch, R. L. (1983).  Factor analysis (2nd ed.).
    Hillsdale, NJ: Lawrence Erlbaum.

Hair, J. F., Jr., Anderson, R. E., Tatham, R. L., & Black,
    W. C. (1992).  Multivariate data analysis with readings
    (3rd ed.).  New York: Macmillan.

Harman, H. H. (1967).  Modern factor analysis (2nd ed.
    rev.).  Chicago: The University of Chicago Press.  One
    of the classic references for factor analysis.

Kachigan, S. K. (1986).  Statistical analysis: An
    interdisciplinary introduction to univariate &
    multivariate methods.  New York: Radius Press.  Chapter
    15.

Kim, J., & Mueller, C. W. (1978).  Factor analysis:
    Statistical methods and practical issues.  SAGE, 07-014.
    Beverly Hills, CA: Sage Publications.

Kim, J., & Mueller, C. W. (1978).  Introduction to factor
    analysis: What it is and how to do it.  SAGE, 07-013.
    Beverly Hills, CA: Sage Publications.

Kroonenberg, P. M., & Lewis, C. (1982).  Methodological
    issues in the search for a factor model: Exploration
    through confirmation.  Journal of Educational
    Statistics, 7, 69-89.

Loehlin, J. C. (1987).  Latent variable models: An
    introduction to factor, path, and structural analysis.
    Hillsdale, NJ: Lawrence Erlbaum.

Lohnes, P. R. (1986).  Correlated factors modeling via weak
    mathematics.  American Educational Research Journal, 23,
    289-302.

Long, J. S. (1983).  Confirmatory factor analysis.  SAGE,
    07-033.  Beverly Hills, CA: Sage Publications.

Magidson, J. (Ed.)  (1979).  Advances in factor analysis and
    structural equation models.  Cambridge, MA: Abt Books.

McDonald, R. P. (1985).  Factor analysis and related
    methods.  Hillsdale, NJ: Lawrence Erlbaum.  Includes
    sections on linear structural relations, confirmatory
    factor analysis.

Mislevy, R. J. (1986).  Recent developments in the factor
    analysis of categorical variables.  Journal of
    Educational Statistics, 11, 3-31.

Overall, J. E., & Klett, C. J. (1972).  Applied multivariate
    analysis.  New York: McGraw-Hill.  Chapters 4-7 include
    discussion of factor analysis topics; the book also
    includes sample FORTRAN programs for factor analysis.

Ramsey, F. L. (1986).  A fable of PCA.  American
    Statistician, 40, 323-324.

Rummel, R. J. (1970).  Applied factor analysis.  Evanston:
    Northwestern University Press.

Stevens, J. (1986).  Applied multivariate statistics for the
    social sciences.  Hillsdale, NJ: Lawrence Erlbaum.
    Chapter 11 covers principal components analysis.

Veldman, D. J. (1967).  FORTRAN programming for the
    behavioral sciences.  New York: Holt, Rinehart and
    Winston.  Chapter 9 provides techniques for principal
    axis, common factor analysis, image analysis, and
    comparison of factor structures.  FORTRAN programs
    included for analyzing data.

Williams, F. (1986).  Reasoning with statistics: How to read
    quantitative research (3rd ed.).  New York: Holt,
    Rinehart and Winston.  Chapter 13 provides an easy
    introduction to factor analysis.

        ................
Also, Harris, R.J. A primer of multivariate statistics.  New York:
Academic Press (now in 2nd edition, about 1988 date, I think).

==========================================================

From: sam@kalessin.Jpl.Nasa.Gov (Sam Sirlin)

This is more usually called singular value decomposition. I have no
idea what you want it for, so can't help with anything but basic
references: 

Dongarra et al, LINPACK manual 

Greville and Ben-Isreal, "Generalized Inverses" had what I thought was
a good treatment of SVD. 

-- 
Sam Sirlin
Jet Propulsion Laboratory         sam@kalessin.jpl.nasa.gov

==========================================================

From: scotbri@rosevax.rosemount.com (Scott Brigham)

The book that I used that I found to be pretty helpful is

Multivariate Calibration by Harald Martens and Tormod Naes

John Wiley and Sons, 1989.  (419 pages)

It discusses PCA as well as PLS (partial least squares) and covers
a lot of the other stuff you'll want to know when doing PCA.

==========================================================

From: ph@physiol.ox.ac.uk (Patrick Haggard)

I Recommend Krzanowski's Book 
Principles of Multivariate Analysis (OUP),
which is very strong on concepts of multivariate
stuff, inc. PCA.
Patrick Haggard --- ph@uk.ac.ox.physiol

==========================================================

From: Steve Smith <bugman@sciborg.uwaterloo.ca>

Try the following for principal components. It's written for end users like
me (a biologist) rather than for mathies, but in my opinion, it's first rate.
If you get several replies, you can post to the group.

Cheers!

	Jackson, J. Edward. 1991.
	A user's guide to principal components.
	John Wiley & Sons, Inc., New York.

==========================================================

From: stewart@cs.umd.edu (G. W. Stewart)

Here is a dump from my bibliography.  Keep in mind that it represents
a numerical analysts view.  A statistician could give you many
more references.

Pete Stewart

\begin{thebibliography}{10}

\bibitem{cham:72}
J.~M. Chambers.
\newblock Stabilizing linear regression against observational error in
  independent variats.
\newblock Manuscript, Bell Laboratories, Murray Hill, New Jersey., 1972.

\bibitem{ecyo:36}
C.~Eckart and G.~Young.
\newblock The approximation of one matrix by another of lower rank.
\newblock {\em Psychometrika}, 1:211--218, 1936.

\bibitem{gaza:79}
K.~R. Gabriel and S.~Zamir.
\newblock Lower rank approximation of matrices by least squares with any choice
  of weights.
\newblock {\em Technometrics}, 21:489--498, 1979.

\bibitem{golu:69}
G.~H. Golub.
\newblock Matrix decompositions and statistical calculations.
\newblock Technical Report 124, Computer Science Department, Stanford
  University, 1969.

\bibitem{golu:69a}
G.~H. Golub.
\newblock Matrix decompositions and statistical computation.
\newblock In R.C. Milton and J.~A. Nelder, editors, {\em Statistical
  Computation}, pages 365--397, New York, 1969. Academic Press.
\newblock Cited in {\AA ke Bj\"orck's} bibliography on least squares, which is
  available by anonymous ftp from {\tt math.liu.se} in {\tt pub/references}.

\bibitem{good:69}
I.~J. Good.
\newblock Some applications of the singular decomposition of a matrix.
\newblock {\em Technometrics}, 11:823--831, 1969.

\bibitem{guns:78}
R.~F. Gunst.
\newblock Similarities among least squares, principal component, and latent
  root regression estimators.
\newblock Manuscript. Department of Statistics, Southern Methodist University,
  1978?

\bibitem{guwm:76}
R.~F. Gunst, J.~T. Webster, and R.~L. Mason.
\newblock A comparison of least squares and latent root regression estimators.
\newblock {\em Technometrics}, 18:75--83, 1976.
\newblock Cited in \cite{govl:89}.

\bibitem{hamm:85}
S.~J. Hammarling.
\newblock The singular value decomposition in multivariate statistics.
\newblock {\em {ACM} {SIGNUM} Newsletter}, 20:2--25, 1985.
\newblock Cited in \cite{govl:89}.

\bibitem{hano:81}
R.~J. Hanson and M.~J. Norris.
\newblock Analysis of measurements based on the singular value decomposition.
\newblock {\em {SIAM} Journal on Scientific and Statistical Computing},
  2:363--374, 1981.
\newblock Cited in \cite{govl:89}.

\bibitem{hawk:73}
D.~M. Hawkins.
\newblock On the investigation of alternative regressions by principal
  component analysis.
\newblock {\em Appl.\ Statist.}, 22:275--286, 1973.

\bibitem{hote:33}
H.~Hotelling.
\newblock Analysis of a complex of statistical variables into principal
  components.
\newblock {\em Journal of Educational Psychology}, 24:417--441 and 498--520,
  1933.

\bibitem{hote:57}
H.~Hotelling.
\newblock Relation of the newer multivariate statistical methods to factor
  analysis.
\newblock {\em Br. J. Static. Psychol.}, 10:69--79, 1957.

\bibitem{hucu:62}
J.~R. Hurley and R.~B. Cattell.
\newblock The {Procrustes} program: {Direct} rotation to test a hypothesized
  factor structure.
\newblock {\em Behavioral Science}, 7:258--262, 1962.

\bibitem{paig:85}
C.~C. Paige.
\newblock The general linear model and the generalized singular value
  decomposition.
\newblock {\em Linear Algebra and its Applications}, 70:269--284, 1985.
\newblock Cited in \cite{govl:89}.

\bibitem{pear:01}
K.~Pearson.
\newblock On lines and planes of closest fit to points in space.
\newblock {\em Philosophical Magazine}, 2:559--572, 1901.
\newblock Cited in \cite{govl:89}.

\bibitem{rao:80}
C.~R. Rao.
\newblock Matrix approximations and reduction of dimensionality in multivariate
  statistical analysis.
\newblock In P.~R. Krishnaiah, editor, {\em Multivariate Analysis--V}.
  North-Holland, Amsterdam, 1980.

\bibitem{vhva:91}
S.~Van~Huffel and J.~Vandewalle.
\newblock {\em The Total Least Squares Problem: Computational Aspects and
  Analysis}.
\newblock SIAM, Philadelphia, 1991.

\bibitem{wegm:74}
J.~Webster, R.~Gunst, and R.~Mason.
\newblock Latent root regression analysis.
\newblock {\em Technometrics}, 16:513--522, 1974.

\bibitem{wegm:76}
J.~Webster, R.~Gunst, and R.~Mason.
\newblock A comparison of least squares and latent root regression estimators.
\newblock {\em Technometrics}, 18:75--83, 1976.

\end{thebibliography}

==========================================================

From: rmyers@ics.uci.edu (Richard E. Myers)

I've become interested in using PCA to analyze a sequence of high
dimensional vectors (speech frames).  My SYSTAT manual and a look on
mlvl turned up the following references:


5. MCDONALD RP.
     COMMON PRINCIPAL COMPONENTS AND RELATED MULTIVARIATE METHODS - FLURY,B.
     Pub type:  Book Review.
     JOURNAL OF CLASSIFICATION, 1990, V7 N2:310-312.

6. Flury, Bernhard, 1951-
     Common principal components and related multivariate models / Bernhard
   Flury.  New York : Wiley, c1988.
     Series title:  Wiley series in probability and mathematical statistics.

8. Jackson, J. Edward.
     A user's guide to principal components / J. Edward Jackson.  New York :
   Wiley, c1991.
     Series title:  Wiley series in probability and mathematical statistics.
   Applied probability and statistics.

2. Harman, Harry Horace, 1913-1976.
     Modern factor analysis.  [Chicago] University of Chicago Press [1960].

1. Mulaik, Stanley A., 1935-
     The foundations of factor analysis [by] Stanley A. Mulaik.  New York,
   McGraw-Hill [1971, c1972].
     Series title:  McGraw-Hill series in psychology.

1. Gnanadesikan, Ram, 1932-
     Methods for statistical data analysis of multivariate observations / R.
   Gnanadesikan.  New York : Wiley, c1977.
     Series title:  Wiley series in probability and mathematical statistics.
     Series title:  A Wiley publication in applied statistics.

40. Mardia, K. V.
      Multivariate analysis / K. V. Mardia, J. T. Kent, J. M. Bibby.  London ;
    New York : Academic Press, 1979.
      Series title:  Probability and mathematical statistics.

==========================================================

From: olevin@random.hwr.arizona.edu

Cattell, r.b. (965) Factor analysis. An introduction to
Essentials. I: the purpose and underlying models. Biometrics 21,
pp. 190-210
II: the role of factor analysis in research, Biometrics, 21,
pp405-435.

Childs d 1970, The essentials of factor analysis. London, Holt,
Rinehart, and Wisnton.

Cooley W.W> and Lohnes (1971) Multivariate data analysis, NY
J-Wiley.

Davis, J./c. 1973) Statistics and data analyiss i geology,
Academic press

Fruchter b. (1954) /Introduction to factor analysis D. van
Nostrand Co.

Seyhan E. 1985 Introduction to multivariate statistical analysis
Free university, Institute of Earth sciences Netherlands


==================================================================

From: whitbeck@equinox.unr.edu (Michael Whitbeck)

 PCA is highly flavored by the area of application
 but a very readable book is "Factor Analysis in Chemistry" by
 E. Malinowski, Wiley-Interscience, 1991.
 Heartily recommended! (despite some typos)

 Not very readable but more mathematical, less science application
 oriented, is "Factor Analysis as a Statistical Method" by 
 Lawley and Maxwell, Butterworths, 1963. But read the HISTORICAL
 INTRO in Malinowski's book if nothing else! 

 Recently matlab code has been posted that will do pca.
 To 'roll' your own in FORTRAN or C or C++.... start with
 a singular value decomposition routine from netlib or NR.
 Sort the eigenvalues, largest first, determine which
 eigenvalues are significant (see Malinowski)... Quite
 easy.

==================================================================

From: jsvrc@rc.rit.edu (J A Stephen Viggiano)

Edward (Ted) Jackson, of Eastman Kodak, wrote a three-part series on PCA for
the _Journal_of_Quality_Technology_ which appeared in 1981. It's a great
introduction. I think he also wrote a book on the subject.

There are a variety of books on a related topic, Factor Analysis. Be advised
that this is only a related topic; PCA is only treated as a special case, if
at all. One of the best is the Third Edition of _Modern_Factor_Analysis_ by
Harry Horace Harmon. BTW, Part 3 of Ted Jackson's series in JQT covers
Factor Analysis.

Finally, there are books on multivariate analysis that discuss PCA.
Gnanadesikan's _Methods_for_Statistical_Data_Analysis_of_Multivariate_
_Observations_ provided me with a starting point for my thesis work on
non-linear PCA about 8 or 9 years ago.

==================================================================
==================================================================

From: beaucham@uxh.cso.uiuc.edu (James Beauchamp)

A summary from a previous query, including the following:

I found the Sage University Papers series Quantitative Applications in the
Social Sciences to provide very accessable introductions to such topics.
The Principal Components Analysis paper is #69 by Dunteman; I have not 
seen the particular paper. If your library doesn't have it, Sage Publications
is out of Newbury Park, California (fax: 805-499-0871).

Peter Palij, Columbia University, pbp1@cunixb.cc.columbia.edu

------------

i'm not an expert on PC, but i have had occassion to use it as a tool in
descriptive data analysis.  i also noticed the multiple methods/explanations
you mention.  if you haven't looked at it already, you might want to look
at a book by morrisson - something like "multivariate analysis".  he talks
a bit about different methods, for example, when PC would makes sense based
on the covariance matrix vs based on the correlation matrix.  

john  watts, University of Chicago, stuw@midway.uchicago.edu

------------

Duda & Hart is a great reference for statistical
pattern recognition techniques.  While PCA is a
general tool, they have a decent explanation,
plus references.  

Dave DeMers, UC San Diego, demers@cs.ucsd.edu

------------

Yes, the stuff on principal components analysis often IS confusing and
technical.  One reason for the confusion is that a lot of analysis techniques
are quite similar.  Principal components, Correspondence Analysis, Canonical
Analysis (and maybe another two or three) are all techniques based on the
exact SAME mathematical manipulation (the singular value decomposition or
SVD).  The differences amongst the techniques comes in how the data are
treated before the SVD is conducted.

I think that the best combination of explanation, technical sophistication
without obfuscation, and examples is (as is often the case) in the little
green Sage book from their series on quantitative analysis.  The title
is something like Classification and Scaling Analysis, but I am not
certain of the title.  The authors, however, are Susan Weber (sp?) and
A Kimball Romney.

Raymond V Liedka, Cornell University, RJOY@cornellc.cit.cornell.edu 

------------

Venables (the guy who followed up with a posting) is good, and 
PC must be one of his specialties, but if you need something more
elementary, try

  Pielou, E.C. 1984
  The interpretation of ecological data: A primer on classification
  and ordination
  Published by Wiley
  LC call number: QH541.15.S72P54 1984

Lots of simple, worked-out examples with diagrams. The different
varieties of PC are identified.

Charles Packer, PACKER@amarna.gsfc.nasa.gov 

------------

This is one well-defined method. For the theory see for example:
Lebert, Morineau and Warwick: Multivariate Descriptive Analysis (Wiley, 1984).
Mardia, Kent and Bibby: Multivariate Analysis (Academic Press, 1979).
SAS/STAT User's Guide 1&2.

Helgi Thorsson, helgith@rhi.hi.is 

------------

Regarding your query about references to the PC techniques, I talked to
Doris (who is the real expert here on that topic), and she suggested:

1) Brillinger, D. R. (1981) Time Series, Data Analysis and Theory.
   Holden Day, San Francisco. (esp. Chapter 9)

2) Molenaar, P. C. M. (1987) Dynamic factor analysis in the frequency
   domain: causal modeling of multivariate psychophysiological time
   series. Multivariate Behavioral Research, 22, 329-353. (this is a dandy
   since it deals explicitly with changing spectra, but I find it rough
   going.)

Fred Wightman,  Univ. of Wisconsin at Madison, WIGHTMAN@waisman.wisc.edu 

------------

A standard reference is Morrison's Multivariate Statistical Methods, 
that shows how PC fits into the scheme of things. Preisendorfer's
PC analysis in meterology and oceanography is also a good book and
has a very thorough list of references of PC applications.

You should read (though it is difficult) Richman's (1986) Rotation of PC
in J. of Climatology, 6, 293-335.

Arthur J. Mariano  U. of Miami, mariano@umigw.miami.edu

------------

One book that may be of use is part of the Sage series:
   Dunteman, George H. (1989). Principal Components Analysis. Sage Publications.
   ISBN number: 0-8039-3014-2

Wynne Chin, The University of Calgary, chin@acs.ucalgary.ca 

------------

Try:

Principal Component Analysis
by I. T. Jolliffe
Springer Series in Statistics
1986

Stephanie Butler, TI Semiconductor Process and Design Center, 
butler%epcot@ti.com 


Aside from Bill Venables remarks, there are several basic statistical
texts which may help.  Here's a reference to one with a more rigorous
text listed second.

Cooley and Lohnes "Multivariate Data Analysis"

Preisendorfer "Principal Component Analysis in Meteorology and
               Oceanography"

Mike Richman, The University of Oklahoma, richman@reepicheep.gcn.uoknor.edu

=====================================================================