🚀 go-pugleaf

In response to my request for references on least-squares type 
algorithms which treat variables in both X and Y as having errors 
associated with them, I received the following replies. 

I wish to thank all the respondents and am posting this for the 
information of others who may find it useful. 

I have not had a chance to read most of the papers so I cannot comment 
on the relative value of these various references. 

I will be happy to receive any further comments or references and will 
try to keep this list available should anyone want a copy 

Gerald

----------------------------------------------------------------------------

Look at odrpack (orthogonal distance regression package) from
netlib.  It also uses a Levenberg-Marquardt algorithm for the 
regression.

Chuck Hales
DuPont Company, Central R&D Dept., Scientific Computing Div. 
E320/103, P.O. Box 80320, Wilmington, DE   19880-0320
Internet: halesch@esvax.dnet.dupont.com   phone: (302) 695-4858
___________________________________________________________________________

ODRPACK is a package for orthogonal distance regression.   This is appropriate
when the errors in the X and Y directions are equal.   ODRPACK is TOMS
algorithm 676 (see TOMS volume 15(4)).   You can get the Fortran source from
either netlib or statlib.   You may be able to scale either the X or the Y
variable so that the standard deviations are the same in both directions.

Alan Miller, Quality Improvement Project
CSIRO Division of Mathematics & Statistics, Melbourne, Australia
Phone: +61 3 542-2266   Fax: +61 3 542-2474  E-mail: alan@mel.dms.csiro.au
Mail: CSIRO DMS, Private Bag 10, Clayton, Vic. 3168, Australia
___________________________________________________________________________

There is a package for doing this called ODRPACK. Look in netlib for it. 
I played around with it once. It worked well. 

Herb Schilling , NASA Lewis Research Center , 21000 Brookpark Road, MS 142-5
Cleveland, Ohio 44135 . (216) 433-8955       sshws@convx1.lerc.nasa.gov
___________________________________________________________________________

There's a new FORTRAN package on 'netlib', (i.e. where LINPACK and
others of it's bretheren live) called ODRPACK, which does what
you want, if you're interested.

-Matt Kennel  mbk@inls1.ucsd.edu
-Institute for Nonlinear Science, University of California, San Diego

___________________________________________________________________________

Try the book "Data Reduction and Error Analysis for the Physical
Sciences"  by Philip R Bevington.  It was printed in 1969, and just this
year, one of his colleagues updated the book.  The 1992 version comes with
a disk of the source code in Pascal.  Both versions have the Fortran code.
I just used the book to write a nonlinear least sqaures program using the
Levenberg-Marquardt algorithm.  Works great!

Magdi N. Azer
magdi@uxh.cso.uiuc.edu
___________________________________________________________________________

magdi@uxh.cso.uiuc.edu (Magdi N Azer) writes:

>Try the book "Data Reduction and Error Analysis for the Physical
>Sciences"  by Philip R Bevington.  It was printed in 1969, and just this
>year, one of his colleagues updated the book.  The 1992 version comes with
>a disk of the source code in Pascal.  Both versions have the Fortran code.
>I just used the book to write a nonlinear least sqaures program using the
>Levenberg-Marquardt algorithm.  Works great!

The 1969 Bevington copy only says that the usual ordinary linear squares can
be used if the errors in x are sufficiently small to ignore and does not offer
anything for the case where the x and y errors are comparable. Some
straight-forward articles for the case where both variables are subject to
errors are the following:

Mandel, John (1984) "Fitting Straight Lines When Both Variables are Subject to
Error", Journal of Quality Technology, v. 16(1):1.

Reilman, Miriam, Richard Gunst, and Mani Lakshminarayanan (1986) "Stochastic
Regression with Errors in Both Variables", Journal of Quality Technology,
v. 18(3): 162.

Feigelson, Eric and Gutti Babu (1992) "Linear Regression in Astronomy II",
Astrophysical Journal--will appear in Sept 20, 1992 (I have a preprint).

This subject has been the topic of heated debate for decades. The last
reference has a listing of the important papers in various disciplines in
its bibliography section. Code can be received by sending a request to
code@stat.psu.edu. Be aware that what algorithm you use depends on the
error structure of the (x,y) values and whether or not you want to find the
slope of the line or you want to predict a y given a x.

nick
strobel@dirac.phys.washington.edu
___________________________________________________________________________

  This may have been mentioned already but I believe that when
you have "errors in both x and y" you have a "Total Least
Squares" problem.  So you might want to check out the literature
on total least squares.  In particular, SIAM has a book, basically
a compilation of articles, out on total least squares.

Mark Inlow
SAIC
(619) 546-6978
inlowm@jupiter.saic.com
___________________________________________________________________________

From: hrubin@snap.stat.purdue.edu (Herman Rubin)
Message-Id: <9208171336.AA23418@snap.stat.purdue.edu>

This was a rather hot topic from the late 1930s to the mid 1950s, and probably 
the most readable general work on the identification and estimation problem is 
by T. A. Jeeves in _Annals of Mathematical Statistics_ 25, pp 714-723.  There 
are further results by Stein in volume 1 of the _Proceedings of the Third 
Berkeley Symposium of Mathematical Statistics and Probability_.  As far as 
making these methods practical, little has been done before or since.  The big 
problem is a lack of identifiability, and if everything is normal, there is no 
solution.  This was certainly known in the 1930s. 

There are situations in which reasonable things can be done, such as if
there are any relevant quantities whose errors can be assumed uncorrelated
with the errors in the variables.  For example, if there is a substantial
time drift, the temporal order of the observations can be used.  If the
observations are roughly linear in time, this will even be quite good.

Much of this was "well known" by those mathematical statisticians working
in the area back then, but it has not remained a major area of research.

----------------------------------------------------------------------------

From: butler@spdc.ti.com (Stephanie Butler)

The method is called Error in Variables.

Try Technometrics, Nov. 1991, 33(4), p405 by E. V. Thomas
for example application.

Also:

Measurement Error Models, Wiley & Sons by W. A. Fuller 91987)
-- This book is actually series of articles.  It looks very
   comprehensive.  Check it out!

Stephanie Watts Butler     butler@spdc.ti.com      
Texas Instruments         

----------------------------------------------------------------------------

From: "Dr. Aivars Celmins" (VLD/GSB) <celms@BRL.MIL>

  Your problem (with arbitrary number of erroneous coordinates) is
  discussed and solved in

   A. Celmins, Least-Squares Optimization with Implicit Model Functions, in
   "Mathematical Programming with Data Perturbations II", Anthony V. Fiacco,
   Ed., Marcel Dekker, 1983, pp. 131-152.

----------------------------------------------------------------------------

From: Nicolas Strobel <strobel@dirac.phys.washington.edu>

You may be opening a big can of worms with this request! There's been a lot of
argument in various disciplines about this one for decades and often people
in different disciplines are not aware of what other disciplines have done
about this problem.  Anyway here are some references to be sure to include in
your reading list:

Mandel, John (1984) "Fitting Straight Lines When Both Variables are Subject to
Error", Journal of Quality Technology, v. 16(1):1.

Reilman, Miriam, Richard Gunst, and Mani Lakshminarayanan (1986) "Stochastic
Regression with Errors in Both Variables", Journal of Quality Technology,
v. 18(3): 162.

Feigelson, Eric and Gutti Babu (1992) "Linear Regression in Astronomy II",
Astrophysical Journal--will appear in Sept 20, 1992 (I have a preprint). This
one has zillions of references including all the important ones from the
various disciplines. Code can be received by sending a request to 
code@stat.psu.edu.

Nick Strobel
strobel@dirac.phys.washington.edu

----------------------------------------------------------------------------

From: bill@bessel.as.utexas.edu (William H. Jefferys)

Our GaussFit program handles this situation. It
(and the manual) can be obtained from this site
by anonymous FTP. Consult Barbara McArthur
(mca@bessel.as.utexas.edu) for implementation
details if you have problems. The manual contains
a number of references for this problem (called
"errors-in-variables" in the literature).

Bill
----------------------------------------------------------------------------

From: sshws@convx1.lerc.nasa.gov (Herb Schilling)
Organization: NASA Lewis Research Center

Since this problem occurs frequently, I am taking the liberty of
appending a rather long list of references from my personal data base.
It is the result of a brute-force search on keywords and certainly
contains some irrelevant articles.  With equal certainty it also
omits some relevant articles.

Pete Stewart


N.N. Abdelmalek.
On the solution of least squares problems and pseudo-inverses.
Computing, 13:215--228, 1974.

T.W. Anderson.
Estimation of linear functional relationships: Approximate
  distributions and connections with simultaneous equations in econometrics.
Journal of the Royal Statistical Society, 38:1--31, 1976.

T.W. Anderson.
Estimating linear statistical relationships.
Annals of Statistics, 12:1--45, 1984.

M.Arioli, I.S. Duff, and P.P.M. van Rijk.
On the augmented system approach to sparse least-squares problems.
Numerische Mathematik, 55:667--685, 1989.

A.E. Beaton, D.B. Rubin, and J.L. Barone.
The acceptability of regression solutions: Another look at
  computational accuracy.
Journal of the American Statistical Association, 71:158--168,
  1976.

D.A. Belsley.
Assessing the presence of harmful collinearity and other forms of
  weak data through a test for signal-to-noise.
Journal of Econometrics, 20:211--253, 1982.

D.A. Belsley.
Demeaning conditioning diagnostics through centering.
The American Statistician, 38:73--77, 1984.

D.A. Belsley, A.E. Kuh, and R.E. Welsch.
Regression Diagnostics: Identifying Influential Data and Sources
  of Collinearity.
John Wiley and Sons, New York, 1980.

J.Berkson.
Are there two regressions.
Journal of the American Statistical Association, 45:164--180,
  1950.

Error analysis of least squares algorithms.
Technical Report LiTH-MAT-R-1988-22, Department of Mathematics,
  Link{\"oping  University, 1988.

Componentwise backward errors and condition estimates for linear
  least square problems.
Technical Report LiTH-MATH-R-1989-13, Department of Mathematics,
  University, 1989.

Component-wise perturbation analysis and errors bounds for linear
  least square solutions.
BIT, 31:238--244, 1991.
Cited in {\AA ke Bj\""orck's bibliography on least squares, which is
  available by anonymous ftp from math.liu.se in  pub/references.

G.F. Brown, J.B. Kadane, and J.G. Ramage.
The asymptotic bias and mean-squared error of double class
  estimators when the distrubances are small.
International Economic Review, 15:667--679, 1974.

Nancy David and G.W. Stewart.
Hypothesis testing with errors in the variables.
Technical Report TR-1735, University of Maryland Department of
  Computer Science, 1988.

R.B. Davies and B.Hutton.
The effects of errors in the independent variables in linear
  regression.
Biometrika, 62:383--391, 1975.

J.Demmel.
The probability that a numerical analysis problem is difficult.
Mathematics of Computation, 50:449--480, 1988.

Perturbation theory for the least squares problem with linear
  equality constraints.
SIAM Journal on Numerical Analysis, 17:338--350, 1980.

M.Feldstein.
Errors in variables: A consistent estimator with smaller {MSE in
  finite samples.
Journal of the American Statistical Association, 69:990--996,
  1974.

W.A. Fuller.
Measurement Error Models.
John Wiley, New York, 1987.

P.P. Gallo.
Consistency of regression estimates when some variables are subject
  to error.
Mimeo Series 1328, Institute of Statistics, University of North
  Carolina at Chapel Hill, 1981.

V.J. Geraci.
Identification of simultaneous equation models with measurement
  error.
Journal of Econometrics, 4:263--283, 1976.

V.J. Geraci.
Estimation of simultaneous equation models with measurement error.
Econometrica, 45:1243--1255, 1977.

G.H. Golub and V.Pereyra.
The differentiation of pseudoinverses and nonlinear least squares
  problems whose variables separate.
SIAM Journal on Numerical Analysis, 10:413--432, 1973.

G.H. Golub and V.Pereyra.
Differentiation of pseudoinverses, separable nonlinear least squares
  problems and other tales.
In M.Z. Nashed, editor, Generalized Inverses and Applications,
  pages 303--324, New York, 1976. Academic Press.

G.H. Golub and C.F. VanLoan.
An analysis of the total least squares problem.
SIAM Journal on Numerical Analysis, 17:883--893, 1980.

G.H. Golub and J.H. Wilkinson.
Note on the iterative refinement of least squares solution.
Numerische Mathematik, 9:139--148, 1966.

Z.Griliches.
Errors in variables and other unobservables.
Econometrica, 42:971--998, 1974.

N.J. Higham.
Computing error bounds for regression problems.
In P.J. Brown and W.A. Fuller, editors, Contemporary
  Mathematics 112: Statistical Analysis of Measurement Error Models and
  Applications, pages 195--210, Providence, Rhode Island, 1990. American
  Mathematical Society.

S.D. Hodges and P.G. Moore.
Data uncertainties and least squares regression.
Applied Statistics, 21:185--195, 1972.

J.B. Kadane.
Comparison of k-class estimators when the disturbances are small.
Econometrica, 39:723--737, 1971.

R.Karlsson, J.-G. Sun, and B.Wald\`en.
Optimal backward perturbation bounds for the linear least squares
  problem.
Technical Report LiTH-MAT-R-1992-06, Department of Mathematics,
  Link{\"oping University, 1992.
Cited in {\AA ke Bj\"orck's bibliography on least squares, which is
  available by anonymous ftp from {\tt math.liu.se in {\tt pub/references.

H.S. Konijn.
Maximum likelihood estimators and confidence intervals for a simple
  errors in variables model.
Communications in Statistics: Part A, 10:983--996, 1981.

C.H. Kummell.
Reduction of observation equations which contain more than one
  observed quantity.
The Analyst, 6:97--105, 1879.

C.L. Lawson and R.J. Hanson.
Solving Least Squares Problems.
Prentice Hall, Englewood Cliffs, New Jersey, 1974.

D.V. Lindley.
Regression lines and the linear functional relationship.
Supplement to the Journal of the Royal Statistical Society,
  9:218--244, 1947.

P.L{\"otstedt.
Perturbation bounds for the linear least squares problem subject to
  linear inequality constraints.
BIT, 23:500--519, 1983.

A.Madansky.
The fitting of straight lines when both variables are subject to
  error.
Journal of the American Statistical Association, 54:173--205,
  1959.

D.W. Marquardt.
Comment on {Smith and {Campbell (1980).
Journal of the American Statistical Association, 75:87--91,
  1980.

P.A.P. Moran.
Estimating structural and functional relationships.
Journal of Multivariate Analysis, 1:232--255, 1971.

M.Paardekooper.
Inverse perturbations for approximations of least squares solutions.
Technical Report 79.083, Katholieke Hogeschool Tilburg, Faculteit der
  Economische Wetenschappen, 1979.

C.C. Paige.
Computer solution and perturbation analysis of generalized linear
  least squares problems.
Mathematics of Computation, 33:171--184, 1979.

O.Reiers{\ol.
Identifiability of a linear relation between variables which are
  subject to error.
Econometrica, 18:375--389, 1950.

K.C. Schmidt.
Selecting subsets of variables in the linear regression problem.
Master's thesis, Department of Mathematics, University of Maryland,
  College Park, 1979.

H.Schwetlick and V.Tiller.
Numerical methods for estimating parameters in nonlinear models with
  errors in the variables.
Technometrics, 27:17--24, 1985.
Cited in {\AA ke Bj\"orck's bibliography on least squares, which is
  available by anonymous ftp from {\tt math.liu.se in {\tt pub/references.

P.Sprent.
A generalized least-squares approach to linear functional
  relationships.
Journal of The Royal Statistical Society, Series B,
  28:278--297, 1966.

V.K. Srivastava.
Disturbance-variance estimation in simultaneous equations when
  disturbances are small.
Journal of the American Statistical Association, 67:164--168,
  1972.

G.W. Stewart.
On the continuity of the generalized inverse.
SIAM Journal on Applied Mathematics, 17:33--45, 1969.

G.W. Stewart.
On the perturbation of pseudo-inverses, projections, and linear least
  squares problems.
SIAM Review, 19:634--662, 1977.

G.W. Stewart.
Sensitivity coefficients for the effects of errors in the independent
  variables in a linear regression.
Technical Report TR-571, Computer Science Department, University of
  Maryland, 1977.

G.W. Stewart.
A generalization of the {Eckart-Young approximation theorem.
Technical Report TR-1325, Department of Computer Science, the
  University of Maryland, 1983.

G.W. Stewart.
A nonlinear version of {Gauss's minimum variance theorem with
  applications to an errors-in-the-variables model.
Technical Report TR-1263, Department of Computer Science, University
  of Maryland, 1983.

G.W. Stewart.
Collinearity and least squares regression.
Statistical Science, 2:68--100, 1987.

G.W. Stewart.
Perturbation theory and least squares with errors in the variables.
In P.J. Brown and W.A. Fuller, editors, Contemporary
  Mathematics 112: Statistical Analysis of Measurement Error Models and
  Applications, pages 171--181, Providence, Rhode Island, 1990. American
  Mathematical Society.

G.W. Stewart.
Stochastic perturbation theory.
SIAM Review, 32:576--610, 1990.

B.F. Swindel and D.R. Bower.
Rounding errors in the independent variables in a general linear
  model.
Technometrics, 14:215--218, 1972.

A.vander Sluis.
Condition numbers and equilibration of matrices.
Numerische Mathematik, 14:14--23, 1969.

A.vander Sluis.
Stability of the solutions of linear least squares problems.
Numerische Mathematik, 23:241--254, 1975.

S.VanHuffel and J.Vandewalle.
Subset selection using the total least squares approach in
  collinearity problems with errors in the variables.
Linear Algebra and its Applications, 88/89:695--714, 1987.

B.Wald\'en, R.Karlson, and J.-G. Sun.
Optimal backward perturbation bounds for the linear least squares
  problem.
Technical Report LiTH-MAT-R-1992-06, Department of Mathematics,
  Link{\"oping University, 1992.

P.-{\AA. Wedin.
On the almost rank deficient case of the least squares problem.
BIT, 13:344--354, 1973.

P.-{\AA. Wedin.
Perturbations of the pseudo-inverse and the linear least squares
  problem analysed with suitable matrix decompositions.
Technical Report UMINF--69.78, Institute of Information Processing,
  University of Ume\aa, 1978.

P.-{\AA. Wedin.
Perturbation theory and condition numbers for generalized and
  constrained linear least squares problems.
Technical Report UMINF--125.85, Institute of Information Processing,
  University of Ume\aa, 1985.

P.-{\AA. Wedin.
Perturbation theory and condition numbers for generalized and
  constrained least squares problems.
Technical Report S-901-87, Institute of Information Processing,
  University of Ume\aa, 1987.

M.Wei.
Perturbation of the least squares problem.
Linear Algebra and Its Applications, 141:177--182, 1990.


-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
diamong@gov.on.ca          | There's never enough time  |  \ 
I spoke only for myself    | to do all the nothing you  |    \ 
and my bike !              | want.  -  Calvin           |      \ ^. .>