🚀 go-pugleaf

bill@alembic.acs.com described a time series problem that included gaps in
data and low probability high amplitude spikes.  There are some reputable
time series techniques for both of these cases separately, and an
integration is probably possible. 

The problem of robust/resistant spectrum estimation in the presence of a
contaminating distribution that might manifest itself with spikes or
"glint" has been well-studied in the last two decades.  The most flexible
approach seems to be to fit a robust autoregression, then infer a spectrum
from the ar time series model.  The LS algorithms for autoregression
can be robustified against outliers in direct analogy to the extensive
work done recently for regular regression.  Fortunately for those of us
who have forgotten how to program, the statistical package Splus has
implemented a number of tools in support of robust autoregression of messy
time series.  Also included in the package is a function for inferring a
spectrum from the estimated robust ar coefficients.  

An accessible classical review article on robust spectral estimation is
"Robust-resistant Spectrum Estimation", R.D. Martin and D.J. Thomson,
Proc. IEEE, vol. 70, no. 9, Sept. 1982.  It is a little long in the tooth,
but lays out the conceptual background in a manner that can be read by
both engineers and statisticians.

There are a number of approaches to missing gaps or uneven spacing of
samples in time series, in both the time and frequency domains.  I'd just
like to mention the very practical and useful work done by Dick Jones, a
Colorado biometrician, over a decade ago.  Jones' approach to both missing
and unevenly spaced data was to embed a continuous ar model inside a
Kalman filter.  The estimates of the ar model parameters are updated by
the Kalman filter only when data occurs in its natural uneven placement in
time.  The Kalman filter-ar model "falls through" sections of missing data
with what amounts to an optimal interpolation, feeding back an error
signal based on the difference between the model prediction and the
observed data when it finally hits a valid data point.  Of course, for
every ar model there is a unique spectrum, so this can be construed to be
a spectral analysis approach.  Jones has had quite a few publications
about this and related topics, for example: 

R.H. Jones, Maximum likelihood fitting of ARMA models to time series with
missing observations, Technometrics, v 22, pp 389-396, 1980

R.H. Jones, Fitting a continuous-time autoregression to discrete data, a
chapter in Applied Time Series Analysis II, D.F. Findley, Ed., New York:
Academic Press, 1980 

(Note also that there was a very influential conference on this topic, the
ONR Symposium on Time Series Analysis of Irregularly Spaced Data, February
10-13, 1983, at Texas A&M.)

Jones very nicely puts source code for his algorithms in appendices of
some of his papers.

But the most modern and accessible implementation of this comes from
Robert Shumway's very practical textbook "Applied Statistical Time Series
Analysis," (Prentice Hall, NJ, 1988), in which he describes the
state-space Kalman filter approach to missing data in a time series in
Chapter 3.  Amazingly enough, he includes software to implement it in a
diskette attached to the back cover. 
-------------------------------------------------------------------------
Robert L. Burr, MSEE, PhD
Research Associate Professor
University of Washington
Seattle, WA 98195
bobburr@u.washington.edu