🚀 go-pugleaf

In article <1992Sep30.132441.10320@netnews.whoi.edu> rich@boreas.whoi.edu writes:
>
>
>I am trying to generate spectral estimates from time series which have
>2 problems:
>
>   1) dropouts, i.e. gaps in the time series (perhaps 20% of the values)
>   2) error spikes (perhaps 2% of the values)
>
>The standard window-fft-and-average techniques seem to be inadequate
>for this - I can't think of a good way to handle the gaps (other than
>ignoring them by removing them from the series), and the spikes tend to
>give a lot of "white" energy.
>
>  .......
>
>Can anyone help me with some good hints or references?
>


 The Nov/Dec 1990 issue of Computers in Physics, Numerical Recipes
article, discusses Savitzky-Golay smoothing filters.  A quick perusal
gives me the impression that this may be a method to consider, though
it may not be the best solution for the gappiness problem.

 complete ref:  Press, W H and S A Teukolsky, 1990, Computers in Physics 4(6)
                pp 669-672


We haven't tried this out yet, so we would appreciate hearing from you
one way or the other.

Bruce Huber
Lamont-Doherty Geological Obs.
Palisades NY 10964

In article <1992Sep30.132441.10320@netnews.whoi.edu>
rich@boreas.whoi.edu writes:
>  The time series is a geophysical one (i.e. reddish spectrum),
> with perhaps a few small peaks (tides and what-not)...

I suggest you get a hold of Duncan Agnew's Ph.D thesis
"Strain Tides at Pinon Flat: Analysis and Interpretation"
[UCSD, 1979]. His chapter-4 treats problems similar to
yours: gaps, jumps, noisy stretches etc. His lucid treatment
is easy to follow and the methods he explicits can be readily
implemented. Enjoy ...

Pierre R.

 

  To deal with gappy data there are of course no perfect
techniques. There is no way to recover information that is not
there. Most of the techniques developed to deal with the data
holes involve some assumptions about what the spectrum looks like,
or using some kind of trial spectrum, and then filling in the holes
with some kind of filter. There are one or two classic references that
are worth looking at:

(1) Thompson, R.O.R.Y, 1971, Spectral estimateion from irregularly
spaced data, IEEE Transactions on Geoscience Electronics, GE-9, 107-110.
(2) Horne, J.H. and S.L. Baliunas, 1986, A prescription for period analysis
of unevenly spaced time series, Astrophysical Journal, 302, 757-763.
(3) Sturges, W., 1992, The spectrum of loop current variability from gappy 
data, Journal of Physical Oceanography, to appear this fall, Oct or Nov
1992.

These references should be enough to work with.

Brad deYoung
hysics Department
Memorial University
St. John's, Newfoundland
A1B 3X7

bdeyoung@kean.ucs.mun.ca (INTERNET)  Phone: 709-737-8839
b.deyoung (OMNET)                    FAX  : 709-737-8739

An excellent and well documented method for forming power spectra from
gappy 
data is to use covariances formed from the existing portions of data.  For
example, the alternative methods (and the fortran code to execute them)
can
be found in the book "Digital Spectral Analysis, with applications" by
S.L.
Marple (Prentice-Hall, 1987).

Covariance-based methods are often referred to as "auto-regressive" models
(or AR-spectra;  I can't say why this name has stuck).  Marple reviews the
relative advantages of AR versus "periodogram" (fft) and a few others
(e.g.
Proni's method).

As for the spikes, your on your own.  A method which has worked less
poorly
than others for me in the past is based on using a threshold applied to
the 
difference between each value and the median value in its neighborhood. 
An
unsatisfactory aspect of it is that it has two adjustable (subjective)
parameters (the size of the neighborhood, and the threshold).  A
redeeming 
feature is that there are enough adjustable parameters to make it reject
most
of the points which you would have chosen by eye, if you had the time.

Good luck.

	-jerry smith