🚀 go-pugleaf

Newsgroups: sci.stat.math
Subject: gappy/noisy time series
Followup-To: Message-ID: <1992Sep29.234100.7104@netnews.whoi.edu>
Distribution: world
Organization: Colman Research


Rich Pawlowicz of Organization: Woods Hole Oceanographic Institution
asks the following:

>I am trying to generate spectral estimates from time series which have
>2 problems:
>
>   1) dropouts, i.e. gaps in the time series (perhaps 20% of the values)
>   2) error spikes (perhaps 2% of the values)
>
>The time series is a geophysical one (i.e. reddish spectrum), with
>perhaps a few small peaks (tides and what-not).
>
>The standard window-fft-and-average techniques seem to be inadequate
>for this - I can't think of a good way to handle the gaps (other than
>ignoring them by removing them from the series), and the spikes tend to
>give a lot of "white" energy.
>
>Median filtering can get rid of some of the spikiness, but it does bad
>things to some of the other parts of the signal.
>

My suggestion is to try the following:

1. take a dft (direct Fourier transform) of the unequally spaced
series. Take the inverse dft at equally spaced intervals.  The real
part will be your recovered "signal" and the imaginary part is
the noise estimate or residuals. I seem to remember that this is
equilivant to doing a least squares estimate.  

2. plot the power spectrum and identify a reasonable cutoff frequency
for a low pass filter to remove the effect of the spikes.  Alternatly,
you could just taper off the high freq. terms before taking the
inverse dft above.

3. let me know how you make out.  Also, if possible, i would like to
get a copy of your ``problem'' time series.  I have put together a
time series package in xlisp (submitted to the statlib archieve) and
i would like to exercise it on some real problems.

bill hatch
-- 
Bill Hatch  (301)470-3839(w)  (301)441-1675(h)
Coleman Research, 14504 Greenview Drive, Laurel MD 20708
All opinions expressed are my own - dont blame my employer or alembic systems