Vostok ice core, part 1

An ongoing development is better handling irregularly sampled data. This is a very hard problem with no pure solution.

After a lot of investigation and experimentation I have concluded that NDFT/NDFT are of little use, solve nothing, kicks straight back into the input data must be good. Usually involved is approximating and other heuristics.

Looks hard. Run away.

Raw data overlaid with a resampled dataset

For what I am doing a good solution is fix up the dataset using signal processing, kind of trivial, although it will seem black magic to outsiders. (why no blue, pink, white, transparent magic?)

A key is keeping the human brain in the loop, each case is likely to be different with no one size fits all.

I’ve coded up the hard part for a human as an extension of one software package.

Seems to work nicely, as above, the test dataset. Vostok original data sampling ranges from 60 through 600 years.

An XLS with the original data and resample dataset is here vostoke-temperature-a

Reference

 http://www.ncdc.noaa.gov/paleo/metadata/noaa-icecore-2453.html

 ftp://ftp.ncdc.noaa.gov/pub/data/paleo/icecore/antarctica/vostok/deutnat.txt

The dataset is now clean and trivial for normal tools.

And for those who like first difference…

Hiding anything? Nope. The 1st diff does of course have high frequency noise but is surprisingly small. Only clear term in the part not shown is ~4044y. No idea what that is if anything.

More to be done, always is.

Advertisements