Engineering topics

Scargle, Jeffrey D.

Publications and source records attributed to Scargle, Jeffrey D..

At least 19 records

NASA NTRS2017

Time and Measurement Days

Questions in data analysis involving the concepts of time and measurement are often pushed into the background or reserved for a philosophical discussion. Some examples are: a) Is causality a consequence of the laws of physics, or can the arrow of time be reversed? b) Can we determine the arrow of time of an event? c) Do we need the continuum hypothesis for the underlying function in any measurement process? d) Can we say anything about the analyticity of the underlying process of an event? e) Would it be valid to model a non-analytical process as function of time? f) What are the implications of all these questions for classical Fourier techniques? However, in the age of big data gathered either from space missions supplying ultra-precise long time series, or e.g. LIGO data from the ground, the moment to bring these questions to the foreground seems arrived. The limitations of our understanding of some fundamental processes is emphasized by the lack of solution for problems open for more than 2 decades, such as the non-detection of solar g-modes, or the modal identification of main sequence stellar pulsators like delta Scuti stars. Flicker noise or 1/f noise, for example, attributed in solar-like stars to granulation, is analyzed mostly only to apply noise reduction techniques, neither considering the classical problem of 1/f noise that was introduced a 100 years ago, nor taking into account ergodic or non-ergodic solutions that make inapplicable spectral analysis techniques in practice. This topic was discussed by Nicholas W. Watkins during the ITISE meeting held in Granada in 2016. There he presented preliminary results of his research on Mandelbrot's related work. We reproduce here his quotation of Mandelbrot (1999) "There is a sharp contrast between a highly anomalous ("non-white") noise that proceeds in ordinary clock time and a noise whose principal anomaly is that it is restricted to fractal time", suggesting a connection with the above proposed topics that could be phrased as the following additional questions:a) Is self-organized criticality (SOC) frequent in astrophysical phenomena? b) Could all fractals in nature be considered stochastic? c) Could we establish mathematical/physical relationships between chaotic and fractal behaviors in time series? d) Could the differences between fractals and chaos in terms of analyticity be used to understand the residuals of the fitting of stellar light curves? In this meeting we would like to approximate these problems from a holistic and multidisciplinary perspective, taking into account not only technical issues but also the deeper implications. In particular the concept of connectivity (introduced in Pascual-Granado et al. A&A, 2015) could be used to implement, within the framework of ARMA processes, an "arrow of time" (see attached document), and so studying the possible implications in the concept of time as envisaged by Watkins.

data analysis↗

NASA NTRS2014

Chromospheric Variability: Analysis of 36 years of Time Series from the National Solar Observatory/Sacramento Peak Ca II K-line Monitoring Program

Analysis of more than 36 years of time series of seven parameters measured in the NSO/AFRL/Sac Peak K-line monitoring program elucidates five elucidates five components of the variation: (1) the solar cycle (period approx. 11 years), (2) quasi-periodic variations (periods approx 100 days), (3) a broad band stochastic process (wide range of periods), (4) rotational modulation, and (5) random observational errors. Correlation and power spectrum analyses elucidate periodic and aperiodic variation of the chromospheric parameters. Time-frequency analysis illuminates periodic and quasi periodic signals, details of frequency modulation due to differential rotation, and in particular elucidates the rather complex harmonic structure (1) and (2) at time scales in the range approx 0.1 - 10 years. These results using only full-disk data further suggest that similar analyses will be useful at detecting and characterizing differential rotation in stars from stellar light-curves such as those being produced by NASA's Kepler observatory. Component (3) consists of variations over a range of timescales, in the manner of a 1/f random noise process. A timedependent Wilson-Bappu effect appears to be present in the solar cycle variations (1), but not in the stochastic process (3). Component (4) characterizes differential rotation of the active regions, and (5) is of course not characteristic of solar variability, but the fact that the observational errors are quite small greatly facilitates the analysis of the other components. The recent data suggest that the current cycle is starting late and may be relatively weak. The data analyzed in this paper can be found at the National Solar Observatory web site http://nsosp.nso.edu/cak_mon/, or by file transfer protocol at ftp://ftp.nso.edu/idl/cak.parameters.

Solar physics↗

NASA NTRS2013

Alanna Conners and the Origins of Principled Data Analysis

Alanna was one of the most important pioneers in the development of not just sophisticated algorithms for analyzing astronomical data but more importantly an overall viewpoint emphasizing the use of statistically sound principles in place of blind application of cook-book recipes, or black boxes. I will outline some of the threads of this viewpoint, emphasizing time series data, with a focus on the importance of these developments for the Age of Digital Astronomy that we are entering.

age of digital astronomy↗

NASA NTRS2013

Studies in Astronomical Time Series Analysis. VI. Bayesian Block Representations

This paper addresses the problem of detecting and characterizing local variability in time series and other forms of sequential data. The goal is to identify and characterize statistically significant variations, at the same time suppressing the inevitable corrupting observational errors. We present a simple nonparametric modeling technique and an algorithm implementing it-an improved and generalized version of Bayesian Blocks [Scargle 1998]-that finds the optimal segmentation of the data in the observation interval. The structure of the algorithm allows it to be used in either a real-time trigger mode, or a retrospective mode. Maximum likelihood or marginal posterior functions to measure model fitness are presented for events, binned counts, and measurements at arbitrary times with known error distributions. Problems addressed include those connected with data gaps, variable exposure, extension to piece- wise linear and piecewise exponential representations, multivariate time series data, analysis of variance, data on the circle, other data modes, and dispersed data. Simulations provide evidence that the detection efficiency for weak signals is close to a theoretical asymptotic limit derived by [Arias-Castro, Donoho and Huo 2003]. In the spirit of Reproducible Research [Donoho et al. (2008)] all of the code and data necessary to reproduce all of the figures in this paper are included as auxiliary material.

signal detection↗

NASA NTRS2011

Heterogeneity in Short Gamma-Ray Bursts

We analyze the Swift/BAT sample of short gamma-ray bursts, using an objective Bayesian Block procedure to extract temporal descriptors of the bursts' initial pulse complexes (IPCs). The sample comprises 12 and 41 bursts with and without extended emission (EE) components, respectively. IPCs of non-EE bursts are dominated by single pulse structures, while EE bursts tend to have two or more pulse structures. The medians of characteristic timescales - durations, pulse structure widths, and peak intervals - for EE bursts are factors of approx 2-3 longer than for non-EE bursts. A trend previously reported by Hakkila and colleagues unifying long and short bursts - the anti-correlation of pulse intensity and width - continues in the two short burst groups, with non-EE bursts extending to more intense, narrower pulses. In addition we find that preceding and succeeding pulse intensities are anti-correlated with pulse interval. We also examine the short burst X-ray afterglows as observed by the Swift/XRT. The median flux of the initial XRT detections for EE bursts (approx 6 X 10(exp -10) erg / sq cm/ s) is approx > 20 x brighter than for non-EE bursts, and the median X-ray afterglow duration for EE bursts (approx 60,000 s) is approx 30 x longer than for non-EE bursts. The tendency for EE bursts toward longer prompt-emission timescales and higher initial X-ray afterglow fluxes implies larger energy injections powering the afterglows. The longer-lasting X-ray afterglows of EE bursts may suggest that a significant fraction explode into more dense environments than non-EE bursts, or that the sometimes-dominant EE component efficiently p()wers the afterglow. Combined, these results favor different progenitors for EE and non-EE short bursts.

Norris, Jay P.↗

NASA NTRS2005

Influence of Coronal Abundance Variations

The PI of this project was Jeff Scargle of NASA/Ames. Co-I's were Alma Connors of Eureka Scientific/Wellesley, and myself. Part of the work was subcontracted to Eureka Scientific via SAO, with Vinay Kashyap as PI. This project was originally assigned grant number NCC2-1206, and was later changed to NCC2-1350 for administrative reasons. The goal of the project was to obtain, derive, and develop statistical and data analysis tools that would be of use in the analyses of high-resolution, high-sensitivity data that are becoming available with new instruments. This is envisioned as a cross-disciplinary effort with a number of "collaborators" including some at SA0 (Aneta Siemiginowska, Peter Freeman) and at the Harvard Statistics department (David van Dyk, Rostislav Protassov, Xiao-li Meng, Epaminondas Sourlas, et al). We have developed a new tool to reliably measure the metallicities of thermal plasma. It is unfeasible to obtain high-resolution grating spectra for most stars, and one must make the best possible determination based on lower-resolution, CCD-type spectra. It has been noticed that most analyses of such spectra have resulted in measured metallicities that were significantly lower than when compared with analyses of high- resolution grating data where available (see, e.g., Brickhouse et al., 2000, ApJ 530,387). Such results have led to the proposal of the existence of so-called Metal Abundance Deficient, or "MAD" stars (e.g., Drake, J.J., 1996, Cool Stars 9, ASP Conf.Ser. 109, 203). We however find that much of these analyses may be systematically underestimating the metallicities, and using a newly developed method to correctly treat the low-counts regime at the high-energy tail of the stellar spectra (van Dyk et al. 2001, ApJ 548,224), have found that the metallicities of these stars are generally comparable to their photospheric values. The results were reported at the AAS (Sourlas, Yu, van Dyk, Kashyap, and Drake, 2000, BAAS 196, v32, #54.02), and at the conference on Statistical Challenges in Modem Astronomy (Sourlas, van Dyk, Kashyap, Drake, and Pease, 2003, SCMA 111, Eds. E.D.Feigelson, G.J.Babu, New York:Springer, p489-490). We also described the limitations of one of the most egregiously misused and misapplied statistical tests in astrophysical literature, the F-test for verifying model components (Protassov, van Dyk, Connors, Kashyap, and Siemiginowska, 2002, ApJ, 571,545). Indeed, a search through the ApJ archives turned up 170 papers in the 5 previous years that used the F-test explicitly in some form or the other, and with the vast majority of them not using it correctly! Indeed, looking at just 4 issues of the ApJ in 2001, we found 13 instances of its use, of which nine were demonstrably incorrect. Clearly, it is difficult to understate the importance of this issue. We also worked on speeding up Bayes Blocks and Sparse Bayes Blocks algorithms to make them more tractable for large searches. We also supported staistics students and postdocs in both explicit physics- model-based (spectra with tens of thousands of atomic lines) and "model-free" -- i.e. non-parametric or semi-parametric -- algorithms. Work on using more of the latter is just beginning; while using multi-scale methods for Poisson imaging has come to hition. In fact, "An Image Restoration Technique with Error Estimates", by D. Esch, A. Connors, M. Karovska, and D. van Dyk, was published by ApJ (Esch et a1.2004, ApJ, 610, 1213). The code has been delivered to M. Karovska for CXC; and is available for beta-testing upon request. The other large project we worked on was on the self-consistent modeling of logN-logs curves in the Poisson limit. logN-logs curves are a fundamental tool in the study of source populations, luminosity functions, and cosmological parameters. However, their determination is hampered by statistical effects such as the Eddington bias, incompleteness due to detection efficiency, faint source flux fluctuations, etc. We have develed a new and powerful method using the full Poisson machinery that allows us to model the logN-logs distribution of X-ray sources in a self-consistent manner. Because we properly account for all the above statistical effects, our modeling is valid over the full range of the data, and not just for strong sources, as is normally done. Using a Bayesian approach and modeling the fluxes with known functional forms such as simple or broken power-laws, and conditioning the expected photon counts on the fluxes, the background contamination, effective area, detector vignetting, and detection probability, we can delve deeply into the low counts regime and extend the usefulness of medium sensitivity surveys such as ChAMP by orders of magnitude. The built-in flexibility of the algorithm also allows a simultaneous analysis of multiple datasets. We have applied this analysis to a set a Chandra observations (Sourlas, Kashyap, Zezas, van Dyk, 2004, HEAD #8, #16.32)

Scargle, Jeffrey D.↗

NASA NTRS2004

Discovery of Planetary Systems With SIM

We are witnessing the birth of a new observational science: the discovery and characterization of extrasolar planetary systems. In the past five years, over 70 extrasolar planets have been discovered by precision Doppler surveys, most by members of this SIM team. We are using the data base of information gleaned from our Doppler survey to choose the best targets for a new SIN planet search. In the same way that our Doppler database now serves SIM, our team will return a reconnaissance database to focus Terrestrial Planet Finder (TPF) into a more productive, efficient mission.

Marcy, Geoffrey W.↗

NASA NTRS2001

Advanced Statistical and Data Analysis Tools for Astrophysics

The goal of the project is to obtain, derive, and develop statistical and data analysis tools that would be of use in the analyses of high-resolution, high-sensitivity data that are becoming available with new instruments. This is envisioned as a cross-disciplinary effort with a number of collaborators.

Kashyap, V.↗

NASA NTRS1999

Publication Bias ( The "File-Drawer Problem") in Scientific Inference

Publication bias arises whenever the probability that a study is published depends on the statistical significance of its results. This bias, often called the file-drawer effect since the unpublished results are imagined to be tucked away in researchers' file cabinets, is potentially a severe impediment to combining the statistical results of studies collected from the literature. With almost any reasonable quantitative model for publication bias, only a small number of studies lost in the file-drawer will produce a significant bias. This result contradicts the well known Fail Safe File Drawer (FSFD) method for setting limits on the potential harm of publication bias, widely used in social, medical and psychic research. This method incorrectly treats the file drawer as unbiased, and almost always miss-estimates the seriousness of publication bias. A large body of not only psychic research, but medical and social science studies, has mistakenly relied on this method to validate claimed discoveries. Statistical combination can be trusted only if it is known with certainty that all studies that have been carried out are included. Such certainty is virtually impossible to achieve in literature surveys.

Scargle, Jeffrey D.↗

NASA NTRS1997

Bayesian Blocks: A New Method to Analyze Photon Counting Data

A Bayesian analysis of photon-counting data leads to a new time-domain algorithm for detecting localized structures (bursts), revealing pulse shapes, and generally characterizing intensity variations. The raw counting data -- time-tag events (TTE), time-to-spill (TTS) data, or binned counts -- is converted to a maximum likelihood segmentation of the observation into time intervals during which the photon arrival rate is perceptibly constant -- i.e. has a fixed intensity without statistically significant variations. The resulting structures, Bayesian Blocks, can be thought of as bins with arbitrary spacing determined by the data. The method itself sets no lower limit to the time scale on which variability can be detected. We have applied the method to RXTE data on Cyg X-1, yielding information on this source's short-time-scale variability.

Scargle, Jeffrey D.↗

NASA NTRS1996

Statistical Challenges in Modern Astronomy II

Wavelets are the central idea of a broad framework for thinking about displaying, and analyzing data - not just a new specialized technique. They are localized basis functions that provide an extremely flexible and efficient representation of time series. Current developments in the theory and application of wavelets are yielding powerful multiresolution techniques for the analysis of time series and image data. Wavelet methods can effect noise removal. characterization of stochastic behavior, depiction of power spectrum time-variations, and data compression - for time series data, image data, and data of higher dimensions. I review published applications of wavelets in astronomy.

Scargle, Jeffrey D.↗

NASA NTRS1995

The Dripping Handrail Model: Transient Chaos in Accretion Systems

We define and study a simple dynamical model for accretion systems, the "dripping handrail" (DHR). The time evolution of this spatially extended system is a mixture of periodic and apparently random (but actually deterministic) behavior. The nature of this mixture depends on the values of its physical parameters - the accretion rate, diffusion coefficient, and density threshold. The aperiodic component is a special kind of deterministic chaos called transient chaos. The model can simultaneously exhibit both the quasiperiodic oscillations (QPO) and very low frequency noise (VLFN) that characterize the power spectra of fluctuations of several classes of accretion systems in astronomy. For this reason, our model may be relevant to many such astrophysical systems, including binary stars with accretion onto a compact object - white dwarf, neutron star, or black hole - as well as active galactic nuclei. We describe the systematics of the DHR's temporal behavior, by exploring its physical parameter space using several diagnostics: power spectra, wavelet "scalegrams," and Lyapunov exponents. In addition, we note that for large accretion rates the DHR has periodic modes; the effective pulse shapes for these modes - evaluated by folding the time series at the known period - bear a resemblance to the similarly- determined shapes for some x-ray pulsars. The pulsing observed in some of these systems may be such periodic-mode accretion, and not due to pure rotation as in the standard pulsar model.

Young, Karl↗

NASA NTRS1994

Dripping handrails and the quasi-periodic oscillations of the AM Herculis objects

AM Her objects exhibit periodic, quasi-periodic, and aperiodic variability on timescales ranging from seconds to years. Here, we investigate a process for the production of aperiodic and quasi-periodic accretion rate fluctuations. We consider the nonlinear dynamical model known as the dripping handrail (DHR). The DHR, basically a model for certain types of spatially extended systems and loosely based on water condensing on and dripping off a handrail, has recently been used as a model for the quasi-periodic oscillations (QPO) and very low frequency noise of the low-mass X-ray binary Sco X-1. Here, we show that (1) the DHR is a robust QPO generation process in that it leads to QPO production under a wide range of conditions and assumptions; (2) the phenomenology of the DHR is consistent with the observed aperiodic and quasi-periodic varibility of the AM Her QPO source VV Pup over timescales ranging from 16 ms to 20 s; and (3) a single DHR model can produce both broadband QPOs and features with quality Q greater than 20 as observed in several AM Her QPO sources.

Steiman-Cameron, Thomas Y.↗

NASA NTRS1993

The quasi-periodic oscillations and very low frequency noise of Scorpius X-1 as transient chaos - A dripping handrail?

We present evidence that the quasi-periodic oscillations (QPO) and very low frequency noise (VLFN) characteristic of many accretion sources are different aspects of the same physical process. We analyzed a long, high time resolution EXOSAT observation of the low-mass X-ray binary (LMXB) Sco X-1. The X-ray luminosity varies stochastically on time scales from milliseconds to hours. The nature of this variability - as quantified with both power spectrum analysis and a new wavelet technique, the scalegram - agrees well with the dripping handrail accretion model, a simple dynamical system which exhibits transient chaos. In this model both the QPO and VLFN are produced by radiation from blobs with a wide size distribution, resulting from accretion and subsequent diffusion of hot gas, the density of which is limited by an unspecified instability to lie below a threshold.

Scargle, Jeffrey D.↗

NASA NTRS1990

Studies in astronomical time series analysis. IV - Modeling chaotic and random processes with linear filters

While chaos arises only in nonlinear systems, standard linear time series models are nevertheless useful for analyzing data from chaotic processes. This paper introduces such a model, the chaotic moving average. This time-domain model is based on the theorem that any chaotic process can be represented as the convolution of a linear filter with an uncorrelated process called the chaotic innovation. A technique, minimum phase-volume deconvolution, is introduced to estimate the filter and innovation. The algorithm measures the quality of a model using the volume covered by the phase-portrait of the innovation process. Experiments on synthetic data demonstrate that the algorithm accurately recovers the parameters of simple chaotic processes. Though tailored for chaos, the algorithm can detect both chaos and randomness, distinguish them from each other, and separate them if both are present. It can also recover nonminimum-delay pulse shapes in non-Gaussian processes, both random and chaotic.

Scargle, Jeffrey D.↗

NASA NTRS1989

Studies in astronomical time series analysis. III - Fourier transforms, autocorrelation functions, and cross-correlation functions of unevenly spaced data

This paper develops techniques to evaluate the discrete Fourier transform (DFT), the autocorrelation function (ACF), and the cross-correlation function (CCF) of time series which are not evenly sampled. The series may consist of quantized point data (e.g., yes/no processes such as photon arrival). The DFT, which can be inverted to recover the original data and the sampling, is used to compute correlation functions by means of a procedure which is effectively, but not explicitly, an interpolation. The CCF can be computed for two time series not even sampled at the same set of times. Techniques for removing the distortion of the correlation functions caused by the sampling, determining the value of a constant component to the data, and treating unequally weighted data are also discussed. FORTRAN code for the Fourier transform algorithm and numerical examples of the techniques are given.

Scargle, Jeffrey D.↗

NASA NTRS1989

An introduction to chaotic and random time series analysis

The origin of chaotic behavior and the relation of chaos to randomness are explained. Two mathematical results are described: (1) a representation theorem guarantees the existence of a specific time-domain model for chaos and addresses the relation between chaotic, random, and strictly deterministic processes; (2) a theorem assures that information on the behavior of a physical system in its complete state space can be extracted from time-series data on a single observable. Focus is placed on an important connection between the dynamical state space and an observable time series. These two results lead to a practical deconvolution technique combining standard random process modeling methods with new embedded techniques.

Scargle, Jeffrey D.↗

NASA NTRS1987

An astrometric facility for planetary detection on the space station

An Astrometric Telescope Facility (ATF) for planetary detection is being studied as a potential space station initial operating capability payload. The primary science objective of this mission is the detection and study of planetary systems around other stars. In addition, the facility will be capable of other astrometric measurements such as stellar motions of other galaxies and highly precise direct measurement of stellar distance within the Milky Way Galaxy. The results of a recently completed ATF preliminary systems definition study are summarized. Results of this study indicate that the preliminary concept for the facility is fully capable of meeting the science objective without the development of any new technologies. A simple straightforward operations approach was developed for the ATF. A real-time facility control is not normally required, but does maintain a near real-time ground monitoring capability for the facility and science data stream on a full-time basis. Facility observational sequences are normally loaded once a week. In addition, the preliminary system is designed to be fail-safe and single-fault tolerant. Routine interactions by the space station crew with the ATF will not be necessary, but onboard controls are provided for crew override as required for emergencies and maintenance.

Nishioka, Kenji↗