Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data reduction pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2017

Data Reduction Pipeline for the CHARIS Integral-Field Spectrograph I: Detector Readout Calibration and Data Cube Extraction

We present the data reduction pipeline for CHARIS, a high-contrast integral-field spectrograph for the Subaru Telescope. The pipeline constructs a ramp from the raw reads using the measured nonlinear pixel response and reconstructs the data cube using one of three extraction algorithms: aperture photometry, optimal extraction, or chi-squared fitting. We measure and apply both a detector flatfield and a lenslet flatfield and reconstruct the wavelength- and position-dependent lenslet point-spread function (PSF) from images taken with a tunable laser. We use these measured PSFs to implement a chi-squared-based extraction of the data cube, with typical residuals of approximately 5 percent due to imperfect models of the under-sampled lenslet PSFs. The full two-dimensional residual of the chi-squared extraction allows us to model and remove correlated read noise, dramatically improving CHARIS's performance. The chi-squared extraction produces a data cube that has been deconvolved with the line-spread function and never performs any interpolations of either the data or the individual lenslet spectra. The extracted data cube also includes uncertainties for each spatial and spectral measurement. CHARIS's software is parallelized, written in Python and Cython, and freely available on github with a separate documentation page. Astrometric and spectrophotometric calibrations of the data cubes and PSF subtraction will be treated in a forthcoming paper.

Brandt, Timothy D.↗
NASA NTRS2015

PISCES High Contrast Integral Field Spectrograph Simulations and Data Reduction Pipeline

The PISCES (Prototype Imaging Spectrograph for Coronagraphic Exoplanet Studies) is a lenslet array based integral field spectrograph (IFS) designed to advance the technology readiness of the WFIRST (Wide Field Infrared Survey Telescope)-AFTA (Astrophysics Focused Telescope Assets) high contrast Coronagraph Instrument. We present the end to end optical simulator and plans for the data reduction pipeline (DRP). The optical simulator was created with a combination of the IDL (Interactive Data Language)-based PROPER (optical propagation) library and Zemax (a MatLab script), while the data reduction pipeline is a modified version of the Gemini Planet Imager's (GPI) IDL pipeline. The simulations of the propagation of light through the instrument are based on Fourier transform algorithms. The DRP enables transformation of the PISCES IFS data to calibrated spectral data cubes.

The PISCES (Prototype Imaging Spectrograph for Cor↗
NASA NTRS2012

New Techniques for High-Contrast Imaging with ADI: The ACORNS-ADI SEEDS Data Reduction Pipeline

We describe Algorithms for Calibration, Optimized Registration, and Nulling the Star in Angular Differential Imaging (ACORNS-ADI), a new, parallelized software package to reduce high-contrast imaging data, and its application to data from the Strategic Exploration of Exoplanets and Disks (SEEDS) survey. We implement seyeral new algorithms, includbg a method to centroid saturated images, a trimmed mean for combining an image sequence that reduces noise by up to approx 20%, and a robust and computationally fast method to compute the sensitivitv of a high-contrast obsen-ation everywhere on the field-of-view without introducing artificial sources. We also include a description of image processing steps to remove electronic artifacts specific to Hawaii2-RG detectors like the one used for SEEDS, and a detailed analysis of the Locally Optimized Combination of Images (LOCI) algorithm commonly used to reduce high-contrast imaging data. ACORNS-ADI is efficient and open-source, and includes several optional features which may improve performance on data from other instruments. ACORNS-ADI is freely available for download at www.github.com/t-brandt/acorns_-adi under a BSD license

Brandt, Timothy D.↗
DOE OSTI2026

Data Release 1 of the Dark Energy Spectroscopic Instrument

In 2021 May the Dark Energy Spectroscopic Instrument (DESI) collaboration began a 5 yr spectroscopic redshift survey to produce a detailed map of the evolving three-dimensional structure of the Universe between z = 0 and z ≈ 4. DESI’s principal scientific objectives are to place precise constraints on the equation of state of dark energy, the gravitationally driven growth of large-scale structure, and the sum of the neutrino masses, and to explore the observational signatures of primordial inflation. We present DESI DR1, which consists of all data acquired during the first 13 months of the DESI main survey, as well as a uniform reprocessing of the DESI Survey Validation data, which were previously made public in the DESI Early Data Release. The DR1 main survey includes high-confidence redshifts for 18.7M objects, of which 13.1M are spectroscopically classified as galaxies, 1.6M as quasars, and 4M as stars, making DR1 the largest sample of extragalactic redshifts ever assembled. We summarize the DR1 observations, the spectroscopic data-reduction pipeline and data products, large-scale structure catalogs, value-added catalogs, and describe how to access and interact with the data. In addition to fulfilling its core cosmological objectives with unprecedented precision, we expect DR1 to enable a wide range of transformational astrophysical studies and discoveries.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2013

The DEEP2 Galaxy Redshift Survey: Design, Observations, Data Reduction, and Redshifts

We describe the design and data analysis of the DEEP2 Galaxy Redshift Survey, the densest and largest high-precision redshift survey of galaxies at z approx. 1 completed to date. The survey was designed to conduct a comprehensive census of massive galaxies, their properties, environments, and large-scale structure down to absolute magnitude MB = −20 at z approx. 1 via approx.90 nights of observation on the Keck telescope. The survey covers an area of 2.8 Sq. deg divided into four separate fields observed to a limiting apparent magnitude of R(sub AB) = 24.1. Objects with z approx. < 0.7 are readily identifiable using BRI photometry and rejected in three of the four DEEP2 fields, allowing galaxies with z > 0.7 to be targeted approx. 2.5 times more efficiently than in a purely magnitude-limited sample. Approximately 60% of eligible targets are chosen for spectroscopy, yielding nearly 53,000 spectra and more than 38,000 reliable redshift measurements. Most of the targets that fail to yield secure redshifts are blue objects that lie beyond z approx. 1.45, where the [O ii] 3727 Ang. doublet lies in the infrared. The DEIMOS 1200 line mm(exp −1) grating used for the survey delivers high spectral resolution (R approx. 6000), accurate and secure redshifts, and unique internal kinematic information. Extensive ancillary data are available in the DEEP2 fields, particularly in the Extended Groth Strip, which has evolved into one of the richest multiwavelength regions on the sky. This paper is intended as a handbook for users of the DEEP2 Data Release 4, which includes all DEEP2 spectra and redshifts, as well as for the DEEP2 DEIMOS data reduction pipelines. Extensive details are provided on object selection, mask design, biases in target selection and redshift measurements, the spec2d two-dimensional data-reduction pipeline, the spec1d automated redshift pipeline, and the zspec visual redshift verification process, along with examples of instrumental signatures or other artifacts that in some cases remain after data reduction. Redshift errors and catastrophic failure rates are assessed through more than 2000 objects with duplicate observations. Sky subtraction is essentially photon-limited even under bright OH sky lines; we describe the strategies that permitted this, based on high image stability, accurate wavelength solutions, and powerful B-spline modeling methods. We also investigate the impact of targets that appear to be single objects in ground-based targeting imaging but prove to be composite in Hubble Space Telescope data; they constitute several percent of targets at z approx. 1, approaching approx. 5%-10% at z > 1.5. Summary data are given that demonstrate the superiority of DEEP2 over other deep high-precision redshift surveys at z approx. 1 in terms of redshift accuracy, sample number density, and amount of spectral information. We also provide an overview of the scientific highlights of the DEEP2 survey thus far.

Galaxy↗
NASA NTRS1982

Principal Components as a Data Reduction and Noise Reduction Technique

The potential of principal components as a pipeline data reduction technique for thematic mapper data was assessed and principal components analysis and its transformation as a noise reduction technique was examined. Two primary factors were considered: (1) how might data reduction and noise reduction using the principal components transformation affect the extraction of accurate spectral classifications; and (2) what are the real savings in terms of computer processing and storage costs of using reduced data over the full 7-band TM complement. An area in central Pennsylvania was chosen for a study area. The image data for the project were collected using the Earth Resources Laboratory's thematic mapper simulator (TMS) instrument.

Imhoff, M. L.↗
NASA NTRS2012

Demystifying Kepler Data: A Primer for Systematic Artifact Mitigation

The Kepler spacecraft has collected data of high photometric precision and cadence almost continuously since operations began on 2009 May 2. Primarily designed to detect planetary transits and asteroseismological signals from solar-like stars, Kepler has provided high quality data for many areas of investigation. Unconditioned simple aperture time-series photometry are however affected by systematic structure. Examples of these systematics are differential velocity aberration, thermal gradients across the spacecraft, and pointing variations. While exhibiting some impact on Kepler's primary science, these systematics can critically handicap potentially ground-breaking scientific gains in other astrophysical areas, especially over long timescales greater than 10 days. As the data archive grows to provide light curves for 10(exp 5) stars of many years in length, Kepler will only fulfill its broad potential for stellar astrophysics if these systematics are understood and mitigated. Post-launch developments in the Kepler archive, data reduction pipeline and open source data analysis software have occurred to remove or reduce systematic artifacts. This paper provides a conceptual primer for users of the Kepler data archive to understand and recognize systematic artifacts within light curves and some methods for their removal. Specific examples of artifact mitigation are provided using data available within the archive. Through the methods defined here, the Kepler community will find a road map to maximizing the quality and employment of the Kepler legacy archive.

Kinemuchi, K.↗
NASA NTRS2005

An XMM-Newton Monitoring Campaign of the Accretion Flow in IGRJ16318-4848

This grant is associated to a successful XMM-Newton-AO3 observational proposal to monitor the spectrum of the X-ray loud component of the recently discovered binary system IGR J16138-4848, to study the conditions of the accretion flows (and their evolution) in binary system. All four EPIC-PN and MOS observations of the target have now been performed (the last one of the 4, only 3 months ago). The four observations were logarithmically spaced, so to cover timescales from days to months. Data from all four pointings have now been reduced, using the XMM-Newton data reduction pipeline, and spectra and lightcurves from the target have been extracted. For the first three observations we have already performed the observation-by-observation data analysis, by fitting the single EPIC spectra with spectral models that include an intrinsic continuum power law (reduced at low energy by neutral absorption), a 6.4 keV iron emission line (detected in all spectra with varying intensity) and a Compton-reflection component. A Compton reflection component is also detected in all spectra, although at lower significance. The analysis of the fourth and last observation of our monitoring campaign has just recently begun. Next, we will (1) stack together the four observations of IGR J16138-4848, to obtain high-accuracy estimates of the average spectral parameters of this object; and then (2) proceed to the time-evolving analysis, of the three spectral parameters: (a) Gamma (the slope of the intrinsic continuum), (b) W(FeK), the equivalent width of the 6.4 keV Iron emission line, and (c) R, the relative amount of Compton reflection. Through this time-resolved spectroscopic analysis we hope to constrain (a) the physical state of the accreting matter and its relation with the X-ray output, and (b) the evolution of the accretion flow geometry, distribution and covering factor.

Mushotzky, Richard↗
DOE OSTI2025

The Simons Observatory: science goals and forecasts for the enhanced Large Aperture Telescope

We describe updated scientific goals for the wide-field, millimeter-wave survey that will be produced by the Simons Observatory (SO). Significant upgrades to the 6-meter SO Large Aperture Telescope (LAT) are expected to be complete by 2028, and will include a doubled mapping speed with 30,000 new detectors and an automated data reduction pipeline. In addition, a new photovoltaic array will supply most of the observatory's power. The LAT survey will cover about 60% of the sky at a regular observing cadence, with five times the angular resolution and ten times the map depth of the Planck satellite. The science goals are to: (1) determine the physical conditions in the early universe and constrain the existence of new light particles; (2) measure the integrated distribution of mass, electron pressure, and electron momentum in the late-time universe, and, in combination with optical surveys, determine the neutrino mass and the effects of dark energy via tomographic measurements of the growth of structure at redshifts z ≲ 3; (3) measure the distribution of electron density and pressure around galaxy groups and clusters, and calibrate the effects of energy input from galaxy formation on the surrounding environment; (4) produce a sample of more than 30,000 galaxy clusters, and more than 100,000 extragalactic millimeter sources, including regularly sampled AGN light-curves, to study these sources and their emission physics; (5) measure the polarized emission from magnetically aligned dust grains in our Galaxy, to study the properties of dust and the role of magnetic fields in star formation; (6) constrain asteroid regoliths, search for Trans-Neptunian Objects, and either detect or eliminate large portions of the phase space in the search for Planet 9; and (7) provide a powerful new window into the transient universe on time scales of minutes to years, concurrent with observations from the Vera C. Rubin Observatory of overlapping sky.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2012

A Peculiar Class of Debris Disks from Herschel/DUNES: A Steep Fall Off in the Far Infrared

Context. The existence of debris disks around old main sequence stars is usually explained by continuous replenishment of small dust grains through collisions from a reservoir of larger objects. Aims. We present photometric data of debris disks around HIP 103389 (HD199260), HIP 100350 (HN Peg, HD206860), and HIP 114948 (HD 219482), obtained in the context of our Herschel Open TIme Key Program DUNES (DUst around NEarby Stars). Methods. We used Herschel/PACS to detect the thermal emission of the three debris disks with a 30 sigma sensitivity of a few mJy at l00 micron and 160 micron. In addition, we obtained Herschel/PACS photometric data at 70 micron for HIP 103389. These observations are complemented by a large variety of optical to far-infrared photometric data. Two different approaches are applied to reduce the Herschel data to investigate the impact of data reduction on the photometry. We fit analytical models to the available spectral energy distribution (SED) data using the fitting method of simulated therma1 annealing as well as a classical grid search method. Results. The SEDs of the three disks potentially exhibit an unusually steep decrease at wavelengths >= 70 micron. We investigate the significance of the peculiar shape of these SEDs and the impact on models of the disks provided it is real. Using grain compositions that have been applied successfully for modeling of many other debris disks, our modeling reveals that such a steep decrease of the SEDs in the long wavelength regime is inconsistent with a power-law exponent of the grain size distribution -3.5 expected from a standard equilibrium collisional cascade. In contrast, a steep grain size distribution or, alternatively an upper grain size in the range of few tens of micrometers are implied. This suggests that a very distinct range of grain sizes would dominate the thermal. emission of such disks. However, we demonstrate that the understanding of the data of faint sources obtained with Herschel is still incomplete and that the significance of our results depends on the version of the data reduction pipeline used. Conclusions. A new mechanism to produce the dust in the presented debris disks, deviations from the conditions required for a standard equilibrium collisional cascade (grain size exponent of -3.5), and/or significantly different dust properties would be necessary to explain the potentially steep SED shape of the three debris disks presented.

Ertel, S.↗
NASA NTRS2017

Fireballs in the Sky: An Augmented Reality Citizen Science Program

Fireballs in the Sky is an innovative Australian citizen science program that connects the public with the research of the Desert Fireball Network (DFN). This research aims to understand the early workings of the solar system, and Fireballs in the Sky invites people around the world to learn about this science, contributing fireball sightings via a user-friendly augmented reality mobile app. Tens of thousands of people have downloaded the app world-wide and participated in the science of meteoritics. The Fireballs in the Sky app allows users to get involved with the Desert Fireball Network research, supplementing DFN observations and providing enhanced coverage by reporting their own meteor sightings to DFN scientists. Fireballs in the Sky reports are used to track the trajectories of meteors - from their orbit in space to where they might have landed on Earth. Led by Phil Bland at Curtin University in Australia, the Desert Fireball Network (DFN) uses automated observatories across Australia to triangulate trajectories of meteorites entering the atmosphere, determine pre-entry orbits, and pinpoint their fall positions. Each observatory is an autonomous intelligent imaging system, taking 1000 by 36 megapixel all-sky images throughout the night, using neural network algorithms to recognize events. They are capable of operating for 12 months in a harsh environment, and store all imagery collected. We developed a completely automated software pipeline for data reduction, and built a supercomputer database for storage, allowing us to process our entire archive. The DFN currently stands at 50 stations distributed across the Australian continent, covering an area of 2.5 million square kilometers. Working with DFN's partners at NASA's Solar System Exploration Research Virtual Institute, the team is expanding the network beyond Australia to locations around the world. Fireballs in the Sky allows a growing public base to learn about and participate in this exciting research.

Day, Brian↗
NASA NTRS2021

Mass Inferencing Model Creation And Deployment To Lunar Excavation Robot, RASSOR

NASA’s Regolith Advanced Surface Systems Operations Robot (RASSOR) Excavator is a teleoperated mobile robotic platform with a unique space regolith excavation capability. This research project developed functionality for inferencing regolith mass ingested during RASSOR operation, enhancing RASSOR’s ability to successfully complete ISRU missions. Radio wave propagation time to the Moon and back is ~2.56 seconds. Though teleoperation is possible with this delay, autonomous capability that enables RASSOR to plan and execute excavation missions intelligently and efficiently is preferred. To teleoperate or run autonomously, it is crucial for the quantity of regolith mass ingested by RASSOR to be available as a system state for efficient operation (e.g. knowledge of whether drums are full informs the task of highest priority, whether it be continuing to dig, or returning to a processing plant to offload regolith). A configurable data reduction and analysis pipeline was created to allow for straightforward incorporation of new data, such as that from lunar excavation, to improve model performance in new environments. Four distinct modeling approaches were employed in developing a mass inferencing approach that could work on RASSOR. All four models take in system states and output a mass prediction for each set of the robot’s bucket drums. Initial results from deployment to RASSOR and testing in a simulated lunar environment show that the models have <10% mean error during robot operation. Future work includes refinement of a model that estimates regolith mass in real-time during excavation as well as further testing of the developed models on the hardware.

ROS↗
NASA NTRS2012

The Atacama Cosmology Telescope: Data Characterization and Map Making

We present a description of the data reduction and mapmaking pipeline used for the 2008 observing season of the Atacama Cosmology Telescope (ACT). The data presented here at 148 GHz represent 12% or the 90 TB collected by ACT from 2007 to 2010. In 2008 we observed for 136 days, producing a total of 142h of data (11 TB for the 148 GHz band only), with a daily average of 10.5 h of observation. From these, 108.5 h were devoted to 850 sq deg stripe (11.2 h by 9 deg.1) centered on a declination of -52 deg.7, while 175 h were devoted to a 280 square deg stripe (4.5 h by 4 deg.8) centered at the celestial equator. We discuss sources of statistical and systematic noise, calibration, telescope pointing and data selection. Out of 1260 survey hours and 1024 detectors per array, 816 h and 593 effective detectors remain after data selection for this frequency band, yielding a 38 % survey efficiency. The total sensitivity in 2008, determined from the noise level between 5 Hz and 20 Hz in the time-ordered data stream (TOD), is 32 muK square root of s in CMB units. Atmospheric brightness fluctuations constitute the main contaminant in the data and dominate the detector and noise covariance at low frequencies in the TOD. The maps were made by solving the lease squares problem using the Preconditioned Conjugate Gradient method, incorporating the details of the detector and noise correlations. Cross-correlation with WMAP sky maps as well as analysis from simulations reveal the our maps are unbiased at l > 300. This paper accompanies the public release of the 148 GHz southern stripe maps from 2008. The techniques described here will be applied to future maps and data releases.

Duenner, Rolando↗
NASA NTRS2003

ISO Key Project: Exploring the Full Range of Quasar/Agn Properties

While most of the work on this program has been completed, as previously reported, the portion of the program dealing with the subtopic of ISO LWS data analysis and reduction for the LWS Extragalactic Science Team and its leader, Dr. Howard Smith, is still active. This program in fact continues to generate results, and newly available computer modeling has extended the value of the datasets. As a result the team requests a one-year no-cost extension to this program, through 31 December 2004. The essence of the proposal is to perform ISO spectroscopic studies, including data analysis and modeling, of star-formation regions using an ensemble of archival space-based data from the Infrared Space Observatory's Long Wavelength Spectrometer and Short Wavelength Spectrometer, but including as well some other spectroscopic databases. Four kinds of regions are considered in the studies: (1) disks around more evolved objects; (2) young, low or high mass pre-main sequence stars in star-formation regions; (3) star formation in external, bright IR galaxies; and (4) the galactic center. One prime focus of the program is the OH lines in the far infrared. The program has the following goals: 1) Refine the data analysis of ISO observations to obtain deeper and better SNR results on selected sources. The ISO data itself underwent 'pipeline 10' reductions in early 2001, and additional 'hands-on data reduction packages' were supplied by the ISO teams in 2001. The Fabry-Perot database is particularly sensitive to noise and slight calibration errors; 2) Model the atomic and molecular line shapes, in particular the OH lines, using revised Monte-Carlo techniques developed by the SWAS team at the Center for Astrophysics; 3) Attend scientific meetings and workshops; 4) Perform E&PO activities related to infrared astrophysics and/or spectroscopy.

Wilkes, Belinda↗
NASA NTRS2002

ISO Key Project: Exploring The Full Range of Quasar/AGN Properties

While most of the work on this program has been completed, as previously reported, the portion of the program dealing with the sub topic of ISO LWS data analysis and reduction for the LWS Extragalactic Science Team and its leader, Dr. Howard Smith, is still active. This program in fact continues to generate results, and newly available computer modeling has extended the value of the datasets, As a result the team has requested and been granted an obtained a no-cost extension to this program, through December 31, 2003. The essence of the proposal is to perform ISO spectroscopic studies, including data analysis and modeling, of star formation regions using an ensemble of archival space-based data from the Infrared Space Observatory's Long Wavelength Spectrometer and Short Wavelength Spectrometer, but including as well some other spectroscopic data bases. Four kinds of regions are considered in the studies: (1) disks around more evolved objects; (2) young, low or high mass pre-main sequence stars in star formation regions; (3) star formation in external, bright IR galaxies; and (4) the galactic center. One prime focus of the program is the OH lines in the far infrared. The program has the following goals: (1) refine the data analysis of ISO observations, to obtain deeper and better SNR results on selected sources. The ISO data itself underwent "pipeline 10" reductions in early 2001, and additional "hands-on data reduction packages" were supplied by the ISO teams in 2001. The Fabry-Perot database in particularly sensitive to noise can slight calibration errors. (2) model the atomic and molecular line shapes, in particular the OH lines, using revised Monte-Carlo techniques developed by the SWAS team at the Center for Astrophysics; (3) attend scientific meetings and workshops; (4) do E&PO activities related to infrared astrophysics and/or spectroscopy.

Wilkes, Belinda↗
NASA NTRS2005

Explore Full Range of QSO/AGN Properties

The goal of the proposal is to perform ISO spectroscopic studies, including data analysis and modeling, of star formation regions using an ensemble of archival space-based data from the Infrared Space Observatory s Long Wavelength Spectrometer and Short Wavelength Spectrometer, but including as well some other spectroscopic databases. Four kinds of regions are considered in the studies: (1) disks around more evolved objects; (2) young, low or high mass pre-main sequence stars in star formation regions; (3) star formation in external, bright IR galaxies; and (4) the galactic center. One prime focus of the program is the OH lines in the far infrared. The program had the following goals: 1) Refine the data analysis of IS0 observations to obtain deeper and better SNR results on selected sources. The IS0 data itself underwent "pipeline 10" reductions in early 2001, and additional "hands-on data reduction packages" were supplied by the IS0 teams in 2001. The Fabry-Perot database is particularly sensitive to noise and slight calibration errors. 2) Model the atomic and molecular line shapes, in particular the OH lines, using revised monte- carlo techniques developed by the SWAS team at the Center for Astrophysics; 3) Attend scientific meetings and workshops; 4) Do E&PO activities related to infrared astrophysics and/or spectroscopy.

Oliversen, Ronald↗