Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
NASA NTRS2016

The Kepler Science Data Processing Pipeline Source Code Road Map

We give an overview of the operational concepts and architecture of the Kepler Science Processing Pipeline. Designed, developed, operated, and maintained by the Kepler Science Operations Center (SOC) at NASA Ames Research Center, the Science Processing Pipeline is a central element of the Kepler Ground Data System. The SOC consists of an office at Ames Research Center, software development and operations departments, and a data center which hosts the computers required to perform data analysis. The SOC's charter is to analyze stellar photometric data from the Kepler spacecraft and report results to the Kepler Science Office for further analysis. We describe how this is accomplished via the Kepler Science Processing Pipeline, including, the software algorithms. We present the high-performance, parallel computing software modules of the pipeline that perform transit photometry, pixel-level calibration, systematic error correction, attitude determination, stellar target management, and instrument characterization.

Kepler pipeline software↗
NASA NTRS1988

Spacelab data processing facility (SLDPF) Quality Assurance (QA)/Data Accounting (DA) expert systems: Transition from prototypes to operational systems

The SLDPF is responsible for the capture, quality monitoring processing, accounting, and shipment of Spacelab and/or Attached Shuttle Payloads (ASP) telemetry data to various user facilities. Expert systems will aid in the performance of the quality assurance and data accounting functions of the two SLDPF functional elements: the Spacelab Input Processing System (SIPS) and the Spacelab Output Processing System (SOPS). Prototypes were developed for each as independent efforts. The SIPS Knowledge System Prototype (KSP) used the commercial shell OPS5+ on an IBM PC/AT; the SOPS Expert System Prototype used the expert system shell CLIPS implemented on a Macintosh personal computer. Both prototypes emulate the duties of the respective QA/DA analysts based upon analyst input and predetermined mission criteria parameters, and recommended instructions and decisions governing the reprocessing, release, or holding for further analysis of data. These prototypes demonstrated feasibility and high potential for operational systems. Increase in productivity, decrease of tedium, consistency, concise historial records, and a training tool for new analyses were the principal advantages. An operational configuration, taking advantage of the SLDPF network capabilities, is under development with the expert systems being installed on SUN workstations. This new configuration in conjunction with the potential of the expert systems will enhance the efficiency, in both time and quality, of the SLDPF's release of Spacelab/AST data products.

Basile, Lisa↗
NASA NTRS1988

Spacelab data processing facility (SLDPF) quality assurance (QA)/data accounting (DA) expert systems - Transition from prototypes to operational systems

The SLDPF is responsible for the capture, quality monitoring processing, accounting, and shipment of Spacelab and/or Attached Shuttle Payloads (ASP) telemetry data to various user facilities. Expert systems will aid in the performance of the quality assurance and data accounting functions of the two SLDPF functional elements: the Spacelab Input Processing System (SIPS) and the Spacelab Output Processing System (SOPS). Prototypes were developed for each as independent efforts. The SIPS Knowledge System Prototype (KSP) used the commercial shell OPS5+ on an IBM PC/AT; the SOPS Expert System Prototype used the expert system shell CLIPS implemented on a Macintosh personal computer. Both prototypes emulate the duties of the respective QA/DA analysts based upon analyst input and predetermined mission criteria parameters, and recommended instructions and decisions governing the reprocessing, release, or holding for further analysis of data. These prototypes demonstrated feasibility and high potential for operational systems. Increase in productivity, decrease of tedium, consistency, concise historical records, and a training tool for new analyses were the principal advantages. An operational configuration, taking advantage of the SLDPF network capabilities, is under development with the expert systems being installed on SUN workstations. This new configuration in conjunction with the potential of the expert systems will enhance the efficiency, in both time and quality, of the SLDPF's release of Spacelab/AST data products.

Basile, Lisa↗
DOE OSTI2024

Global Geo-processed Data of Aquifer Properties by 0.5° Grid, Country and Water Basins

This repository of global hydrogeologic datasets contains aquifer properties on 0.5° scale, including depth to groundwater (Fan et al., 2013), aquifer thickness (de Graaf et al., 2015), WHYMap aquifer classes (Richts et al., 2011), porosity and permeability (Gleeson et al., 2014), digitized and geo-processed from their respective sources. Globally gridded aquifer properties could be used independently to estimate global groundwater availability or used as critical inputs to the superwell model to simulate groundwater extraction and provide estimates of pumped volumes and unit costs under user-specific scenarios. Key resources related to this data are: Niazi, H., Ferencz, S., Graham, N., Yoon, J., Wild, T., Hejazi, M., Watson, D., & Vernon, C. (2024; In-prep). Long-term Hydro-economic Assessment Tool for Evaluating Global Groundwater Cost and Supply: Superwell v1. Geoscientific Model Development. superwell model repository which uses this data to simulate groundwater extraction and provides estimates of the global extractable volumes and unit-costs ($/km3) of accessible groundwater production under user-specified extraction scenarios. Repository Overview Main output: aquifer_properties.csv contains all processed outputs, including aquifer properties like porosity, permeability, aquifer thickness, and depth to groundwater. shapefiles.zip: contains all digitized GIS databases and shapefile for all aquifer properties prep_inputs.R: R script that processes the shapefiles to produce the aquifer_properties file plot_inputs.R: R script for plotting the maps and conducting preliminary analysis on the available groundwater volume basin_to_country_mapping.csv, basin_country_region_mapping.csv and continent_county_mapping.csv provide the mapping between continents, 32 energy-economic macro regions, countries, and water basins for post-processing aquifer_properties.csv Maps: Each map visualizes the spatial distribution of one of the aquifer properties across the globe map_in_Porosity.png map_in_Permeability.png map_in_Aquifer_thickness.png map_in_Depth_to_water.png map_in_Grid_area_km.png map_in_WHYClass.png Sample inputs sample_inputs.py: this script samples inputs from the aquifer_properties dataset, ensuring the sampled and original inputs maintain the same distributions sampled_data_100.csv contains 100 sampled data points and sampled_data_100.png compares their distributions Dataset Overview The main outputs are consolidated in a comprehensive aquifer_properties.csv file and include the following fields: GridCellID: Unique identifier for each (roughly 0.5°) grid cell Continent: Continent name Country: Country name GCAM_basin_ID: Identifier for GCAM hydrologic basin Basin_long_name: Full name of the basin WHYClass: Hydrogeologic classification based on WHYMap aquifer classes (Richts et al., 2011) Porosity: Soil porosity (%) (Gleeson et al., 2014) Permeability: Soil permeability (in square meters; Gleeson et al., 2014) Aquifer_thickness: Thickness of the aquifer (in meters; de Graaf et al., 2015) Depth_to_water: Depth to groundwater (in meters; Fan et al., 2013) Grid_area: Area of the grid cell (in square meters) Key References The datasets are digitized versions of global hydrogeologic properties from the following key literature sources: Depth to Groundwater: Fan, Y., Li, H., & Miguez-Macho, G. (2013). Global Patterns of Groundwater Table Depth. Science, 339(6122), 940-943. https://doi.org/10.1126/science.1229881 Aquifer Thickness: de Graaf, I. E. M., Sutanudjaja, E. H., van Beek, L. P. H., & Bierkens, M. F. P. (2015). A high-resolution global-scale groundwater model. Hydrol. Earth Syst. Sci., 19(2), 823-837. https://doi.org/10.5194/hess-19-823-2015 Porosity and Permeability: Gleeson, T., Moosdorf, N., Hartmann, J., & van Beek, L. P. H. (2014). A glimpse beneath earth's surface: GLobal HYdrogeology MaPS (GLHYMPS) of permeability and porosity. Geophysical Research Letters, 41(11), 3891-3898. https://doi.org/10.1002/2014GL059856 Aquifer classes: Richts, A., Struckmeier, W. F., & Zaepke, M. (2011). WHYMAP and the Groundwater Resources Map of the World 1:25,000,000. In J. A. A. Jones (Ed.), Sustaining Groundwater Resources: A Critical Element in the Global Water Crisis (pp. 159-173). Springer Netherlands. https://doi.org/10.1007/978-90-481-3426-7_10 Cite as Niazi, H., Watson, D., Hejazi, M., Yonkofski, C., Ferencz, S., Vernon, C., Graham, N., Wild, T., & Yoon, J. (2024). Global Geo-processed Data of Aquifer Properties by 0.5° Grid, Country and Water Basins. MSD-LIVE Data Repository. https://doi.org/10.57931/2307831 Contact Reach out to Hassan Niazi or open an issue in superwell repository for questions or suggestions.

aquifer thickness↗
NASA NTRS2013

Using NASA's Reference Architecture: Comparing Polar and Geostationary Data Processing Systems

The JPSS and GOES-R programs are housed at NASA GSFC and jointly implemented by NASA and NOAA to NOAA requirements. NASA's role in the JPSS Ground System is to develop and deploy the system according to NOAA requirements. NASA's role in the GOES-R ground segment is to provide Systems Engineering expertise and oversight for NOAA's development and deployment of the system. NASA's Earth Science Data Systems Reference Architecture is a document developed by NASA's Earth Science Data Systems Standards Process Group that describes a NASA Earth Observing Mission Ground system as a generic abstraction. The authors work within the respective ground segment projects and are also separately contributors to the Reference Architecture document. Opinions expressed are the author's only and are not NOAA, NASA or the Ground Projects' official positions.

Ullman, Richard↗
NASA NTRS2009

Data Processing for Atmospheric Phase Interferometers

This paper presents a detailed discussion of calibration procedures used to analyze data recorded from a two-element atmospheric phase interferometer (API) deployed at Goldstone, California. In addition, we describe the data products derived from those measurements that can be used for site intercomparison and atmospheric modeling. Simulated data is used to demonstrate the effectiveness of the proposed algorithm and as a means for validating our procedure. A study of the effect of block size filtering is presented to justify our process for isolating atmospheric fluctuation phenomena from other system-induced effects (e.g., satellite motion, thermal drift). A simulated 24 hr interferometer phase data time series is analyzed to illustrate the step-by-step calibration procedure and desired data products.

Acosta, Roberto J.↗
NASA NTRS1985

Atmospheric phenomena data processing and display

Several related efforts which deal with the analysis and display of data pertaining to various atmospheric phenomena are reported. Spectrometer data analysis, with refinements to the preexisting programs, is discussed. A discussion of a Monte Carlo simulation of a photon transfer problem is presented. Raw data gathered in the field by a high-flying U2 aircraft is usually pulse code modulated (PCM). Techniques to reduce the data into a computer-amenable format are described. Several utility programs and other work are outlined.

Tatom, F. B.↗
NASA NTRS1976

Architecture and data processing alternatives for Tse computer. Volume 1: Tse logic design concepts and the development of image processing machine architectures

Digital computer components which perform two dimensional array logic operations (Tse logic) on binary data arrays are described. The properties of Golay transforms which make them useful in image processing are reviewed, and several architectures for Golay transform processors are presented with emphasis on the skeletonizing algorithm. Conventional logic control units developed for the Golay transform processors are described. One is a unique microprogrammable control unit that uses a microprocessor to control the Tse computer. The remaining control units are based on programmable logic arrays. Performance criteria are established and utilized to compare the various Golay transform machines developed. A critique of Tse logic is presented, and recommendations for additional research are included.

Rickard, D. A.↗
DOE OSTI2012

The Dark Energy Survey Data Processing and Calibration System

The Dark Energy Survey (DES) is a 5000 deg2 grizY survey reaching characteristic photometric depths of 24th magnitude (10 sigma) and enabling accurate photometry and morphology of objects ten times fainter than in SDSS. Preparations for DES have included building a dedicated 3 deg2 CCD camera (DECam), upgrading the existing CTIO Blanco 4m telescope and developing a new high performance computing (HPC) enabled data management system (DESDM). The DESDM system will be used for processing, calibrating and serving the DES data. The total data volumes are high (~2PB), and so considerable effort has gone into designing an automated processing and quality control system. Special purpose image detrending and photometric calibration codes have been developed to meet the data quality requirements, while survey astrometric calibration, coaddition and cataloging rely on new extensions of the AstrOmatic codes which now include tools for PSF modeling, PSF homogenization, PSF corrected model fitting cataloging and joint model fitting across multiple input images. The DESDM system has been deployed on dedicated development clusters and HPC systems in the US and Germany. An extensive program of testing with small rapid turn-around and larger campaign simulated datasets has been carried out. The system has also been tested on large real datasets, including Blanco Cosmology Survey data from the Mosaic2 camera. In Fall 2012 the DESDM system will be used for DECam commissioning, and, thereafter, the system will go into full science operations.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Global Geo-processed Data of Aquifer Properties by 0.5° Grid, Country and Water Basins

This repository of global hydrogeologic datasets contains aquifer properties on 0.5° scale, including depth to groundwater (Fan et al., 2013), aquifer thickness (de Graaf et al., 2015), WHYMap aquifer classes (Richts et al., 2011), recharge (Döll and Fiedler, 2008; Gleeson et al., 2016), lakes (Messager et al., 2016), porosity and permeability (Gleeson et al., 2014), digitized and geo-processed from their respective sources. Globally gridded aquifer properties could be used independently to estimate global groundwater availability or used as critical inputs to the superwell model to simulate groundwater extraction and provide estimates of pumped volumes and unit costs under user-specific scenarios. Key resources related to this data are: Niazi, H., Ferencz, S. B., Graham, N. T., Yoon, J., Wild, T. B., Hejazi, M., Watson, D. J., & Vernon, C. R. (2025). Long-term hydro-economic analysis tool for evaluating global groundwater cost and supply: Superwell v1.1. Geoscientific Model Development, 18(5), 1737-1767. https://doi.org/10.5194/gmd-18-1737-2025 superwell model repository which uses this data to simulate groundwater extraction and provides estimates of the global extractable volumes and unit-costs ($/km3) of accessible groundwater production under user-specified extraction scenarios. Repository Overview Main output: aquifer_properties_rec.csv contains all processed outputs, including aquifer properties like porosity, permeability, recharge, lake areas, aquifer thickness, and depth to groundwater. shapefiles.zip: contains all digitized GIS databases and shapefile for all aquifer properties prep_inputs.R and prep_inputs_recharge_lakes.R: R scripts that process the shapefiles to produce the aquifer_properties_rec.csv file plot_inputs.R: R script for plotting the maps and conducting preliminary analysis on the available groundwater volume basin_to_country_mapping.csv, basin_country_region_mapping.csv and continent_county_mapping.csv provide the mapping between continents, 32 energy-economic macro regions, countries, and water basins for post-processing aquifer_properties_rec.csv Maps: Each map visualizes the spatial distribution of one of the aquifer properties across the globe map_in_Porosity.png map_in_Permeability.png map_in_Aquifer_thickness.png map_in_Depth_to_water.png map_in_Recharge.png map_in_Grid_area_km.png map_in_Lake_area_km.png map_in_WHYClass.png Sample inputs sample_inputs.py: this script samples inputs from the aquifer_properties_rec dataset, ensuring the sampled and original inputs maintain the same distributions sampled_data_100.csv contains 100 sampled data points and sampled_data_100.png compares their distributions Dataset Overview The main outputs are consolidated in a comprehensive aquifer_properties_rec.csv file and include the following fields: GridCellID: Unique identifier for each (roughly 0.5°) grid cell Continent: Continent name Country: Country name GCAM_basin_ID: Identifier for GCAM hydrologic basin Basin_long_name: Full name of the basin WHYClass: Hydrogeologic classification based on WHYMap aquifer classes (Richts et al., 2011) Porosity: Soil porosity (%) (Gleeson et al., 2014) Permeability: Soil permeability (in square meters; Gleeson et al., 2014) Aquifer_thickness: Thickness of the aquifer (in meters; de Graaf et al., 2015) Depth_to_water: Depth to groundwater (in meters; Fan et al., 2013) Recharge: long-term annual averaged recharge rates (in m/yr; Döll and Fiedler, 2008; Gleeson et al., 2016) Grid_area: Area of the grid cell (in square meters) Lakes_area: Area of inland lakes (in square meters; Messager et al., 2016) Key References The datasets are digitized versions of global hydrogeologic properties from the following key literature sources: Depth to Groundwater: Fan, Y., Li, H., & Miguez-Macho, G. (2013). Global Patterns of Groundwater Table Depth. Science, 339(6122), 940-943. https://doi.org/10.1126/science.1229881 Aquifer Thickness: de Graaf, I. E. M., Sutanudjaja, E. H., van Beek, L. P. H., & Bierkens, M. F. P. (2015). A high-resolution global-scale groundwater model. Hydrol. Earth Syst. Sci., 19(2), 823-837. https://doi.org/10.5194/hess-19-823-2015 Porosity and Permeability: Gleeson, T., Moosdorf, N., Hartmann, J., & van Beek, L. P. H. (2014). A glimpse beneath earth's surface: GLobal HYdrogeology MaPS (GLHYMPS) of permeability and porosity. Geophysical Research Letters, 41(11), 3891-3898. https://doi.org/10.1002/2014GL059856 Aquifer classes: Richts, A., Struckmeier, W. F., & Zaepke, M. (2011). WHYMAP and the Groundwater Resources Map of the World 1:25,000,000. In J. A. A. Jones (Ed.), Sustaining Groundwater Resources: A Critical Element in the Global Water Crisis (pp. 159-173). Springer Netherlands. https://doi.org/10.1007/978-90-481-3426-7_10 Recharge: Döll, P., & Fiedler, K. (2008). Global-scale modeling of groundwater recharge. Hydrol. Earth Syst. Sci., 12(3), 863-885. https://doi.org/10.5194/hess-12-863-2008; Gleeson, T., Befus, K. M., Jasechko, S., Luijendijk, E., & Cardenas, M. B. (2016). The global volume and distribution of modern groundwater. Nature Geoscience, 9(2), 161-167. https://doi.org/10.1038/ngeo2590 Inland Lakes: Messager, M. L., Lehner, B., Grill, G., Nedeva, I., & Schmitt, O. (2016). Estimating the volume and age of water stored in global lakes using a geo-statistical approach. Nature Communications, 7(1), 13603. https://doi.org/10.1038/ncomms13603 Cite as Niazi, H., Watson, D., Hejazi, M., Yonkofski, C., Ferencz, S., Vernon, C., Graham, N., Wild, T., & Yoon, J. (2024). Global Geo-processed Data of Aquifer Properties by 0.5° Grid, Country and Water Basins. MultiSector Dynamics-Living, Intuitive, Value-adding, Environment. https://doi.org/10.57931/2484226 Contact Reach out to Hassan Niazi or open an issue in the superwell repository for questions or suggestions.

aquifer thickness↗
NASA NTRS1975

Data processing large quantities of multispectral information

Method is combination of digital and optical techniques. Multispectral data is coded into binary matrix format and then encoded onto photographic film. Film is holographically correlated with spectral signature to generate single-class classification map. Number of maps are optically superimposed to produce full-color, multiclass classification map.

Haskell, R. E.↗
NASA NTRS1974

Multispectral scanner data processing over Sam Houston National Forest

The Edit 9 forest scene, a computer processing technique, and its capability to map timber types in the Sam Houston National Forest, are evaluated. Special efforts were made to evaluate existing computer processing techniques in mapping timber types using ERTS-1 and aircraft data, and to provide an opportunity to open up new research and development areas in forestry data.

Reeves, C. A.↗
NASA NTRS1976

Radar data processing and analysis

Digitized four-channel radar images corresponding to particular areas from the Phoenix and Huntington test sites were generated in conjunction with prior experiments performed to collect X- and L-band synthetic aperture radar imagery of these two areas. The methods for generating this imagery are documented. A secondary objective was the investigation of digital processing techniques for extraction of information from the multiband radar image data. Following the digitization, the remaining resources permitted a preliminary machine analysis to be performed on portions of the radar image data. The results, although necessarily limited, are reported.

Ausherman, D.↗
DOE OSTI2018

Improving the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems with Eley

Here, burst buffer is an effective solution for reducing the data transfer time and the I/O interference in HPC systems. Extending Burst Buffers (BBs) to handle Big Data applications is challenging because BBs must account for the large data inputs of Big Data applications and the Quality-of-Service (QoS) of HPC applications which are considered as first-class citizens in HPC systems. Existing BBs focus on only intermediate data of Big Data applications and incur a high performance degradation of both Big Data and HPC applications. We present Eley , a burst buffer solution that helps to accelerate the performance of Big Data applications while guaranteeing the QoS of HPC applications. To achieve this goal, Eley embraces interference-aware prefetching technique that makes reading data input faster while introducing low interference for HPC applications. Evaluations using a wide range of Big Data and HPC applications demonstrate that Eley improves the performance of Big Data applications by up to 30% compared to existing BBs while maintaining the QoS of HPC applications.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1981

Digital ultrasonics signal processing: Flaw data post processing use and description

A modular system composed of two sets of tasks which interprets the flaw data and allows compensation of the data due to transducer characteristics is described. The hardware configuration consists of two main units. A DEC LSI-11 processor running under the RT-11 sngle job, version 2C-02 operating system, controls the scanner hardware and the ultrasonic unit. A DEC PDP-11/45 processor also running under the RT-11, version 2C-02, operating system, stores, processes and displays the flaw data. The software developed the Ultrasonics Evaluation System, is divided into two catagories; transducer characterization and flaw classification. Each category is divided further into two functional tasks: a data acquisition and a postprocessor ask. The flaw characterization collects data, compresses its, and writes it to a disk file. The data is then processed by the flaw classification postprocessing task. The use and operation of a flaw data postprocessor is described.

Buel, V. E.↗
DOE OSTI2020

K-means-driven Gaussian Process data collection for angle-resolved photoemission spectroscopy

Abstract We propose the combination of k-means clustering with Gaussian Process (GP) regression in the analysis and exploration of 4D angle-resolved photoemission spectroscopy (ARPES) data. Using cluster labels as the driving metric on which the GP is trained, this method allows us to reconstruct the experimental phase diagram from as low as 12% of the original dataset size. In addition to the phase diagram, the GP is able to reconstruct spectra in energy-momentum space from this minimal set of data points. These findings suggest that this methodology can be used to improve the efficiency of ARPES data collection strategies for unknown samples. The practical feasibility of implementing this technology at a synchrotron beamline and the overall efficiency implications of this method are discussed with a view on enabling the collection of more samples or rapid identification of regions of interest.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗