Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,423 records · Page 79

Automated Precision Orbit Determination for TOPEX/Poseidon with GPS

A highly automated GPS data processing system for the orbit determination of TOPEX/Poseidon is described. The orbit is recovered to an estimated accuracy of better than 4 cm in altitude, 6 cm crosstrack, and 11 cm down track. The RMS postfit residuals on the ionospherically calibrated carrier phase observable are less than 5 mm. The RMS difference over a 4.5-hour overlap period between two 30-hour data arcs is 1 cm in altitude, 5 cm cross track, and 4 cm down track. These results can be obtained within two days of onboard GPS data collection. Most of the data processing for a 30-hour arc of GPS data can be performed on a single workstation in less than 6 hours of CPU time. The estimation scenarios are explained, the automated data processing steps are described, and means to assess solution quality are discussed.

automation GPS data processing orbit determination↗

Study to determine cloud motion from meteorological satellite data

Processing techniques were tested for deducing cloud motion vectors from overlapped portions of pairs of pictures made from meteorological satellites. This was accomplished by programming and testing techniques for estimating pattern motion by means of cross correlation analysis with emphasis placed upon identifying and reducing errors resulting from various factors. Techniques were then selected and incorporated into a cloud motion determination program which included a routine which would select and prepare sample array pairs from the preprocessed test data. The program was then subjected to limited testing with data samples selected from the Nimbus 4 THIR data provided by the 11.5 micron channel.

Clark, B. B.↗

Performance of the Lidar Design and Data Algorithms for the GLAS Global Cloud and Aerosol Measurements

The Geoscience Laser Altimeter System (GLAS) launched in early 2003 is the first polar orbiting satellite lidar. The instrument design includes high performance observations of the distribution and optical scattering cross sections of atmospheric clouds and aerosol. The backscatter lidar operates at two wavelengths, 532 and 1064 nm. For the atmospheric cloud and aerosol measurements, the 532 nm channel was designed for ultra high efficiency with solid state photon counting detectors and etalon filtering. Data processing algorithms were developed to calibrate and normalize the signals and produce global scale data products of the height distribution of cloud and aerosol layers and their optical depths and particulate scattering cross sections up to the limit of optical attenuation. The paper will concentrate on the effectiveness and limitations of the lidar channel design and data product algorithms. Both atmospheric receiver channels meet and exceed their design goals. Geiger Mode Avalanche Photodiode modules are used for the 532 nm signal. The operational experience is that some signal artifacts and non-linearity require correction in data processing. As with all photon counting detectors, a pulse-pile-up calibration is an important aspect of the measurement. Additional signal corrections were found to be necessary relating to correction of a saturation signal-run-on effect and also for daytime data, a small range dependent variation in the responsivity. It was possible to correct for these signal errors in data processing and achieve the requirement to accurately profile aerosol and cloud cross section down to 10-7 llm-sr. The analysis procedure employs a precise calibration against molecular scattering in the mid-stratosphere. The 1064 nm channel detection employs a high-speed analog APD for surface and atmospheric measurements where the detection sensitivity is limited by detector noise and is over an order of magnitude less than at 532 nm. A unique feature of the GLAS is a full acquisition of the surface return pulse, which has important application to the atmospheric transmission retrieval.

Spinhirne, James D.↗

DDS-Suite - A Dynamic Data Acquisition, Processing, and Analysis System for Wind Tunnel Testing

Wind Tunnels have optimized their steady-state data systems for acquisition and analysis and even implemented large dynamic-data acquisition systems, however development of near real-time processing and analysis tools for dynamic-data have lagged. DDS-Suite is a set of tools used to acquire, process, and analyze large amounts of dynamic data. Each phase of the testing process: acquisition, processing, and analysis are handled by separate components so that bottlenecks in one phase of the process do not affect the other, leading to a robust system. DDS-Suite is capable of acquiring 672 channels of dynamic data at rate of 275 MB / s. More than 300 channels of the system use 24-bit analog-to-digital cards and are capable of producing data with less than 0.01 of phase difference at 1 kHz. System architecture, design philosophy, and examples of use during NASA Constellation and Fundamental Aerodynamic tests are discussed.

Burnside, Jathan J.↗

ERBE and CERES broadband scanning radiometers

Broadband scanning radiometers have been used extensively on earth-orbiting satellites to measure the Earth's outgoing radiation. The resulting estimates of longwave and shortwave fluxes have played an important role in helping to understand the Earth's radiant energy balance or budget. The Clouds and the Earth Radiant Energy System (CERES) experiment is expected to include instruments with three broadband scanning radiometers. The design of the CERES instrument will draw heavily from the flight-proven Earth Radiation Budget Experiment (ERBE) scanner instrument technology and will benefit from the several years of ERBE experience in mission operations and data processing. The discussion starts with a description of the scientific objectives of ERBE and CERES. The design and operational characteristics of the ERBE and CERES instrument are compared and the two ground-based data processing systems are compared. Finally, aspects of the CERES data processing which might be performed in near real-time aboard a spacecraft platform are discussed, and the types of algorithms and input data requirements for the onboard processing system are identified.

Weaver, William L.↗

A Multi-mission Event-Driven Component-Based System for Support of Flight Software Development, ATLO, and Operations first used by the Mars Science Laboratory (MSL) Project

This paper details an architectural description of the Mission Data Processing and Control System (MPCS), an event-driven, multi-mission ground data processing components providing uplink, downlink, and data management capabilities which will support the Mars Science Laboratory (MSL) project as its first target mission. MPCS is developed based on a set of small reusable components, implemented in Java, each designed with a specific function and well-defined interfaces. An industry standard messaging bus is used to transfer information among system components. Components generate standard messages which are used to capture system information, as well as triggers to support the event-driven architecture of the system. Event-driven systems are highly desirable for processing high-rate telemetry (science and engineering) data, and for supporting automation for many mission operations processes.

Mission Data Processing and Control System (MPCS)↗

Ground-based_transient_electromagnetic_data_and_resistivity_models_for_SubTER_BelleCreek_2017-2018

Ground-based transient electromagnetic (TEM) data were acquired in selected locations around Belle Creek, Montana to define the resistivity structure of the near surface. Data were acquired using the ABEM WalkTEM system (Guideline Geo Ab, Sundbyberg, Sweden). TEM data were processed and numerically inverted to derive one-dimensional resistivity structure using SPIA (Aarhus Geosoftware, Aarhus, Denmark). This release includes raw and processed TEM data and resistivity models for each data location using the following structure: Ground-based_transient_electromagnetic_data_and_resistivity_models_for_SubTER_BelleCreek_2017-2018.xml SBC_TEM_2019_SoundingLocationsModels.gdb.zip - file geodatabase with all models and plots attached. This is a mirror of the contents of the model directory. Ground-based_transient_electromagnetic_data_and_resistivity_models_for_SubTER_BelleCreek_2017-2018.zip Data SBC_USF - raw data USF_format_description.pdf - manual describing the USF file format SBC_SoundingID-USF_lookup.csv - table describing the USF file naming convention and channel mapping *.usf files - raw, unprocessed data files SBC_TEM - processed data AarhusInv_manual.pdf - manual describing the TEM file format *.tem files - processed data files Models SBC_Model_Plots - plots of the data and the models for each sounding location TEM_model_data_dictionary.csv - data dictionary describing tabulated models SBC_TEM_models.csv - tabulated models

geophysical surveying↗

STS-56, RSRM-031, 360L031 KSC processing configuration and data report

KSC Processing Configuration and Data Report is being provided as a historical document and as an enhancement to future RSRM manufacturing and processing operations. The following sections provide information on segment receipt, aft booster build-up, booster assembly, and closeout for STS-56, RSRM flight set 36OL031. Section 2.0 contains a summary of RSRM-031 processing. Section 3.0 discusses any significant problems or special issues that require special attention. Sections 4.0 through 6.0 contain narrative descriptions of all key events, including any related processing problems. Appendix A provides engineering specifications and changes. A list and matrix of all problem reports (PR's) pertinent to this flight set is provided in Appendix B. The matrix was provided by the Thiokol LSS Quality Engineering office. Copies of the PR's generated during the processing of RSRM-031 will be provided upon request. Appendix C contains the motor set status matrix, which provides milestone dates for the RSRM-031 flow. Section 7.0 provides recommendations, if any, for the improvement of flight hardware processing. Section 8.0 contains data sheets that provide flight hardware parts and consumables information installed during the booster build-up and stacking operations by location, lot/serial number, expiration and cure dates/times, and installation dates.

Source record↗

STS-51, RSRM-033, 360T033 KSC processing configuration and data report

KSC Processing Configuration and Data Report is being provided as a historical document and as an enhancement to future RSRM manufacturing and processing operations. The following sections provide information on segment receipt, aft booster build up, motor assembly, and closeout for STS-51, RSRM flight set 360T033. Section 2.0 contains a summary of RSRM-033 processing. Section 3.0 discusses any significant problems or special issues that require special attention. Sections 4.0 through 6.0 contain narrative descriptions of all key events, including any related processing problems. Appendix A provides engineering specifications and changes. A list and matrix of all problem reports (PR's) pertinent to this flight set is provided in Appendix B. The matrix was provided by the Thiokol LSS Quality Engineering office. Copies of the PR's generated during the processing of RSRM-033 will be provided upon request. Appendix C contains the motor set status matrix, which provides milestone dates for the RSRM-033 flow. Section 7.0 provides recommendations for the improvement of flight hardware processing. Section 8.0 contains data sheets that provide flight hardware parts and consumable information installed during the booster build-up and stacking operations by location, lot/serial number, expiration and cure dates/times, and installation dates.

Hillard, Robert C.↗

Space Shuttle data acquisition and processing

The techniques of minimizing the amount of Space Shuttle data processed by using real-time ground data and on board computer to prescreen the data are presented. With the advent of long duration space flight and the large amounts of data generated, it became necessary to limit the amount of data to be processed and also to present data in a manner that highlights the most important data. The Shuttle operational data system with output of 128 kilobits/sec, the developmental flight instrumentation system, and screening of data by the analysts, flight crew, or onboard computers is described, noting that by using these monitoring methods the mission evaluation team can rapidly determine time periods for which postflight data processing is required. Postflight processing systems, batch system output products, data compression techniques, the interactive computer system, and the special telemetry conversion system are discussed, and it is concluded that the combination of Shuttle on board and ground data processing systems enhances the ability to perform a functional evaluation of the Space Shuttle vehicle.

Foster, G. B., Jr.↗

Data and Scripts associated with a manuscript on ecosystem responses to wildfires in the Columbia River Basin

This data package is associated with the publication “Ecosystem leaf area, gross primary production, and evapotranspiration responses to wildfire in the Columbia River Basin” submitted to Biogeosciences (Shi et al., 2024; doi: 10.22541/au.171053013.30286044/v1). In this research, data products, leaf area index (LAI), gross primary production (GPP), and evapotranspiration (ET), from the Moderate Resolution Imaging Spectroradiometer (MODIS) are used to quantify the resistance and resilience of different ecosystem types in the Columbia River Basin (CRB). A machine learning algorithm, random forest (RF), was used to examine the impacts of precipitation, vapor pressure deficit (VPD), and burn severity from Monitoring Trends in Burn Severity (MTBS) on ecosystem resilience. The data package includes the processed MODIS data products, precipitation, VPD, and burn severity in 138 fire regions in CRB and the input files for RF model training. This data package includes six folders. The MODIS products are included in three MODIS_* folders with shell scripts for data clipping and *ncl files for data processing: (1) “/MODIS_LAI_CRB”; (2) “/MODIS_GPP_CRB”; and (3) “/MODIS_ET_CRB”. All the processed data for each fire event are NetCDF formatted. The MTBS burn severity data and the shell and *ncl scripts used for data processing are in the folder named (4) “MTBS_fire”. The ERA meteorological fields and the data processing scritps are in (5) “ERA_Var_CR”. All the scripts for figure development are in the format of *ncl and in the folder (6) “paper_scripts”. See the file ending in “flmd.csv” for a list of all files contained in this data package and descriptions for each. Tabular column headers and units are described in the data dictionary file ending in “dd.csv”.

54 ENVIRONMENTAL SCIENCES↗

Kepler Science Operations Center Architecture

We give an overview of the operational concepts and architecture of the Kepler Science Data Pipeline. Designed, developed, operated, and maintained by the Science Operations Center (SOC) at NASA Ames Research Center, the Kepler Science Data Pipeline is central element of the Kepler Ground Data System. The SOC charter is to analyze stellar photometric data from the Kepler spacecraft and report results to the Kepler Science Office for further analysis. We describe how this is accomplished via the Kepler Science Data Pipeline, including the hardware infrastructure, scientific algorithms, and operational procedures. The SOC consists of an office at Ames Research Center, software development and operations departments, and a data center that hosts the computers required to perform data analysis. We discuss the high-performance, parallel computing software modules of the Kepler Science Data Pipeline that perform transit photometry, pixel-level calibration, systematic error-correction, attitude determination, stellar target management, and instrument characterization. We explain how data processing environments are divided to support operational processing and test needs. We explain the operational timelines for data processing and the data constructs that flow into the Kepler Science Data Pipeline.

Middour, Christopher↗

Development of Predictive Models for Advanced Reactor Autonomous Control

Advanced reactor designs including microreactors and small modular reactors will contribute to the clean production of cheap energy, and autonomous control for advanced reactors is an appealing option for reducing cost. However, there is a lack of industry experience applying autonomous control for advanced nuclear reactors. To accelerate the development and industry acceptance of autonomous control software for nuclear reactors, we aim to demonstrate autonomous control of the Purdue University research reactor (PUR-1) using INL-developed model predictive control (MPC) methods. To prepare for this demonstration, data-driven predictive models based on process data collected from PUR-1 have been developed and integrated with MPC and used to control a physics-based model of PUR-1. A data-driven dynamics model and a gated recurrent unit (GRU) network were both trained on process data from PUR-1. The dynamics model was shown to effectively control the reactor model with MPC when provided reactivity as a control variable but failed to control the model through the control rod positions. The GRU network produced more accurate predictions than the dynamics model when evaluated on operational data, and future work will include the evaluation of the GRU network in the controller.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Bringing heterogeneity to the CMS software framework

The advent of computing resources with co-processors, for example Graphics Processing Units (GPU) or Field-Programmable Gate Arrays (FPGA), for use cases like the CMS High-Level Trigger (HLT) or data processing at leadership-class supercomputers imposes challenges for the current data processing frameworks. These challenges include developing a model for algorithms to offload their computations on the co-processors as well as keeping the traditional CPU busy doing other work. The CMS data processing framework, CMSSW, implements multithreading using the Intel Threading Building Blocks (TBB) library, that utilizes tasks as concurrent units of work. In this paper we will discuss a generic mechanism to interact effectively with non-CPU resources that has been implemented in CMSSW. In addition, configuring such a heterogeneous system is challenging. In CMSSW an application is configured with a configuration file written in the Python language. The algorithm types are part of the configuration. The challenge therefore is to unify the CPU and co-processor settings while allowing their implementations to be separate. We will explain how we solved these challenges while minimizing the necessary changes to the CMSSW framework. We will also discuss on a concrete example how algorithms would offload work to NVIDIA GPUs using directly the CUDA API.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

VA EDH Advanced Software Pipeline Framework Report: Enhancing Automation and Scalability

The VA Environmental Determinants of Health (EDH) Advanced Software Pipeline Framework is designed to enhance the efficiency, scalability, and security of geospatial data processing workflows. This framework integrates modern data orchestration and containerization technologies, including Prefect for workflow automation, Docker for containerization, and PostgreSQL/PostGIS for geospatial data storage and analysis. It ensures standardized, reproducible, and automated data processing, supporting VA objectives related to substance use risk assessment and recovery research. The pipeline addresses key scalability and performance challenges through horizontal and vertical scaling, high-performance computing (HPC) integration, parallel processing, task caching, and dynamic resource allocation. These optimizations improve throughput and reduce latency, allowing the system to efficiently manage large and complex datasets. Additionally, security and compliance measures—such as data encryption (SSL), Role-Based Access Control (RBAC), and adherence to GDPR and HIPAA standards—safeguard sensitive information throughout data transmission and storage. A key implementation of this framework includes the automation of shelter list geolocation workflows, ensuring that up-to-date data is readily available for VA decision-making. Lessons learned from this project include the transition from in-memory processing to incremental storage writes, improving resource management and reliability. Future enhancements aim to expand automation, integrate AI-driven anomaly detection, and incorporate high-performance computing resources. This framework provides a scalable, secure, and adaptable solution for managing geospatial datasets, reinforcing the VA’s ability to support clinical and strategic initiatives through data-driven decision-making.

97 MATHEMATICS AND COMPUTING↗

Shallow Marine High-Resolution 3D Seismics for Above-Zone CO2 Monitoring: A Case Study in Offshore Japan

Shallow marine high-resolution 3D (HR3D) seismic utilizes a novel seismic acquisition technology for imaging fine geological features not revealed by conventional 3D surveys. HR3D technology has been previously used for shallow subsurface studies, however is rarely implemented for monitoring injected fluids, and has not previously been used to monitor an active CO2 storage project. In this study, we for the first time demonstrate the acquisition, processing and initial interpretation of a HR3D seismic survey in Tomakomai, offshore Japan, where a large-scale CO2 storage project is being undertaken. As a seismic monitoring tool, the HR3D survey images the geologic overburden (< 1000 ms), aiming to detect potential CO2 leakage pathways above the injection reservoir. The HR3D acquisition system comprises a 210 in3 GI air gun and 4 streamers that are 25 m in length with 10 m inline separation. Each streamer has 8 channels with a 3.125 m group interval, yielding a very small final bin size of 3 x 3 m. The short offset and low-fold coverage of the HR3D data creates new challenges in seismic data processing, which demands a different data processing workflow tailored to the HR3D acquisition. We present a data processing workflow involving multiple software processing packages, which tackled high-resolution related static issues and significantly reduced various noise elements. Dip-steering and structural filtering were employed on the stacked volume to further enhance the lateral coherence and to remove random noise and suppress acquisition footprints. The final volume shows flat and coherent above-zone stratigraphy, suggesting a good seal quality above the CO2 reservoir. The successful implementation of HR3D acquisition and the customized processing workflow verify the feasibility of HR3D seismics for the above- zone monitoring of offshore CO2 injection and storage projects.

Feng, Ye Erickson↗

SHALLOW MARINE HIGH-RESOLUTION 3D SEISMICS FOR ABOVE- ZONE CO2MONITORING: A CASE STUDY IN OFFSHORE JAPAN

Shallow marine high-resolution 3D (HR3D) seismic utilizes a novel seismic acquisition technology for imaging fine geological features not revealed by conventional 3D surveys. HR3D technology has been previously used for shallow subsurface studies, however is rarely implemented for monitoring injected fluids, and has not previously been used to monitor an active CO2 storage project. In this study, we for the first time demonstrate the acquisition, processing and initial interpretation of a HR3D seismic survey in Tomakomai, offshore Japan, where a large-scale CO2 storage project is being undertaken. As a seismic monitoring tool, the HR3D survey images the geologic overburden (< 1000 ms), aiming to detect potential CO2 leakage pathways above the injection reservoir. The HR3D acquisition system comprises a 210 in3 GI air gun and 4 streamers that are 25 m in length with 10 m inline separation. Each streamer has 8 channels with a 3.125 m group interval, yielding a very small final bin size of 3 x 3 m. The short offset and low-fold coverage of the HR3D data creates new challenges in seismic data processing, which demands a different data processing workflow tailored to the HR3D acquisition. We present a data processing workflow involving multiple software processing packages, which tackled high-resolution related static issues and significantly reduced various noise elements. Dip-steering and structural filtering were employed on the stacked volume to further enhance the lateral coherence and to remove random noise and suppress acquisition footprints. The final volume shows flat and coherent above-zone stratigraphy, suggesting a good seal quality above the CO2 reservoir. The successful implementation of HR3D acquisition and the customized processing workflow verify the feasibility of HR3D seismics for the above-zone monitoring of offshore CO2 injection and storage projects.

Feng, Ye E.↗

1000 Soils Pilot Dataset, version 8, May 2025

This record hosts data generated by the 1000 Soils Pilot. Data will be updated as more become available. Please see the most recent data upload for current data. A beta visualization tool is available for some data types at https://shinyproxy.emsl.pnnl.gov/app/1000soils. Please submit any suggestions or comments through the 'contact' tab. We are actively working to improve visualizations and value all feedback. Data completed include: Geochemistry, texture, respiration, and enzyme activities FTICR-MS organic matter chemistry Microbial biomass C and N TOC/TDN of water-extractable OM X-ray computed tomography (derived metrics available here, raw data available upon request) Metagenomes; a variety of data formats are available upon request Soil hydraulic properties Data in progress: LC-MS/MS in development, timeline TBD, inquire for status 1000S_processed_BGC_summary.csv contains all available biogeochemical data; microbial biomass C and N; and TOC/TDN of water-extractable OM; and 1000S_Tomography.xslx contains a summary of data generated via X-ray computed tomography. icr_v2_corems2.csv contains FTICR-MS data processed by CoreMS version 2. These data are merged by formula across instrument runs to enable cross-sample comparisons. Technical replicates are merged by retaining peaks present in 2 out of 3 replicates. 1000Soils_Metadata_Site_Mastersheet_v1.csv contains site information. Soil Hydraulics_corrected_02042025.xlsx contains soil hydraulics information. Readme File_v4.xlsx is the readme file. Please contact the MONet project (monet.emsl@pnnl.gov) or Emily Graham (emily.graham@pnnl.gov) with questions. The following file and all raw data are available upon request: icr_by_mass_for_single_sample_analysis_only.csv contains FTICR-MS data processed by CoreMS and is intended for usage in the calculation of biochemical transformations within samples only. These data are not acceptable for cross-sample comparison of masses because they are from multiple instrument runs. For more information, please see: https://www.emsl.pnnl.gov/monet and https://sc-data.emsl.pnnl.gov/monet Acknowledgment: Soil data were provided by the Molecular Observation Network (MONet) at the Environmental Molecular Sciences Laboratory (https://ror.org/04rc0xn13), a DOE Office of Science user facility sponsored by the Biological and Environmental Research program under Contract No. DE-AC05-76RL01830. The work (proposal: 10.46936/10.25585/60008970) conducted by the U.S. Department of Energy, Joint Genome Institute (https://ror.org/04xm1d337), a DOE Office of Science user facility, is supported by the Office of Science of the U.S. Department of Energy operated under Contract No. DE-AC02-05CH11231. The Molecular Observation Network (MONet) database is an open, FAIR, and publicly available compilation of the molecular and microstructural properties of soil. Data in the MONet open science database can be found at https://sc-data.emsl.pnnl.gov/.

biogeochemistry↗