SEARCH · Engineering Papers

Results for “Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

All sources NASA NTRS DOE OSTI

DOE OSTI2018

Workflow-based automatic processing for Internet of Floating Things crowdsourced data

Data from sensors incorporated into mobile devices, such as networked navigational sensors, can be used to capture detailed environmental information. We describe here a workflow and framework for using sensors on boats to construct unique new datasets of underwater topography (bathymetry). Starting with a large number of measurements of position, depth, etc., obtained from such an Internet of Floating Things, we illustrate how, with a specialized protocol, data can be communicated to cloud resources, even when using delayed, intermittent, or disconnected networks. We then propose a method for automatic sensor calibration based on a novel reputation approach. Sampled depth data are interpolated efficiently on a cloud computing platform in order to provide a continuously updated bathymetric database. Our prototype implementation uses the FACE-IT Galaxy workflow engine to manage network communication and exploits the computational power of GPGPUs in a virtualized cloud environment, working with a CUDA-parallel algorithm, for efficient data processing. We report on an initial evaluation involving data from a sailing vessel in Italian coastal waters.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

2017 Western Passage Tidal Energy Resource Characterization Measurements

These data are from tidal resource characterization measurements collected between April and July 2017 in Western Passage near Eastport, Maine, USA. The dataset contains the following four sub-datasets, each of which is described in more detail in the README.pdf. 1. A bottom-mounted Teledyne RDI Workhorse 600 kHz acoustic Doppler current profiler (ADCP) was deployed at 44.92015 N, 66.98915 W in ~50 m of water from 3 April to 18 July (106 days). Data were recorded in 6-minute increments in the ENU (East, magnetic North, Up) coordinate system with bin-mapping enabled. 2. A bottom-mounted Nortek Signature 500 kHz ADCP was deployed at 44.92192 N, 66.98913 W in ~50 m of water from 4 April to 18 July (105 days). Data were sampled and recorded at 2 Hz and recorded in the ENU (East, magnetic North, Up) coordinate system. 3. Between those stations along a cross-channel transect, a Stable Tidal Turbulence Mooring (STTM) positioned ~10 m above the seabed was deployed for one week during a spring tide. The STTM was outfitted with two Nortek Vector acoustic Doppler velocimeters equipped with inertial motion units (ADVs), a bottom-tracking downward-looking Teledyne RDI Workhorse 600 kHz ADCP to provide motion-corrected flow and turbulence characteristics at high temporal resolution, and an upward-looking Teledyne RDI Sentinel V20 ADCP. The STTM was deployed at 44.92098 N, 66.98922 W from 24-31 May. 4. A vessel-mounted Teledyne RDI Workhorse 300 kHz ADCP collected current data along three transects over two days, 4-5 April. The data processing used DOLfYN version 0.11.2. All hdf5 files (i.e., files ending in `.h5`) contained here can be opened using that version of DOLfYN (e.g., `dat = dolfyn.load('')`). All distances are in meters (e.g., depth, range, MLLW, hab, eta, z_), and all velocities in m/s. See the DOLfYN documentation https://lkilcher.github.io/dolfyn/), and/or the Nortek and Teledyne RDI documentation for additional details. Additional details on the dataset can be found in the README.pdf, including: - Format details of each data file. - How to regenerate the data-processing (using the files in the `wp2017_processing.zip` archive).

16 TIDAL AND WAVE POWER↗

DOE OSTI2015

Applying Hanford Tank Mixing Data to Define Pulse Jet Mixer Operation

Pulse jet mixed (PJM) process vessels are being developed for storing, blending, and chemical processing of nuclear waste slurries at the Waste Treatment and Immobilization Plant (WTP) to be built at Hanford, Washington. These waste slurries exhibit variable process feed characteristics including Newtonian to non-Newtonian rheologies over a range of solids loadings. Waste feed to the WTP from the Hanford Tank Farms will be accomplished via the Waste Feed Delivery (WFD) system which includes million-gallon underground storage double-shell tanks (DSTs) with dual-opposed jet mixer pumps. Experience using WFD type jet mixer pumps to mobilize actual Hanford waste in DSTs may be used to establish design threshold criteria of interest to pulse jet mixed process vessel operation. This paper describes a method to evaluate the pulse jet mixed vessel capability to process waste based on information obtained during mobilizing and suspending waste by the WFD system jet mixer pumps in a DST. Calculations of jet velocity and wall shear stress in a specific pulse jet mixed process vessel were performed using a commercial computational fluid dynamics (CFD) code. The CFD-modelled process vessel consists of a 4.9-m- (16-ft-) diameter tank with a 2:1 semi-elliptical head, a single, 10-cm (4-in.) downward facing 60-degree conical nozzle, and a 0.61-m (24-in.) inside diameter PJM. The PJM is located at 70% of the vessel radius with the nozzle stand-off-distance 14 cm (6 in.) above the vessel head. The CFD modeled fluid velocity and wall shear stress can be used to estimate vessel waste-processing performance by comparison to available actual WFD system process data. Test data from the operation of jet mixer pumps in the 23-m (75-ft) diameter DSTs have demonstrated mobilization, solid particles in a sediment matrix were moved from their initial location, and suspension, mobilized solid particles were moved to a higher elevation in the vessel than their initial location, of waste solids. Jet mixer pumps were used in Hanford waste tank 241-AZ-101, and at least 95% of the 0.46-m (18-in.) deep sediment, with a shear strength of 1,500 to 4,200 Pa, was mobilized. Solids with a median particle size of 43 μm, 90th percentile of 94μm, were suspended in tank 241-AZ-101 to at least 5.5 m (216 in.) above the vessel bottom. Analytical calculations for this jet mixer pump test were used to estimate the velocities and wall shear stress that mobilized and suspended the waste. These velocities and wall shear stresses provide design threshold criteria which are metrics for system performance that can be evaluated via testing. If the fluid motion in a specific pulse jet mixed process vessel meets or exceeds the fluid motion of the demonstrated performance in the WFD system, confidence is provided that that vessel will similarly mobilize and suspend those solids if they were within the WTP. The single PJM CFD-calculated jet velocity and wall shear stress compare favorably with the design threshold criterion estimated for the tank 241-AZ-101 process data. Therefore, for both mobilization and suspension, the performance data evaluated from the WFD system testing increases confidence that the performance of the pulse jet mixed process vessels will be sufficient to process that waste even if that waste is not fully characterized.

mixing' slurry↗

DOE OSTI2012

Raman Lidar Profiles Best Estimate Value-Added Product Technical Report

The ARM Raman lidars are semi-autonomous ground-based systems that transmit at a wavelength of 355 nm with 300 mJ, ~5 ns pulses, and a pulse repetition frequency of 30Hz. Signals from the various detection channels are processed to produce time- and height-resolved estimates of several geophysical quantities, such as water vapor mixing ratio, relative humidity, aerosol scattering ratio, backscatter, optical depth, extinction, and depolarization ratio. Data processing is currently handled by a suite of six value-added product (VAP) processes. Collectively, these processes are known as the Raman Lidar Profiles VAP (RLPROF). The top-level best-estimate (BE) VAP process was introduced in order to bring together the most relevant information from the intermediate-level VAPs. As such, the BE process represents the final stage in data processing for the Raman lidar. Its principal function is to extract the primary variables from each of the intermediate-level VAPs, perform additional quality control, and combine all of this information into a single output file for the end-user. The focus of this document is to describe the processing performed by the BE VAP process.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

TEAMER: Pitching Foil Crossflow Turbine Efficiency Data

This dataset documents the efficiency testing of a pitching foil crossflow turbine, conducted at the University of New Hampshire's (UNH) Chase Ocean Engineering Laboratory tow tank facility. The tests explored various pitch phases and amplitudes, ranging from 0 to 18 degrees, across different flow speeds and turbine RPMs. Specifically, testing was performed at pitch amplitudes of 0,3,6,9,12,15 and 18 degrees. The results indicate that turbine efficiency improves with increased pitch amplitude, reaching optimal performance at 12 degrees. Modifications were made to the test frame by UNH to enhance the stiffness of the support structure. The testing specifically mapped the efficiency of a single quadrant of the Ocean Renewable Power Company (ORPC) Autonomous Turbine Generator Unit (ATGU), covering the full range of allowable pitching amplitudes and phases. The dataset provides both raw and processed data, including drag and torque measurements, and contains Python scripts used for data processing and visualization. This research was supported by funding from TEAMER RFTS 1 (Request for Technical Support).

16 TIDAL AND WAVE POWER↗

DOE OSTI2021

In Situ Transmission Electron Microscopy: Signal processing challenges and examples

Transmission electron microscopy (TEM) is a powerful tool for imaging material structure and characterizing material chemistry. Recent advances in data collection technology for TEM have enabled high-volume and high-resolution data collection at a microsecond frame rate. Here, taking advantage of these advances in data collection rates requires the development and application of data processing tools, including image analysis, feature extraction, and streaming data processing techniques. In this article, we highlight a few areas in materials science that have benefited from combining signal processing and statistical analysis with data collection capabilities in TEM and present a future outlook on opportunities of integrating signal processing with automated TEM data analysis.

36 MATERIALS SCIENCE↗

DOE OSTI2019

BigPanDA: PanDA Workload Management System and its Applications beyond ATLAS

Modern experiments collect peta-scale volumes of data and utilize vast, geographically distributed computing infrastructure that serves thousands of scientists around the world. Requirements for rapid, near real-time data processing, fast analysis cycles and need to run massive detector simulations to support data analysis pose special premium on efficient use of available computational resources. A sophisticated Workload Management System (WMS) is needed to coordinate the distribution and processing of data and jobs in such environment. The ATLAS experiment at CERN uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. While PanDAcurrently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, it runs around 2 million jobs per day on hundreds of Grid sites and serving thousands of ATLAS users. In 2017 about 1.5 exabytes of data were processed with PanDA.In 2012 BigPanDA project project was started with aim to introduce new types of computing resources into ATLAS computing infrastructure, but also to offering PanDA features to different data-intensive applications for projects and experiments outside of ATLAS and High-Energy and Nuclear Physics. In this article we will present accomplishments and discuss possible directions for future work.

Svirin, Pavlo↗

DOE OSTI2022

Crossing the Finish Line: Integration of Data-Driven Process Control for Maximization of Energy and Resource Efficiency in Advanced Water Resource Recovery Facilities

Improvements in process monitoring and control at water resource recovery facilities (WRRFs) could result in reductions in electricity consumption, chemical inputs, and greenhouse gas emissions, as well as improved energy recovery. Many current WRRF data collection, monitoring, and control approaches use 20th century process monitoring and control systems, which require large design safety factors to ensure reliability in the absence of more advanced, precise controls. Implementation of more modern data-driven control tools could lead to more efficient operations that provide intrinsic reliability with better overall process performance at full-scale. This presentation provides an overview of a recently initiated project "Crossing the Finish Line: Integration of Data-Driven Process Control for Maximization of Energy and Resource Efficiency in Advanced Water Resource Recovery Facilities" which will (1) develop and demonstrate data-driven process controls at full-scale facilities for five promising WRRF Applications (i.e., process technologies) that provide whole-plant approaches and offer substantial energy and resource recovery benefits, and (2) create a toolbox of new process control approaches and an implementation guide including five examples for application at utilities. The presentation also provides a detailed overview of the research approach and progress being made on one of the five Applications, namely Application 2: Biological Nutrient Removal (BNR): ammonium-based aeration control (ABAC) / ammonia vs. NOx (AvN) + partial denitration with anammox (PdNA), which is being implemented at Hampton Roads Sanitation District. This project is a collaboration of work being conducted by DC Water, Hampton Roads Sanitation District, Metro Water Recovery, University of Michigan, Northwestern University, US Military Academy - West Point, Black & Veatch, and Oak Ridge National Laboratory. Research partner: U.S. Department of Energy.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2017

Open Source Release of NJOY2016 and NJOY21

NJOY has been the standard for nuclear data processing for many decades. NJOY is a comprehensive computer code used to produce point-wise and multigroup nuclear data application libraries from cross sections and related quantities obtained from evaluated nuclear data in the Evaluated Nuclear Data Format (ENDF). NJOY was first publicly released in 1977 and distributed by what is now the Radiation Safety Information Computational Center (RSICC) at Oak Ridge National Laboratory and the National Energy Software Center (NESC) at Argonne National Laboratory. From that time until the release of NJOY2012, NJOY has been distributed by RSICC and made available to researchers outside the United States by agreements between RSICC and the NEA Databank. NJOY2012 was distributed by the Feynman Center at Los Alamos National Laboratory which did not have agreements with the NEA Databank, thus making it more difficult for non US citizens to obtain copies of NJOY2012. Furthermore, NJOY has had export control restrictions preventing some researchers from obtaining NJOY at all. Two new versions of NJOY have been released, NJOY2016 and NJOY21, in January of 2017. Both NJOY2016 and NJOY21 have been released with no export control restrictions and are distributed with an open-source license. This is a big departure from previous releases and impacts not only who has access to NJOY, but also how updates are made available to users and how collaboration is facilitated among interested developers. In addition to distributing NJOY2016 and NJOY21 with an open-source license, these two new versions provide new capabilities for NJOY which will maintain NJOY as the standard for nuclear data processing into the foreseeable future. We report in this summary about the changes that come with these two new versions of NJOY and how this impacts the users and long-time contributors to NJOY.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2026

Poisson Log-Normal Process for Count Data Prediction

Modeling count data is important in physics and other scientific disciplines, where measurements often involve discrete, non-negative quantities such as photon or neutrino detection events. Traditional parametric approaches can be trained to generate integer-count predictions but may struggle with capturing complex, non-linear dependencies often observed in the data. Gaussian process (GP) regression provides a robust non-parametric alternative to modeling continuous data; however, it cannot generate integer outputs. We propose the Poisson Log-Normal (PoLoN) process, a framework that employs GP to model Poisson log-rates. As in GP regression, our approach relies on the correlations between data points captured via GP kernel structure rather than explicit functional parameterizations. We demonstrate that the PoLoN predictive distribution is Poisson-LogNormal and provide an algorithm for optimizing kernel hyperparameters. Furthermore, we adapt the PoLoN approach to the problem of detecting weak localized signals superimposed on a smoothly varying background - a task of considerable interest in many areas of science and engineering. Our framework allows us to predict the strength, location and width of the detected signals. We evaluate PoLoN's performance using both synthetic and real-world datasets, including the open dataset from CERN which was used to detect the Higgs boson at the Large Hadron Collider. Our results indicate that the PoLoN process can be used as a non-parametric alternative for analyzing, predicting, and extracting signals from integer-valued data.

Saha, Anushka [Rutgers U., Piscataway]↗

DOE OSTI2025

Compactly‐Supported Nonstationary Kernels for Computing Exact Gaussian Processes on Big Data

The Gaussian process (GP) is a widely used method for analyzing large-scale data sets, including spatio-temporal measurements of nonlinear processes that are now commonplace in the environmental sciences. Traditional implementations of GPs involve stationary kernels (also termed covariance functions) that limit their flexibility, and exact methods for inference that prevent application to data sets with more than about 10,000 points. Modern approaches to address stationarity assumptions generally fail to accommodate large data sets, while all attempts to address scalability focus on approximating the Gaussian likelihood, which can involve subjectivity and lead to inaccuracies. In this work, we explicitly derive an alternative kernel that can discover and encode both sparsity and nonstationarity. We embed the kernel within a fully Bayesian GP model and leverage high-performance computing resources to enable the analysis of massive data sets. We demonstrate the favorable performance of our novel kernel relative to existing exact and approximate GP methods across a variety of synthetic data examples. Furthermore, we conduct space–time prediction based on more than 1 million measurements of daily maximum temperature and verify that our results outperform state-of-the-art methods in the Earth sciences. More broadly, having access to exact GPs that use ultra-scalable, sparsity-discovering, nonstationary kernels allows GP methods to truly compete with a wide variety of machine learning methods.

Gaussian processes↗

DOE OSTI2017

Applying Service-Oriented Architecture to Archiving Data in Control and Monitoring Systems

Current trends in the architecture of software systems focus our attention on building systems using a set of loosely coupled components, each providing a specific functionality known as service. It is not much different in control and monitoring systems, where a functionally distinct sub-system can be identified and independently designed, implemented, deployed and maintained. One functionality that renders itself perfectly to becoming a service is archiving the history of the system state. The design of such a service and our experience of using it are the topic of this article. The service is built with responsibility segregation in mind, therefore, it provides for reducing data processing on the data viewer side and separation of data access and modification operations. The service architecture and the details concerning its data store design are discussed. An implementation of a service client capable of archiving EPICS process variables (PV) and LabVIEW shared variables is presented. Data access tools, including a browser-based data viewer and a mobile viewer, are also presented.

Nogiec, J. M.↗

DOE OSTI2025

A Cloud-Tracking Data Set for the CSAPR2 Adaptive Scanning during TRACER

The U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) User Facility (Mather and Voyles 2013) deployed the first ARM Mobile Facility (AMF1; Miller et al. 2016) near LaPorte, Texas to support the Tracking Aerosol Convection Interactions Experiment (TRACER) (Jensen et al. 2025) near Houston, Texas. From October 2021 to September 2022, AMF1 was deployed to 29.67° N, 95.06° W near LaPorte, Texas and the 2nd Generation C-band Scanning ARM Precipitation Radar (CSAPR2) was deployed to a supplementary site at 29.53° N, 95.28° W (Figure 1). During an intensive operational period (IOP) from 1 June to 30 September 2022, the CSAPR2 sampled precipitation echoes in an adaptive scanning mode following the Multisensor Agile Adaptive Scanning (MAAS) framework (Kollias et al. 2020). MAAS helped optimize the CSAPR2 scan strategy to perform frequent plan position indicator (PPI) and range height indicator (RHI) scans (Lamer et al. 2023). Details of the CSAPR2 scanning, data processing, and calibration procedures used by the principal investigator (PI), and the PI data files are described by Oue et al. (2023). Details of the CSAPR2 operational performance, ARM data processing and correction procedures, and data quality masks are described by Feng et al. (2024a).

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Machine learning-driven predictive resource management in complex science workflows

Here, the collaborative efforts of large communities in science experiments, often comprising thousands of global members, reflect a monumental commitment to exploration and discovery. Recently, advanced and complex data processing has gained increasing importance in science experiments. Data processing workflows typically consist of multiple intricate steps, and the precise specification of resource requirements is crucial for each step to allocate optimal resources for effective processing. Estimating resource requirements in advance is challenging due to a wide range of analysis scenarios, varying skill levels among community members, and the continuously increasing spectrum of computing options. One practical approach to mitigate these challenges involves initially processing a subset of each step to measure precise resource utilization from actual processing profiles before completing the entire step. While this two-staged approach enables processing on optimal resources for most of the workflow, it has drawbacks such as initial inaccuracies leading to potential failures and suboptimal resource usage, along with overhead from waiting for initial processing completion, which is critical for fast-turnaround analyses. In this context, our study introduces a novel pipeline of machine learning models within a comprehensive workflow management system, the Production and Distributed Analysis (PanDA) system. These models employ advanced machine learning techniques to predict key resource requirements, overcoming challenges posed by limited upfront knowledge of characteristics at each step. Accurate forecasts of resource requirements enable informed and proactive decision-making in workflow management, enhancing the efficiency of handling diverse, complex workflows across heterogeneous resources.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

Gemini frontier fields: wide-field adaptive optics K{sub s}-band imaging of the galaxy clusters MACS J0416.1-2403 and ABELL 2744

We have observed two of the six Frontier Fields galaxy clusters, MACS J0416.1-2403 and Abell 2744, using the Gemini Multi-Conjugate Adaptive Optics System (GeMS) and the Gemini South Adaptive Optics Imager (GSAOI). With 0.″ 08–0.″ 10 FWHM our data are nearly diffraction-limited over a 100{sup ′′}×100{sup ′′} wide area. GeMS/GSAOI complements the Hubble Space Telescope (HST) redwards of 1.6 μm with twice the angular resolution. We reach a 5σ depth of K{sub s}∼25.6 mag (AB) for compact sources. In this paper, we describe the observations, data processing, and initial public data release. We provide fully calibrated, co-added images matching the native GSAOI pixel scale as well as the larger plate scales of the HST release, adding to the legacy value of the Frontier Fields. Our work demonstrates that even for fields at high galactic latitude where natural guide stars are rare, current multi-conjugated adaptive optics technology at 8 m telescopes has opened a new window on the distant universe. Observations of a third Frontier Field, Abell 370, are planned.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

16S and ITS Amplicon Sequencing Fastq files and metadata from PARCHED Panama Tropical Forest soils, 2019-2020,

Model projections predict tropical forests will experience longer periods of drought and more intense precipitation cycles under a changing climate. Such transitions have implications for structure-function relationships within microbial communities. We examine how chronic drying might reshape prokaryotic and fungal communities across four lowland forests in Panama with a wide variation in mean annual precipitation and soil fertility. Four sites were established across a 1000 mm span in mean annual precipitation (2335 to 3300 mm). We expected microbial communities at sites with lower MAP to be less sensitive to chronic drying than sites with higher MAP; while fungal communities to be more resistant to disturbance than prokaryotes. At each location, partial throughfall exclusion structures were established over 10 x 10 m plots to reduce direct precipitation input. Raw demultiplexed sequences (bacteria, archaea, fungal) from soil samples taken from PARCHED Panama Tropical Forest throughfall exclusion experiments. Files that contain 16S are sequences from prokaryotes, ITS indicates sequences from fungi. Compressed fastq files are contained in Field_PARCHED_ITS_fastq_2020.zip, Field_PARCHED_ITS_fastq_2019.zip, Field_PARCHED_16S_fastq_2020.zip, Field_PARCHED_16S_fastq_2019.zip.There are four sites across the isthmus of Panama, each with 4 control and 4 exclusion plots. Throughfall exclusion shelters were built to intercept 50% of throughfall precipitation that hits the soil. Samples were taken on May 2019 and Jan 2020 from 0-10 cm and 10-20 cm depth approximately 9 and 18 months after shelter installation. Metadata and sample IDs for fastq files for 2019 sampling are within Field_PARCHED_Metadata_2019_ITS.csv and Field_PARCHED_Metadata_2019_16S.csv. The metadata for both 16S and ITS fastq files for January 2020 is included in Field_PARCHED_Metadata_2020.csv. No data processing or QA/QC was done on the raw data. Data processing example provided in R notebook file.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2016

Using NJOY to Create MCNP ACE Files and Visualize Nuclear Data

We provide lecture materials that describe the input requirements to create various MCNP ACE files (Fast, Thermal, Dosimetry, Photo-nuclear and Photo-atomic) with the NJOY Nuclear Data Processing code system. Input instructions to visualize nuclear data with NJOY are also provided.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2017

Impacts of Carbon Capture on Life Cycle Inventory of Ammonia and Petroleum Products for Comparison to Thermoelectric Power Generation - LCAXVII

This work was presented at the American Center for Life Cycle Assessment (ACLCA) LCAXVII Conference in Portsmouth, NH on October 5, 2017. The objectives of this study are to create an updated baseline life cycle inventory for petroleum products considering the full suite of TRACI impacts that expands upon Cooney et al. and to understand the effect of implementing carbon capture on the environmental profile of refineries. This inventory adds toxic air emissions and releases to water and soil to NETL’s previous unit process data. The approach demonstrates how detailed national inventory data can be used to improve LCA unit process data. Facility-level variability is presented and the implications for uncertainty analysis and interpretation of results are discussed.

Young, Ben↗