Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data requirements”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Eureka: Enabling Fine-Grained Access and Range Queries on Compressed Scientific Data via Data-Index Co-Compression

Handling large-scale scientific data in high-performance computing (HPC) environments poses significant challenges, including excessive I/O, high storage costs, and slow query performance. Traditional approaches often require full data decompression and scans, making them impractical for real-time or interactive analysis. To address these limitations, we introduce Eureka, a unified data-index co-compression framework that enables fine-grained access and efficient range queries on compressed scientific datasets. Eureka integrates spatial domain decomposition with block-wise error-bounded lossy compression to support selective decompression. It constructs a hierarchical AVL-tree index during compression to capture block-level value ranges, enabling fast pruning during query execution. To reduce metadata overhead, the index itself is also compressed while ensuring recall-preserving results. Experiments on six diverse HPC simulation datasets show that Eureka achieves up to 25x data compression and over 300x index compression, surpassing state-of-the-art compressors such as SZ3 and ZFP in rate-distortion performance. Additionally, Eureka delivers over 30x speedup for low-selectivity range queries, making it a scalable and efficient solution for modern scientific data analysis.

Yan, Ning↗
DOE OSTI2020

Identifying Suspect Instrument Intervals Using Midnight Noise Time Histories

With an intense work ethic, and high levels of devotion to their craft, seismologists continue to battle the elements and wrestle with complex logistics in their efforts to field instrumentation over the entire globe. We are fortunate to see data volumes accelerating in size, and quality continuing to improve; for example, universal timing issues can now be considered rare. However, instrument response issues are still common, and can hinder research that seeks to understand amplitudes of seismic wavefields. We are interested in using the accumulations of seismic data to develop models that will predict high frequency (0.2-20 Hz, and higher) signal amplitudes for explosion monitoring purposes over broad areas, which requires large data sets and extensive quality control. To identify response and station health issues, we have collected noise time histories for global seismic data, focusing on measurements near (but not restricted to) midnight to eliminate diurnal variations, and have manually determined time intervals that appear inconsistent with background behavior. We assign descriptive labels, but do not attempt to diagnose causes. We use these intervals to discard data. To date, we have examined 39,260 channels from 11,105 stations, heavily weighted toward IRIS holdings, dates through 2017 (depending on station, roughly the date we started our manual review), bands between 1 and 8 Hz, finding 24,733 anomalous time intervals. The great majority (90%) of these intervals appear to be shifts of constant offset, often bounded by times of known instrument changes, likely the result of poor documentation of response parameters at one of many stages between the field and the plotter. We hope these results can be of use to our colleagues, and would encourage community efforts to diagnose anomalous behavior, and fix poor responses. We also hope these results will support automation efforts, including application of supervised learning techniques.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Applying Transfer Learning for Street-Scale Nuisance Flood Forecasting in Coastal-Urban Cities

An important challenge with Machine Learning (ML) is its transferability; that is, whether a ML model trained on one set of data can be applied to a second set of data without requiring a full re-training of the model. Transfer Learning (TL) addresses this challenge by transferring knowledge learned in the source domain (the data it was trained on) to the target domain (a second set of data that is statistically different but related, which the model was not trained on). This study investigates the use of TL for street-scale nuisance flood forecasting by exploring whether a ML model trained on data collected for one set of streets can effectively forecast flooding for another set of streets in the same city using TL. The envisioned use case is a city deploying a new flood depth monitoring sensor on a street and using TL to apply a ML model, trained on sensor data from an existing flood depth sensor network, to this new street. Eventually, the new flood depth sensor will have a sufficient dataset for training its own ML model, but TL can be used to fill the gap in time while this new dataset is being generated. This method is explored using a Long Short-Term Memory (LSTM) model trained on data for the flood-prone streets of Norfolk City, Virginia. The data used for training includes environmental time series (rainfall, tide), topographic features (Digital Elevation Model (DEM), Topographic Wetness Index (TWI), Depth To Water (DTW)), and street-scale flood depth time series obtained from a high-fidelity physics-based model, acting as a synthetic street-scale stream depth sensor dataset since actual stream depth sensor data is generally unavailable for most cities. A set of 180 flood-prone streets was used to train a base model, while another set of 180 flood-prone streets was used to re-train that model using different TL strategies. The results show that full-weight re-training proved most effective and minimal re-training of only the output layer was insufficient. The advantage of TL was most pronounced when target data was limited, meaning data collected at the new water depth sensor location included generally less than 18 flood events. As target data increased beyond 18 flood events, the benefit of TL diminished relative to training a ML model directly on the local flood events. These findings can assist cities as they implement street-scale flood sensing systems to create accurate forecasts for new sensing locations that do not yet have sufficient data records to train a local ML model.

Roy, Binata [Univ. of Virginia, Charlottesville, V↗
DOE OSTI2022

Semi-Automated, Object-Based Tomography of Dislocation Structures

The characterization of the three-dimensional arrangement of dislocations is important for many analyses in materials science. Dislocation tomography in transmission electron microscopy is conventionally accomplished through intensity-based reconstruction algorithms. Although such methods work successfully, a disadvantage is that they require many images to be collected over a large tilt range. Here, we present an alternative, semi-automated object-based approach that reduces the data collection requirements by drawing on the prior knowledge that dislocations are line objects. Our approach consists of three steps: (1) initial extraction of dislocation line objects from the individual frames, (2) alignment and matching of these objects across the frames in the tilt series, and (3) tomographic reconstruction to determine the full three-dimensional configuration of the dislocations. Drawing on innovations in graph theory, we employ a node-line segment representation for the dislocation lines and a novel arc-length mapping scheme to relate the dislocations to each other across the images in the tilt series. We demonstrate the method for a dataset collected from a dislocation network imaged by diffraction-contrast scanning transmission electron microscopy. Based on these results and a detailed uncertainty analysis for the algorithm, we discuss opportunities for optimizing data collection and further automating the method.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

ZMPY3D: accelerating protein structure volume analysis through vectorized 3D Zernike moments and Python-based GPU integration

Abstract Motivation Volumetric 3D object analyses are being applied in research fields such as structural bioinformatics, biophysics, and structural biology, with potential integration of artificial intelligence/machine learning (AI/ML) techniques. One such method, 3D Zernike moments, has proven valuable in analyzing protein structures (e.g., protein fold classification, protein–protein interaction analysis, and molecular dynamics simulations). Their compactness and efficiency make them amenable to large-scale analyses. Established methods for deriving 3D Zernike moments, however, can be inefficient, particularly when higher order terms are required, hindering broader applications. As the volume of experimental and computationally-predicted protein structure information continues to increase, structural biology has become a “big data” science requiring more efficient analysis tools. Results This application note presents a Python-based software package, ZMPY3D, to accelerate computation of 3D Zernike moments by vectorizing the mathematical formulae and using graphical processing units (GPUs). The package offers popular GPU-supported libraries such as CuPy and TensorFlow together with NumPy implementations, aiming to improve computational efficiency, adaptability, and flexibility in future algorithm development. The ZMPY3D package can be installed via PyPI, and the source code is available from GitHub. Volumetric-based protein 3D structural similarity scores and transform matrix of superposition functionalities have both been implemented, creating a powerful computational tool that will allow the research community to amalgamate 3D Zernike moments with existing AI/ML tools, to advance research and education in protein structure bioinformatics. Availability and implementation ZMPY3D, implemented in Python, is available on GitHub (https://github.com/tawssie/ZMPY3D) and PyPI, released under the GPL License.

Lai, Jhih-Siang (ORCID:0000000156775890)↗
DOE OSTI2024

Performance Analysis of Data Processing in Distributed File Systems with Near Data Processing

In the era of big data, the escalating volume and velocity of data generation pose significant challenges in data processing. Traditional systems like Spark and Hadoop manage the increasing amount and velocity of data by improving data placement and processing speeds. However, they face inherent limitations due to the essential data movement required for processing. In this paper, we explore the Skyhook framework, a novel extension of the Ceph distributed system, which significantly reduces the need for data movement. We present an extensive case study using the Skyhook framework, applying it with the TPC-H and K-means clustering algorithms. More specifically, we leverage the TPC-H benchmark to distinguish between CPU-intensive and I/O-intensive tasks. We explore the integration of K-means clustering into SQL, coupled with a near-data processing system to offload the computational burden of the K-means clustering algorithm to storage nodes. We conduct a comprehensive performance evaluation of distributed data processing applications across three processing approaches: traditional layout (baseline), optimized layout, and near-data processing. Additionally, we introduce the use of the FIO tool to simulate real-world system workloads, enabling the measurement of performance metrics such as average latency and CPU utilization. Our research is a significant advance in understanding how to optimize data processing systems to meet the demands of the modern data landscape.

Hou, Shiyue↗
DOE OSTI2021

Characterizing the Sample Selection for Supernova Cosmology

Type Ia supernovae (SNe Ia) are used as distance indicators to infer the cosmological parameters that specify the expansion history of the universe. Parameter inference depends on the criteria by which the analysis SN sample is selected. Only for the simplest selection criteria and population models can the likelihood be calculated analytically, otherwise it needs to be determined numerically, a process that inherently has error. Numerical errors in the likelihood lead to errors in parameter inference. This article presents toy examples where the distance modulus is inferred given a set of SNe at a single redshift. Parameter estimators and their uncertainties are calculated using Monte Carlo techniques. The relationship between the number of Monte Carlo realizations and numerical errors is presented. The procedure can be applied to more realistic models and used to determine the computational and data management requirements of the transient analysis pipeline.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Accurate and Data‐Efficient Micro X‐ray Diffraction Phase Identification Using Multitask Learning: Application to Hydrothermal Fluids

Traditional analysis of highly distorted micro X‐ray diffraction (μ‐XRD) patterns from hydrothermal fluid environments is a time‐consuming process, often requiring substantial data preprocessing and labeled experimental data. Herein, the potential of deep learning with a multitask learning (MTL) architecture to overcome these limitations is demonstrated. MTL models are trained to identify phase information in μ‐XRD patterns, minimizing the need for labeled experimental data and masking preprocessing steps. Notably, MTL models show superior accuracy compared to binary classification convolutional neural networks. Additionally, introducing a tailored cross‐entropy loss function improves MTL model performance. Most significantly, MTL models tuned to analyze raw and unmasked XRD patterns achieve close performance to models analyzing preprocessed data, with minimal accuracy differences. This work indicates that advanced deep learning architectures like MTL can automate arduous data handling tasks, streamline the analysis of distorted XRD patterns, and reduce the reliance on labor‐intensive experimental datasets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Specifying Calibration of Energy-Measuring Equipment

Instruments and standards need to be calibrated periodically to ensure that their use yields accurate measurements. Calibration needs, however, vary in sophistication, based on user expectations. This white paper reviews calibration terminology and standards, and describes best practices that have been developed for a) calibrating measurement equipment to ensure some known level of accuracy, and b) accrediting calibration service providers. Calibration is discussed in the context of metrological traceability, and excerpts from laboratory scopes of accreditation are shared to reveal the diversity of terminology and format among them. In an effort to aid those who currently calibrate measuring equipment or who have new or changing needs for calibrating measuring equipment, rationale is provided for why a specification might be used to request calibration services that meet specific test needs. Given the growing interest in using data to manage the energy consumption of systems, and the increasing number and variety of systems (e.g., building, computing) that can report such information, use cases requiring energy data are used to facilitate further discussion and provide examples. Commercially available service providers who are accredited for energy-measuring equipment calibration are compared and contrasted, and a specification template that might be used for requesting this calibration is presented. The specification template should be tailored to meet each user’s needs. To illustrate, an example set of energy-measuring-equipment test conditions (reflecting the planned usage of the device to be calibrated) is used to develop a customized calibration specification, and commercially available service providers are assessed in terms of their qualifications for calibration to that particular implementation of the specification template.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Workshop Summary: Bridging the Gap Between Atmospheric Science and Grid Integration

The need for dedicated, accurate, expertly curated weather data is increasingly important as the share of variable renewable energy increases on the power system. Projections for futures with very high (50+% annual energy) shares of variable generation require ongoing assessment of data requirements from industry stakeholders in their power system operation and planning contexts. In March 2024, NREL organized a workshop entitled "Bridging the Gap Between Atmospheric Science and Grid Integration Workshop", which brought atmospheric scientists and power system experts together to refine the requirements of atmospheric datasets for grid integration, and to describe a holistic approach to creating new and regularly updated national scale wind datasets for power system planning and operations. The results of this workshop are being used to inform the near-term development and a longer-term strategy for DOE to produce relevant wind resource datasets and inform wider use of wind/solar/load data sets in power system planning. This presentation provides an overview of a preworkshop survey, an assessment of current state of the art of national-scale datasets for wind resource assessment and grid integration, insights on appropriate uses of the WTK-LED, power system perspectives on data needs, as well as recommended next steps as discussed in the workshop and how these steps support longer-term strategies.

17 WIND ENERGY↗
DOE OSTI2023

Environmental and Social Justice Considerations in a Circular Economy

A consideration of environmental justice (EJ) and social justice (SJ) is critical to minimize the impacts of technology deployment on local communities. SJ and EJ impacts occur in specific geographic locations but can cover a wide range of effects (e.g., air pollution, access to clean water, jobs, wages, and education), making it challenging to determine which metrics are appropriate to evaluate and which data are required for qualitative or quantitative analysis. This issue is only exacerbated when the technology in question is at an early technology readiness level (TRL) without community specific data. Here, we present a framework for evaluating the human health, local environment, and job implications of processes that are at early to middle TRLs. Using a case study on enzymatic polyethylene terephthalate (PET) recycling (middle TRL), we demonstrate how to qualitatively and quantitatively assess these EJ and SJ metrics for a circular economy context and how to communicate the results in a manner beneficial to both researchers and local communities.

circular economy↗
DOE OSTI2023

Adaptation of Temperature Dependent Thermal Neutron Libraries for Fast On-The-Fly Monte Carlo Sampling at Arbitrary Temperatures

Thermal neutron scattering data is necessary for reactor simulations of thermal reactors, but the data storage required can be unwieldy. This is exacerbated when simulations require a large temperature range, as is the case for accident scenarios and reactor start-up. Previous work conducted at RPI has explored and demonstrated a proof of concept of On-The-Fly (OTF) data storage with light water and crystalline graphite based on ENDF/B-VII.1. This provides an updated data storage paradigm for incoherent inelastic cross-sections that allow the data to be used to generate necessary sampling Cumulative Density Functions (CDFs) for any desired temperature. The work presented in this paper extends the previous work by implementing the proof of concept based on ENDF/B-VIII.0, instead of ENDF/B-VII.1, and to include additional materials, such as beryllium and oxygen in beryllium oxide and the new reactor grades of porous graphite. Completion of this work allows a single thermal data file to represent any thermal scattering event for a given material OTF for the temperature of interest, which greatly reduces the data storage needed with only a marginal cost in the calculation of the CDFs OTF. This paper will review the basic methodologies in generating the OTF data, as well as present the updates and improvements made in the generation process. Application of the methodologies covered in this paper show a significant reduction in the storage requirements for inelastic incoherent scattering events. In which, the evaluated ENDF/B-VIII.0 thermal scattering library can be reduced from 563.1 MB to 10.9 MB for light water and 339.3 MB to 13.6 MB for Be in BeO.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2025

Describing hadronization via histories and observables for Monte-Carlo event reweighting

We introduce a novel method for extracting a fragmentation model directly from experimental data without requiring an explicit parametric form, called Histories and Observables for Monte-Carlo Event Reweighting (HOMER), consisting of three steps: the training of a classifier between simulation and data, the inference of single fragmentation weights, and the calculation of the weight for the full hadronization chain. We illustrate the use of HOMER on a simplified hadronization problem, a q\bar{q} q q ‾ string fragmenting into pions, and extract a modified Lund string fragmentation function f(z) f ( z ) . We then demonstrate the use of HOMER on three types of experimental data: (i) binned distributions of high-level observables, (ii) unbinned event-by-event distributions of these observables, and (iii) full particle cloud information. After demonstrating that f(z) f ( z ) can be extracted from data (the inverse of hadronization), we also show that, at least in this limited setup, the fidelity of the extracted f(z) f ( z ) suffers only limited loss when moving from (i) to (ii) to (iii). Public code is available at https://gitlab.com/uchep/mlhad.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

Targeted Biomining and Machine Learning Approaches in Critical Minerals Revealed by a Biogeochemical Survey of a Coal Mine Drainage Remediation System

Abandoned coal mine drainage (AMD) remediation systems in Pennsylvania can concentrate critical minerals and materials (CMM) at levels comparable to mining-grade ores. Remediation systems have varying engineering features and are open to the environment, resulting in diverse microbial colonization and seasonal climate influences that may impact CMM speciation. The location of CMMs, the types of bacterial communities tolerant of these pollutant conditions, and the influence of localized climate on CMM rich remediation systems are not well characterized. Through a one-year spatiotemporal survey of biogeochemistry at a remediation system, we have initiated the process to address these questions. Rare Earth Elements (REE) ranged 180-1,200 ppm and greater than 1,500 bacterial ASVs were classified via 16S sequencing. Analyses indicate biogeochemical differences are heavily influenced by engineering features. Additionally, REE precipitants correlate strongly with the elements Al, Cu, Zn, Be, and U. Unearthing these trends has refined our line of inquiry to explore biological mining opportunities more closely with these metals. Furthermore, we created a Machine Learning Model for predicting AMD REE content, with 89% accuracy, using the data from this study and several others. Further training data is required to create a more reputable model. Recently, global research efforts have prioritized modeling work or the use of the few historical surveys to design experiments. Through our data, we challenge this approach, emphasizing the importance of expanding fundamental survey efforts prior to advanced product design and experimentation.

critical minerals↗
DOE OSTI2024

Robust unfolding of MeV x-ray spectra from filter stack spectrometer data

Here, we present an inversion method capable of robustly unfolding MeV x-ray spectra from filter stack spectrometer (FSS) data without requiring an a priori specification of a spectral shape or arbitrary termination of the algorithm. Our inversion method is based upon the perturbative minimization (PM) algorithm, which has previously been shown to be capable of unfolding x-ray transmission data, albeit for a limited regime in which the x-ray mass attenuation coefficient of the filter material increases monotonically with x-ray energy. Our inversion method improves upon the PM algorithm through regular smoothing of the candidate spectrum and by adding stochasticity to the search. With these additions, the inversion method does not require a physics model for an initial guess, fitting, or user-selected termination of the search. Instead, the only assumption made by the inversion method is that the x-ray spectrum should be near a smooth curve. Testing with synthetic data shows that the inversion method can successfully recover the primary large-scale features of MeV x-ray spectra, including the number of x-rays in energy bins of several-MeV widths to within 10%. Fine-scale features, however, are more difficult to recover accurately. Examples of unfolding experimental FSS data obtained at the Texas Petawatt Laser Facility and the OMEGA EP laser facility are also presented.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

Indoor air quality in California homes with code-required mechanical ventilation

Data were collected in 70 detached houses built in 2011-2017 in compliance with the mechanical ventilation requirements of California's building energy efficiency standards. Each home was monitored for a 1-week period with windows closed and the central mechanical ventilation system operating. Pollutant measurements included time-resolved fine particulate matter (PM 2.5 ) indoors and outdoors and formaldehyde and carbon dioxide (CO 2 ) indoors. Time-integrated measurements were made for formaldehyde, NO 2 , and nitrogen oxides (NO X ) indoors and outdoors. Operation of the cooktop, range hood, and other exhaust fans was continuously recorded during the monitoring period. Onetime diagnostic measurements included mechanical airflows and envelope and duct system air leakage. All homes met or were very close to meeting the ventilation requirements. On average, the dwelling unit ventilation fan moved 50% more airflow than the minimum requirement. Pollutant concentrations were similar to or lower than those reported in a 2006-2007 study of California new homes built in 2002-2005. Mean and median indoor concentrations were lower by 44% and 38% for formaldehyde and 44% and 54% for PM 2.5 . Ventilation fans were operating in only 26% of homes when first visited, and the control switches in many homes did not have informative labels as required by building standards.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

Scalable algorithms for physics-informed neural and graph networks

Physics-informed machine learning (PIML) has emerged as a promising new approach for simulating complex physical and biological systems that are governed by complex multiscale processes for which some data are also available. In some instances, the objective is to discover part of the hidden physics from the available data, and PIML has been shown to be particularly effective for such problems for which conventional methods may fail. Unlike commercial machine learning where training of deep neural networks requires big data, in PIML big data are not available. Instead, we can train such networks from additional information obtained by employing the physical laws and evaluating them at random points in the space–time domain. Such PIML integrates multimodality and multifidelity data with mathematical models, and implements them using neural networks or graph networks. Here, we review some of the prevailing trends in embedding physics into machine learning, using physics-informed neural networks (PINNs) based primarily on feed-forward neural networks and automatic differentiation. For more complex systems or systems of systems and unstructured data, graph neural networks (GNNs) present some distinct advantages, and here we review how physics-informed learning can be accomplished with GNNs based on graph exterior calculus to construct differential operators; we refer to these architectures as physics-informed graph networks (PIGNs). We present representative examples for both forward and inverse problems and discuss what advances are needed to scale up PINNs, PIGNs and more broadly GNNs for large-scale engineering problems.

42 ENGINEERING↗
DOE OSTI2020

Cryogenic C-band wavelength division multiplexing system using an AIM Photonics Foundry process design kit

Cryogenic environments make superconducting computing possible by reducing thermal noise, electrical resistance and heat dissipation. Heat generated by the electronics and thermal conductivity of electrical transmission lines to the outside world constitute two main sources of thermal load in such systems. As a result, higher data rates require additional transmission lines which come at an increasingly higher cooling power cost. Hybrid or monolithic integration of silicon photonics with the electronics can be the key to higher data rates and lower power costs in these systems. We present a 4-channel wavelength division multiplexing photonic integrated circuit (PIC) built from modulators in the AIM Photonics process development kit (PDK) that operate at 25 Gbps at room temperature and 10 Gbps at 40 K. We further demonstrate 2-channel operation for 20 Gbps aggregate data rate at 40 K using two different modulators/wavelengths, with the potential for higher aggregate bit rates by utilizing additional channels.

42 ENGINEERING↗