Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “scientific data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 595 records · Page 33
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

X-Ray Imaging and Spectroscopy Mission

The X-Ray Imaging and Spectroscopy Mission (XRISM) is a joint mission between the Japan Aerospace Exploration Agency (JAXA) and the National Aeronautics and Space Administration (NASA) in collaboration with the European Space Agency (ESA). In addition to the three space agencies, universities and research institutes from Japan, North America, and Europe have joined to contribute to developing satellite and onboard instruments, data-processing software, and the scientific observation program. XRISM is the successor to the ASTRO-H (Hitomi) mission, which ended prematurely in 2016. Its primary science goal is to examine astrophysical problems with precise, high-resolution X-ray spectroscopy. XRISM promises to discover new horizons in X-ray astronomy. It carries a 6 × 6 pixelized X-ray microcalorimeter on the focal plane of an X-ray mirror assembly (Resolve) and a co-aligned X-ray CCD camera (Xtend) that covers the same energy band over a large field of view. XRISM utilizes the Hitomi heritage, but all designs were reviewed. The attitude and orbit control system was improved in hardware and software. The spacecraft was launched from the JAXA Tanegashima Space Center on 2023 September 6 (UTC). During the in-orbit commissioning phase, the onboard components were activated. Although the gate valve protecting the Resolve sensor with a thin beryllium X-ray entrance window was not yet opened, scientific observation started in 2024 February with the planned performance verification observation program. The nominal observation program commenced with the following guest observation program beginning in 2024 September.

Astronomy and AstroPhysics↗
DOE OSTI2025

A Tale from the Trenches: Applying Metamorphic and Differential Testing to Bioinformatics Software

Metamorphic and differential testing have been proposed as best practices for testing software that is difficult to test, such as for programs in scientific domains. An assumption is that these approaches can be easily customized and applied to almost any domain. However, scientific software is often data-driven, and metamorphic relations may require significant domain knowledge to develop. In addition, tools are often written for ad-hoc experimentation by the scientists and often embed many assumptions about the importance and representation of different natural phenomena. In this paper, we present our experience applying both metamorphic and differential testing to a set of four computational biology tools that predict the growth of an organism. While our original goal was to evaluate these techniques to improve our system-level testing, we encountered multiple roadblocks along the way. Although we did find faults (some confirmed by developers), we also uncovered a set of challenges, including the considerable manual effort required for (a) defining domain-specific tests, (b) validating correctness, and (c) distinguishing between issues stemming from poor data and those arising from incorrect software.

Marsh, Alexis L [Iowa State University/Ames Labora↗
DOE OSTI2021

A High Performance Sparse Tensor Algebra Compiler in MLIR

Sparse tensor algebra is widely used in many applications, including scientific computing, machine learning, and data analytics. The performance of sparse tensor algebra kernels strongly depends on the intrinsic characteristics of the input tensors, hence many storage formats are designed for tensors to achieve optimal performance for particular applications/architectures, which makes it challenging to implement and optimize every tensor operation of interest on a given architecture. We propose a tensor algebra domain-specific language (DSL) and compiler framework to automatically generate kernels for mixed sparse-dense tensor algebra operations. The proposed DSL provides high-level programming abstractions that resemble the familiar Einstein notation to represent tensor algebra operations. The compiler introduces a new Sparse Tensor Algebra dialect built on top of LLVM's extensible MLIR compiler infrastructure for efficient code generation while covering a wide range of tensor storage formats. Our compiler also leverages input-dependent code optimization to enhance data locality for better performance. Our results show that the performance of automatically generated kernels outperforms the state-of-the-art sparse tensor algebra compiler, with up to 20.92x, 6.39x, and 13.9x performance improvement over state-of-the-art tensor algebra compilers, for parallel SpMV, SpMM, and TTM, respectively.

Tian, Ruiqin↗
DOE OSTI2020

Consistent Space Runtime (CSPACER) v1.0

This software provides a runtime for communication-bound applications, especially with irregular communication patterns. The target programming abstraction for the runtime is the space consistency model, which defines consistency guarantees at the granularity of memory spaces. This model has relaxed consistency semantics that enables a wide range of runtime optimizations. The runtime leverages threading to accelerate communication primitives, especially collective operations. It also allows efficient pipelining of communication operations and enable constructing a consistent state of multiple unordered communication activities targeting a memory space. The runtime uses a reduced API design that decomposes complex communication primitives in traditional general-purpose runtime into a sequence of simpler steps. To improve the productivity of using this runtime, we provide communication patterns commonly used for regular scientific computing applications and irregular data analytics. These communication patterns offer skeletons for the integration with application computation to allow efficient overlap.

Ibrahim, Khaled↗
DOE OSTI2020

TRACER Overview 200220 [Slides]

The Atmospheric Radiation Measurement (ARM) User Facility is a U.S. Department of Energy Office of Science asset to be used by scientists to further our scientific knowledge about the atmosphere. The DOE ARM program has maintained facilities for use by the scientific community for 25 years. Data is freely available at www.arm.gov. The ARM science objective is to provide a detailed and accurate description of the Earth’s atmosphere to resolve uncertainties in earth system models. This report provides information on TRACER: TRacking Aerosol Convection Interations Experiment.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

A comprehensive and synthetic dataset for global, regional, and national greenhouse gas emissions by sector 1970–2018 with an extension to 2019

To track progress towards keeping global warming well below 2 °C or even 1.5 °C, as agreed in the Paris Agreement, comprehensive up-to-date and reliable information on anthropogenic emissions and removals of greenhouse gas (GHG) emissions is required. Here we compile a new synthetic dataset on anthropogenic GHG emissions for 1970–2018 with a fast-track extension to 2019. Our dataset is global in coverage and includes CO 2 emissions, CH 4 emissions, N 2 O emissions, as well as those from fluorinated gases (F-gases: HFCs, PFCs, SF 6 , NF 3 ) and provides country and sector details. We build this dataset from the version 6 release of the Emissions Database for Global Atmospheric Research (EDGAR v6) and three bookkeeping models for CO 2 emissions from land use, land-use change, and forestry (LULUCF). We assess the uncertainties of global greenhouse gases at the 90 % confidence interval (5th–95th percentile range) by combining statistical analysis and comparisons of global emissions inventories and top-down atmospheric measurements with an expert judgement informed by the relevant scientific literature. We identify important data gaps for F-gas emissions. The agreement between our bottom-up inventory estimates and top-down atmospheric-based emissions estimates is relatively close for some F-gas species (~ 10 % or less), but estimates can differ by an order of magnitude or more for others. Our aggregated F-gas estimate is about 10 % lower than top-down estimates in recent years. However, emissions from excluded F-gas species such as chlorofluorocarbons (CFCs) or hydrochlorofluorocarbons (HCFCs) are cumulatively larger than the sum of the reported species. Using global warming potential values with a 100-year time horizon from the Sixth Assessment Report by the Intergovernmental Panel on Climate Change (IPCC), global GHG emissions in 2018 amounted to 58 ± 6.1 GtCO 2 eq. consisting of CO 2 from fossil fuel combustion and industry (FFI) 38 ± 3.0 GtCO 2 , CO 2 -LULUCF 5.7 ± 4.0 GtCO 2 , CH 4 10 ± 3.1 GtCO 2 eq., N2O 2.6 ± 1.6 GtCO 2 eq., and F-gases 1.3 ± 0.40 GtCO 2 eq. Initial estimates suggest further growth of 1.3 GtCO 2 eq. in GHG emissions to reach 59 ± 6.6 GtCO 2 eq. by 2019. Our analysis of global trends in anthropogenic GHG emissions over the past 5 decades (1970–2018) highlights a pattern of varied but sustained emissions growth. There is high confidence that global anthropogenic GHG emissions have increased every decade, and emissions growth has been persistent across the different (groups of) gases. There is also high confidence that global anthropogenic GHG emissions levels were higher in 2009–2018 than in any previous decade and that GHG emissions levels grew throughout the most recent decade. While the average annual GHG emissions growth rate slowed between 2009 and 2018 (1.2 % yr –1 ) compared to 2000–2009 (2.4 % yr –1 ), the absolute increase in average annual GHG emissions by decade was never larger than between 2000–2009 and 2009–2018. Our analysis further reveals that there are no global sectors that show sustained reductions in GHG emissions. There are a number of countries that have reduced GHG emissions over the past decade, but these reductions are comparatively modest and outgrown by much larger emissions growth in some developing countries such as China, India, and Indonesia. There is a need to further develop independent, robust, and timely emissions estimates across all gases. As such, tracking progress in climate policy requires substantial investments in independent GHG emissions accounting and monitoring as well as in national and international statistical infrastructures. The data associated with this article (Minx et al., 2021) can be found at https://doi.org/10.5281/zenodo.5566761.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Enabling discovery data science through cross-facility workflows

Experimental and observational instruments for scientific research (such as light sources, genome sequencers, accelerators, telescopes and electron microscopes) increasingly require High Performance Computing (HPC) scale capabilities for data analysis and workflow processing. Next-generation instruments are being deployed with higher resolutions and faster data capture rates, creating a big data crunch that cannot be handled by modest institutional computing resources. Often these big data analysis pipelines also require near real-time computing and have higher resilience requirements than the simulation and modeling workloads more traditionally seen at HPC centers. While some facilities have enabled workflows to run at a single HPC facility, there is a growing need to integrate capabilities across HPC facilities to enable cross-facility workflows, either to provide resilience to an experiment, increase analysis throughput capabilities, or to better match a workflow to a particular architecture. In this paper we describe the barriers to executing complex data analysis workflows across HPC facilities and propose an architectural design pattern for enabling scientific discovery using cross-facility workflows that includes orchestration services, application programming interfaces (APIs), data access and co-scheduling.

Antypas, Katerina B.↗
DOE OSTI2026

Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics

Context: Scientific open-source software (SciOSS) plays a foundational role in research and engineering, yet its long-term sustainability has often been overlooked and remains a significant concern. Objective: This study investigates the long-term sustainability of SciOSS through code and test quality metrics. Method: We analyze CASS Software Portfolio projects, classifying them by sustainability and comparing their code structure, test coverage, and links between code quality and testing across the dataset. Results: Sustainable projects show higher, more consistent test coverage and clearer code-test correlations, while unsustainable ones show weaker patterns. Overall, test coverage is low in scientific software, and high complexity and coupling reduce testability. Conclusion: In this study, we present a practical, data-driven approach for assessing sustainability in scientific software, offering a foundation for evaluating long-term software health and supporting future efforts in quality assurance and sustainability monitoring.

Md mushfiqur rahman, Sheikh [University of Tenness↗
DOE OSTI2020

An Integrated Indexing and Search Service for Distributed File Systems

Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this article, we present TagIt, a scalable data management service framework aimed at scientific datasets, which can be integrated into prevalent distributed file system architectures. A key feature of TagIt is a scalable, distributed metadata indexing framework, which facilitates a flexible tagging capability to support data discovery. Furthermore, the tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. We have integrated TagIt into two popular distributed file systems, i.e., GlusterFS and CephFS. Our evaluation demonstrates that TagIt can expedite data search operation by up to 10× over the extant decoupled approach.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗
DOE OSTI2025

Basic Research Needs for Inverse Methods for Complex Systems under Uncertainty

Inverse problems, which aim to infer unknown properties of a system using experimental and observational data, are central to addressing many of the U.S. Department of Energy’s (DOE) most critical scientific and engineering challenges. Accurate, computationally efficient, and data-efficient solutions to inverse problems are essential for advancing DOE mission-critical science drivers, including analyzing data from large-scale experimental facilities, optimizing fusion reactor performance, accelerating materials discovery, enhancing geophysical imaging, improving wildfire predictions, and enabling autonomous systems and digital twins. However, these problems are becoming increasingly complex, often involving nonlinear, highdimensional, and interconnected systems and models that span multiple physics and scales, while relying on data with varying quantity, quality, and information content. Compounding these challenges is the uncertainty inherent in DOE-relevant systems, where errors in inputs, noise in data, incompleteness of data, and discrepancies between models and reality constrain the accuracy and precision of solutions. At the same time, the convergence of recent scientific computing trends—scientific machine learning, artificial intelligence, and computing advances such as exascale computing—is creating unprecedented opportunities for tackling these challenges. The cross-cutting nature of inverse problems, combined with their growing complexity and rapidly evolving data and algorithmic demands, strongly motivates the formulation of a prioritized research agenda to maximize their capabilities and impact. In response to this need, DOE’s Advanced Scientific Computing Research (ASCR) program in the Office of Science convened the Workshop on Basic Research Needs for Inverse Problems for Complex Systems Under Uncertainty in June 2025. This workshop brought together experts across disciplines to identify grand challenges and major opportunities in the field. Through collaborative discussions, the workshop defined transformative research directions aimed at addressing the mathematical, statistical, and computational challenges posed by inverse problems under uncertainty. As a result of these efforts, four priority research directions (PRDs) were identified to guide future research and development in this area. These PRDs, summarized below, represent a roadmap for advancing the foundational science and mathematics of inverse problems, enabling robust, scalable, and uncertainty-aware solutions that are critical for DOE applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Veridical data science

Building and expanding on principles of statistics, machine learning, and scientific inquiry, we propose the predictability, computability, and stability (PCS) framework for veridical data science. Our framework, composed of both a workflow and documentation, aims to provide responsible, reliable, reproducible, and transparent results across the data science life cycle. The PCS workflow uses predictability as a reality check and considers the importance of computation in data collection/storage and algorithm design. It augments predictability and computability with an overarching stability principle. Stability expands on statistical uncertainty considerations to assess how human judgment calls impact data results through data and model/algorithm perturbations. As part of the PCS workflow, we develop PCS inference procedures, namely PCS perturbation intervals and PCS hypothesis testing, to investigate the stability of data results relative to problem formulation, data cleaning, modeling decisions, and interpretations. We illustrate PCS inference through neuroscience and genomics projects of our own and others. Moreover, we demonstrate its favorable performance over existing methods in terms of receiver operating characteristic (ROC) curves in high-dimensional, sparse linear model simulations, including a wide range of misspecified models. Finally, we propose PCS documentation based on R Markdown or Jupyter Notebook, with publicly available, reproducible codes and narratives to back up human choices made throughout an analysis. The PCS workflow and documentation are demonstrated in a genomics case study available on Zenodo.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Final Targeting Strategy for the Sloan Digital Sky Survey IV Apache Point Observatory Galactic Evolution Experiment 2 North Survey

The Apache Point Observatory Galactic Evolution Experiment 2 (APOGEE-2) is a dual-hemisphere, near-infrared (NIR), spectroscopic survey with the goal of producing a chemodynamical mapping of the Milky Way. The targeting for APOGEE-2 is complex and has evolved with time. In this paper, we present the updates and additions to the initial targeting strategy for APOGEE-2N presented in Zasowski et al. (2017). These modifications come in two implementation modes: (i) “Ancillary Science Programs” competitively awarded to Sloan Digital Sky Survey IV PIs through proposal calls in 2015 and 2017 for the pursuit of new scientific avenues outside the main survey, and (ii) an effective 1.5 yr expansion of the survey, known as the Bright Time Extension (BTX), made possible through accrued efficiency gains over the first years of the APOGEE-2N project. For the 23 distinct ancillary programs, we provide descriptions of the scientific aims, target selection, and how to identify these targets within the APOGEE-2 sample. The BTX permitted changes to the main survey strategy, the inclusion of new programs in response to scientific discoveries or to exploit major new data sets not available at the outset of the survey design, and expansions of existing programs to enhance their scientific success and reach. After describing the motivations, implementation, and assessment of these programs, we also leave a summary of lessons learned from nearly a decade of APOGEE-1 and APOGEE-2 survey operations. Finally, a companion paper, F. Santana et al. (submitted; AAS29036), provides a complementary presentation of targeting modifications relevant to APOGEE-2 operations in the Southern Hemisphere.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific ap- plications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Net- works (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate band- widths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer perfor- mance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application- level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher pre- diction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗
DOE OSTI2020

On Performance Prediction of Big Data Transfer in High-performance Networks

Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.

Liu, Wuji↗
DOE OSTI2024

HERMES

HERMES - High-speed Event Retrieval and Management for Enhanced Spectral imaging code. This code is meant to unpack and process neutron imaging data from the TPX3Cam made by Amsterdam Scientific Instruments. The TPX3Cam utilizes the Timepix3 chip in a single photon counting mode of image acquisition. The photon counting data streaming off the TPX3Cam needs to be process and analyzed in order to create final images. Therefore we are developing both python and cpp codes to allow for users of the TPX3Cam to efficiently analyze data and create images.

Long, Alexander↗
DOE OSTI2020

Flow-based likelihoods for non-Gaussian inference

We investigate the use of data-driven likelihoods to bypass a key assumption made in many scientific analyses, which is that the true likelihood of the data is Gaussian. In particular, we suggest using the optimization targets of flow-based generative models, a class of models that can capture complex distributions by transforming a simple base distribution through layers of nonlinearities. We call these flow-based likelihoods (FBL). We analyze the accuracy and precision of the reconstructed likelihoods on mock Gaussian data, and show that simply gauging the quality of samples drawn from the trained model is not a sufficient indicator that the true likelihood has been learned. We nevertheless demonstrate that the likelihood can be reconstructed to a precision equal to that of sampling error due to a finite sample size. We then apply FBLs to mock weak lensing convergence power spectra, a cosmological observable that is significantly non-Gaussian (NG). We find that the FBL captures the NG signatures in the data extremely well, while other commonly used data-driven likelihoods, such as Gaussian mixture models and independent component analysis, fail to do so. This suggests that works that have found small posterior shifts in NG data with data-driven likelihoods such as these could be underestimating the impact of non-Gaussianity in parameter constraints. By introducing a suite of tests that can capture different levels of NG in the data, we show that the success or failure of traditional data-driven likelihoods can be tied back to the structure of the NG in the data. Here, unlike other methods, the flexibility of the FBL makes it successful at tackling different types of NG simultaneously. Because of this, and consequently their likely applicability across datasets and domains, we encourage their use for inference when sufficient mock data are available for training.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. As such, parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new applications, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performing parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries.

97 MATHEMATICS AND COMPUTING↗