Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “scientific data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,189 records · Page 66
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

The Artificial Scientist: in-Transit Machine Learning of Plasma Simulations

Large-scale simulations or scientific experiments produce petabytes of data per run. This poses massive challenges for I/O and storage when scientific analysis workflows are run manually offline. Unsupervised deep learning-based techniques to extract patterns and non-linear relations from these large amounts of data provide a way to build scientific understanding from raw data, reducing the need for manual pre-selection of analysis steps, but require exascale compute and memory to process the full dataset available. In this paper, we demonstrate a heterogeneous streaming workflow in which plasma simulation data is streamed directly to a Machine Learning (ML) application training a model on the simulation data in-transit, completely circumventing the capacity-constrained filesystem bottleneck. This workflow employs openPMD to provide a high level interface to describe scientific data and also uses ADIOS2, to transfer volumes of data that exceed the capabilities of the filesystem. We employ experience replay to avoid catastrophic forgetting in learning from this non-steady state process in a continual manner and adapt it to improve model convergence while learning in-transit. As a proof-of-concept, we approach the ill-posed inverse problem of predicting particle dynamics from radiation in a particle-incell (PIConGPU) simulation of the Kelvin-Helmholtz instability (KHI). We detail hardware-software co-design challenges as we scale PIConGPU to full Frontier, the Top-1 system as of June 2024 Top500 list.

Kelling, Jeffrey [Helmholtz-Zentrum Dresden Rossen↗
NASA NTRS1999

Telescience Operations on International Space Station

This paper describes the concept of telescience operations for the International Space Station (ISS). The extended duration microgravity environment of the ISS will enable microgravity science research to enter into a new era of increased scientific and technological data return. The National Aeronautics and Space Administration (NASA) has a vision of distributed ground operations which enables the Principal Investigator direct interaction with his/her on-board experiment from his/her home location. This is the concept of telescience and is essential for maximizing the use of the long duration science environment that ISS provides. The goal of telescience is to provide the capability to fully tele-operate an experiment from any ground location in such a way as to increase the amount and quality of scientific and technological data return and decrease the operations cost of an individual experiment relative to the era of Space Shuttle experiments. This paper also describes the NASA Lewis Research Center (LeRC) implementation approach for the LeRC Telescience Support Center (TSC) and Principal Investigator Science Operations Sites (SOS) which will fully meet the concept of telescience as prescribed by the Agency.

Schubert, Kathleen E.↗
NASA NTRS2003

Intelligent Systems Technologies to Assist in Utilization of Earth Observation Data

With the launch of several Earth observing satellites over the last decade, we are now in a data rich environment. From NASA's Earth Observing System (EOS) satellites alone, we are accumulating more than 3 TB per day of raw data and derived geophysical parameters. The data products are being distributed to a large user community comprising scientific researchers, educators and operational government agencies. Notable progress has been made in the last decade in facilitating access to data. However, to realize the full potential of the growing archives of valuable scientific data, further progress is necessary in the transformation of data into information, and information into knowledge that can be used in particular applications. Sponsored by NASA s Intelligent Systems Project within the Computing, Information and Communication Technology (CICT) Program, a conceptual architecture study has been conducted to examine ideas to improve data utilization through the addition of intelligence into the archives in the context of an overall knowledge building system. Potential Intelligent Archive concepts include: 1) Mining archived data holdings using Intelligent Data Understanding algorithms to improve metadata to facilitate data access and usability; 2) Building intelligence about transformations on data, information, knowledge, and accompanying services involved in a scientific enterprise; 3) Recognizing the value of results, indexing and formatting them for easy access, and delivering them to concerned individuals; 4) Interacting as a cooperative node in a web of distributed systems to perform knowledge building (i.e., the transformations from data to information to knowledge) instead of just data pipelining; and 5) Being aware of other nodes in the knowledge building system, participating in open systems interfaces and protocols for virtualization, and collaborative interoperability. This paper presents some of these concepts and identifies issues to be addressed by research in future intelligent systems technology.

Ramapriyan, Hampapuram K.↗
NASA NTRS1988

A Mars orbital laser altimeter for rover trafficability: Instrument concept and science potential

Limited information on the types of geologic hazards (boulders, troughs, craters etc.) that will affect rover trafficability on Mars are available for the two Viking Lander sites, and there are no prospects for increasing this knowledge base in the near future. None of the instrument payloads on the upcoming Mars Observer or Soviet PHOBOS missions can directly measure surface obstacles on the scales of concern for rover safety (a few meters). Candidate instruments for the Soviet Mars 92 orbiter/balloon/rover mission such as balloon-borne stereo imaging, rover panoramic imaging, and orbital synthetic aperature imaging (SAR) are under discussion, but data from this mission may not be available for target areas of interest for the U.S. Mars Rover Sample Return (MRSR) mission. In an effort to determine how to directly measure the topography of surface obstacles that could affect rover trafficability on Mars, we are studying how to design a laser altimeter with extremely high spatial and vertical resolution that would be suitable for a future Mars Orbiter spacecraft (MRSR precursor or MRSR orbiter). This report discusses some of the design issues associated with such an instrument, gives examples of laser altimeter data collected for Mars analog terrains on Earth, and outlines the scientific potential of data that could be obtained with the system.

Garvin, J. B.↗
NASA NTRS1989

Impact of low gravity on water electrolysis operation

Advanced space missions will require oxygen and hydrogen utilities for several important operations including the following: (1) propulsion; (2) electrical power generation and storage; (3) environmental control and life support; (4) extravehicular activity; (5) in-space manufacturing and (6) in-space science activities. An experiment suited to a Space Shuttle standard middeck payload has been designed for the Static Feed Water Electrolysis technology which has been viewed as being capable of efficient, reliable oxygen and hydrogen generation with few subsystem components. The program included: end use design requirements, phenomena to be studied, Space Shuttle Orbiter experiment constraints, experiment design and data requirements, and test hardware requirements. The objectives are to obtain scientific and engineering data for future research and development and to focus on demonstrating and monitoring for safety of a standard middeck payload.

Powell, F. T.↗
NASA NTRS1992

Satcom systems and technologies into the 21st century

The role of communications satellites in the domestic U.S. telecommunications market for the next 25 years is examined, the focus being on the new satellite communications systems concepts and their enabling technologies. A simple procedure for the identification of critical satellite technologies in the years 2000-2015 is described. Study results indicate that by the year 2015, satellites will find commercial applications in the distribution of space-gathered data, supercomputer network, scientific visualization to remote areas, mobile services, VSAT, broadband ISDN, and data services.

Kwan, Robert K.↗
DOE OSTI2026

QProR: An Efficient Framework for Quantity-of-Interest Based Progressive Retrieval with Guaranteed Error Control

Scientific applications generate an unprecedented volume of data, overwhelming the network and file systems’ bandwidth and posing challenges for efficient and scalable data retrieval and analysis. Progressive data compression offers a promising solution by enabling on-demand retrieval at reduced size. However, existing progressive methods either fail to bound the errors in essential quantities of interest (QoIs) derived from raw data or suffer from suboptimal retrieval efficiency. In this work, we propose QProR, an efficient QoI-based progressive framework that optimizes progressive retrieval for target QoIs. Our key contributions include: (1) a systematic framework that integrates error-controlled lossy compressors with bitplane encoding while decoupling the two processes for high flexibility and adaptability; (2) a novel weighted bitplane encoding method which incorperates QoI knowledge into data refactoring to enhance retrieval efficiency; (3) an optimized retrieval strategy that accounts for the varying impacts of different variables on multivariate QoIs; (4) comprehensive evaluations using six real-world datasets from multiple scientific applications and thorough comparisons against state of the arts. Experimental results demonstrate that QProR achieves up to 80.38% reduction in the retrieval size under the same requested QoI error tolerance, when compared with the best-performing existing methods. When transferring 384 GB of scientific data to remote sites, QProR delivers up to 1.68 × speedup in the end-to-end data transfer performance.

Li, Wenbo [University of Kentucky]↗
DOE OSTI2026

An Autonomous MCP Bridge to Rucio: Enhancing Data Management Accessibility for High Energy Physics

The Rucio Data Management System [1] is an important tool used by High Energy Physics experiments, including those at Fermi National Accelerator Laboratory, to store and manage exabyte-scale scientific datasets. Despite its central role in coordinating data across globally distributed storage sites, Rucio's command line interface (CLI) presents a steep learning curve, and makes it difficult for scientists to navigate through. To solve this issue, a containerized Model Context Protocol (MCP) [2] server was built that connects Large Language Models directly to Rucio, allowing AI agents to handle data tasks by using simple, natural language rather than memorized terminal commands. The core engineering focus of this project was moving the server away from slow terminal commands that require text parsing and replacing them with a native Python Client API toolset and a planned REST API framework. Moving to the Python API handles data operations directly in memory, which helps clear up formatting errors, provides the AI with clean, structured JSON data and speeds up tool execution. To prove that the system actually works, a benchmarking pipeline was also built with various questions to test the AI across four different model configurations. The questions included finding data scopes, tracking down specific datasets, and checking replication rules. Through benchmarking, early runs showed that with raw terminal text, the model would get confused and stuck, whereas switching to the Python API to feed the AI clean, structured data yielded massive improvement. By creating an intelligent and autonomous bridge to a storage network, this project shows how AI can be implemented in scientific data management, which ultimately helps scientists at Fermilab spend less time sorting through data and more time focusing on their experiments and analysis.

Akella, Kashyap [William Rainey Harper Coll.]↗
DOE OSTI2024

Unpaired image translation to mitigate domain shift in liquid argon time projection chamber detector responses

Deep learning algorithms often are developed and trained on a training dataset and deployed on test datasets. Any systematic difference between the training and a test dataset may severely degrade the final algorithm performance on the test dataset—what is known as the domain shift problem . This issue is prevalent in many scientific domains where algorithms are trained on simulated data but applied to real-world datasets. Typically, the domain shift problem is solved through various domain adaptation (DA) methods. However, these methods are often tailored for a specific downstream task, such as classification or semantic segmentation, and may not easily generalize to different tasks. This work explores the feasibility of using an alternative way to solve the domain shift problem that is not specific to any downstream algorithm. The proposed approach relies on modern Unpaired Image-to-Image (UI2I) translation techniques, designed to find translations between different image domains in a fully unsupervised fashion. In this study, the approach is applied to a domain shift problem commonly encountered in Liquid Argon Time Projection Chamber (LArTPC) detector research when seeking a way to translate samples between two differently distributed LArTPC detector datasets deterministically. This translation allows for mapping real-world data into the simulated data domain where the downstream algorithms can be run with much less domain-shift-related performance degradation. Conversely, using the translation from the simulated data to a real-world domain can increase the realism of the simulated dataset and reduce the magnitude of any systematic uncertainties. To evaluate the quality of the translations, we use both pixel-wise metrics and a downstream task to measure the effectiveness of UI2I methods for mitigating the domain shift problem. We adapted several popular UI2I translation algorithms to work on scientific data and demonstrated the viability of these techniques for solving the domain shift problem with LArTPC detector data. To facilitate further development of DA techniques for scientific datasets, the ‘Simple Liquid-Argon Track Samples’ dataset used in this study is also published.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2000

Corongraphic Observations and Analyses of The Ultraviolet Solar Corona

The activities supported under NASA Grant NAG5-613 included the following: 1) reduction and scientific analysis of data from three sounding rocket flights of the Rocket Ultraviolet Coronagraph Spectrometer, 2) development of ultraviolet spectroscopic diagnostic techniques to provide a detailed empirical description of the extended solar corona, 3) extensive upgrade of the rocket instrument to become the Ultraviolet Coronal Spectrometer (UVCS) for Spartan 201,4) instrument scientific calibration and characterization, 5) observation planning and mission support for a series of five Spartan 201 missions (fully successful except for STS 87 where the Spartan spacecraft was not successfully deployed and the instruments were not activated), and 6) reduction and scientific analysis of the UVCS/Spartan 201 observational data. The Ultraviolet Coronal Spectrometer for Spartan 201 was one unit of a joint payload and the other unit was a White Light Coronagraph (WLC) provided by the High Altitude Observatory and the Goddard Space Flight Center. The two instruments were used in concert to determine plasma parameters describing structures in the extended solar corona. They provided data that could be used individually or jointly in scientific analyses. The WLC provided electron column densities in high spatial resolution and high time resolution. UVCS/Spartan provided hydrogen velocity distributions, and line of sight hydrogen velocities. The hydrogen intensities from UVCS together with the electron densities from WLC were used to determine hydrogen outflow velocities. The UVCS also provided O VI intensities which were used to develop diagnostics for velocity distributions and outflow velocities of minor ions.

Kohl, John L.↗
DOE OSTI2020

Scalable Data-Intensive Geocomputation: A Design for Real-Time Continental Flood Inundation Mapping

The convergence of data-intensive and extreme-scale computing enables an integrated software and data ecosystem for scientific discovery. Developments in this realm will fuel transformative research in data-driven interdisciplinary domains. Geocomputation provides computing paradigms in Geographic Information Systems (GIS) for interactive computing of geographic data, processes, models, and maps. Because GIS is data-driven, the computational scalability of a geocomputation workflow is directly related to the scale of the GIS data layers, their resolution and extent, as well as the velocity of the geo-located data streams to be processed. Unique in high user interactivity and low end-to-end latency requirements, geocomputation applications will dramatically benefit from the convergence of high-end data analytics (HDA) and high-performance computing (HPC). The application level challenge, however, is to identify and eliminate computational bottlenecks that arise along a geocomputation workflow. Indeed, poor scalability at any of the workflow components is detrimental to the entire end-to-end pipeline. Here, we study a large geocomputation use case in flood inundation mapping that handles multiple national-scale geospatial datasets and targets low end-to-end latency. We discuss benefits and challenges for harnessing both HDA and HPC for data-intensive geospatial data processing and intensive numerical modeling of geographic processes. We propose an HDA+HPC geocomputation architecture design that couples HDA (e.g., Spark)-based spatial data handling and HPC-based parallel data modeling. Key techniques for coupling HDA and HPC to bridge the two different software stacks are reviewed and discussed.

Liu, Yan↗
DOE OSTI2023

A Data Deposition Platform for Sharing Nuclear Magnetic Resonance Data

Nuclear magnetic resonance (NMR) data are rarely deposited in open databases, leading to loss of critical scientific knowledge. Existing data reporting methods (images, tables, lists of values) contain less information than raw data, and are poorly standardized. Together, these issues limit FAIR (findable, accessible, interoperable, reusable) access to these data, which in turn creates barriers for compound dereplication and the development of new data-driven discovery tools. Existing NMR databases are either not designed for natural products data, or employ complex deposition interfaces that disincentivize deposition. Journals, including the Journal of Natural Products (JNP), are now requiring data submission as part of the publication process, creating the need for a streamlined, user-friendly mechanism to deposit and distribute NMR data. Recently, our team reported the development of the Natural Products Magnetic Resonance Database (NP-MRD; www.np-mrd.org). Here in this paper we present a new data deposition platform for the NP-MRD project that is designed to enable users to deposit NMR data for published or submitted manuscripts in under five minutes. This platform includes a suite of automated data extraction and standardization tools, together with a simple-to-use web-based interface and detailed error reporting to simplify the data deposition process and is available at www.np-mrd.org/submissions.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗
DOE OSTI2021

Hyper Suprime-Cam Legacy Archive

Abstract We present the launch of the Hyper Suprime-Cam Legacy Archive (HSCLA), a public archive of processed, science-ready data from the Hyper Suprime-Cam (HSC). HSC is an optical wide-field imager installed at the prime focus of the Subaru Telescope and has been in operation since 2014. While ∼1/3 of the total observing time of HSC has been used for the Subaru Strategic Program (SSP), the remainder of the time is used for Principal Investigator (PI)-based programs. We have processed the data from these PI-based programs and make the processed, high-quality data available to the community through HSCLA. The current version of HSCLA includes data taken in the first year of science operation, 2014. We provide both individual and coadd images as well as photometric catalogs. The photometric catalog from the coadd is loaded to the database, which offers a fast access to the large catalog. There are other online tools such as an image browser and an image cutout tool and they will be useful for science analyses. The coadd images reach 24–27th magnitudes at 5σ for point sources and cover approximately 580 square degrees in at least one filter with 150 million objects in total. We perform extensive quality assurance tests and verify that the photometric and astrometric quality of the data is good enough for most scientific explorations. However, the data are not without problems and users are referred to the list of known issues before exploiting the data for science. All the data and documentations can be found at the data release site, 〈https://hscla.mtk.nao.ac.jp/〉.

Tanaka, Masayuki↗
DOE OSTI2026

Discovery of Probabilistic Dirichlet-to-Neumann Maps on Graphs

Dirichlet-to-Neumann maps enable the coupling of multiphysics simulations across computational subdomains by ensuring continuity of state variables and fluxes at artificial interfaces. We present a novel method for learning Dirichlet-to-Neumann maps on graphs using Gaussian processes, specifically for problems where the data obey a conservation law arising from an underlying partial differential equation. Our approach combines discrete exterior calculus and nonlinear optimal recovery to infer relationships between vertex and edge values. This framework yields data-driven predictions with uncertainty quantification across the entire graph, even when observations are limited to a subset of vertices and edges. By minimizing the reproducing kernel Hilbert space norm while penalizing kernel complexity through maximum likelihood estimation, our method ensures that the resulting surrogate strictly enforces conservation laws without overfitting. We demonstrate our method on two representative applications: subsurface flow in fracture networks and arterial blood flow. Finally, the results demonstrate that the method maintains high accuracy and well-calibrated uncertainty estimates even under severe data scarcity, highlighting its potential for scientific applications where limited data and reliable uncertainty quantification are critical.

Dirichlet-to-Neumann map↗
DOE OSTI2024

Data associated with the manuscript “Radiative impact of record-breaking wildfires from integrated ground-based data” collected in Richland, Washington in September 2020

This data package is associated with the publication “Radiative impact of record-breaking wildfires from integrated ground-based data” submitted to Nature Scientific Reports (Kassianov et al., 2024). Data from ground-based measurements of shortwave and spectrally resolved irradiance and aerosol optical depth (AOD) in the visible and near-infrared spectral ranges were assessed to quantify the radiative impact of the September 2020 wildfires that occurred in the Western United States. Data were collected in September 2020 by several ground-based instruments at the Atmospheric Measurements Laboratory (AML) located in Richland, Washington (46.3451, -119.2792). These data include (1) Aerosol Optical Depth (AOD); (2) spectrally resolved and shortwave (SW) irradiances; (3) backscatter profiles; (4) total sky images; and (5) near-surface ambient air temperatures.The data package consists of five sub-directories: (1) “AML_Ceilometer_”; (2)” AML_CSPHOT_”; (3) “AML_MFRSR_irradiances_”; (4) “AML_SW_irradiances_and_Temp_”; (5) “AML_TSI_images_”; and 6 files stored at the directory level, including the readme, file-level metadata file, and data dictionary. The file-level metadata file (the file ending in “_flmd.csv”) lists all files contained in this data package and descriptions for each. The data dictionary (the file ending in “_dd.csv”) describes each tabular column header’s unit, definition, and structure. Below are descriptions of each sub-directory:“AML_Ceilometer_” includes ceilometer data collected at the AML. These files contain the corresponding narratives of data. Details related to the ceilometer data can be found in Morris (2016). “AML_CSPHOT_” includes ascii files with high-temporal resolution (about 10-15 min) AML CSPHOT data and their daily-averaged counterparts. These two files contain the corresponding narratives of data. Details related to the CSPHOT data can be found in Gregory (2011). “AML_MFRSR_irradiances_” includes ascii files with the AML MFRSR-measured diffuse, normal, and total spectrally resolved irradiance. Details related to the MFRSR data can be found in Hodges and Michalsky (2016) and Koontz et al. (2013). “AML_SW_irradiances_+_Temp_” includes near-surface ambient air temperature and SW irradiances, namely direct normal, diffuse hemispherical, and total hemispheric (global), measured at the AML. These files also incorporate the corresponding narratives of data. Details related to the SW irradiances can be found in Andreas et al. (2018). “AML_TSI_images_” includes Total Sky Images (TSIs) collected at the AML. Details related to the TSI data can be found in Morris (2005).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Data-Driven Supervised Dimension Reduction for Scientific Discovery (LDRD QTI Report)

This report summarizes the findings of a four months FY24 Advanced Science & Technology (AS&T) LDRD Quick Targeted Investigation (QTI) project focused on the exploration of supervised dimension reduction approaches based on autoencoders. Autoencoders have been extensively employed in literature for unsupervised learning tasks, however, their use for supervised regression tasks, which are common within scientific applications, has been limited. Motivated by linear dimension reduction strategies like Active Subspaces and Adaptive Basis, we explored the possibility of employing autoencoders to discover a non-linear manifold able to represent the original function in fewer dimensions. In this report, we discuss a neural network architecture and we perform a numerical campaign on several problems ranging from simple two-dimensional functions to a model problem for magnetohydrodynamics in five dimensions. In our preliminary results, we show that the proposed approach is found to be superior to linear dimension reduction strategies in representing the target function even with a single latent variable.

97 MATHEMATICS AND COMPUTING↗