Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Open data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 901 records · Page 50
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

In-Water Data Acquisition Tool Supports Four Marine Energy Projects

The NREL-developed Modular Ocean Data Acquisition (MODAQ) system is built from open-source hardware and software. The tool can both collect and store data and even share curated information through the cloud. Marine energy developers can work with NREL's engineers to design their own customized MODAQ and capture high-quality field data to help monitor and improve their technology designs. For example, users could assess how much power their device produces at sea, analyze the durability of a specific device component, or even control their device from a desk halfway around the world.

data acquisition↗
DOE OSTI2018

CMS Analysis and Data Reduction with Apache Spark

Experimental Particle Physics has been at the forefront of analyzing the world's largest datasets for decades. The HEP community was among the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems for distributed data processing, collectively called "Big Data" technologies have emerged from industry and open source projects to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and tools, promising a fresh look at analysis of very large datasets that could potentially reduce the time-to-physics with increased interactivity. Moreover these new tools are typically actively developed by large communities, often profiting of industry resources, and under open source licensing. These factors result in a boost for adoption and maturity of the tools and for the communities supporting them, at the same time helping in reducing the cost of ownership for the end-users. In this talk, we are presenting studies of using Apache Spark for end user data analysis. We are studying the HEP analysis workflow separated into two thrusts: the reduction of centrally produced experiment datasets and the end-analysis up to the publication plot. Studying the first thrust, CMS is working together with CERN openlab and Intel on the CMS Big Data Reduction Facility. The goal is to reduce 1 PB of official CMS data to 1 TB of ntuple output for analysis. We are presenting the progress of this 2-year project with first results of scaling up Spark-based HEP analysis. Studying the second thrust, we are presenting studies on using Apache Spark for a CMS Dark Matter physics search, comparing Spark's feasibility, usability and performance to the ROOT-based analysis.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

Harnessing large language models’ zero-shot and few-shot learning capabilities for regulatory research

Abstract Large language models (LLMs) are sophisticated AI-driven models trained on vast sources of natural language data. They are adept at generating responses that closely mimic human conversational patterns. One of the most notable examples is OpenAI's ChatGPT, which has been extensively used across diverse sectors. Despite their flexibility, a significant challenge arises as most users must transmit their data to the servers of companies operating these models. Utilizing ChatGPT or similar models online may inadvertently expose sensitive information to the risk of data breaches. Therefore, implementing LLMs that are open source and smaller in scale within a secure local network becomes a crucial step for organizations where ensuring data privacy and protection has the highest priority, such as regulatory agencies. As a feasibility evaluation, we implemented a series of open-source LLMs within a regulatory agency’s local network and assessed their performance on specific tasks involving extracting relevant clinical pharmacology information from regulatory drug labels. Our research shows that some models work well in the context of few- or zero-shot learning, achieving performance comparable, or even better than, neural network models that needed thousands of training samples. One of the models was selected to address a real-world issue of finding intrinsic factors that affect drugs' clinical exposure without any training or fine-tuning. In a dataset of over 700 000 sentences, the model showed a 78.5% accuracy rate. Our work pointed to the possibility of implementing open-source LLMs within a secure local network and using these models to perform various natural language processing tasks when large numbers of training examples are unavailable.

Biochemistry & Molecular Biology↗
DOE OSTI2013

Sluiceway Operations to Pass Juvenile Salmonids at The Dalles Dam, Columbia River, USA

Existing ice and trash sluiceways are commonly used to pass juvenile salmonids downstream at hydropower dams through a benign, non-turbine route. At The Dalles Dam on the Columbia River, managers undertook optimizing operations of sluiceway weirs to maximize survival of juvenile salmonids at the powerhouse. We applied fixed-location hydroacoustic methods to compare fish passage rates and sluiceway efficiencies for two weir configurations during 2004 and 2005: three weirs versus six weirs, located at the mid- versus east powerhouse, respectively. We also analyzed horizontal distributions of passage at the sluiceway and turbines and the effects of operating turbines beneath open sluiceway gates to provide supporting data relevant to operations optimization. Based on the findings, we recommend the following for long-term operations for the sluiceway at The Dalles Dam: open six rather than three sluiceway weirs to take advantage of the maximum hydraulic capacity of the sluiceway; open the three weirs above the western-most operating main turbine unit (MU) and the three weirs at MU 8 where turbine passage rates are relatively high; operate the turbine units below open sluiceway weirs as a standard procedure; operate the sluiceway 24 h/d year-round to maximize its benefits to juvenile salmonids; and use the same operations for spring and summer emigrants. These operational concepts are transferable to dams where sluiceway surface flow outlets are used protect downstream migrating fishes.

juvenile salmonid passage↗
DOE OSTI2022

A Meteorology and Snow Data Set From Adjacent Forested and Meadow Sites at Crested Butte, CO, USA

We present meteorology and snow observation data collected at sites in the southwestern Colorado Rocky Mountains (USA) over three consecutive water years with different amounts of snow water equivalent (SWE) accumulation: A year with above average SWE (2019), a year with average SWE (2020), and a year with below average SWE (2021). This data set is distinguished by its emphasis on paired open-forest sites in a continental snow climate. Approximately once a month during February–May, we collected data from 15 to 20 snow pits and took 8 to 19 snow depth transects. Our sampling sites were in open and adjacent forested areas at 3,100 m and in a lower elevation aspen (3,035 m) and higher elevation conifer stand (3,395 m). In total, we recorded 270 individual snow pit density and temperature profiles and over 4,000 snow depth measurements. These data are complimented by continuous meteorological measurements from two weather stations: One in the open and one in the adjacent forest. Meteorology data—including incoming shortwave and longwave radiation, outgoing shortwave radiation, relative humidity, wind speed, snow depth, and air and infrared surface temperature—were quality controlled and the forcing data were gap-filled. These data are available to download from Bonner, Smyth, et al. at three levels of processing, including a level with downscaled, adjusted precipitation based on data assimilation using observed snow depth and a process-based snow model. We demonstrate the utility of these data with a modeling experiment that explores open-forest differences and identifies opportunities for improvements in model representation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

The DECADE cosmic shear project IV: cosmological constraints from 107 million galaxies across 5,400 deg$^2$ of the sky

We present cosmological constraints from the Dark Energy Camera All Data Everywhere (DECADE) cosmic shear analysis. This work uses shape measurements for 107 million galaxies measured through Dark Energy Camera (DECam) imaging of $5,\!412$ deg$^2$ of sky that is outside the Dark Energy Survey (DES) footprint. We derive constraints on the cosmological parameters $S_8 = 0.791^{+0.027}_{-0.032}$ and $Ω_{\rm m} =0.269^{+0.034}_{-0.050}$ for the $Λ$CDM model, which are consistent with those from other weak lensing surveys and from the cosmic microwave background. We combine our results with cosmic shear results from DES Y3 at the likelihood level, since the two datasets span independent areas on the sky. The combined measurements, which cover $\approx\! 10,\!000$ deg$^2$, prefer $S_8 = 0.791 \pm 0.023$ and $Ω_{\rm m} = 0.277^{+0.034}_{-0.046}$ under the $Λ$CDM model. These results are the culmination of a series of rigorous studies that characterize and validate the DECADE dataset and the associated analysis methodologies (Anbajagane et. al 2025a,b,c). Overall, the DECADE project demonstrates that the cosmic shear analysis methods employed in Stage-III weak lensing surveys can provide robust cosmological constraints for fairly inhomogeneous datasets. This opens the possibility of using data that have been previously categorized as ``unusable'' for cosmic shear analyses, thereby increasing the statistical power of upcoming weak lensing surveys.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Drone Flight Data Logs

This dataset represents the open-air tests for the drones when testing different flight scenarios. For some flights we created and tested with a set of onboard sensors. For others we used the native logs for the drones. We recorded relevant conditions for each of the flights to examine environmental issues and weight impacts. We also looked at segmentations of flights to investigate the energy used in each type of flight.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Revealing complex subsurface dynamics with continuous seismic monitoring: Observations using distributed acoustic sensing and surface orbital vibrators during hydraulic fracturing

Understanding hydraulic fracturing is crucial to improving the stimulation of unconventional reservoirs and increasing fluid production. This study develops a novel seismic monitoring technology using distributed acoustic sensing (DAS) and surface orbital vibrators (SOV) to capture fracture seismic response and mechanical properties at high temporal intervals. We analyze continuous time-lapse vertical seismic profiling (VSP) data acquired every hour during the first nine days of treatment of an unconventional reservoir in the Austin Chalk/Eagle Field Laboratory. The VSP data contain clear seismic signals scattered from the activated fractures. The spatiotemporal changes of the fracture reflectivity revealed by the SOV/DAS data correlate well with the observations of fracture locations inferred from low-frequency DAS data. These results capture the fracture opening and closure processes, as well as highlight potential prestage activations of the fractures due to hydraulic connectivity with preexisting fracture systems. Therefore, analysis of the presented data set provides a unique opportunity to understand fracture initiation and subsequent evolution, not only in the context of unconventional resources but also in enhanced geothermal systems.

Correa, Julia↗
DOE OSTI2025

DOE Repository Metadata Profile (DRMP): A Metadata Framework for Advancing Interoperability and AI Readiness Across Scientific Repositories

The Department of Energy (DOE) funds a diverse and distributed ecosystem of repositories that steward scientific data, publications, and software across its research programs, user facilities, and national laboratories. While significant progress has been made in standardizing dataset-level metadata, the metadata describing repositories themselves (their identity, governance, access interfaces, policies, and technical capabilities) remains inconsistent and fragmented across DOE-funded systems. This variability limits discoverability, interoperability, automated validation, and AI-driven analysis, all of which are increasingly essential for modern scientific workflows. To address this gap, the DOE Data Curation Working Group (DCWG) developed the DOE Repository Metadata Profile (DRMP). The DRMP is a practical, community-driven framework that defines how repositories can describe themselves in a consistent, machine-actionable, and scalable manner. The DRMP is not a new metadata schema. Instead, it is a mapping profile and structured element set capturing the essential characteristics of DOE repositories. It harmonizes repository-level metadata across six widely adopted community schemas: RE3Data; DCAT-US v3; Schema.org; Dublin Core; DataCite 4.6; and PREMIS 3.0. This harmonization eliminates reinvention and enables interoperability within DOE and across the broader scientific ecosystem. A core objective of the DRMP is to reduce burden on repositories by allowing them to reuse their existing metadata through a Rosetta-style crosswalk rather than redesigning local implementations. The profile introduces a three-level conformance model that supports incremental adoption: • Level 1 – Minimum Viable Record (MVR): foundational identification elements required for workflows, project registration, and basic repository presence. • Level 2 – Interoperable: structured metadata enabling alignment with national and international discovery systems. • Level 3 – AI-Ready: enhanced provenance, policy transparency, fixity, semantic context, and capabilities that support automated reasoning, model training governance, and machine-assisted curation. To support implementation, the DRMP includes JSON Schema definitions, OpenAPI patterns, and MCP templates that allow repositories to publish machine-readable metadata directly within existing platforms. These resources are modular and lightweight, enabling adoption without major architectural change. Adopting the DRMP enables repositories to: • Enhance discoverability and interoperability by aligning identifiers, classifications, and descriptive elements across widely used schema standards. • Support federated discovery and cross-registration across DOE systems, Data.gov, and international catalogs. • Enable AI agents and workflow orchestration systems to interpret repository-level metadata within the American Science Cloud (AmSC) through Model Context Protocol (MCP)-based context publication. • Demonstrate alignment with DOE’s open science, stewardship, and FAIR data priorities. This guidance represents a community-driven step forward. Through voluntary adoption and continued feedback, the DRMP advances a cohesive, machine-actionable description of DOE repositories that supports FAIR data practices, preparing the infrastructure for AI-enabled research, and strengthening the discoverability and reuse of DOE’s scientific outputs.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2019

Lipid Mini-On: mining and ontology tool for enrichment analysis of lipidomic data

Abstract Summary Here we introduce Lipid Mini-On, an open-source tool that performs lipid enrichment analyses and visualizations of lipidomics data. Lipid Mini-On uses a text-mining process to bin individual lipid names into multiple lipid ontology groups based on the classification (e.g. LipidMaps) and other characteristics, such as chain length. Lipid Mini-On provides users with the capability to conduct enrichment analysis of the lipid ontology terms using a Shiny app with options of five statistical approaches. Lipid classes can be added to customize the user’s database and remain updated as new lipid classes are discovered. Visualization of results is available for all classification options (e.g. lipid subclass and individual fatty acid chains). Results are also visualized through an editable network of relationships between the individual lipids and their associated lipid ontology terms. The utility of the tool is demonstrated using biological (e.g. human lung endothelial cells) and environmental (e.g. peat soil) samples. Availability and implementation Rodin (R package: https://github.com/PNNL-Comp-Mass-Spec/Rodin), Lipid Mini-On Shiny app (https://github.com/PNNL-Comp-Mass-Spec/LipidMiniOn) and Lipid Mini-On online tool (https://omicstools.pnnl.gov/shiny/lipid-mini-on/). Supplementary information Supplementary data are available at Bioinformatics online.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2019

GRACC: GRid ACcounting Collector

The OSG has long maintained a central accounting system called Gratia. It uses small probes on each computing and storage resource in order to collect resource usage. The probes report to a central collector which stores the usage in a database. The database is then queried to generate reports. As the OSG aged, the size of the database grew very large. It became too large for the database technology to efficiently query to generate detailed reports. The design of a replacement requires data storage that could be queried efficiently to generate multi-year reports. Additionally, it requires flexibilityto add new attributes to the collected data.In this paper we will describe updates to the GRACC architecture in the last 18 months. GRACC uses modern web technologies that were designed for large data storage, query, and visualization. That includes the open source database Elasticsearch, message broker software RabbitMQ, and Grafana and Kibana as data visualization platforms. It uses multiple agents that perform operations on the data to transform it for easier querying and summarization.

Weitzel, Derek↗
DOE OSTI2019

Density Calculations of Actinide Solutions using the Pitzer Method

This report summarizes an exhaustive literature search for density data of actinide solutions in support of the Nuclear Criticality Safety Program (NCSP). This work comprises an extension of work originally begun in 2003 that implemented an advanced density calculation scheme in the SCALE material input processor. This method demonstrated improved criticality calculations for aqueous systems of UO 2 (NO 3 ) 2 and Pu(NO 3 ) 4 in excess acid over a considerable range of concentrations and temperatures. It also provided for UO 2 F 2 in acid at room temperature, although with higher uncertainty. The current study was intended to search all available sources for additional data on uranium and plutonium solutions, including the open literature, reports with limited distribution, and unpublished sources such as laboratory notebooks. Data for some of the systems has been regressed to obtain model parameters for the Pitzer formalism of modeling electrolyte solutions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Vulnerabilities in Artificial Intelligence and Machine Learning Applications and Data

Artificial intelligence (AI) applications driven by machine learning (ML) are transformational technologies within the international nuclear security regime. Advancements realized by AI—faster and improved data insights, more efficient and automated processes, reductions in human error—enable nuclear security applications such as behavior analysis for insider threat mitigation, source tracking of stolen nuclear material, and facial recognition software for physical protection. In addition to the advantages, however, there are also inherent vulnerabilities and threats associated with its use and risk mitigations must be built into any AI/ML-enabled systems. This work provides a background on AI and ML and different data types used in the field, including open-source intelligence information (OSINT) that is discoverable by AI tools and application data that are used by AI tools for decision-making and automation. Current and potential AI applications and vulnerabilities related to their use within the nuclear security regime are also discussed.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2024

Database schema design: Energy Flexibility Environmental Tradeoffs Tool

The Energy Flexibility-Environment Tradeoff Toolset is designed using Streamlit framework, with Python as the programming language. Data storage is facilitated through the use of SQLite. Streamlit is an open-source Python framework for machine learning and data science teams, and SQLite is the most used database engine.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2016

An Adaptable Seismic Data Format

We present ASDF, the Adaptable Seismic Data Format, a modern and practical data format for all branches of seismology and beyond. The growing volume of freely available data coupled with ever expanding computational power opens avenues to tackle larger and more complex problems. Current bottlenecks include inefficient resource usage and insufficient data organization. Properly scaling a problem requires the resolution of both these challenges, and existing data formats are no longer up to the task. ASDF stores any number of synthetic, processed or unaltered waveforms in a single file. A key improvement compared to existing formats is the inclusion of comprehensive meta information, such as event or station information, in the same file. Additionally, it is also usable for any non-waveform data, for example, cross-correlations, adjoint sources or receiver functions. Last but not least, full provenance information can be stored alongside each item of data, thereby enhancing reproducibility and accountability. Any data set in our proposed format is self-describing and can be readily exchanged with others, facilitating collaboration. The utilization of the HDF5 container format grants efficient and parallel I/O operations, integrated compression algorithms and check sums to guard against data corruption. To not reinvent the wheel and to build upon past developments, we use existing standards like QuakeML, StationXML, W3C PROV and HDF5 wherever feasible. Usability and tool support are crucial for any new format to gain acceptance. We developed mature C/Fortran and Python based APIs coupling ASDF to the widely used SPECFEM3D_GLOBE and ObsPy toolkits.

58 GEOSCIENCES↗
DOE OSTI2023

Soil architecture and subsurface stratigraphic heterogeneities across land-lake interfaces along Lake Erie.

Uploaded raw datasets from electrical resistivity tomography (ERT), ground penetrating radar (GPR), electromagnetic induction (EMI) and insitu sensors were acquired across land-lake interfaces along Lake Erie. These datasets were useful to delineate the soil architecture and subsurface stratigraphic heterogeneities. We combined multiple geophysical methods with lithostratigraphic logs from borehole cores and data from soil and borehole sensors to investigate spatial heterogeneities across land-lake interfaces. The apparent electrical conductivity (Eca) maps matched closely with the soil maps from public database and also detected additional soil units that were missing in the traditional soil maps. This implies that electromagnetic induction (EMI) could be relied upon for non invasive characterization of soils in sampling restricted sites where only non-invasive measurements are feasible. Results from electrical resistivity tomography (ERT) and Ground penetrating radar (GPR) are consistent with the surficial geology of the study area, and revealed that the vertical variabilities at the sites are mainly driven by stratigraphy.The ERT data (.stg) were acquired from three different sites portage river (PTR), crane creek (CRC) and old woman creek (OWC). The ".stg" files can be opened with ERT inversion software such as EarthImager, ResIpy or Res2DInv. The GPR data (.GPZ) were also acquired from the three sites, and the ".GPZ" data files can be opened with EKKO_project 5 software from Sensors & Software Inc., or with any other standard GPR software. The EM data (.csv) were acquired from the three sites while the soil sensor data (.csv) here were acquired only from crane creek site. The ".csv) files can be opened with MS Excel or any other spread sheet software.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

Using KBase to Assemble and Annotate Prokaryotic Genomes

The DOE Systems Biology Knowledgebase is an open–access bioinformatics software and data platform for analyzing plants, microbes, and their communities. KBase enables scientists to create, execute, collaborate on, and share reproducible analyses of their biological data in the context of public data and private collaborator data. For microbiologists researching prokaryotes, KBase offers analysis tools for performing quality control and assessment of Next–Generation Sequencing reads, de novo assembly, genome annotation, and tools for analyzing structural and functional features of genomes. As a result, this unit demonstrates an example workflow for taking a comparative and iterative approach to assembly and annotation of prokaryotic genomes using KBase that can be used by microbiologists seeking to perform isolate analysis in a rapid and reproducible fashion.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Machine-Learning Accelerated Studies of Materials with High Performance and Edge Computing

In the studies of materials, experimental measurements often serve as the reference to verify physics theory and modeling; while theory and modeling provide a fundamental understanding of the physics and principles behind. However, the interactions and cross validation between them have long been a challenge even to-date. Not only that inferring a physics model from experimental data is itself a difficult inverse problem, another major challenge is the orders-of-magnitude longer wall-clock time required to carry out high-fidelity computer modeling to match the timescale of experiments. We envisage that by combining high performance computing, data science, and edge computing technology, the current predicament can be alleviated, and a new paradigm of data-driven physics research will open up. For example, we can accelerate computer simulations by first performing the large-scale modeling on high performance computers and train a machine-learned surrogate model. This computationally inexpensive surrogate model can then be transferred to the computing units residing closely to the experimental facilities to perform high-fidelity simulations at a much higher throughout. The model will also be more amenable to analyzing and validating experimental observations in comparable time scales at a much lower computational cost. Further integration of these accelerated computer simulations with an outer machine learning loop can also inform and direct future experiments, while making the inverse problem of physics model inference more tractable. We will demonstrate a proof-of-concept by using a quantum Monte Carlo application, Dynamical Cluster Approximation (DCA++), to machine-learn a surrogate model and accelerate the study of quantum correlated materials.

Li, Ying Wai↗