Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software: data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

LipidOz enables automated elucidation of lipid carbon–carbon double bond positions from ozone-induced dissociation mass spectrometry data

Lipids play essential roles in many biological processes and disease pathology, but unambiguous identification of lipids is complicated by the presence of multiple isomeric species differing by fatty acyl chain length, stereospecifically numbered (sn) position, and position/stereochemistry of double bonds. Conventional liquid chromatography-mass spectrometry (LC-MS/MS) analyses enable the determination of fatty acyl chain lengths (and in some cases sn position) and number of double bonds, but not carbon-carbon double bond positions. Ozone-induced dissociation (OzID) is a gas-phase oxidation reaction that produces characteristic fragments from lipids containing double bonds. OzID can be incorporated into ion mobility spectrometry (IMS)-MS instruments for the structural characterization of lipids, including additional isomer separation and confident assignment of double bond positions. The complexity and repetitive nature of OzID data analysis and lack of software tool support have limited the application of OzID for routine lipidomics studies. Here, we present an open-source Python tool, LipidOz, for the automated determination of lipid double bond positions from OzID-IMS-MS data, which employs a combination of traditional automation and deep learning approaches. Our results demonstrate the ability of LipidOz to robustly assign double bond positions for lipid standard mixtures and complex lipid extracts, enabling practical application of OzID for future lipidomics.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Evaluating Software Tools for Lipid Identification from Ion Mobility Spectrometry–Mass Spectrometry Lipidomics Data

The unambiguous identification of lipids is a critical component of lipidomics studies and greatly impacts the interpretation and significance of analyses as well as the ultimate biological understandings derived from measurements. The level of structural detail that is available for lipid identifications is largely determined by the analytical platform being used. Mass spectrometry (MS) coupled with liquid chromatography (LC) is the predominant combination of analytical techniques used for lipidomics studies, and these methods can provide fairly detailed lipid identification. More recently, ion mobility spectrometry (IMS) has begun to see greater adoption in lipidomics studies thanks to the additional dimension of separation that it provides and the added structural information that can support lipid identification. At present, relatively few software tools are available for IMS-MS lipidomics data analysis, which reflects the still limited adoption of IMS as well as the limited software support. This fact is even more pronounced for isomer identifications, such as the determination of double bond positions or integration with MS-based imaging. In this review, we survey the landscape of software tools that are available for the analysis of IMS-MS-based lipidomics data and we evaluate lipid identifications produced by these tools using open-access data sourced from the peer-reviewed lipidomics literature.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Deep Learning for Automated Identification of Eels in Sonar Data

Freshwater eels, such as the American eel (Anguilla rostrata) present numerous challenges related to safe downstream fish passage at hydroelectric facilities. One of those challenges is effective monitoring of their abundance, movements, and behavior to facilitate design and operation of eel protection and passage facilities. A previous EPRI study documented the ability of human analysts to reliably identify American eels in data obtained with a 1100/1800 kHz, multibeam sonar. This report describes a project to develop deep learning (a subset of artificial intelligence) tools to automate the time-consuming, subjective process of eel identification in multibeam sonar data. The project exploited new data collected in the laboratory and the existing data from the prior EPRI field study to develop and test deep learning and other data analytic tools, including wavelet filtering, differencing for static object removal, and convolutional neural network analysis. The analysis of the laboratory data demonstrated feasibility of the approach, revealed object characteristics observed with the sonar that distinguish eels from similarly sized and shaped acoustic targets, and provided additional data for algorithm selection and training. Deep learning algorithms trained and tested on the laboratory data alone achieved accuracy rates of greater than 98% when classifying acoustic images of eels and similar-sized neutrally buoyant sticks. The algorithm trained and tested on the pre-existing field data alone, and yielded classification accuracy of 9.3% false positives and 13.3% false negatives when distinguishing between eels and sticks/PVC pipes based on video clips (i.e., multiple, consecutive images). This performance is comparable to the classification accuracy achieved by human analysts in the prior study. The deep learning algorithm trained on a combination of video clips obtained in the laboratory and the field and tested on video clips from the field, was able to distinguish eels from sticks and PVC pipes (a river debris analog) of similar size with 100% accuracy. Outreach to the hardware, software, and end-user communities early in the project helped to identify needs and specify the application space. Outreach to those communities at the end of the project communicated project results and opportunities for further development. The project achieved proof of concept for automated identification of eel in multibeam sonar data. Future work should focus on acquisition of additional data for more robust algorithm training and testing; modification of the software tools to accommodate multiple acoustic targets in the acoustic field at a given time; identification of additional object classes; incorporation of motion in the object identification and classification algorithms; operationalizing the software tools, including integration with other existing sonar data analysis tools; and partnering with hardware and software providers for distribution of the software tools with their commercial products.

13 HYDRO ENERGY↗
DOE OSTI2020

XtalCAMP: a comprehensive program for the analysis and visualization of scanning Laue X-ray micro-/nanodiffraction data

XtalCAMP is a software package based on the MATLAB platform, which is suitable for, but not limited to, the analysis and visualization of scanning Laue X-ray micro-/nanodiffraction data. The main objective of the software is to provide complementary functionalities to the Laue indexing software packages used at several synchrotron beamlines. Here, the graphical user interfaces allow the easy analysis of characteristic microstructure features, including real-time intensity mapping for a quick examination of phase, grain and defect distribution, 2D color-coded mapping of microstructural properties from the output of other Laue indexing software, crystal orientation visualization, grain boundary characterization based on orientation/misorientation calculation, principal strain/stress analysis, and strain ellipsoid representation, as well as a series of additional toolkits. As an example, XtalCAMP is applied to the microstructural investigation of a solution-heat-treated Ni-based superalloy manufactured using a laser 3D-printing technique, and a deformed natural quartzite from Val Bregaglia in the Central Alps.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Managing Software Provenance to Enhance Reproducibility in Computational Research

Scientific processes rely on software as an important tool for data acquisition, analysis, and discovery. Over the years, sustainable software development practices have made progress in being considered as an integral component of research. However, management of computation-based scientific studies is often left to individual researchers who design their computational experiments based on personal preferences and the nature of the study. Here, we believe that the quality, efficiency, and reproducibility of computation-based scientific research can be improved by explicitly creating an execution environment that allows researchers to provide a clear record of traceability. This is particularly relevant to complex computational studies in high-performance computing (HPC) environments. In this article, we review the documentation required to maintain a comprehensive record of HPC computational experiments for reproducibility. We also provide an overview of tools and practices that we have developed to perform such studies around Flash-X, a multiphysics scientific software.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Data from: "Towards CONUS-Wide ML-Augmented Conceptually-Interpretable Modeling of Catchment-Scale Precipitation-Storage-Runoff Dynamics"

This data package was generated to support the manuscript “Towards CONUS-Wide Machine Learning-Augmented Conceptually Interpretable Modeling of Catchment-Scale Precipitation-Storage-Runoff Dynamics.” It provides input files, model outputs, plotting data, scripts, notebooks, and documentation used to develop, evaluate, and reproduce Mass-Conserving Perceptron (MCP)-based hydrologic modeling experiments across 513 selected Catchment Attributes and Meteorology for Large-sample Studies in the United States (CAMELS-US) basins. The files are organized by modeling component and analysis purpose, including rainfall–runoff experiments, snow module experiments, coupled hydrologic-snow experiments, Long Short-Term Memory (LSTM) benchmark results, model skill metrics, initialization and epoch records, cell-state normalization files, Akaike Information Criterion (AIC)-based model comparison files, and data used to generate manuscript figures. Tabular files can be opened using standard spreadsheet software or Python/R data-analysis tools. Python scripts, Jupyter notebooks, and selected MATLAB scripts are included for model execution, postprocessing, plotting, and statistical analysis. Quality assurance and quality control were conducted through the source-data selection and modeling workflow. Meteorological forcing, streamflow, and static catchment attributes were derived from the CAMELS-US dataset, and snow water equivalent data were derived from the University of Arizona (UA) Snow Water Equivalent dataset. Selected basins and time periods were screened during the associated research workflow to avoid missing observations or poor-quality cases. Static geospatial features were processed primarily using Quantum Geographic Information System (QGIS) and Geospatial Data Abstraction Library (GDAL) workflows. Additional details are provided in the associated manuscript and documentation.

ESS-DIVE CSV File Formatting Guidelines Reporting ↗
DOE OSTI2021

Scalable Comparative Visualization of Ensembles of Call Graphs

Optimizing the performance of large-scale parallel codes is critical for efficient utilization of computing resources. Code developers often explore various execution parameters, such as hardware configurations, system software choices, and application parameters, and are interested in detecting and understanding bottlenecks in different executions. They often collect hierarchical performance profiles represented as call graphs, which combine performance metrics with their execution contexts. The crucial task of exploring multiple call graphs together is tedious and challenging because of the many structural differences in the execution contexts and significant variability in the collected performance metrics (e.g., execution runtime). In this paper, we present Ensemble CallFlow to support the exploration of ensembles of call graphs using new types of visualizations, analysis, graph operations, and features. We introduce ensemble-Sankey , a new visual design that combines the strengths of resource-flow (Sankey) and box-plot visualization techniques. Whereas the resource-flow visualization can easily and intuitively describe the graphical nature of the call graph, the box plots overlaid on the nodes of Sankey convey the performance variability within the ensemble. Our interactive visual interface provides linked views to help explore ensembles of call graphs, e.g., by facilitating the analysis of structural differences, and identifying similar or distinct call graphs. Finally, we demonstrate the effectiveness and usefulness of our design through case studies on large-scale parallel codes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

py4DSTEM: A Software Package for Four-Dimensional Scanning Transmission Electron Microscopy Data Analysis

Scanning transmission electron microscopy (STEM) allows for imaging, diffraction, and spectroscopy of materials on length scales ranging from microns to atoms. By using a high-speed, direct electron detector, it is now possible to record a full two-dimensional (2D) image of the diffracted electron beam at each probe position, typically a 2D grid of probe positions. These 4D-STEM datasets are rich in information, including signatures of the local structure, orientation, deformation, electromagnetic fields, and other sample-dependent properties. However, extracting this information requires complex analysis pipelines that include data wrangling, calibration, analysis, and visualization, all while maintaining robustness against imaging distortions and artifacts. In this paper, we present py4DSTEM, an analysis toolkit for measuring material properties from 4D-STEM datasets, written in the Python language and released with an open-source license. We describe the algorithmic steps for dataset calibration and various 4D-STEM property measurements in detail and present results from several experimental datasets. We also implement a simple and universal file format appropriate for electron microscopy data in py4DSTEM, which uses the open-source HDF5 standard. We hope this tool will benefit the research community and help improve the standards for data and computational methods in electron microscopy, and we invite the community to contribute to this ongoing project.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

Data from: Lowland Tropical Forests Remain a Methane Sink Under Warming and Long-Term Hurricane Disturbance Recovery

The repository folder contains spreadsheets and script for soil greenhouse gas (GHG) fluxes, soil moisture, soil temperature, air temperature, and precipitation measurements collected from the Tropical Responses to Altered Climate Experiment (TRACE) at the Sabana Research Field Station, El Yunque National Forest (USDA Forest Service; 18°19′28.74″ N, 65°43′50.09″ W) — an open-air field warming experiment located in a lowland tropical forest in Puerto Rico within the Luquillo Experimental Forest (LEF) — six to seven years after Hurricanes Irma and Maria (2017). All spreadsheets for soil and air microclimate data, as well as soil greenhouse gas data, are included as csv files. Air temperature data are also included as Excel spreadsheets (.xlsx). The script is built in R Studio, which is the only software required to run data analysis. This dataset is associated with the manuscript “Larocca Conte G ; Zuvela L ; Cruz-Pérez R ; Barreto-Vélez T ; Becerra-Santillan N ; Campbell S ; Chu H ; Dam T ; Grullón-Penkova I ; Kleit M ; Ortiz-Iglesias D ; Rubio-Lebrón L ; Cavaleri M ; Reed S ; Sihi D ; Wood T ; O'Connell C., 2026. Lowland Tropical Forests Remain a Methane Sink Under Warming and Long-Term Hurricane Disturbance Recovery. Agricultural and Forest Meteorology. In review". The dataset was used to test the effect of warming on soil CH4 dynamics following long-term legacy effects of hurricane disturbance. The dataset includes: - An overall README file in word and pdf format describing methodology and spreadsheets’ structure. - Continuous measurements of soil temperature and moisture from January 2023 to July 2024 measured with Campbell CS655 probes (“TRACE_soil_temperature_and_moisture_2023_cleaned(in).csv” and “TRACE_soil_temperature_and_moisture_2024_cleaned. csv”). - Air temperature data measured with a HOBO MX23O1A data logger (“Hobo air temperature 2023 Sep 2024” and “Hobo air temperature 2023 Sep 2024” – “CSV FILES folders”). - Precipitation data from a nearby weather tower downloaded from González et al. (2025; “sabana_2020-2025.csv”). - Soil CH4 and CO2 effluxes measured intermittently in two summer campaigns (June – August 2023 and June – July 2024) with a LI-COR 8200-01S Portable Smart Chamber coupled with a LI-COR LI-7810 CH4/ CO2/H2O Trace Gas Analyzer (“23_24COMBO2.0.csv”). - R markdown script for data analysis (“Trace new_PLOTS.Rmd”).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Archetype-based Redshift Estimation for the Dark Energy Spectroscopic Instrument Survey

We present a computationally efficient galaxy archetype-based redshift estimation and spectral classification method for the Dark Energy Survey Instrument (DESI) survey. The DESI survey currently relies on a redshift fitter and spectral classifier using a linear combination of principal component analysis–derived templates, which is very efficient in processing large volumes of DESI spectra within a short time frame. However, this method occasionally yields unphysical model fits for galaxies and fails to adequately absorb calibration errors that may still be occasionally visible in the reduced spectra. Our proposed approach improves upon this existing method by refitting the spectra with carefully generated physical galaxy archetypes combined with additional terms designed to absorb data reduction defects and provide more physical models to the DESI spectra. We test our method on an extensive data set derived from the survey validation (SV) and Year 1 (Y1) data of DESI. Our findings indicate that the new method delivers marginally better redshift success for SV tiles while reducing catastrophic redshift failure by 10%–30%. At the same time, results from millions of targets from the main survey show that our model has relatively higher redshift success and purity rates (0.5%–0.8% higher) for galaxy targets while having similar success for QSOs. These improvements also demonstrate that the main DESI redshift pipeline is generally robust. Additionally, it reduces the false-positive redshift estimation by 5%–40% for sky fibers. We also discuss the generic nature of our method and how it can be extended to other large spectroscopic surveys, along with possible future improvements.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

The Astropy Project: Sustaining and Growing a Community-oriented Open-source Project and the Latest Major Release (v5.0) of the Core Package

The Astropy Project supports and fosters the development of open-source and openly developed Python packages that provide commonly needed functionality to the astronomical community. A key element of the Astropy Project is the core package astropy, which serves as the foundation for more specialized projects and packages. In this article, we summarize key features in the core package as of the recent major release, version 5.0, and provide major updates on the Project. We then discuss supporting a broader ecosystem of interoperable packages, including connections with several astronomical observatories and missions. We also revisit the future outlook of the Astropy Project and the current status of Learn Astropy. We conclude by raising and discussing the current and future challenges facing the Project.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Considerations for Optimizing the Photometric Classification of Supernovae from the Rubin Observatory

The Vera C. Rubin Observatory will increase the number of observed supernovae (SNe) by an order of magnitude; however, it is impossible to spectroscopically confirm the class for all SNe discovered. Thus, photometric classification is crucial, but its accuracy depends on the not-yet-finalized observing strategy of Rubin Observatory's Legacy Survey of Space and Time (LSST). We quantitatively analyze the impact of the LSST observing strategy on SNe classification using simulated multiband light curves from the Photometric LSST Astronomical Time-Series Classification Challenge (PLAsTiCC). First, we augment the simulated training set to be representative of the photometric redshift distribution per SNe class, the cadence of observations, and the flux uncertainty distribution of the test set. Then we build a classifier using the photometric transient classification library snmachine, based on wavelet features obtained from Gaussian process fits, yielding a similar performance to the winning PLAsTiCC entry. We study the classification performance for SNe with different properties within a single simulated observing strategy. We find that season length is important, with light curves of 150 days yielding the highest performance. Cadence also has an important impact on SNe classification; events with median inter-night gap <3.5 days yield higher classification performance. Interestingly, we find that large gaps (>10 days) in light-curve observations do not impact performance if sufficient observations are available on either side, due to the effectiveness of the Gaussian process interpolation. This analysis is the first exploration of the impact of observing strategy on photometric SN classification with LSST.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Impact of Rubin Observatory Cadence Choices on Supernovae Photometric Classification

The Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) will discover an unprecedented number of supernovae (SNe), making spectroscopic classification for all the events infeasible. LSST will thus rely on photometric classification, whose accuracy depends on the not-yet-finalized LSST observing strategy. In this work, we analyze the impact of cadence choices on classification performance using simulated multiband light curves. First, we simulate SNe with an LSST baseline cadence, a nonrolling cadence, and a presto-color cadence, which observes each sky location three times per night instead of twice. Each simulated data set includes a spectroscopically confirmed training set, which we augment to be representative of the test set as part of the classification pipeline. Then we use the photometric transient classification library snmachine to build classifiers. We find that the active region of the rolling cadence used in the baseline observing strategy yields a 25% improvement in classification performance relative to the background region. This improvement in performance in the actively rolling region is also associated with an increase of up to a factor of 2.7 in the number of cosmologically useful Type Ia SNe relative to the background region. However, adding a third visit per night as implemented in presto-color degrades classification performance due to more irregularly sampled light curves. Overall, our results establish desiderata on the observing cadence related to classification of full SNe light curves, which in turn impacts photometric SNe cosmology with LSST.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Large Area Airborne Contamination Monitoring

The individual components of a Large Area Airborne Contamination Monitoring (LAACM) system have been demonstrated as functional and the concept of a low-cost, rapidly deployable, and rapidly expandable system has been validated. All components of a previously demonstrated system were re-evaluated. Based on this evaluation, recommendations were made for air collection equipment, scintillating material was synthesized and optimized for application, and improved images were acquired. A preliminary concept of operations was developed, outlining the need for an automated scintillation application system and software for data acquisition and analysis. The work conducted under this program in FY24 has raised the Technology Readiness Level (TRL) of the LAACM from TRL 3 to TRL 4.

Whiteside, Tad S. [Savannah River National Laborat↗
DOE OSTI2021

Milestone M6 Report: Reducing Excess Data Movement Part 1

This is the second in a sequence of three Hardware Evaluation milestones that provide insight into the following questions: What are the sources of excess data movement across all levels of the memory hierarchy, going out to the network fabric? What can be done at various levels of the hardware/software hierarchy to reduce excess data movement? How does reduced data movement track application performance? The results of this study can be used to suggest where the DOE supercomputing facilities, working with their hardware vendors, can optimize aspects of the system to reduce excess data movement. Quantitative analysis will also benefit systems software and applications to optimize caching and data layout strategies. Another potential avenue is to answer cost-benefit questions, such as those involving memory capacity versus latency and bandwidth. This milestone focuses on techniques to reduce data movement, quantitatively evaluates the efficacy of the techniques in accomplishing that goal, and measures how performance tracks data movement reduction. We study a small collection of benchmarks and proxy mini-apps that run on pre-exascale GPUs and on the Accelsim GPU simulator. Our approach has two thrusts: to measure advanced data movement reduction directives and techniques on the newest available GPUs, and to evaluate our benchmark set on simulated GPUs configured with architectural refinements to reduce data movement.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Optical emissivity dataset of multi-material heterogeneous designs generated with automated figure extraction

Optical device design is typically an iterative optimization process based on a good initial guess from prior reports. Optical properties databases are useful in this process but difficult to compile because their parsing requires finding relevant papers and manually converting graphical emissivity curves to data tables. Here, we present two contributions: one is a dataset of thermal emissivity records with design-related parameters, and the other is a software tool for automated colored curve data extraction from scientific plots. We manually collected 64 papers with 176 figures reporting thermal emissivity and automatically retrieved 153 colored curve data records. The automated figure analysis software pipeline uses Faster R-CNN for axes and legend object detection, EasyOCR for axes numbering recognition, and k-means clustering for colored curve retrieval. Additionally, we manually extracted geometry, materials, and method information from the text to add necessary metadata to each emissivity curve. Finally, we analyzed the dataset to determine the dominant classes of emissivity curves and determine the underlying design parameters leading to a type of emissivity profile.

47 OTHER INSTRUMENTATION↗