Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data processing and standardization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

Towards a Comparative Assessment of Data-Driven Process Models in Health Information Technology

Process mining for conformance analysis focuses on comparing a reference process model against a data-driven process model that is generated via log files from information technology systems. While this approach is helpful when there is an existing process model in an organization, it leaves the question of what to do in the absence of a complete reference process model unanswered. In this paper, we present a comparative assessment approach that combines process mining, process mapping for dimensionality reduction, and statistical analysis. Our goal is to find similarities and dissimilarities in data-driven process models among U.S. Veterans Health Administration (VHA) facilities to assess process conformance among different healthcare facilities, which can help assess the standardization of care. We illustrate our approach by applying it to two clinical radiology order process models generated by two similar facilities. Our results demonstrate statistical similarities in the standardization of care among those two facilities.

Klasky, Hilda↗

PV Performance Modeling and Stakeholder Engagement (Final Technical Report)

This core capability project’s objective is to increase the value of photovoltaic (PV) performance models by improving their functionality, demonstrating, and quantifying their validity, and offering a wide range of stakeholder engagement opportunities. In FY22-24, we developed new and improved modeling algorithms and functions to represent PV performance more accurately in a variety of environments and conditions. The “Model parameter toolkit” was developed and includes functions to translate between different module temperature models, incidence angle modifier models, and single-diode models. A new modeling capability named “PV Atlas” was also developed leveraging Sandia’s High Performance Computing resources. This capability allows us to investigate several questions and provide climate-specific best practices and geographic data files; all these are hosted on an interactive website on Sandia’s GitHub and can be used for training, system optimization, or to provide best practices for uncertainty reduction. For model validation, we published high-quality PV performance, and weather data; these data are well documented, filtered, and processed for quality and include examples on how to run PV simulations. We also developed well documented, standardized methods for validating PV models and ran independent model validation and 2 blind modeling intercomparisons engaging with 49 organizations from 17 countries. We co-led and contributed to a growing, well documented and maintained suite of open-source functions for PV modeling (i.e., the pvlib-python) and we outreached to the PV modeling stakeholders via the PVPMC workshops and web resources. In addition, this project supported US representation and leadership for the International Energy Agency (IEA) PVPS Task 13; specifically, members of our team led and supported 3 subtasks on: 1) Best practices for the optimization of bifacial photovoltaic tracking, 2) Extreme weather events and their multiple impact on PV power plants: Risks, failure mechanisms and mitigation strategies, and 3) Best practice guidelines for the use of economic and technical Key Performance Indicators (KPIs). This project resulted in the publications of 14 peer reviewed journal papers, 37 conference presentations, 6 SAND reports, 5 public datasets and 6 new webpages on the PVPMC website. It supported the release of 13 pvlib-python versions where 28 enhancements were from this PV Performance Modeling project. We co-organized 5 PVPMC workshops in FY22-24 with the participation of 214 unique institutions and around 700 participants. The PVPMC website was redesigned, and its reliability was improved; it receives over 50,000 visitors/year from 202 unique countries.

14 SOLAR ENERGY↗

Coupling of high-resolution mass spectrometer and photosynthesis system for comprehensive leaf volatile metabolite profiling

Background Leaf-level biogenic volatile organic compounds (BVOCs) emissions represent a major source of organic gases in the atmosphere, influencing both climate and air quality. These emissions are strongly driven by environmental perturbations, which affect individual plant- to ecosystem-level processes. Uncovering all the BVOCs and understanding how their emissions respond to altered environmental conditions provide critical insights into vegetation-driven changes in atmospheric chemistry. We developed a tandem instrumentation setup that integrates a proton transfer reaction time-of-flight mass spectrometer (PTR-ToF-MS) with parts-per-trillion detection limits and a photosynthetic infrared gas exchange system for the untargeted survey of all the BVOCs. This novel system enables simultaneous, real-time monitoring of BVOC emissions and photosynthetic parameters at the leaf level, offering new opportunities to disentangle the physiological and environmental drivers of VOC release. Furthermore, we established the VOC Analysis and Processing Optimization Resource (VAPOR), an open-access software tool designed for rapid data post-processing and the analysis of the variability of hundreds of BVOCs. We assessed the performance of the tandem system under varying background conditions, using standard gas mixtures and a range of environmental factors. Results Blank emissions were substantially lower for major BVOCs (e.g., isoprene) compared to those observed in plant emissions. Despite this, the observation of background-level VOCs highlights the importance of routinely acquiring and accounting for blank measurements in analyses using the coupled instrumentation. Introduction of known VOC concentrations to the system demonstrated a linear response across different compounds with varying molecular compositions, indicating minimal gas loss regardless of chemical moieties within the coupled instrumentation. We applied the optimized system to investigate the physiological mechanisms driving BVOC emissions across different genotypes of poplar and pennycress. The high mass resolution capabilities of the PTR-ToF-MS, coupled with comprehensive VAPOR-driven data analysis, enabled the identification of several important BVOCs, including methanol and methanethiol; these BVOCs displayed substantial variation across pennycress genotypes and showed concentrations ~ 100–350% higher than the blank. Moreover, isoprene emissions varied significantly among poplar genotypes grown in different potting media. Conclusions Tandem instrumentation offers a powerful tool for profiling volatile molecular markers and elucidating their genetic and environmental underpinnings. This approach enhances our ability to predict BVOC emissions in response to genotype by environmental interactions and contributes to a deeper understanding of vegetation responses to environmental changes.

Biogenic volatile organic compounds↗

Innovating the next generation of commercial smart building software

Nearly 30% of commercial building energy use is wasted due to equipment faults and HVAC controls problems. The result is increased emissions, compromised comfort and productivity, and less reliable coordination of building power needs with a clean grid. The energy impact alone represents $17 billion in potential savings. Today’s smart building software provides a robust solution to address these operational deficiencies. Energy management and information systems (EMIS) are saving up to 9% on average, with two-year paybacks. They are being incorporated into energy management processes, commissioning services, and utility programs. As effective as they are, two barriers prevent even deeper benefits; limited personnel to fix problems once they are identified, and the expense and time to manually implement changes in control systems. In partnership with the research community, the EMIS industry is developing new capabilities to overcome these barriers. Moving beyond siloed products for either fault detection and diagnostics, or optimal control, these new capabilities empower users to not only automatically identify faults, but also to push corrective action, and control improvements to their buildings. In this paper, several areas for enhancements are documented: ‘one-time’ correction of faults such as setpoints, schedules, and economizer lockouts; short-term active testing for automated proportional integral derivative (PID) loop tuning and functional testing; and continuous supervisory control for demand flexibility and year-round efficiency. Results are presented from a pair of partner implementations out of a dozen providers integrating these enhancements into their products, including field tests from across the country, and insights into operator acceptance and integration into operations and maintenance practices.

Casillas, Armando↗

High-resolution leaf area index maps generated from unoccupied aerial system, Teller Mile 27, Seward Peninsula, Alaska

Leaf area index (LAI), a measure of the amount of one-side leaf area per ground unit, is an important indicator of plant carbon, energy, and water cycle. In the heterogeneous Arctic landscapes, it has been challenging to accurately measure LAI across species and space needed for Earth system model validation. Here, we use multispectral unoccupied aerial systems (UASs) to scale up and map leaf area index (LAI) , in a low-Arctic tundra landscape on the Seward Peninsula, Alaska. We linked previous published LAI measurements with high-resolution, UAS-collected multispectral data collected over the region of Next Generation Ecosystem Experiments in the Arctic (NGEE Arctic)’s Teller Mile Maker 27 site in 2022 to develop random forest (RF) machine learning models to predict and map LAI. 100 RF models were developed to account for uncertainties in ground LAI plot measurements and process scaling. This dataset includes a raster (*.tif) map of the mean LAI value of the 100 RF models, a raster (*.tif) map of the standard deviation of the RF-modeled LAI data, and a user guide (*.pdf).

54 ENVIRONMENTAL SCIENCES↗

Leveraging Large Language Models for Real-World Data Evidence: A Framework for Automated Treatment Extraction and Data Harmonization

Background: The ability to comprehensively collect treatment information from cancer patient medical records would enable studies to evaluate real-world benefits and risks tied to specific treatments. Currently, it is difficult to system- atically collect high-quality treatment information because it is often stored in unstructured text. Manually extracting and standardizing drug and regimen data is time-intensive. Recent advances in large language models (LLMs) offer a potential solution for automated extraction of structured treatment information from clinical text. Objective: This study systematically evaluates the utility of four LLMs from the Llama family for automated extraction of oncology treatment information from clinical text. This information can guide researchers using cancer registry data to provide insights into cancer care and outcomes beyond clinical trials. Methods: Four instruction-tuned Llama models with varying parameter counts (1B, 3B, 8B, and 70B) were evaluated for their ability to extract treatment information from clinical documents. A unified oncology knowledge base integrating seven major public data sources was developed to standardize and normalize extracted entities—a critical step for harmonizing data from diverse sources. Extracted treatment data were compared against expert-annotated ground truth. Model performance was assessed using accuracy metrics (Precision, Recall, F1-Score) and opera- tional feasibility metrics, including processing speed and structural compliance of the output. Results: A strong positive correlation was observed between model size and extraction accuracy. F1-score improved from 0.609 for the 1B model to 0.710 (3B), 0.807 (8B), and 0.828 (70B). While larger models demonstrated superior accuracy and compliance, they incurred higher computational costs. The modest performance difference between 8B and 70B suggests diminishing returns with increasing model size. Conclusions: LLMs represent a viable technology for automating oncology treatment extraction. The 8B-parameter model emerged as a highly effective option, balancing high accuracy and computational efficiency. Selecting an appropriate LLM for deployment in cancer registries involves a trade-off between desired accuracy and available operational resources. Harmonizing extracted entities with the oncology knowledge base facilitates standardized integration into common data models, enhancing data quality for real-world evidence analyses.

artificial intelligence↗

Ectomycorrhizal effects on decomposition are highly dependent on fungal traits, climate, and litter properties: A model-based assessment. Dataset.

To simulate the effects of mycorrhizal fungi on soil organic matter cycling, we incorporated mycorrhizal processes into the Carbon, Organisms, Rhizosphere, and Protection in the Soil Environment (CORPSE) model to develop a new soil model Myco-CORPSE. The new model was calibrated and evaluated against soil measurements taken at temperate forests in New Hampshire (NH) and Georgia (GA). A series of scenario analysis were also conducted to explore the conditions under which ectomycorrhizal (ECM) N acquisition processes can induce different soil C accumulation in ECM systems compared to arbuscular (AM) systems.In this data package, we included:-The Python codes of the standard Myco-CORPSE model we developed: "Standard Myco_CORPSE python codes.zip". The main program is the "gradient_sim.py" which calculates the bulk soil microbes and CN content along a user defined gradient of clay, soil temperature, soil moisture and mycorrhizal dominance, and relies on two subprograms "CORPSE_deriv.py" and "CORPSE_integrate.py". "CORPSE_deriv.py" calculated the changes in all simulated soil stock within every time step and "CORPSE_integrate.py" integrate the changes in all simulated soil stock within simulated time period. The program "Plot.py" is used to plot the major outputs produced by the main program "gradient_sim.py".-The modified Python codes of Myco-CORPSE models with site-level environmental inputs (in NH and GA) used to conduct simulations in NH and GA sites: "NH_GA model simulations.zip". -The Python codes used to evaluate the Myco-CORPSE simulation outputs in NH and GA sites against site-level measurements: "Plot NH_GA simulation against measurements.zip". It includes both the evaluation Python code, the model outputs on NH and GA sites, and the measured soil properties in both sites.-The modified Python codes of Myco-CORPSE models "Scenario analysis_model simulations.zip" that is used to conduct scenario analysis of how different litter properties, mycorrhizal fungal traits, climate, and seasonal variation in temperature and vegetation phenology impact the mycorrhizal effects on soil CN properties. The sub file folder "Scenario analysis_litter traits" contains the codes for scenario analysis of different litter properties; The sub file folder "Scenario analysis_ECM types" contains the codes for scenario analysis of different ECM fungal traits; The sub file folder "Scenario analysis_climate&seasonality" contains the codes for scenario analysis of different climate and seasonalities;-"Scenario analysis_model results and plotting codes.zip" contains all the output files from the the scenario analysis of Myco-CORPSE model as described above and the plotting codes used to the generate the heatmaps and scatterplots shown in the manuscript "Ectomycorrhizal effects on decomposition are highly dependent on fungal traits, climate, and litter properties: A model-based assessment"The majority of the model outputs did not have specific geographic information or temporal coverage because the analysis we conducted are mainly hypothetical model simulations. We only provided geographic description, coordinates and temporal coverage for those soil measurements which we used for model evaluations (included in the "Plot NH_GA simulation against measurements.zip").

54 ENVIRONMENTAL SCIENCES↗

Ptychography at all wavelengths

Ptychography is a computational imaging technique that operates across multiple wavelength regimes, from electron (picometres) to X-ray (~0.1 nm), extreme ultraviolet (~10 nm) and visible light (micrometres). By reconstructing both amplitude and phase from diffraction patterns, ptychography enables high-resolution, quantitative imaging without conventional limitations imposed by lens-based optics. Ptychography has enabled advances across a range of scales: achieving deep-sub-angstrom resolution with electron microscopy, becoming an indispensable tool at X-ray synchrotron facilities worldwide and overcoming the trade-offs between resolution and field-of-view in optical imaging. This Primer provides a unified treatment of ptychography across these wavelength regimes. First, we discuss theoretical foundations, reconstruction algorithms, experimental considerations and wavelength-specific challenges. We then give examples of raw and processed data from various configurations and wavelengths. Next, we highlight key applications of ptychography in life sciences, materials science and industry. We also discuss data standards, open-source software implementations and best practices for ensuring reproducibility across different wavelength regimes. Finally, we consider limitations and future opportunities for ptychography. Together with accompanying datasets and code implementations, this Primer aims to serve newcomers and experienced practitioners in the field, facilitating broader adoption of ptychography across different disciplines.

47 OTHER INSTRUMENTATION↗

Benchmark Tracking System for Performance Monitoring

Benchmarking is essential for high-performance software development, particularly for monitoring performance across code iterations. This project focused on enhancing the benchmarking process for Lamellar, an asynchronous runtime for High-Performance Computing (HPC) systems developed at Pacific Northwest National Laboratory. Prior to this work, benchmark results were difficult to track and compare across code versions, presenting significant challenges in identifying performance regressions and long-term trends. The primary objective was to establish a systematic, reproducible approach for measuring performance and detecting regressions following code commits. Our methodology involved three key components: standardizing benchmark outputs, implementing data versioning, and developing analysis tools. We standardized the benchmark output format to JSON Line records containing specific fields (execution time, hardware specifications, and environmental variables). To address data management challenges, we evaluated several options and eventually chose a git repository dedicated to benchmark data. We developed a suite of Python tools that processed benchmark results, enriched them with metadata, and facilitated search in the repository. The resulting system enables more efficient filtering and comparison of performance metrics across commit histories, hardware configurations, and benchmark variants through a unified query interface. Our implementation reduces computational overhead by first checking for existing results through configuration matching before initiating new benchmark runs, thereby conserving resources. The system has been validated by Lamellar developers. It organizes results by benchmark type and build configurations for efficient retrieval. Future developments include a planned Large Language Model interface for predicting benchmark performance, incorporating the criterion package for statistical analysis, which will enable automated detection of statistically significant performance changes, and integration with continuous integration pipelines. Despite these enhancements being reserved for future work, this project has successfully provided the Lamellar development team with a framework for maintaining consistent performance standards and identifying optimization opportunities across workloads and hardware environments.

97 MATHEMATICS AND COMPUTING↗

Region-based convolutional neural network for wind turbine wake characterization from scanning lidars

A convolutional neural network is applied to lidar scan images from three experimental campaigns to identify and characterize wind turbine wakes. Initially developed as a proof-of-concept model and applied to a single data set in complex terrain, the model is now improved and generalized and applied to two other unique lidar data sets, one located near an escarpment and one located offshore. The model, initially developed using lidar scans collected in predominantly westerly flow, exhibits sensitivity to wind flow direction. The model is thus successfully generalized through implementing a standard rotation process to scan images before input into the convolutional neural network to ensure the flow is westerly. The sample size of lidar scans used to train the model is increased, and along with the generalization process, these changes to the model are shown to enhance accuracy and robustness when characterizing dissipating and asymmetric wakes. Applied to the offshore data set in which nearly 20 wind turbine wakes are included per scan, the improved model exhibits a 95% success rate in characterizing wakes and a 74% success rate in characterizing dissipating wake fragments. The improved model is shown to generalize well to the two new data sets, although an increase in wake characterization accuracy is offset by an increase in model sensitivity and false positive wake identifications.

17 WIND ENERGY↗

Tidal Energy Resource Characterization, Bottom Lander Measurements, Cook Inlet, AK, 2021

These datasets are from tidal resource characterization measurements collected on the Terrasond High Energy Oceanographic Mooring (THEOM) from 1 July 2021 to 30 August 2021 (60 days) in Cook Inlet, Alaska. The lander was deployed at 60.7207031 N, 151.4294998 W in ~50 m of water. The dataset contains raw and processed data from the following two instruments: 1. A Nortek Signature 500 kHz acoustic Doppler current profiler (ADCP). Data were recorded in 4 Hz in the beam coordinate system from all 5 beams. Processed data has been averaged into 5 minutes bins and converted to the East-North-Up (ENU) coordinate system. 2. A Nortek Vector acoustic Doppler velocimeter (ADV). Data were recorded at 8 Hz in the beam coordinate system. Processed data has been averaged into 5 minutes bins and converted to the Streamwise - Cross-stream - Vertical (Principal) coordinate system. Turbulence statistics were calculated from 5-minute bins, with an FFT length equal to the bin length, and saved in the processed dataset. Data was read and analyzed using the DOLfYN (version 1.0.2) python package and saved in MATLAB (.mat) and netCDF (.nc) file formats. Files containing analyzed data (".b1") were standardized using the TSDAT (version 0.4.2) python package. NetCDF files can be opened using DOLfYN (e.g., `dat = dolfyn.load(''*.nc")`) or the xarray python package (e.g. `dat = xarray.open_dataset("*.nc"). All distances are in meters (e.g., depth, range, etc), and all velocities in m/s. See the DOLfYN documentation linked in the submission, and/or the Nortek documentation for additional details.

16 TIDAL AND WAVE POWER↗

Using Meteorological Data to Evaluate Worldwide PV Degradation Rates

Here we automate the process to enable the creation of world maps for specific PV durability metrics. The end goal is to create a tool for the creation of world maps predicting the sensitivity of modules to degradation to inform the development and utilization of module testing standards and to provide information to module designers. TMY (Typical Meteorological Year) Data was aggregated from sites around the world to render insight into factors contributing to the degradation of solar modules. The focus of aggregating data was to create a single dataset to work with utilizing Solar Irradiance values combined with Ambient Temperature, Relative Humidity, Wind Speed and Dew-Point Temperature.

14 SOLAR ENERGY↗

Metadata Schemas and Ontologies for Building Energy Applications: A Critical Review and Use Case Analysis

With the increasing digitalization of processes throughout the lifecycle of buildings, data exchanged between stakeholders and between building systems has grown significantly. However, a lack of semantic interoperability between data in different systems is still prevalent, hindering the development of applications that can be reused across buildings and limiting the scalability of innovative solutions. Semantics refers to the description of the meaning of the data in a way that can be consistently understood by applications. Recently, several competing initiatives have been developing metadata schemas and ontologies to express this semantic information for different applications in the building domain. This paper systematically reviews these schemas and conducts an analysis of five of them to evaluate their applicability to three high-value use cases for building operations: energy audits, automated fault detection and diagnostics and optimal control. The survey finds 40 schemas published in the last 10 years but but their actual use in industry is difficult to estimate. Among the five selected ontologies, several gaps are highlighted in relation to the three use cases. Recommendations for the future include better harmonization of these initiatives, more centralized repositories and search engines for these schemas as well as better industry engagement to facilitate their adoption.

Smart Building, Sematic, Metadata, Ontology, Data ↗

Timely Reporting of Heavy Hitters Using External Memory

Given an input stream S of size N, a Φ-heavy hitter is an item that occurs at least ΦN times in S. The problem of finding heavy-hitters is extensively studied in the database literature. In this work, we study a real-time heavy-hitters variant in which an element must be reported shortly after we see its T = Φ N-th occurrence (and hence it becomes a heavy hitter). We call this the Timely Event Detection (TED) Problem. The TED problem models the needs of many real-world monitoring systems, which demand accurate (i.e., no false negatives) and timely reporting of all events from large, high-speed streams with a low reporting threshold (high sensitivity). Like the classic heavy-hitters problem, solving the TED problem without false-positives requires large space (Ω (N) words). Thus in-RAM heavy-hitters algorithms typically sacrifice accuracy (i.e., allow false positives), sensitivity, or timeliness (i.e., use multiple passes). We show how to adapt heavy-hitters algorithms to external memory to solve the TED problem on large high-speed streams while guaranteeing accuracy, sensitivity, and timeliness. Our data structures are limited only by I/O-bandwidth (not latency) and support a tunable tradeoff between reporting delay and I/O overhead. With a small bounded reporting delay, our algorithms incur only a logarithmic I/O overhead. We implement and validate our data structures empirically using the Firehose streaming benchmark. Multi-threaded versions of our structures can scale to process 11M observations per second before becoming CPU bound. In comparison, a naive adaptation of the standard heavy-hitters algorithm to external memory would be limited by the storage device’s random I/O throughput, i.e., ≈100K observations per second.

97 MATHEMATICS AND COMPUTING↗

HopPyBar

HopPyBar is a python program to import, analyze, and export split-Hopkinson pressure bar (SHPB, also known as Kolsky bar) data. Traditional analysis offers a black box approach, where input data is converted to analyzed output by performing a series of calculations without user involvement. This program serves as a developmental platform to "white box" the data analysis process. Data streams can be captured (in-situ) to enable advanced or unconventional analyses, statistics, and comparisons. Additionally, the program is geared towards the standardized forms of input and output used at LANL to streamline analysis, but the open nature of the program makes additional input/output schemes straightforward to add. General workflow will import SHPB data in one of a number of formats, identify relevant portions of data signals, and convert to stress-strain-strain rate to show material behavior as a function of dynamic testing.

Morrow, Benjamin↗

Confidentiality-preserving machine learning algorithms for soft-failure detection in optical communication networks

Automated fault management is at the forefront of next-generation optical communication networks. The increase in complexity of modern networks has triggered the need for programmable and software-driven architectures to support the operation of agile and self-managed systems. In these scenarios, the European Telecommunications Standards Institute zero-touch network and service management approach is imperative. The need for machine learning algorithms to process the large volume of telemetry data brings safety concerns as distributed cloud-computing solutions become the preferred approach for deploying reliable communication network automation. This paper’s contribution is twofold. First, we propose a simple yet effective method to guarantee the confidentiality of the telemetry data based on feature scrambling. The method allows the operation of third-party computational services without direct access to the full content of the collected data. Additionally, the effectiveness of four unsupervised machine learning algorithms for soft-failure detection is evaluated when applied to the scrambled telemetry data. The methods are based on factor analysis, principal component analysis, nonlinear principal component analysis, and singular value decomposition. Most dimensionality reduction algorithms have the common property that they can maintain similar levels of fault classification performance while hiding the data structure from unauthorized access. Evaluations of the proposed algorithms demonstrate this capability.

97 MATHEMATICS AND COMPUTING↗

SBND Detector Status and First Results

The Short-Baseline Near Detector (SBND) is one of three liquid argon time projection chamber (LArTPC) neutrino detectors positioned along the axis of the Booster Neutrino Beam (BNB) at Fermilab, and serves as the near detector in the Short-Baseline Neutrino (SBN) Program. The detector just completed its second year of running, collecting over 6.5e20 POT, equivalent to an unprecedented sample of over 5 million neutrino interactions. Initial data has demonstrated superb performance of the detector’s subsystems, enabling precise tracking and calorimetric reconstruction of events. With this large statistical data set, SBND is performing multi-dimensional cross section measurements of inclusive and exclusive topologies and precise searches for beyond the Standard Model (BSM) processes. As the near detector in the SBN Program, it will enable the full potential of the joint sterile neutrino measurements program by precisely characterizing the unoscillated neutrino beam, constraining BNB flux and neutrino-argon cross-section systematic uncertainties. In this talk, the current status of the experiment, first results from SBND’s neutrino interaction program, as well as status and prospects for BSM and oscillation searches are discussed.

Paton, Josephine [Fermilab] (ORCID:000000032651489↗