Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data analysis, visualization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

ExaWind at NREL: Upping the Ante

The objective of the ExaWind component of the Exascale Computing Project is to deliver many-turbine blade-resolved simulations in complex terrain. These simulations bring new challenges to both compute and analysis of the resulting data. In this paper/video, we visually explore the impact of ExaWind on wind simulations through two studies of a small wind farm under two atmospheric conditions. We then turn to analysis and review tools that visualization researchers at NREL use to answer the challenges that ExaWind brings.

collaborative visualization↗
DOE OSTI2025

Enabling Command-and-Control in Advanced In Situ Workflows

Scientific discovery is progressing towards autonomous science with the combination of scientific instruments, high-performance computing, and artificial intelligence in complex workflows. This evolution introduces new requirements for managing scientific workflows, including feedback loops, near real-time constraints, and the ability to dynamically control workflow execution. In situ workflows that analyze and visualize data as it is generated are well-suited to satisfy stringent time constraints and their iterative nature offers greater opportunities for command-and-control. However, only a few of the many workflow management systems available have been specifically designed to manage in situ workflows and often lack support for automated feedback loops that allow analysis and visualization components to interact with the main scientific data producer. To address this need, we present in this paper how to add command-and-control capabilities to a workflow management system. We identify the functional design requirements of such a command-and-control system, detail its architecture, interface, and core mechanisms, and illustrate how advanced in situ workflows can leverage command-and-control in three use cases: graceful termination with checkpoint, dynamic and adaptive data reduction, and event-triggered analysis.

Mehta, Kshitij [ORNL] (ORCID:0000000297149981)↗
DOE OSTI2020

GIS Visualization of Transportation Energy Consumption

Transportation is witnessing unprecedented transformation, where emerging technologies are disrupting the way we travel. Be it sharing economy, or micro-mobility, the landscape of urban transportation is undergoing a much-needed paradigm shift. In order to capture and model these shifts, researchers need to be agile in their studies of the current and future transportation landscape. The need for agility in turn calls for sophisticated, intuitive, and reliable means to visualize and share transportation data. Travel is inherently spatial, so methods of visualizing transportation information should also be rooted in spatial analysis. Geographic Information Systems (GIS) are the premier technological framework to analyze and display spatial data. GIS tools have been used in the past to depict flow of vehicles, and display of network conditions (speed, congestion, etc.). However, in an ever-changing technological landscape, spanning advancements in vehicle as well as information systems, depicting vehicle flows falls short of providing a comprehensive picture of the impact these advancements have on travel related energy consumption. To address this issue, this research effort presents a web-based mapping application to visualize how travel related energy flows across a city. The application is being developed by National Renewable Energy Laboratory researchers to integrate various transportation energy consumption models within a GIS schema. This has the goal of enabling rapid analysis of energy impacts of the dramatically evolving transportation environment. The application is being developed using a Python framework for ArcGIS Online. Using road network data from the City of Columbus, Ohio and traffic data from CATT Laboratory's Regional Integrated Transportation Information System, transportation energy consumption will be modeled at a macro level. The application aims to provide highly accurate data while still maintaining the flexibility needed to adapt the model when new technologies arise. The methodology presented through this effort is expected to provide insights into how light-duty vehicles use energy on a large scale based on the road network they use. The application will also provide data exports to enable sharing and collaboration. " need for agility in turn calls for sophisticated, intuitive, and reliable means to visualize and share transportation data. Travel is inherently spatial, so methods of visualizing transportation information should also be rooted in spatial analysis. Geographic Information Systems (GIS) are the premier technological framework to analyze and display spatial data. GIS tools have been used in the past to depict flow of vehicles, and display of network conditions (speed, congestion, etc.). However, in an ever-changing technological landscape, spanning advancements in vehicle as well as information systems, depicting vehicle flows falls short of providing a comprehensive picture of the impact these advancements have on travel related energy consumption. To address this issue, this research effort presents a web-based mapping application to visualize how travel related energy flows across a city. The application is being developed by National Renewable Energy Laboratory researchers to integrate various transportation energy consumption models within a GIS schema. This has the goal of enabling rapid analysis of energy impacts of the dramatically evolving transportation environment. The application is being developed using a Python framework for ArcGIS Online. Using road network data from the City of Columbus, Ohio and traffic data from CATT Laboratory's Regional Integrated Transportation Information System, transportation energy consumption will be modeled at a macro level. The application aims to provide highly accurate data while still maintaining the flexibility needed to adapt the model when new technologies arise. The methodology presented through this effort is expected to provide insights into how light-duty vehicles use energy on a large scale based on the road network they use. The application will also provide data exports to enable sharing and collaboration.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2020

Deep learning for in situ data compression of large turbulent flow simulations

As the size of turbulent flow simulations continues to grow, in situ data compression is becoming increasingly important for visualization, analysis, and restart checkpointing. For these applications, single-pass compression techniques with low computational and communication overhead are crucial. In this paper we present a deep-learning approach to in situ compression using an autoencoder architecture that is customized for three-dimensional turbulent flows and is well suited for contemporary heterogeneous computing resources. The autoencoder is compared against a recently introduced randomized single-pass singular value decomposition (SVD) for three different canonical turbulent flows: decaying homogeneous isotropic turbulence, a Taylor-Green vortex, and turbulent channel flow. Our proposed fully convolutional autoencoder architecture compresses turbulent flow snapshots by a factor of 64 with a single pass, allows for arbitrarily sized input fields, is cheaper to compute than the randomized single-pass SVD for typical simulation sizes, performs well on unseen flow configurations, and has been made publicly available. The results reported here show that the autoencoder dramatically outperforms a randomized single-pass SVD with similar compression ratio and yields comparable performance to a higher-rank decomposition with an order of magnitude less compression in regard to preserving a number of important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Refining HPCToolkit for application performance analysis at exascale

As part of the US Department of Energy’s Exascale Computing Project (ECP), Rice University has been refining its HPCToolkit performance tools to better support measurement and analysis of applications executing on exascale supercomputers. To efficiently collect performance measurements of GPU-accelerated applications, HPCToolkit employs novel non-blocking data structures to communicate performance measurements between tool threads and application threads. To attribute performance information in detail to source lines, loop nests, and inlined call chains, HPCToolkit performs parallel analysis of large CPU and GPU binaries involved in the execution of an exascale application to rapidly recover mappings between machine instructions and source code. To analyze terabytes of performance measurements gathered during executions at exascale, HPCToolkit employs distributed-memory parallelism, multithreading, sparse data structures, and out-of-core streaming analysis algorithms. To support interactive exploration of profiles up to terabytes in size, HPCToolkit’s hpcviewer graphical user interface uses out-of-core methods to visualize performance data. The result of these efforts is that HPCToolkit now supports collection, analysis, and presentation of profiles and traces of GPU-accelerated applications at exascale. These improvements have enabled HPCToolkit to efficiently measure, analyze and explore terabytes of performance data for executions using as many as 64K MPI ranks and 64K GPU tiles on ORNL’s Frontier supercomputer. HPCToolkit’s support for measurement and analysis of GPU-accelerated applications has been employed to study a collection of open-science applications developed as part of ECP. This paper reports on these experiences, which provided insight into opportunities for tuning applications, strengths and weaknesses of HPCToolkit itself, as well as unexpected behaviors in executions at exascale.

Adhianto, Laksono↗
DOE OSTI2025

Advanced Interactive 3D Visualization Tool for Customizable Analyses of Tomography Datasets in Material Science

Current methods for visualizing and analyzing 3D tomography datasets in materials science often lack the interactivity and depth required for detailed structural insights. This limitation restricts a researchers' ability to accurately interpret complex data, which is critical for advancing material innovations and understanding structural properties. To address this issue, we have developed a novel, web-based interactive 3D visualization and analysis tool from the Trame framework that offers customizable features to enhance data interpretability. The tool allows users to adjust parameters such as visible range, slice planes, data rotation, and layering, providing a more detailed and dynamic view of complex structures. Its user-friendly web interface increases the accessibility and ease of use for both novice and experienced researchers, to visualize large volumetric datasets. The tool supports a diverse range of data formats, making it versatile for various research applications. Unique capabilities include real-time data manipulation, automated feature detection, context-sensitive feedback, and real-time volume calculations and distributions per sliced region or layer, alongside the ability to quickly generate high-quality screenshots and videos for presentations and reports. These advancements offer a comprehensive solution for enhanced 3D data exploration, significantly improving the analysis process and communication of results in materials science.

36 - MATERIALS SCIENCE↗
DOE OSTI2024

SARS-CoV-2 wastewater variant surveillance: pandemic response leveraging FDA’s GenomeTrakr network

ABSTRACT Wastewater surveillance has emerged as a crucial public health tool for population-level pathogen surveillance. Supported by funding from the American Rescue Plan Act of 2021, the FDA‘s genomic epidemiology program, GenomeTrakr, was leveraged to sequence SARS-CoV-2 from wastewater sites across the United States. This initiative required the evaluation, optimization, development, and publication of new methods and analytical tools spanning sample collection through variant analyses. Version-controlled protocols for each step of the process were developed and published on protocols.io. A custom data analysis tool and a publicly accessible dashboard were built to facilitate real-time visualization of the collected data, focusing on the relative abundance of SARS-CoV-2 variants and sub-lineages across different samples and sites throughout the project. From September 2021 through June 2023, a total of 3,389 wastewater samples were collected, with 2,517 undergoing sequencing and submission to NCBI under the umbrella BioProject,PRJNA757291. Sequence data were released with explicit quality control (QC) tags on all sequence records, communicating our confidence in the quality of data. Variant analysis revealed wide circulation of Delta in the fall of 2021 and captured the sweep of Omicron and subsequent diversification of this lineage through the end of the sampling period. This project successfully achieved two important goals for the FDA’s GenomeTrakr program: first, contributing timely genomic data for the SARS-CoV-2 pandemic response, and second, establishing both capacity and best practices for culture-independent, population-level environmental surveillance for other pathogens of interest to the FDA. IMPORTANCE This paper serves two primary objectives. First, it summarizes the genomic and contextual data collected during a Covid-19 pandemic response project, which utilized the FDA’s laboratory network, traditionally employed for sequencing foodborne pathogens, for sequencing SARS-CoV-2 from wastewater samples. Second, it outlines best practices for gathering and organizing population-level next generation sequencing (NGS) data collected for culture-free, surveillance of pathogens sourced from environmental samples.

Microbiology↗
DOE OSTI2026

Toward Unified Autonomous Scattering Experiments: A Cross-Facility Case Study at ALS and PETRA III

Autonomous experiments rely on the integration of control, data acquisition, analysis, and decision-making frameworks. While such systems have been demonstrated at individual facilities, adapting them to additional instruments remains challenging due to differences in local infrastructure. We present a modular workflow that connects existing open-source tools for data access (Tiled), workflow orchestration (Prefect), analysis and visualization (pyFAI, Plotly Dash), and Gaussian-process-based adaptive sampling (gpCAM) into a unified framework for autonomous scattering experiments. The same configuration operates across two synchrotron beamlines (ALS 7.3.3 and PETRA III P03) with only minimal facility-specific adjustments, as shown in proof-of-concept demonstrations. This validates that a consistent design emphasizing modularity and shared interfaces can ease deployment across diverse experimental environments. The resulting framework provides a flexible foundation for extending autonomous control and analysis capabilities beyond a single beamline or instrument.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Physical discovery in representation learning via conditioning on prior knowledge

Recent advances in electron, scanning probe, optical, and chemical imaging and spectroscopy yield bespoke data sets containing the information of structure and functionality of complex systems. In many cases, the resulting data sets are underpinned by low-dimensional simple representations encoding the factors of variability within the data. The representation learning methods seek to discover these factors of variability, ideally further connecting them with relevant physical mechanisms. However, generally, the task of identifying the latent variables corresponding to actual physical mechanisms is extremely complex. Here, we present an empirical study of an approach based on conditioning the data on the known (continuous) physical parameters and systematically compare it with the previously introduced approach based on the invariant variational autoencoders. The conditional variational autoencoder (cVAE) approach does not rely on the existence of the invariant transforms and hence allows for much greater flexibility and applicability. Interestingly, cVAE allows for limited extrapolation outside of the original domain of the conditional variable. However, this extrapolation is limited compared to the cases when true physical mechanisms are known, and the physical factor of variability can be disentangled in full. We further show that introducing the known conditioning results in the simplification of the latent distribution if the conditioning vector is correlated with the factor of variability in the data, thus allowing us to separate relevant physical factors. We initially demonstrate this approach using 1D and 2D examples on a synthetic data set and then extend it to the analysis of experimental data on ferroelectric domain dynamics visualized via piezoresponse force microscopy.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

Improved Method for Evaluating and Specifying the Chromaticity of Light Sources

This article recommends a method for calculating and specifying light source chromaticity using the CIE 2015 10° color matching functions (CMFs), which, according to analysis of existing psychophysical experiment data, can reduce visual mismatch compared to specifications based on the traditional CIE 1931 2° CMFs. Specifically, this work evaluates, documents, and recommends for adoption a supporting system of measures to be used with the CIE 2015 10° CMFs: a new uniform chromaticity scale (UCS) diagram with coordinates (s, t), a measure of correlated color temperature (CCTst), and a measure of distance from the Planckian locus (Dst). It also presents options for updating nominal classification quadrangles. A complete method of this nature has not yet been standardized, which may be contributing to the slow uptake of the CIE 2015 CMFs. The proposed tools are analogous to u, v, CCT, Duv, and the ANSI C78.377 nominal classification quadrangles that are currently specified in the CIE 1960 UCS using the CIE 1931 2° CMFs. While conceptually equivalent, the differences between the current standard method and the new st system are important for reducing unintended mismatch in the chromaticity of light. Furthermore, the implications of changing chromaticity specification methods are identified by a comparison over a diverse set of real light source SPDs. Finally, as a step toward improving the design and specification of light sources for illumination purposes, we propose a process for implementing the recommendations.

42 ENGINEERING↗
DOE OSTI2023

AEflow (Autoencoder fluid flow compression network) [SWR-22-29]

As the size of turbulent flow simulations continues to grow, in situ data compression is becoming increasingly important for visualization, analysis, and restart checkpointing. For these applications, single-pass compression techniques with low computational and communication overhead are crucial. In this paper we present a deep-learning approach to in situ compression using an autoencoder architecture that is customized for three-dimensional turbulent flows and is well suited for contemporary heterogeneous computing resources. The autoencoder is compared against a recently introduced randomized single-pass singular value decomposition (SVD) for three different canonical turbulent flows: decaying homogeneous isotropic turbulence, a Taylor-Green vortex, and turbulent channel flow. Our proposed fully convolutional autoencoder architecture compresses turbulent flow snapshots by a factor of 64 with a single pass, allows for arbitrarily sized input fields, is cheaper to compute than the randomized single-pass SVD for typical simulation sizes, performs well on unseen flow configurations, and has been made publicly available. The results reported here show that the autoencoder dramatically outperforms a randomized single-pass SVD with similar compression ratio and yields comparable performance to a higher-rank decomposition with an order of magnitude less compression in regard to preserving a number of important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses.

King, Ryan↗
DOE OSTI2022

Web application for machining parameter recommendation using visual analytics

The computer codes create a web application including multiple user interfaces, visualizations, interaction modules between the user, and visualizations. The visualizations will be interactive, allowing users to query the data directly, and they will be arranged in a dashboard. When selections are made in one visualization, filters will be applied to all other visualizations providing a coordinated system for asking questions about the data. The visualization can be utilized for manufacturing data management, analysis, and recommendation.

Chae, Junghoon↗
DOE OSTI2023

ML for microbiomes

The software provides machine learning analysis and visualization to detect patterns in microbiome data, including topic modeling, probabilistic graphical modeling, conventional machine learning methods, and deep learning. The software is written in python and R, it uses some python and R libraries as well as big open-source libraries like sklearn, networkX, pytorch (python), pgmpy (python), and bnlearn (R). It also has a script to use for MALLET and DTM (open-source packages for topic modeling, written in Java).

Kim, Anastasiia↗
DOE OSTI2024

Machine learning tools for epigenetics

The software provides machine learning analysis and visualization to detect patterns in epigenetic data, including conventional machine learning and statistical methods, and open-source packages like pyBigWig (https://github.com/deeptools/pyBigWig) for data processing. The software is written in python, it uses some python libraries.

Kim, Anastasiia↗
DOE OSTI2020

RETRO RX

RETRO Rx is an R&D 100 Gold Award-winning suite of tools that provides a new class of analytics in the space between traditional epidemiological models and unanalyzed spreadsheets of data. Unlike traditional models, these tools provide analysts, scientists, practitioners, and decision makers a way to easily and effectively do sophisticated analyses using a simple user interface. AIDO guides users at the early stages of a disease outbreak by comparing the current parameters to historic outbreaks. RED Alert constructs a broad historical picture of disease incidence (recent outbreak included) to detect disease reemergence at its earliest stages, locally and globally. RETRO Rx compiles data and presents analysis in a clear, concise visual format to support disease mitigation and prevention planning. Both are available currently as web-based apps. A prototype of the RED Alert mobile app with enhanced functionality is available for beta testing and AIDO mobile app development is planned.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

The ARM Data-oriented Metrics and Diagnostics Package for Climate Models - A New Tool for Evaluating Climate Models with Field Data

The U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) program User Facility produces ground-based long-term continuous unique measurements for atmospheric state, precipitation, turbulent fluxes, radiation, aerosol, cloud and the land surface, which are collected at multiple sites. These comprehensive datasets have been widely used to calibrate climate models and are proven to be invaluable for climate model development and improvement. This article introduces an evaluation package to facilitate the use of ground-based ARM measurements in climate model evaluation. The ARM data-oriented metrics and diagnostics package (ARM-DIAGS) includes both ARM observational datasets and a Python-based analysis toolkit for computation and visualization. The observational datasets are compiled from multiple ARM data products and specifically tailored for use in climate model evaluation. In addition, ARM-DIAGS also includes simulation data from models participating the Coupled Model Inter-comparison Project (CMIP), which will allow climate-modeling groups to compare a new, candidate version of their model to existing CMIP models. The analysis toolkit is designed to make the metrics and diagnostics quickly available to the model developers.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Blue Keanu: A Scientific Visualization Tool For Network Data

This software allows the user to visualize complex PCAP-ng files captured from network capture software such as Wireshark. The visualization runs in a GUI window that can be zoomed or moved to areas of interest in a waterfall type display. The user then can see an area of interest that looks different than the typical traffic visually, such as a human interaction or non-repetitive area of data. The program will tell the user the packet number and byte offset of interest for fast analysis of discrete atomic or non-random events. This is particularly useful for visualization of unknown binary format data, such as in PLC or SCADA protocols that may have human or other non-repetitive activity for further analysis, reverse engineering, or fast forensic analysis.

Durller, MichaelGeorge↗
DOE OSTI2020

A terminology for in situ visualization and analysis systems

The term "in situ processing" has evolved over the last decade to mean both a specific strategy for visualizing and analyzing data and an umbrella term for a processing paradigm. The resulting confusion makes it difficult for visualization and analysis scientists to communicate with each other and with their stakeholders. To address this problem, a group of over 50 experts convened with the goal of standardizing terminology. This paper summarizes their findings and proposes a new terminology for describing in situ systems. An important finding from this group was that in situ systems are best described via multiple, distinct axes: integration type, proximity, access, division of execution, operation controls, and output type. This paper discusses these axes, evaluates existing systems within the axes, and explores how currently used terms relate to the axes.

97 MATHEMATICS AND COMPUTING↗