Ease of use and deployment for a fast, scalable data movement infrastructure
Explore the source record for details and available documents.
42 ENGINEERING↗
SEARCH · Engineering Papers
Results for “data analysis, visualization”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A Practical Solver for Scalar Data Topological Simplification
This paper presents a practical approach for the optimization of topological simplification, a central pre-processing step for the analysis and visualization of scalar data. Given an input scalar field f and a set of “signal” persistence pairs to maintain, our approaches produces an output field g that is close to f and which optimizes (i) the cancellation of “non-signal” pairs, while (ii) preserving the “signal” pairs. In contrast to pre-existing simplification algorithms, our approach is not restricted to persistence pairs involving extrema and can thus address a larger class of topological features, in particular saddle pairs in three-dimensional scalar data. Our approach leverages recent generic persistence optimization frameworks and extends them with tailored accelerations specific to the problem of topological simplification. Extensive experiments report substantial accelerations over these frameworks, thereby making topological simplification optimization practical for real-life datasets. Our approach enables a direct visualization and analysis of the topologically simplified data, e.g., via isosurfaces of simplified topology (fewer components and handles). We apply our approach to the extraction of prominent filament structures in three-dimensional data. Specifically, we show that our pre-simplification of the data leads to practical improvements over standard topological techniques for removing filament loops. Here, we also show how our approach can be used to repair genus defects in surface processing. Finally, we provide a C++ implementation for reproducibility purposes.
97 MATHEMATICS AND COMPUTING↗
epicsuite(EAS)
Software, Documentation, Tutorials, Testing data, Example data for processing, analysis, filtering, querying, visualization and otherwise transforming genomic data for scientific analysis and discovery.
Rogers, David H.↗
Continuous Emulation and Multiscale Visualization of Traffic Flow Using Stationary Roadside Sensor Data
With the advent of the next-generation traffic monitoring systems, there has been a significant increase in the spatial-temporal resolution of vehicle mobility data in many cities. Effective analysis and visualization of such data can provide transportation planners with data-driven insights, which can facilitate the understanding of multiscale traffic dynamics. In this paper, we present a web-based traffic emulator for emulating and visualizing near-real-time and historical traffic flows on highways using data from road-side sensors. To construct a continuous traffic flow, the emulator adopts an analytical pipeline that can (a) integrate traffic data collected from discrete road-side radar detection sensors, (b) interpolate traffic conditions (vehicle speed and volume) on unmeasured road segments based on traffic flow theory, and (c) generate lane-specific vehicle trajectories and movements using a mathematically optimized representation of the road network. Our app also provides an integrated visual workflow that allows users to explore the interconnected traffic dynamics using an appropriate traffic flow visualization selected based on the level of detail. We devise two innovative geo-visualization techniques that utilize an animated strips-network representation and a lane usage matrix to visualize lane performances. To ensure a smooth emulation of large-scale traffic flow in an easy-to-access web environment, we implement the emulator using client-side GPU-accelerated techniques. Lastly, we close with a case study that visualizes traffic dynamics of two scenarios - an afternoon peak hour and a traffic accident - in Chattanooga, Tennessee. Our app visualizes the responses of traffic dynamics during different traffic conditions, and to the presence of the traffic accident at different spatial scales.
42 ENGINEERING↗
Visualization Within the Department of Energy: NREL IEEE VIS Application Spotlight
This presentation highlights the role of advanced visualization techniques at the National Renewable Energy Laboratory (NREL) in supporting cutting-edge research across diverse energy domains. From immersive analytics and uncertainty visualization to high-resolution and real-time data analysis, NREL's visualization capabilities enable scientists to explore complex datasets more effectively. These tools are critical for advancing research in materials science, renewable energy technologies, biofuels, electric vehicle infrastructure, energy efficiency - from industrial processes to entire communities - and then bringing these innovations to practice through energy systems integration. NREL's visualization tools drive innovation across renewable energy and grid modernization efforts by providing deeper insights and improving decision-making.
grid modernization↗
3D Model Visual Verification and Mesh-Based Data Analysis in Fulcrum
Explore the source record for details and available documents.
Lefebvre, Robert Alexander↗
Sensitivity Analysis of Drivers Water Shortage in the Los Angeles Region During Drought
The code and detailed step-by-step instructions for generating the model output data, processing results, and analysis and plotting are provided at https://github.com/IMMM-SFA/Ferencz_et_al_2026_ER_Water. The PyArtes model is a python adaptation of the Artes model. PyArtes uses many of the same input data and optimization model architecture as Artes. Documentation for the PyArtes model is provided in the Supplement to the paper. The primary data product are simulated monthly water shortages for indoor and outdoor demand under a large ensemble of drought scenarios (>13,000). The droughts are hypothetical and are not based on historical time series data of supply sources - though historical data did help inform ranges explored for supply parameters. Demands are informed by recent 2017-2021 water supply data. Demands used for the model can be accessed at https://github.com/IMMM-SFA/Ferencz_et_al_2026_ER_Water. Simulations resolve demand for over 90 water providers in the study region. The results report 36 months of water shortage data for each indoor and outdoor demand node. The study also developed a multilayer perceptron (MLP) neural network trained on a subset of the simulated shortage ensemble to emulate worst annual water shortage for a given set of parameter multipliers -- provided the parameter values fall within the ranges sampled in the ensemble. Emulated water shortages for synthetic ensembles are in the MLP-generated shortages folder. The MLP model was used to generate larger ensembles to support Sobol analysis that would have been extremely computationally expensive to simulate. Datasets provided in this repository*: Simulated shortages. These results are used for the analysis for Figures 5, 8, and 9 in the paper, and also to train the MLP emulator. .zip file containing outputs for the 13,312 scenario ensemble. Separate .csv files for indoor and outdoor shortage for each scenario. Rows = demand ids (~100), Columns = months (36) Units = acre-feet/month of shortage (shortage = monthly demand - supply). 1 acft = 1233.48 m^3 .csv files of aggregated shortages derived from the 13,312 ensemble Rows = scenarios (13,312), Columns = demand ids (~100) Units = acre-feet/year (either worst annual shortage or total shortage over the 3-year drought) .csv file of the parameter multipliers scenarios for the ensemble .csv file of the parameter ranges and baseline values the multipliers were applied to MLP-generated shortages. These results are used for Figures 4, 6, and 7 in the paper. mwd higher folder: scenario ensembles, emulated worst year total shortages (acft), and Sobol results Emulated shortages. Rows = scenarios, columns = demand ids, units acft Sobol results. Rows = demand ids, columns Sobol (S1, ST, or 95% confidence interval) value for each parameter mwd lower folder: scenario ensembles, emulated worst year total shortages (acft), and Sobol results same organization as mwd higher MLP performance: performance metrics (R^2, RMSE, BIAS, MAPE) for the testing subset (20% or 2,662 scenarios) and simulated vs emulated worst year shortage (acre-feet/year) for every demand node, MWD wholesale regions, and the entire study region (LAC). Supporting data for figures. Figure plotting scripts in the associated GitHub repo. These files support analysis and visualization. Geospatial Data used for plotting simulated water shortages and Sobol results. Dictionary of full names for demand nodes in the model and estimates of water supply by source type informed by Artes input files and California Urban Water Management Planning data: https://water.ca.gov/Programs/Water-Use-And-Efficiency/Urban-Water-Use-Efficiency/Urban-Water-Management-Plans *Readme files provided for each folder.
drought↗
Machine learning to discover mineral trapping signatures due to CO 2 injection
Mineral trapping is pursued as a geological CO 2 sequestration (GCS) mechanism because it permanently stores CO 2 in solid phases or minerals. However, CO 2 mineral-trapping mechanisms are poorly understood due to (1) lack of sufficient field and laboratory data characterizing these complex processes, and (2) challenges to develop site-specific reactive-transport models coupling fluid flow and geochemical reactions occurring at various temporal (from milliseconds to years) and spatial (from pore (millimeters) to field (kilometers)) scales. Reactive transport with additional complexities such as heterogeneity can make the simulation outputs even more difficult to interpret because of complex nonlinearity and multi-scale interdependencies. Furthermore, the values of model outputs such as concentrations can vary by several orders of magnitude, making it harder to correlate and characterize the impact of the variables via traditional data interpretation techniques such as exploratory data analyses. Recently, machine learning (ML) has shown promise in feature discovery and in highlighting hidden mechanisms that cannot be obtained by existing data-analytics and statistical methods. In this study, we applied an unsupervised ML approach, non-negative matrix factorization with custom -means clustering (NMF) to the data generated by reactive-transport simulations of GCS. The reactive-transport data consisted of 19 attributes, including four physio-chemical variables (pH, porosity, aqueous CO 2 , and sequestered CO 2 ), six chemical species (K + , Na + , HCO, Ca 2+ , Mg 2+ , Fe 2+ ), and four carbonate minerals (calcite, dolomite, siderite, and ankerite), a feldspar mineral (albite), and four clay minerals (illite, clinochlore, kaolinite, and smectite) over a period of 200 years of simulation time. Furthermore, the simulation data used was for Morrow B sandstone at the Farnsworth hydrocarbon unit in Texas. Data are sampled at two locations within the model domain: (1) at the injection well and (2) 200 m west of the injection well. The injection was performed for a period of 10 years. Using NMF, we estimated the temporal interdependencies among the 19 attributes over a span of 200 years. We found that NMF was able to identify four reaction stages and their dominant attributes; these cannot be directly discerned through traditional visualization (e.g., line plots, Pareto analysis, Glyph-based visualization methods) or exploratory data analysis tools of the simulation data. The four stages were: reactions in the injection phase followed by short-, mid-, and long-term reactions. The NMF analysis also revealed that 10 among the 19 attributes are dominant. These dominant attributes for mineral trapping include calcite, dolomite at injection well, siderite at 200 m away from the injection well, clinochlore, kaolinite, Na + , K + , Ca 2+ , Mg 2+ , pH, and aqeuous CO 2 . Finally, at late times (65–200 years), our results showed that calcite plays a major role in mineral trapping with insignificant contribution from siderite, ankerite, and clay minerals. These findings make the proposed unsupervised ML-model attractive for reactive-transport sensing towards real-time GCS monitoring.
54 ENVIRONMENTAL SCIENCES↗
XtalCAMP: a comprehensive program for the analysis and visualization of scanning Laue X-ray micro-/nanodiffraction data
XtalCAMP is a software package based on the MATLAB platform, which is suitable for, but not limited to, the analysis and visualization of scanning Laue X-ray micro-/nanodiffraction data. The main objective of the software is to provide complementary functionalities to the Laue indexing software packages used at several synchrotron beamlines. Here, the graphical user interfaces allow the easy analysis of characteristic microstructure features, including real-time intensity mapping for a quick examination of phase, grain and defect distribution, 2D color-coded mapping of microstructural properties from the output of other Laue indexing software, crystal orientation visualization, grain boundary characterization based on orientation/misorientation calculation, principal strain/stress analysis, and strain ellipsoid representation, as well as a series of additional toolkits. As an example, XtalCAMP is applied to the microstructural investigation of a solution-heat-treated Ni-based superalloy manufactured using a laser 3D-printing technique, and a deformed natural quartzite from Val Bregaglia in the Central Alps.
36 MATERIALS SCIENCE↗
Near real-time streaming analysis of big fusion data
Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.
70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
A Scoping Review of Mixed Initiative Visual Analytics in the Automation Renaissance
Artificial agents are increasingly integrated into data analysis workflows, carrying out tasks that were primarily done by humans. Our research explores how the introduction of automation recalibrates the dynamic between humans and automating technology. To explore this question, we conducted a scoping review encompassing twenty years of mixed-initiative visual analytic systems. To describe and contrast the relationship between humans and automation, we developed an integrated taxonomy to delineate the objectives of these mixed-initiative visual analytics tools, how much automation they support, and the assumed roles of humans. Here, we describe our qualitative approach of integrating existing theoretical frameworks with new codes we developed. Our analysis shows that the visualization research literature lacks consensus on the definition of mixed-initiative systems and explores a limited potential of the collaborative interaction landscape between people and automation. Our research provides a scaffold to advance the discussion of human-AI collaboration during visual data analysis. Our integrated taxonomy is available in the form of a web application on https://smonadjemi.github.io/miva.
Monadjemi, Shayan [ORNL] (ORCID:0000000293855969)↗
Optimization and Augmentation for Data Parallel Contour Trees
Contour trees are used for topological data analysis in scientific visualization. While originally computed with serial algorithms, recent work has introduced a vector-parallel algorithm. Furthermore, this algorithm is relatively slow for fully augmented contour trees which are needed for many practical data analysis tasks. We therefore introduce a representation called the hyperstructure that enables efficient searches through the contour tree and use it to construct a fully augmented contour tree in data parallel, with performance on average 6 times faster than the state-of-the-art parallel algorithm in the TTK topological toolkit.
97 MATHEMATICS AND COMPUTING↗
Top Research Challenges and Opportunities for Near Real-Time Extreme-Scale Visualization of Scientific Data
The rapid advancement in scientific simulations and experimental facilities has resulted in the generation of vast amounts of data at unprecedented scales. The analysis and visualization of large amounts of data is a challenge in and of itself, but the requirements for timeliness significantly magnify these difficulties. Near real-time visualization is critical to monitor and analyze the data produced by these large facilities, but current production tools are not well-suited to these requirements. In this position paper, we share our perspective on some of the challenges, and thus, opportunities for research that stand in the way of near-real-time visualization of large scientific data.
Pugmire, Dave↗
Thermally driven phase transition of halide perovskites revealed by big data-powered in situ electron microscopy
Halide perovskites are promising light-absorbing materials for high-efficiency solar cells, while the crystalline phase of halide perovskites may influence the device’s efficiency and stability. In this work, we investigated the thermally driven phase transition of perovskite (CsPbIxBr3—x), which was confirmed by electron diffraction and high-resolution transmission electron microscopy results. CsPbIxBr3—x transitioned from δ phase to α phase when heated, and the γ phase was obtained when the sample was cooled down. The γ phase was stable as long as it was isolated from humidity and air. A template matching-based data analysis method enabled visualization of the thermally driven phase evolution of perovskite during heating. Here, we also proposed a possible atomic movement in the process of phase transition based on our in situ heating experimental data. The results presented here may improve our understanding of the thermally driven phase transition of perovskite as well as provide a protocol for big-data analysis of in situ experiments.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Rubin Observatory LSST Tutorials
A collection of tutorials -- both Jupyter notebooks and documentation-based --- demonstrating data access, analysis, and visualization techniques for the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST). These tutorials are intended for astronomers, educators, and data scientists working with LSST data products.
Vera C. Rubin Observatory, NSF-DOE↗
FREDA: A Web Application for the Processing, Analysis, and Visualization of Fourier‐Transform Mass Spectrometry Data
The high-resolution measurement capability of Fourier-transform mass spectrometry (FT-MS) has made it a necessity for exploring the molecular composition of complex organic mixtures, like soil, plant, aquatic, and petroleum samples. This demand has driven a need for informatics tools to explore and analyze FT-MS data in a robust and reproducible manner. FREDA is an interactive web application developed to enable spectrometrists to format, process, and explore their FT-MS data without the need for statistical programming expertise. FREDA was built to explore outputs from a molecular identification tool, like CoreMS, and provide a suite of methods to filter data, compute chemical properties of peaks, statistically compare samples and groups of samples, conduct exploratory data analysis, and download the results with a report detailing all steps conducted. To demonstrate the utility of FREDA, an example analysis was conducted using FT-MS data from a soil microbiology study of samples collected in two different soil depths at the Sphagnum bog forest north of Grand Rapids, Minnesota. Differences between the two depths are observed using Kendrick, Gibbs free energy, and van Krevelen plots. G-tests are used to quantify a significant difference between the groups. All analyses and plotting are conducted using only the FREDA application. FREDA is an open-source and readily available web application that allows users to explore and make statistically valid conclusions about their FT-MS data. The application is available online (https://map.emsl.pnnl.gov/app/freda) with a tutorial web series (https://youtu.be/k5HLE2kNSBY?si=yB6sGoyvzxrFf5MP) and freely accessible code on Github (https://github.com/EMSL-Computing/FREDA).
47 OTHER INSTRUMENTATION↗
Streaming Data in HPC Workflows Using ADIOS
The “IO Wall” problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS “engine”, accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.
Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗
PyKrev: A Python Library for the Analysis of Complex Mixture FT-MS Data
In this study, we present PyKrev, a Python library for the analysis of complex mixture Fourier transform mass spectrometry (FT-MS) data. PyKrev is a comprehensive suite of tools for analysis and visualization of FT-MS data after formula assignment has been performed. These comprise formula manipulation and calculation of chemical properties, intersection analysis between multiple lists of formulas, calculation of chemical diversity, assignment of compound classes to formulas, multivariate analysis, and a variety of visualization tools producing van Krevelen diagrams, class histograms, PCA score, and loading plots, biplots, scree plots, and UpSet plots. The library is showcased through analysis of hot water green tea extracts and Scotch whisky FT-ion cyclotron resonance-MS data sets. PyKrev addresses the lack of a single, cohesive toolset for researchers to perform FT-MS analysis in the Python programming environment encompassing the most recent data analysis techniques used in the field.
47 OTHER INSTRUMENTATION↗