Programming and Performance Visualization Tools
Explore the source record for details and available documents.
Computer science↗
SEARCH · Engineering Papers
Results for “Visual Analytics Environments”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Build and Execute Environment
At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.
Guan, Qiang↗
Build and Execute Environment
At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.
Guan, Qiang↗
Simulation Exploration through Immersive Parallel Planes
We present a visualization-driven simulation system that tightly couples systems dynamics simulations with an immersive virtual environment to allow analysts to rapidly develop and test hypotheses in a high-dimensional parameter space. To accomplish this, we generalize the two-dimensional parallel-coordinates statistical graphic as an immersive 'parallel-planes' visualization for multivariate time series emitted by simulations running in parallel with the visualization. In contrast to traditional parallel coordinate's mapping the multivariate dimensions onto coordinate axes represented by a series of parallel lines, we map pairs of the multivariate dimensions onto a series of parallel rectangles. As in the case of parallel coordinates, each individual observation in the dataset is mapped to a polyline whose vertices coincide with its coordinate values. Regions of the rectangles can be 'brushed' to highlight and select observations of interest: a 'slider' control allows the user to filter the observations by their time coordinate. In an immersive virtual environment, users interact with the parallel planes using a joystick that can select regions on the planes, manipulate selection, and filter time. The brushing and selection actions are used to both explore existing data as well as to launch additional simulations corresponding to the visually selected portions of the input parameter space. As soon as the new simulations complete, their resulting observations are displayed in the virtual environment. This tight feedback loop between simulation and immersive analytics accelerates users' realization of insights about the simulation and its output.
97 MATHEMATICS AND COMPUTING↗
Visual HPC Workflows for the Analysis of System Dynamics Models
Visual analytics supported by high performance computing (HPC) accelerates and enhances the discovery, exploration, and analysis of causal patterns in complex system dynamics (SD) models. We present a suite of visualization-assisted ensemble-based techniques for hypothesis generation and testing, and for sensitivity analysis. By employing HPC to provide parallel, on-demand simulation of SD models, one can “steer” an ensemble of simulated scenarios in real time as one first formulates and then informally tests those hypotheses: this provides rapid feedback for analysts to refine their understanding of the causal relationships emergent from a model. Such understandings can be followed and augmented by rigorous application of statistical methods, namely global variance-based sensitivity analysis, Monte-Carlo filtering, adaptive regional sensitivity analysis, and self-organized maps: here timely computation relies on HPC, while effective presentation emphasizes high-dimensional multivariate data visualization. Immersive visualization in virtual 3D environments provides an excellent adjunct to the traditional 2D graphics typically used for SD models, as it generates an embodied understanding of model behavior and facilitates an active, collaborative critique of model structure and output. Finally, we summarize prospects for HPC-enabled visual analytics applied to SD modeling.
29 ENERGY PLANNING, POLICY, AND ECONOMY↗
Edge ML for CAN bus intrusion detection in AVs
Autonomous Vehicles (AVs) are revolutionizing transportation, but their reliance on interconnected cyber-physical systems exposes them to unprecedented cybersecurity risks. This study addresses the critical challenge of detecting real-time cyber intrusions in self-driving vehicles by leveraging a dataset from the Udacity self-driving car project. We simulate four high-impact attack vectors, Denial of Service (DoS), spoofing, replay, and fuzzy attacks, by injecting noise into spatial features (e.g., bounding box coordinates) to replicate adversarial scenarios. We develop and evaluate two lightweight neural network architectures (NN-1 and NN-2) alongside a logistic regression baseline (LG-1) for intrusion detection. The models achieve exceptional performance, with NN-2 attaining an AUC score of 93.15% and 93.15% accuracy, demonstrating their suitability for edge deployment in AV environments. Through explainable AI techniques, we uncover unique forensic fingerprints of each attack type, such as spatial corruption in fuzzy attacks and temporal anomalies in replay attacks, offering actionable insights for feature engineering and proactive defense. Visual analytics, including confusion matrices, ROC curves, and feature importance plots, validate the models' robustness and interpretability. This research sets a new benchmark for AV cybersecurity, delivering a scalable, field-ready toolkit for Original Equipment Manufacturers (OEMs) and policymakers. By aligning intrusion fingerprints with SAE J3061 automotive security standards, we provide a pathway for integrating machine learning into safety-critical AV systems. Our findings underscore the urgent need for security-by-design AI, ensuring that AVs not only drive autonomously but also defend autonomously. This work bridges the gap between theoretical cybersecurity and life-preserving engineering, offering a leap toward safer, more secure autonomous transportation.
97 MATHEMATICS AND COMPUTING↗
Chapter 21: The Utility of Virtual Reality for Science and Engineering
In our daily usage of the large-scale immersive virtual environment at the National Renewable Energy Laboratory (NREL), we have observed how this VR system can be a useful tool to enhance scientific and engineering workflows. On multiple occasions, we have observed scientists and engineers discover features in their data using immersive environments that they had not seen in prior investigations of their data on traditional desktop displays. We have embedded more information into our analytics tools, allowing engineers to explore complex multivariate spaces. We have observed natural interactions with 3D objects and how those interactions seem to catalyze understanding. And we have seen improved collaboration with groups of stakeholders. In this chapter, we discuss these practical advantages of immersive visualization in the context of several real-world examples.
97 MATHEMATICS AND COMPUTING↗
GUIDE: A Scalable Information Directory Service to Collect, Federate, and Analyze Logs for Operational Insights into a Leadership HPC Facility
In this paper, we describe the GUIDE framework to collect, federate, and analyze log data from the Oak Ridge Leadership Computing Facility's (OLCF), and how we use it derive insights into center operations. We collectsystem logs and extract monitoring data at every level of the various OLCF subsystems, and have developed a suite of pre-processing tools to make the raw data consumable. The cleansed logs are then ingested and federated into a central, scalable data warehouse, Splunk, that offers storage, indexing, querying, and visualization capabilities. We have further developed and deployed a set of analytics tools to analyze these multiple disparate log streams in concert, and derive operational insights. We describe our experience from developing and deploying the GUIDE infrastructure, and deriving valuable insights on the various subsystems, based on two years of operations in the production OLCF environment.
Vazhkudai, Sudharshan↗
Immersive Analytics in Critical Spatial Domains: From Materials to Energy Systems
Immersive analytics (IA) leverages virtual reality, augmented reality, and mixed reality to transform how users interact with complex datasets across domains such as science, industry, and education. These immersive technologies offer spatial and multimodal environments that foster intuitive exploration, but they also introduce challenges related to cognitive load, interface design, and system performance. Here, this article presents a comprehensive review of visualization techniques, interaction models, and multimodal inputs utilized in IA. Drawing on case studies in scientific visualization, industrial training, and educational communication, we examine both the potential and limitations of current systems. Finally, we propose future research directions, focusing on real‐time collaboration, adaptive user interfaces, and scalable data exploration strategies to advance the field.
99 - GENERAL AND MISCELLANEOUS↗
PIPER: Performance Insights for Programmers and Exascale Runtimes
During the lifetime of the project, the PIPER at the University of Utah worked in collaboration with the team from the partner institutions to establish a set of software tools for analyzing and improving the performance parallel computing tools. These activities complement and complete the work performed by the other X-Stack 2 projects with the definition of a common data models and computing infrastructures, which have been a central element in the targeted performance analysis pipeline and new programming models. The University of Utah team have focused mainly on developing data analysis and visualization systems to facilitate exploration, comparison and evaluation of the behavior and performance of complex algorithms in modern HPC infrastructures. This is involved several demonstrations of the performance achieved and presentation of the results in a way that facilitates the users understand bottlenecks and ways to overcome them. One major focus of the University of Utah activities has been the analysis of the performance characteristics of Fat-Tree and Dragonfly topology networks. The work involved the development of practical visualization systems especially in collaboration with the team at LLNL, and published several papers and gave invited talks at prestigious venues such as the Salishan Conference on High-Speed Computing or the International Conference on High Performance Computing, Data, and Analytics. In addition, the university of Utah team has advanced the core activities on analyzing the performance characteristics of I/O and other data movement systems and their practical use for scaling Uintah and other simulation codes poised for use in the exascale computing environments of the future.
42 ENGINEERING↗
Automated Transformation of Lignin Topologies into Atomic Structures with LigninBuilder
Lignin is an abundant aromatic heteropolymer found in secondary plant cell walls and is a potential feedstock for conversion into bioderived fuels and chemicals. Lignin chemical diversity complicates traditional structural studies, and so, relatively little experimental evidence exists for how lignin structure exists in aqueous solution or how lignin polymers respond to changes in their chemical environment. Molecular modeling can address these concerns; however, prior computational structural lignin models typically did not capture lignin heterogeneity, as only a few polymers were considered. LigninBuilder creates a framework for building structural libraries for lignin from existing topological libraries, permitting significantly greater diversity of lignin structures to be sampled at atomic detail. As a demonstration of its capabilities, LigninBuilder was applied to three libraries of lignin from hardwood, softwood, and grass, and the resulting polymer structures were simulated in an aqueous environment. The lignins adopted compact globular structures, as would be expected for polymers in poor solvents. The differences between the libraries were largest when quantifying the packing of nonadjacent aromatic residues, with greater branching within the polymer resulting in poorer aromatic packing. Individual lignin polymers were also found to undergo rapid conformational changes, with the dwell time within a state growing as the square of the molecular weight. This first application of LigninBuilder demonstrates the potential for atomic-level insight into lignin interactions. LigninBuilder is distributed as a plugin to the visualization software VMD, lowering the barrier for modeling lignin structure in diverse environments.
09 BIOMASS FUELS↗
Updates to the Spatial Analysis and Decision Assistance (SADA) Freeware Tool - 16583
SADA is a collaboration between the University of Tennessee and Oak Ridge National Laboratory to present a freeware desktop application integrating environmental risk analytics, spatial modeling, and decision sciences for use at EPA, NRC, DOE, and DoD contaminated sites. The software embeds risk assessment, uncertainty modeling, and downstream decision processes within a spatial context. Updates to Version 6 will provide will include a rich GIS environment and a new 3-D visualization model which permits immersive visualization of contamination, with topology, and the built environment. The risk models presented are developed in conjunction with Superfund to implement all scenarios in their Regional Screening Levels (RSLs) and radionuclide preliminary remediation goals (PRGs). In addition, SADA will be able to ingest high resolution LandScan population data providing insights into the proximity of population to contamination. A brief overview of the progress in provided along with anticipated next steps. (authors)
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Predictive Big Data Analytics: A Study of Parkinson’s Disease Using Large, Complex, Heterogeneous, Incongruent, Multi-Source and Incomplete Observations
Background A unique archive of Big Data on Parkinson’s Disease is collected, managed and disseminated by the Parkinson’s Progression Markers Initiative (PPMI). The integration of such complex and heterogeneous Big Data from multiple sources offers unparalleled opportunities to study the early stages of prevalent neurodegenerative processes, track their progression and quickly identify the efficacies of alternative treatments. Many previous human and animal studies have examined the relationship of Parkinson’s disease (PD) risk to trauma, genetics, environment, co-morbidities, or life style. The defining characteristics of Big Data–large size, incongruency, incompleteness, complexity, multiplicity of scales, and heterogeneity of information-generating sources–all pose challenges to the classical techniques for data management, processing, visualization and interpretation. We propose, implement, test and validate complementary model-based and model-free approaches for PD classification and prediction. To explore PD risk using Big Data methodology, we jointly processed complex PPMI imaging, genetics, clinical and demographic data. Methods and Findings Collective representation of the multi-source data facilitates the aggregation and harmonization of complex data elements. This enables joint modeling of the complete data, leading to the development of Big Data analytics, predictive synthesis, and statistical validation. Using heterogeneous PPMI data, we developed a comprehensive protocol for end-to-end data characterization, manipulation, processing, cleaning, analysis and validation. Specifically, we (i) introduce methods for rebalancing imbalanced cohorts, (ii) utilize a wide spectrum of classification methods to generate consistent and powerful phenotypic predictions, and (iii) generate reproducible machine-learning based classification that enables the reporting of model parameters and diagnostic forecasting based on new data. We evaluated several complementary model-based predictive approaches, which failed to generate accurate and reliable diagnostic predictions. However, the results of several machine-learning based classification methods indicated significant power to predict Parkinson’s disease in the PPMI subjects (consistent accuracy, sensitivity, and specificity exceeding 96%, confirmed using statistical n-fold cross-validation). Clinical (e.g., Unified Parkinson's Disease Rating Scale (UPDRS) scores), demographic (e.g., age), genetics (e.g., rs34637584, chr12), and derived neuroimaging biomarker (e.g., cerebellum shape index) data all contributed to the predictive analytics and diagnostic forecasting. Conclusions Model-free Big Data machine learning-based classification methods (e.g., adaptive boosting, support vector machines) can outperform model-based techniques in terms of predictive precision and reliability (e.g., forecasting patient diagnosis). We observed that statistical rebalancing of cohort sizes yields better discrimination of group differences, specifically for predictive analytics based on heterogeneous and incomplete PPMI data. UPDRS scores play a critical role in predicting diagnosis, which is expected based on the clinical definition of Parkinson’s disease. Even without longitudinal UPDRS data, however, the accuracy of model-free machine learning based classification is over 80%. The methods, software and protocols developed here are openly shared and can be employed to study other neurodegenerative disorders (e.g., Alzheimer’s, Huntington’s, amyotrophic lateral sclerosis), as well as for other predictive Big Data analytics applications.
Science & Technology - Other Topics↗
Simulation process and data flow for a large system dynamics model
This paper documents the workflow and supporting technologies that a large system dynamics model, the biomass scenario model, employs to streamline the data preparation, simulation, quality control, and analysis process at the National Renewable Energy Laboratory. The workflow centers on automation of routine aspects of the flow of data between data stores, simulations, and visualizations. It enforces quality checks on data, reproducibility of computations, and traceability of results, while maintaining complete archives of modeling and analysis artifacts. The resulting frictionless simulation/analysis environment supports large-scale sensitivity analysis, interactive creation of ensembles of simulations, and rapid visualization-based exploration of simulation results.
09 BIOMASS FUELS↗
Towards Adaptive Decision Support: A Perspective from Intelligent and Annotated Visual Analytics for Exploring Big Urban Mobility Data
Explore the source record for details and available documents.
Xu, Haowen↗
Exploration and Analysis of Legacy Data at the Hanford Site Using PHOENIX - 19044
Radioactive contaminants pose risks to human health and the environment. These risks are influenced by factors such as disposal/release and transport mechanisms, location in the environment, potential exposure receptors, and the presence of comingled contaminants. Understanding the behavior, transport, and distribution of radionuclides in the environment is essential to facilitating risk-informed decision-making related to remedial design, action, and monitoring. These decisions are based on significant amounts of complex data that must be accessed, integrated, and analyzed to effectively evaluate data and formulate appropriate conclusions and recommendations. Often, data volume, complexity, and legacy access issues pose severe challenges for analysts, regulators, and stakeholders interested in evaluating scientific data to support decision-making. To address these challenges at the U.S. Department of Energy Hanford Site, Pacific Northwest National Laboratory (PNNL) has developed PHOENIX (PNNL-Hanford Online Environmental Information exchange) to facilitate risk-informed decision-making by streamlining investigative and analytical processes through the integration and visualization of previously isolated datasets created under distinct regulatory or operational regimes, each serving a specific purpose. With the large volume of data dispersed across many Hanford environmental databases, finding the correct sources of relevant data, accessing them, and relating them in a meaningful way can prove challenging and time consuming. By analyzing Hanford databases, performing substantial optimization using modern data processing methods and developing extensive queries on the refactored data, PHOENIX enables users to access, integrate, display, visualize, and retrieve data that would have been historically unfeasible to examine. The data and information accessible through PHOENIX help inform remediation-related assessments and evaluations, predict and quantify the risks that radioactive and chemical contaminants pose to human health and the environment, and communicate this information to decision makers responsible for risk-based remediation actions. (authors)
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Machine Intelligence-Centered System for Automated Characterization of Functional Materials and Interfaces
Classic design of experiment relies on a time-intensive workflow that requires planning, data interpretation, and hypothesis building by experienced researchers. Here, in this paper, we describe an integrated, machine-intelligent experimental system which enables simultaneous dynamic tests of electrical, optical, gravimetric, and viscoelastic properties of materials under a programmable dynamic environment. Specially designed software controls the experiment and performs on-the-fly extensive data analysis and dynamic modeling, real-time iterative feedback for dynamic control of experimental conditions, and rapid visualization of experimental results. The system operates with minimal human intervention and enables time-efficient characterization of complex dynamic multifunctional environmental responses of materials with simultaneous data processing and analytics. The system provides a viable platform for artificial intelligence (AI)-centered material characterization, which, when coupled with an AI-controlled synthesis system, could lead to accelerated discovery of multifunctional materials.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Visualizing the iron atom exchange front in the Fe(II)-catalyzed recrystallization of goethite by atom probe tomography
The autocatalytic redox interaction between aqueous Fe(II) and Fe(III)-(oxyhydr)oxide minerals such as goethite and hematite leads to rapid recrystallization marked, in principle, by an atom exchange (AE) front, according to bulk iron isotopic tracer studies. However, direct evidence for this AE front has been elusive given the analytical challenges of mass-resolved imaging at the nanoscale on individual crystallites. We report successful isolation and characterization of the AE front in goethite microrods by 3D atom probe tomography (APT). The microrods were reacted with Fe(II) enriched in tracer 57 Fe at conditions consistent with prior bulk studies. APT analyses and 3D reconstructions on cross-sections of the microrods reveal an AE front that is spatially heterogeneous, at times penetrating several nanometers into the lattice, in a manner consistent with defect-accelerated exchange. Evidence for exchange along microstructural domain boundaries was also found, suggesting another important link between exchange extent and initial defect content. The findings provide an unprecedented view into the spatial and temporal characteristics of Fe(II)-catalyzed recrystallization at the atomic scale, and substantiate speculation regarding the role of defects controlling the dynamics of electron transfer and AE interaction at this important redox interface.
Taylor, Sandra D.↗