Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Visualization and Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

A Visual Analytics Approach to Structured Data Analysis to Enhance Nonproliferation and Arms Control Verification Activities

Analysis activities for Nonproliferation and Arms Control verification require the use of many types of data. Tabular structured data, such as Excel spreadsheets and relational databases, have traditionally been used for data mining activities, where specific queries are issued against data to look for matching results. The application of visual analytics tools to structured data enables further exploration of datasets to promote discovery of previously unknown results. This paper discusses the application of a specific visual analytics tool to datasets related to the field of Arms Control and Nonproliferation to promote the use of visual analytics more broadly in this domain. Visual analytics focuses on analytical reasoning facilitated by interactive visual interfaces (Wong and Thomas 2004). It promotes exploratory analysis of data, and complements data mining technologies where known patterns can be mined for. Also with a human in the loop, they can bring in domain knowledge and subject matter expertise. Visual analytics has not widely been applied to this domain. In this paper, we will focus on one type of data: structured data, and show the results of applying a specific visual analytics tool to answer questions in the Arms Control and Nonproliferation domain. We chose to use the T.Rex tool, a visual analytics tool developed at PNNL, which uses a variety of visual exploration patterns to discover relationships in structured datasets, including a facet view, graph view, matrix view, and timeline view. The facet view enables discovery of relationships between categorical information, such as countries and locations. The graph tool visualizes node-link relationship patterns, such as the flow of materials being shipped between parties. The matrix visualization shows highly correlated categories of information. The timeline view shows temporal patterns in data. In this paper, we will use T.Rex with two different datasets to demonstrate how interactive exploration of the data can aid an analyst with arms control and nonproliferation verification activities. Using a dataset from PIERS (PIERS 2014), we will show how container shipment imports and exports can aid an analyst in understanding the shipping patterns between two countries. We will also use T.Rex to examine a collection of research publications from the IAEA International Nuclear Information System (IAEA 2014) to discover collaborations of concern. We hope this paper will encourage the use of visual analytics structured data analytics in the field of nonproliferation and arms control verification. Our paper outlines some of the challenges that exist before broad adoption of these kinds of tools can occur and offers next steps to overcome these challenges.

arms control↗

LevSeq: Rapid Generation of Sequence-Function Data for Directed Evolution and Machine Learning

Sequence-function data provides valuable information about the protein functional landscape but is rarely obtained during directed evolution campaigns. Here, we present Long-read every variant Sequencing (LevSeq), a pipeline that combines a dual barcoding strategy with nanopore sequencing to rapidly generate sequence-function data for entire protein-coding genes. LevSeq integrates into existing protein engineering workflows and comes with open-source software for data analysis and visualization. The pipeline facilitates data-driven protein engineering by consolidating sequence-function data to inform directed evolution and provide the requisite data for machine learning-guided protein engineering (MLPE). LevSeq enables quality control of mutagenesis libraries prior to screening, which reduces time and resource costs. Simulation studies demonstrate LevSeq’s ability to accurately detect variants under various experimental conditions. Lastly, we show LevSeq’s utility in engineering protoglobins for new-to-nature chemistry. Widespread adoption of LevSeq and sharing of the data will enhance our understanding of protein sequence-function landscapes and empower data-driven directed evolution.

59 BASIC BIOLOGICAL SCIENCES↗

Empirical Study of Focus-Plus-Context and Aggregation Techniques for the Visualization of Streaming Data

Analysis of streaming data often involves both real-time monitoring of incoming data as well as contextual awareness of data history. A focus-plus-context approach can support both goals, with variable levels of visual aggregation making it possible to provide a high level of detail for incoming and recent data while providing contextual information about recent history. Visual aggregation reduces data resolution in order to show the context of data over large periods of time within a limited display space. With a controlled experiment, we evaluated the effectiveness of different types of aggregation for four types of stream-analysis tasks. Overall, the results show that a focus-plus-context design has little negative impact on the ability to successfully monitor and analyze streaming data, making it possible to show longer periods of time than other approaches. However, visual aggregation can be problematic for trend recognition tasks. This research demonstrates how the effectiveness of the visualization depends on the specifics of the analysis task.

Ragan, Eric↗

Using GIS for Processing, Analysis and Visualization of Hydrological Model Data - 16202

The Applied Research Center at Florida International University (ARC-FIU) is supporting the soil and groundwater remediation efforts of the U.S. Department of Energy's (DOE's) Savannah River Site (SRS) by developing a surface water model to simulate the hydrology and the fate and transport of contaminants and sediment in the Tims Branch watershed. Advances in ArcGIS software through the development of spatial GIS database (geo-database) technology, coupled with the development of data models such as ArcHydro which possesses a spatial relational database management system (RDMS) schema and relationship structure specific to hydrologic systems, provides modelers with tools and applications to assist in the processing, analysis and visualization of flow and contaminant transport data. In addition, the coupling of this type of geo-database structure with a numerical modeling package such as MIKE SHE/MIKE 11 can serve as an efficient tool that significantly reduces the time needed for data preparation. The MIKE modeling system has a geographic information systems (GIS) user interface built into its system that can directly use GIS data for model inputs. GIS-based hydrologic models can provide a spatial element that other hydrologic models lack. GIS is important for processing and analysis of large spatio-temporal datasets derived from multiple sources that are being used for setting up of models, model calibration and simulation of flow and contaminant fate and transport. GIS-based approaches in hydrological modeling can also provide the benefit of combining different layers of geographic data to create new integrated information which can be quite useful for creating dependent or independent hydrological variables. ARC-FIU has developed an ArcGIS geo-database to support the hydrological modeling work being conducted at SRS. The SRS geo-database was developed based on the ArcHydro and ArcGIS Base Map data models with modifications made for project specific input parameters. The significance of this approach was to ensure its replicability for potential application at other DOE sites. The SRS geo-database assists in the storage and management of GIS and time series data being used for hydrological model development at Savannah River Site. Other GIS tools such as ArcGIS Model Builder coupled with Python scripting have enabled the automation of many of the repetitive geo-processing tasks required for pre- and post-processing of the hydrological modeling data. Automation of many of the data processing tasks has facilitated faster and hence more complex analyses of field test data. The toolbox created is also a scalable and reusable application that can be implemented in other watersheds. This paper describes the use of ArcGIS tools for the pre-processing, spatial delineation and analysis of hydrological model data; for automation of repetitive geo-processing tasks; and for visualization of different hydrological components of the Tims Branch watershed. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

The ADAQ framework: An integrated toolkit for data acquisition and analysis with real and simulated radiation detectors

Here, the ADAQ framework is a collection of software tools that is designed to streamline the acquisition and analysis of radiation detector data produced in modern digital data acquisition (DAQ) systems and in Monte Carlo detector simulations. The purpose of the framework is to maximize user scientific pro-ductivity by minimizing the effort and expertise required to fully utilize radiation detectors in a variety of scientific and engineering disciplines. By using a single set of tools to span the real and simulation domains, the framework eliminates redundancy and provides an integrated workflow for high-fidelity comparison between experimental and simulated detector performance. Built on the ROOT data analysis framework, the core of the ADAQ framework is a set of Cþþ and Python libraries that enable high-level control of digital DAQ systems and detector simulations with data stored into standardized binary ROOT files for further analysis. Two graphical user interface programs utilize the libraries to create powerful tools: ADAQAcquisition handles control and readout of real-world DAQ systems and ADAQAnalysis provides data analysis and visualization methods for experimental and simulated data. At present, the ADAQ framework supports digital DAQ hardware from CAEN S.p.A. and detector simulations performed in Geant4; however, the modular design will facilitate future extension to other manufacturers and simulation platforms.

detector simulation↗

GNPS Dashboard: collaborative exploration of mass spectrometry data in the web browser

Access to web-based platforms has enabled scientists to perform research remotely. A critical aspect of mass spectrometry data analysis is the inspection, analysis, and visualization of the raw data to validate data quality and confirm statistical observations. We developed the GNPS Dashboard, a web-based data visualization tool, to facilitate synchronous collaborative inspection, visualization, and analysis of private and public mass spectrometry data remotely.

59 BASIC BIOLOGICAL SCIENCES↗

Distributed Data-Flow for In-Situ Visualization and Analysis at Petascale

We conducted a feasibility study to research modifications to data-flow architectures to enable data-flow to be distributed across multiple machines automatically. Distributed data-flow is a crucial technology to ensure that tools like the VisIt visualization application can provide in-situ data analysis and post-processing for simulations on peta-scale machines. We modified a version of VisIt to study load-balancing trade-offs between light-weight kernel compute environments and dedicated post-processing cluster nodes. Our research focused on memory overheads for contouring operations, which involves variable amounts of generated geometry on each node and computation of normal vectors for all generated vertices. Each compute node independently decided whether to send data to dedicated post-processing nodes at each stage of pipeline execution, depending on available memory. We instrumented the code to allow user settable available memory amounts to test extremely low-overhead compute environments. We performed initial testing of this prototype distributed streaming framework, but did not have time to perform scaling studies at and beyond 1000 compute-nodes.

97 MATHEMATICS AND COMPUTING↗

Conquering Data Chaos: Research Data Management with Kubernetes

Managing massive volumes of data and effectively making it accessible to researchers poses significant challenges and is a barrier to scientific discovery. In many cases, critical data is locked up in unwieldy file formats or one-off databases and is too large to effectively process on a single machine. This talk explores the role of Kubernetes, an open-source container orchestration platform, in addressing research data management challenges. I will discuss how we are using a set of publicly available open-source and home-grown tools in the National Renewable Energy Lab (NREL) Data, Analysis, and Visualization (DAV) group to help researchers overcome data-related bottlenecks. The talk will begin by providing an overview of the data challenges faced in research data management, including data storage, processing, and analysis. I will highlight Kubernetes' ability to handle large-scale data by leveraging containerization and distributed computing, including distributed storage. Kubernetes allows researchers to encapsulate data processing infrastructure and workflows into portable containers, enabling reproducibility and ease of deployment. Kubernetes can then schedule and manage the resource allocation of these containers to enable efficient utilization of limited computing resources, leading to more efficient data processing and analysis. I will discuss some limitations of traditional, siloed approaches to dealing with data and emphasize the need for solutions which foster collaboration. I will highlight how we are using Kubernetes at NREL to facilitate data sharing and cooperation among research teams. Kubernetes' flexible architecture enables the deployment of shared computing environments, such as Apache Superset, where researchers can seamlessly access and analyze shared datasets. Providing the ability to have one research team easily consume data generated by another, utilizing Kubernetes' as a central data platform, is one of the major wins we've encountered by adopting the platform. Finally, I will showcase real-world use cases from NREL where we have used Kubernetes to solve some persistent data challenges involving large volumes of sensor and monitoring data. I will discuss the challenges we encountered when creating our cluster and making it available as a production-ready resource. I will also discuss the specific suite of tools, including Postgres and Apache Druid for columnar and timeseries data, and Redpanda Kafka for streaming data we have deployed in our infrastructure, and the process that went into the selection of these tools.

collaborative environment↗

The Neurodata Without Borders ecosystem for neurophysiological data science

The neurophysiology of cells and tissues are monitored electrophysiologically and optically in diverse experiments and species, ranging from flies to humans. Understanding the brain requires integration of data across this diversity, and thus these data must be findable, accessible, interoperable, and reusable (FAIR). This requires a standard language for data and metadata that can coevolve with neuroscience. We describe design and implementation principles for a language for neurophysiology data. Our open-source software (Neurodata Without Borders, NWB) defines and modularizes the interdependent, yet separable, components of a data language. We demonstrate NWB’s impact through unified description of neurophysiology data across diverse modalities and species. NWB exists in an ecosystem, which includes data management, analysis, visualization, and archive tools. Thus, the NWB data language enables reproduction, interchange, and reuse of diverse neurophysiology data. More broadly, the design principles of NWB are generally applicable to enhance discovery across biology through data FAIRness.

59 BASIC BIOLOGICAL SCIENCES↗

Parallel In Situ Coupling of a Simulation with a Fully Featured Visualization System

There is a widening gap between compute performance and the ability to store computation results. Complex scientific codes are the most affected since they must save massive files containing meshes and fields for offline analysis. Time and storage costs instead dictate that data analysis and visualization be combined with the simulations themselves, being done in situ so data are transformed to a manageable size before they are stored. Earlier approaches to in situ processing involved combining specific visualization algorithms into the simulation code, limiting flexibility. We introduce a new library which instead allows a fully-featured visualization tool, VisIt, to request data as needed from the simulation and apply visualization algorithms in situ with minimal modification to the application code.

99 GENERAL AND MISCELLANEOUS↗

Introduction

Welcome to the Conference on Visualization and Data Analysis (VDA) 2011. The scope of VDA covers all topics and areas of data visualization and visual analytics. Since the first VDA conference in 1994, the annual event has grown steadily into a major venue for visualization researchers and practitioners from around the world to present their work and share their experiences.

97 MATHEMATICS AND COMPUTING↗

Towards Exascale: High Performance Visualization and Analytics – Project Status Report

The LBNL Visualization research program focuses on a key set of problems facing our community as we move towards the exascale regime of computing. First, we aim to better understand how to effectively take advantage of an evolving architectural landscape where there are an increasing number of cores per processor by examining algorithmic reformulation and alternative methods of programming these increasingly complex platforms. Second, as the gap widens between computational and I/O capacity, it is increasingly imperative to perform more visual data exploration and analysis operations concurrent with the simulation to avoid increasingly expensive I/O. Our approach is to examine several interrelated issues aimed at minimizing the memory footprint, data movement, and design/implementation issues related to deploying tools that can have broad applicability across many different simulation codes and science domains. Third, increasing computational power produces larger and more complex scientific data, and our research program focuses on new technologies aimed at enabling scientific data understanding with ever-larger data. Here, we explore how multi-resolution data representation techniques can reduce data movement, and study novel approaches that focus visualization and analysis processing on subsets of data that are “scientifically interesting,” thereby also minimizing data movement while accelerating scientific knowledge discovery.

97 MATHEMATICS AND COMPUTING↗

Fiber Uncertainty Visualization for Bivariate Data With Parametric and Nonparametric Noise Models

Visualization and analysis of multivariate data and their uncertainty are top research challenges in data visualization. Constructing fiber surfaces is a popular technique for multivariate data visualization that generalizes the idea of level-set visualization for univariate data to multivariate data. Here, in this paper, we present a statistical framework to quantify positional probabilities of fibers extracted from uncertain bivariate fields. Specifically, we extend the state-of-the-art Gaussian models of uncertainty for bivariate data to other parametric distributions (e.g., uniform and Epanechnikov) and more general nonparametric probability distributions (e.g., histograms and kernel density estimation) and derive corresponding spatial probabilities of fibers. In our proposed framework, we leverage Green's theorem for closed-form computation of fiber probabilities when bivariate data are assumed to have independent parametric and nonparametric noise. Additionally, we present a nonparametric approach combined with numerical integration to study the positional probability of fibers when bivariate data are assumed to have correlated noise. For uncertainty analysis, we visualize the derived probability volumes for fibers via volume rendering and extracting level sets based on probability thresholds. We present the utility of our proposed techniques via experiments on synthetic and simulation datasets.

97 MATHEMATICS AND COMPUTING↗

BASTet: Shareable and Reproducible Analysis and Visualization of Mass Spectrometry Imaging Data via OpenMSI

Mass spectrometry imaging (MSI) is a transformative imaging method that supports the untargeted, quantitative measurement of the chemical composition and spatial heterogeneity of complex samples with broad applications in life sciences, bioenergy, and health. While MSI data can be routinely collected, its broad application is currently limited by the lack of easily accessible analysis methods that can process data of the size, volume, diversity, and complexity generated by MSI experiments. The development and application of cutting-edge analytical methods is a core driver in MSI research for new scientific discoveries, medical diagnostics, and commercial-innovation. However, the lack of means to share, apply, and reproduce analyses hinders the broad application, validation, and use of novel MSI analysis methods. To address this central challenge, we introduce the Berkeley Analysis and Storage Toolkit (BASTet), a novel framework for shareable and reproducible data analysis that supports standardized data and analysis interfaces, integrated data storage, data provenance, workflow management, and a broad set of integrated tools. Based on BASTet, we describe the extension of the OpenMSI mass spectrometry imaging science gateway to enable web-based sharing, reuse, analysis, and visualization of data analyses and derived data products. Here, we demonstrate the application of BASTet and OpenMSI in practice to identify and compare characteristic substructures in the mouse brain based on their chemical composition measured via MSI.

97 MATHEMATICS AND COMPUTING↗

Intuitive Visualizations Through Multi-Domain Projections for Performance Analysis at Scale

Performance analysis of parallel scientific codes is becoming increasingly difficult due to the rapidly growing complexity of applications and architectures. Existing tools fall short in providing intuitive views to reveal the root causes of performance problems. We have developed a new paradigm of projecting and visualizing performance data obtained from one domain onto other domains for faster, more intuitive analysis of applications. We gather performance data in three domains: hardware, application, and communication. For each domain, we define projections that allow the data to be viewed in the other domains. This framework is called the HAC model. The model not only allows us to directly compare the data across domains, but also allows us to use data visualization and analysis tools available in the other domains. Using the methodologies and visualization techniques mentioned above, we have demonstrated the careful unscrambling of otherwise tangled measurements caused by adaptive systems. By attributing performance measurements directly to their causes, we are able to visualize performance measurements in the domains most intuitive to the user, which are not necessarily those in which measurements are collected.

97 MATHEMATICS AND COMPUTING↗

Synchrophasor Visual Integration and Event Evaluation for Utilities (SynchroVIEEU) with High Penetration of Renewables

The Hawaiian Electric Companies (Companies), operating independent grid systems on the islands of Oahu, Hawaii, Maui, Molokai and Lanai, have been contending with some of the highest levels of renewable generation from both utility-scale and distributed generation resources. Renewable penetration levels exceeding 50% of generation are some of the highest in the nation and are changing the system characteristics and requiring operational changes to grid management. The Companies have aggressively leveraged federal, multi-state and industry grant funding to demonstrate new technologies for monitoring, visualization, forecasting and analysis to inform transformation needs and to improve operational awareness, especially at the distribution level. As part of the American Recovery and Reinvestment Act (ARRA) supported efforts to facilitate integration of renewables, synchrophasor devices were deployed on the islands of Hawaii and Maui in 2010. The devices have potential to provide Hawaii Electric Light Company (HELCO) on Hawaii Island and Maui Electric Company (MECO) with additional information to support post-event analysis and to help quantify observed area-wide changes in voltage and frequency on the systems. Consistent with the U.S. Department of Energy’s (DOE) Office of Electricity Delivery and Energy Reliability (OE) FOA objectives, the goal of this project was to accelerate the integration of synchrophasor information into production grade data visualization and analysis platforms/models to inform timely, efficient and reliable operations of a modern grid in light of high penetrations of renewable resources. The project comprised a two-year development effort to investigate the integration and visualization of synchrophasor field data into production-ready transmission and distribution proactive modeling efforts and a common data analysis platform application currently being used by the Companies. Conceptually, the project was about getting the right data to the right people at the right time in an easily accessible and timely environment.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DynaM: Dynamic Multiresolution Data Representation for Large-Scale Scientific Analysis

Fast growing large-scale systems enable scientific applications to run at a much larger scale and accordingly produce gigantic volumes of simulation output. Such data imposes a grand challenge to post-processing tasks such as visualization and data analysis, because these tasks are often performed at a host machine that is remotely located and equipped with much less memory and storage resources. During the simulation runs, it is also desirable for scientists to be able to interactively monitor and steer the progress of simulation. This requires scientific data to be represented in an efficient form for initial exploration and computation steering. In this paper, we propose DynaM a software framework that can represent scientific data in a multiresolution form, and dynamically organize data blocks into an optimized layout for efficient scientific analysis. DynaM supports a convolution-based multiresolution data representation for abstracting scientific data for visualization at a wide spectrum of resolution. To support the efficient generation and retrieval of different data granularities from such representation, a dynamic data organization in DynaM is enabled to cater distinct peculiarities of different size data blocks for efficient and balanced I/O performance. Our experimental results demonstrate that DynaM can efficiently represent large scientific dataset and speed up the visualization of multidimensional scientific data. An up to 29 times speedup is achieved on Jaguar supercomputer at Oak Ridge National Laboratory.

Tian, Yuan↗