Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data visualization and analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Synchrophasor Visual Integration and Event Evaluation for Utilities (SynchroVIEEU) with High Penetration of Renewables

The Hawaiian Electric Companies (Companies), operating independent grid systems on the islands of Oahu, Hawaii, Maui, Molokai and Lanai, have been contending with some of the highest levels of renewable generation from both utility-scale and distributed generation resources. Renewable penetration levels exceeding 50% of generation are some of the highest in the nation and are changing the system characteristics and requiring operational changes to grid management. The Companies have aggressively leveraged federal, multi-state and industry grant funding to demonstrate new technologies for monitoring, visualization, forecasting and analysis to inform transformation needs and to improve operational awareness, especially at the distribution level. As part of the American Recovery and Reinvestment Act (ARRA) supported efforts to facilitate integration of renewables, synchrophasor devices were deployed on the islands of Hawaii and Maui in 2010. The devices have potential to provide Hawaii Electric Light Company (HELCO) on Hawaii Island and Maui Electric Company (MECO) with additional information to support post-event analysis and to help quantify observed area-wide changes in voltage and frequency on the systems. Consistent with the U.S. Department of Energy’s (DOE) Office of Electricity Delivery and Energy Reliability (OE) FOA objectives, the goal of this project was to accelerate the integration of synchrophasor information into production grade data visualization and analysis platforms/models to inform timely, efficient and reliable operations of a modern grid in light of high penetrations of renewable resources. The project comprised a two-year development effort to investigate the integration and visualization of synchrophasor field data into production-ready transmission and distribution proactive modeling efforts and a common data analysis platform application currently being used by the Companies. Conceptually, the project was about getting the right data to the right people at the right time in an easily accessible and timely environment.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DynaM: Dynamic Multiresolution Data Representation for Large-Scale Scientific Analysis

Fast growing large-scale systems enable scientific applications to run at a much larger scale and accordingly produce gigantic volumes of simulation output. Such data imposes a grand challenge to post-processing tasks such as visualization and data analysis, because these tasks are often performed at a host machine that is remotely located and equipped with much less memory and storage resources. During the simulation runs, it is also desirable for scientists to be able to interactively monitor and steer the progress of simulation. This requires scientific data to be represented in an efficient form for initial exploration and computation steering. In this paper, we propose DynaM a software framework that can represent scientific data in a multiresolution form, and dynamically organize data blocks into an optimized layout for efficient scientific analysis. DynaM supports a convolution-based multiresolution data representation for abstracting scientific data for visualization at a wide spectrum of resolution. To support the efficient generation and retrieval of different data granularities from such representation, a dynamic data organization in DynaM is enabled to cater distinct peculiarities of different size data blocks for efficient and balanced I/O performance. Our experimental results demonstrate that DynaM can efficiently represent large scientific dataset and speed up the visualization of multidimensional scientific data. An up to 29 times speedup is achieved on Jaguar supercomputer at Oak Ridge National Laboratory.

Tian, Yuan↗

Extreme-Scale Distribution-Based Data Analysis

Scientists overview and identify regions of interest by transforming data into compact information descriptors that characterize simulation results and allow detailed analysis on demand. Among many existing feature descriptors, statistical information derived from data samples is a promising approach to taming the big data avalanche because data distributionscomputed from a population can compactly describe the presence and characteristics of salient data features with minimal data movement. The ability to computationally summarize and process data using distributions also provides an efficient and representative capture of information that can adjust to size and resource constraints, with the added benefit that uncertainty associated with results can be quantified and communicated. In this project, we have developed a distribution-based data analysis and visualization framework for optimizing in situ processing and analysis of extreme-scale scientific data.

97 MATHEMATICS AND COMPUTING↗

Architecture for interoperable software in biology

Understanding biological complexity demands a combination of high-throughput data and interdisciplinary skills. One way to bring to bear the necessary combination of data types and expertise is by encapsulating domain knowledge in software and composing that software to create a customized data analysis environment. To this end, simple flexible strategies are needed for interconnecting heterogeneous software tools and enabling data exchange between them. Drawing on our own work and that of others, we present several strategies for interoperability and their consequences, in particular, a set of simple data structures—list, matrix, network, table and tuple—that have proven sufficient to achieve a high degree of interoperability. We provide a few guidelines for the development of future software that will function as part of an interoperable community of software tools for biological data analysis and visualization.

59 BASIC BIOLOGICAL SCIENCES↗

Final Technical Report for: A Unified Data-Driven Approach for Programming In Situ Analysis and Visualization

The placement and movement of data is becoming the key limiting factor on both performance and energy efficiency of high performance computations. As systems generate more data, it is becoming increasingly difficult to actually move that data elsewhere for post-processing, as the rate of improvements in supporting I/O infrastructure is not keeping pace. Together, these trends are creating a shift in how we think about exascale computations, from a viewpoint that focuses on FLOPS to one that focuses on data and data-centric operations as fundamental to the reasoning about, and optimization of, scientific workflows on extreme-scale architectures. The overarching goal of our effort was the study of a unified data-driven approach for programming applications and in situ analysis and visualization. Our work was to understand the interplay between data-centric programming model requirements at extreme-scale and the overall impact of those requirements on the design, capabilities, flexibility, and implementation details for both applications and the supporting in situ infrastructure. In this context, we made many improvements to the Legion programming system (one of the leading data-centric models today) and demonstrated in situ analyses on real application codes using these improvements.

97 MATHEMATICS AND COMPUTING↗

FY08 LDRD Final Report LOCAL: Locality-Optimizing Caching Algorithms and Layouts

This project investigated layout and compression techniques for large, unstructured simulation data to reduce bandwidth requirements and latency in simulation I/O and subsequent post-processing, e.g. data analysis and visualization. The main goal was to eliminate the data-transfer bottleneck - for example, from disk to memory and from central processing unit to graphics processing unit - through coherent data access and by trading underutilized compute power for effective bandwidth and storage. This was accomplished by (1) designing algorithms that both enforce and exploit compactness and locality in unstructured data, and (2) adapting offline computations to a novel stream processing framework that supports pipelining and low-latency sequential access to compressed data. This report summarizes the techniques developed and results achieved, and includes references to publications that elaborate on the technical details of these methods.

99 GENERAL AND MISCELLANEOUS↗

Improving Performance of M-to-N Processing and Data Redistribution in In Transit Analysis and Visualization

In an in transit setting, a parallel data producer, such as a numerical simulation, runs on one set of ranks M, while a data consumer, such as a parallel visualization application, runs on a different set of ranks N. One of the central challenges in this in transit setting is to determine the mapping of data from the set of M producer ranks to the set of N consumer ranks. This is a challenging problem for several reasons, such as the producer and consumer codes potentially having different scaling characteristics and different data models. The resulting mapping from M to N ranks can have a significant impact on aggregate application performance. In this work, we present an approach for performing this M-to-N mapping in a way that has broad applicability across a diversity of data producer and consumer applications. We evaluate its design and performance with a study that runs at high concurrency on a modern HPC platform. By leveraging design characteristics, which facilitate an “intelligent” mapping from M-to-N, we observe significant performance gains are possible in terms of several different metrics, including time-to-solution and amount of data moved.

Loring, Burlen↗

Improving Performance of M-to-N Processing and Data Redistribution in In Transit Analysis and Visualization

In an in transit setting, a parallel data producer, such as a numerical simulation, runs on one set of ranks M, while a data consumer, such as a parallel visualization application, runs on a different set of ranks N: One of the central challenges in this in transit setting is to determine the mapping of data from the set of M producer ranks to the set of N consumer ranks. This is a challenging problem for several reasons, such as the producer and consumer codes potentially having different scaling characteristics and different data models. The resulting mapping from M to N ranks can have a significant impact on aggregate application performance. In this work, we present an approach for performing this M-to-N mapping in a way that has broad applicability across a diversity of data producer and consumer applications. We evaluate its design and performance with a study that runs at high concurrency on a modern HPC platform. By leveraging design characteristics, which facilitate an ''intelligent'' mapping from M-to-N, we observe significant performance gains are possible in terms of several different metrics, including time-to-solution and amount of data moved.

Loring, Burlen↗

Xi-CAM v1.2.3

Xi-CAM aims to provide a community driven platform for multimodal analysis in synchrotron science. The platform core provides a robust plugin infrastructure for extensibility, allowing continuing development to simply add further functionality. Current modules include tools for characterization with (GI)SAXS, Tomography, and XAS. This will continue to serve as a development base as algorithms for multimodal analysis develop. Seamless remote data access, visualization and analysis are key elements of Xi-CAM, and will become critical to synchrotron data infrastructure as expectations for future data volume and acquisition rates rise with continuously increasing throughputs. The highly interactive design elements of Xi-cam will similarly support a generation of users which depend on immediate data quality feedback during high-throughput or burst acquisition modes.

Pandolfi, Ronald↗

SCULPT (Supervised Clustering and Uncovering Latent Patterns with Training) v1

SCULPT (Supervised Clustering and Uncovering Latent Patterns with Training) is a comprehensive data visualization and analysis application focused on working with COLTRIMS (COLd Target Recoil Ion Momentum Spectroscopy) data, which is used in atomic and molecular physics experiments. The application offers several powerful features: - Data uploading and processing capabilities for COLTRIMS files - Multiple visualization methods using UMAP (Uniform Manifold Approximation and Projection) for dimensionality reduction - Interactive selection of data points across multiple views - Feature engineering through various methods: - Manual feature selection from calculated physics parameters - Deep autoencoder for dimension reduction - Genetic programming for discovering meaningful features - Mutual information-based feature selection - Multiple clustering approaches (DBSCAN, KMeans, Agglomerative) - Quality metrics for evaluating clustering results - Export capabilities for selections and generated features

Daoud, Hazem [Lawrence Berkeley National Laborator↗

Visual Analytics: Building a Vibrant and Resilient National Science

Five years after the science of visual analytics was formally established, we attempt to use two different studies to assess the current state of the community and evaluate the progress the community has made in the past few years. The first study involves a comparison analysis of intellectual and scholastic accomplishments recently made by the visual analytics community. The second one aims to measure the degree of community reach and internet penetration of visual-analytics-related resources. This paper describes our efforts to harvest the study data, conduct analysis, and make interpretations based on parallel comparisons with five other established computer science areas.

97 MATHEMATICS AND COMPUTING↗

A Topological Framework for the Interactive Exploration of Large Scale Turbulent Combustion

The advent of highly accurate, large scale volumetric simulations has made data analysis and visualization techniques an integral part of the modern scientific process. To develop new insights from raw data, scientists need the ability to define features of interest in a flexible manner and to understand how changes in the feature definition impact the subsequent analysis of the data. Therefore, simply exploring the raw data is not sufficient. This paper presents a new topological framework for the analysis of large scale, time-varying, turbulent combustion simulations. It allows the scientists to explore interactively the complete parameter space of fuel consumption thresholds for an entire time-dependent combustion simulation. By computing augmented merge trees and their corresponding data segmentations, the system allows the user complete flexibility to segment, select, and track burning cells through time thanks to a linked view interface. We developed this technique in the context of low-swirl turbulent pre-mixed flame simulation analysis, where the topological abstractions enable an efficient tracking through time of the burning cells and provide new qualitative and quantitative insights into the dynamics of the combustion process.

99 GENERAL AND MISCELLANEOUS↗

Rendering the Topological Spines

Many tools to analyze and represent high dimensional data already exits yet most of them are not flexible, informative and intuitive enough to help the scientists make the corresponding analysis and predictions, understand the structure and complexity of scientific data, get a complete picture of it and explore a greater number of hypotheses. With this in mind, N-Dimensional Data Analysis and Visualization (ND²AV) is being developed to serve as an interactive visual analysis platform with the purpose of coupling together a number of these existing tools that range from statistics, machine learning, and data mining, with new techniques, in particular with new visualization approaches. My task is to create the rendering and implementation of a new concept called topological spines in order to extend ND²AV's scope. Other existing visualization tools create a representation preserving either the topological properties or the structural (geometric) ones because it is challenging to preserve them both simultaneously. Overcoming such challenge by creating a balance in between them, the topological spines are introduced as a new approach that aims to preserve them both. Its render using OpenGL and C++ and is currently being tested to further on be implemented on ND²AV. In this paper I will present what are the Topological Spines and how they are rendered.

97 MATHEMATICS AND COMPUTING↗

Improving Scientific Data Analysis Through Multi-touch Enabled Interactive Data Visualization with Applications to Neutron Science

Scientific data analysis is an inherently exploratory process that requires flexible human interaction with data visualization and anal- ysis tools. In this paper, we describe challenges in neutron scattering data analysis and present our approach for allowing more effective human interaction. In addition to multi-touch enabled data visualiza- tion techniques, we describe color scale enhancements, layouts, and visual analytics approaches for 3D volume slicing. Since challenges in neutron science are echoed in most scientific fields, we generalize the project objective and present a case for bridging the gap between the latest advances in interactive data visualization and practical scientific scenarios where new critical knowledge is buried in large volumes of complex data.

Steed, Chad↗

In-situ sampling of a large-scale particle simulation for interactive visualization and analysis

We propose storing a random sampling of data from large scale particle simulations, such as the Roadrunner Universe MC{sup 3} cosmological simulation, to be used for interactive post-analysis and visualization. Simulation data generation rates will continue to be far greater than storage bandwidth rates and other limiting technologies by many orders of magnitude. This implies that only a very small fraction of data generated by the simulation can ever be stored and subsequently post-analyzed. The limiting technology in this situation is analogous to the problem in many population surveys: there aren't enough human resources to query a large population. To cope with the lack of resources, statistical sampling techniques are used to create a representative data set of a large population. Mirroring that situation, we propose to store a simulation-time random sampling of the particle data to cope with the bOlllenecks and support interactive, exploratory post-analysis. The particle samples are immediately stored in a level-ol-detail format for post-visualization and analysis, which amortizes the cost of post-processing for interactive visualization. Additionally, we incorporate a system for recording and visualizing sample approximation error information for confidence and importance highlighting.

97 MATHEMATICS AND COMPUTING↗

Scalable Computation of Topological Abstractions for Scalar Data

Topological data analysis has become an important tool for large scale scalar data analysis and visualization, efficiently extracting the inherent structure and features of interest of the data. However, with growing dataset sizes and complexity, it is increasingly becoming infeasible to compute topological abstractions of interest in serial and on single machines. This paper presents the state of the art in the scalable computation of topological abstractions on scalar data, in shared memory parallel on single machines, and in distributed memory parallel on multiple machines. We highlight results for set‐based, graph‐based and complex‐based abstractions and organize the state of the art based on this taxonomy. The paper identifies parallelization and distribution techniques common in topological algorithms and highlights further areas of interest with underdeveloped efforts.

97 MATHEMATICS AND COMPUTING↗

Parallel visualization and analysis with paraview on a Cray XT4

Scientific data sets produced by modern supercomputers like ORNL's Cray XT 4, Jaguar, can be extremely large, making visualization and analysis more difficult as moving large resultant data to dedicated analysis systems can be prohibitively expensive. We share our continuing work of integrating a parallel visualization system, ParaView, on ORNL's Jaguar system and our efforts to enable extreme scale interactive data visualization and analysis. We will discuss porting challenges and present performance numbers. Jaguar is a Cray XT4 with 7,832 compute nodes each with 4 cores and 8GB of memory, located in Oak Ridge, Tennessee. Jaguar is currently ranked No.8 in the Top 500. Jaguar, and other machines like it, enable increasingly larger data sets. Whether scientists are producing extremely massive data or small data they still have a requirement for analysis. Analysis is often difficult for larger data and it is further complicated when the user is distant to the data. Data sets that are too large to ship and/or analyze on a single desktop require a parallel analysis tool and a high performance computer to run on. Running ParaView interactively on Jaguar is a positive improvement for both local and remote users. We have shown that it is feasible for our user at LANL to visualize his simulation data created on jaguar. Local users may face fewer obstacles and get improved performance, due to their higher bandwidth to Jaguar. For future work, we would like to see higher priority queues that enable interactive visualization. A more interactive user-friendly qsub that estimates time to allocation would be very useful. Hopefully, our work will lead to more users attempting interactive visualization on Jaguar, resulting in quicker turnaround times for visualization and analysis. As users migrate to using supercomputing platforms for visualization, we expect future research possibilities in this area.

00↗

A browser-based tool for visualization and analysis of diffusion MRI data

Human neuroscience research faces several challenges with regards to reproducibility. While scientists are generally aware that data sharing is important, it is not always clear how to share data in a manner that allows other labs to understand and reproduce published findings. Here we report a new open source tool, AFQ-Browser, that builds an interactive website as a companion to a diffusion MRI study. Because AFQ-Browser is portable—it runs in any web-browser—it can facilitate transparency and data sharing. Moreover, by leveraging new web-visualization technologies to create linked views between different dimensions of the dataset (anatomy, diffusion metrics, subject metadata), AFQ-Browser facilitates exploratory data analysis, fueling new discoveries based on previously published datasets. In an era where Big Data is playing an increasingly prominent role in scientific discovery, so will browser-based tools for exploring high-dimensional datasets, communicating scientific discoveries, aggregating data across labs, and publishing data alongside manuscripts.

60 APPLIED LIFE SCIENCES↗