Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data visualization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Predictive Big Data Analytics: A Study of Parkinson’s Disease Using Large, Complex, Heterogeneous, Incongruent, Multi-Source and Incomplete Observations

Background A unique archive of Big Data on Parkinson’s Disease is collected, managed and disseminated by the Parkinson’s Progression Markers Initiative (PPMI). The integration of such complex and heterogeneous Big Data from multiple sources offers unparalleled opportunities to study the early stages of prevalent neurodegenerative processes, track their progression and quickly identify the efficacies of alternative treatments. Many previous human and animal studies have examined the relationship of Parkinson’s disease (PD) risk to trauma, genetics, environment, co-morbidities, or life style. The defining characteristics of Big Data–large size, incongruency, incompleteness, complexity, multiplicity of scales, and heterogeneity of information-generating sources–all pose challenges to the classical techniques for data management, processing, visualization and interpretation. We propose, implement, test and validate complementary model-based and model-free approaches for PD classification and prediction. To explore PD risk using Big Data methodology, we jointly processed complex PPMI imaging, genetics, clinical and demographic data. Methods and Findings Collective representation of the multi-source data facilitates the aggregation and harmonization of complex data elements. This enables joint modeling of the complete data, leading to the development of Big Data analytics, predictive synthesis, and statistical validation. Using heterogeneous PPMI data, we developed a comprehensive protocol for end-to-end data characterization, manipulation, processing, cleaning, analysis and validation. Specifically, we (i) introduce methods for rebalancing imbalanced cohorts, (ii) utilize a wide spectrum of classification methods to generate consistent and powerful phenotypic predictions, and (iii) generate reproducible machine-learning based classification that enables the reporting of model parameters and diagnostic forecasting based on new data. We evaluated several complementary model-based predictive approaches, which failed to generate accurate and reliable diagnostic predictions. However, the results of several machine-learning based classification methods indicated significant power to predict Parkinson’s disease in the PPMI subjects (consistent accuracy, sensitivity, and specificity exceeding 96%, confirmed using statistical n-fold cross-validation). Clinical (e.g., Unified Parkinson's Disease Rating Scale (UPDRS) scores), demographic (e.g., age), genetics (e.g., rs34637584, chr12), and derived neuroimaging biomarker (e.g., cerebellum shape index) data all contributed to the predictive analytics and diagnostic forecasting. Conclusions Model-free Big Data machine learning-based classification methods (e.g., adaptive boosting, support vector machines) can outperform model-based techniques in terms of predictive precision and reliability (e.g., forecasting patient diagnosis). We observed that statistical rebalancing of cohort sizes yields better discrimination of group differences, specifically for predictive analytics based on heterogeneous and incomplete PPMI data. UPDRS scores play a critical role in predicting diagnosis, which is expected based on the clinical definition of Parkinson’s disease. Even without longitudinal UPDRS data, however, the accuracy of model-free machine learning based classification is over 80%. The methods, software and protocols developed here are openly shared and can be employed to study other neurodegenerative disorders (e.g., Alzheimer’s, Huntington’s, amyotrophic lateral sclerosis), as well as for other predictive Big Data analytics applications.

Science & Technology - Other Topics↗

Automating galaxy morphology classification using k -nearest neighbours and non-parametric statistics

ABSTRACT Morphology is a fundamental property of any galaxy population. It is a major indicator of the physical processes that drive galaxy evolution and in turn the evolution of the entire Universe. Historically, galaxy images were visually classified by trained experts. However, in the era of big data, more efficient techniques are required. In this work, we present a k-nearest neighbours based approach that utilizes non-parametric morphological quantities to classify galaxy morphology in Sloan Digital Sky Survey images. Most previous studies used only a handful of morphological parameters to identify galaxy types. In contrast, we explore 1023 morphological spaces (defined by up to 10 non-parametric statistics) to find the best combination of morphological parameters. Additionally, while most previous studies broadly classified galaxies into early types and late types or ellipticals, spirals, and irregular galaxies, we classify galaxies into 11 morphological types with an average accuracy of ${\sim} 80\!-\!90 \, {{\rm per\, cent}}$ per T-type. Our method is simple, easy to implement, and is robust to varying sizes and compositions of the training and test samples. Preliminary results on the performance of our technique on deeper images from the Hyper Suprime-Cam Subaru Strategic Survey reveal that an extension of our method to modern surveys with better imaging capabilities might be possible.

Mukundan, Kavya↗

Learning Global Proliferation Expertise Evolution Using AI-Driven Analytics and Public Information

Detecting and anticipating global proliferation expertise and capability evolution from unstructured, noisy, and incomplete public data streams is a highly desired, but extremely challenging task. Here, in this article, we present our pioneering data-driven approach to support the non-proliferation mission to detect and explain the evolution of proliferation expertise and capability development globally from terabytes of publicly available information (PAI), focusing on our knowledge extraction pipeline and descriptive analytics. We first discuss how we fuse nine open-source data streams, including multilingual data, to convert 4 TB of unstructured data to structured knowledge and encode dynamically evolving proliferation expertise representations—content and context graphs. For this, we rely on natural language processing (NLP) and deep learning (DL) models to perform information extraction, topic modeling, and distributed text representation (aka embedding) learning. We then present interactive, usable, and explainable descriptive analytics to refine domain knowledge and present it in a human-understandable form. Finally, we introduce future work avenues that will leverage our dynamic knowledge representations and descriptive analytics to enable predictive and prescriptive inferences to achieve real-time domain understanding and contextual reasoning about global proliferation expertise and capability evolution.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Wind Wake Watcher v. 1.0

This software enables the user to produce Google Earth visualizations of turbine wake effects for wind farms. The visualizations are based on computations of statistical quantities that vary with wind direction and help quantify the effects on power production of upwind turbines on turbines in their wakes. The results of the software are plot images and kml files that can be loaded into Google Earth. The statistics computed are described in greater detail in the paper: S. Martin, C. H. Westergaard, and J. White (2016), Visualizing Wind Farm Wakes Using SCADA Data, in Wither Turbulence and Big Data in the 21st Century? Eds. A. Pollard, L. Castillo, L. Danaila, and M. Glauser. Springer, pgs. 231-254.

Martin, Shawn↗

VEDA Visualization Exploration & Data Analysis

Why? - Interdisciplinary science depends on large amount of Earth science data and computational resources - Working with these datasets is non-trivial - Big data science requires advanced distributed computing knowledge What? VEDA is an open platform that brings key Earth science datasets next to open source tools for data processing, analysis, visualization, and exploration in a managed and more accessible computing environment.

Manil Maskey↗

GES DISC Datalist Improves Earth Science Data Discoverability

At American Geophysical Union(AGU) 2016 Fall Meeting, Goddard Earth Sciences Data Information Services Center (GES DISC) unveiled a novel way to access data: Datalist. Currently, datalist is a collection of predefined data variables from one or more archived datasets, curated by our subject matter expert (SME). Our science support team has curated a predefined Hurricane Datalist and received very positive feedback from the user community. Datalist uses the same architecture our new website uses and have the same look and feel as other datasets on our web site. and also provides a one-stop shopping for data, metadata, citation, documentation, visualization and other available services. Since the last AGU Meeting, we have further developed a few new datalists corresponding to the Big Earth Data Initiative (BEDI) Societal Benefit Areas and A-Train data. We now have four datalists: Hurricane, Wind Energy, Greenhouse Gas and A-Train. We have also started working with our User Working Group members to create their favorite datalists and working with other DAAC to explore the possibility to include their products in our datalists that may also lead to a future of potential federated (cross-DAAC) datalists. Since our datalist prototype effort was a success, we are planning to make datalist operational. It's extremely important to have a common metadata model to support datalist, this will also be the foundation of federated datalist. We mapped our datalist metadata model to the unpublished UMM(Universal Metadata Model)-Var (Variable) (June version) and found that the UMM-var together with UMM-C (Collection) and possible UMM-S (Service) will meet our basic requirements. For example: Dataset shortname, and version are already specified in UMM-C, variable name, long name, units, dimensions are all specified in UMM-Var. UMM-Var also facilitates Science Keywords to allow tagging at variable level and Characteristics for optional variable characteristics. Measurements is useful for grouping of the variables and Set is promising to define datalist. And finally, the UMM-Service model to specify the available services for the variable will be very beneficial. In summary, UMM-Var, UMM-C and UMM-S are the basis of federated datalist and the development and deployment of datalist will contribute to the evolution of the UMM.

datalist↗

High-Performance Computing Unlocks Innovation at NREL

Need to fly around a wind farm? Or step inside a molecule? NREL scientists use a super powerful (and highly energy-efficient) computer to visualize and solve big problems in renewable energy research.

17 WIND ENERGY↗

Abstract for CRADA between VariGrid Explorations Inc. and NETL (AGMT-0863)

As the energy industry continues to invest in data, big data, and advanced analytics to improve production, reduce operation and maintenance costs, improve safety, and reduce risks, uncertainty will serve as a limiting factor in the success of these operations. Intuitive communication of uncertainty can provide more transparency in the decision-making process; offering the ability to more accurately evaluate the risk and impact of decisions, identify data and knowledge gaps, and evaluate trade-offs. The issue of uncertainty will thus play a large role in the success of investments that rely on big data and data analytics. However, no robust method for representation of uncertainty in spatial data visuals currently exists. As such, uncertainty is, at best, communicated using multiple independent visuals and aggregated in final displays, or frequently omitted altogether. Failure to accurately and effectively quantify and communicate uncertainty poses a barrier, limiting critical information that would better allow businesses to make more accurate and intelligent decisions.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

An Interactive Visualization Tool for Large-Scale Building Stock Modeling

Recent advancements in data science and high-performance computing are making it easier to run millions of building simulations, but meaningful visualization of such large datasets remains a challenge. This paper presents a new tool developed to view the results of large-scale OpenStudio simulations of national, regional, or local building stocks. The tool processes millions of simulations to calculate measure savings, utility bills, carbon emissions, primary energy, and cost-effectiveness metrics at a high geographic resolution. Interactive visualizations of the building characteristics, consumption, and measure savings data include proportional symbol maps and histogram plots and can be filtered by any building characteristic.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Open Source Visualization and Analysis Platform for 3D Reconstructions of Materials by Transmission Electron Microscopy

Three-dimensional characterization of materials at the nano- and meso-scale has become possible with transmission and scanning transmission electron microscopes (S/TEM). Its importance has extended to a wide class of nanomaterials such as hydrogen fuel cells, solar cells, industrial catalysts, new battery materials and semiconductor devices, as well as spanning high-tech industry, universities, and national labs. While capable instrumentation is abundant, this rapidly expanding demand for high-resolution tomography is bottlenecked by software that is instead tailored for lower-dose, biological applications and not optimized for higher-resolution materials applications. Existing tomography fails to utilize the chemical information provided by S/TEM spectrometers. To address this problem, this project delivered a scalable, fully functional, freely-distributable, open Source materials tomography package with a modern user interface that enables automated acquisition, alignment, and real-time reconstruction of raw tomography data, and provides advanced segmentation, three-dimensional chemical visualization and analysis optimized for materials applications. It has established an extendable framework capable of automation for high-throughput for the tomography of materials from data acquisition to visualization. Phase I and II delivered a full-featured, cross-platform, clean and integrated application for S/TEM materials tomography. It can read projection data from the microscope, with graphical tools for alignment of data, tomographic reconstruction, segmentation, and visualization of the reconstructed 3D volume. The entire pipeline can be saved to an XML state file, enabling fully reproducible data processing and analysis with a full Python environment for the development of custom algorithms, processing, and analysis requirements. Phase IIB extended the application to provide real-time tomographic reconstruction, "live updates" as data is processed, analysis of big data, and multi-channel tomography. Here, quantitative assessment of nanomaterials can occur as data is being recorded on a 3D visualization platform that accommodates additional information from multiple channels. We provide unmatched high-throughput tomography, where the complete tomographic pipeline from measurement to 3D visualization can occur rapidly. The project was extended to offer capabilities for micro-CT, X-ray, neutron, focused ion beam, atomic electron tomography and atom probe tomography. With over 600 transmission electron microscopes worldwide and approximately 50 coming online each year, the demand and impact of an open-source tomography tool is large. Significant opportunities exist in high-tech industry, universities, and national labs to enable or enhance three-dimensional imaging at the nanoscale and bring automated high-throughput approaches that will accelerate progress in materials characterization and metrology. The project supports a service based business model by enabling lab-specific acquisition and processing customization and integration-support and development that will be provided into Phase III and beyond.

36 MATERIALS SCIENCE↗

Architecture of a Data Portal for Publishing and Delivering Open Data for Atmospheric Measurement

Atmospheric data are collected by researchers every day. Campaigns such as GOAmazon 2014/2015 and the Amazon Tall Tower Observatory collect essential data on aerosols, gases, cloud properties, and meteorological parameters in the Brazilian Amazon basin. These data products provide insights and essential information for analyzing and predicting natural processes. However, in Brazil, it is estimated that more than 80% of the scientific data collected are not published due to the lack of web portals that collect and store these data. This makes it difficult, or even impossible, to access and integrate the data, which can result in the loss of significant amounts of information and significantly affect the understanding of the overall data. To address this problem, we propose a data portal architecture and open data deployment that enable Big Data processing, human interaction, and download-oriented approaches with tools that help users catalog, publish and visualize atmospheric data. Thus, we describe the architecture developed, based on the experience of the Atmospheric Radiation Measurement Data Center, which incorporates the principles of FAIR, the infrastructure and content management system for managing scientific data. The portal partial results were tested with environmental data from contaminated areas at the University of São Paulo. Overall, this data portal creates more shared knowledge about atmospheric processes by providing users with access to open environmental data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Interactive Visualization of Multi-Group Cross Sections on High-Fidelity Spatial Meshes

Core simulation tools utilizing Monte Carlo transport methods represent the 'Holy Grail' for nuclear reactor analysis. However, deterministic transport methods pose a number of advantages to Monte Carlo from a computational perspective, which seems likely to make these methods more viable in the near to intermediate term. Deterministic methods almost universally discretize the energy domain in a few to hundreds of energy groups. One of the key challenges for deterministic transport-based methods to be viable is accurate, reactor agnostic multi-group cross section generation. This paper describes tools for data processing and visualization of multigroup cross sections generated on high-fidelity spatial meshes from the OpenMC Monte Carlo code. Next generation deterministic transport reactor core modeling tools will require multi-group cross section (MGXS) generation on high-fidelity spatial meshes. High-fidelity MGXS are necessary for the accurate simulation of neutron physics in nuclear reactors with a high degree of spectral heterogeneity - e.g., large localized neutron flux gradients throughout a reactor core. Although many local (e.g., pin cell) approximations have been developed to model the scalar flux to compute spectral-averaged MGXS, these techniques cannot account for complex intra-pin and intra-assembly spatial self-shielding effects. Monte Carlo presents the most accurate method for reactor agnostic multi-group cross section generation since it does not require the use of local approximations to the flux. Monte Carlo has been proposed as a highly desirable framework for reactor agnostic cross section generation for many years. Monte Carlo methods can effectively embed all of the relevant spectral effects into the MGXS through the random particle tracking and interaction sampling processes. Previous work has applied MCNP and Serpent to compute few-group constants for diffusion calculations, and OpenMC to compute multi-group cross sections for transport-based methods. The primary difficulty with using Monte Carlo for MGXS generation is the tremendous computational resources required to adequately converge multi-group cross sections in each spatial region and energy group. Recent development of the OpenMC Monte Carlo code for high-performance computing platforms has made multigroup cross section generation from Monte Carlo more tractable in the foreseeable future. The recent release of OpenMC v0.7.1 includes a new Python-based module for MGXS generation. The MGXS generation module leverages many of the algorithms and features for 'big data' analysis of large tally datasets on complex spatial meshes. This paper describes the data management workflow and visualization tools which have been developed to explore the plethora of MGXS data generated with OpenMC. It is the authors' hope that the development of the visualization tool described in this paper will ultimately enable the design of machine learning algorithms to accelerate the convergence rate of high-fidelity multi-group cross section generated with Monte Carlo. (authors)

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Big Cypress Water Resources: Using Earth Observations to Assess Water Quality in Big Cypress Reservation, FL

Upstream agricultural development and runoff are driving harmful algal blooms in the Big Cypress Reservation. The Seminole Tribe of Florida Environmental Resource Management Department monitors water quality in the Big Cypress Reservation, located on the north end of Big Cypress National Preserve. The Environmental Resource Management Department aims to incorporate Earth observations into its investigative approach, which consists of Geographic Information Systems and in situ water sampling. We assessed the feasibility of using Earth observations to measure harmful algal blooms over time and identify vulnerable areas. We analyzed data from multiple remote sensing platforms and sensors, including Landsat 8 Thermal Infrared Spectrometer(TIRS), Landsat 9 TIRS-2, Landsat 9 Operational Land Imager 2, Sentinel-2 MultiSpectral Instrument, and Sentinel-3 Ocean and Land Color Instrument. We determined that it was challenging to use Sentinel-3, Landsat 8, and Landsat 9 imagery for monitoring the canals within the Reservation, and that Sentinel-2 was the most capable platform for the study area. Using Sentinel-2, we created spectral indices for the detection of algal blooms. We used these indices to measure algal blooms at 8 stations across the Big Cypress Reservation. We then created time series and seasonal decompositions of in situ water sample data and indices to visualize relationships between datasets, along with correlation matrices. Our correlations were not conclusive. However, we found that the spectral indices have the potential to detect algal blooms. Lastly, we evaluated Maxar WorldView-3 imagery and found that using data from sensors with higher spatial resolutions would improve results.

Remote sensing↗

ADIOS Visualization Schema: A First Step Towards Improving Interdisciplinary Collaboration in High Performance Computing

Scientific communities have benefitted from a significant increase of available computing and storage resources in the last few decades. For science projects that have access to leadership scale computing resources, the capacity to produce data has been growing exponentially. Teams working on such projects must now include, in addition to the traditional application scientists, experts in various disciplines including applied mathematicians for development of algorithms, visualization specialists for large data, and I/O specialists. Sharing of knowledge and data is becoming a requirement for scientific discovery, providing useful mechanisms to facilitate this sharing is a key challenge for e-Science. Our hypothesis is that in order to decrease the time to solution for application scientists we need to lower the barrier of entry into related computing fields. We aim at improving users' experience when interacting with a vast software ecosystem and/or huge amount of data, while maintaining focus on their primary research field. In this context we present our approach to bridge the gap between the application scientists and the visualization experts through a visualization schema as a first step and proof of concept for a new way to look at interdisciplinary collaboration among scientists dealing with big data. The key to our approach is recognizing that our users are scientists who mostly work as islands. They tend to work in very specialized environment but occasionally have to collaborate with other researchers in order to take full advantage of computing innovations and get insight from big data. We present an example of identifying the connecting elements between one of such relationships and offer a liaison schema to facilitate their collaboration.

Computer Science↗

An Interactive Visualization Tool for Large-Scale Building Stock Modeling: Preprint

Recent advancements in data science and high-performance computing are making it easier to run millions of building simulations, but meaningful visualization of such large datasets remains a challenge. This paper presents a new tool developed to view the results of large-scale OpenStudio simulations of national, regional, or local building stocks. The tool processes millions of simulations to calculate measure savings, utility bills, carbon emissions, primary energy, and cost-effectiveness metrics at a high geographic resolution. Interactive visualizations of the building characteristics, consumption, and measure savings data include proportional symbol maps and histogram plots and can be filtered by any building characteristic.

big data↗

The Importance of Data Visualization: Incorporating Storytelling into the Scientific Presentation

From its inception in 2000, one of the primary tasks of the Biomedical Data Reduction Analysis (BDRA) group has been translation of large amounts of data into information that is relevant to the audience receiving it. BDRA helps translate data into an integrated model that supports both operational and research activities. This data integrated model and subsequent visual data presentations have contributed to BDRA's success in delivering the message (i.e., the story) that its customers have needed to communicate. This success has led to additional collaborations among groups that had previously not felt they had much in common until they worked together to develop solutions in an integrated fashion. As more emphasis is placed on working with "big data" and on showing how NASA's efforts contribute to the greater good of the American people and of the world, it becomes imperative to visualize the story of our data to communicate the greater message we need to share. METHODS To create and expand its data integrated model, BDRA has incorporated data from many different collaborating partner labs and other sources. Data are compiled from the repositories of the Lifetime Surveillance of Astronaut Health and the Life Sciences Data Archive, and from the individual laboratories at Johnson Space Center that support collection of data from medical testing, environmental monitoring, and countermeasures, as designated in the Medical Requirements Integration Documents. Ongoing communication with the participating collaborators is maintained to ensure that the message and story of the data are retained as data are translated into information and visual data presentations are delivered in different venues and to different audiences. RESULTS We will describe the importance of storytelling through an integrated model and of subsequent data visualizations in today's scientific presentations and discuss the collaborative methods used. We will illustrate the discussion with examples of graphs from BDRA's past work supporting operations and/or research efforts.

Babiak-Vazquez, A.↗

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗

NASA's Big Earth Data Initiative Accomplishments

The goal of NASA's effort for BEDI is to improve the usability, discoverability, and accessibility of Earth Observation data in support of societal benefit areas. Accomplishments: In support of BEDI goals, datasets have been entered into Common Metadata Repository(CMR), made available via the Open-source Project for a Network Data Access Protocol (OPeNDAP), have a Digital Object Identifier (DOI) registered for the dataset, and to support fast visualization many layers have been added in to the Global Imagery Browse Services (GIBS).

CMR↗