Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data visualization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2014

SciDAC Visualization and Analytics Center for Enabling Technologies

This project focuses on leveraging scientific visualization and analytics software technology as an enabling technology for increasing scientific productivity and insight. Advances in computational technology have resulted in an "information big bang," which in turn has created a significant data understanding challenge. This challenge is widely acknowledged to be one of the primary bottlenecks in contemporary science. The vision for our Center is to respond directly to that challenge by adapting, extending, creating when necessary and deploying visualization and data understanding technologies for our science stakeholders. Using an organizational model as a Visualization and Analytics Center for Enabling Technologies (VACET), we are well positioned to be responsive to the needs of a diverse set of scientific stakeholders in a coordinated fashion using a range of visualization, mathematics, statistics, computer and computational science and data management technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Galaxy morphological classification in deep-wide surveys via unsupervised machine learning

ABSTRACT Galaxy morphology is a fundamental quantity, which is essential not only for the full spectrum of galaxy-evolution studies, but also for a plethora of science in observational cosmology (e.g. as a prior for photometric-redshift measurements and as contextual data for transient light-curve classifications). While a rich literature exists on morphological-classification techniques, the unprecedented data volumes, coupled, in some cases, with the short cadences of forthcoming ‘Big-Data’ surveys (e.g. from the LSST), present novel challenges for this field. Large data volumes make such data sets intractable for visual inspection (even via massively distributed platforms like Galaxy Zoo), while short cadences make it difficult to employ techniques like supervised machine learning, since it may be impractical to repeatedly produce training sets on short time-scales. Unsupervised machine learning, which does not require training sets, is ideally suited to the morphological analysis of new and forthcoming surveys. Here, we employ an algorithm that performs clustering of graph representations, in order to group image patches with similar visual properties and objects constructed from those patches, like galaxies. We implement the algorithm on the Hyper-Suprime-Cam Subaru-Strategic-Program Ultra-Deep survey, to autonomously reduce the galaxy population to a small number (160) of ‘morphological clusters’, populated by galaxies with similar morphologies, which are then benchmarked using visual inspection. The morphological classifications (which we release publicly) exhibit a high level of purity, and reproduce known trends in key galaxy properties as a function of morphological type at z < 1 (e.g. stellar-mass functions, rest-frame colours, and the position of galaxies on the star-formation main sequence). Our study demonstrates the power of unsupervised machine learning in performing accurate morphological analysis, which will become indispensable in this new era of deep-wide surveys.

Martin, G.↗
NASA NTRS2019

Sherlock Data Warehouse

This slide deck provides an overview of the data and resources available in the Sherlock Data Warehouse. Sherlock was developed and is currently maintained by the Aviation Systems Division at NASA Ames Research Center. Sherlock contains a valuable collection of flight, air traffic management, and weather data. But Sherlock is not just a data archive. Sherlock also includes tools and resources to access, download, and visualize data, as well as resources to process the data. This overview summarizes Sherlock data sources, demonstrates data analytics and visualization with MicroStrategy, illustrates disparate data integration using the ATM Knowledge graph, and presents a machine learning use case using the Big Data system.

data warehouse↗
DOE OSTI2014

SCALING AN URBAN EMERGENCY EVACUATION FRAMEWORK: CHALLENGES AND PRACTICES

Critical infrastructure disruption, caused by severe weather events, natural disasters, terrorist attacks, etc., has significant impacts on urban transportation systems. We built a computational framework to simulate urban transportation systems under critical infrastructure disruption in order to aid real-time emergency evacuation. This framework will use large scale datasets to provide a scalable tool for emergency planning and management. Our framework, World-Wide Emergency Evacuation (WWEE), integrates population distribution and urban infrastructure networks to model travel demand in emergency situations at global level. Also, a computational model of agent-based traffic simulation is used to provide an optimal evacuation plan for traffic operation purpose [1]. In addition, our framework provides a web-based high resolution visualization tool for emergency evacuation modelers and practitioners. We have successfully tested our framework with scenarios in both United States (Alexandria, VA) and Europe (Berlin, Germany) [2]. However, there are still some major drawbacks for scaling this framework to handle big data workloads in real time. On our back-end, lack of proper infrastructure limits us in ability to process large amounts of data, run the simulation efficiently and quickly, and provide fast retrieval and serving of data. On the front-end, the visualization performance of microscopic evacuation results is still not efficient enough due to high volume data communication between server and client. We are addressing these drawbacks by using cloud computing and next-generation web technologies, namely Node.js, NoSQL, WebGL, Open Layers 3 and HTML5 technologies. We will describe briefly about each one and how we are using and leveraging these technologies to provide an efficient tool for emergency management organizations. Our early experimentation demonstrates that using above technologies is a promising approach to build a scalable and high performance urban emergency evacuation framework that can improve traffic mobility and safety under critical infrastructure disruption in today s socially connected world.

cloud computing↗
DOE OSTI2012

Canopy 2.1 User Guide

Canopy is a visual analytic software suite, developed by researchers at the Pacific Northwest National Laboratory (PNNL), to support intelligence analysts’ ability to efficiently analyze massive amounts of multimedia data. Canopy incorporates state-of-the-art computer vision algorithms and interactive visualizations so that analysts can comprehend and articulate the big picture, thereby reducing their workload and ultimately the effort of identifying the critical intelligence for decision makers.

Canopy↗
DOE OSTI2015

Distributed computing strategies for processing of FT-ICR MS imaging datasets for continuous mode data visualization

High-resolution Fourier transform ion cyclotron resonance (FT-ICR) mass spectrometry imaging enables the spatial mapping and identification of biomolecules from complex surfaces. The need for long time-domain transients, and thus large raw file sizes, results in a large amount of raw data (“big data”) that must be processed efficiently and rapidly. This can be compounded by largearea imaging and/or high spatial resolution imaging. For FT-ICR, data processing and data reduction must not compromise the high mass resolution afforded by the mass spectrometer. The continuous mode “Mosaic Datacube” approach allows high mass resolution visualization (0.001 Da) of mass spectrometry imaging data, but requires additional processing as compared to featurebased processing. We describe the use of distributed computing for processing of FT-ICR MS imaging datasets with generation of continuous mode Mosaic Datacubes for high mass resolution visualization. An eight-fold improvement in processing time is demonstrated using a Dutch nationally available cloud service.

Environmental Molecular Sciences Laboratory↗
NASA NTRS2011

NASA's Hyperwall Revealing the Big Picture

NASA:s hyperwall is a sophisticated visualization tool used to display large datasets. The hyperwall, or video wall, is capable of displaying multiple high-definition data visualizations and/or images simultaneously across an arrangement of screens. Functioning as a key component at many NASA exhibits, the hyperwall is used to help explain phenomena, ideas, or examples of world change. The traveling version of the hyperwall is typically comprised of nine 42-50" flat-screen monitors arranged in a 3x3 array (as depicted below). However, it is not limited to monitor size or number; screen sizes can be as large as 52" and the arrangement of screens can include more than nine monitors. Generally, NASA satellite and model data are used to highlight particular themes in atmospheric, land, and ocean science. Many of the existing hyperwall stories reveal change across space and time, while others display large-scale still-images accompanied by descriptive, story-telling captions. Hyperwall content on a variety of Earth Science topics already exists and is made available to the public at: eospso.gsfc.nasa.gov/hyperwall. Keynote and PowerPoint presentations as well as Summary of Story files are available for download on each existing topic. New hyperwall content and accompanying files will continue being developed to promote scientific literacy across a diverse group of audience members. NASA invites the use of content accessible through this website but requests the user to acknowledge any and all data sources referenced in the content being used.

Sellers, Piers↗
DOE OSTI2026

Expanding Access to Science Participation: A FAIR Framework for Petascale Data Visualization and Analytics

The massive data generated by scientists daily serve as both a major catalyst for new discoveries and innovations, as well as a significant roadblock that restricts access to the data. Here, our paper introduces a new approach to removing Big Data barriers and democratizing access to petascale data for the broader scientific community. Our novel data fabric abstraction layer allows user-friendly querying of scientific information while hiding the complexities of dealing with file systems or cloud services. We enable FAIR (Findable, Accessible, Interoperable, and Reusable) access to datasets such as NASA’s petascale climate datasets. Our paper presents an approach to managing, visualizing, and analyzing petabytes of data within a browser on equipment ranging from the top NASA supercomputer to commodity hardware like a laptop. Our novel data fabric abstraction utilizes state-of-the art progressive compression algorithms and machine-learning insights to power scalable visualization dashboards for petascale data. The result provides users with the ability to identify extreme events or trends dynamically, expanding access to scientific data and further enabling discoveries. We validate our approach by improving the ability of climate scientists to visually explore their data via three fully interactive dashboards. We further validate our approach by deploying the dashboards and simplified training materials in the classroom at a minority-serving institution. These dashboards, released in simplified form to the general public, contribute significantly to a broader push to democratize the access and use of climate data.

Computer science↗
NASA NTRS2017

Detection of Hail Storms in Radar Imagery Using Deep Learning

In 2016, hail was responsible for 3.5 billion and 23 million dollars in damage to property and crops, respectively, making it the second costliest weather phenomenon in the United States. In an effort to improve hail-prediction techniques and reduce the societal impacts associated with hail storms, we propose a deep learning technique that leverages radar imagery for automatic detection of hail storms. The technique is applied to radar imagery from 2011 to 2016 for the contiguous United States and achieved a precision of 0.848. Hail storms are primarily detected through the visual interpretation of radar imagery (Mrozet al., 2017). With radars providing data every two minutes, the detection of hail storms has become a big data task. As a result, scientists have turned to neural networks that employ computer vision to identify hail-bearing storms (Marzbanet al., 2001). In this study, we propose a deep Convolutional Neural Network (ConvNet) to understand the spatial features and patterns of radar echoes for detecting hailstorms.

natural hazard↗
NASA NTRS2019

NASA GES DISC Giovanni: Current and Future

Giovanni (Geospatial Interactive Online Visualization and Analysis Infrastructure), developed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), has established a reputation among NASA users for easy access, analysis, and visualization of NASA Earth science data. Currently, Giovanni supports over 1900 variables in eight disciplinary areas. Like any other enterprise application, Giovanni faces big data challenges, such as servicing increasingly large data volumes and more complex data types, while at the same time addressing the demands of a more diverse user community, e.g., placing requests for long-term time series from multiple spatially and temporally dense data records. I will present how Giovanni has been evolving from an on-premises, monolithic software application towards a cloud-enabled implementation to address these challenges.

Analytics↗
DOE OSTI2015

Fleet DNA Phase 1 Refinement & Phase 2 Implementation; NREL (National Renewable Energy Laboratory)

Fleet DNA acts as a secure data warehouse for medium- and heavy-duty vehicle data. It demonstrates that vehicle drive cycle data can be collected and stored for large-scale analysis and modeling applications. The data serve as a real-world data source for model development and validation. Storage of the results of past/present/future data collection efforts improves analysis efficiency through pooling of shared data and provides the opportunity for 'big data' type analyses. Fleet DNA shows it is possible to develop a common database structure that can store/analyze/report on data sourced from multiple parties, each with unique data formats/types. Data filtration and normalization algorithms developed for the project allow for a wide range of data types and inputs, expanding the project’s potential. Fleet DNA demonstrates the power of integrating Big Data with existing and future tools and analyses: it provides an enhanced understanding and education of users, users can explore greenhouse gases and economic opportunities via AFLEET and ADOPT modeling, drive cycles can be characterized and visualized using DRIVE, high-level vehicle modeling can be performed using real-world drive cycles via FASTSim, and data reporting through Fleet DNA Phase 1 and 2 websites provides external users access to analysis results and gives the opportunity to explore on their own.

33 ADVANCED PROPULSION SYSTEMS↗
NASA NTRS2018

Restructuring Big Data to Improve Data Access and Performance in Analytic Services Making Research More Efficient for the Study of Extreme Weather Events and Application User Communities

By developing and enhancing various services and tools, the GES DISC provides users with the capability to access and visualize data, and to make comparisons of data from multiple sensor and models via a number of cross-discipline projects. Discovering Data via Faceted Web Interface Web interface to data products and services Search and Download mechanisms Dataset Landing Pages Accessing Data through Interoperable Services: GDS – GrADS Data Server OPeNDAP - Open-source Project for a Network Data Access Protocol WMS – OGC service GIS connector – allowing IS tools to access data easier (coming soon) HTTPS -- direct online access Downloading Data Basics: Subset and egridding Service – Parameter, Spatial, Time, Vertical, Mean averaging, format conversion, and regridding for L3/L4 gridded data Swath Data Subsetter – Parameter, spatial subset of L2 /L1 data. Visualizing Data Online: Giovanni –Visualization and Analysis L3/L4 gridded data AIRS NRT Viewer – AIRS near-real-time DQVis – L2 data quality visualization

data cube↗
NASA NTRS2019

New Era, New Opportunity, Is GES DISC Ready for Big Data Challenge?

The new era of Big Data has opened doors for many new opportunities, as well as new challenges, for both Earth science research/application and data communities. As one of the twelve NASA data centers - Goddard Earth Sciences Data and Information Services Center (GES DISC), one of our great challenges has been how to help research/application community efficiently (quickly and properly) accessing, visualizing and analyzing the massive and diverse data in natural hazard research, management, or even prediction. GES DISC has archived over 2000 TB data on premises and distributed over 23,000 TB of data since 2010. Our data has been widely used in every phase of natural hazard management and research, i.e. long term risk assessment and reduction, forecasting and predicting, monitoring and detection, early warning, damage assessment and response. The big data challenge is not just about data storage, but also about data discoverability and accessibility, and even more, about data migration/mirroring in the cloud. This paper is going to demonstrate GES DISC’s efforts and approaches of evolving our overall Web services and powerful Giovanni (Geospatial Interactive Online Visualization ANd aNalysis Infrastructure) tool into further improving data discoverability and accessibility. Prototype works will also be presented.

Li, A.↗
DOE OSTI2017

Digital archives, big data and image-based culturomics for social impact assessment: Opportunities and challenges

Social impact assessment (SIA) is well-established but uses conventional approaches that have become less effective in recent decades, particularly in relation to declining survey response rates and a lack of youth engagement. Images from digital archives and social media sources are poised to advance the research and practice of SIA by transcending text-based methods with insights into changing landscapes, and human engagement with them. This viewpoint describes progress, challenges and cautions toward the development of such tools (defined as culturomics), using hydroelectricity cases to illustrate potential approaches. These tools build on foundational work in a range of disciplines, including the humanities and computer science. We describe necessary advances in machine learning, image digitization, and data aggregation and visualization techniques, as well as ways to ensure that such tools are carefully tested, applied and interpreted. Challenges include the automation, acquisition and management of datasets, and using these tools appropriately and equitably. Critically, culturomics of any kind must not be used as a replacement for engagement with people, but as complementary to inclusive stakeholder engagement. - Highlights: •Image-based digital archives and social media present opportunities for new SIA tools. •Application-ready big data approaches are emerging across many fields. •SIA scholars, practitioners and stakeholders should engage with culturomics. •Challenges include automation, digitization, interpretation and justice. •Culturomics of any kind must not replace engagement with people.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2018

Integrated Multi Scale Data Analytics and Machine Learning for the Grid; Benchmarking Algorithms and Data Quality Analysis

Machine Learning and Data Analytics are terms which are utilized now ubiquitously throughout grid analytics forums, as a potential solution to the growing big data challenges for the distribution grid. As sensors become inexpensive and common, and communication networks grow, the analytics themselves must adapt and enable consumers and providers to visualize and utilize the data provided effectively. Machine learning is one of the techniques proposed in order to manage this data deluge. This report is a subset of the work in this activity, funded through the DOE Grid Modernization Initiative, to summarize the benchmarking and review of data quality impact on the algorithm sets being developed within each use case area.

42 ENGINEERING↗
DOE OSTI2018

Optimization of Energy Flow Through Synthetic Metabolic Modules and Regulatory Networks in a Model Photosynthetic Eukaryotic Microbe

Diatoms are important oceanic photosynthetic microorganisms, responsible for approximately 20% of all carbon fixation on Earth. Additionally, diatoms are attractive metabolic engineering candidates. This investigation leveraged Systems Biology techniques to advance our understanding of and potential to engineer diatoms for bioproducts of societal interest, such as biofuels. Systems Biology combines computational and experimental techniques to understand an organism at the whole-cell level as well as characterize the interface between the organism and its environment. Our research resulted in major contributions to the understanding of systems-level light-driven metabolism. Diatoms have a unique evolutionary history. As a result, existing understanding of photosynthetic metabolism may not be applicable to diatom physiology. We generated, validated, and published a high-quality metabolic model of diatom metabolism facilitating novel insights into these unique organisms. Additionally, we overcame a consistent challenge in the modeling of photosynthetic organisms by integrating light into metabolic models. This result allows realistic assessments of how the light environment affects cell physiology and metabolism, a necessary advancement for bioengineering. Next, we generated several methodologies for integrating large-scale datasets with models of metabolism. These “big data” approaches begin to address a constant challenge in biology, which is translating data into information and understanding. Finally, we built web-based databases for housing, distributing, and visualizing systems-level biology data. These projects created an online ecosystem for sharing, analyzing, and characterizing metabolic information and is currently being used by public and private institutions. Photosynthetic bioengineering has the promise to create high-energy products with minimal inputs. Systems Biology has a long history of facilitating bioengineering of cellular metabolism. However, photosynthetic organisms are only sparsely present in this history. This project and its outputs have accelerated the application of proven Systems Biology techniques to light-driven metabolism. The resulting methodologies will accelerate public and private endeavors leveraging photosynthetic systems in the bioengineering of bioproducts such as biofuels or carbon sequestration.

09 BIOMASS FUELS↗
NASA NTRS2014

Using ISERV and Commercial Satellite Imagery to Assess and Monitor Recovery Efforts in Urban Damaged Areas

NASA's Short-term Prediction, Research, and Transition (SPoRT) Center uses a wide array of satellites to monitor and assess the impacts of natural disasters, with support from NASA's Applied Sciences Program. One of the newest sensors SPoRT is utilizing in these activities is the International Space Station (ISS) SERVIR Environmental Research and Visualization System (ISERV) instrument. ISERV provides a unique view of the areas impacted and will play a big role in monitoring the recovery these areas. High‐resolution commercial satellite data is also used to monitor urban areas that have been impacted by natural disasters. SPoRT is developing techniques to measure the extent of these disasters and to monitor recovery. Several of these techniques include semi‐automatic feature detection and change as well as developing an experimental damage assessment based upon the visible damage observed by the satellites. Furthermore, throughout these activities SPoRT hopes to provide additional data to the NOAA National Weather Service Damage Assessment Toolkit, which will help to supplement those activities being performed in the field.

Bell, Jordan R.↗
DOE OSTI2015

PlanetSense: A Real-time Streaming and Spatio-temporal Analytics Platform for Gathering Geo-spatial Intelligence from Open Source Data

Geospatial intelligence has traditionally relied on the use of archived and unvarying data for planning and exploration purposes. In consequence, the tools and methods that are architected to provide insight and generate projections only rely on such datasets. Albeit, if this approach has proven effective in several cases, such as land use identification and route mapping, it has severely restricted the ability of researchers to inculcate current information in their work. This approach is inadequate in scenarios requiring real-time information to act and to adjust in ever changing dynamic environments, such as evacuation and rescue missions. In this work, we propose PlanetSense, a platform for geospatial intelligence that is built to harness the existing power of archived data and add to that, the dynamics of real-time streams, seamlessly integrated with sophisticated data mining algorithms and analytics tools for generating operational intelligence on the fly. The platform has four main components – i) GeoData Cloud – a data architecture for storing and managing disparate datasets; ii) Mechanism to harvest real-time streaming data; iii) Data analytics framework; iv) Presentation and visualization through web interface and RESTful services. Using two case studies, we underpin the necessity of our platform in modeling ambient population and building occupancy at scale.

big data architecture↗