Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 973 records · Page 54
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Complex optimization for big computational and experimental neutron datasets

Here, we present a framework to use high performance computing to determine accurate solutions to the inverse optimization problem of big experimental data against computational models. We demonstrate how image processing, mathematical regularization, and hierarchical modeling can be used to solve complex optimization problems on big data. We also demonstrate how both model and data information can be used to further increase solution accuracy of optimization by providing confidence regions for the processing and regularization algorithms. Finally, we use the framework in conjunction with the software package SIMPHONIES to analyze results from neutron scattering experiments on silicon single crystals, and refine first principles calculations to better describe the experimental data.

36 MATERIALS SCIENCE↗
DOE OSTI2018

SharP Data Constructs: Data Constructs to Enable Data-Centric Computing

Extreme-scale applications (i.e., Big-Compute) are becoming increasingly data-intensive, i.e., producing and consuming increasingly large amounts of data. The HPC systems traditionally used for these applications are now used for Big-Data applications such as data analytics, social network analysis, machine learning, and genomics. As a consequence of these trends, the system architecture should be flexible and data-centric. This can already be witnessed in the pre-exascale systems with TBs of on-node hierarchical and heterogeneous memories, PBs of system memory, low-latency, high-throughput networks, and many threaded cores. As such, the pre-exascale systems suit the needs of both Big-Compute and Big-Data applications. Though the system architecture is flexible enough to support both Big-Compute and Big-Data, we argue there is a software gap. Particularly, we need data-centric abstractions to leverage the full potential of the system, i.e., there is a need for native support for data resilience, the ability to express data locality and affinity, mechanisms to reduce data movement, the ability to share data, and abstractions to express User's data usage and data access patterns. In this paper, we (i) show the need for taking a holistic approach towards data-centric abstractions, (ii) show how these approaches were realized in the SHARed data-structure centric Programming abstraction (SharP) library, a data-structure centric programming abstraction, and (iii) apply these approaches to a variety of applications that demonstrate its usefulness. Particularly, we apply these approaches to QMCPack and the Graph500 benchmark and demonstrate the advantages of this approach on extreme-scale systems.

Aderholdt, William Ferrol↗
DOE OSTI2016

SPECTRUM AND ANISOTROPY OF TURBULENCE FROM MULTI-FREQUENCY MEASUREMENT OF SYNCHROTRON POLARIZATION

We consider turbulent synchrotron-emitting media that also exhibit Faraday rotation and provide a statistical description of synchrotron polarization fluctuations. In particular, we consider these fluctuations as a function of the spatial separation of the direction of the measurements and as a function of wavelength for the same line of sight. On the basis of our general analytical approach, we introduce several measures that can be used to obtain the spectral slopes and correlation scales of both the underlying magnetic turbulence responsible for emission and the spectrum of the Faraday rotation fluctuations. We show the synergetic nature of these measures and discuss how the study can be performed using sparsely sampled interferometric data. We also discuss how additional characteristics of turbulence can be obtained, including the turbulence anisotropy and the three-dimensional direction of the mean magnetic field. In addition, we consider the cases when the synchrotron emission and Faraday rotation regions are spatially separated. Appealing to our earlier study, we explain that our new results are applicable to a wide range of spectral indexes of relativistic electrons responsible for synchrotron emission. We expect wide application of our techniques, both with existing synchrotron data sets and with big forthcoming data sets from LOFAR and SKA.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2024

Combining Large Datasets - Cancer Moonshot Task Group Final Summary

In February 2022, President Biden re-ignited the Cancer Moonshot with bold new goals: to reduce the cancer death rate by half within 25 years and improve the lives of people with cancer and cancer survivors. To achieve these ambitious goals, the White House convened the first-ever Cancer Cabinet, bringing together departments and agencies from across the federal government to end cancer as we know it.The Cancer Cabinet convened three task forces and supporting task groups, including the Data and Innovation Task Force, which supported the Cancer Moonshot priority to “Deliver innovation to patients and communities.” In early 2023, the “Combining Large Datasets” (CoLD) Task Group was created within the Data and Innovation Task Force. The scope of the CoLD Task Group was how federal agencies combine large datasets for broad applications across cancer prevention and control, including nutrition, epidemiology, and military/Veteran health. Within this scope, the group sought to better leverage the immense potential of data and power of data tools to increase our understanding of cancer incidence, causes, mortality, treatments, prevention, outcomes, costs, and all other aspects of the burden of cancer.

data integration↗
DOE OSTI2023

Nuclear Data: What is the big deal? [Slides]

This presentation begins with a look at the practical application: differential and integral data. It also includes a nuclear data evaluation: cross section formalisms, data processing and cross section library generation, computer tools, and criticality safety assessment: nuclear data. The presentation ends with some concluding remarks.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

Tsdat: An Open-Source Data Standardization Framework for Marine Energy and Beyond

Many organizations are tasked with the collection and processing of large quantities of data from various measurement devices. Data reported from these sources are often not interoperable with datasets and software used by analysts and other organizations in the same domain, introducing barriers for collaboration on large-scale projects. This poses a particular problem for cross-device comparisons and machine learning applications, which rely on large quantities of data from multiple sources. To address these challenges, the open-source Time-Series Data Pipelines (Tsdat) Python framework was developed by Pacific Northwest National Laboratory, with strategic guidance and direction provided by the National Renewable Energy Laboratory and Sandia National Laboratories to facilitate collaboration and accelerate advancements in the marine energy domain through the development of an open-source ecosystem of tools. This paper will describe the Tsdat framework and the data standards within which it operates. A beta version of Tsdat has been released and is being used by several projects in marine energy, wind energy, and building energy systems.

big data↗
DOE OSTI2018

Quadri-joint inversion: Method and application to the Big Sky 9C 3D data set in northern Montana

The Kevin Dome [Formula: see text] storage project, located in northern Montana, attempted to characterize the Duperow Formation as a potential long-term storage zone for injected [Formula: see text]. A multicomponent (9C) seismic survey was acquired for the Big Sky Carbon Sequestration Partnership over a portion of the Kevin Dome using P- and S-wave sources. Prestack migrated PP, PS, SH, and SV data sets were generated. We then applied several stratigraphic inversion workflows using one or several kinds of seismic wavefield at the same time resulting in joint inversions of each data set. The aim of our study is to demonstrate the benefits of doing quadri-joint inversion of PP-, PS-, SH-, and SV-wavefields for the recovery of the elastic earth parameters, especially the S-wave impedance and density. These are crucial parameters because they can help determine lithology and porefill in the reservoir characterization workflow. Because the inversion workflow always uses the original seismic data recorded in its own time domain, it is necessary to compute registration laws between PP-PS-, PP-SH-, and PP-SV-wavefields using a time shift computation procedure (warping) based on inverted S-wave impedances from inversion of a single wavefield. This generated a significant improvement over methods that rely on attempting to match trace waveforms that may have a different phase, frequency content, and polarity. Finally, we wanted to investigate the reliability of the quadri-joint inversion results in the Bakken/Banff Formations, which have less lateral geologic variation than the underlying Duperow target. This interval shares many of the geophysical characterization challenges common to shale reservoirs in other North American basins. We computed geomechanical parameters, such as Poisson’s ratio and Young’s modulus, which are a proxy for brittleness. Comparison of these results with independent laboratory measurements in the Bakken interval demonstrates the superiority of the quadri-joint inversion method to the traditional inversion using P-wave data only.

Geochemistry & Geophysics↗
DOE OSTI2019

Time-Resolved Electrical Scanning Probe Microscopy of Layered Perovskites Reveals Spatial Variations in Photoinduced Ionic and Electronic Carrier Motion

We study light-induced dynamics in thin films comprising Ruddlesden–Popper phases of the layered 2D perovskite (C 4 H 9 NH 3 ) 2 PbI 4 . We probe ionic and electronic carrier dynamics using two complementary scanning probe methods, time-resolved G-mode Kelvin probe force microscopy and fast free time-resolved electrostatic force microscopy, as a function of position, time, and illumination. Here, we show that the average surface photovoltage sign is dominated by the band bending at the buried perovskite–substrate interface. However, the film exhibits substantial variations in the spatial and temporal response of the photovoltage. Under illumination, the photovoltage equilibrates over hundreds of microseconds, a time scale associated with ionic motion and trapped electronic carriers. Surprisingly, we observe that the surface photovoltage of the 2D grain centers evolves more rapidly in time than at the grain boundaries. We propose that the slower evolution at grain boundaries is due to a combination of ion migration occurring between PbI 4 planes, as well as electronic carriers traversing grain boundary traps, thereby changing the time-dependent band unbending at grain boundaries. These results provide a model for the photoinduced dynamics in 2D perovskites and are a useful basis for interpreting photovoltage dynamics on hybrid 2D/3D structures.

14 SOLAR ENERGY↗
NASA NTRS2016

Earth Observing Data System Data and Information System (EOSDIS) Overview

The National Aeronautics and Space Administration (NASA) acquires and distributes an abundance of Earth science data on a daily basis to a diverse user community worldwide. The NASA Big Earth Data Initiative (BEDI) is an effort to make the acquired science data more discoverable, accessible, and usable. This presentation will provide a brief introduction to the Earth Observing System Data and Information System (EOSDIS) project and the nature of advances that have been made by BEDI to other Federal Users.

Earth Science↗
DOE OSTI2017

Canopus: A Paradigm Shift Towards Elastic Extreme-Scale Data Analytics on HPC

Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking on the current HPC software ecosystem. To that end, this paper develops Canopus, a progressive JPEGlike data management scheme for storing and analyzing big scientific data. It co-designs the data decimation, compression and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data into a much smaller, reduced-accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, Canopus provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-the-fly. We evaluate the impact of Canopus on unstructured triangular meshes, a pervasive data model used by scientific modeling and simulations. In particular, we demonstrate the progressive data exploration of Canopus using the “blob detection” use case on the fusion simulation data.

Lu, Tao↗
DOE OSTI2018

SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data

Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking in the current HPC software ecosystem. To that end, this paper develops SIRIUS, a progressive JPEG-like data management scheme for storing and analyzing big scientific data. It co-designs data decimation, compression, and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data, using either topological or uniform decimation, into a much smaller, reduced-accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, SIRIUS provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-the-fly. This paper further develops algorithms to preserve statistics for data decimation, a common requirement for reducing data. Here, we assess the impact of SIRIUS on unstructured triangular meshes, a pervasive data model used in scientific simulations. In particular, we evaluate two realistic use cases: the blob detection in fusion and high-pressure area extraction in computational fluid dynamics.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2017

Metadata Evaluation and Improvement: Evolving Analysis and Reporting

ESIP Community members create and manage a large collection of environmental datasets that span multiple decades, the entire globe, and many parts of the solar system. Metadata are critical for discovering, accessing, using and understanding these data effectively and ESIP community members have successfully created large collections of metadata describing these data. As part of the White House Big Earth Data Initiative (BEDI), ESDIS has developed a suite of tools for evaluating these metadata in native dialects with respect to recommendations from many organizations. We will describe those tools and demonstrate evolving techniques for sharing results with data providers.

metadata recommendations↗
DOE OSTI2016

Towards Real-Time Detection and Tracking of Spatio-Temporal Features: Blob-Filaments in Fusion Plasma [Spatio-Temporal Features in Large Irregular Data: Blob-Filaments in Fusion Plasma]

This work was originally motivated the need to identify and track blob filaments, a feature often associated with instability in magnetically confined fusion plasma. Understanding and mitigating such instability would improve fusion reactors and make fusion a truly inexhaustible source of clean energy. Similar spatio-temporal features are important in many other applications, for example, ignition kernels in combustion and tumor cells in a medical image. This work presents an approach for extracting these features by dividing the overall task into three steps: local identification of feature cells, grouping feature cells into extended feature, and tracking movement of feature through overlapping in space. Through our extensive work in parallelization, we demonstrate that this approach can effectively make use of a large number of compute nodes to detect and track blobs in fusion plasma. On a set of 4.3TB data, we observed linear speedup on 1024 processes and completed blob detection in less than three milliseconds using Edison, a Cray XC30 system at NERSC.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2016

Big, Deep, and Smart Data in Scanning Probe Microscopy

Scanning probe microscopy techniques open the door to nanoscience and nanotechnology by enabling imaging and manipulation of structure and functionality of matter on nanometer and atomic scales. We analyze the discovery process by SPM in terms of information flow from tip-surface junction to the knowledge adoption by scientific community. Furthermore, we discuss the challenges and opportunities offered by merging of SPM and advanced data mining, visual analytics, and knowledge discovery technologies.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
NASA NTRS2012

Revamping Spacecraft Operational Intelligence with Splunk

So what is Splunk? Instead of giving the technical details, which you can find online, I'll tell you what it did for me. Splunk slapped everything into one place, with one uniform format, and gave me the ability to forget about all these annoying details of where it is, how to parse it, and all that. Instead, I only need to interact with Splunk to find the data I need. This sounds simple and obvious, but it's surprising what you can do once you all of your data is indexed in one place. By having your data organized, querying becomes much easier. Let's say that I want to search telemetry for a sensor_name gtemp_1 h and to return all data that is at most five minutes old. And because Splunk can hook into a real ]time stream, this data will always be up-to-date. Extending the previous example, I can now aggregate all types of data into one view based in time. In this picture, I've got transaction logs, telemetry, and downlinked files all in one page, organized by time. Even though the raw data looks completely than this, I've defined interfaces that transform it into this uniform format. This gives me a more complete picture for the question what was the spacecraft doing at this particular time? And because querying data is simple, I can start with a big block of data and whiddle it down to what I need, rather than hunting around for the individual pieces of data that I need. When we have all the data we need, we can begin widdling down the data with Splunk's Unix-like search syntax. These three examples highlights my trial-and-error attempts to find large temperature changes. I begin by showing the first 5 temperatures, only to find that they're sorted chronologically, rather than from highest temperatures to lowest temperatures. The next line shows sorting temperatures by their values, but I find that that fs not really what I want either. I want to know the delta temperatures between readings. Looking through Splunk's user manual, I find the delta function, which lets me dynamically generate new information to use in my query. With that extra piece of information, I can now return only the telemetry readings where the temperature changed by at least 10. One other useful feature I'll mention is that all of these queries can be run through Splunk's API. So any scripting language you can think of can plug right in and make these queries. This gives us the ability to build a lot of new tools.

operational intelligence↗