Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

Global Oil & Gas Infrastructure Features Database EDX Spatial Web Map

This submission offers a link to a web mapping application hosted instance of the Global Oil & Gas Features Database (GOGI), via EDX Spatial. This offers users with the ability to visualize, interact, and create maps with data of their choice, as well as download specific attributes or fields of view from the database. This data can also be downloaded as a File Geodatabse from EDX at https://edx.netl.doe.gov/dataset/global-oil-gas-features-database. Access the technical report describing how this database was produced using the following link: https://edx.netl.doe.gov/dataset/development-of-an-open-global-oil-and-gas-infrastructure-inventory-and-geodatabase” This data was developed using a combination of big data computing, custom search and data integration algorithms, and expert driven search to collect open oil and gas data resources worldwide. This approach identified over 380 data sets and integrated more than 4.8 million features into the GOGI database. Acknowledgements: This work was funded under the Climate and Clean Air Coalition (CCAC) Oil and Gas Methane Science Studies. The studies are managed by United Nations Environment in collaboration with the Office of the Chief Scientist, Steven Hamburg of the Environmental Defense Fund. Funding was provided by the Environmental Defense Fund, OGCI Companies (Shell, BP, ENI, Petrobras, Repsol, Total, Equinor, CNPC, Saudi Aramco, Exxon, Oxy, Chevron, Pemex) and CCAC.

gas↗
DOE OSTI2014

Petascale Hierarchical Modeling VIA Parallel Execution

The research allows more effective model building. By allowing researchers to fit complex models to large datasets in a scalable manner, our algorithms and software enable more effective scientific research. In the new area of “big data,” it is often necessary to fit “big models” to adjust for systematic differences between sample and population. For this task, scalable and efficient model-fitting tools are needed, and these have been achieved with our new Hamiltonian Monte Carlo algorithm, the no-U-turn sampler, and our new C++ program, Stan. In layman’s terms, our research enables researchers to create improved mathematical modes for large and complex systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Assessing Teleconnections-Induced Predictability of Regional Water Cycle on Seasonal to Decadal Timescales Using Machine Learning Approaches

Focal Area: (3) Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: Predictive understanding of the regional water cycle based on teleconnections of climate modes of variability using AI methods.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Development of Explainable, Knowledge-Guided AI Models to Enhance the E3SM Land Model Development and Uncertainty Quantification

Focal Area(s): (2)Predictive modeling using AI techniques and AI-derived model components; use of AI and other tools to design a prediction system comprising of a hierarchy of models. (3) Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge- guided AI. Science Challenge: The Energy Exascale Earth System Model (E3SM) is a fully coupled, state-of-the-science Earth system model that uses code optimized for DOE's advanced computers to address the most critical scientific questions facing our nation and society (Golaz et al., 2019). The E3SM Land model (ELM) is designed to understand how the changes in terrestrial land surfaces will interact with other Earth system components and has been used to understand hydrologic cycles, biogeophysics, and ecosystem dynamics. In spite of great successes, the ELM has several known issues that restrain rapid improvements. For example, the ELM uses equilibrium models to simulate dynamic land-climate interactions and it requires long model spin-up time to identify suitable initial conditions for transient simulations. The ELM lacks built-in uncertainty mechanisms that can improve the robustness of model predictions. The ELM is a holistic, deterministic model system with a rigid design, and in many situations, it is hard to modify the ELM system to incorporate new theory/hypothesis and new data across scales to address emerging science problems (such as predicting the impacts of water cycle extremes). In addition, The ELM is technically optimized for traditional CPU-centric computers and it cannot fully utilize the current and incoming leadership computers for model simulations and uncertainty quantification (UQ). The success of artificial intelligence (AI) has inspired scientists to use AI models to discover intrinsic features from simulation data (Chattopadhyay et al., 2020) and observational data (Reichstein et al., 2019) to gain further process understanding of Earth science problems. However, autonomous AI model training through deep learning usually requires a huge amount of annotated data. To overcome the limitations from the data and computing resources, knowledge-guided AI models are necessary where human-knowledge is ingested in model construction (Banino et al., 2018) and training process (Silver et al., 2016) for efficient learning. Herein, we present a new way that leverages the process understanding from the ELM to guide AI model development for the ELM enhancement and UQ. We hope this study can inspire further Earth and environmental system model developments and transformations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Facilitating better and faster simulations of aerosol-cloud interactions in Earth system models

Focal Area(s): 1. Predictive modeling through the use of AI techniques and AI-derived model components; the use of AI and other tools to design a prediction system comprising a hierarchy of models. 2. Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: One major challenge that Earth system models (ESMs) face in providing credible prediction of the Earth system and its water cycle characteristics (e.g., mean state, variability, and extreme events) is to accurately simulate aerosol-cloud interactions (ACI). The physical, chemical, and dynamical processes affecting ACI are extremely complex and they range from nanoscale to planetary scale. In each model development cycle, scientists spend significant efforts investigating model deficiencies and uncertainties associated with aerosols (e.g., emissions, chemical processes, aerosol microphysics, and transport) and clouds (e.g., macrophysics, microphysics, turbulence, and large-scale circulation) in order to develop improved treatments. However, despite decades of active research, ACI is still a major source of uncertainty in climate projections, even though great progress has been made. Specific scientific challenges include: (i) Parameterizations are developed based on limited data; (ii) The complexity of a parameterization required for accurate predictions is not understood; (iii) Incomplete and unknown physics leads to errors in the fully coupled Earth system; and (iv) Complex physics is computationally too expensive to employ in ESMs.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Geophysical Retrievals in an Artificial Intelligence (AI) Framework for Illuminating Processes Controlling Water Cycle

Focal Area(s): This white paper responds to Focal Area #3: Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: This white paper addresses the water-cycle and data-model integration grand challenge. It leverages data from the Atmospheric Radiation Measurement (ARM) Climate Research Facility, and Next-Generation Ecosystem Experiment (NGEE), and Science Focus Area (SFA). The white paper focuses on controlling cloud, precipitation, and radiative properties as observed and simulated by the Earth System Models (ESM). The described framework can be readily applied to any other ensemble of instruments, including satellites and other ground-based networks.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Improve wildfire predictability driven by extreme water cycle with interpretable physically-guided ML/AI

Focal Area(s): Predictive modeling using AI techniques and AI-derived model components; use of AI and other tools to design a prediction system comprising of a hierarchy of models (Primary); insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics or knowledge-guided AI (Secondary). Science Challenge: Wildfires modify land surface characteristics, such as vegetation composition, soil and litter carbon stocks, and surface albedo, with significant consequences for the regional carbon cycle. For example, tropical regions (i.e., African and South America) are particularly vulnerable to wildfire, account for more than 80% of the global burned area, and emit ~1.4 PgC y -1 into the atmosphere together with other dust and aerosols that strongly affect regional climate.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Representing the Unrepresented Impact of River Ice on Hydrology, Biogeochemistry, Vegetation, and Geomorphology: A Hybrid Physics-Machine Learning Approach

Focal areas include: Predictive modeling through the use of AI techniques and AI-derived model components; the use of AI and other tools to design a prediction system composed of a hierarchy of models (e.g., AI driven model/component/parameterization selection). Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Striped Data Analysis Framework

Traditionally, High Energy data analysis is based on the model where data are stored in files and analyzed by running multiple analysis processes, each reading one or more of the data files. This process involves repeated data reduction step, which produces smaller files, which is time consuming and leads to data duplication. We propose an alternative approach to data storage and analysis, based on the Big Data technologies. The idea is to store each element of data once and only once in a distributed scalable database and analyze data by reading only "interesting" pieces of data from the database. To make this approach possible, we developed columnar Striped Data Representation Format as the basis of the framework. Traditional columnar approach allows for efficient analysis of complex data structures. While keeping all the benefits of columnar data representation, striped mechanism goes further by enabling efficient parallelization of computations and flexible dist ribution of data analysis. The framework includes scalable and elastic data storage, compute and user analysis backend components. The framework uses off-the shelf web services and data caching technologies as the compute/data co-location mechanism. Flexible architecture allows the framework run in the cloud using container technologies. The framework offers Python/Jupyter as the user analysis backend platform, but can also run in command line or batch mode. In the article, we will present the results of the FNAL-LDRD-2016-032 FNAL LDRD project, the design, implementation, features and performance characteristics of the Striped Data Analysis Framework.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2017

Why wait? Let's start computing while data is still on the wire

In this era of Big Data, computing useful information from data is becoming increasingly complicated, particularly due to the ever increasing volumes of data that need to travel over the network to data centers to be stored and processed, all highly expensive operations in the long haul. In this paper we suggest that we can do computing and analysis of data "on the wire," i.e., while data is still in transit. The nature of these computations include analysis, visualization, pattern recognition, and prediction, or forecasting, on the streaming data. We follow a service function chaining architecture to implement this, assuming that the data packets arrive within a single network administrative domain. As a demonstration of this new computing paradigm, we present three examples. Firstly, we demonstrate pattern recognition and data visualization on streaming forex data, which can be used for lucrative trading in the forex market. In our second example, we analyze and learn user buying patterns from clickstream data streaming from multiple websites. Finally, we monitor solar sensors for a zero reading while the packets are still on their way to the data center, to schedule any maintenance and requisite repairs with no time delay.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Integrated Large-Scale Data Management Platform for Photovoltaic Power Conversion Equipment (PCE) Reliability Data: Preprint

To meet the demand for accuracy and real-time capability of PV system degradation evaluation, massive volume data is needed to run high-fidelity and high-efficiency simulations and perform advanced data analysis. However, PV farm operators have a series of difficulties with PV inverter data, such as data collection from multiple channels, massive data storage, data management and massive data analysis. To address these challenges, we developed an integrated data management platform capable of data acquisition, processing, storage, query, and performing big data analysis utilizing AI algorithms. The platform can also achieve data correctness verification and provide an effective distributed data management solution to retrieve massive data and establish a connection to distributed computational frameworks.

data management↗
DOE OSTI2020

FY2019 Energy Efficient Mobility Systems Annual Progress Report

EEMS Program activities during FY 2019 focused on analytical research to understand the impacts that new mobility technologies and services will have at the vehicle, traveler, and overall transportation system-level. This research included the development of vehicle and transportation system simulation models and tools to evaluate the complex interactions among the various actors within the mobility landscape, analysis of empirical data to characterize which solutions may provide the largest benefits, and development of new control systems and algorithms that use vehicle connectivity and automation to improve the performance and efficiency of individual vehicles as well as the overall traffic system. This document presents a brief overview of the EEMS Program and documents progress and results for projects within four of the five EEMS activity areas: (1) the SMART (Systems and Modeling for Accelerated Research in Transportation) Mobility Lab Consortium, (2) High Performance Computing and Big Data Solutions for Mobility Data, (3) Advanced R&D Projects conducted by industry and academia, and (4) Core Modeling, Simulation, and Evaluation, Similarly, the remaining EEMS activity area – (5) Living Labs (managed under VTO’s Technology Integration Program). Each of the individual progress reports provide a project overview and highlights of the technical results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2019

Energy Efficient Mobility Systems 2018 Annual Progress Report

Energy Efficient Mobility Systems (EEMS) Program activities during FY 2018 focused on analytical research to understand the impacts that new mobility technologies and services will have at the vehicle, traveler, and overall transportation system-level. This research included the development of vehicle and transportation system simulation models and tools to evaluate the complex interactions among the various actors within the mobility landscape, analysis of empirical data to characterize which solutions may provide the largest benefits, and development of new control systems and algorithms that use vehicle connectivity and automation to improve the performance and efficiency of individual vehicles as well as the overall traffic system This document presents a brief overview of the EEMS Program and documents progress and results for projects within four of the five EEMS activity areas: (1) the SMART (Systems and Modeling for Accelerated Research in Transportation) Mobility Lab Consortium, (2) High Performance Computing and Big Data Solutions for Mobility Data, (3) Advanced R&D Projects conducted by industry and academia, and (4) Core Modeling, Simulation, and Evaluation, Similarly, the remaining EEMS activity area – (5) Living Labs (managed under VTO’s Technology Integration Program). Each of the individual progress reports provide a project overview and highlights of the technical results.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS2015

Supercomputing in the Age of Discovering Superearths, Earths and Exoplanet Systems

With the Kepler Mission, exoplanet science has entered the era of Big Data and has benefited tremendously from the availability of the NAS (NASA Advanced Supercomputing Division) Pleiades supercomputer. This presentation provides an overview of the history of exoplanet discoveries and highlights the key enabling technologies for Kepler. I provide an overview of the Kepler science pipeline and the algorithms used to analyze the data and detect transiting planets, all of which run routinely on the NAS Pleiades system. The presentation also provides a description of the Blender analysis program used to validate Kepler planet discoveries statistically, and the effort to discover exomoons lurking in Kepler data, both of which also run on Pleiades. Finally, I describe NASA's TESS (Transiting Exoplanet Survey Satellite) Mission, which leverages Kepler heritage to provide a science pipeline similar to that for Kepler and that will also run routinely on the NAS Pleiades supercomputer.

big data↗
DOE OSTI2018

Autonomous Energy Grids

Current frameworks to monitor, control, and optimize large-scale energy systems are becoming increasingly inadequate because of significantly high penetration levels of variable generation and distributed energy resources being integrated into electric power systems; the deluge of data from pervasive metering of energy grids; and a variety of new market mechanisms, including multilevel ancillary services. This paper outlines the concept of autonomous energy grids (AEGs). These systems are supported by a scalable, reconfigurable, and self-organizing information and control infrastructure, are extremely secure and resilient (self-healing), and can self-optimize in real time to ensure economic and reliable performance while systematically integrating energy in all forms. AEGs rely on cellular building blocks that can self-optimize when isolated from a larger grid and participate in optimal operation when interconnected to a larger grid. This paper describes the key concepts and research necessary in the broad domains of optimization theory, control theory, big data analytics, and complex system theory and modeling to realize the AEG vision.

autonomous energy grids↗