Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “big data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Findings on Subtask 3.1 - Bakken Rich Gas Enhanced Oil Recovery Project

Total in-place oil for the Bakken petroleum system (BPS) (which includes the Bakken and Three Forks Formations) has been estimated to be 600 billion barrels (bbl). However, BPS wells have decline rates as high as 85% over the first 3 years of their lives, and primary recovery factors typically range from 3% to 10% of original oil in place. Given the low initial recovery rates, even small incremental productivity improvements could dramatically increase technically recoverable oil in the BPS. One potential solution is enhanced oil recovery (EOR) using gas injection, such as carbon dioxide (CO2) or hydrocarbon (HC) gases. While commonly used in conventional reservoirs, CO2 EOR in unconventional tight oil reservoirs has been limited to pilot tests. EOR using rich gas (mixture of methane, ethane, and propane) has also been employed in numerous pilots in several unconventional plays and has recently been successfully applied in the Eagle Ford play. If successful, large-scale gas-based EOR in the BPS could dramatically increase oil productivity and recovery factors and extend the life of the play for decades. While CO2 may be a technically suitable working fluid for EOR in the BPS, supplies are limited and costs for using CO2 in EOR pilots are prohibitively high. Meanwhile, produced gas flaring has presented challenges for BPS operators in North Dakota. Analysis conducted by the North Dakota Pipeline Authority indicates that the current gas-gathering infrastructure in North Dakota is insufficient to accommodate all of the associated gas that is produced from the BPS. The geographically isolated location of North Dakota relative to large natural gas markets, combined with suppressed natural gas prices, has made it economically challenging for industry to invest capital in expanding gas-gathering infrastructure in the state. These circumstances led to a research program conducted by the Energy & Environmental Research Center (EERC) in partnership with Liberty Resources Management Company LLC (LR) to examine the potential to use rich gas injection for EOR and mitigate flaring. A rich gas EOR pilot test was designed and executed by LR at its Stomping Horse development area in Williams County, North Dakota. From July 2018 through May 2019, a total of 160 million standard cubic feet (MMscf) of rich produced gas was injected into the BPS using five different wells in a sequential injection strategy. LR’s Leon–Gohrick drill spacing unit (DSU) was used as the test site. Regulatory oversight was provided by the North Dakota Industrial Commission (NDIC). Technical support was provided by the EERC through a series of laboratory, modeling, and field-based activities, and additional post-pilot research activities incorporated learnings from the test, developed new laboratory data, improved fracture modeling methods, and developed machine learning and big data analytics. The results from the Stomping Horse rich gas EOR pilot activities indicate that developing an effective, economical EOR approach for the BPS will require more field tests. Another key lesson learned from the Stomping Horse tests is that detailed pre- and posttest data on reservoir conditions and fluids production are essential. Robust reservoir characterization provides information that is crucial to creating realistic geomodels and conducting valid dynamic simulations of potential EOR scenarios. A detailed understanding of the completions and production history of offset wells is also necessary for valid test result interpretations. This knowledge is essential to designing the operational parameters of injectivity tests and interpreting the results. A conformance control strategy is also essential to success. Laboratory-based examinations of rich gas interactions with reservoir fluids and rocks were conducted, with an emphasis on determining the ability to mobilize oil in the tight reservoir rocks and shales of the BPS. Injection fluid composition was shown to have a positive impact on reducing reservoir oil minimum miscibility pressure (MMP), reducing interfacial tension (IFT), and altering wettability. IFT and contact angle measurements demonstrated that wettability can be altered in the presence of rich gas, suggesting the potential to improve oil recovery. Iterative modeling of surface infrastructure and reservoir performance using data generated by the various project activities was conducted. A geologic model of the Stomping Horse area was built; history-matched oil, gas, and water production was used in simulations of various EOR scenarios. Early programmatic modeling results were used to support LR’s design and operation of the EOR pilot and to provide insight regarding optimization of future commercial-scale BPS EOR design and operations. Post-pilot modeling focused on alternative methods of understanding complex fracture networks and accelerating simulation time. These led to improved simulation run times and provide excellent history-matching results. Several of these iterative models were used as the bases for developing algorithms into machine learning and big data analytics. History matching in reservoir simulation is time-consuming and computer processing-intensive. Machine learning algorithms were created, and an automated history-matching tool was developed. A large set of synthetic reservoir simulations were created to generate well responses (oil, gas, and water production, well bottomhole pressure [BHP], and tracer or propane breakthrough) for a set of EOR operating parameters that included offset well status (open or closed), injectate (rich gas or propane), injection rate, and injection well BHP. A user interface was developed to provide real-time visualization. Machine learning-based models were developed to provide rapid forecasting of well performance given a set of user-defined EOR operating parameters. These predictive models allow the user to modify the offset well status, injection rate, and injection well BHP and rapidly forecast future production performance. The combination of real-time visualization tools with real-time forecasting tools provides a framework for real-time control—operational changes that the EOR site operator can enact (e.g., changing gas injection rates) to affect the observed performance and potentially improve the EOR outcome. There is great reason to be optimistic about the future of EOR in the Bakken. The results of the laboratory studies suggest significant potential for high rates of oil mobilization using produced field gas injection under the right conditions. The results of the lab studies, combined with rigorous statistical analysis of well production data and associated modeling efforts, confirm the notion that fluid mobility within the reservoir is controlled by fractures. As more knowledge is gained about the nature and distribution of fracture networks in the Bakken, the industry will be in a better position to predict and, ultimately, influence fluid mobility. New field tests are necessary to develop a more complete understanding of those conditions. Thoughtful and creatively engineered field tests within a well-characterized geologic setting will yield the fundamental knowledge needed to take Bakken oil production to the next level. This subtask was cofunded through the EERC–U.S. Department of Energy Joint Program on Research and Development for Fossil Energy-Related Resources Cooperative Agreement No. DE-FE0024233. Nonfederal funding was provided by the North Dakota Industrial Commission’s Oil and Gas Research Program and Computer Modelling Group.

04 OIL SHALES AND TAR SANDS↗
DOE OSTI2024

Efficient First-Order Algorithms for Large-Scale, Non-Smooth Maximum Entropy Models with Application to Wildfire Science

Maximum entropy (MaxEnt) models are a class of statistical models that use the maximum entropy principle to estimate probability distributions from data. Due to the size of modern data sets, MaxEnt models need efficient optimization algorithms to scale well for big data applications. State-of-the-art algorithms for MaxEnt models, however, were not originally designed to handle big data sets; these algorithms either rely on technical devices that may yield unreliable numerical results, scale poorly, or require smoothness assumptions that many practical MaxEnt models lack. In this paper, we present novel optimization algorithms that overcome the shortcomings of state-of-the-art algorithms for training large-scale, non-smooth MaxEnt models. Our proposed first-order algorithms leverage the Kullback–Leibler divergence to train large-scale and non-smooth MaxEnt models efficiently. For MaxEnt models with discrete probability distribution of n elements built from samples, each containing m features, the stepsize parameter estimation and iterations in our algorithms scale on the order of O(mn) operations and can be trivially parallelized. Moreover, the strong ℓ1 convexity of the Kullback–Leibler divergence allows for larger stepsize parameters, thereby speeding up the convergence rate of our algorithms. To illustrate the efficiency of our novel algorithms, we consider the problem of estimating probabilities of fire occurrences as a function of ecological features in the Western US MTBS-Interagency wildfire data set. Our numerical results show that our algorithms outperform the state of the art by one order of magnitude and yield results that agree with physical models of wildfire occurrence and previous statistical analyses of wildfire drivers.

Physics↗
DOE OSTI2019

A Machine Learning Approach for the Discovery of Ligand-Specific Functional Mechanisms of GPCRs

G protein-coupled receptors (GPCRs) play a key role in many cellular signaling mechanisms, and must select among multiple coupling possibilities in a ligand-specific manner in order to carry out a myriad of functions in diverse cellular contexts. Much has been learned about the molecular mechanisms of ligand-GPCR complexes from Molecular Dynamics (MD) simulations. However, to explore ligand-specific differences in the response of a GPCR to diverse ligands, as is required to understand ligand bias and functional selectivity, necessitates creating very large amounts of data from the needed large-scale simulations. This becomes a Big Data problem for the high dimensionality analysis of the accumulated trajectories. Here we describe a new machine learning (ML) approach to the problem that is based on transforming the analysis of GPCR function-related, ligand-specific differences encoded in the MD simulation trajectories into a representation recognizable by state-of-the-art deep learning object recognition technology. We illustrate this method by applying it to recognize the pharmacological classification of ligands bound to the 5-HT 2A and D2 subtypes of class-A GPCRs from the serotonin and dopamine families. The ML-based approach is shown to perform the classification task with high accuracy, and we identify the molecular determinants of the classifications in the context of GPCR structure and function. This study builds a framework for the efficient computational analysis of MD Big Data collected for the purpose of understanding ligand-specific GPCR activity.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Virtual Log-Structured Storage for High-Performance Streaming

Over the past decade, given the higher number of data sources (e.g., Cloud applications, Internet of things) and critical business demands, Big Data transitioned from batch-oriented to real-time analytics. Stream storage systems, such as Apache Kafka, are well known for their increasing role in real-time Big Data analytics. For scalable stream data ingestion and processing, they logically split a data stream topic into multiple partitions. Stream storage systems keep multiple data stream copies to protect against data loss while implementing a stream partition as a replicated log. This architectural choice enables simplified development while trading cluster size with performance and the number of streams optimally managed. This paper introduces a shared virtual log-structured storage approach for improving the cluster throughput when multiple producers and consumers write and consume in parallel data streams. Stream partitions are associated with shared replicated virtual logs transparently to the user, effectively separating the implementation of stream partitioning (and data ordering) from data replication (and durability). We implement the virtual log technique in the KerA stream storage system. When comparing with Apache Kafka, KerA improves the cluster ingestion throughput by up to 4x when multiple producers write over hundreds of data streams.

consistent stream ordering↗
DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

ASCR@40

One page flyers on: A Revolution in Modeling and Simulation; Applying Equations to Complex Problems; Uncertainty Quantification; Moving Big Data; Grid Computing for High-Speed Collaboration; Making Sense of Big Data; Overcoming Scaling Challenges; Building Better Computers; World-Leading Computing Facilities; Supporting Science Through Open-Source Software; and Building the Computational Workforce. Content provided by Department of Energy multiprogram laboratory researchers. Prepared by the Krell Institute for the ASCAC Subcommittee on the 40-year History of ASCR.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

A Survey of Singular Value Decomposition Methods for Distributed Tall/Skinny Data

The Singular Value Decomposition (SVD) is one of the most important matrix factorizations, enjoying a wide variety of applications across numerous application domains. In statistics and data analysis, the common applications of SVD inclue Principal Components Analysis (PCA) and regression. Usually these applications arise on data that has far more rows than columns, so-called "tall/skinny" matrices. In the big data analytics context, this may take the form of hundreds of millions to billions of rows with only a few hundred columns. There is a need, therefore, for fast, accurate, and scalable tall/skinny SVD implementations which can fully utilize modern computing resources. To that end, we present a survey of three different algorithms for computing the SVD for these kinds of tall/skinny data layouts using MPI for communication. We contextualize these with common big data analytics techniques. Finally, we present both CPU and GPU timing results from the Summit supercomputer, and discuss possible alternative approaches.

Schmidt, Drew↗
DOE OSTI2022

Reviews and syntheses: The promise of big diverse soil data, moving current practices towards future potential

Abstract. In the age of big data, soil data are more available and richer than ever, but – outside of a few large soil survey resources – they remain largely unusable for informing soil management and understanding Earth system processes beyond the original study. Data science has promised a fully reusable research pipeline where data from past studies are used to contextualize new findings and reanalyzed for new insight. Yet synthesis projects encounter challenges at all steps of the data reuse pipeline, including unavailable data, labor-intensive transcription of datasets, incomplete metadata, and a lack of communication between collaborators. Here, using insights from a diversity of soil, data, and climate scientists, we summarize current practices in soil data synthesis across all stages of database creation: availability, input, harmonization, curation, and publication. We then suggest new soil-focused semantic tools to improve existing data pipelines, such as ontologies, vocabulary lists, and community practices. Our goal is to provide the soil data community with an overview of current practices in soil data and where we need to go to fully leverage big data to solve soil problems in the next century.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

Data Transfer Advisor with Transport Profiling Optimization

The network infrastructures have been rapidly upgraded in many high-performance networks (HPNs). However, such infrastructure investment has not led to corresponding performance improvement in big data transfer, especially at the application layer, largely due to the complexity of optimizing transport control on end hosts. We design and implement ProbData, a PRofiling Optimization Based DAta Transfer Advisor, to help users determine the most effective data transfer method with the most appropriate control parameter values to achieve the best data transfer performance. ProbData employs a profiling optimization based approach to exploit the optimal operational zone of various data transfer methods in support of big data transfer in extreme scale scientific applications. We present a theoretical framework of the optimized profiling approach employed in ProbData as wellas its detailed design and implementation. The advising procedure and performance benefits of ProbData are illustrated and evaluated by proof-of-concept experiments in real-life networks.

Rao, Nageswara S.↗
DOE OSTI2014

MIC-SVM: Designing A Highly Efficient Support Vector Machine For Advanced Modern Multi-Core and Many-Core Architectures

Support Vector Machine (SVM) has been widely used in data-mining and Big Data applications as modern commercial databases start to attach an increasing importance to the analytic capabilities. In recent years, SVM was adapted to the field of High Performance Computing for power/performance prediction, auto-tuning, and runtime scheduling. However, even at the risk of losing prediction accuracy due to insufficient runtime information, researchers can only afford to apply offline model training to avoid significant runtime training overhead. To address the challenges above, we designed and implemented MICSVM, a highly efficient parallel SVM for x86 based multi-core and many core architectures, such as the Intel Ivy Bridge CPUs and Intel Xeon Phi coprocessor (MIC).

42 ENGINEERING↗
DOE OSTI2022

A Multi-Branch Decoder Network Approach to Adaptive Temporal Data Selection and Reconstruction for Big Scientific Simulation Data

A key challenge in scientific simulation is that the simulation outputs often require intensive I/O and storage space to store the results for effective post hoc analysis. This article focuses on a quality-aware adaptive temporal data selection and reconstruction problem where the goal is to adaptively select simulation data samples at certain key timesteps in situ and reconstruct the discarded samples with quality assurance during post hoc analysis. This problem is motivated by the limitation of current solutions that a significant amount of simulation data samples are either discarded or aggregated during the sampling process, leading to inaccurate modeling of the simulated phenomena. Two unique challenges exist: 1) the sampling decisions have to be made in situ and adapted to the dynamics of the complex scientific simulation data; 2) the reconstruction error must be strictly bounded to meet the application requirement. To address the above challenges, we develop DeepSample , an error-controlled convolutional neural network framework, that jointly integrates a set of coherent multi-branch deep decoders to effectively reconstruct the simulation data with rigorous quality assurance. The results on two real-world scientific simulation applications show that DeepSample significantly outperforms other state-of-the-art methods on both sampling efficiency and reconstructed simulation data quality.

Zhang, Yang↗
DOE OSTI2018

A Framework for Privacy-Preserving Data Publishing with Enhanced Utility for Cyber-Physical Systems

Cyber-physical systems have enabled the collection of massive amounts of data in an unprecedented level of spatial and temporal granularity. Publishing these data can prosper big data research, which, in turn, helps improve overall system efficiency and resiliency. The main challenge in data publishing is to ensure the usefulness of published data while providing necessary privacy protection. In our previous work (Jia et al. 2017a), we presented a privacy-preserving data publishing framework (referred to as PAD hereinafter), which can guarantee k-anonymity while achieving better data utility than traditional anonymization techniques. PAD learns the information of interest to data users or features from their interactions with the data publishing system and then customizes data publishing processes to the intended use of data. However, our previous work is only applicable to the case where the desired features are linear in the original data record. In this article, we extend PAD to nonlinear features. Our experiments demonstrate that for various data-driven applications, PAD can achieve enhanced utility while remaining highly resilient to privacy threats.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2021

An AI-Enabled MODEX Framework for Improving Predictability of Subsurface Water Storage across Local and Continental Scales

Focal Area: (2) Predictive modeling through the use of AI techniques and AI-derived model components. (3) Insight gleaned from complex data using AI, big data analytics, and other advanced methods. We propose an AI-enabled model-experiment (MODEX) framework to improve the predictability of subsurface water storage (SWS) from local to conus scales in a changing environment by taking advantage of DOE’s observation and simulation capabilities, as well as to inform the model and the observation development.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

In Situ Inference for Earth System Predictability

Focal Area: Focal Area 3: Insight gleaned from complex simulated data using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: An understanding of future evolution in precipitation extremes is critical to numerous DOE mission questions. Extreme events are by nature short time-scale events that are difficult to diagnose in available model data. Accurate modeling of extreme events necessarily requires high spatial resolution at the storm scale locally. However, the environment in which storms grow is dependent on global, remote, processes. These complex spatiotemporal relationships are impossible to diagnose at resolutions required to accurately model storms responsible for extreme precipitation. At exascale, climate simulations will produce results at fine enough resolution to investigate these relationships. However, the resulting data from these simulations will be far too large to save for post-simulation analysis. We advocate for fitting statistical models inside the simulations as they run, a context known as in situ, which will facilitate scientific investigations using the full fine-scale data stream. Figure 1 shows an example of the type of model we could consider, a Bayesian hierarchical spatial regression model. Precipitation extremes at each grid cell are modeled using extreme value distributions. Since extremes are rare, fitting models to individual grid cells can result in high variance and poor estimates. Instead, the model can be made more robust by smoothing the parameters of the extreme value model across space. Additionally, the parameters themselves can be functionally linked to other variables elsewhere in the simulation. Thus, we can use the fine-scale data to build more robust models for extremes that link extreme behavior to other climate patterns.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2014

Centralized Analysis of Local Data, With Dollars and Lives on the Line: Lessons From The Home Radon Experience

In this chapter we elucidate four main themes. The first is that modern data analyses, including "Big Data" analyses, often rely on data from different sources, which can present challenges in constructing statistical models that can make effective use of all of the data. The second theme is that although data analysis is usually centralized, frequently the final outcome is to provide information or allow decision-making for individuals. Third, data analyses often have multiple uses by design: the outcomes of the analysis are intended to be used by more than one person or group, for more than one purpose. Finally, issues of privacy and confidentiality can cause problems in more subtle ways than are usually considered; we will illustrate this point by discussing a case in which there is substantial and effective political opposition to simply acknowledging the geographic distribution of a health hazard. A researcher analyzes some data and learns something important. What happens next? What does it take for the results to make a difference in people's lives? In this chapter we tell a story - a true story - about a statistical analysis that should have changed government policy, but didn't. The project was a research success that did not make its way into policy, and we think it provides some useful insights into the interplay between locally-collected data, statistical analysis, and individual decision making.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2015

mdtmFTP

To address the high-performance challenges of data transfer in big data era, we research, develop, and implement mdtmFTP: a High-performance Data Transfer Tool in Bigdata Era. DOE’s Advanced Scientific Computing Research (ASCR) office has funded Fermilab and Brookhaven National Laboratory to collaboratively work on the Multicore-Aware Data Transfer Middleware (MDTM) project. MDTM aims to accelerate data movement toolkits on multicore systems. mdtmFTP is the latest outcome of this continued research effort. mdtmFTP is a high-performance data transfer tool that builds upon the MDTM middleware. Initial tests show that mdtmFTP performs better than existing data transfer tools

Zhang, L.↗