SEARCH · Engineering Papers

Results for “big data transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

Big Data Challenges at CCMC

Like other research centers, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) at NASA Goddard Space Flight Center (GSFC) is also experiencing the big data challenges. CCMC hosts over 80 space weather models for Runs On Request (ROR), Continuous Runs and Instant Runs simulation services for the research community. In addition, CCMC has started to support simulation output onboarding in response to the Open Science initiative. Overall, we have accumulated over petabytes of simulation output data and are rapidly growing. In this presentation, we will discuss our data and storage challenges. We will present our attempts to address our challenges and any associated lessons learned. CCMC uses Apache Airflow to ensure data transfer is consistent. We will give a brief overview on how we leverage Apache Airflow to enhance our environment.

space weather↗

NASA NTRSDate not supplied

Big Data Challenges at the Community Coordinated Modeling Center (CCMC)

space weather↗

DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

NASA NTRS2022

Cosmological constraints on dark matter interactions with ordinary matter

Dark matter interactions with electrons or protons during the early Universe leave imprints on the cosmic microwave background and the matter power spectrum, and can be probed through cosmological and astrophysical observations. These interactions lead to momentum and heat exchange between the ordinary and dark matter components, which in turn results in a transfer of pressure from the ordinary to the dark matter. We explore these interactions using a diverse suite of data: cosmic microwave background anisotropies, baryon acoustic oscillations, the Lyman-α forest, and the abundance of Milky-Way subhalos. We derive constraints using model-independent parameterizations of the dark matter–electron and dark matter–proton interaction cross sections and map these constraints onto concrete dark matter models. Our constraints are complementary to other probes of dark matter interactions with ordinary matter, such as direct detection, big bang nucleosynthesis, various astrophysical systems, and accelerator based experiments. They exclude sufficiently large cross sections for a large range of dark matter masses, which cannot be accessed by direct-detection experiments due to the overburden from the Earth’s atmosphere or crust.

Manuel A. Buen-Abad↗

DOE OSTI2022

Cosmological constraints on dark matter interactions with ordinary matter

Dark matter interactions with electrons or protons during the early Universe leave imprints on the cosmic microwave background and the matter power spectrum, and can be probed through cosmological and astrophysical observations. These interactions lead to momentum and heat exchange between the ordinary and dark matter components, which in turn results in a transfer of pressure from the ordinary to the dark matter. Here, we explore these interactions using a diverse suite of data: cosmic microwave background anisotropies, baryon acoustic oscillations, the Lyman-α forest, and the abundance of Milky-Way subhalos. We derive constraints using model-independent parameterizations of the dark matter–electron and dark matter–proton interaction cross sections and map these constraints onto concrete dark matter models. Our constraints are complementary to other probes of dark matter interactions with ordinary matter, such as direct detection, big bang nucleosynthesis, various astrophysical systems, and accelerator-based experiments. They exclude sufficiently large cross sections for a large range of dark matter masses, which cannot be accessed by direct-detection experiments due to the overburden from the Earth’s atmosphere or crust.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Geo Thermal Cloud: Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources

The project is motivated by the challenges, risks, and costs associated with geothermal exploration and production. Many processes and parameters impacting geothermal conditions are poorly understood. Diverse datasets are available to help characterize subsurface geothermal conditions (public and proprietary; satellite, airborne surveys, vegetation/water sampling, geological, geophysical, etc.). Yet, it is not clear how to properly leverage these datasets for geothermal exploration due to an incomplete understanding of how physical processes impacting subsurface geothermal conditions are represented in these observations. Recent advancements in machine learning (ML) provide great promise to resolve these issues. The tremendous challenges and risks of geothermal exploration and production bring the demand for novel ML methods and tools that can (1) analyze large field datasets, (2) assimilate model simulations (large inputs and outputs), (3) process sparse datasets, (4) perform transfer learning (between sites with different exploratory levels), (5) extract hidden geothermal signatures in the field and simulation data, (6) label geothermal resources and processes, (7) identify high-value data acquisition targets, and (8) guide geothermal exploration and production by selecting optimal exploration, production, and drilling strategies. Our goals and work under Phases 1 and 2 (as proposed) of this project address all these needs.

15 GEOTHERMAL ENERGY↗

DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗

DOE OSTI2024

GeoThermalCloud: Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources

58 GEOSCIENCES↗

DOE OSTI2024

Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources

The primary goals of this project are identifying hidden geothermal resources in the USA and designing profitable enhanced geothermal systems (EGS). Many non-obvious processes and parameters could characterize geothermal resources and could control the ultimate energy potential of geothermal fields. Diverse datasets (e.g., geology, geochemistry, geophysics, satellite, airborne geophysics) are available to help characterize geothermal resources, but this data is sparse and multi-scale. This has hindered attempts to leverage the datasets for geothermal exploration and profitable EGS design. Recent advancements in machine learning (ML) give promise to overcome these issues. Modern ML methods and tools can (1) analyze large datasets, (2) assimilate model ensembles that include a multitude of inputs and outputs, (3) process sparse datasets, (4) perform transfer learning between sites with different data quality, (5) extract hidden geothermal signatures from field and simulation data, (6) label geothermal resources and processes, (7) identify high-value data acquisition targets, and (8) guide geothermal exploration and production by selecting optimal exploration, production, and drilling strategies. In this work, we implement ML-based geothermal exploration and an enhanced geothermal systems (EGS) design tool to achieve the above goals. Our exploration tool is GeoThermalCloud (GTC) EGS design tool is GeoDT-ML. GTC (github.com/SmartTensors/GeoThermalCloud.jl) utilizes a LANL unsupervised ML platform called SmartTensors (https://tensors.lanl.gov/) to automate data analyses and interpretations by extracting hidden signatures to identify geothermal prospects. It enables the identification of critical measurements needed to identify geothermal resource signatures. GeoDT-ML (github.com/SmartTensors/GeoThermalCloud.jl/tree/master/) adds coupling to GeoDT (https://github.com/GeoDesignTool/GeoDT.git) for stochastic EGS design optimization and performance prediction. GeoDT-ML leverages recent advances in deep learning and high-performance computing. Contributors to this effort include LANL, PNNL, Google, Stanford, and Julia Computing.

15 GEOTHERMAL ENERGY↗

NASA NTRS2001

Telemetry Tracking & Control (TT&C) - First TDRSS, then Commercial GEO & Big LEO and Now Through LEO

The advent of low earth orbit (LEO) commercial communication satellites provides an opportunity to dramatically reduce Telemetry, Tracking and Control (TT&C) costs of launch vehicles, Unpiloted Aerial Vehicles (UAVs), Research Balloons and spacecraft by reducing or eliminating ground infrastructure. Personnel from the Goddard Space Flight Center's Wallops Flight Facility (GSFC\WFF) have successfully used commercial Geostationary Earth Orbit (GEO) and Big LEO communications satellites for Long Duration Balloon Flight TT&C. The Flight Modem is a GSFC\WFF Advanced Range Technology initiative (ARTI) designed to streamline TT&C capability in the user community of these scientific data gathering platforms at low cost. Making use of existing LEO satellites and adapting and ruggedized commercially available components; two-way, over the horizon communications may be established with these vehicles at great savings due to reduced infrastructure. Initially planned as a means for permitting GPS data for tracking and recovery of sounding rocket and balloon payloads, expectations are that the bandwidth can soon be expanded to allow more comprehensive data transfer. The system architecture which integrates antennas, GPS receiver, commercial satellite packet data modem and a single board computer with custom software is described and technical challenges are discussed along with the plan for their resolution. A three-phase testing and development plan is outlined and the current results are reported. Results and status of ongoing flight tests on aircraft and sounding rockets are reported. Future applications on these platforms and the potential for satellite support are discussed along with an analysis of cost effectiveness of this method vs. other tracking and data transmission schemes.

Morgan, Dwayne R.↗

NASA NTRSDate not supplied

Assessment of Sensor Footprint Size and Comparison of Commercial Smallsat Images

Science users of commercial satellite data build their studies on the properties of the satellite data they work with, pixel size being one of the determining factors. However, pixel size and nominal image footprint size can be different, impacting the scale of features discernable in the satellite images. Here, we assess geometric properties such as nominal image footprint size for Planet Labs' SuperDove series, MAXAR's WorldView series, and potentially BlackSky. Additionally, we assess temporal change in nominal footprint size for the SuperDove series. Nominal Image footprint size is assessed over the CalVal sites in Baotou, China; Shadnagar, India; and Big Spring, TX. Edge spread functions are constructed along the black/white transitions, from which the line spread functions are constructed and modulation transfer functions of the scene are estimated. Scene full width half maximum (FWHM), which represents sensor footprint size, is estimated from the line spread function. The average nominal footprint size is 3.3 pixels for SuperDove series, 1.5 pixels for WorldView-2, and 1.3 pixels for WorldView-3. The SuperDove series nominal footprint size improves with time in orbit, from an average of 3.4 pixels soon after launch to an average of 3.2 pixels one or more years after launch.

PlanetScope↗

NASA NTRS1982

Contrast reduction by the atmosphere and retrieval of nonuniform surface reflectance

A radiative transfer model is developed which gives the upward radiance at nadir for any 1-D Lambertian surface reflectance. This model is used to depict the atmospheric effect on the transmittance of contrast for any 1-D surface reflectance. Here by contrast we mean a general variation of the radiation field across the image. With the aid of this model an inversion algorithm is developed for retrieval of true surface reflectance from high resolution satellite data (e.g., Landsat). This inversion technique can be a useful tool for extraction of surface reflectance from satellite data in the case of a surface reflectance variable in one dimension only (e.g., seashore or near borders of big fields). A sensitivity study of the inversion procedure on the knowledge of atmospheric parameters and sensor calibration was performed. It is shown that this inversion technique is stable even in the presence of errors in the sensor calibration and the atmospheric parameters. The method was applied to Landsat data in two wavelengths. The results show reasonable dependence of the derived surface reflectance on the distance from the seashore.

Mekler, Y.↗

DOE OSTI2024

Convergence in simulating global soil organic carbon by structurally different models after data assimilation

Abstract Current biogeochemical models produce carbon–climate feedback projections with large uncertainties, often attributed to their structural differences when simulating soil organic carbon (SOC) dynamics worldwide. However, choices of model parameter values that quantify the strength and represent properties of different soil carbon cycle processes could also contribute to model simulation uncertainties. Here, we demonstrate the critical role of using common observational data in reducing model uncertainty in estimates of global SOC storage. Two structurally different models featuring distinctive carbon pools, decomposition kinetics, and carbon transfer pathways simulate opposite global SOC distributions with their customary parameter values yet converge to similar results after being informed by the same global SOC database using a data assimilation approach. The converged spatial SOC simulations result from similar simulations in key model components such as carbon transfer efficiency, baseline decomposition rate, and environmental effects on carbon fluxes by these two models after data assimilation. Moreover, data assimilation results suggest equally effective simulations of SOC using models following either first‐order or Michaelis–Menten kinetics at the global scale. Nevertheless, a wider range of data with high‐quality control and assurance are needed to further constrain SOC dynamics simulations and reduce unconstrained parameters. New sets of data, such as microbial genomics‐function relationships, may also suggest novel structures to account for in future model development. Overall, our results highlight the importance of observational data in informing model development and constraining model predictions.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2019

Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams

Data quality assessment, management and improvement is integral part of any big data intensive scientific research to ensure accurate, reliable and reproducible science discoveries. The task of maintaining the quality of data, however, is non-trivial and pose a challenge for a program like Department of Energy’s Atmospheric Radiation Measurement (ARM) that collects data from hundreds of instruments across the world,and distributes thousands of streaming data products that are continuously growing in near-real-time in an archive 1.7 Petabyte in size and growing. We present a computational data processing workflow to collect the data quality issue via an easy and intuitive web-based portal that allows reporting of any quality issues for any site, facility or instruments at a granularity down to individual variable in the data files. Portal allows instrument specialists and scientists to provide corrective actions in form of symbolic equation. A parallel processing framework applies the data improvement to large volume of data in efficient parallel environment, while optimizing data transfer and file I/O operations. Corrected files are systematically versioned and archived. A provenance tracking module tracks and records any change made to the data during its entire life cycle which are communicated transparently to the scientific users. Developed in Python using open source technologies, software architecture enables efficient and fast management and improvement of data in an operational data center environment.

Kumar, Jitendra↗

DOE OSTI2024

Investigating resource-efficient neutron/gamma classification ML models targeting eFPGAs

There has been considerable interest and resulting progress in implementing machine learning (ML) models in hardware over the last several years from the particle and nuclear physics communities. A big driver has been the release of the Python package, hls4ml, which has enabled porting models specified and trained using Python ML libraries to register transfer level (RTL) code. So far, the primary end targets have been commercial field-programmable gate arrays (FPGAs) or synthesized custom blocks on application specific integrated circuits (ASICs). However, recent developments in open-source embedded FPGA (eFPGA) frameworks now provide an alternate, more flexible pathway for implementing ML models in hardware. These customized eFPGA fabrics can be integrated as part of an overall chip design. In general, the decision between a fully custom, eFPGA, or commercial FPGA ML implementation will depend on the details of the end-use application. In this work, we explored the parameter space for eFPGA implementations of fully-connected neural network (fcNN) and boosted decision tree (BDT) models using the task of neutron/gamma classification with a specific focus on resource efficiency. We used data collected using an AmBe sealed source incident on Stilbene, which was optically coupled to an OnSemi J-series silicon photomultiplier (SiPM) to generate training and test data for this study. We investigated relevant input features and the effects of bit-resolution and sampling rate as well as trade-offs in hyperparameters for both ML architectures while tracking total resource usage. The performance metric used to track model performance was the calculated neutron efficiency at a gamma leakage of 10 -3 . The results of the study will be used to aid the specification of an eFPGA fabric, which will be integrated as part of a test chip.

47 OTHER INSTRUMENTATION↗

DOE OSTI2020

State-of-the-art on research and applications of machine learning in the building life cycle

Fueled by big data, powerful and affordable computing resources, and advanced algorithms, machine learning has been explored and applied to buildings research for the past decades and has demonstrated its potential to enhance building performance. This study systematically surveyed how machine learning has been applied at different stages of building life cycle. By conducting a literature search on the Web of Knowledge platform, we found 9579 papers in this field and selected 153 papers for an in-depth review. The number of published papers is increasing year by year, with a focus on building design, operation, and control. However, no study was found using machine learning in building commissioning. There are successful pilot studies on fault detection and diagnosis of HVAC equipment and systems, load prediction, energy baseline estimate, load shape clustering, occupancy prediction, and learning occupant behaviors and energy use patterns. None of the existing studies were adopted broadly by the building industry, due to common challenges including (1) lack of large scale labeled data to train and validate the model, (2) lack of model transferability, which limits a model trained with one data-rich building to be used in another building with limited data, (3) lack of strong justification of costs and benefits of deploying machine learning, and (4) the performance might not be reliable and robust for the stated goals, as the method might work for some buildings but could not be generalized to others. Finally, findings from the study can inform future machine learning research to improve occupant comfort, energy efficiency, demand flexibility, and resilience of buildings, as well as to inspire young researchers in the field to explore multidisciplinary approaches that integrate building science, computing science, data science, and social science.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2019

Transfer Request from Rutgers University to University of Illinois Urbana Champaign

Shortly after the Big Bang the entire universe was filled with a nearly perfect fluid known as the Quark Gluon Plasma. Relativistic heavy ion collisions can now reproduce this fluid in the laboratory where the Quark Gluon Plasma exhibits a rapid but smooth cross-over phase transition into hadrons at vanishing net-baryon densities. Recent experiments plan to explore finite baryon densities where a critical point is expected. If found, this would mark the first discovery of a critical point in a relativistic system described by a fundamental theory of nature, which would have far reaching consequences for high-energy nuclear physics and nuclear astrophysics (such as in neutron star mergers). Characteristic temperatures of equilibrium (e.g. the inflection point of the entropy density) and transport coefficients (e.g. minimum of the shear viscosity over entropy density) vary widely at a cross-over phase transition but converge at a critical point, and extracting the behavior of these characteristic temperatures is a major focal point of this research. Specifically, the interplay between strange and light hadrons will be exploited to study the flavor hierarchy in the cross-over region. To investigate this, a viscous relativistic hydrodynamics framework with two conserved charges must be developed into a new open-source code along with initial conditions that contain baryon number and strangeness. Flow observables sensitive to the equation of state and transport coefficients will be calculated across beam energies. New techniques will be developed to study this flavor hierarchy from first principles and to extract the characteristic temperatures from experimental data. Through this new dynamical framework, this project will provide essential guidance to the upcoming Beam Energy Scan II runs at Relativistic Heavy-Ion Collider and the future Facility for Antiproton and Ion Research facility in the search for the Quantum Chromodynamic critical point and subsequent investigation of the baryon-rich Quark Gluon Plasma.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2026

Unified Modeling Architecture for Load Management in Extreme Heat: The New York City Case

Integration of renewable resources to meet growing energy demand is becoming a global priority under decarbonization mandates. This study contributes to ongoing efforts on this key subject by assessing the feasibility of using coastal-urban renewable energy resources, namely, offshore wind and rooftop photovoltaic systems, to meet electricity demand of New York City during the intense recent heat wave period of June 2025. A unified modeling framework, based on the urbanized weather research and forecasting model, is used to simulate climate, renewable resources, and energy demand variables. Findings show significant energy load mismatch of approximately 1150 GWh over the month, between the demand and the combined renewable generation outcome. Three storage integration scenarios are analyzed to mitigate the deficits, reducing said deficits by a minimum of approximately 9% over the duration of the month. This study provides a transferable modeling framework tool for evaluating renewable integration in dense urban environments that can be used by grid operators to support grid resilience during extreme heat events.

54 ENVIRONMENTAL SCIENCES↗