SEARCH · Engineering Papers

Results for “frameworks and databases”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 559 records · Page 31

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗

DOE OSTI2020

Global patterns of terrestrial nitrogen and phosphorus limitation

Nitrogen (N) and phosphorus (P) limitation constrains the magnitude of terrestrial carbon uptake in response to elevated carbon dioxide and climate change. However, global maps of nutrient limitation are still lacking. Here we examined global N and P limitation using the ratio of site-averaged leaf N and P resorption efficiencies of the dominant species across 171 sites. We evaluated our predictions using a global database of N- and P-limitation experiments based on nutrient additions at 106 and 53 sites, respectively. Globally, we found a shift from relative P to N limitation for both higher latitudes and precipitation seasonality and lower mean annual temperature, temperature seasonality, mean annual precipitation and soil clay fraction. Excluding cropland, urban and glacial areas, we estimate that 18% of the natural terrestrial land area is significantly limited by N, whereas 43% is relatively P limited. The remaining 39% of the natural terrestrial land area could be co-limited by N and P or weakly limited by either nutrient alone. This work provides both a new framework for testing nutrient limitation and a benchmark of N and P limitation for models to constrain predictions of the terrestrial carbon sink.

58 GEOSCIENCES↗

DOE OSTI2026

Modernizing GlideinWMS Factory Monitoring with Prometheus & Grafana

Large-scale scientific experiments like CMS and DUNE rely on the distributed workload management system GlideinWMS to efficiently utilize computing resources across heterogeneous computing environments. GlideinWMS currently records Factory statistics using Round Robin Databases (RRDBs), XML, and JSON files, and these statistics are displayed via custom monitoring Web pages, thereby limiting integration with modern observability platforms. This project investigates the use of Prometheus-based instrumentation to expose Factory metrics using OpenTelemetry principles. Factory statistics related to Glidein submission and job execution are exported as Prometheus metrics through the Prometheus Python Client Library and are served via an HTTP metrics endpoint. The collected metrics are inspected using the Prometheus web-based interface and are visualized through Grafana dashboards within the Landscape monitoring infrastructure at Fermilab. This project significantly streamlines the integration of modern monitoring technologies into GlideinWMS and establishes a framework for extending observability across additional system components.

Appiah, Gideon [Grambling State U.]↗

NASA NTRS2024

Automatic Detection and Classification of Aurora in THEMIS All‐Sky Images

We report a novel machine-learning algorithm for automatically detecting and classifying aurora in all–sky images (ASI) that is largely trained without requiring ground–truth labels. By including a small number of labeled images, we are able to automatically label all of the approximately 700 million images in the Time History of Events and Macroscale Interactions during Substorms (THEMIS) ASI data set from 2008 to 2022. We use a two–stage approach. In the first stage, we adapt the Simple framework for Contrastive Learning of Representations (SimCLR) algorithm to learn latent representations of THEMIS all–sky images. We then finetune a classifier network on the latent representations our model learns of the manually labeled Oslo aurora THEMIS (OATH) data set. We demonstrate that this two–stage approach achieves excellent classification results on data for which there is no current ML classification benchmark. The outcome of this work will facilitate efficient information retrieval for researchers interested in specific categories of aurora and will enable large scale statistical studies and machine learning analyses of THEMIS all–sky images that have not previously been possible. To demonstrate possible ways to utilize this database, we performed a statistical analysis of the occurrence rates of auroral labels with respect to solar wind parameters, interplanetary magnetic field vector, and geomagnetic indices. We further investigate the occurrence rates of auroral phenomena in the annotated data set and their geoeffectiveness by utilizing the co–located THEMIS ground magnetometer data set.

Jeremiah W Johnson↗

DOE OSTI2021

Physics-Guided Machine Learning for Prediction of Cloud Properties in Satellite-Derived Solar Data

With over 20 years of high-resolution surface irradiance data covering most of the western hemisphere, the National Solar Radiation Database (NSRDB) is a vital public data asset. The NSRDB uses a two-step Physical Solar Model (PSM) that explicitly considers the effects of clouds and other atmospheric variables on radiative transfer. High-quality physical and optical cloud properties derived from satellite imagery are perhaps the most important data inputs to the PSM, representing the greatest source of radiation attenuation and scattering. However, traditional methods for cloud property retrieval have their own limitations and are unable to accurately predict cloud properties outside of nominal conditions. We introduce a physics-guided neural network that can accurately predict cloud properties when traditional methods fail or are inaccurate. Using this framework, we show reductions in relative Root Mean Square Error (RMSE) for Global Horizontal Irradiance (GHI) up to 13 percentage points for timesteps that previously had missing or low-quality cloud property data. We expect that this methodology will be effective in improving the quality of cloud property and solar irradiance data in the NSRDB.

cloud properties↗

NASA NTRS2014

Soil Moisture Data Assimilation in the NASA Land Information System for Local Modeling Applications and Improved Situational Awareness

As part of the NASA Soil Moisture Active Passive (SMAP) Early Adopter (EA) program, the NASA Shortterm Prediction Research and Transition (SPoRT) Center has implemented a data assimilation (DA) routine into the NASA Land Information System (LIS) for soil moisture retrievals from the European Space Agency's Soil Moisture Ocean Salinity (SMOS) satellite. The SMAP EA program promotes application‐driven research to provide a fundamental understanding of how SMAP data products will be used to improve decision‐making at operational agencies. SPoRT has partnered with select NOAA/NWS Weather Forecast Offices (WFOs) that use output from a real‐time regional configuration of LIS, without soil moisture DA, to initialize local numerical weather prediction (NWP) models and enhance situational awareness. Improvements to local NWP with the current LIS have been demonstrated; however, a better representation of the land surface through assimilation of SMOS (and eventually SMAP) retrievals is expected to lead to further model improvement, particularly during warm‐season months. SPoRT will collaborate with select WFOs to assess the impact of soil moisture DA on operational forecast situations. Assimilation of the legacy SMOS instrument data provides an opportunity to develop expertise in preparation for using SMAP data products shortly after the scheduled launch on 5 November 2014. SMOS contains a passive L‐band radiometer that is used to retrieve surface soil moisture at 35‐km resolution with an accuracy of 0.04 cu cm cm (exp -3). SMAP will feature a comparable passive L‐band instrument in conjunction with a 3‐km resolution active radar component of slightly degraded accuracy. A combined radar‐radiometer product will offer unprecedented global coverage of soil moisture at high spatial resolution (9 km) for hydrometeorological applications, balancing the resolution and accuracy of the active and passive instruments, respectively. The LIS software framework manages land surface model (LSM) simulations and includes an Ensemble Kalman Filter for conducting land surface DA. SPoRT has added a module to read, quality‐control and bias‐correct swaths of Level II SMOS soil moisture retrievals prior to assimilation within LIS. The impact of SMOS DA is being tested using the Noah LSM. Experiments are being conducted to examine the impacts of SMOS soil moisture DA on the resulting LISNoah fields and subsequent NWP simulations using the Weather Research and Forecasting (WRF) model initialized with LIS‐Noah output. LIS‐Noah soil moisture will be validated against in situ observations from Texas A&M's North American Soil Moisture Database to reveal the impact and possible improvement in soil moisture trends through DA. WRF model NWP case studies will test the impacts of DA on the simulated near‐surface and boundary‐layer environments, and precipitation during both quiescent and disturbed weather scenarios. Emphasis will be placed on cases with large analysis increments, especially due to contributions from regional irrigation patterns that are not represented by precipitation input in the baseline LIS‐Noah run. This poster presentation will describe the soil moisture DA methodology and highlight LIS‐Noah and WRF simulation results with and without assimilation.

Case, Jonathan L.↗

DOE OSTI2021

Physics-Guided Machine Learning for Prediction of Cloud Properties in Satellite-Derived Solar Data: Preprint

With over 20 years of high-resolution surface irradiance data covering most of the western hemisphere, the National Solar Radiation Database (NSRDB) is a vital public data asset. The NSRDB uses a two-step Physical Solar Model (PSM) that explicitly considers the effects of clouds and other atmospheric variables on radiative transfer. High-quality physical and optical cloud properties derived from satellite imagery are perhaps the most important data input to the PSM, representing the greatest source of radiation attenuation and scattering. However, traditional methods for cloud property retrieval have their own limitations and are unable to accurately predict cloud properties outside of nominal conditions. We introduce a physics-guided neural network that can accurately predict cloud properties when traditional methods fail or are inaccurate. Using this framework, we show reductions in relative Root Mean Square Error (RMSE) for Global Horizontal Irradiance (GHI) up to 13 percentage points for timesteps that previously had missing or low-quality cloud property data. We expect that this methodology will be effective in improving the quality of cloud property and solar irradiance data in the NSRDB.

41 EE - Solar Energy Technologies Office (EE-4S)↗

DOE OSTI2021

DEEPEN Data Catalog for Magmatic Geothermal Systems in the United States

This data catalog contains information related to the Training Site Analysis for the Geothermica project "DE-risking Exploration of geothermal Plays in magmatic ENvironments (DEEPEN)." The DEEPEN project aims to reduce exploration risk for geothermal fluids in magmatic systems by developing improved an improved framework for interpretation of exploration data using the Play Fairway Analysis (PFA) methodology. The Training Site Analysis performed for DEEPEN leverages existing datasets to develop a customized PFA approach to exploration for multiple geothermal resource types in magmatic systems (conventional hydrothermal resources, supercritical fluid and superheated steam resources, and superhot EGS resources). This data catalog contains links to publicly available data files related to 8 training sites in the United States. US training sites are: the Cascades/Aleutians PFA project; the Hawaii PFA project, the Oregon Cascades PFA project, the Snake River Plain, Idaho PFA project, the Washington State PFA project, Newberry Volcano, Coso Geothermal Field, and the Geysers Geothermal field. This database contains an overview of these training sites, data sources, and links to publicly available exploration datasets. For the five PFA projects, details on exploration data related to PFA components (heat, fluid, permeability, sometimes seal) are provided, including a summary of data weighting methodologies.

15 GEOTHERMAL ENERGY↗

NASA NTRS2022

Sustainable Aviation Operations and the Role of Information Technology and Data Science: Background, Current Status and Future Directions

This paper reviews the achievements of the international community towards environmentally friendly aviation operations, also referred to as Sustainable Aviation Operations in the last 25 years and the aspirations and goals to limit the impact of aviation and climate in the future. The framework for achieving global progress is provided by the International Civil Aviation Organization. NASA and FAA supported research and development to advance ATM concepts, and implemented the technology, concepts, and procedures that were responsible for creating fuel efficient flights. Historically aviation operations have been analyzed using physics-based models and provide information for making operational decisions. Future developments in aviation operations require new concepts, procedure, modeling, and analysis techniques. There is an increasing interest in applying methods based on Machine Learning Techniques to problems in Air Traffic Management. Aviation operations involving many decision makers, multiple objectives, poor or unavailable physics-based models and the availability of a rich historical database provide opportunities to exploit the richness of data-driven methods. The promises and challenges in applying Machine Learning Techniques to Air Traffic Management are discussed in the paper along with the testing and trustworthiness required for adoption of the techniques in operations.

Sustainable Aviation, Data Science, Machine Learni↗

NASA NTRS2022

Sustainable Aviation Operations and the Role of Information Technology and Data Science: Background, Current Status and Future Directions

Sustainable Aviation, Data Science, Machine Learni↗

DOE OSTI2023

Linearized Pair-Density Functional Theory for Vertical Excitation Energies

Multiconfiguration pair-density functional theory (MC-PDFT) is a computationally efficient method that computes the energies of electronic states in a state specific or state average framework via an on-top functional. However, MC-PDFT does not include state interaction among these states since the final energies do not come from the diagonalization of an effective model-space Hamiltonian. Recently, multistate extensions such as linearized PDFT (L-PDFT) have been developed to accurately model the potentials near conical intersections and avoided crossings. However, there has not been any systematic study evaluating their performance for predicting vertical excitations at the equilibrium geometry of a molecule, when the excited states are generally well separated. In this paper, we report the performance of L-PDFT on the extensive QUESTDB data set of vertical excitations using a database of automatically selected active spaces. We show that L-PDFT performs well on all these excitations and successfully reproduces the performance of MC-PDFT. These results further demonstrate the potential of L-PDFT, as its scaling is constant with the number of states included in the state-average manifold, whereas MC-PDFT scales linearly in this regard.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2023

pyGWBSE: a high throughput workflow package for GW-BSE calculations

Abstract We develop an open-source python workflow package, py GWBSE to perform automated first-principles calculations within the GW-BSE (Bethe-Salpeter) framework. GW-BSE is a many body perturbation theory based approach to explore the quasiparticle (QP) and excitonic properties of materials. GW approximation accurately predicts bandgaps of materials by overcoming the bandgap underestimation issue of the more widely used density functional theory (DFT). BSE formalism produces absorption spectra directly comparable with experimental observations. py GWBSE package achieves complete automation of the entire multi-step GW-BSE computation, including the convergence tests of several parameters that are crucial for the accuracy of these calculations. py GWBSE is integrated with Wannier90 , to generate QP bandstructures, interpolated using the maximally-localized wannier functions. py GWBSE also enables the automated creation of databases of metadata and data, including QP and excitonic properties, which can be extremely useful for future material discovery studies in the field of ultra-wide bandgap semiconductors, electronics, photovoltaics, and photocatalysis.

36 MATERIALS SCIENCE↗

DOE OSTI2023

BuildingsBench: A Benchmark for Universal Building Load Forecasting [SWR-23-51]

The residential and commercial building stock in the United States is responsible for a significant percentage of energy consumption and greenhouse gas emissions. Electrification of end-uses, as well as decarbonizing the electrical grid through renewable energy sources such as solar and wind, constitutes the pathway to zero-emission buildings. Forecasting day-ahead building energy consumption is an integral part of this solution. Currently, specialized forecasting models are hand-made for each individual building, which is time-consuming, expensive, and leads to duplicated efforts. BuildingsBench is a Python software framework for training and comparing generalized machine learning models for universal building load forecasting. This challenge tasks a single foundational model to generalize its forecasts for a wide variety of buildings, across geographic regions, building types, weather patterns, and more. This software provide code for pre-training such models and subsequently evaluating their performance on a suite of hundreds of diverse real and synthetic buildings. BuildingsBench is a platform for: - Large-scale pretraining with the synthetic Buildings-900K dataset for short-term load forecasting (STLF). Buildings-900K is statistically representative of the entire U.S. building stock and is extracted from the NREL End-Use Load Profiles database. - Benchmarking on two tasks evaluating generalization: zero-shot STLF and transfer learning for STLF. We provide an index-based PyTorch Dataset for large-scale pretraining, easy data loading for multiple real building energy consumption datasets as PyTorch Tensors or Pandas DataFrames, simple (persistence) to advanced (transformer) baselines, metrics management, and more.

Emami, Patrick↗

DOE OSTI2023

Process-based cost estimation framework for assessing economic viability of environmentally and socially sustainable rare earth element feedstocks

Rare Earth Elements (REEs) are critical raw material inputs that are required to enable the energy transition. Constraints on the U.S. supply of conventional REE feedstocks and the desire to diversify U.S. reliance on foreign supply chains have motivated the pursuit of REE recovery from unconventional feedstocks including waste products. Recovery of REEs from unconventional sources is also motivated by environmental and social considerations, including the potential to offset new mining activity and subsidize the remediation of REE-containing wastes from coal, natural gas, and other mineral extraction activities. Sourcing REEs from unconventional feedstocks would reduce supply chain constraints and minimize the environmental impacts of REE extraction, but current processes to exploit these feedstocks have uncertain technical and economic performance and the processes have not been demonstrated at scale. This work aims to reduce uncertainty in evaluating the performance of new REE production pathways. A database of operating and capital expenses for REE recovery and establish consistent process unit costs for common stages in the conventional supply chain are developed. Market prices are used to develop well-defined methods for determining product values for a diverse set of products that can be generated (e.g. oxalates, alloys, end-use products such as magnets) but are not typically considered in technoeconomic analyses. Finally, to provide insight on the effects of economies of scale and feedstock purity on capital and operating costs, process unit level cost functions relating total costs per kilogram of REE produced are reported. These tools can be applied to assess and facilitate scale-up of REE production from diverse sources of REEs.

Fritz, Alison↗

DOE OSTI2024

Jackson, L., Johnson, M.B., Latrach, A., Grimes, D., Martinez, C., and Mclaughlin, J.F., 2024, Multidisciplinary geotechnical data collection, curation, and analysis for conformity with the regulatory framework for geologic carbon storage in Wyoming, USA: Geological Society of America Abstracts with Programs. Vol. 56, No. 5, 2024, doi: 10.1130/abs/2024AM-405024

Title: Multidisciplinary Geotechnical Data Collection, Curation, and Analysis for Conformity with the Regulatory Framework for Geologic Carbon Storage in Wyoming, USA. Text: Construction and operation of wells for geologic sequestration of carbon dioxide necessitate that they are permitted under the Environmental Protection Agency’s Underground Injection Control Class VI requirements. Class VI wells conform to stringent requirements to ensure long-term safety and integrity of the storage site and the protection of Underground Sources of Drinking Water. Entities pursuing Class VI permitting must provide comprehensive geologic site characterization, including regional geologic structure and stratigraphy, aquifer information, reservoir and confining unit geomechanical properties, geochemical analyses, assessment of trapping capacity and mechanisms, and a variety of other of multidisciplinary geotechnical data. The Wyoming Class VI Site Characterization Database Project is focused on developing a geologic site characterization database of geotechnical information, which has been compiled and verified from established, public databases/entities and scientific literature to expedite Class VI permitting in Sweetwater County within the Greater Green River Basin of southern Wyoming. The preliminary suite of compiled data from 14,000 wells includes 8,000 wells with logs and 7,250 wells with formation tops, ~70 wells with core data (e.g., X-Ray diffraction, petrographic, and petrophysical data), ~2,500 water analyses, ~740 seismic events data, and ~520 bottom-hole temperature measurements. Future work on—and stemming from—this project will include new core analyses, calculation and interpolation of subsurface temperature gradients, mechanical earth models, geochemical simulations, storage capacity estimation, stratigraphic column generation and correlation, and construction of subsurface maps. Finally, this work will help to inspire and facilitate subsurface data compilation and curation beyond Sweetwater County, Wyoming.

42 ENGINEERING↗

NASA NTRS2024

Rhode Island Ecological Conservation: Methods for Monitoring Rhode Island Habitats: Contributing to a Framework for Targeted Conservation and Management

Global avian population decline since the 1970s is largely attributable to habitat loss and degradation from anthropogenic disturbances. NASA DEVELOP’s Rhode Island Ecological Conservation team partnered with the Audubon Society of Rhode Island to compute land use land cover (LULC) maps of Rhode Island to aid in the conservation of the state’s 140 bird species. This project aimed to support the partner’s land acquisition strategies with updated and specific LULC classifications showing potential bird-habitat locations across the state. We incorporated remotely sensed data from Landsat 8 and 9 Operational Land Imager (OLI) into LULC maps using unsupervised classification techniques in ArcGIS Pro and supervised classification in Google Earth Engine. We generated six land classifications for 2023, which showed land cover dominated by upland habitats (forests, scrub/shrub, and grasslands), followed by development. We used TerrSet’s Land Change Modeler to forecast LULC change through 2043, using 2011 and 2021 National Land Cover Database (NLCD) land cover maps derived from Landsat 8 and 9 imagery. Project results suggest that non-urban upland and wetland habitats will decrease over time, while development will continue to encroach on non-urban avian habitats. Our maps and associated data will allow for more efficient land acquisition and management efforts to support avian habitat conservation across Rhode Island. Our study shows that data acquisition and processing from open data sources is feasible and further analysis can be done through GIS classification tools. More analysis is needed beyond this study to obtain more detailed land cover maps, though Audubon can aid its targeted conservation efforts with our current, historic, and forecasted LULC maps.

Remote sensing↗

NASA NTRS1986

Transportability, distributability and rehosting experience with a kernel operating system interface set

For the past two years, PRC has been transporting and installing a software engineering environment framework, the Automated Product control Environment (APCE), at a number of PRC and government sites on a variety of different hardware. The APCE was designed using a layered architecture which is based on a standardized set of interfaces to host system services. This interface set called the APCE Interface Set (AIS), was designed to support many of the same goals as the Common Ada Programming Support Environment (APSE) Interface Set (CAIS). The APCE was developed to provide support for the full software lifecycle. Specific requirements of the APCE design included: automation of labor intensive administrative and logistical tasks: freedom for project team members to use existing tools: maximum transportability for APCE programs, interoperability of APCE database data, and distributability of both processes and data: and maximum performance on a wide variety of operating systems. A brief description is given of the APCE and AIS, a comparison of the AIS and CAIS both in terms of functionality and of philosophy and approach and a presentation of PRC's experience in rehosting AIS and transporting APCE programs and project data. Conclusions are drawn from this experience with respect to both the CAIS efforts and Space Station plans.

Blumberg, F. C.↗

DOE OSTI2021

Quinoa Phenotyping Methodologies: An International Consensus

Quinoa is a crop originating in the Andes but grown more widely and with the genetic potential for significant further expansion. Due to the phenotypic plasticity of quinoa, varieties need to be assessed across years and multiple locations. To improve comparability among field trials across the globe and to facilitate collaborations, components of the trials need to be kept consistent, including the type and methods of data collected. Here, an internationally open-access framework for phenotyping a wide range of quinoa features is proposed to facilitate the systematic agronomic, physiological and genetic characterization of quinoa for crop adaptation and improvement. Mature plant phenotyping is a central aspect of this paper, including detailed descriptions and the provision of phenotyping cards to facilitate consistency in data collection. High-throughput methods for multi-temporal phenotyping based on remote sensing technologies are described. Tools for higher-throughput post-harvest phenotyping of seeds are presented. A guideline for approaching quinoa field trials including the collection of environmental data and designing layouts with statistical robustness is suggested. To move towards developing resources for quinoa in line with major cereal crops, a database was created. The Quinoa Germinate Platform will serve as a central repository of data for quinoa researchers globally.

59 BASIC BIOLOGICAL SCIENCES↗