Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “resource potential”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Markets, Resources, and Environmental and Energy Justice (MarkeRs-EEJ)

This project will support the development of CO2U technologies and a CO2U industry by assessing the resource and market potential and infrastructure requirements for midterm (-2030) and long-term (-2050) deployment of CO2U technologies, as well as evaluating implications on energy justice factors such as air quality, gross domestic product (GDP), and jobs. Addressing both of these topics enables the project to provide context that can be used across a large portfolio of CO2U technologies so stakeholders can make decisions informed by both economic and societal factors.

BIOMASS FUELS↗
DOE OSTI2024

Investigation of Scale Deposition and Wellbore Corrosion in Carbonated Brine Injection: A Simulation Study

Carbonated brine injection (CBI), which involves dissolving CO2 in produced water to be injected into deep geologic reservoirs, is a method to manage produced brine, carbon dioxide (CO2) sequestration, and reduce anthropogenic greenhouse gas emissions. This low-risk CO2 storage strategy can leverage the existing saltwater disposal infrastructure and well resources. However, the potential for corrosion and scale deposition are a major challenge for carbonated brine which may result in subsidence, wellbore failure, casing failure, and cement failure. The main objective of this research was to evaluate the wellbore materials compatibility with carbonated brine water during the injection process, for repurposing Class II saltwater disposal wells for CBI.

Belarbi, Zineb↗
DOE OSTI2023

Insight into the Radiolytic Degradation Mechanism of TODGA

Partitioning and transmutation schemes, where americium is separated from other components of used nuclear fuel and burned in a fast neutron reactor, offer a path to dramatically decrease the space requirements for storage of nuclear fuel cycle wastes, allowing more efficient usage of potentially scarce storage resources.[1] Improvements in the efficiency of processes for separating americium from used nuclear fuels can thus have a significant impact on the costs of future nuclear fuel cycles. Understanding separation ligand radiation chemistry is important for development of new ligands and processes, as the ligand degradation products can have a deleterious effect on a separation process. Tetraoctyl diglycolamide (TODGA) has been studied as a ligand for lanthanide/minor actinide extraction for partitioning and transmutation schemes. However, the mechanism of initial energy transfer from products of solvent radiolysis to TODGA is still under debate. One proposed mechanism is attack by a n-dodecane radical cation resulting in electron abstraction and formation of a TODGA radical cation,[2] while later work has proposed hydrogen abstraction from sites adjacent to the central ether oxygen by other radical species.[3] The latter mechanism is proposed to result in degradation products from rupture of the ether bond. However, radiolytic degradation of TODGA in n-dodecane shows products that would result from rupture of all the backbone bonds. There is little information on what degradation products would be expected to form from the TODGA radical cation. In this work, we have begun to investigate this by examining the electron impact (EI) ionization mass spectrum of TODGA, which initially produces a TODGA radical cation in the gas phase. The EI spectrum shows fragments that would result from most backbone bond cleavages, similar to what is observed in the radiolysis of TODGA in n-dodecane. As the lifetime of the solvent radical cation decreases from n-dodecane to n-hexane,[4] irradiation of TODGA in n-hexane should have less solvent radical cations available for reaction with TODGA, and instead should favor reactions with longer-lived radicals. In contrast to n-dodecane, radiolytic degradation of TODGA in n-hexane produces only N,N-dioctylacetamide, which would result from rupture of the ether bond. The n-dodecane radical cation lifetime is long enough to produce TODGA radical cations but the n-hexane radical cation is mostly consumed before reaction with TODGA, so other radical processes that abstract hydrogen from TODGA dominate. This suggests that electron abstraction by the n-dodecane radical cation is an important component of TODGA radiolysis, although it does not rule out the presence of mechanisms involving hydrogen abstraction adjacent to the ether oxygen. References [1] J. Serp et al., Energies 2017, 10 (9), DOI: 10.3390/en10091445. [2] C. Zarzana et al., Solvent Extr. Ion Exch. 2015, 33 (5), 431–447, DOI: 10.1080/07366299.2015.1012885. [3] T. Koubský et al., Prog. Nucl. Energy 2017, 94, 208–215, DOI: 10.1016/j.pnucene.2016.07.010. [4] F. Sviridenko et al., Chem. Phys. Lett. 1998, 297 (3), 343–349, DOI: 10.1016/S0009-2614(98)01099-9.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗
DOE OSTI2024

Sustainable Aviation Fuel: Reducing Emissions from Commercial Flight

Demand for jet fuel is expected to balloon over the next three decades, which adds to the aviation industry’s challenge of meeting goals for cutting aviation greenhouse gas (GHG) emissions in half by 2050.1 Sustainable aviation fuels (SAFs) made from renewable biomass and waste resources have the potential to deliver the performance of petroleum-based jet fuel but with a fraction of its carbon footprint, giving airlines solid footing for reducing GHG emissions from flight.

09 BIOMASS FUELS↗
DOE OSTI2024

Community structure and function during periods of high performance and system upset in a full-scale mixed microalgal wastewater resource recovery facility

Microalgae have the potential to exceed current nutrient recovery limits from wastewater, enabling water resource recovery facilities (WRRFs) to achieve increasingly stringent effluent permits. The use of photobioreactors (PBRs) and the separation of hydraulic retention and solids residence time (HRT/SRT) further enables increased biomass in a reduced physical footprint while allowing operational parameters (e.g., SRT) to select for desired functional communities. However, as algal technology transitions to full-scale, there is a need to understand the effect of operational and environmental parameters on complex microbial dynamics among mixotrophic microalgae, bacterial groups, and pests (i.e., grazers and pathogens) and to implement robust process controls for stable long-term performance. Here, we examine a full-scale, intensive WRRF utilizing mixed microalgae for tertiary treatment in the US (EcoRecover, Clearas Water Recovery Inc.) during a nine-month monitoring campaign. We investigated the temporal variations in microbial community structure (18S and 16S rRNA genes), which revealed that stable system performance of the EcoRecover system was marked by a low-diversity microalgal community (D INVSIMPSON = 2.01) dominated by Scenedesmus sp. (MRA = 55 %-80 %) that achieved strict nutrient removal (effluent TP < 0.04 mg·L -1 ) and steady biomass concentration (TSS monthly avg . = 400–700 mg·L −1 ). Operational variables including pH, alkalinity, and influent ammonium (NH 4 + ), correlated positively (p < 0.05, method = Spearman) with algal community during stable performance. Further, the use of these parameters as operational controls along with N/P loading and SRT allowed for system recovery following upset events. Importantly, the presence or absence of bacterial nitrification did not directly impact algal system performance and overall nutrient recovery, but partial nitrification (potentially resulting from NO 2 − accumulation) inhibited algal growth and should be considered during long-term operation. The microalgal communities were also adversely affected by zooplankton grazers (ciliates, rotifers) and fungal parasites (Aphelidium), particularly during periods of upset when algal cultures were experiencing culture turnover or stress conditions (e.g., nitrogen limitation, elevated temperature). Altogether, the active management of system operation in order to maintain healthy algal cultures and high biomass productivity can result in significant periods (>4 months) of stable system performance that achieve robust nutrient recovery, even in winter months in northern latitudes (WI, USA).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Time to Use Dendrohydrological Data in Water Resources Management?

Despite a wide consensus on the potential importance of streamflow reconstructions to water resources planning and management activities, applications of dendrohydrology have been piecemeal. This review article describes barriers to widespread use of dendrohydrological data and suggests research needs for operationalizing streamflow records reconstructed from these proxy data.

42 ENGINEERING↗
DOE OSTI2024

New Resource Adequacy Criteria for the Energy Transition: Modernizing Reliability Requirements

The transition toward a cleaner and more weather-dependent power system brings with it unprecedented challenges and opportunities for maintaining resource adequacy—namely, a future that includes rapid load growth, plant retirements, and a shift toward variable and energy-limited resources. Resource adequacy analyses, capacity accreditation, and the resource adequacy planning criteria are becoming increasingly important for power system planning and investment decisions. While previous ESIG reports evaluated the changing resource adequacy risks and planning practices created by the energy transition, none specifically examined potential changes to the resource adequacy criterion used for planning. The resource adequacy criterion sets the level of supply- and demand-side resources that are required for a given power system to meet reliability objectives. It is a pivotal standard that influences billions of dollars of investment decisions.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS1991

Survey of resource opportunities and critical evaluation of economic requirements

A series of mission analyses were performed to evaluate human mission to Mars and the moon with and without the aid of planetary resource utilization. The types of trade studies that are considered include the use of resources to manufacture propellant, food, habitat atmospheric gases, and lander habitat structure. Also, the potential for export of resources from the moon, Mars, Phobos, Deimos, and selected asteroids is also examined. In all cases, mass leveraging is evaluated. For certain cases, economic factors are evaluated as well. It is concluded that some uses are highly leveraging on the mission, whereas others have lesser impact and, therefore, should be afforded lesser priority in resource utilization studies. This survey is made with a consistent set of scaling laws for spacecraft propulsion and habitation systems and subsystems, and therefore, provides a rational basis for comparing different resource locations and use strategies.

Clark, Benton C.↗
NASA NTRS1985

Enabling technologies for transition to utilization of space-based resources and operations

This article explores a potential scenario for the further development of space infrastructure resources and operations management. It is a scenario that transitions from the current ground-based system to an architecture that is predominantly space-based by exploiting key mission systems in an operational support role. If this view is accurate, an examination of the range of potential infrastructure elements and how they might interact in a maximally productive space-based operations complex is needed, innovative technologies beyond the current Shuttle and Space Station legacy need to be identified, and research programs pursued. Development of technologies within the areas of telerobotics, machine autonomy, human autonomy, in-space manufacturing and construction, propulsion and energy is discussed.

Sadin, S. R.↗
DOE OSTI2024

Geothermal Uncertainty Representation in reV: the Renewable Energy Potential Model

We present a preliminary methodology for including geothermal resource uncertainty into the Renewable Energy Potential model, which estimates potential capacity and costs on a gridded surface at the national scale. The uncertainty outputs characterize the 10th, 50th and 90th percentile for geothermal resources using two energy capacity estimation equations. We then present a method and results that demonstrate how other geologic data layers, which may be indicative of permeability, can be used to inform the mean and standard deviation of the geothermal capacity. We demonstrate how the mean and standard deviation can be defined or partially informed by using collocated regression estimates and estimate errors, respectively . These regression results are from 36 observed geothermal power plants in the Great Basin region and are also used to benchmark the P10-P90 calculations.

exclusions↗
DOE OSTI2024

GeoThermalCloud: Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources

The primary goals of this project are exploring hidden geothermal resources in the U.S.A. and designing profitable enhanced geothermal systems (EGS). Many processes and parameters control geothermal exploration and energy production from geothermal fields. Diverse datasets (e.g., geology, geochemistry, geophysics, satellite, airborne geophysics) are available to help characterize subsurface geothermal conditions. Sparse and multi-scale characteristics of these datasets prohibit properly leveraging these datasets for geothermal exploration and profitable EGS design. Recent advancements in machine learning (ML) promise to resolve these issues. The tremendous challenges and risks of geothermal exploration and production bring the demand for novel ML methods and tools that can (1) analyze large field datasets, (2) assimilate model simulations (large inputs and outputs), (3) process sparse datasets, (4) perform transfer learning (between sites with different exploratory levels), (5) extract hidden geothermal signatures in the field and simulation data, (6) label geothermal resources and processes, (7) identify high-value data acquisition targets, and (8) guide geothermal exploration and production by selecting optimal exploration, production, and drilling strategies. To address these necessities, ML-based geothermal resources exploration and enhanced geothermal systems (EGS) design tools have been developed. The exploration tool is called GeoThermalCloud and EGS design tool is called GeoDT-ML. GeoThermalCloud (https://github.com/SmartTensors/GeoThermalCloud.jl) utilizes a LANL unsupervised ML platform called SmartTensors (https://tensors.lanl.gov/) to automate data analyses and interpretations by extracting hidden signatures to identify geothermal prospects. Also, it enables the identification of critical measurements needed to identify geothermal resource signatures. Alternatively, GeoDT-ML (https://github.com/SmartTensors/GeoThermalCloud.jl/tree/master/EGS) is an ML-based alternative to GeoDT (https://github.com/GeoDesignTool/GeoDT.git), a fast, simplified multi-physics solver to evaluate EGS project designs in uncertain geologic systems. GeoDT-ML leverages recent advances in deep learning and high-performance computing. It is a faster and simpler version of GeoDT. To make this project a success, we used capabilities of LANL, PNNL, Google, Stanford, and Julia Computing. We analyzed eight datasets of the U.S.A. using GeothermalCloud and demonstrated potential highly prospective geothermal resources and identified key factors defining highly prospective sites. The first data set includes 44 locations in southwest New Mexico and 18 geological, hydrogeological, geophysical, geothermal, geochemical attributes. We defined low- and medium-temperature hydrothermal systems and discovered a new highly prospective site. The second data set analyzed 18 shallow water chemistry attributes at 14,342 locations in the Great Basin. It demarcated modestly, moderately, and highly prospective sites including key attributes for each type of prospectivity. The third data set analyzed Utah FORGE data including satellite (InSAR), geophysical (gravity, seismic), geochemical, and geothermal attributes. Here, we performed prospectivity analysis to identify future drilling locations using geological, geochemical, and geophysical attributes. Maps of temperature at depth and heat flow are constructed based on the available data. Prospectivity maps were generated, and drilling locations were proposed for future geothermal field exploration. The fourth data set analyzed 21 attributes at 120 locations in Tularosa Basin, New Mexico; data comes from past play fairway analyses in this region. ML analyses identified geothermal signatures associated with modestly, moderately, and highly hydrothermal systems. We also defined dominant attributes and spatial distribution of the geothermal signatures. The fifth, sixth, seventh, and eighth datasets include Tohatchi Springs, New Mexico, Hawaii, Brady site, Nevada, and EGS Collab, respectively. Moreover, we coupled GeothermalCloud and magnetotellurics data to pinpoint drilling locations for developing geothermal projects in the Tularosa Basin, New Mexico. GeothermalCloud found potential prospective locations for geothermal resources near White Sands Missile Range and McGregor Range at Fort Bliss. Magnetotellurics data determined the potential depth (~1800m) of geothermal prospects at McGregor Range based on apparent resistivity structures/layers in the subsurface. The McGregor Range consists of three resistivity layers and two resistivity structures. Magnetotellurics data also helps identify that the western portion of the McGregor Range has thick and low-resistivity earth materials. The low resistivity to the west is most likely for a fault system. Assuming temperature is consistent with a geothermal reservoir, the west-central part of the McGregor Range has the highest geothermal potential because of the increase in porosity and associated permeability attributed to the interpreted fault system. Also, we devised a coupling strategy between a process model and GeothermalCloud to characterize hydrogeological conditions and geothermal conditions, respectively. The process model characterizes hydrogeological and geothermal conditions on highly prospective geothermal sites provided by GeothermalCloud. We developed a physics-informed neural network (PINN) version of the Burns equation that can be easily coupled with GeothermalCloud. Furthermore, we performed an optimal design decision maximizing the economic value of an EGS power plant. This study optimized the range of well spacing between injection and production wells maximizing net present value in dollars (NPV). For this task, we used the GeoDT to simulate the Utah FORGE EGS development cycle from the initial well design to the end of production. Next, we accomplished another crucial task, which is predicting permeability of geothermal reservoirs. Predicting permeability of geothermal reservoirs is a non-trivial task because of huge computational runtime of simulation and lack of measurements. To avoid these limitations, we used easy-to-measure chemical concentrations in the subsurface as measurement data and convolutional neural network based ML model of a high-fidelity model. Next, we predicted permeability using Markov chain Monte Carlo simulation. We found that Markov chain Monte Carlo simulation predicts permeability with a high certainty if the prediction zone in the simulation area has chemical concentration data. Finally, we analyzed the DOE funded INGENIOUS and GeoDAWN projects data. For discovering hidden geothermal systems in the Great Basin, the INGENIOUS project accumulated old data, collected new data, and released them in 2022. The dataset includes a total of 24 geological, geophysical, and geochemical attributes. Data resolution and scale significantly vary prohibiting an appropriate usage. To avoid such limitations, we brought all data in the same resolution and scale by applying the inverse distance weighting interpolation technique for predicting data in unsampled locations. Subsequently, we analyzed LiDAR data of the GeoDAWN project. We received data in tiles format. The DOE’s overarching goal is to use ML on LiDAR data for finding favorable geological structures (e.g., step up faults in Brady, Nevada). To serve the purpose, we need to label favorable geologic structures that correspond to LiDAR data. We wrote an algorithm to label the LiDAR data with the favorable geologic structures.

15 GEOTHERMAL ENERGY↗
DOE OSTI2023

Enabling Scalable VQE Simulation on Leading HPC Systems

Large-scale simulations of quantum circuits pose significant challenges, especially in the context of quantum chemistry, due to the number of qubits, circuit depth, and the number of circuits needed per problem. High-performance computing (HPC) systems offer massive computational capabilities that could help overcome these obstacles. We developed a high-performance quantum circuit simulator, called NWQ-Sim, and demonstrate its capability to simulate large quantum chemistry problems on NERSC's Perlmutter supercomputer. Integrating NWQ-Sim with XACC, we have executed QPE and VQE algorithms for downfolded quantum chemistry systems at unprecedented scales. Our work demonstrates the potential of leveraging HPC resources to advance quantum chemistry and other applications of near-term quantum devices.

Wang, Meng↗
NASA NTRSDate not supplied

In-Situ Reaction Monitoring for Lunar Applications Utilizing a Single Quadrupole Residual Gas Analyzer

With a revived focus to create a permeant presence on the moon and in preparation for future Mars exploration, it is imperative that all re-sources are utilized to their fullest potential. In-Situ Resource Utilization (ISRU) will be critical for future mission success as it would enable independent operation of missions while minimally relying on the complex supply chain created between the Earth, moon and Mars. One of the most critical resources that have been identified for ISRU is the creation of liquid oxygen (O2) for not only breathable air, but also for rocket fuel. On the moon and potentially Mars, one of the most O2 rich resources is in the presence of regolith. Although there are numerous minerals within regolith and various processes to extract O2, this is not the scope of this paper. Since the O2 extraction method can vary, a critical standardized analytical method is needed to verify O2 (or oxygenated precursor compounds) for extraction efficiency and purity. Our group utilizes a modified consumer off-the-shelf (COTS), Residual Gas Analyzer (RGA) Single Quadrupole Mass Spectrometer (QMS). Due to the current field mission requirements, the gas analysis module will need to rely only on mass-to-charge ratios (m/z) and peak intensities to differentiate and quantitate targeted gas-phase reactants or products, i.e. carbon monoxide (CO), carbon dioxide (CO2), methane (CH4) and hydrogen (H2). However, due to solar wind implanted volatiles and various other compounds present in lunar regolith, such as nitrogen (N2), helium (He), and deuterium (D2), deconvolution and quantification of isobaric compounds becomes quite difficult. Specifically, the presence of N2 makes quantification of CO, an oxygenated precursor, difficult due to both compounds having a peak at m/z 28. Approaches to differentiate isobaric components: Initially, the solution to overcome isobaric overlap was to use isotopic ratios, particularly be-tween carbon-13 and nitrogen-15. Although limited literature discuss isotopic composition and attempt to quantify the ratios, variation is particularly high and limited to a few lunar regolith samples. The following were identified as potential approaches to quantify isobaric species, N2 and CO, using our RGA. Threshold Ionization. CO and N2 both have peaks at m/z 28, but only N2 is capable of producing a peak at m/z 7. While a N2 peak at m/z 7 is generally rare, instrument parameters can be optimized in order to maximize the peak intensity. Typically a standard ionization energy of 70 eV is used to monitor a batch process and would produce reliable results for a majority of the reactants or products that we are interested in quantifying. However, to analyze the rare molecular fragment of N2 (N2+ at m/z 7), a different ionization energy may be beneficial. Threshold Ionization Mass Spectrometry (TIMS) will be explored to optimize the N2+ peak at m/z 7 by varying parameters such as ionization energy and pressure. Therefore, a quantitative value at m/z 7 will be used to determine the product gas concentration of N2, which will be subtracted from the total concentration determined at m/z 28 enabling us to approximate the amount of CO. Secondary Detector (infrared gas detector, IR). Another approach to quantitate and differentiate be-tween isobaric components is to leverage alternative modes of detection. Unlike mass spectrometry, which generally uses a separation step prior to analysis (i.e. gas chromatography), IR produces energy that is specifically absorbed depending on the bond and configuration of the molecule. This technique could be leveraged since it is generally only active on non-symmetric molecules, i.e. CO. Utilizing concentrations of CO determined from another detector could help correct concentrations of CO obtained from the QMS. However, the inclusion of another detector adds additional mission requirements, such as power, software changes and avionics updates that add additional cost and schedule requirements.

mass spectrometry↗
NASA NTRS2009

The Integrated Medical Model - Optimizing In-flight Space Medical Systems to Reduce Crew Health Risk and Mission Impacts

The Integrated Medical Model (IMM) is a decision support tool used by medical system planners and designers as they prepare for exploration planning activities of the Constellation program (CxP). IMM provides an evidence-based approach to help optimize the allocation of in-flight medical resources for a specified level of risk within spacecraft operational constraints. Eighty medical conditions and associated resources are represented in IMM. Nine conditions are due to Space Adaptation Syndrome. The IMM helps answer fundamental medical mission planning questions such as What medical conditions can be expected? What type and quantity of medical resources are most likely to be used?", and "What is the probability of crew death or evacuation due to medical events?" For a specified mission and crew profile, the IMM effectively characterizes the sequence of events that could potentially occur should a medical condition happen. The mathematical relationships among mission and crew attributes, medical conditions and incidence data, in-flight medical resources, potential clinical and crew health end states are established to generate end state probabilities. A Monte Carlo computational method is used to determine the probable outcomes and requires up to 25,000 mission trials to reach convergence. For each mission trial, the pharmaceuticals and supplies required to diagnose and treat prevalent medical conditions are tracked and decremented. The uncertainty of patient response to treatment is bounded via a best-case, worst-case, untreated case algorithm. A Crew Health Index (CHI) metric, developed to account for functional impairment due to a medical condition, provides a quantified measure of risk and enables risk comparisons across mission scenarios. The use of historical in-flight medical data, terrestrial surrogate data as appropriate, and space medicine subject matter expertise has enabled the development of a probabilistic, stochastic decision support tool capable of optimizing in-flight medical systems based on crew and mission parameters. This presentation will illustrate how to apply quantitative risk assessment methods to optimize the mass and volume of space-based medical systems for a space flight mission given the level of crew health and mission risk.

Kerstman, Eric↗
DOE OSTI2022

Task 12 PV Sustainability - Mineral Resource Use Footprints of Residential PV Systems

Resource use intensity is often mentioned as one of the main characteristics of PV systems and PV electricity. Recently, the International Energy Agency published a report on the role of critical minerals in clean energy transitions. The Product Environmental Footprint pilot study on PV electricity quantified (among other environmental impacts) its abiotic depletion potential. So far, a comprehensive assessment of resource use impacts highlighting the different facets of its impacts is however lacking. For the first time, the resource use impacts of PV electricity are quantified simultaneously with four impact category indicators recommended or suggested by the Life Cycle Initiative hosted at UN Environment. The indicators cover distinctly different aspects of resource use, namely resource depletion with the Abiotic Depletion Potential, ultimate reserves (ADP UR ), economic resource scarcity with the Abiotic Depletion Potential, economic reserves (ADP ER ), resource quality with the Surplus Ore Potential, Ultimate Recoverable Resources (SOP URR ) and re-source criticality with the ESSENZ method. The resource use impacts caused from the generation of 1 kWh electricity with a residential scale photovoltaic (PV) system installed in Central Europe using mono- and multi-crystalline silicon panels and CdTe panels, respectively are quantified. The product system includes manufacture, use and end of life treatment (take back and recycling) of the PV panels, cabling, inverter and supporting structure, the supply chains of the raw materials and energy used in PV panel and inverter manufacture as well as transport logistics.

14 SOLAR ENERGY↗
DOE OSTI2025

Machine learning-driven predictive resource management in complex science workflows

Here, the collaborative efforts of large communities in science experiments, often comprising thousands of global members, reflect a monumental commitment to exploration and discovery. Recently, advanced and complex data processing has gained increasing importance in science experiments. Data processing workflows typically consist of multiple intricate steps, and the precise specification of resource requirements is crucial for each step to allocate optimal resources for effective processing. Estimating resource requirements in advance is challenging due to a wide range of analysis scenarios, varying skill levels among community members, and the continuously increasing spectrum of computing options. One practical approach to mitigate these challenges involves initially processing a subset of each step to measure precise resource utilization from actual processing profiles before completing the entire step. While this two-staged approach enables processing on optimal resources for most of the workflow, it has drawbacks such as initial inaccuracies leading to potential failures and suboptimal resource usage, along with overhead from waiting for initial processing completion, which is critical for fast-turnaround analyses. In this context, our study introduces a novel pipeline of machine learning models within a comprehensive workflow management system, the Production and Distributed Analysis (PanDA) system. These models employ advanced machine learning techniques to predict key resource requirements, overcoming challenges posed by limited upfront knowledge of characteristics at each step. Accurate forecasts of resource requirements enable informed and proactive decision-making in workflow management, enhancing the efficiency of handling diverse, complex workflows across heterogeneous resources.

97 MATHEMATICS AND COMPUTING↗