Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Knowledge Discovery”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 883 records · Page 49
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Blast from the Past: ASDC Curation for NASA Suborbital Legacy Missions to Promote Data Discovery and Accessibility

NASA has an extensive history of conducting suborbital field campaigns to further advances in atmospheric sciences. Beginning with the Chemical Instrument Test and Evaluation (CITE) conducted in 1983-1984, NASA has completed many suborbital campaigns over the past three decades. Since the early 2010s, suborbital missions are typically assigned to a NASA Distributed Active Archive Center (DAAC) prior to the mission for long-term archival and distribution. Efforts are being made by NASA’s Earth Science Data and Information System (ESDIS) Project and the Airborne Data Management Group (ADMG) to assign legacy missions to DAACs for permanent archival and distribution, so that these valuable datasets remain to be available to the scientific community. NASA’s Atmospheric Science Data Center (ASDC) has been named the assigned DAAC for nearly 20 atmospheric composition legacy missions, including missions conducted as part of the Global Tropospheric Experiment (GTE) and expects to be named the assigned DAAC for more of these missions over the next few years. The primary goal of the ASDC is to provide access to the datasets as they are currently formatted to the broad user community and enhance their findability and accessibility. However, data reporting standards have evolved significantly since 1983 and the datasets span a wide variety of file formats, including text, Ames, GTE, and ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), and the amount of metadata and relevant information included in the files also varies greatly and can not be readily extracted without subject matter knowledge. This has caused challenges for the ASDC’s suborbital metadata extraction pipeline in ensuring that accurate and necessary metadata is being provided for the missions by all the ASDC’s existing search mechanisms. To make the data more findable and accessible, the ASDC has begun researching ways to further enhance the datasets, including distributing value-added products (i.e. consistent file format such as ICARTT or netCDF), adding standard names from the ESDIS Standards Coordination Office (ESCO)-approved Atmospheric Composition Variable Standard Names Convention (ACVSNC), and creating outreach materials such as ArcGIS StoryMaps, User Guides, and Micro Articles, providing overviews of the missions and what type of data was collected during the missions. These efforts also help support NASA’s Open-Source Science by enhancing the FAIRness of the legacy data products. This presentation will review the ASDC’s ongoing efforts, progress made, and future plans for legacy missions.

Megan Buzanowicz↗
DOE OSTI2026

Oakland University Cybersecurity Center (Final Scientific/Technical Report)

This report summarizes the outcomes of Award DE-CR0000023, “Oakland University Cybersecurity Center,” a 31-month project funded by the U.S. Department of Energy Office of Cybersecurity, Energy Security, and Emergency Response (CESER). The project addressed cybersecurity risks facing small and medium-sized manufacturers (SMMs) transitioning to Industry 4.0. The project integrated customer discovery, applied research, and cybersecurity training development. A total of 51 cybersecurity assessments identified significant gaps in baseline practices, incident response, and workforce capability. Research efforts produced a scalable mitigation framework tailored to SMM environments, and workforce analysis identified persistent talent gaps. Eight cybersecurity training modules were developed and deployed via Oakland University’s Professional and Continuing Education (PACE) platform. All objectives were completed, with 98.93% federal budget utilization and cost share exceeding requirements. The project establishes a scalable model for strengthening cybersecurity resilience and workforce capacity across U.S. manufacturing supply chains.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1991

Short-period comets

The spacecraft flybys of Comet Halley in 1986 confirmed Whipple's icy conglomerate hypothesis for cometary nuclei and showed that comets are far richer in volatiles than any other class of solar system bodies. Water is the most abundant volatile, comprising roughly 80 percent of the gas flowing out from the nucleus. Carbon monoxide is next with a content of 15 percent relative to water, though with approximately half of that coming from an extended source in the cometary coma, i.e., hydrocarbon dust grains. The detection of large numbers of hydrocarbon CHON grains was one of the more significant discoveries of the Halley flybys, as was the ground-based observation that CN occurs in jets, again indicating an extended source. Evidence was also found for more complex hydrocarbons. Estimates of the total dust-to-gas ratio for Halley range as high as 2:1, indicating that a substantial fraction of the volatile material may be tied up in solid hydrocarbons rather than ices. The role of clathrates in trapping more volatile ices is not yet understood. If Halley can be taken to be representative of all short-period comets, then the short-period comets may provide a significant source of volatiles in near-earth space. This resource is more difficult to reach dynamically than the near-earth asteriods, but the high volatile content may justify the additional effort necessary. In addition, there is considerable evidence that at least some fraction of the near-earth asteriods are extinct cometary nuclei which have evolved into asteroid orbits, and which may contain significant volatiles buried beneath an insulating lag-deposit crust of nonvolatiles. Knowledge of comets will be greatly enhanced in the near future by the Comet Rendezvous Flyby mission now under development by NASA, and by the proposed Rosetta mission.

Weissman, Paul R.↗
NASA NTRS2012

NASA's Space Launch System: A Heavy-Lift Platform for Entirely New Missions

The National Aeronautics and Space Administration's (NASA's) Space Launch System (SLS) will contribute a new capability for human space flight and scientific missions beyond low-Earth orbit (LEO). The SLS Program, managed at NASA s Marshall Space Flight Center, will develop the heavy-lift vehicle that will launch the Orion Multi-Purpose Crew Vehicle (MPCV), equipment, supplies, and major science missions for exploration and discovery. Orion will carry crews to space, provide emergency abort capability, sustain the crew during space travel, and provide safe reentry from deep-space return velocities. Supporting Orion s first autonomous flight to lunar orbit and back in 2017 and its first crewed flight in 2021, the SLS ultimately offers a flexible platform for both human and scientific exploration. The SLS plan leverages legacy infrastructure and hardware in NASA s inventory, as well as continues with advanced technologies now in development, to deliver an initial 70 metric ton (t) lift capability in 2017, evolving to a 130-t capability, using a block upgrade approach. This paper will give an overview of the SLS design and management approach against a backdrop of the missions it will support. It will detail the plan to deliver the initial SLS capability to the launch pad in the near term, as well as summarize the innovative approaches the SLS team is applying to deliver a safe, affordable, and sustainable long-range capability for entirely new missions-opening a new realm of knowledge and a world of possibilities for multiple partners. Design reference missions that the SLS is being planned to support include Mars, Jupiter, Lagrange Points, and near-Earth asteroids (NEAs), among others. The Agency is developing its mission manifest in parallel with the development of a heavy-lift flagship that will dramatically increase total lift and volume capacity beyond current launch vehicle options, reduce trip times, and provide a robust platform for conducting new missions destined to rewrite textbooks with the information they deliver, while creating a framework for further collaboration among domestic and international partners, and potentially spurring economic expansion into new markets.

Creech, Stephen D.↗
DOE OSTI2015

DISCOVERY OF GAMMA-RAY PULSATIONS FROM THE TRANSITIONAL REDBACK PSR J1227-4853

The 1.69 ms spin period of PSR J1227−4853 was recently discovered in radio observations of the low-mass X-ray binary XSS J12270−4859 following the announcement of a possible transition to a rotation-powered millisecond pulsar state, inferred from decreases in optical, X-ray, and gamma-ray flux from the source. We report the detection of significant (5σ) gamma-ray pulsations after the transition, at the known spin period, using ∼1 year of data from the Large Area Telescope (LAT) on board the Fermi Gamma-ray Space Telescope. The gamma-ray light curve of PSR J1227−4853 can be fit by one broad peak, which occurs at nearly the same phase as the main peak in the 1.4 GHz radio profile. The partial alignment of light-curve peaks in different wavebands suggests that at least some of the radio emission may originate at high altitude in the pulsar magnetosphere, in extended regions co-located with the gamma-ray emission site. We folded the LAT data at the orbital period, both pre- and post-transition, but find no evidence for significant modulation of the gamma-ray flux. Analysis of the gamma-ray flux over the mission suggests an approximate transition time of 2012 November 30. Continued study of the pulsed emission and monitoring of PSR J1227−4853, and other known redback systems, for subsequent flux changes will increase our knowledge of the pulsar emission mechanism and transitioning systems.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

2024 NMDC Ambassador Training Materials [Slides]

The NMDC is a sustainable data discovery platform that promotes open science and shared-ownership across a broad and diverse community of researchers, funders, publishers, societies, and other collaborators. The NMDC aims to enable multi-omic microbiome research to accelerate scientific discovery. The NMDC is a Department of Energy funded program that is a collaboration between 3 National Laboratories: Lawrence Berkeley National Laboratory (LBNL), Los Alamos National Laboratory (LANL), and Pacific Northwest National Laboratory (PNNL).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2014

Measuring tides and binary parameters from gravitational wave data and eclipsing timings of detached white dwarf binaries

The discovery of the most compact detached white dwarf (WD) binary SDSS J065133.33+284423.3 has been discussed in terms of probing the tidal effects in WDs. This system is also a verification source for the space-based gravitational wave (GW) detector, eLISA, or the evolved Laser Interferometer Space Antenna, which will observe short-period compact Galactic binaries with P {sub orb} ≲ 5 hr. We address the prospects of performing tidal studies using eLISA binaries by showing the fractional uncertainties in the orbital decay rate, f-dot , and the rate of that decay, f{sup ¨} expected from both the GW and electromagnetic (EM) data for some of the high-f binaries. We find that f-dot and f{sup ¨} can be measured using GW data only for the most massive WD binaries observed at high frequencies. From timing the eclipses for ∼10 yr, we find that f-dot can be known to ∼0.1% for J0651. We find that from GW data alone, measuring the effects of tides in binaries is (almost) impossible. We also investigate the improvement in the knowledge of the binary parameters by combining the GW amplitude and inclination with EM data with and without f-dot . In our previous work, we found that EM data on distance constrained the 2σ uncertainty in chirp mass to 15%-25% whereas adding f-dot reduces it to 0.11%. EM data on f-dot also constrain the 2σ uncertainty in distance to 35%-19%. EM data on primary mass constrain the secondary mass m {sub 2} to factors of two to ∼40% whereas adding f-dot reduces this to 25%. Finally, using single-line spectroscopic data constrains 2σ uncertainties in both the m {sub 2}, d to factors of two to ∼40%. Adding EM data on f-dot reduces these 2σ uncertainties to ≤25% and 6%-19%, respectively. Thus we find that EM measurements of f-dot and radial velocity are valuable in constraining eLISA binary parameters.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

Genome-resolved metagenomics of eukaryotic populations during early colonization of premature infants and in hospital rooms

Background: Fungal infections are a significant cause of mortality and morbidity in hospitalized preterm infants, yet little is known about eukaryotic colonization of infants and of the neonatal intensive care unit as a possible source of colonizing strains. This is partly because microbiome studies often utilize bacterial 16S rRNA marker gene sequencing, a technique that is blind to eukaryotic organisms. Knowledge gaps exist regarding the phylogeny and microdiversity of eukaryotes that colonize hospitalized infants, as well as potential reservoirs of eukaryotes in the hospital room built environment. Results: Genome-resolved analysis of 1174 time-series fecal metagenomes from 161 premature infants revealed fungal colonization of 10 infants. Relative abundance levels reached as high as 97% and were significantly higher in the first weeks of life (p = 0.004). When fungal colonization occurred, multiple species were present more often than expected by random chance (p = 0.008). Twenty-four metagenomic samples were analyzed from hospital rooms of six different infants. Compared to floor and surface samples, hospital sinks hosted diverse and highly variable communities containing genomically novel species, including from Diptera (fly) and Rhabditida (worm) for which genomes were assembled. With the exception of Diptera and two other organisms, zygosity of the newly assembled diploid eukaryote genomes was low. Interestingly, Malassezia and Candida species were present in both room and infant gut samples. Conclusions: Increased levels of fungal co-colonization may reflect synergistic interactions or differences in infant susceptibility to fungal colonization. Discovery of eukaryotic organisms that have not been sequenced previously highlights the benefit of genome-resolved analyses, and low zygosity of assembled genomes could reflect inbreeding or strong selection imposed by room conditions.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2019

A Quantitative Study of Deep Learning Training on Heterogeneous Supercomputers

Deep learning (DL) has become a key technique for solving complex problems in scientific research and discovery. DL training for science is substantially challenging because it has to deal with massive quantities of multi-dimensional data. High- performance computing (HPC) supercomputers are increasingly being employed for meeting the exponentially growing demand for DL. Multiple GPUs and high-speed interconnect network are needed for supporting DL on HPC systems. However, the excessive use of GPUs without considering effective benefits leads to inefficient resource utilization of these expensive setups. In this paper, we conduct a quantitative analysis to gauge the efficacy of DL workloads on the latest HPC system and identify viability of next-generation DL-optimized heterogeneous supercomputers for enabling researchers to develop more efficient resource management and distributed DL middleware. We evaluate well- known DL models with large-scale datasets using the popular TensorFlow framework, and provide a thorough evaluation in- cluding scalability, accuracy, variability, storage resource, GPU- GPU/GPU-CPU data transfer, and GPU utilization. Our analysis reveals that the latest heterogeneous supercomputing cluster shows varying performance trend as compared to the existing literature for single- and multi-node training. To the best of our knowledge, this is the first work to conduct such a quantitative and comprehensive study of DL training on a supercomputing system with multiple GPUs.

Han, Jingoo↗
DOE OSTI2016

Conservation of the C-type lectin fold for accommodating massive sequence variation in archaeal diversity-generating retroelements

Diversity-generating retroelements (DGRs) provide organisms with a unique means for adaptation to a dynamic environment through massive protein sequence variation. The potential scope of this variation exceeds that of the vertebrate adaptive immune system. DGRs were known to exist only in viruses and bacteria until their recent discovery in archaea belonging to the ‘microbial dark matter’, specifically in organisms closely related to Nanoarchaeota. However, Nanoarchaeota DGR variable proteins were unassignable to known protein folds and apparently unrelated to characterized DGR variable proteins. To address the issue of how Nanoarchaeota DGR variable proteins accommodate massive sequence variation, we determined the 2.52 Å resolution limit crystal structure of one such protein, AvpA, which revealed a C-type lectin (CLec)-fold that organizes a putative ligand-binding site that is capable of accommodating 1013 sequences. This fold is surprisingly reminiscent of the CLec-folds of viral and bacterial DGR variable protein, but differs sufficiently to define a new CLec-fold subclass, which is consistent with early divergence between bacterial and archaeal DGRs. The structure also enabled identification of a group of AvpA-like proteins in multiple putative DGRs from uncultivated archaea. These variable proteins may aid Nanoarchaeota and these uncultivated archaea in symbiotic relationships. Our results have uncovered the widespread conservation of the CLec-fold in viruses, bacteria, and archaea for accommodating massive sequence variation. In addition, to our knowledge, this is the first report of an archaeal CLec-fold protein.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Deep-branching acetogens in serpentinized subsurface fluids of Oman

Little is known of acetogens in contemporary serpentinizing systems, despite widely supported theories that serpentinite-hosted environments supported the first life on Earth via acetogenesis. To address this knowledge gap, genome-resolved metagenomics was applied to subsurface fracture water communities from an area of active serpentinization in the Samail Ophiolite, Sultanate of Oman. Two deeply branching putative bacterial acetogen types were identified in the communities belonging to the Acetothermia (hereafter, types I and II) that exhibited distinct distributions among waters with lower and higher water–rock reaction (i.e., serpentinization influence), respectively. Metabolic reconstructions revealed contrasting core metabolic pathways of type I and II Acetothermia, including in acetogenic pathway components (e.g., bacterial- vs. archaeal-like carbon monoxide dehydrogenases [CODH], respectively), hydrogen use to drive acetogenesis, and chemiosmotic potential generation via respiratory (type I) or canonical acetogen ferredoxin-based complexes (type II). Notably, type II Acetothermia metabolic pathways allow for use of serpentinization-derived substrates and implicate them as key primary producers in contemporary hyperalkaline serpentinite environments. Phylogenomic analyses indicate that 1) archaeal-like CODH of the type II genomes and those of other serpentinite-associated Bacteria derive from a deeply rooted horizontal transfer or origin among archaeal methanogens and 2) Acetothermia are among the earliest evolving bacterial lineages. The discovery of dominant and early-branching acetogens in subsurface waters of the largest near-surface serpentinite formation provides insight into the physiological traits that likely facilitated rock-supported life to flourish on a primitive Earth and possibly on other rocky planets undergoing serpentinization.

Science & Technology - Other Topics↗
NASA NTRS1986

FASAC Technical Assessment Report: Soviet Space Science Research

This report is the work of a panel of eight US scientists who surveyed and assessed Soviet research in the spare sciences. All of the panelists were very familiar with Soviet research through their knowledge of the published scientific literature and personal contacts with Soviet and other foreign colleagues. In addition, all of the panelists reviewed considerable additional open literature--scientific, and popular, including news releases. The specific disciplines of Soviet space science research examined in detail for the report were: solar-terrestrial research, lunar and planetary research, space astronomy and astrophysics, and, life sciences. The Soviet Union has in the past carried out an ambitious program in lunar exploration and, more recently, in studies of the inner planets, Mars and especially Venus. The Soviets have provided scientific data about the latter planet which has been crucial for studies of the planet's evolution. Future programs envision an encounter with Halley's Comet, in March 1986, and missions to Mars and asteroids. The Soviet programs in the life sciences and solar-terrestrial research have been long-lasting and systematically pursued. Much of the ground-based and space-based research in these two disciplines appears to be motivated by the requirement to establish long-term human habitation in near-Earth space. The Soviet contributions to new discoveries and understanding in observational space astronomy and astrophysics have been few. This is in significant contrast to the very excellent theoretical work contributed by Soviet scientists in this discipline.

Lanzerotti, L. J.↗
DOE OSTI2024

DOE BSSD Performance Management Metrics Report Q1

Microbes play key roles in our biosphere, from driving global nutrient cycling to impacting plant, animal and human health and disease. Complex data from microbial genomes, proteins, and metabolites provide a window into these tiny engines that drive life on our planet. Yet these data are dispersed among researchers’ laboratories and various repositories, making it difficult to access. This calls for new ways of managing data, improving data interoperability, advancing community standards, and creating an infrastructure where data are shared efficiently. We have built the National Microbiome Data Collaborative (NMDC) to advance how scientists create, use, and reuse data to redefine the way we understand and harness the power of microbes. The vision of the National Microbiome Data Collaborative (NMDC) is to drive a microbiome data sharing network connecting data, people, and ideas to advance microbiome innovation and discovery. The NMDC was launched in 2019 and brought together DOE National Laboratories to collaborate across resources, capabilities, and expertise. The NMDC team was strategically assembled to include software developers, microbial researchers, metadata experts, and multi-omics specialists. The diversity of the NMDC team reflects the inherently interdisciplinary nature of microbiome science, and we leverage the strengths of the DOE National Laboratory system. Towards BER’s goal of advancing an iterative systems biology approach to the understanding of microbial genomes, the NMDC serves as a foundation for infrastructure, data standards, and community building. Together with the flagship DOE User Facilities, the Joint Genome Institute (JGI) and the Environmental Molecular Sciences Laboratory (EMSL), we are developing core capabilities in metadata standards for environmental descriptors and sample handling and processing; standardized bioinformatic workflows; an interface for data search and access; and robust community engagement activities. The NMDC production platform supports long-term data infrastructure and community building for BER’s bioenergy and environmental research goals. Our approach leverages lessons learned and an ambitious framework for collaborative, interdisciplinary data infrastructure to support microbiome research. The NMDC supports data, information, and knowledge access through three defined software tools – the Submission Portal, NMDC EDGE, and the Data Portal – driven by community needs. Herein, we describe the value proposition for the microbiome research community, our overarching strategy, and challenges and opportunities for developing the NMDC as both an infrastructure and community engagement program.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Surface basicity controls C–C coupling rates during carbon dioxide-assisted methane coupling over bifunctional Ca/ZnO catalysts

Carbon dioxide-assisted coupling of methane offers an approach to chemically upgrade two greenhouse gases and components of natural gas to produce ethylene and syngas. Prior research on this reaction has concentrated efforts on catalyst discovery, which has indicated that composites comprised of both reducible and basic oxides are especially promising. There is a need for detailed characterization of these bifunctional oxide systems to provide a more fundamental understanding of the active sites and their roles in the reaction. We studied the dependence of physical and electronic properties of Ca-modified ZnO materials on Ca content via X-ray photoelectron and absorption spectroscopies, electron microscopy, and infrared spectroscopic temperature-programmed desorption (IR-TPD). It was found that introduction of only 0.6 mol% Ca onto a ZnO surface is necessary to induce significant improvement in the catalytic production of C 2 species: C 2 selectivity increases from 5% on un-modified ZnO to 58%, at similar conversions. Evidence presented shows that this selectivity increase results from the formation of an interface between the basic CaO and reducible ZnO phases. Furthermore, the basicity of these interface sites correlates directly with catalytic activity over a wide composition range, and this relationship indicates that moderate CO 2 adsorption strength is optimal for CH 4 coupling. These results demonstrate, for the first time to our knowledge, a volcano-type relationship between CO 2 -assisted CH 4 coupling activity and catalyst surface basicity, which can inform further catalyst development.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Plant science decadal vision 2020–2030: Reimagining the potential of plants for a healthy and sustainable future

Abstract Plants, and the biological systems around them, are key to the future health of the planet and its inhabitants. The Plant Science Decadal Vision 2020–2030 frames our ability to perform vital and far‐reaching research in plant systems sciences, essential to how we value participants and apply emerging technologies. We outline a comprehensive vision for addressing some of our most pressing global problems through discovery, practical applications, and education. The Decadal Vision was developed by the participants at the Plant Summit 2019, a community event organized by the Plant Science Research Network. The Decadal Vision describes a holistic vision for the next decade of plant science that blends recommendations for research, people, and technology. Going beyond discoveries and applications, we, the plant science community, must implement bold, innovative changes to research cultures and training paradigms in this era of automation, virtualization, and the looming shadow of climate change. Our vision and hopes for the next decade are encapsulated in the phrase reimagining the potential of plants for a healthy and sustainable future. The Decadal Vision recognizes the vital intersection of human and scientific elements and demands an integrated implementation of strategies for research (Goals 1–4), people (Goals 5 and 6), and technology (Goals 7 and 8). This report is intended to help inspire and guide the research community, scientific societies, federal funding agencies, private philanthropies, corporations, educators, entrepreneurs, and early career researchers over the next 10 years. The research encompass experimental and computational approaches to understanding and predicting ecosystem behavior; novel production systems for food, feed, and fiber with greater crop diversity, efficiency, productivity, and resilience that improve ecosystem health; approaches to realize the potential for advances in nutrition, discovery and engineering of plant‐based medicines, and "green infrastructure." Launching the Transparent Plant will use experimental and computational approaches to break down the phytobiome into a "parts store" that supports tinkering and supports query, prediction, and rapid‐response problem solving. Equity, diversity, and inclusion are indispensable cornerstones of realizing our vision. We make recommendations around funding and systems that support customized professional development. Plant systems are frequently taken for granted therefore we make recommendations to improve plant awareness and community science programs to increase understanding of scientific research. We prioritize emerging technologies, focusing on non‐invasive imaging, sensors, and plug‐and‐play portable lab technologies, coupled with enabling computational advances. Plant systems science will benefit from data management and future advances in automation, machine learning, natural language processing, and artificial intelligence‐assisted data integration, pattern identification, and decision making. Implementation of this vision will transform plant systems science and ripple outwards through society and across the globe. Beyond deepening our biological understanding, we envision entirely new applications. We further anticipate a wave of diversification of plant systems practitioners while stimulating community engagement, underpinning increasing entrepreneurship. This surge of engagement and knowledge will help satisfy and stoke people's natural curiosity about the future, and their desire to prepare for it, as they seek fuller information about food, health, climate and ecological systems.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Evaluation of a scientific data search infrastructure

The ability to search over large scientific datasets has become crucial to next-generation scientific discoveries as data generated from scientific facilities grow dramatically. In previous work, we developed and deployed ScienceSearch, a search infrastructure for scientific data which uses machine learning to automate metadata creation. Our current deployment is deployed atop a container based platform at a HPC center. In this article, we present an evaluation and discuss our experiences with the ScienceSearch infrastructure. Specifically, we present a performance evaluation of ScienceSearch's infrastructure focusing on scalability trends. The obtained results show that ScienceSearch is able to serve up to 130 queries/min with latency under 3 s. We discuss our infrastructure setup and evaluation results to provide our experiences and a perspective on opportunities and challenges of our search infrastructure.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

Machine learning-driven predictive resource management in complex science workflows

Here, the collaborative efforts of large communities in science experiments, often comprising thousands of global members, reflect a monumental commitment to exploration and discovery. Recently, advanced and complex data processing has gained increasing importance in science experiments. Data processing workflows typically consist of multiple intricate steps, and the precise specification of resource requirements is crucial for each step to allocate optimal resources for effective processing. Estimating resource requirements in advance is challenging due to a wide range of analysis scenarios, varying skill levels among community members, and the continuously increasing spectrum of computing options. One practical approach to mitigate these challenges involves initially processing a subset of each step to measure precise resource utilization from actual processing profiles before completing the entire step. While this two-staged approach enables processing on optimal resources for most of the workflow, it has drawbacks such as initial inaccuracies leading to potential failures and suboptimal resource usage, along with overhead from waiting for initial processing completion, which is critical for fast-turnaround analyses. In this context, our study introduces a novel pipeline of machine learning models within a comprehensive workflow management system, the Production and Distributed Analysis (PanDA) system. These models employ advanced machine learning techniques to predict key resource requirements, overcoming challenges posed by limited upfront knowledge of characteristics at each step. Accurate forecasts of resource requirements enable informed and proactive decision-making in workflow management, enhancing the efficiency of handling diverse, complex workflows across heterogeneous resources.

97 MATHEMATICS AND COMPUTING↗