Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “science data systems operation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Multi-system analysis of offshore geologic carbon storage: a review of open-source data science solutions

Geologic carbon storage projects are maturing worldwide and the footprint of deployment in the offshore is expanding. At present, there are ten projects in operation or that have been completed, more than 50 in construction and development, and dozens of characterization studies completed or underway. Offshore geologic carbon storage offers potential benefits over onshore geologic carbon storage. These offshore projects are generally remote in location, distant from population centers, and avoid complicated pore space rights while having abundant prospective storage potential. Some offshore fields targeted for carbon storage have comparatively fewer prior borehole penetrations except for areas that have been explored for petroleum production, minimizing potential issues such as pressure interference and infrastructure impacts. Yet offshore geologic carbon storage projects face distinctive technical and economic challenges, such as seafloor geohazards (e.g., seabed instability), expensive maritime transport, and meteorological-oceanographic conditions that can damage infrastructure and impact operations. Analytical capabilities and improved computational speeds have advanced engineering, earth and energy sciences in the wake of the arrival of modern data science over the last decade. These advancements have created an opportunity for integrated, multi-systems modeling approaches utilizing artificial intelligence and machine learning that are no longer limited by computational issues. Analytical tools developed alongside this advancement in data science can be leveraged to calibrate the potential advantages and challenges of carbon storage operations in the offshore. New methods and approaches that incorporate data science to analyze multiple aspects of engineered and natural systems can provide insights that complement the characterization and onsite engineering that traditional commercial and operational software addresses. These new methods and approaches can potentially improve the outcome of energy operations and carbon storage. Providing multi-system, science-driven data analytics enhances the knowledge base that offshore developers, operators, and regulatory bodies may draw from to improve offshore site selection and operational efficiency. Here, we provide a brief synopsis of geologic carbon storage efforts to date, an overview of the engineered and natural systems involved in offshore geologic carbon storage, and a review of publicly available, open-source, offshore and/or carbon storage related data- and science-driven tools developed by 2010 or later that are suitable for screening and assessing regions for offshore geologic carbon storage.

artificial intelligence↗
DOE OSTI2024

UAS remote sensing (Osprey platform): Red-green-blue (RGB) imagery, thermal infrared (TIR) imagery, and canopy reflectance, Seward Peninsula, Alaska, 2018

Airborne remote sensing data collected using the Brookhaven National Laboratory's (BNL) heavy-lift unoccupied aerial system (UAS) octocopter platform - the Osprey - operated by the Terrestrial Ecosystem Science and Technology (TEST) group. This package includes data from 34 flights flown over the NGEE-Arctic Council Mile Maker 72 (MM72), Kougarok MM64, Kougarok MM80, and Teller MM27 sites in July, 2018. The Osprey is a multi-sensor UAS platform that simultaneously measures very high spatial resolution optical red/green/blue (RGB) and thermal infrared (TIR) surface "skin" temperature imagery, as well as surface reflectance at 1 nm intervals in the visible to near-infrared spectral range from ~350-1000 nm measured at regular intervals along each flight path. This package provide the Level 0 (raw, unprocessed) data collected by the Osprey platform. Ancillary aircraft data, flight mission parameters, and general flight conditions provided by the onboard flight and data collection computers are also included. Data and metadata are provided as text (*.txt, *.json), tabular (*.dat, *.csv, *.waypoint), and image (*.jpg) formats. This metadata document contains flight campaign, instrument and file metadata, along with a description of the L0 data, and file naming scheme. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

UAS remote sensing (Osprey platform): Red-green-blue (RGB) imagery, thermal infrared (TIR) imagery, and canopy reflectance, Seward Peninsula, Alaska, 2019

Airborne remote sensing data collected using the Brookhaven National Laboratory's (BNL) heavy-lift unoccupied aerial system (UAS) octocopter platform - the Osprey - operated by the Terrestrial Ecosystem Science and Technology (TEST) group (https://www.bnl.gov/testgroup). This package includes data from 17 flights flown over the NGEE-Arctic Council Mile Maker 64 (MM64) and Teller MM27 sites in July, 2019. The Osprey is a multi-sensor UAS platform that simultaneously measures very high spatial resolution optical red/green/blue (RGB) and thermal infrared (TIR) surface "skin" temperature imagery, as well as surface reflectance at 1 nm intervals in the visible to near-infrared spectral range from ~350-1000 nm measured at regular intervals along each flight path. This package provide the Level 0 (raw, unprocessed) data collected by the Osprey platform. Ancillary aircraft data, flight mission parameters, and general flight conditions provided by the onboard flight and data collection computers are also included. Data and metadata are provided as text (*.txt, *.json), tabular (*.dat, *.csv, *.waypoint), and image (*.jpg) formats. This metadata document contains flight campaign, instrument and file metadata, along with a description of the L0 data, and file naming scheme. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Hands-On, Heads-Up: Blending Cyber T&E with Data Science-Driven Training in Jupyter Notebooks

In an era of increasingly sophisticated threats to critical infrastructure, cybersecurity professionals must be more than just aware; they must be immersed, agile, and equipped to operate in environments where failure is not an option. Nowhere is this truer than in the nuclear sector, where cyber-physical systems, regulatory scrutiny, and insider threat potential demand a new generation of hands-on, technically fluent defenders. This paper presents a unified training approach that integrates Cybersecurity Test and Evaluation (T&E) with data science techniques using Jupyter Notebooks as the interactive lab environment. The program centers on a modular, scenario-driven curriculum designed to build not just knowledge but practical capability in the assessment and defense of radiation detection systems, firmware interfaces, and operational security postures.

98 - NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL↗
DOE OSTI2025

One-shot learning for solution operators of partial differential equations

Learning and solving governing equations of a physical system, represented by partial differential equations (PDEs), from data is a central challenge in many areas of science and engineering. Traditional numerical methods can be computationally expensive for complex systems and require complete governing equations. Existing data-driven machine learning methods require large datasets to learn a surrogate solution operator, which could be impractical. Here, we propose a solution operator learning method that requires only one PDE solution, i.e., one-shot learning, along with suitable initial and boundary conditions. Leveraging the locality of derivatives, we define a local solution operator in small local domains, train it using a neural network, and use it to predict solutions of new input functions via mesh-based fixed-point iteration or meshfree neural-network based approaches. We test our method on various PDEs, complex geometries, and a practical spatial infection spread application, demonstrating its effectiveness and generalization capabilities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

AI Driven Optimization of Public Transit

This project explores the application of AI-driven methods to optimize public transit operations for the Chattanooga Area Regional Transportation Authority (CARTA). By leveraging data analytics, machine learning, and predictive modeling, the initiative seeks to enhance system efficiency, improve rider experience, and support sustainability goals. This research, supported by the National Science Foundation and the U.S. Department of Energy, integrates real-time transit data with advanced computational tools to inform decision-making, optimize routes, and balance operational demands. The work exemplifies a forward-looking model for mid-sized cities aiming to modernize mobility systems through intelligent technology integration.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Transforming Energy Through Computational Excellence: NREL's Computational Science Center

Computational methods underpin advancing the science and engineering of energy efficiency, sustainable transportation, renewable power technologies, and developing a knowledge base to optimize energy systems. NREL's Computational Science Center (CSC) proudly focuses on providing the service of computing, advancing the science of computing, and enabling NREL's clean energy mission.

applied mathematics↗
DOE OSTI2025

Toward Drilling the Perfect Geothermal Well: An International Research Coordination Network for Geothermal Drilling Optimization Supported by Deep Machine Learning and Cloud Based Data Aggregation

The EDGE project, supported by the U.S. Department of Energy Geothermal Technologies Office under award DE-EE0008793, established a data-driven framework for improving the efficiency, cost-effectiveness, and reliability of geothermal well drilling. The project focused on developing scalable data infrastructure, advanced machine learning and probabilistic models, and integrated analytics tools to support continuous drilling optimization. A central objective was to reduce geothermal drilling costs by up to seventy percent while minimizing the risk of well failure through predictive diagnostics and adaptive planning. Over the project period, a comprehensive data repository was designed and deployed, incorporating records from over one hundred geothermal wells across varied geological settings. This repository supported both structured and unstructured data and adhered to FAIR data principles, enabling provenance tracking, quality control, and standardized metadata. The project introduced automated ingestion pipelines and a cloud-hosted platform that facilitated access to raw, processed, and derived datasets. This infrastructure served as the foundation for model development and analysis. Machine learning workflows were developed to predict key drilling metrics including rate of penetration, non-productive time, and total drilling costs. Self-organizing maps and dimensionality reduction methods were used to uncover operational patterns and outliers, while supervised learning algorithms such as random forests and deep neural networks were applied to forecast performance outcomes. The models were validated on heterogeneous datasets from both U.S. and Icelandic fields, demonstrating variable but significant predictive accuracy. The results indicated that finer temporal resolution, inclusion of lithological data, and consistency in operational annotations could substantially improve model performance. The project also implemented process mining techniques to reconstruct state-transition models from drilling event logs. These models enabled the identification of deviations from optimal workflows and provided insights into recurring failure modes. Analysis of non-productive time highlighted the impact of equipment failures, geological challenges, and human factors, offering opportunities for targeted mitigation strategies. The EDGE Dashboard was developed as a web-based expert system integrating data visualization, model outputs, and user-driven queries. It provided an accessible interface for operators to explore historical data, evaluate predicted outcomes, and compare drilling scenarios. Initial feedback from project partners suggested that the dashboard could serve as a foundation for more advanced advisory and optimization tools. Overall, the EDGE project demonstrated the feasibility and value of applying modern data science techniques to geothermal drilling. It delivered a set of interoperable tools and models that can support more efficient, lower-risk well development. The findings point toward a viable path for transitioning from advisory analytics to semi-autonomous drilling systems, contingent on continued collaboration, expanded datasets, and field validation. The project results have immediate relevance for drilling operations, data management practices, and future geothermal R&D efforts aimed at achieving reliable, cost-competitive geothermal energy at scale.

15 GEOTHERMAL ENERGY↗
DOE OSTI2025

RTN-011: Rubin Observatory Plans for an Early Science Program

This document outlines Rubin Observatory's plans for a dedicated \emph{Early Science Program} to enable high-impact science prior to the first annual data release of the Legacy Survey of Space and Time (LSST). Components of the Early Science Program include releasing science-grade commissioning data products via a series of ``Data Previews,'' ramping up of the transient alert stream during commissioning, implementing a program of incremental template generation to augment alert production in the early phases of the survey, and the first LSST Data Release, DR1, based on the first 6 months of data from the LSST. A detailed breakdown of which data products can be expected when is provided. The Rubin Operations team is working closely with the science community to optimize the Early Science Program for the time-domain and solar system science achievable in the first year of operations. This is a living document; both it and the Early Science Program will continue to evolve over the course of commissioning and pre-operations in response to the state of the as-built system and to community guidance.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Leveraging ARM Data to Improve Models for Predictive Understanding of Energy and Security Challenges

Extreme weather and natural hazards can disrupt the energy sector, affecting demand, generation, transmission, distribution, consumption and operational planning at regional and national scales. These disruptions stem from a broad range of atmospheric phenomena, including winter storms, freezing rain, wet snow loading, severe convection, flooding and landslides, wildfires, prolonged heat, and drought. Many of these same phenomena can also affect national security through impacts to transportation and infrastructure. To support the U.S. Department of Energy (DOE) focus on energy resilience and national security, the Atmospheric Radiation Measurement (ARM) User Facility is uniquely positioned to contribute measurement data, analyses, and modeling frameworks that can significantly improve predictive understanding of these hazards to mitigate their effects. To explore this opportunity, ARM convened a two-part virtual workshop in November 2025. The workshop engaged interdisciplinary experts in atmospheric science, energy systems, modeling, and operations. The goal of the meeting was to engage with these interdisciplinary experts to address three questions: • What are examples of atmospheric processes that represent significant risks to energy security or national security and where are those risks greatest? • What measurements or measurement strategies would improve ARM’s capacity to address these issues? • How can ARM and users of the ARM facility better work with the Energy Exascale Earth System Model (E3SM) and multi-sector modeling communities to apply ARM data to improving E3SM simulations of these phenomena? Participants were asked to submit white papers ahead of the meeting to initiate thinking on these themes and to help organize discussions. Workshop sessions were then organized around themes identified in the white papers. First from the white papers and then through subsequent discussions, workshop participants identified many examples that address the three questions listed above. Participants called out energy system vulnerabilities to weather phenomena such as the impact of freezing rain, strong winds, and excessive heat on power grids. They also noted the effects that weather phenomena could have on energy demand or supply (e.g., through effects of extreme temperatures). They called out security vulnerabilities such as impacts to crops from aerosol-borne pathogens and risks to industry due to melting permafrost in the Arctic. In all, over a dozen meteorological phenomena were linked to energy or security vulnerabilities. For many of the identified phenomena, participants pointed out where ARM was well poised to address issues (e.g., through measurements of cloud microphysics to inform studies of freezing rain) but also noted needs for additional measurements or modified measurement strategies. For example, adaptive scanning of severe weather would be valuable for probing winter storms or severe convection. Participants pointed out the value in integrating external observations with ARM measurements and with applying artificial intelligence (AI) to ARM observation analysis and they advocated for using model simulations to help optimize measurement strategies through Observing System Simulation Experiments (OSSEs). It was clear from the workshop that there are many ways that ARM observations can be used to mitigate energy and security concerns, but meeting participants were also asked to identify what they considered to be the greatest opportunities by ranking issues pertaining to the three workshop questions. This was accomplished through a survey administered to participants between the two virtual sessions. The highest-priority phenomena identified were winter storms, severe convection, and arctic processes. Discussion in the second session, therefore, focused primarily on these three areas, which were most fully developed in exploring ARM opportunities. Nevertheless, it was also clear that ARM has opportunities to contribute to all the identified topics. This report describes the workshop, including input from discussion and white papers (Sections 2 and 3) and a list of priority recommendations (section 4). Many other ideas for ARM contributions are discussed in individual white papers (Appendix D).

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2025

CROCUS Dual Polarization Ceilometer Data at Northeastern Illinois University Rooftop

This dataset is from the Department of Energy Office of Science funded project, CROCUS Urban Integrated Field Laboratory (https://crocus-urban.org/). The dual-polarization ceilometer (Vaisala CL61) is an autonomous lidar system operating at 910 nm wavelength, providing valuable measurements for understanding atmospheric boundary layer evolution, air quality, and cloud-aerosol interactions. The CL61 measures the backscattered signal intensity alternating between parallel- and cross-polarization signals. The unique depolarization measurement capability improves discrimination between different particle types, such as liquid droplets, ice crystals, and aerosols. The depolarization is highly dependent on the scatterer shape and orientation (spherical vs non-spherical particles), with the linear depolarization ratio providing a measure of dominant backscatter signal component from atmospheric particles at various heights, essentially allowing discrimination between liquid and solid particles. With its efficient optical system, CL61’s improved signal-to-noise ratio compared to traditional ceilometers allows studying detailed vertical profiles of aerosols and clouds up to 15 km height.Datasets are stored in a netCDF data format, and we we encourage users to make use of the associated toolkits available from Unidata (https://www.unidata.ucar.edu/software/netcdf/), Project Pythia (https://foundations.projectpythia.org/core/data-formats/netcdf-cf.html), and our “Instrument Cookbooks” (https://crocus-urban.github.io/instrument-cookbooks) for more information on how to process the metadata-rich datasets.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Nuclear Physics Network Requirements Review (Final Report)

The Energy Sciences Network (ESnet) is the high-performance network user facility for the US Department of Energy (DOE) Office of Science (SC) and delivers highly reliable data transport capabilities optimized for the requirements of data-intensive science. In essence, ESnet is the circulatory system that enables the DOE science mission by connecting all its laboratories and facilities in the US and abroad. ESnet is funded and stewarded by the Advanced Scientific Computing Research (ASCR) program and managed and operated by the Scientific Networking Division at Lawrence Berkeley National Laboratory (LBNL). ESnet is widely regarded as a global leader in the research and education networking community. ESnet interconnects DOE national laboratories, user facilities, and major experiments so that scientists can use remote instruments and computing resources as well as share data with collaborators, transfer large datasets, and access distributed data repositories. ESnet is specifically built to provide Between July 2023 and October 2023, ESnet and the Nuclear Physics program (NP) of the DOE SC organized an ESnet requirements review of NP-supported activities. Preparation for these events included identification of key stakeholders: program and facility management, research groups, and technology providers. Each stakeholder group was asked to prepare formal case study documents about its relationship to the NP program to build a complete understanding of the current, near-term, and long-term status, expectations, and processes that will support the science going forward.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Improving Self-Driving Labs: Quantifying System-Level Experiment Repeatability and Broadening Instrument-Level Compatibility

Modular Autonomous Research System (MARS) is a self-driving laboratory (SDL) which performs wet-lab science with peptide-lanthanide combinations in an automated and, ultimately, an autonomous manner to aid in soil analysis for domestic lithium mining. Autonomous experimentation involves automated experimentation, experiment planning, and active learning. MARS consists of a 6-axis robotic arm (UR5e) on a linear rail, pipette robots (Opentrons 2), and microplate readers. These components transport, operate on, and collect data with chemical solutions in standard labware. For effective autonomy, MARS must perform system-level labware operations repeatably, plan experiments autonomously, and be portable between research-domains. Repeatability is evaluated by labware placement precision, such that future operations can properly locate labware, as well as the elapsed time, so that low variance mean estimates of experiment duration can inform high-level researcher decision making. Autonomous experiment planning is the next step to decouple experimentation from human management; however, there is a conflict between the ideal system-level experiment goals and the constraints imposed by instruments’ limitations. Sub-domain portability is a long-term goal to extend MARS’ research beyond the chemistry of peptide-lanthanide binding to other sub-domains without having to invest significant overhead to system retrofitting. To address these goals, we manually trained the robotic arm labware placement and modelled statistical failurerate and uncertainty Additionally, we benchmarked the duration and variance of each experiment sub-operation as a heuristic for research decision making. Next, we use a parameterized geometric program (PGP) approach to design experiments that optimize system-level objectives and satisfy instrument-level constraints. Lastly, we proposed a Python framework to maximize MARS’ extensibility to other scientific sub-domains through a JSON-based experiment specification.

36 MATERIALS SCIENCE↗
DOE OSTI2024

2024 OES-Environmental 2024 State of the Science Report, Chapter 8: Marine Renewable Energy Data and Information Systems

As the marine renewable energy (MRE) sector grows, large amounts of environmental and technical data and information are being collected. When these data and information are openly available, they can be used to guide research and development, inform responsible siting and consenting of projects, and increase stakeholder understanding through transparency. For example, quality environmental data collected during the siting, consenting, construction, operation, and decommissioning of MRE projects can all play key roles in better characterizing baseline conditions, developing effective monitoring and mitigation strategies, and retiring environmental risks through data transferability (see Chapter 6). Ensuring that these data and information are easily discoverable and accessible will help the MRE sector make informed decisions and coexist in an increasingly busy ocean environment.

16 TIDAL AND WAVE POWER↗
DOE OSTI2024

SpecFIDLER User Manual (Software V.2.6.0)

The Spectroscopic Field Instrument for Detection of Low Energy Radiation (SpecFIDLER) allows response teams to detect and quantify plutonium contamination on the ground. Notional scenarios include dispersion from a weapon accident, or the launch failure of a space probe containing a radioisotope thermoelectric generator. Unlike other instruments, the thin-window sodium iodide detector is sensitive to the low-energy gamma rays emitted by plutonium isotopes. The system supports both mobile survey as well as stationary sampling. This manual provides information about installing, maintaining, and troubleshooting the SpecFIDLER. The scope of this document includes the physical hardware, software for data acquisition, and algorithms for data analysis. Recent changes to the software and algorithms aim to streamline the operation of the system.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

Robotics for HVAC applications: A critical review and future perspectives

Recent advances in artificial intelligence (AI), enhanced computational capabilities, and innovations in sensors and hardware have driven the increasing development and application of robots in heating, ventilation, and air conditioning (HVAC) systems. We selected and reviewed 101 studies published between 2005 and 2025, sourced from IEEE Xplore, Scopus, Web of Science, and the ACM Digital Library. To analyze these works, we developed a five-dimensional analytical framework (morphology, sensing, navigation, task execution, and system integration), inspired by the Springer Handbook of Robotics and tailored specifically for robotic applications in HVAC. Based on the reviewed studies, six distinct tasks spanning the entire HVAC lifecycle have been identified. Among the six tasks, inspection and maintenance dominate (59 %), followed by indoor monitoring and auditing (21 %), whereas leakage detection, comfort support, and installation/retrofit remain less explored. To address the identified gaps, this review proposes future research directions including investigating robot-aware HVAC design principles, developing multimodal HVAC sensing and data fusion techniques, enhancing robot training and hardware capabilities, and expanding robotic applications beyond Maintenance and Operations (M&O). The findings from this review inform future robotics research for HVAC applications and ultimately enhance system affordability, energy efficiency, resilience or reliability, and occupant environmental comfort. Moreover, it seeks to inspire researchers to explore the intersections of robotics, computer science, building science, and HVAC engineering fostering advancements in this multidisciplinary field.

AI↗
DOE OSTI2024

Employing artificial intelligence to steer exascale workflows with colmena

Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. In conclusion, our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing.

Workflows↗
DOE OSTI2025

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗