Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Software pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Report on ISR-1 High-Altitude Balloon Flight

To test small technologies at lower cost for space science applications, LANL has developed a small high altitude balloon payload that could, in the future, be regularly and inexpensively launched from LANL. A neutron detector, NEMO, was integrated to evaluate its performance in a space-like mixed-radiation environment and collect neutron data in the atmosphere. In collaboration with EES-14, a high-altitude balloon payload was launched from LANL Technical Area 51 on February 27, 2023 and April 17, 2023. For real-time geolocation, a SAM-M8Q M8 GNSS module was used to get position and time, and an Iridium RockBLOCK 9603 was used to communicate with the ground using the Iridium satellite fleet. These modules were all controlled using an Iteaduino Mega microcontroller board. Finally, a High Altitude Science Eagle Flight Computer with a temperature pressure sensor ran independently, writing data to an SD card. All of these modules were powered by a 5 mAh lithium polymer battery. The battery was attached to the bottom of the payload while the remaining electronics were embedded in the underside of the top of the payload. These modules were wired as seen in Figure 1-2. The Iteaduino Mega microcontroller board was programmed to use the RockBLOCK to send a message once every 10 minutes containing neutron and GPS data read off the NEMO and SAM M8Q, respectively. Once the message send attempt finished, the RockBLOCK would be slept for the rest of the 10 minute interval. The Eagle Flight Computer ran continuously throughout the flight, taking data every 6 seconds. The RockBLOCK message data was set up to be delivered from the Iridium satellite fleet to a website, where it was stored and parsed to create live maps and plots for analysis and balloon retrieval. The RockBLOCK message data was additionally configured to be sent to an email as a fail-safe. The payload was ground-tested successfully for over 50 hours, with multiple revisions occurring to best prepare for conditions at altitude and improve the software and firmware to fix any issues that cropped up with the data pipeline. Additional to the balloon payload, the flight had an attached iMet-4 radiosonde and Garmin T5 GPS Dog Collar. The radiosonde provided GPS and meteorological data. The T5 dog collar is used along with a Garmin Astro 430 to track the balloon at a range of up to 9 miles for retrieval. The balloon itself was initially a 1600 g meteorological balloon with an attached High Altitude Science parachute, both of which can be seen in Figure 1-3. After the first flight, the EES team swapped to a Rocketman parachute.

42 ENGINEERING↗

Open Reproducible Electron Microscopy Data Analysis

Electron microscopy (EM) is a cornerstone technique in the materials and biological sciences capable of imaging structures at nano- to atomic-scale resolution. Advances in technologies mean that one acquires datasets at increasing data rates and sizes. These advancements present enormous opportunities for researchers to understand complex systems. However, processing the resulting large-scale, complex data in a reproducible and shareable way is a real challenge for researchers. The building, managing, and maintaining complex workflows in a reproducible manner requires extensive knowledge in several areas outside the researchers’ core skill sets, such as software engineering, data science, and high-performance computing (HPC). Our work demonstrates an innovative approach to solving these problems, enabling reproducible EM data analysis through container encapsulated pipelines. Using modern container technologies, we encapsulate processing elements and connect them using shared memory. We expose user-friendly, advanced algorithms and tools to allow end users to utilize without expert programming skills. The platform enables reproducible, scalable, shareable pipelines for the analysis and visualization of EM data. Focusing on interoperability, we leverage the DOE and other agencies’ existing investments to provide a powerful software platform for EM data analysis.

Harris, Christopher↗

Supercomputing Pipelines Search for Therapeutics Against COVID-19

The urgent search for drugs to combat SARS-CoV-2 has included the use of supercomputers. The use of general-purpose graphical processing units (GPUs), massive parallelism, and new software for high-performance computing (HPC) has allowed researchers to search the vast chemical space of potential drugs faster than ever before. We developed a new drug discovery pipeline using the Summit supercomputer at Oak Ridge National Laboratory to help pioneer this effort, with new platforms that incorporate GPU-accelerated simulation and allow for the virtual screening of billions of potential drug compounds in days compared to weeks or months for their ability to inhibit SARS-COV-2 proteins. Here, this effort will accelerate the process of developing drugs to combat the current COVID-19 pandemic and other diseases.

60 APPLIED LIFE SCIENCES↗

Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory

The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.

Yang, Wei [SLAC]↗

EJFAT Scientific Perspective

Presented new computing model to the test by deploying the EJFAT system alongside a data-stream processing framework running the production-level CLAS12 event reconstruction application. In this experiment, a continuous stream of CLAS12 Level-1 identified events was processed in real-time using the EJFAT load balancer, distributing the workload across 90 computing nodes located across the U.S. This marks the first-ever large-scale, real-time distributed data stream processing experiment, demonstrating that scientific data-streaming pipelines can efficiently scale across four dimensions, thanks to EJFAT’s advanced hardware and software capabilities.

Gyurjyan, Vardan [Thomas Jefferson National Accele↗

gRNA-SeqRET: a universal tool for targeted and genome-scale gRNA design and sequence extraction for prokaryotes and eukaryotes

High-throughput genetic screening is frequently employed to rapidly associate gene with phenotype and establish sequence-function relationships. With the advent of CRISPR technology, and the ability to functionally interrogate previously genetically recalcitrant organisms, non-model organisms can be investigated using pooled guide RNA (gRNA) libraries and sequencing-based assays to quantitatively assess fitness of every targeted locus in parallel. To aid the construction of pooled gRNA assemblies, we have developed an in silico design workflow for gRNA selection using the gRNA Sequence Region Extraction Tool (gRNA-SeqRET). Built upon the previously developed CCTop, gRNA-SeqRET enables automated, scalable design of gRNA libraries that target user-specified regions or whole genomes of any prokaryote or eukaryote. Additionally, gRNA-SeqRET automates the bulk extraction of any regions of sequence relative to genes or other features, aiding in the design of homology arms for insertion or deletion constructs. We also assess in silico the application of a designed gRNA library to other closely related genomes and demonstrate that for very closely related organisms Average Nucleotide Identity (ANI) > 95% a large fraction of the library may be of relevance. The gRNA-SeqRET web application pipeline can be accessed at https://grna.jgi.doe.gov. The source code is comprised of freely available software tools and customized Python scripts, and is available at https://bitbucket.org/berkeleylab/grnadesigner/src/master/ under a modified BSD open-source license (https://bitbucket.org/berkeleylab/grnadesigner).

59 BASIC BIOLOGICAL SCIENCES↗

Summary of Work on a Software Adapter for Project Requirements and MBSE

The National Reactor Innovation Center (NRIC) has funded the development of a digital pipeline to pull a project’s requirements from IBM’s Rational Dynamic Object Oriented Requirements System (DOORS), save them to Deep Lynx, then push them into the corresponding project in Innoslate. These utilities are accessible from a web browser and allow a project’s functional and business requirements to be tied to its tree of model-based systems engineering (MBSE) data.

99 GENERAL AND MISCELLANEOUS↗

wa-hls4ml: A GNN Surrogate Model for hls4ml

Recent advancements in use of machine learning techniques on field-programmable gate arrays (FPGAs) have allowed for implementation of embedded neural networks with extremely low latency. This is invaluable for particle detectors at the Large Hadron Collider, where latency and used area must be strictly bounded. The hls4ml framework is a procedure for converting from trained machine learning model software, to a synthesis result that can be used on an FPGA. However, running the pipeline is a time-consuming procedure, and there is a strong risk of failure. In particular, it is possible that the model is unable to be converted into a synthesis result, or that the resource consumption of the model will exceed the resources of the target FPGA. To aid with this development, we introduce wa-hls4ml, a surrogate model which uses a graph neural network to emulate the structure of the source models. The goal is to estimate the chance of success and resource consumption of an arbitrary model when passed through the hls4ml procedure, without the time consumption of actually running the pipeline.

43 PARTICLE ACCELERATORS↗

A Tool Kit for Generating Simulated Radiation Measurements for Advanced Reactor Safeguards and Security

A tool kit was developed to simulate and analyze passive radiation measurements of molten salt reactor (MSR) operations to support development of nuclear safeguards approaches for this emerging reactor technology. A Transient Simulation Framework of Reconfigurable Modules (TRANSFORM) multiphysics simulation of an MSR produces time-dependent isotopic inventories at user-selected locations within the model. The tool kit implements the Gamma Detector Response and Analysis Software (GADRAS) application programming interface to inject the TRANSFORM isotopic inventories extracted/processed by a Python pipeline into GADRAS models of user-defined geometries. The TRANSFORM inventories are the source terms used to obtain synthetic measurements from GADRAS-defined detectors. The speed of TRANSFORM and GADRAS simulations enables surveying the large design space of MSRs (e.g., fuel type, fuel salt composition, number of loops) and the plethora of measurements (e.g., location, detector type, and collimation) within the reactor. This has enabled timely assessment of the various measurement locations and detectors to identify the most effective and efficient safeguards approach for a specific MSR design. Lastly, the tool kit also simulates extracted samples that can be aged to a desired dose, enabling stakeholders to optimize a measurement plan to use sample analysis as an element within a broader material accountancy plan.

Westphal, Greg↗

A Graph Neural Network Surrogate Model for hls4ml

Recent advancements in use of machine learning (ML) techniques on field-programmable gate arrays (FPGAs) have allowed for the implementation of embedded neural networks with extremely low latency. This is invaluable for particle detectors at the Large Hadron Collider, where latency and used area are strictly bounded. The hls4ml framework is a procedure that converts trained ML model software to a synthesis result to can be used on an FPGA. However, running the pipeline is a time-consuming procedure, and there is a strong risk of failure. In particular, it may not be possible to successfully convert a model into a synthesis result, or the resource consumption of the model may exceed the resources of the target FPGA. To aid with this development, we introduce wa-hls4ml, a surrogate model using a graph neural network to emulate the structure of the source models. The goal is to estimate the chance of success and resource consumption of a given model when passed through the hls4ml pipeline, without needing to run the pipeline.

Plotnikov, Dennis↗

AutoCCS: automated collision cross-section calculation software for ion mobility spectrometry–mass spectrometry

Abstract Motivation Ion mobility spectrometry (IMS) separations are increasingly used in conjunction with mass spectrometry (MS) for separation and characterization of ionized molecular species. Information obtained from IMS measurements includes the ion’s collision cross section (CCS), which reflects its size and structure and constitutes a descriptor for distinguishing similar species in mixtures that cannot be separated using conventional approaches. Incorporating CCS into MS-based workflows can improve the specificity and confidence of molecular identification. At present, there is no automated, open-source pipeline for determining CCS of analyte ions in both targeted and untargeted fashion, and intensive user-assisted processing with vendor software and manual evaluation is often required. Results We present AutoCCS, an open-source software to rapidly determine CCS values from IMS-MS measurements. We conducted various IMS experiments in different formats to demonstrate the flexibility of AutoCCS for automated CCS calculation: (i) stepped-field methods for drift tube-based IMS (DTIMS), (ii) single-field methods for DTIMS (supporting two calibration methods: a standard and a new enhanced method) and (iii) linear calibration for Bruker timsTOF and non-linear calibration methods for traveling wave based-IMS in Waters Synapt and Structures for Lossless Ion Manipulations. We demonstrated that AutoCCS offers an accurate and reproducible determination of CCS for both standard and unknown analyte ions in various IMS-MS platforms, IMS-field methods, ionization modes and collision gases, without requiring manual processing. Availability and implementation https://github.com/PNNL-Comp-Mass-Spec/AutoCCS. Supplementary information Supplementary data are available at Bioinformatics online. Demo datasets are publicly available at MassIVE (Dataset ID: MSV000085979).

47 OTHER INSTRUMENTATION↗

WETO Software Stack Best Practices

Wind energy researchers typically share one key characteristic: a passion for increasing wind energy in the global energy mix. The U.S. Department of Energy (DOE) supports this mission in a number of ways including allocating funding directly to various aspects of wind energy research through the Office of Energy Efficiency and Renewable Energy (EERE) via the Wind Energy Technologies Office (WETO). While the traditional output of research is academic publication, software development efforts are increasingly a major focus. Software tools in the research environment allow researchers to describe an idea and quickly increase the scope and scale as they study it further. As a product of research, these tools represent a direct pipeline from researcher to industry practitioners since they are the implementation of ideas described in academic publications. Given this vital role in wind energy research and commercial development, the broad research software portfolio supported by WETO must maintain a minimum level of quality to support the wind energy field in the growing transition to renewable energy. This report outlines a series o f best practices to be adopted by all WETO-supported software projects, as well as expectations that the communities interacting with these projects should have of the developers and tools themselves. Wind energy research software has a unique standing in the field of scientific software. The stakeholders are varied with a subset being: (1) DOE EERE leadership, (2) DOE WETO leadership and program managers, (3) National lab leadership, (4) Associated project principle investigators, (5) Research software engineers, (6) Wind energy researchers in academia (including graduate students, post docs, and national lab staff), (7) Industry researchers and practitioners, (8) Commercial software developers, and (9) The general public interested in wind energy. These software are typically the end-user of other generic software libraries, so the funding cycles are often tied to applied research rather than the development of the software itself. Since the developers are also wind energy researchers, these tools are typically designed in a way that closely resembles the application in which they're used. Additionally, the expertise and incentives for the developers have a high variability, and often neither are aligned with software engineering or computer science. Given the unique environment in which wind energy research software is produced and consumed, it is critical for model owners to understand the context of their software. A framework for developing this understanding is to answer the following questions of a given software project: What is it's purpose? What is its role in the field of wind energy? What is the profile of the expected users? For how long will it be relevant? What is the expected impact? These questions allow model owners to identify the appropriate methods for the design, development, and long term maintenance of their software. Additionally, the answer provide context for future planners to understand why particular decisions were made and discern the consequences of changing course. The information is aggregated from experience within WETO-supported software development groups as well as external organizations and efforts to define the craft of research software engineering. These best practices aim to make the collaborative development process efficient and effective while improving the model understanding across stakeholders. Additionally, the general adoption of a common framework for software quality ensures that the end users of WETO software can trust these tools and accurately understand the risks to workflow integration.

17 WIND ENERGY↗

Report of the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science

This report summarizes insights from the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science, which convened more than 40 experts from national laboratories, academia, industry, and community organizations to chart a path toward more powerful, sustainable, and collaborative scientific software ecosystems. To address urgent challenges at the intersection of high-performance computing (HPC), AI, and scientific software, participants envisioned agile, robust ecosystems built through socio-technical co-design—the intentional integration of social and technical components as interdependent parts of a unified strategy. This approach combines advances in AI, HPC, and software with new models for cross-disciplinary collaboration, training, and workforce development. Key recommendations include building modular, trustworthy AI-enabled scientific software systems; enabling scientific teams to integrate AI systems into their workflows while preserving human creativity, trust, and scientific rigor; and creating innovative training pipelines that keep pace with rapid technological change. Pilot projects were identified as near-term catalysts, with initial priorities focused on hybrid AI/HPC infrastructure, cross-disciplinary collaboration and pedagogy, responsible AI guidelines, and prototyping of public-private partnerships. This report presents a vision of next-generation ecosystems for scientific computing where AI, software, hardware, and human expertise are interwoven to drive discovery, expand access, strengthen the workforce, and accelerate scientific progress.

97 MATHEMATICS AND COMPUTING↗

Introducing NOODLES: Collaborative Visualization in a Flavorful Package

We introduce NOODLES, a lightweight collaborative visualization and analysis protocol. NOODLES was developed to enable new scientific workflows while enhancing existing pipelines. Using a technology-minimal approach to strengthen applicability, this protocol allows researchers to tie disparate software together as well as making domain specific visualizations accessible across various platforms and form factors. In this talk we discuss background, challenges, and gaps in the current tool landscape that lead to the development of the protocol. We then describe the design of the protocol and present several use cases to demonstrate the efficacy of NOODLES in addressing complex visualization challenges and how it co-exists with existing tools. We conclude by outlining in-progress work and potential avenues for future development within the NOODLES framework.

collaborative analysis↗

Open Source Platform for Live Processing of High Data Rate Electron Microscopy (SBIR Phase I Award Topic 15, Subtopic B, Final Scientific / Technical Report)

4D scanning transmission electron microscopy (STEM) is experiencing a revolution both in terms of data size, and data veracity, as are many other areas of experimental data acquisition. This presents enormous opportunities to make new scientific discoveries, but is also hamstrung by aging software that is no longer fit for purpose. Experimental data acquisition platforms must make the same investments in software infrastructure that the simulation community made decades ago to take advantage of high-performance computing (HPC) if they hope to realize the full potential of the latest advances made in detector technology. This requires rethinking the design and deployment of hardware, moving away from small, fixed commodity computing capabilities with closed interfaces to open, modular, scalable systems designed for parallelism and throughput. The technical approach will leverage best-in-class open source technologies, and develop new approaches strategically to build an open source platform for experimental data acquisition targeting high data rate electron microscopy. The project will develop a software platform that uses the power and performance of modern HPC resources to ingest the large volume of data produced by electron microscopes using direct electron detectors. A set of configurable pipelines will be used to enable the processing to be tailored to the problem being investigated with builtin visualization of data at interactive rates. This project will leverage investments made by the DOE and other agencies to provide a powerful software platform for the experimental data community working with most powerful electron microscopes, such as 4D STEM detectors. It will create a platform that serves as a reference for the community. Permissive open source licensing will make the platform available to all, with a simple deployment strategy offering the ability to keep data close to where it is generated. This advanced software platform will require customization for each deployment, creating opportunities for software services that are typically offered to extend open source software platforms.

97 MATHEMATICS AND COMPUTING↗

PtychoShelves , a versatile high-level framework for high-performance analysis of ptychographic data

Over the past decade, ptychography has been proven to be a robust tool for non-destructive high-resolution quantitative electron, X-ray and optical microscopy. It allows for quantitative reconstruction of the specimen's transmissivity, as well as recovery of the illuminating wavefront. Additionally, various algorithms have been developed to account for systematic errors and improved convergence. With fast ptychographic microscopes and more advanced algorithms, both the complexity of the reconstruction task and the data volume increase significantly. PtychoShelves is a software package which combines high-level modularity for easy and fast changes to the data-processing pipeline, and high-performance computing on CPUs and GPUs.

97 MATHEMATICS AND COMPUTING↗

Distributed Many-to-Many Protein Sequence Alignment using Sparse Matrices

Identifying similar protein sequences is a core step in many computational biology pipelines such as detection of homologous protein sequences, generation of similarity protein graphs for downstream analysis, functional annotation, and gene location. Performance and scalability of protein similarity search have proven to be a bottleneck in many bioinformatics pipelines due to increase in cheap and abundant sequencing data. This work presents a new distributed-memory software PASTIS. PASTIS relies on sparse matrix computations for efficient identification of possibly similar proteins. We use distributed sparse matrices for scalability and show that the sparse matrix infrastructure is a great fit for protein similarity search when coupled with a fully-distributed dictionary of sequences that allow remote sequence requests to be fulfilled. Our algorithm incorporates the unique bias in amino acid sequence substitution in search without altering basic sparse matrix model, and in turn, achieves ideal scaling up to millions of protein sequences.

97 MATHEMATICS AND COMPUTING↗

MetaboDirect: an analytical pipeline for the processing of FT-ICR MS-based metabolomic data

Background: Microbiomes are now recognized as the main drivers of ecosystem function ranging from the oceans and soils to humans and bioreactors. However, a grand challenge in microbiome science is to characterize and quantify the chemical currencies of organic matter (i.e., metabolites) that microbes respond to and alter. Critical to this has been the development of Fourier transform ion cyclotron resonance mass spectrometry (FT-ICR MS), which has drastically increased molecular characterization of complex organic matter samples, but challenges users with hundreds of millions of data points where readily available, user-friendly, and customizable software tools are lacking. Results: Here, we build on years of analytical experience with diverse sample types to develop MetaboDirect, an open-source, command-line-based pipeline for the analysis (e.g., chemodiversity analysis, multivariate statistics), visualization (e.g., Van Krevelen diagrams, elemental and molecular class composition plots), and presentation of direct injection high-resolution FT-ICR MS data sets after molecular formula assignment has been performed. When compared to other available FT-ICR MS software, MetaboDirect is superior in that it requires a single line of code to launch a fully automated framework for the generation and visualization of a wide range of plots, with minimal coding experience required. Among the tools evaluated, MetaboDirect is also uniquely able to automatically generate biochemical transformation networks (ab initio) based on mass differences (mass difference network-based approach) that provide an experimental assessment of metabolite connections within a given sample or a complex metabolic system, thereby providing important information about the nature of the samples and the set of microbial reactions or pathways that gave rise to them. Finally, for more experienced users, MetaboDirect allows users to customize plots, outputs, and analyses. Conclusion: Application of MetaboDirect to FT-ICR MS-based metabolomic data sets from a marine phage-bacterial infection experiment and a Sphagnum leachate microbiome incubation experiment showcase the exploration capabilities of the pipeline that will enable the research community to evaluate and interpret their data in greater depth and in less time. It will further advance our knowledge of how microbial communities influence and are influenced by the chemical makeup of the surrounding system. The source code and User’s guide of MetaboDirect are freely available through (https://github.com/Coayala/MetaboDirect) and (https://metabodirect.readthedocs.io/en/latest/), respectively.

54 ENVIRONMENTAL SCIENCES↗