SEARCH · Engineering Papers

Results for “Software pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

All sources NASA NTRS DOE OSTI

DOE OSTI2024

wa-hls4ml: A GNN Surrogate Model for hls4ml

Recent advancements in use of machine learning techniques on field-programmable gate arrays (FPGAs) have allowed for implementation of embedded neural networks with extremely low latency. This is invaluable for particle detectors at the Large Hadron Collider, where latency and used area must be strictly bounded. The hls4ml framework is a procedure for converting from trained machine learning model software, to a synthesis result that can be used on an FPGA. However, running the pipeline is a time-consuming procedure, and there is a strong risk of failure. In particular, it is possible that the model is unable to be converted into a synthesis result, or that the resource consumption of the model will exceed the resources of the target FPGA. To aid with this development, we introduce wa-hls4ml, a surrogate model which uses a graph neural network to emulate the structure of the source models. The goal is to estimate the chance of success and resource consumption of an arbitrary model when passed through the hls4ml procedure, without the time consumption of actually running the pipeline.

43 PARTICLE ACCELERATORS↗

NASA NTRS2017

Improved Reference Sampling and Subtraction: A Technique for Reducing the Read Noise of Near-Infrared Detector Systems

Near-infrared array detectors, like the James Webb Space Telescope (JWST) NIRSpec's Teledyne's H2RGs, often provide reference pixels and a reference output. These are used to remove correlated noise. Improved reference sampling and subtraction (IRS(exp 2)) is a statistical technique for using this reference information optimally in a leastsquares sense. Compared with the traditional H2RG readout, IRS(exp 2) uses a different clocking pattern to interleave many more reference pixels into the data than is otherwise possible. Compared with standard reference correction techniques, IRS(exp 2) subtracts the reference pixels and reference output using a statistically optimized set of frequency dependent weights. The benefits include somewhat lower noise variance and much less obvious correlated noise. NIRSpec's IRS(exp 2) images are cosmetically clean, with less 1/f banding than in traditional data from the same system. This article describes the IRS(exp 2) clocking pattern and presents the equations needed to use IRS(exp 2) in systems other than NIRSpec. For NIRSpec, applying these equations is already an option in the calibration pipeline. As an aid to instrument builders, we provide our prototype IRS(exp 2) calibration software and sample JWST NIRSpec data. The same techniques are applicable to other detector systems, including those based on Teledyne's H4RG arrays. The H4RG's interleaved reference pixel readout mode is effectively one IRS(exp 2) pattern.

Methods-statistical↗

DOE OSTI2022

A Tool Kit for Generating Simulated Radiation Measurements for Advanced Reactor Safeguards and Security

A tool kit was developed to simulate and analyze passive radiation measurements of molten salt reactor (MSR) operations to support development of nuclear safeguards approaches for this emerging reactor technology. A Transient Simulation Framework of Reconfigurable Modules (TRANSFORM) multiphysics simulation of an MSR produces time-dependent isotopic inventories at user-selected locations within the model. The tool kit implements the Gamma Detector Response and Analysis Software (GADRAS) application programming interface to inject the TRANSFORM isotopic inventories extracted/processed by a Python pipeline into GADRAS models of user-defined geometries. The TRANSFORM inventories are the source terms used to obtain synthetic measurements from GADRAS-defined detectors. The speed of TRANSFORM and GADRAS simulations enables surveying the large design space of MSRs (e.g., fuel type, fuel salt composition, number of loops) and the plethora of measurements (e.g., location, detector type, and collimation) within the reactor. This has enabled timely assessment of the various measurement locations and detectors to identify the most effective and efficient safeguards approach for a specific MSR design. Lastly, the tool kit also simulates extracted samples that can be aged to a desired dose, enabling stakeholders to optimize a measurement plan to use sample analysis as an element within a broader material accountancy plan.

Westphal, Greg↗

DOE OSTI2024

A Graph Neural Network Surrogate Model for hls4ml

Recent advancements in use of machine learning (ML) techniques on field-programmable gate arrays (FPGAs) have allowed for the implementation of embedded neural networks with extremely low latency. This is invaluable for particle detectors at the Large Hadron Collider, where latency and used area are strictly bounded. The hls4ml framework is a procedure that converts trained ML model software to a synthesis result to can be used on an FPGA. However, running the pipeline is a time-consuming procedure, and there is a strong risk of failure. In particular, it may not be possible to successfully convert a model into a synthesis result, or the resource consumption of the model may exceed the resources of the target FPGA. To aid with this development, we introduce wa-hls4ml, a surrogate model using a graph neural network to emulate the structure of the source models. The goal is to estimate the chance of success and resource consumption of a given model when passed through the hls4ml pipeline, without needing to run the pipeline.

Plotnikov, Dennis↗

NASA NTRS2019

The Cumulus Ecosystem: Open Source and Beyond to Foster Collaboration

NASA’s Earth Observing System Data and Information System (EOSDIS) open source Cumulus software is designed as a common set of code and services that can be used to create a pipeline to deliver and manage earth science data in the cloud. Cumulus strives to create an ecosystem on the foundation of open source that unites those with shared problems and goals by encouraging users to contribute solutions back to the platform. Large parts of ingesting and managing data are common and much of what is created can be used by others. Our goal is to maximize collaboration and code reuse while allowing users to design a custom solution that meets their needs without having to take on extraneous functionality. In this talk we will describe how the Cumulus ecosystem works beyond just open source software. We will review the technology, the successes and challenges, and the evolution and future of Cumulus as an ecosystem.

Cumulus↗

NASA NTRS2012

New Techniques for High-Contrast Imaging with ADI: The ACORNS-ADI SEEDS Data Reduction Pipeline

We describe Algorithms for Calibration, Optimized Registration, and Nulling the Star in Angular Differential Imaging (ACORNS-ADI), a new, parallelized software package to reduce high-contrast imaging data, and its application to data from the Strategic Exploration of Exoplanets and Disks (SEEDS) survey. We implement seyeral new algorithms, includbg a method to centroid saturated images, a trimmed mean for combining an image sequence that reduces noise by up to approx 20%, and a robust and computationally fast method to compute the sensitivitv of a high-contrast obsen-ation everywhere on the field-of-view without introducing artificial sources. We also include a description of image processing steps to remove electronic artifacts specific to Hawaii2-RG detectors like the one used for SEEDS, and a detailed analysis of the Locally Optimized Combination of Images (LOCI) algorithm commonly used to reduce high-contrast imaging data. ACORNS-ADI is efficient and open-source, and includes several optional features which may improve performance on data from other instruments. ACORNS-ADI is freely available for download at www.github.com/t-brandt/acorns_-adi under a BSD license

Brandt, Timothy D.↗

DOE OSTI2021

AutoCCS: automated collision cross-section calculation software for ion mobility spectrometry–mass spectrometry

Abstract Motivation Ion mobility spectrometry (IMS) separations are increasingly used in conjunction with mass spectrometry (MS) for separation and characterization of ionized molecular species. Information obtained from IMS measurements includes the ion’s collision cross section (CCS), which reflects its size and structure and constitutes a descriptor for distinguishing similar species in mixtures that cannot be separated using conventional approaches. Incorporating CCS into MS-based workflows can improve the specificity and confidence of molecular identification. At present, there is no automated, open-source pipeline for determining CCS of analyte ions in both targeted and untargeted fashion, and intensive user-assisted processing with vendor software and manual evaluation is often required. Results We present AutoCCS, an open-source software to rapidly determine CCS values from IMS-MS measurements. We conducted various IMS experiments in different formats to demonstrate the flexibility of AutoCCS for automated CCS calculation: (i) stepped-field methods for drift tube-based IMS (DTIMS), (ii) single-field methods for DTIMS (supporting two calibration methods: a standard and a new enhanced method) and (iii) linear calibration for Bruker timsTOF and non-linear calibration methods for traveling wave based-IMS in Waters Synapt and Structures for Lossless Ion Manipulations. We demonstrated that AutoCCS offers an accurate and reproducible determination of CCS for both standard and unknown analyte ions in various IMS-MS platforms, IMS-field methods, ionization modes and collision gases, without requiring manual processing. Availability and implementation https://github.com/PNNL-Comp-Mass-Spec/AutoCCS. Supplementary information Supplementary data are available at Bioinformatics online. Demo datasets are publicly available at MassIVE (Dataset ID: MSV000085979).

47 OTHER INSTRUMENTATION↗

DOE OSTI2024

WETO Software Stack Best Practices

Wind energy researchers typically share one key characteristic: a passion for increasing wind energy in the global energy mix. The U.S. Department of Energy (DOE) supports this mission in a number of ways including allocating funding directly to various aspects of wind energy research through the Office of Energy Efficiency and Renewable Energy (EERE) via the Wind Energy Technologies Office (WETO). While the traditional output of research is academic publication, software development efforts are increasingly a major focus. Software tools in the research environment allow researchers to describe an idea and quickly increase the scope and scale as they study it further. As a product of research, these tools represent a direct pipeline from researcher to industry practitioners since they are the implementation of ideas described in academic publications. Given this vital role in wind energy research and commercial development, the broad research software portfolio supported by WETO must maintain a minimum level of quality to support the wind energy field in the growing transition to renewable energy. This report outlines a series o f best practices to be adopted by all WETO-supported software projects, as well as expectations that the communities interacting with these projects should have of the developers and tools themselves. Wind energy research software has a unique standing in the field of scientific software. The stakeholders are varied with a subset being: (1) DOE EERE leadership, (2) DOE WETO leadership and program managers, (3) National lab leadership, (4) Associated project principle investigators, (5) Research software engineers, (6) Wind energy researchers in academia (including graduate students, post docs, and national lab staff), (7) Industry researchers and practitioners, (8) Commercial software developers, and (9) The general public interested in wind energy. These software are typically the end-user of other generic software libraries, so the funding cycles are often tied to applied research rather than the development of the software itself. Since the developers are also wind energy researchers, these tools are typically designed in a way that closely resembles the application in which they're used. Additionally, the expertise and incentives for the developers have a high variability, and often neither are aligned with software engineering or computer science. Given the unique environment in which wind energy research software is produced and consumed, it is critical for model owners to understand the context of their software. A framework for developing this understanding is to answer the following questions of a given software project: What is it's purpose? What is its role in the field of wind energy? What is the profile of the expected users? For how long will it be relevant? What is the expected impact? These questions allow model owners to identify the appropriate methods for the design, development, and long term maintenance of their software. Additionally, the answer provide context for future planners to understand why particular decisions were made and discern the consequences of changing course. The information is aggregated from experience within WETO-supported software development groups as well as external organizations and efforts to define the craft of research software engineering. These best practices aim to make the collaborative development process efficient and effective while improving the model understanding across stakeholders. Additionally, the general adoption of a common framework for software quality ensures that the end users of WETO software can trust these tools and accurately understand the risks to workflow integration.

17 WIND ENERGY↗

NASA NTRS2008

Enabling and Enhancing Space Mission Success and Reduction of Risk through the Application of an Integrated Data Architecture

The engineering phases of design, development, test, and evaluation (DDT and E) and subsequent planning, preparation, and operation (Ops) of space vehicles in a complex and distributed environment requires massive and continuous flows of information across the enterprise and across temporal stages of the vehicle lifecycle. The resulting capabilities at each subsequent stage depend in part on the capture, preparation, storage, and subsequent provision of information from prior stages. The United States National Aeronautics and Space Administration (NASA) is currently designing a fleet of new vehicles that will replace the Space Shuttle and expand space operations and exploration capabilities. This includes the 2 stage human rated lift vehicle Ares 1 and its associated crew vehicle the Orion, and a service module; the heavy lift cargo vehicle, Ares 5, and an associated cargo stage known as the Earth Departure Stage; and a Lunar Lander vehicle that contains a descent stage, and ascent stage, and a habitation module. A variety of concurrent assorted ground operations infrastructure including software and facilities are also being developed, assorted technology and assembly designs and development for equipment such as EVA suits, life support systems, command and control technologies are also in the pipeline. The development is occurring in a distributed manner, with project deliverables being contributed by a large and diverse assortment of vendors and most space faring nations. Critical information about all of the components, software, and procedures must be shared during the DDT and E phases and then made readily available to the mission operations staff for access during the planning, preparation, and operations phases, and also need to be readily available for system to system interactions. The Constellation Data Systems Project (CxDS) is identifying the needs, and designing and deploying systems and processes to support these needs. This paper details the steps and processes that NASA is applying within the Constellation Program to manage this data and information, and to insure that the correct information is available, correctly annotated, and can be provisioned digitally to enhance response times, and support engineering analysis and anomaly resolution.

Brummett, Robert C.↗

DOE OSTI2025

Report of the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science

This report summarizes insights from the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science, which convened more than 40 experts from national laboratories, academia, industry, and community organizations to chart a path toward more powerful, sustainable, and collaborative scientific software ecosystems. To address urgent challenges at the intersection of high-performance computing (HPC), AI, and scientific software, participants envisioned agile, robust ecosystems built through socio-technical co-design—the intentional integration of social and technical components as interdependent parts of a unified strategy. This approach combines advances in AI, HPC, and software with new models for cross-disciplinary collaboration, training, and workforce development. Key recommendations include building modular, trustworthy AI-enabled scientific software systems; enabling scientific teams to integrate AI systems into their workflows while preserving human creativity, trust, and scientific rigor; and creating innovative training pipelines that keep pace with rapid technological change. Pilot projects were identified as near-term catalysts, with initial priorities focused on hybrid AI/HPC infrastructure, cross-disciplinary collaboration and pedagogy, responsible AI guidelines, and prototyping of public-private partnerships. This report presents a vision of next-generation ecosystems for scientific computing where AI, software, hardware, and human expertise are interwoven to drive discovery, expand access, strengthen the workforce, and accelerate scientific progress.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1989

Autonomous sensor-based dual-arm satellite grappling

Dual-arm satellite grappling involves the integration of technologies developed in the Sensing and Perception (S&P) Subsystem for object acquisition and tracking, and the Manipulator Control and Mechanization (MCM) Subsystem for dual-arm control. S&P acquires and tracks the position, orientation, velocity, and angular velocity of a slowly spinning satellite, and sends tracking data to the MCM subsystem. MCM grapples the satellite and brings it to rest, controlling the arms so that no excessive forces or torques are exerted on the satellite or arms. A 350-pound satellite mockup which can spin freely on a gimbal for several minutes, closely simulating the dynamics of a real satellite is demonstrated. The satellite mockup is fitted with a panel under which may be mounted various elements such as line replacement modules and electrical connectors that will be used to demonstrate servicing tasks once the satellite is docked. The subsystems are housed in three MicroVAX II microcomputers. The hardware of the S&P Subsystem includes CCD cameras, video digitizers, frame buffers, IMFEX (a custom pipelined video processor), a time-code generator with millisecond precision, and a MicroVAX II computer. Its software is written in Pascal and is based on a locally written vision software library. The hardware of the MCM Subsystem includes PUMA 560 robot arms, Lord force/torque sensors, two MicroVAX II computers, and unimation pneumatic parallel grippers. Its software is written in C, and is based on a robot language called RCCL. The two subsystems are described and test results on the grappling of the satellite mockup with rotational rates of up to 2 rpm are provided.

Wilcox, Brian↗

NASA NTRS1999

The ISO-IRAS Faint Galaxy Survey

As part of the ISO-IRAS Faint Galaxy Survey ISO Satellite observations of over 600 IRAS sources have been obtained with the ISOCAM instrument. Because our survey strategy involved relatively short integrations, great care was required in developing analysis software including cosmic-ray and transient removal and calibration. These observations have now been through final pipeline processing at IPAC and ground-based follow-up is ongoing. The observations are for sources from two samples: a " Filler' sample selected to be at z greater than 0.1 and a fainter sample which selected for the highest redshift galaxies in the IRAS survey, with redshifts 0.2 less than z less than 1.0. I now have obtained ground-based follow-up spectrophotometry at Lick and Palomar observatories for 100 LFIRGs with 0.1 less than z less than 0.7. Our observations have confirmed that these systems are comparable to nearby LFIRGs such as Arp 220, with L (sub -)(fir) greater than 10(exp 11) L(sub -) sun and typically HII/Liner optical excitation. About 10% of the galaxies show true AGN (Sy2) excitation. Based on our work on a nearby complete sample of LFIRGS, we believe that the majority of these systems are luminous Starbursts, thus this project is tracing the luminous end of the galaxy star-forming luminosity function - the (infrared) star-formation history of the Universe to z approx. 1, a topic of some considerable recent interest. A by-product of these ISOCAM observations is approximately 1 square degree of deep 2 microns pointings outside the IRAS error boxes, allowing us an independent estimate of the mid-infrared log N - log S relation. Ground-based observations of this sample are continuing.

Smith, Harding E.↗

DOE OSTI2024

Introducing NOODLES: Collaborative Visualization in a Flavorful Package

We introduce NOODLES, a lightweight collaborative visualization and analysis protocol. NOODLES was developed to enable new scientific workflows while enhancing existing pipelines. Using a technology-minimal approach to strengthen applicability, this protocol allows researchers to tie disparate software together as well as making domain specific visualizations accessible across various platforms and form factors. In this talk we discuss background, challenges, and gaps in the current tool landscape that lead to the development of the protocol. We then describe the design of the protocol and present several use cases to demonstrate the efficacy of NOODLES in addressing complex visualization challenges and how it co-exists with existing tools. We conclude by outlining in-progress work and potential avenues for future development within the NOODLES framework.

collaborative analysis↗

NASA NTRS1985

A software simulation study of a (255,223) Reed-Solomon encoder-decoder

A set of software programs which simulates a (255,223) Reed-Solomon encoder/decoder pair is described. The transform decoder algorithm uses a modified Euclid algorithm, and closely follows the pipeline architecture proposed for the hardware decoder. Uncorrectable error patterns are detected by a simple test, and the inverse transform is computed by a finite field FFT. Numerical examples of the decoder operation are given for some test codewords, with and without errors. The use of the software package is briefly described.

Pollara, F.↗

DOE OSTI2020

Open Source Platform for Live Processing of High Data Rate Electron Microscopy (SBIR Phase I Award Topic 15, Subtopic B, Final Scientific / Technical Report)

4D scanning transmission electron microscopy (STEM) is experiencing a revolution both in terms of data size, and data veracity, as are many other areas of experimental data acquisition. This presents enormous opportunities to make new scientific discoveries, but is also hamstrung by aging software that is no longer fit for purpose. Experimental data acquisition platforms must make the same investments in software infrastructure that the simulation community made decades ago to take advantage of high-performance computing (HPC) if they hope to realize the full potential of the latest advances made in detector technology. This requires rethinking the design and deployment of hardware, moving away from small, fixed commodity computing capabilities with closed interfaces to open, modular, scalable systems designed for parallelism and throughput. The technical approach will leverage best-in-class open source technologies, and develop new approaches strategically to build an open source platform for experimental data acquisition targeting high data rate electron microscopy. The project will develop a software platform that uses the power and performance of modern HPC resources to ingest the large volume of data produced by electron microscopes using direct electron detectors. A set of configurable pipelines will be used to enable the processing to be tailored to the problem being investigated with builtin visualization of data at interactive rates. This project will leverage investments made by the DOE and other agencies to provide a powerful software platform for the experimental data community working with most powerful electron microscopes, such as 4D STEM detectors. It will create a platform that serves as a reference for the community. Permissive open source licensing will make the platform available to all, with a simple deployment strategy offering the ability to keep data close to where it is generated. This advanced software platform will require customization for each deployment, creating opportunities for software services that are typically offered to extend open source software platforms.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

PtychoShelves , a versatile high-level framework for high-performance analysis of ptychographic data

Over the past decade, ptychography has been proven to be a robust tool for non-destructive high-resolution quantitative electron, X-ray and optical microscopy. It allows for quantitative reconstruction of the specimen's transmissivity, as well as recovery of the illuminating wavefront. Additionally, various algorithms have been developed to account for systematic errors and improved convergence. With fast ptychographic microscopes and more advanced algorithms, both the complexity of the reconstruction task and the data volume increase significantly. PtychoShelves is a software package which combines high-level modularity for easy and fast changes to the data-processing pipeline, and high-performance computing on CPUs and GPUs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Distributed Many-to-Many Protein Sequence Alignment using Sparse Matrices

Identifying similar protein sequences is a core step in many computational biology pipelines such as detection of homologous protein sequences, generation of similarity protein graphs for downstream analysis, functional annotation, and gene location. Performance and scalability of protein similarity search have proven to be a bottleneck in many bioinformatics pipelines due to increase in cheap and abundant sequencing data. This work presents a new distributed-memory software PASTIS. PASTIS relies on sparse matrix computations for efficient identification of possibly similar proteins. We use distributed sparse matrices for scalability and show that the sparse matrix infrastructure is a great fit for protein similarity search when coupled with a fully-distributed dictionary of sequences that allow remote sequence requests to be fulfilled. Our algorithm incorporates the unique bias in amino acid sequence substitution in search without altering basic sparse matrix model, and in turn, achieves ideal scaling up to millions of protein sequences.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

MetaboDirect: an analytical pipeline for the processing of FT-ICR MS-based metabolomic data

Background: Microbiomes are now recognized as the main drivers of ecosystem function ranging from the oceans and soils to humans and bioreactors. However, a grand challenge in microbiome science is to characterize and quantify the chemical currencies of organic matter (i.e., metabolites) that microbes respond to and alter. Critical to this has been the development of Fourier transform ion cyclotron resonance mass spectrometry (FT-ICR MS), which has drastically increased molecular characterization of complex organic matter samples, but challenges users with hundreds of millions of data points where readily available, user-friendly, and customizable software tools are lacking. Results: Here, we build on years of analytical experience with diverse sample types to develop MetaboDirect, an open-source, command-line-based pipeline for the analysis (e.g., chemodiversity analysis, multivariate statistics), visualization (e.g., Van Krevelen diagrams, elemental and molecular class composition plots), and presentation of direct injection high-resolution FT-ICR MS data sets after molecular formula assignment has been performed. When compared to other available FT-ICR MS software, MetaboDirect is superior in that it requires a single line of code to launch a fully automated framework for the generation and visualization of a wide range of plots, with minimal coding experience required. Among the tools evaluated, MetaboDirect is also uniquely able to automatically generate biochemical transformation networks (ab initio) based on mass differences (mass difference network-based approach) that provide an experimental assessment of metabolite connections within a given sample or a complex metabolic system, thereby providing important information about the nature of the samples and the set of microbial reactions or pathways that gave rise to them. Finally, for more experienced users, MetaboDirect allows users to customize plots, outputs, and analyses. Conclusion: Application of MetaboDirect to FT-ICR MS-based metabolomic data sets from a marine phage-bacterial infection experiment and a Sphagnum leachate microbiome incubation experiment showcase the exploration capabilities of the pipeline that will enable the research community to evaluate and interpret their data in greater depth and in less time. It will further advance our knowledge of how microbial communities influence and are influenced by the chemical makeup of the surrounding system. The source code and User’s guide of MetaboDirect are freely available through (https://github.com/Coayala/MetaboDirect) and (https://metabodirect.readthedocs.io/en/latest/), respectively.

54 ENVIRONMENTAL SCIENCES↗