Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributed workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

Deep-learning-based canopy height model generation from sub-meter resolution panchromatic satellite imagery

Canopy height models (CHMs) with sufficient resolution to distinguish individual trees are useful for a variety of applications. However, standard techniques to acquire such data, such as airborne lidar surveying, are often prohibitively expensive. Deep learning techniques for generating CHMs from high-resolution imagery are an attractive option to reduce costs. To date, success with these methods has been demonstrated using multichannel aerial photography and specialized satellite data products derived from multiple sensors, neither of which is commonly available at temporal resolutions finer than one year. Here we demonstrate a method to generate sub-meter resolution CHMs in three forests in California using a more abundant data source: sub-meter resolution, panchromatic satellite imagery from a single sensor. We show that phenology and species composition play important roles in model transferability; when trained using imagery from a single conifer forest in autumn, the model performs well on autumn imagery from a second conifer forest several hundred kilometers distant with no re-training. With modest additions to the training dataset, the same model generates minimally biased estimates of canopy height in both conifer and deciduous forests during multiple seasons. Because the model operates on satellite data with global coverage and a relatively short return interval, we propose its suitability to extrapolate tree-level canopy height data to remote regions and conduct high-temporal resolution monitoring of forest structure. We furthermore demonstrate the workflow’s applicability to fire modeling by conducting simulations in forests populated by trees measured using both this approach and airborne lidar surveying. We find minimal differences in fire behavior relative to a baseline case in which only statistical distributions of tree height and crown area are known. This result underscores the value of forest structural information derived from our workflow for improving the fidelity of wildland fire simulations, among other ecological applications.

54 ENVIRONMENTAL SCIENCES↗

A Scientific Data Provenance Harvester for Distributed Applications

Data provenance provides a way for scientists to observe how experimental data originates, conveys process history, and explains influential factors such as experimental rationale and associated environmental factors from system metrics measured at runtime. The US Department of Energy Office of Science Integrated end-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) project has developed a provenance harvester that is capable of collecting observations from file based evidence typically produced by distributed applications. To achieve this, file based evidence is extracted and transformed into an intermediate data format inspired in part by W3C CSV on the Web recommendations, called the Harvester Provenance Application Interface (HAPI) syntax. This syntax provides a general means to pre-stage provenance into messages that are both human readable and capable of being written to a provenance store, Provenance Environment (ProvEn). HAPI is being applied to harvest provenance from climate ensemble runs for Accelerated Climate Modeling for Energy (ACME) project funded under the U.S. Department of Energy’s Office of Biological and Environmental Research (BER) Earth System Modeling (ESM) program. ACME informally provides provenance in a native form through configuration files, directory structures, and log files that contain success/failure indicators, code traces, and performance measurements. Because of its generic format, HAPI is also being applied to harvest tabular job management provenance from Belle II DIRAC scheduler relational database tables as well as other scientific applications that log provenance related information.

Stephan, Eric G.↗

Genarris 2.0: A Random Structure Generator for Molecular Crystals

Genarris is an open source Python package for generating random molecular crystal structures with physical constraints for seeding crystal structure prediction algorithms and training machine learning models. Here we present a new version of the code, containing several major improvements. A MPI-based parallelization scheme has been implemented, which facilitates the seamless sequential execution of user-defined workflows. A new method for estimating the unit cell volume based on the single molecule structure has been developed using a machine-learned model trained on experimental structures. A new algorithm has been implemented for generating crystal structures with molecules occupying special Wyckoff positions. A new hierarchical structure check procedure has been developed to detect unphysical close contacts efficiently and accurately. New intermolecular distance settings have been implemented for strong hydrogen bonds. To demonstrate these new features, we study two specific cases: benzene and glycine. Genarris finds the experimental structures of the two polymorphs of benzene and the three polymorphs of glycine. Program summary Program Title: Genarris 2.0 Program Files doi: http://dx.doi.org/10.17632/grx6mz4pjn.1 Licensing provisions: BSD-3 Clause Programming language: Python, C External routines/libraries: Spglib, ASE, pymatgen, SciPy, mpi4py, scikit-learn, PyTorch, FHI-aims. Nature of problem: Molecular crystal structure prediction. Solution method: Genarris 2.0 generates molecular crystal structures over the 230 space groups, on general and special Wyckoff positions, using physical constraints. Down-sampling of the generated structures may be performed subsequently, based on molecular crystal packing descriptors and an unsupervised machine learning algorithm. Lastly, ab initio structure relaxation may be performed for the final pool. Depending on the user-defined workflow implemented, Genarris may be used to generate diverse molecular crystal datasets to seed evolutionary algorithms or to train machine learning algorithms or as a standalone crystal structure prediction method. Restrictions: For crystal structure generation, the molecule of interest must be semi-rigid with no bond rotational degrees of freedom. Unusual features: Genarris 2.0 is a highly distributed program, making use of MPI for Python parallelization. The user has the ability to design and implement workflows by executing a user-defined list of procedures. Genarris 2.0 offers new features including a machine learning model for estimating the molecular volume in the solid state from the single molecule structure, structure generation in special Wyckoff positions of space groups, hierarchical structure checks including rigorous treatment of non-orthogonal structures, and clustering and down-selection workflows combining first principles simulations with machine learning. (C) 2020 Elsevier B.V. All rights reserved.

Crystal structure prediction↗

Data transfer for STAR grid jobs

The Solenoidal Tracker at RHIC (STAR) is a multipurpose experiment at the Relativistic Heavy Ion Collider (RHIC) with the primary goal to study the formation and properties of the quark-gluon plasma. STAR is an international collaboration of member institutions and laboratories from around the world. Yearly data-taking period produces PBytes of raw data collected by the experiment. STAR primarily uses its dedicated facility at BNL to process this data, but has routinely leveraged distributed systems, both high throughput (HTC) and high performance (HPC) computing clusters, to significantly augment the processing capacity available to the experiment. The ability to automate the efficient transfer of large data sets on reliable, scalable, and secure infrastructure is critical for any large-scale distributed processing campaign. For more than a decade, STAR computing has relied upon GridFTP with its x509-based authentication to build such data transfer systems and integrate them into its larger production workflow. The end of support by the community for both GridFTP and the x509 standard requires STAR to investigate other approaches to meet its distributed processing needs. In this study we investigate two multi-purpose data distribution systems, Globus.org and XRootD, as alternatives to GridFTP. We compare both their performance and the ease by which each service is integrated into the type of secure and automated data transfer systems STAR has previously built using GridFTP. The presented approach and study may be applicable to other distributed data processing use cases beyond STAR.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

TU-AB-303-03: CBCT-Based On-Site Simulation, Planning, and Delivery (OSPD) for Whole Brain Radiotherapy

Purpose: To demonstrate the feasibility of a CBCT-based on-site simulation, planning, and delivery (OSPD) for whole brain radiotherapy, in which all steps from imaging, planning to treatment delivery are performed at the treatment unit in one appointment time slot. This work serves as the proof of concept for future OSPD single fraction radiation therapy. Methods: An integrated on-site imaging, planning and delivery workflow was developed and tested for whole brain radiotherapy. An automated two-opposed-oblique-beam plan is created by utilizing the treatment planning system scripting and simple field-in-field IMRT. The IMRT plan is designed with maximum 8 control points to cover the target volume consisting of the brain to C1/C2 of the spinal cord, with dose homogeneity criteria from −5% to +7% of the prescription dose. Due to inaccuracy of reconstructed Hounsfield unit numbers in CBCT images, the dose distribution is calculated with non-heterogeneity correction introducing only clinically insignificant dose discrepancy. A coherent and synchronized workflow was designed for a team of attending physician, physicist, therapists, and dosimetrist to work closely with the ability to quickly modify, approve, and implement the treatment. Results: Thirty-one patients have been treated with this OSPD treatment, without compromising the plan quality compared to our regular clinically used parallel apposed 2D plans. The average time for these procedures are 48.02 ±11.55 minutes from the time patient entered the treatment room until s/he exited, and 35.09 ±10.35 minutes from starting CBCT until last beam delivered. This time duration is comparable to the net time when individual tasks are summed up during our regular CT- based whole brain planning and delivery. Conclusions: The OSPD whole brain treatment has been tested to be clinically feasible. The next step is to further improve the efficiency and to streamline the workflow. Other disease sites will be also tested with this new technology.

60 APPLIED LIFE SCIENCES↗

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗

interactEM v1.0

An interactive, container-based workflow tool for creating and spawning directed acyclic graphs (DAGs) of operators in a distributed environment. It has a microservices architecture, and flow-based programming model. Current tools like this do not enable streaming of data directly between operators.

Welborn, Sam [Lawrence Berkeley National Laborator↗

Implementing efficient data compression and encryption in a persistent key-value store for HPC

Recently, persistent data structures, like key-value stores (KVSs), which are stored in a high-performance computing (HPC) system’s nonvolatile memory, provide an attractive solution for a number of emerging challenges like limited I/O performance. Data compression and encryption are two well-known techniques for improving several properties of such data-oriented systems. This article investigates how to efficiently integrate data compression and encryption into persistent KVSs for HPC with the ultimate goal of hiding their costs and complexity in terms of performance and ease of use. Our compression technique exploits deep memory hierarchy in an HPC system to achieve both storage reduction and performance improvement. Our encryption technique provides a practical level of security and enables sharing of sensitive data securely in complex scientific workflows with nearly imperceptible cost. We implement the proposed techniques on top of a distributed embedded KVS to evaluate the benefits and costs of incorporating these capabilities along different points in the dataflow path, illustrating differences in effective bandwidth, latency, and additional computational expense on Swiss National Supercomputing Centre’s Grand Tavé and National Energy Research Scientific Computing Center’s Cori.

97 MATHEMATICS AND COMPUTING↗

Working with Workflows: Highlights from 5 years Building Scientific Workflows

In 2006, the SciDAC Scientific Data Management (SDM) Center proposed to continue its work deploying leading edge data management and analysis capabilities to scientific applications. One of three thrust areas within the proposed center was focused on Scientific Process Automation (SPA) using workflow technology. As a founding member of the Kepler consortium [LAB+09], the SDM Center team was well positioned to begin deploying workflows immediately. We were also keenly aware of some of the deficiencies in Kepler when applied to high performance computing workflows, which allowed us to focus our research and development efforts on critical new capabilities which were ultimately integrated into the Kepler open source distribution, benefiting the entire community. Significant work was required to ensure Kepler was capable of supporting large-scale production runs for SciDAC applications. Our work on generic actors and templates have improved the portability of workflows across machines and provided a higher level of abstraction for workflow developers. Fault tolerance and provenance tracking were obvious areas for improvement within Kepler given the longevity and complexity of our target workflows. To monitor workflow execution, we developed and deployed a web-based dashboard. We then generalized this interface and released it so it could be deployed at other locations. Outreach has always been a primary focus of our work and we had many successful deployments across a number of scientific domains while continually publishing and presenting our work. This short paper describes our most significant accomplishments over the past 5 years. Additional information about the SDM Center can be found in the companion paper: The Scientific Data Management Center: Available Technologies and Highlights.

99 GENERAL AND MISCELLANEOUS↗

Synthetic Soil Aggregates: Bioprinted Habitats for High-Throughput Microbial Metaphenomics

The dynamics of microbial processes are difficult to study in natural soil, owing to the small spatial scales on which microorganisms operate and to the opacity and chemical complexity of the soil habitat. To circumvent these challenges, we have created a 3D-bioprinted habitat that mimics aspects of natural soil aggregates while providing a chemically defined and translucent alternative culturing method for soil microorganisms. Our Synthetic Soil Aggregates (SSAs) retain the porosity, permeability, and patchy resource distribution of natural soil aggregates—parameters that are expected to influence emergent microbial community interactions. We demonstrate the printability and viability of several different microorganisms within SSAs and show how the SSAs can be integrated into a multi-omics workflow for single SSA resolution genomics, metabolomics, proteomics, lipidomics, and biogeochemical assays. We study the impact of the structured habitat on the distribution of a model co-culture microbial community and find that it is significantly different from the spatial organization of the same community in liquid culture, indicating a potential for SSAs to reproduce naturally occurring emergent community phenotypes. The SSAs have the potential as a tool to help researchers quantify microbial scale processes in situ and achieve high-resolution data from the interplay between environmental properties and microbial ecology.

59 BASIC BIOLOGICAL SCIENCES↗

Architectures Toward Reusable Science Data Systems

Science Data Systems (SDS) comprise an important class of data processing systems that support product generation from remote sensors and in-situ observations. These systems enable research into new science data products, replication of experiments and verification of results. NASA has been building systems for satellite data processing since the first Earth observing satellites launched and is continuing development of systems to support NASA science research and NOAAs Earth observing satellite operations. The basic data processing workflows and scenarios continue to be valid for remote sensor observations research as well as for the complex multi-instrument operational satellite data systems being built today. System functions such as ingest, product generation and distribution need to be configured and performed in a consistent and repeatable way with an emphasis on scalability. This paper will examine the key architectural elements of several NASA satellite data processing systems currently in operation and under development that make them suitable for scaling and reuse. Examples of architectural elements that have become attractive include virtual machine environments, standard data product formats, metadata content and file naming, workflow and job management frameworks, data acquisition, search, and distribution protocols. By highlighting key elements and implementation experience we expect to find architectures that will outlast their original application and be readily adaptable for new applications. Concepts and principles are explored that lead to sound guidance for SDS developers and strategists.

Data Processing↗

Data management and processing workflow for the Materials Physics and Engineering group beamlines at the Advanced Photon Source

The ability to store, organize, process and distribute experimental data effectively, efficiently and securely is particularly important for large user facilities like the Advanced Photon Source. In this article, the deployment of the APS Data Management System (DM) at the 1-ID and 6-BM beamlines of the APS is described. These two beamlines support a wide range of experimental techniques and generate data at relatively high rates, making them ideal candidates to illustrate the deployment and customization of the DM system and its tools. Using several usage examples at these beamlines, various capabilities of the DM system are described.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

MITRE Domain Specific Language (DSL) for synthetic biology workflows (CRADA Final Report)

MITRE is currently developing BioNet, a network designed to facilitate the work of biologist collaborators that are distributed across multiple organizations. BioNet is envisaged to break down traditional barriers in biology, allowing for an integrated, service-based approach to projects which can utilize expertise from any participating entity. This disaggregation fosters innovation by enabling contributions from multiple sources. The public will benefit from the development of the BioNet (to which this project contributes), in that this fostered innovation could positively contribute to our economy.

59 BASIC BIOLOGICAL SCIENCES↗

Data Services for Visualization and Analysis - ASC Level II Milestone (7186)

A new in transit Data Service is presented and compared to the traditional file-based workflow and the newly refactored in situ Catalyst workflow. Each workflow is enabled by the IOSS mesh interface equipped with data management layers for Exodus and CGNS (file-based), Catalyst (in situ), and FAODEL (in transit). FAODEL is a distributed object store that can transmit data across MPI allocations. Catalyst is a Para View-based visualization capability developed as part of the CSSE Data Services effort. The workflows considered here take SPARC data into Catalyst for visualization post-processing. Although still in unoptimized form, we show that the in transit approach is a viable alternative to file-based and in situ workflows and offers several advantages to both simulation and post-processing developers. Since IOSS is a mature interface with wide adoption across Sandia and externally, each workflow can be reconfigured to use different simulations that generate mesh data and post-processing tools that consume it.

97 MATHEMATICS AND COMPUTING↗

SU-F-J-110: MRI-Guided Single-Session Simulation, Online Adaptation, and Treatment

Purpose: To develop a combined simulation and treatment workflow for MRI-guided radiation therapy using the ViewRay treatment planning and delivery system. Methods: Several features of the ViewRay MRIdian planning and treatment workflows are used to simulate and treat patients that require emergent radiotherapy. A simple “pre-plan” is created on diagnostic imaging retrieved from radiology PACS, where conformal fields are created to target a volume defined by a physician based on review of the diagnostic images and chart notes. After initial consult in radiation oncology, the patient is brought to the treatment room, immobilized, and imaged in treatment position with a volumetric MR. While the patient rests on the table, the pre-plan is applied to the treatment planning MR and dose is calculated in the treatment geometry. After physician review, modification of the plan may include updating the target definition, redefining fields, or re-balancing beam weights. Once an acceptable treatment plan is finalized and approved, the patient is treated. Results: Careful preparation and judicious choices in the online planning process allow conformal treatment plans to be created and delivered in a single, thirty-minute session. Several advantages have been identified using this process as compared to conventional urgent CT simulation and delivery. Efficiency gains are notable, as physicians appreciate the predictable time commitment and patient waiting time for treatment is decreased. MR guidance in a treatment position offers both enhanced contrast for target delineation and reduction of setup uncertainties. The MRIdian system tools designed for adaptive radiotherapy are particularly useful, enabling plan changes to be made in minutes. Finally, the resulting plans, typically 6 conformal beams, are delivered as quickly as more conventional AP/PA beam arrangements with comparatively superior dose distributions. Conclusion: The ViewRay treatment planning software and delivery system can accommodate a fast simulation and treatment workflow.

60 APPLIED LIFE SCIENCES↗

Machine learning for automated experimentation in scanning transmission electron microscopy

Abstract Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centric experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows, as well as the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation.

36 MATERIALS SCIENCE↗

A Step-by-Step Protocol from METASPACE to Biological Interpretation

Mass spectrometry imaging (MSI) represents an exceptional tool for exploring complex biological systems spatially at the molecular level. However, due to its multidimensional nature and large-scale data output, it presents considerable challenges when it comes to extracting meaningful biological insights. Recent advancements, such as the METASPACE platform, have enabled researchers to efficiently process, annotate, and interpret MSI datasets by leveraging machine learning and cloud-based infrastructure. In this tutorial, we present a detailed and user-friendly R-pipeline designed to help METASPACE users navigate untargeted metabolomic annotations and transform them into practical insights about their biological systems. By combining METASPACE annotations with rapid R-based screening, this workflow not only streamlined the analytical process but also enhanced the understanding of spatial molecular distribution, especially for complex systems. Here, this easy-to-follow approach has the potential for applications in diagnostics, drug discovery, environmental and ecological processes, and more. We envision this pipeline to be particularly useful for newcomers to the field of MSI and

Moreno Pedraza, Abigail↗

Seeking Repeating Anthropogenic Seismic Sources: Implications for Seismic Velocity Monitoring at Fault Zones

Abstract Seismic velocities in rocks are highly sensitive to changes in permanent deformation and fluid content. The temporal variation of seismic velocity during the preparation phase of earthquakes has been well documented in laboratories but rarely observed in nature. It has been recently found that some anthropogenic, high‐frequency (>1 Hz) seismic sources are powerful enough to generate body waves that travel down to a few kilometers and can be used to monitor fault zones at seismogenic depth. Anthropogenic seismic sources typically have fixed spatial distribution and provide new perspectives for velocity monitoring. In this work, we propose a systematic workflow to seek such powerful seismic sources in a rapid and straightforward manner. We tackle the problem from a statistical point of view, considering that persistent, powerful seismic sources yield highly coherent correlation functions (CFs) between pairs of seismic sensors. The algorithm is tested in California and Japan. Multiple sites close to fault zones show high‐frequency CFs stable for an extended period of time. These findings have great potential for monitoring fault zones, including the San Jacinto Fault and the Ridgecrest area in Southern California, Napa in Northern California, and faults in central Japan. However, extra steps, such as beamforming or polarization analysis, are required to determine the dominant seismic sources and study the source characteristics, which are crucial to interpreting the velocity monitoring results. Train tremors identified by the present approach have been successfully used for seismic velocity monitoring of the San Jacinto Fault in previous studies.

58 GEOSCIENCES↗