Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

Transparent runtime parallelization of the R scripting language

Scripting languages such as R and Matlab are widely used in scientific data processing. As the data volume and the complexity of analysis tasks both grow, sequential data processing using these tools often becomes the bottleneck in scientific workflows. We describe pR, a runtime framework for automatic and transparent parallelization of the popular R language used in statistical computing. Recognizing scripting languages interpreted nature and data analysis codes use pattern, we propose several novel techniques: (1) applying parallelizing compiler technology to runtime, whole-program dependence analysis of scripting languages, (2) incremental code analysis assisted with evaluation results, and (3) runtime parallelization of file accesses. Our framework does not require any modification to either the source code or the underlying R implementation. Experimental results demonstrate that pR can exploit both task and data parallelism transparently and overall has better performance as well as scalability compared to an existing parallel R package that requires code modification.

99 GENERAL AND MISCELLANEOUS↗

Simulation of Enhanced Geothermal Systems: A Benchmarking and Code Intercomparison Study

Numerical simulation codes have become critical tools for understanding complex geologic processes, as applied to technology assessment, system design, monitoring, and operational guidance. Recently the need for quantitatively evaluating coupled Thermodynamic, Hydrologic, geoMechanical, and geoChemical (THMC) processes has grown, driven by new applications such as geologic sequestration of greenhouse gases and development of unconventional energy sources. Here we focus on Enhanced Geothermal Systems (EGS), which are man-made geothermal reservoirs created where hot rock exists but there is insufficient natural permeability and/or pore fluids to allow efficient energy extraction. In an EGS, carefully controlled subsurface fluid injection is performed to enhance the permeability of pre-existing fractures, which facilitates fluid circulation and heat transport. EGS technologies are relatively new, and pose significant simulation challenges. To become a trusted analytical tool for EGS, numerical simulation codes must be tested to demonstrate that they adequately represent the coupled THMC processes of concern. This presentation describes the approach and status of a benchmarking and code intercomparison effort currently underway, supported by the U. S. Department of Energy’s Geothermal Technologies Program. This study is being closely coordinated with a parallel international effort sponsored by the International Partnership for Geothermal Technology (IPGT). We have defined an extensive suite of benchmark problems, test cases, and challenge problems, ranging in complexity and difficulty, and a number of modeling teams are applying various simulation tools to these problems. The descriptions of the problems and modeling results are being compiled using the Velo framework, a scientific workflow and data management environment accessible through a simple web-based interface.

benchmark↗

Build Less Code, Deliver More Science: An Experience Report on Composing Scientific Environments using Component-based and Commodity Software Platforms

Modern scientific software is daunting in its diversity and complexity. From massively parallel simulations running on the world’s largest supercomputers, to visualizations and user support environments that manage ever growing complex data collections, the challenges for software engineers are plentiful. While high performance simulators are necessarily specialized codes to maximize performance on specific supercomputer architectures, we argue the vast majority of supporting infrastructure, data management and analysis tools can leverage commodity open source and component-based technologies. This approach can significantly drive down the effort and costs of building complex, collaborative scientific user environments, as well as increase their reliability and extensibility. In this paper we describe our experiences in creating an initial user environment for scientists involved in modeling the detailed effects of climate change on the environment of selected geographical regions. Our approach composes the user environment using the Velo scientific knowledge management platform and the MeDICi Integration Framework for scientific workflows. These established platforms leverage component-based technologies and extend commodity open source platforms with abstractions and capabilities that make them amenable for broad use in science. Using this approach we were able to deliver an operational user environment capable of running thousands of simulations in a 7 month period, and achieve significant software reuse.

Gorton, Ian↗

HPC s Pivot to Data

Computer centers such as NERSC and OLCF have traditionally focused on delivering computational capability that enables breakthrough innovation in a wide range of science domains. Accessing that computational power has required services and tools to move the data from input and output to computation and storage. A ''pivot to data'' is occurring in HPC. Data transfer tools and services that were previously peripheral are becoming integral to scientific workflows. Emerging requirements from high-bandwidth detectors, high-throughput screening techniques, highly concur- rent simulations, increased focus on uncertainty quantification, and an emerging open-data policy posture toward published research are among the data-drivers shaping the networks, file systems, databases, and overall compute and data environment. In this paper we explain the pivot to data in HPC through user requirements and the changing resources provided by HPC with particular focus on data movement. For WAN data transfers we present the results of a study of network performance between centers.

97 MATHEMATICS AND COMPUTING↗

F3D Image Processing and Analysis for Many - and Multi-core Platforms

F3D is written in OpenCL, so it achieve[sic] platform-portable parallelism on modern mutli-core CPUs and many-core GPUs. The interface and mechanims to access F3D core are written in Java as a plugin for Fiji/ImageJ to deliver several key image-processing algorithms necessary to remove artifacts from micro-tomography data. The algorithms consist of data parallel aware filters that can efficiently utilizes[sic] resources and can work on out of core datasets and scale efficiently across multiple accelerators. Optimizing for data parallel filters, streaming out of core datasets, and efficient resource and memory and data managements over complex execution sequence of filters greatly expedites any scientific workflow with image processing requirements. F3D performs several different types of 3D image processing operations, such as non-linear filtering using bilateral filtering and/or median filtering and/or morphological operators (MM). F3D gray-level MM operators are one-pass constant time methods that can perform morphological transformations with a line-structuring element oriented in discrete directions. Additionally, MM operators can be applied to gray-scale images, and consist of two parts: (a) a reference shape or structuring element, which is translated over the image, and (b) a mechanism, or operation, that defines the comparisons to be performed between the image and the structuring element. This tool provides a critical component within many complex pipelines such as those for performing automated segmentation of image stacks. F3D is also called a "descendent" of Quant-CT, another software we developed in the past. These two modules are to be integrated in a next version. Further details were reported in: D.M. Ushizima, T. Perciano, H. Krishnan, B. Loring, H. Bale, D. Parkinson, and J. Sethian. Structure recognition from high-resolution images of ceramic composites. IEEE International Conference on Big Data, October 2014.

Source record↗

Introducing Triquetrum, A Possible Future for Kepler and Ptolemy II

Triquetrum is an open platform for managing and executing scientific workflows that is under development as an Eclipse project. Both Triquetrum and Kepler use Ptolemy II as their execution engine. Triquetrum presents opportunities and risks for the Kepler community. The opportunities include a possibly larger community for interaction and a path for Kepler to move from Kepler's one-off ant-based build environment towards a more common OSGi-based environment and a way to maintain a stable Ptolemy II core. The risks include the fact that Triquetrum is a fork of Ptolemy II that would result in package name changes and other possible changes. In addition, Triquetrum is licensed under the Eclipse Public License v1.0, which includes a patent clause that could conflict with the University of California patent clause. This paper describes these opportunities and risks.

Brooks, Christopher↗

Modeling the Virtual Machine Launching Overhead under Fermicloud

FermiCloud is a private cloud developed by the Fermi National Accelerator Laboratory for scientific workflows. The Cloud Bursting module of the FermiCloud enables the FermiCloud, when more computational resources are needed, to automatically launch virtual machines to available resources such as public clouds. One of the main challenges in developing the cloud bursting module is to decide when and where to launch a VM so that all resources are most effectively and efficiently utilized and the system performance is optimized. However, based on FermiCloud’s system operational data, the VM launching overhead is not a constant. It varies with physical resource (CPU, memory, I/O device) utilization at the time when a VM is launched. Hence, to make judicious decisions as to when and where a VM should be launched, a VM launch overhead reference model is needed. The paper is to develop a VM launch overhead reference model based on operational data we have obtained on FermiCloud and uses the reference model to guide the cloud bursting process.

Garzoglio, Gabriele↗

X.509 Authentication/Authorization in FermiCloud

We present a summary of how X.509 authentication and authorization are used with OpenNebula in FermiCloud. We also describe a history of why the X.509 authentication was needed in FermiCloud, and review X.509 authorization options, both internal and external to OpenNebula. We show how these options can be and have been used to successfully run scientific workflows on federated clouds, which include OpenNebula on FermiCloud and Amazon Web Services as well as other community clouds. We also outline federation options being used by other commercial and open-source clouds and cloud research projects.

Kim, Hyunwoo↗

A Case Study of MPI Over Long Distance Connections

Scientific workflows are increasingly being distributed across wide-area networks, and their code executions are expected to span across geographically dispersed computing systems. MPI has been extensively used to support communications for distributed computations, typically, over compute clusters and high-performance systems within a single facility. We present a case study of performance of MPI basic operations over long distance connections, wherein TCP is used for the underlying transport. We present measurements of execution times of MPI codes that utilize MPI Sendrecv operations over emulated 10Gbps connections with 0-366ms round-trip times, including the longest one spanning the globe. They demonstrate that basic MPI codes can be sustained over long distance connections under external packet loss rates up to 10%. They also highlight the qualitative effects of losses which manifest as increased execution times as a consequence of TCP’s loss recovery process.

Rao, Nageswara S.↗

The National Ignition Facility Data Repository

NIF is the world's largest and most energetic laser experimental system, providing a scientific center to study inertial confinement fusion and matter at extreme energy densities and pressures. This presentation discusses the design, architecture, and implementation of the NIF Data Repository (NDR), which provides for the capture and long-term digital storage of peta-scale datasets produced by conducting experimental campaigns. The NDR is a federated database that provides for the capture of: experimental campaign plans, machine configuration & calibration data, raw experimental results and the processed results produced by scientific workflows. The NDR provides for metadata, pedigree, quality, effectivity, versioning and access control for each of the data categories. A critical capability of the NDR is its extensive data provisioning capabilities and protocols that enable scientists, local and remote alike, to review the results of analysis produced by the NDR's analysis pipeline or to download datasets for offline analysis. The NDR provides for the capture of these locally-produced analysis results to enable both peer review and follow-on automated analysis.

42 ENGINEERING↗

hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices

Accessible machine learning algorithms, software, and diagnostic tools for energy-efficient devices and systems are extremely valuable across a broad range of application domains. In scientific domains, real-time near-sensor processing can drastically improve experimental design and accelerate scientific discoveries. To support domain scientists, we have developed hls4ml, an open-source software-hardware codesign workflow to interpret and translate machine learning algorithms for implementation with both FPGA and ASIC technologies. We expand on previous hls4ml work by extending capabilities and techniques towards low-power implementations and increased usability: new Python APIs, quantization-aware pruning, end-to-end FPGA workflows, long pipeline kernels for low power, and new device backends include an ASIC workflow. Taken together, these and continued efforts in hls4ml will arm a new generation of domain scientists with accessible, efficient, and powerful tools for machine-learning-accelerated discovery.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Accelerating Multivariate Functional Approximation Computation with Domain Decomposition Techniques⋆

Modeling large datasets through Multivariate Functional Approximations (MFA) provide an elegant way to handle many visualization and scientific analysis workflows. The process necessitates scalable data partitioning methods to compute MFA representations efficiently without compromising the accuracy or continuity of the reconstructed solution. We propose a domain -decomposed method for computing the MFA with B -spline bases, which reduces the total work per task and uses a restricted Additive Schwarz (RAS) method to converge the control point data degrees -of -freedom along subdomain boundaries. We provide an in-depth analysis of the parallel approach with domain decomposition solvers, aiming to minimize local subdomain error residuals and recover high -order continuity at subdomain interfaces with appropriate choices of knot overlaps. The communication cost, determined by the overlap regions in the RAS implementation, is optimized to recover the numerical error profile of the single subdomain case. Our proposed method stands in contrast to previous methods, which typically only recover either C 0 or at best C 1 continuity for arbitrary B -spline degree expansions, or those that require post -processing to blend discontinuities in the reconstructed data. We demonstrate the effectiveness of our approach using analytical and real -world datasets in 1D, 2D, and 3D through both strong and weak scaling studies. The performance results indicate that the overall cost of computing the approximation is directly proportional to the underlying nearest -neighbor communication implementation, and is only weakly dependent on the overlap region size that determines the size of the messages. This finding underscores the efficiency and scalability of our proposed method, making it a promising solution for handling large datasets in scientific workflows.

additive Schwarz solvers↗

Experiences with user-centered design for the tigres workflow API

Scientific data volumes have been growing exponentially. This has resulted in the need for new tools that enable users to operate on and analyze data. Cyber infrastructure tools, including workflow tools, that have been developed in the last few years has often fallen short if user needs and suffered from lack of wider adoption. User-centered Design (UCD) process has been used as an effective approach to develop usable software with high adoption rates. However, UCD has largely been applied for user-interfaces and there has been limited work in applying UCD to application program interfaces and cyber infrastructure tools. We use an adapted version of UCD that we refer to as Scientist-Centered Design (SCD) to engage with users in the design and development of Tigres, a workflow application programming interface. Tigres provides a simple set of programming templates (e.g., sequence, parallel, split, merge) that can be can used to compose and execute computational and data transformation pipelines. In this paper, we describe Tigres and discuss our experiences with the use of UCD for the initial development of Tigres. Our experience-to-date is that the UCD process not only resulted in better requirements gathering but also heavily influenced the architecture design and implementation details. User engagement during the development of tools such as Tigres is critical to ensure usability and increase adoption.

Ramakrishnan, L↗

Finite elements for Matérn-type random fields: Uncertainty in computational mechanics and design optimization

This work highlights an approach for incorporating realistic uncertainties into scientific computing workflows based on finite elements, focusing on prevalent applications in computational mechanics and design optimization. We leverage Matérn-type Gaussian random fields (GRFs) generated using the SPDE method to model aleatoric uncertainties, including environmental influences, variating material properties, and geometric ambiguities. Our focus lies on delivering practical GRF realizations that accurately capture imperfections and variations and understanding how they impact the predictions of computational models as well as the shape and topology of optimized designs. Here we describe a numerical algorithm based on solving a generalized SPDE to sample GRFs on arbitrary meshed domains. The algorithm leverages established techniques and integrates seamlessly with the open-source finite element library MFEM and associated scientific computing workflows, like those found in industrial and national laboratory settings. Our solver scales efficiently for large-scale problems and supports various domain types, including surfaces and embedded manifolds. We showcase its versatility through biomechanics and topology optimization applications, emphasizing the potential to influence these domains. The flexibility and efficiency of SPDE-based GRF generation empowers us to run large-scale optimization problems on 2D and 3D domains, including finding optimized designs on embedded surfaces, and to generate design features and topologies beyond the reach of conventional techniques. Moreover, these capabilities allow us to model and quantify geometric uncertainties on reconstructed submanifolds, such as the interpolated surfaces of cerebral aneurysms provided by postprocessing CT scans. In addition to offering benefits in these specific domains, the proposed techniques transcend specific applications and generalize to arbitrary forward and backward problems in uncertainty quantification involving finite elements.

97 MATHEMATICS AND COMPUTING↗

Workflows for Science: A comprehensive guide for ensemble workflow tools usage with applications on OLCF systems

The growing demand for robust computational and workflow environments for scientific applications and user communities at the Oak Ridge Leadership Computing Facility (OLCF) has prompted collaboration with ensemble tools development teams and facility users to produce this technical paper. We connect science applications to the RADICAL-Pilot (RP) workflow tool to execute ensemble instantiations using the Frontier supercomputer. The documented installation, usage, and execution demonstrates how RP streamlines scientific workflows at OLCF. We outline the specific steps OLCF users can follow to integrate this tool with their applications and advance their research. This document stands as a comprehensive guide to OLCF users of ensemble workflow tools with examples on real applications using the Frontier supercomputer.

97 MATHEMATICS AND COMPUTING↗

Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.

Etz, Brian [ORNL] (ORCID:0000000208554863)↗

Data-intensive management and analysis for scientifc simulations.

Scientific simulations can produce enormous amounts of data, making the analysis of results and management of files a difficult task for scientists. The simulation management and analysis system (Smaash) described here is designed to allow scientists to easily capture, store, organize, monitor, and analyze simulation results. The system is automatic, standardized, and secure. Smaash was built using open-source tools and modularized to be independent of the scientific simulation. The web-based front-end allows the scientist to easily interact with the data, and has proved its usefulness in improving the efficiency of a scientific team's workflow. High performance parallel computing allows scientists to solve complex physical problems through computer simulation. However, the massive amounts of data generated and the complex computing environment can create additional complications. A recent review by Ludaescher et al.(2009) describes how scientific workflows can assist scientists in extracting knowledge from these data-intensive operations by automating components within pipelines. Within the fusion community, Klasky et al.(2008) and colleagues have developed a system that handles the storage management, data movement, metadata generation and management, and a means to analyze the results. In response to scientists needs, a simulation management and analysis system (Smaash) was developed at the University of Chicago and Argonne National Laboratory (USA). Smaash provides an integrated way to monitor simulations and analyze computational results; catalog, store, and retrieve simulations; and prepare output for publications. The system is independent of the particular simulation code, accessible from many HPC and browser-based platforms, and built around open-source software tools. Data security and provenance is considered throughout. The analysis components are hidden behind a web-based front end, enabling scientists to focus on their results and not get bogged down by information overload.

99 GENERAL AND MISCELLANEOUS↗