Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “robust execution”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

PRIMA-X Final Report

The PRIMA-X (Performance Retargeting of Instrumentation, Measurement, and Analysis Technologies for Exascale Computing) project is the successor of the DOE PRIMA (Performance Refactoring of Instrumentation, Measurement, and Analysis Technologies for Petascale Computing) project, which addressed the challenge of creating a core measurement infrastructure that would serve as a common platform for both integrating leading parallel performance systems (notably TAU and Scalasca) and developing next-generation scalable performance tools. The PRIMA-X project shifts the focus away from refactorization of robust performance tools towards a re-targeting of the parallel performance measurement and analysis architecture for extreme scales. The massive concurrency, asynchronous execution dynamics, hardware heterogeneity, and multi-objective prerequisites (performance, power, resilience) that identify exascale systems introduce fundamental constraints on the ability to carry forward existing performance methodologies. In particular, there must be a deemphasis of per-thread observation techniques to significantly reduce the otherwise unsustainable flood of redundant performance data. Instead, it will be necessary to assimilate multi-level resource observations into macroscopic performance views, from which resilient performance metrics can be attributed to the computational features of the application. This requires a scalable framework for node-level and system-wide monitoring and runtime analyses of dynamic performance information. Also, the interest in optimizing parallelism parameters with respect to performance and energy drives the integration of tool capabilities in the exascale environment further. Initially, PRIMA-X was a collaborative project between the University of Oregon (lead institution) and the German Research School for Simulation Sciences (GRS). Because Prof. Wolf, the PI at GRS, accepted a position as full professor at Technische Universität Darmstadt (TU Darmstadt) starting February 1st, 2015, the project ended at GRS on January 31st, 2015. This report reflects the work accomplished at GRS until then. The work of GRS is expected to be continued at TU Darmstadt. The first main accomplishment of GRS is the design of different thread-level aggregation techniques. We created a prototype capable of aggregating the thread-level information in performance profiles using these techniques. The next step will be the integration of the most promising techniques into the Score-P measurement system and their evaluation. The second main accomplishment is a substantial increase of Score-P’s scalability, achieved by improving the design of the system-tree representation in Score-P’s profile format. We developed a new representation and a distributed algorithm to create the scalable system tree representation. Finally, we developed a lightweight approach to MPI wait-state profiling. Former algorithms either needed piggy-backing, which can cause significant runtime overhead, or tracing, which comes with its own set of scaling challenges. Our approach works with local data only and, thus, is scalable and has very little overhead.

97 MATHEMATICS AND COMPUTING↗

PRIMA-X - Performance Retargeting of Instrumentation, Measurement, and Analysis Technologies for Exascale Computing

The PRIMA-X (Performance Retargeting of Instrumentation, Measurement, and Analysis Technologies for Exascale Computing) project is the successor of the DOE PRIMA (Performance Refactoring of Instrumentation, Measurement, and Analysis Technologies for Petascale Computing) project, which addressed the challenge of creating a core measurement infrastructure that would serve as a common platform for both integrating leading parallel performance systems (notably TAU and Scalasca) and developing next-generation scalable performance tools. The PRIMA-X project shifts the focus away from refactorization of robust performance tools towards a re-targeting of the parallel performance measurement and analysis architecture for extreme scales. The massive concurrency, asynchronous execution dynamics, hardware heterogeneity, and multi-objective prerequisites (performance, power, resilience) that identify exascale systems introduce fundamental constraints on the ability to carry forward existing performance methodologies. In particular, there must be a deemphasis of per-thread observation techniques to significantly reduce the otherwise unsustainable flood of redundant performance data. Instead, it will be necessary to assimilate multi-level resource observations into macroscopic performance views, from which resilient performance metrics can be attributed to the computational features of the application. This requires a scalable framework for node-level and system-wide monitoring and runtime analyses of dynamic performance information. Also, the interest in optimizing parallelism parameters with respect to performance and energy drives the integration of tool capabilities in the exascale environment further. Initially, PRIMA-X was a collaborative project between the University of Oregon (lead institution) and the German Research School for Simulation Sciences (GRS). Because Prof. Wolf, the PI at GRS, accepted a position as full professor at Technische Universitaet Darmstadt (TU Darmstadt) starting February 1st, 2015, the work of GRS was continued at TU Darmstadt. This report covers the work at TU Darmstadt after the transition. The first main accomplishment of TU Darmstadt is the development of different techniques to aggregate performance data on the level of threads. TU Darmstadt evaluated different schemes designed at GRS during the first phase of this project and integrated them into Score-P, a widely used performance measurement framework for HPC applications. The second main accomplishment is a substantial increase of Score-P’s scalability, achieved by improving its internal representation of the underlying system. Third, we extended our performance-modeling tool Extra-P to automatically create models with more than one input parameter (e.g., process count and input size), which was previously impossible. These models allow the performance behavior of an application to be extrapolated to larger machines. Specific applications of the new multi-parameter capability realized in this project include modeling the isoefficiency function of task-based applications and a novel co-design methodology. Further extensions of Extra-P comprise online modeling functionality, automatic refinement of the model search space, and segmented performance modeling. Further contributions include new locality metrics for both shared- and distributed-memory applications, a new way of measuring the impact of inter-application inference on the execution time of an application, and finally an automatic method that selects appropriate OpenMP constructs for the parallelization of sequential code and classifies the affected variables based on their sharing semantics.

97 MATHEMATICS AND COMPUTING↗

DOE BSSD Performance Management Metrics Report Q2

The vision of the National Microbiome Data Collaborative (NMDC) centers on the concept of connecting data, people, and ideas to advance microbiome innovation and discovery. Building data infrastructure, while key to NMDC’s ability to execute on our vision, can only go so far in creating scientific impact. By fostering strong community partnerships and developing a set of robust community outreach and training programs, we are able to turn our products – the Submission Portal, NMDC EDGE, and the Data Portal – into tools that empower the scientific community. Our multi-pronged community building approach spans individual researchers, research teams, consortia and scientific societies, and institutions and federal agencies. To foster a collaborative and inclusive community-centered environment, we have identified three strategic objectives to promote an inclusive and connected community: (1) recognize and support the diverse research needs and perspectives of the microbiome research community; (2) promote best practices across the microbiome community, from researchers to funders, through community-driven practices (FAIR, CARE, and TRUST); and (3) build a microbiome ecosystem that enables scientific discovery and innovation across stakeholders. These strategic objectives allow our team to focus on impact across a diverse range of activities, from launching the American Society for Microbiology (ASM) Microbiome Data Prize to supporting the Ambassador and Champions programs fostering learning and building a collaborative network. We broadly communicate our work through social media (X/Twitter, LinkedIn, and Instagram), The Microbiome Standard (our quarterly newsletter), and Annual Reports. All our work is underpinned by a strong commitment to diversity, equity, and inclusion as articulated in our Action Plan that tracks progress towards key metrics. A core component of our engagement strategy is user research. User research ensures the Submission Portal, NMDC EDGE, Data Portal, and the new Field Notes mobile app are designed with and for the scientific community. Our user research efforts consist of asking researchers exploratory questions to collect information on researcher priorities, methodologies, and perceptions to ensure that we are aware of the current state of microbiome research. Our usability testing provides researchers with prototypes or test environments of the NMDC products, and we capture valuable information on how users interact with the products to make improvements. Given the diverse nature of microbiome work, we acknowledge that we are not aware of all pressing data challenges and thus rely on the research community to help us identify the most important issues to prioritize. To date, we have conducted 24 interviews and one beta-testing call with 10 participants across all NMDC products, which have generated 321 insights and 120 action items. Herein, we describe the ways we engage with the microbiome research community to advance the NMDC mission.

59 BASIC BIOLOGICAL SCIENCES↗

Capability Building Progression of an Insider Threat Mitigation Program at an International Research Reactor

The nuclear industry recognizes the difficulties involved in developing effective managerial and leadership skills in a highly technical and proficient workforce such as that found in nuclear facilities. Implementing an insider threat mitigation program (ITMP) within the nuclear industry is a complex and ongoing process that demands a comprehensive understanding of human behavior, an organization’s security culture, and rigorous regulatory requirements yet also accounts for facility characteristics, physical security, material flow, and activities involving nuclear material. Given the high-consequence nature of research reactor operations, even minor lapses can lead to safety, security, and reputational risks. An effective ITMP requires a defense-in-depth approach that incorporates behavioral analysis, robust vetting procedures, continuous monitoring, and cross-disciplinary coordination. It must also promote a culture of vigilance and accountability at all levels up to and including executive leadership but be flexible enough to adapt to evolving global threats and technological advances. Insider threat mitigation is not a one-time effort but rather a sustained commitment to excellence in safety and security. Establishing a culture in which personnel proactively report incidents and issues that could affect nuclear safety and security is vital to maintaining a safe and secure operational environment. This document was developed to guide senior management and research reactor organizations in creating comprehensive programs to effectively manage and mitigate insider threat behaviors and actions. It focuses on the key pillars of an effective ITMP, including the national legal framework, security culture, preventive and protective measures, cyber security, and performance evaluation. By using a systematic approach during implementation, facilities can foster environments conducive to insider threat detection and support long-term program sustainability. The document also provides strategies for improving communication across all levels of an organization, helping to eliminate barriers that hinder the development of robust ITMPs and enhance overall security culture. In today’s organizations, the concept of leveraging safety and security culture lessons to facilitate knowledge transfer is rapidly evolving to expedite insider threat management and security culture improvements. This document outlines the rationale for evaluating an ITMP based on national customs, culture, and stakeholders. The elements are all germane to reliability and trustworthiness and relate to security concerns that states may encounter. The document focuses not only on individual perceptions regarding security issues and capability building but also on team building and how to resolve concerns. The implementers of a facility’s ITMP may zero in on indicators of insider threats within their enterprise. This material will benefit organizations when it is applied using a systematic and structured approach as demonstrated throughout the document.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

PolyCheck: Dynamic Verification of Iteration Space Transformations on Affine Programs

High-level compiler transformations, especially loop transformations, are widely recognized as critical optimizations to restructure programs to improve data locality and expose parallelism. Guaranteeing the correctness of program transformations is essential, and to date three main approaches have been developed: proof of equivalence of affine programs, matching the execution traces of programs, and checking bit-by-bit equivalence of the outputs of the programs. Each technique suffers from limitations in either the kind of transformations supported, space complexity, or the sensitivity to the testing dataset. In this paper, we take a novel approach addressing all three limitations to provide an automatic bug checker to verify any iteration reordering transformations on affine programs, including non-affine transformations, with space consumption proportional to the original program data, and robust to arbitrary datasets of a given size. We achieve this by exploiting the structure of affine program control- and data-flow to generate at compile-time lightweight checker code to be executed within the transformed program. Experimental results assess the correctness and effectiveness of our method, and its increased coverage over previous approaches.

affine programs↗

Accelerating error correction in tomographic reconstruction

Abstract Spurred by recent advances in detector technology and X-ray optics, upgrades to scanning-probe-based tomographic imaging have led to an exponential growth in the amount and complexity of experimental data and have created a clear opportunity for tomographic imaging to approach single-atom sensitivity. The improved spatial resolution, however, is highly susceptible to systematic and random experimental errors, such as center of rotation drifts, which may lead to imaging artifacts and prevent reliable data extraction. Here, we present a model-based approach that simultaneously optimizes the reconstructed specimen and sinogram alignment as a single optimization problem for tomographic reconstruction with center of rotation error correction. Our algorithm utilizes an adaptive regularizer that is dynamically adjusted at each alternating iteration step. Furthermore, we describe its implementation in a software package targeting high-throughput workflows for execution on distributed-memory clusters. We demonstrate the performance of our solver on large-scale synthetic problems and show that it is robust to a wide range of noise and experimental drifts with near-ideal throughput.

Ali, Sajid (ORCID:0000000321864636)↗

A neural networks framework for real-time unfolding of neutron spectroscopic data at JET

A determination of fast ion population parameters such as intensity and kinetic temperature is important for fusion reactors. This becomes more challenging with finer time resolution of the measurements, since the limited data in each time slice cause increasing statistical variations in the data. This paper describes a framework using Bayesian-regularized neural networks (NNs) designed for such a task. The method is applied to the TOFOR 2.5 MeV fusion neutron spectrometer at JET. NN training data are generated by random sampling of variables in neutron spectroscopy models. Ranges and probability distributions of the parameters are chosen to match the experimental data. Results have shown good performance both on synthetic and experimental data. The latter was assessed by statistical considerations and by examining the robustness and time consistency of the results. The regularization of the training algorithm allowed for higher time resolutions than simple forward methods. The fast execution time makes this approach suitable for real-time analysis with a time resolution limit in the microsecond time scale.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Weapon Material Program: FY24 Annual Report

The Weapon Material Program (WMP) framework was started in July of Fiscal Year (FY) 2022, along with other Mission Support organizations. The first year was about building the organization with the right people, identifying needs, and developing a vision for expectations. It was also a year to move forward with new equipment and technology for weapon material processes. With numerous improvements, WMP is building world-class systems for production, warehousing, and analytical testing. FY 2023 was a year of development. The WMP was a new organization defining its identity, its focus, and its mission. At the end of FY 2023, WMP evaluated its current position and its immediate needs along with its future goals. The entire organization set out on this FY 2024 journey to success. The lack of funding, staffing, and visibility dramatically impacted weapon material operations, testing, and qualifications. This neglect required numerous areas to be addressed and improvement plans to be developed and implemented. WMP processes, procedures, and overall business operations were dramatically in need of updating, revision, and formal documentation. This included all areas: technology, equipment, facility modifications/upgrades, and testing improvements. To ensure organizational improvement and forward momentum in FY 2024, WMP executed a multi-faceted strategic vision. Action items included the following: • Filling open positions with capable personnel who would contribute to a more robust organization • Establishing more formal operations and problem-solving techniques • Improving departmental procedures to align with expanding scope • Improving training • Establishing collaboration meetings with other departments/organizations • Focusing on the importance of identifying and mitigating concerns with At-Risk Materials (@RM) • Leading modernization eff orts for Blending and Packaging (B&P) We are on the verge of a transformational improvement in all of our processes in support of the mission. The following area achievements outlined in this report reflect the hard work toward achieving the goal of “being the material Subject Matter Experts in the nuclear enterprise”.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Practical methods to improve the development of computational software

The use of computation has become ubiquitous in science and engineering. As the complexity of computer codes has increased, so has the need for robust methods to minimize errors. Past work has show that the number of functional errors is related the number of commands that a code executes. Since the late 1960's, major participants in the field of computation have encouraged the development of best practices for programming to help reduce coder induced error, and this has lead to the emergence of 'software engineering' as a field of study. Best practices for coding and software production have now evolved and become common in the development of commercial software. These same techniques, however, are largely absent from the development of computational codes by research groups. Many of the best practice techniques from the professional software community would be easy for research groups in nuclear science and engineering to adopt. This paper outlines the history of software engineering, as well as issues in modern scientific computation, and recommends practices that should be adopted by individual scientific programmers and university research groups. (authors)

97 MATHEMATICS AND COMPUTING↗

Hybrid Programming for Near-Term Quantum Computing Systems

Recent computations involving quantum processing units (QPUs) have demonstrated a series of challenges inherent to hybrid classical-quantum programming, compilation, execution, and verification and validation. Despite considerable progress, system-level noise, limited low-level instructions sets, remote access models, and an overall lack of portability and classical integration presents near-term programming challenges that must be overcome in order to enable reliable scientific quantum computing and support robust hardware benchmarking. In this work, we draw on our experience in programming QPUs to identify common concerns and challenges, and detail best practices for mitigating these challenges within the current hybrid classical-quantum computing paradigm. Following this discussion, we introduce the XACC quantum compilation and execution framework as a hardware and language independent solution that addresses many of these hybrid programming challenges. XACC supports extensible methodologies for managing a variety of programming, compilation, and execution concerns across the increasingly diverse set of QPUs. We use recent nuclear physics simulations to illustrate how the framework mitigates programming, compilation, and execution challenges and manages the complex workflow present in QPU-enhanced scientific applications. Finally, we codify the resulting hybrid scientific computing workflow in order to identify key areas requiring future improvement.

Mccaskey, Alex↗

Near-Field Imaging of Shallow Chemical Detonations in Granite using Change Detection Methods of Borehole Seismic Data

As part of the Source Physics Experiment (SPE) Phase I shallow chemical detonation series, multiple surface and borehole active-source seismic campaigns were executed to perform high resolution imaging of seismic velocity changes in the granitic substrate. Cross-correlation data processing methods were implemented to efficiently and robustly perform semi-automated change detection of first-arrival times between campaigns. The change detection algorithm updates the arrival times, and consequently the velocity model, of each campaign. The resulting tomographic imagery reveals the evolution of the subsurface velocity structure as the detonations progressed.

42 ENGINEERING↗

Near-Field Imaging of Shallow Chemical Detonations in Granite using Change Detection Methods of Borehole Seismic Data

As part of the Source Physics Experiment (SPE) Phase I shallow chemical detonation series, multiple surface and borehole active-source seismic campaigns were executed to perform high-resolution imaging of seismic velocity changes in the granitic substrate. Cross-correlation data processing methods were implemented to efficiently and robustly perform semi-automated change detection of first-arrival times between campaigns. The change detection algorithm updates the arrival times, and consequently the velocity model, of each campaign. The resulting tomographic imagery reveals the evolution of the subsurface velocity structure as the detonations progressed.

42 ENGINEERING↗

Complexity in scalable computing.

The rich history of scalable computing research owes much to a rapid rise in computing platform scale in terms of size and speed. As platforms evolve, so must algorithms and the software expressions of those algorithms. Unbridled growth in scale inevitably leads to complexity. This special issue grapples with two facets of this complexity: scalable execution and scalable development. The former results from efficient programming of novel hardware with increasing numbers of processing units (e.g., cores, processors, threads or processes). The latter results from efficient development of robust, flexible software with increasing numbers of programming units (e.g., procedures, classes, components or developers). The progression in the above two parenthetical lists goes from the lowest levels of abstraction (hardware) to the highest (people). This issue's theme encompasses this entire spectrum. The lead author of each article resides in the Scalable Computing Research and Development Department at Sandia National Laboratories in Livermore, CA. Their co-authors hail from other parts of Sandia, other national laboratories and academia. Their research sponsors include several programs within the Department of Energy's Office of Advanced Scientific Computing Research and its National Nuclear Security Administration, along with Sandia's Laboratory Directed Research and Development program and the Office of Naval Research. The breadth of interests of these authors and their customers reflects in the breadth of applications this issue covers. This article demonstrates how to obtain scalable execution on the increasingly dominant high-performance computing platform: a Linux cluster with multicore chips. The authors describe how deep memory hierarchies necessitate reducing communication overhead by using threads to exploit shared register and cache memory. On a matrix-matrix multiplication problem, they achieve up to 96% parallel efficiency with a three-part strategy: intra-node multithreading, non-blocking inter-node message passing, and a dedicated communications thread to facilitate concurrent communications and computations. On a quantum chemistry problem, they spawn multiple computation threads and communication threads on each node and use one-sided communications between nodes to minimize wait times. They reduce software complexity by evolving a multi-threaded factory pattern in C++ from a working, message-passing program in C.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Effectiveness of State-Level Policies on Solar Market Development in Different State Contexts

In response to public interest in customer-sited distributed solar photovoltaics (PV), state and local policymakers have implemented policy initiatives with the goal of encouraging private investment and building a robust PV market. Policymakers face challenges, including limited budgets and incomplete information about the effectiveness of the various policy options in their specific situation, in crafting and executing policy that supports market development goals. Recent work investigated the effect of the order in which policies are implemented (referred to as 'policy stacking') and the presence of low-cost enabling policies, such as interconnection standards and net metering, can have on the success of states in promoting PV markets. Findings indicate that implementation of interconnection standards and policy related to the valuation of excess electricity (e.g., net metering), along with indicators of long term government support for a solar PV market (e.g., RPS) and a non-policy determinant (population), explain about 70% of the variation among states in new PV capacity. This paper builds on that research to determine the most effective policy strategies for different types of states, as determined by their physical, demographic and macroeconomic context. A number of researchers have investigated the effectiveness of state-level policy using various statistical methods to determine relationships between installed solar PV projects and policy initiatives. In this study, the grouping of states by non-policy factors adds dimension to these analyses by identifying how policies function in different non-policy environments.

14 SOLAR ENERGY↗

Analysis into Asymptotic Convergence to Full Nonlinear Solutions and Exploration of the Implication of Numerical Operator Mutation of Differential Systems

A robust, sufficiently accurate and practical hydrodynamic simulation toolset is required as a key component of the modeling and simulation of air-gap electrostatic discharge events. This work was performed to complement these ongoing efforts. In particular, hydrodynamic simulations must be vetted to ensure they are robust and sufficiently accurate over relevant characteristic scales. Verification models were generated in order to cultivate the technical knowledge and expertise needed to properly create, implement and execute numerical simulations. Furthermore, this effort was utilized extensively to educate students on the mathematical and numerical principles underlying hydrodynamic simulations. This education opportunity, provided in a holistic and rigorous manner, has greatly benefited developing scientists and engineers with the necessary understandings and toolsets required to excel at accomplishing the task at hand, and, more generally, it has enabled them to generate key programmatic deliverables. This report articulates several subtilties; specifically, how perturbations, nonlinear behavior, and dissipative mechanisms influence numerical stability, how to properly structure mathematical and numerical solutions, and how to properly generate error estimation/assignment. A more rigorous discussion of the consequences of such topics can be found in the body of this report in Chapters 2 and 3 with qualitative findings discussed in Chapter 4.

97 MATHEMATICS AND COMPUTING↗

Scalable Federated Learning for Scientific Foundation Models on Leadership-Class Systems

Federated learning (FL) at leadership-class HPC systems remains largely unexplored, despite growing interest in deploying federated workflows on modern HPC systems. This paper provides the first system-level empirical characterization of federated fine-tuning of pretrained foundation models on an exascale supercomputer under a multi-node deployment. Using up to 96 concurrent FL clients deployed across Frontier nodes, we study the impact of client scale, model size, data heterogeneity, partial participation, and differential privacy on runtime, communication overhead, and convergence stability. Our results show that pretrained transformer models remain robust to heterogeneity, client dropout, and privacy noise, while system efficiency degrades rapidly with scale as synchronizat and orchestration dominate runtime. We further demonstrate that system-aware execution strategies, including intra-node aggregation and early aggregation, significantly reduce wall-clock time without degrading model quality. These findings establish a practical performance baseline and inform the design of communication-efficient FL systems on leadership-class HPC platforms.

Kotevska, Olivera [ORNL] (ORCID:0000000316772243)↗

Installing python software packages : the good, the bad and the ugly.

These slides describe different strategies for installing Python software. Although I am a big fan of Python software development, robust strategies for software installation remains a challenge. This talk describes several different installation scenarios. The Good: the user has administrative privileges - Installing on Windows with an installer executable, Installing with Linux application utility, Installing a Python package from the PyPI repository, and Installing a Python package from source. The Bad: the user does not have administrative privileges - Using a virtual environment to isolate package installations, and Using an installer executable on Windows with a virtual environment. The Ugly: the user needs to install an extension package from source - Installing a Python extension package from source, and PyCoinInstall - Managing builds for Python extension packages. The last item referring to PyCoinInstall describes a utility being developed for the COIN-OR software, which is used within the operations research community. COIN-OR includes a variety of Python and C++ software packages, and this script uses a simple plug-in system to support the management of package builds and installation.

99 GENERAL AND MISCELLANEOUS↗

Middleware Case Study: MeDICi

In many application domains in science and engineering, data produced by sensors, instruments and networks is naturally processed by software applications structured as a pipeline . Pipelines comprise a sequence of software components that progressively process discrete units of data to produce a desired outcome. For example, in a Web crawler that is extracting semantics from text on Web sites, the first stage in the pipeline might be to remove all HTML tags to leave only the raw text of the document. The second step may parse the raw text to break it down into its constituent grammatical parts, such as nouns, verbs and so on. Subsequent steps may look for names of people or places, interesting events or times so documents can be sequenced on a time line. Each of these steps can be written as a specialized program that works in isolation with other steps in the pipeline. In many applications, simple linear software pipelines are sufficient. However, more complex applications require topologies that contain forks and joins, creating pipelines comprising branches where parallel execution is desirable. It is also increasingly common for pipelines to process very large files or high volume data streams which impose end-to-end performance constraints. Additionally, processes in a pipeline may have specific execution requirements and hence need to be distributed as services across a heterogeneous computing and data management infrastructure. From a software engineering perspective, these more complex pipelines become problematic to implement. While simple linear pipelines can be built using minimal infrastructure such as scripting languages, complex topologies and large, high volume data processing requires suitable abstractions, run-time infrastructures and development tools to construct pipelines with the desired qualities-of-service and flexibility to evolve to handle new requirements. The above summarizes the reasons we created the MeDICi Integration Framework (MIF) that is designed for creating high-performance, scalable and modifiable software pipelines. MIF exploits a low friction, robust, open source middleware platform and extends it with component and service-based programmatic interfaces that make implementing complex pipelines simple. The MIF run-time automatically handles queues between pipeline elements in order to handle request bursts, and automatically executes multiple instances of pipeline elements to increase pipeline throughput. Distributed pipeline elements are supported using a range of configurable communications protocols, and the MIF interfaces provide efficient mechanisms for moving data directly between two distributed pipeline elements.

99 GENERAL AND MISCELLANEOUS↗