Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗

The InSAR Scientific Computing Environment 3.0: A Flexible Framework for NISAR Operational and User-Led Science Processing

The InSAR Scientific Computing Environment (ISCE) was first developed under the NASA Advanced Information Systems Technology as a flexible, extensible object-oriented framework for Interferometric Synthetic Aperture Radar (InSAR) processing. The ISCE framework uses Python 3 at the workflow level, controlling modules of compiled code for functional processing, and managing inputs, outputs, and other flow control services. The currently released version, called ISCE 2.1, is distributed to the research community through the Western North America InSAR Consortium under a research license. The ISCE team is working on the next generation of the code in order to prepare for the NASAISRO SAR (NISAR) mission operational processing. Innovations in this code include augmentation or conversion of the custom Python framework elements in ISCE with the Pyre framework, new workflows for interferometric and polarimetric stack processing, a more intuitive and graphically based user interface, and flow control for hybrid computing environments including CPU/GPU clusters, logging and error tracking facilities, and new more efficient computational modules that exploit graphical processor units (GPUs) when available. The ISCE 3.0 framework is designed to work in an operational environment as well as on a single user’s laptop or compute cluster, with services to discover capabilities and scale computations accordingly.

Buckley, Sean M.↗

Toward a smart data transfer node

Scientific computing systems are becoming significantly more complex, with distributed teams and complex workflows spanning resources from telescopes and light sources to fast networks and Internet of Things sensor systems. In such settings, no single, centralized administrative team and software stack can coordinate and manage all resources used by a single application. Indeed, we have reached a critical limit in manageability using current human-in-the-loop techniques. We therefore argue that resources must begin to respond automatically, adapting and tuning their behavior in response to observed properties of scientific workflows. Over time, machine learning methods can be used to identify effective strategies for autonomic, goal-driven management behaviors that can be applied end-to-end across the scientific computing landscape. Using the data transfer nodes that are widely deployed in modern research networks as an example, we explore the architecture, methods, and algorithms needed for a smart data transfer node to support future scientific computing systems that self-tune and self-manage.

97 MATHEMATICS AND COMPUTING↗

MARS: Malleable Actor-Critic Reinforcement Learning Scheduler

In this paper, we introduce MARS, a new scheduling system for HPC-cloud infrastructures based on a cost-aware, flexible reinforcement learning approach, which serves as an intermediate layer for next generation HPC-cloud resource manager. MARS ensembles the pre-trained models from heuristic workloads and decides on the most cost-effective strategy for optimization. A whole workflow application would be split into several optimizable dependent sub-tasks, then based on the pre- defined resource management plan, a reward will be generated after executing a scheduled task. Lastly, MARS updates the Deep Neural Network (DNN) model based on the reward. MARS is designed to optimize the existing models through reinforcement mechanisms. MARS adapts to the dynamics of workflow applications, selects the most cost-effective scheduling solution among pre-built scheduling strategies (backfilling, SJF, etc.) and self- learning deep neural network model at run-time. We evaluate MARS with different real-world workflow traces. MARS can achieve 5%-60% increased performance compare to state-of-the- art approaches.

Baheri, Betis↗

The Globus Galaxies Platform. Delivering Science Gateways as a Service

We use public cloud computers to host sophisticated scientific data; software is then used to transform scientific practice by enabling broad access to capabilities previously available only to the few. The primary obstacle to more widespread use of public clouds to host scientific software (‘cloud-based science gateways’) has thus far been the considerable gap between the specialized needs of science applications and the capabilities provided by cloud infrastructures. We describe here a domain-independent, cloud-based science gateway platform, the Globus Galaxies platform, which overcomes this gap by providing a set of hosted services that directly address the needs of science gateway developers. The design and implementation of this platform leverages our several years of experience with Globus Genomics, a cloud-based science gateway that has served more than 200 genomics researchers across 30 institutions. Building on that foundation, we have also implemented a platform that leverages the popular Galaxy system for application hosting and workflow execution; Globus services for data transfer, user and group management, and authentication; and a cost-aware elastic provisioning model specialized for public cloud resources. We describe here the capabilities and architecture of this platform, present six scientific domains in which we have successfully applied it, report on user experiences, and analyze the economics of our deployments. Published 2015. This article is a U.S. Government work and is in the public domain in the USA.

97 MATHEMATICS AND COMPUTING↗

A2SD: Accelerating Scientific Innovation Through Autonomous Discovery Systems

The 2025 Advancing Autonomous Scientific Discovery (A2SD) workshop convened researchers from academia, national laboratories, and industry to explore the transformative role of autonomy in scientific discovery. The workshop highlighted a convergence of artificial intelligence, robotics, and computational workflows into autonomous systems capable of accelerating the scientific process. Presentations and discussions spanned autonomous experimentation, intelligent workflow orchestration, digital twins, and agent-based systems for managing complex research ecosystems. Key challenges discussed included interoperability across heterogeneous infrastructures, near real-time data management under FAIR principles, reproducibility, and the integration of human oversight. The workshop also emphasized the need for modular software interfaces, federated learning models, and education initiatives to support a next-generation scientific workforce.

Taufer, Michela [University of Tennessee, Knoxvill↗

WarpIV: In situ visualization and analysis of ion accelerator simulations

The generation of short pulses of ion beams through the interaction of an intense laser with a plasma sheath offers the possibility of compact and cheaper ion sources for many applications--from fast ignition and radiography of dense targets to hadron therapy and injection into conventional accelerators. To enable the efficient analysis of large-scale, high-fidelity particle accelerator simulations using the Warp simulation suite, the authors introduce the Warp In situ Visualization Toolkit (WarpIV). WarpIV integrates state-of-the-art in situ visualization and analysis using VisIt with Warp, supports management and control of complex in situ visualization and analysis workflows, and implements integrated analytics to facilitate query- and feature-based data analytics and efficient large-scale data analysis. WarpIV enables for the first time distributed parallel, in situ visualization of the full simulation data using high-performance compute resources as the data is being generated by Warp. The authors describe the application of WarpIV to study and compare large 2D and 3D ion accelerator simulations, demonstrating significant differences in the acceleration process in 2D and 3D simulations. WarpIV is available to the public via https://bitbucket.org/berkeleylab/warpiv. The Warp In situ Visualization Toolkit (WarpIV) supports large-scale, parallel, in situ visualization and analysis and facilitates query- and feature-based analytics, enabling for the first time high-performance analysis of large-scale, high-fidelity particle accelerator simulations while the data is being generated by the Warp simulation suite. Furthermore, this supplemental material https://extras.computer.org/extra/mcg2016030022s1.pdf provides more details regarding the memory profiling and optimization and the Yee grid recentering optimization results discussed in the main article.

43 PARTICLE ACCELERATORS↗

Towards Cross-Facility Workflows Orchestration through Distributed Automation

Modern science relies on end-to-end workflows that incorporate experimental instruments and utilize edge, cloud, or high-performance computing and storage resources. These components are geographically dispersed across various user facilities and interconnected through high-speed networks. In this paper, we present Zambeze, an automated distributed framework designed to facilitate this new class of cross-facility workflows. Utilizing swarm intelligence principles, Zambeze orchestrates science campaigns by managing distributed autonomous agents. These agents can offer a suite of services, including computing, storage, and data management. We demonstrate the feasibility of Zambeze through a real-world application involving electron microscopy, enhanced with Artificial Intelligence capabilities.

Skluzacek, Tyler↗

Expanding Biological Repository Data Available for Sharing and Knowledge Discovery

Biology has developed next-generation data science and alternative analytical approaches with methodologies which require principal investigator (PI) experimental assay data be re-used. This new approach involves mining multiple datasets at once from various hierarchical organizations of biological complexity, while concurrently evaluating how experimental factors affect endpoints of standard assays. The purpose of the NASA Ames Life Sciences Data Archive (ALSDA) is to collect, curate, and make findable, accessible, interoperable, and reusable (FAIR) all non-human space-relevant biological data. These data include mission metadata, subject metadata, assay metadata (parameters), raw and processed assay data, assay imagery, and subject-experienced telemetry (radiation, temperature, humidity, acoustics, vibrations). ALSDA has transformed to bring current biological repository data and all future collected data into this new scientific data mining reality. It has integrated into the ‘NASA Open Science’ group of projects to facilitate a suite of new tools and workflows to improve data accessibility and reusability by implementing data management plans, automating data submission agreements, and adopting the single-point-of-entry data submission portal, originally developed by NASA GeneLab. These systems required ALSDA to develop science assay configurations for the submission portal, capturing essential assay parameters according to established norms in each sub-field within biology. The submission portal expedites data collection by enhancing ease of PI data submission, providing a user interface and specificity for which data is to be submitted. ALSDA datasets are curated to maintain rich metadata, accuracy of datasets, data transparency, provenance, and additionally ensure data are machine-readable (e.g., R and Python languages). ALSDA integration with GeneLab and its analysis portals enable higher-order physiological-level datasets be mined in conjunction with -omics datasets. As ALSDA physiological-level datasets are published (micro-computed tomography, histology, intraocular pressure, hormonal assays, immunostaining, ultrasonography), the merging of hierarchical organizations of biological complexity from spaceflight will enable new knowledge discovery approaches.

Ryan T Scott↗

Electronic Derivative Classifier/Reviewing Official

The electronic Derivative Classifier, Reviewing Official (eDC/RO) is a web based document management and routing system that reduces security risks and increases workflow efficiencies. The system automates the upload, notification review request, and document status tracking of documents for classification review on a secure server. It supports a variety of document formats (i.e., pdf, doc, docx, xls, xlsx, xlsm, ppt, pptx, vsd, vsdx and txt), and allows for the dynamic placement of classification markings such as the classification level, category and caveats on the document, in addition to a document footer and digital signature.

Harris, Joshua C↗

Policy, Permit, Perform: Using City Benchmarking Data and Building Construction Permit History to Identify Energy Performance Improvements

The City of Portland (City) and Multnomah County 2015 Climate Action Plan targets a 40 percent reduction in carbon emissions below 1990 levels by 2030. To reach this goal, the plan includes a key objective to reduce the total energy use of existing buildings by 25 percent. Buildings are responsible for one-half of carbon emissions in Portland, Oregon, U.S., and improving their performance is critical to achieving the City’s climate goals. As part of its climate action planning, the City passed an ordinance on April 22, 2015, requiring commercial buildings 20,000 ft2 and larger to benchmark and disclose annual energy performance metrics through ENERGY STAR® Portfolio Manager® (ESPM), including greenhouse gas emissions, building energy use intensity (EUI) and ENERGY STAR scores. Twenty-five cities in the U.S. have adopted similar policies requiring energy benchmarking. These policies are generating a treasure trove of data about the drivers of building performance, but practitioners are only beginning to link benchmarking data to other datasets to produce actionable insight into improving building performance. In 2016, the City received a financial assistance award from the U.S. Department of Energy (DOE) through the Cities Leading through Energy Analysis and Planning project (Award Number DE-0007737). Portland’s project evaluated the application of the DOE Building Energy Asset Score Tool and rating system developed by Pacific Northwest National Laboratory (PNNL) to commercial buildings that report to the City. The City used building construction permit history to develop Asset Scores for a set of office buildings that reported energy performance for 2015 – the first year the City mandated energy benchmarking and reporting for commercial buildings 50,000 ft2 and larger. The primary project objective was to link energy benchmarking data and commercial building permit data to analyze commercial buildings systems that present the best, specific opportunities to improve energy performance. Integrating these datasets from two City departments, the Bureau of Development Services and the Bureau of Planning and Sustainability, could expand Portland’s use of analytics to inform climate policy decisions and strategic targeting of financial incentives for building owners and managers who are best positioned to improve the performance of their buildings. The project began with the commercial benchmarking data received for 340 commercial buildings that submitted calendar year 2015 energy reports. The City retained Research Into Action and Energy 350 (RIA team) to evaluate this self-reported benchmarking data. Based on a survey of 53 building managers, the RIA team found that most respondents used the City’s compliance tools, including a How-to Guide and Help Desk, and found access to utility data to be straightforward. However, almost one-third of the respondents encountered challenges when estimating the gross floor area (GFA), and some experienced confusion regarding the input of property use types and details into ESPM. Based on the RIA team’s recommendations, the City refined its How-to Guide, created online answers to frequently asked questions, and hosted office hour sessions to help building managers accurately characterize their building’s physical characteristics and operations in ESPM. Concurrently with the RIA team’s evaluation, the City worked with PNNL to estimate DOE Asset Score Preview (Preview) scores for 181 of the 340 commercial buildings that submitted ESPM reports to the City. Asset Score Tool is a web-based tool developed by PNNL for DOE to evaluate and rate the as-built energy efficiency of a building. Asset Score Tool’s full analysis capability runs a detailed energy model based on building geometry, age, envelope, lighting and mechanical systems using standard operating assumptions to give the building a score, similar to a miles per gallon rating for a car. Preview, a simplified deployment of the full Asset Score Tool, provides a preliminary score range using only seven data points, six of which are readily available in the ESPM benchmarking data. Analysis of the combined Preview scores and the operational ESPM scores identified buildings with a high potential for energy improvements through either retro-commissioning and retuning of existing operations or capital investments into building systems. The City contracted Whole Building Solutions (WBS) to mine building permit history and develop full Asset Scores for a subset of 26 office buildings that were eligible for both ENERGY STAR scores and Asset Scores. WBS researched permits and building plans to identify mechanical, lighting and envelope systems and other building characteristics necessary to complete Asset Scores. This historical data was recorded using the web-based Asset Score Tool, which evaluates the as designed energy efficiency of existing buildings. Much of this City information was available online after 2010; however, WBS had to search through microfiche files and the City’s database to obtain permit and plan history prior to 2010. Although building geometry, mechanical system type and basic envelope information was consistently identifiable, WBS found permit data gaps that prevented complete and accurate Asset Scores for most of the office buildings. If the City decides to develop Asset Scores based on building permit information in the future, WBS recommended requiring lighting plans, control diagrams and as-built drawings that reflect how the buildings are actually constructed. In addition, WBS recommended uploading plans and permits from microfiche storage into a more comprehensive, searchable database consistent with the City’s current online permit information. To identify the actual lighting types, WBS and City staff conducted site visits of all 26 office buildings. As part of managing the flow of ESPM and Asset Score data for the project, Earth Advantage, a sub-recipient of the Cities-LEAP award, created a DOE Standard Energy Efficiency Data (SEED) (USDOE 2018) instance for the City and used Asset Score’s Application Programming Interface (API) (Asset Score 2018) capabilities to develop two new features in the SEED Platform™: (1) automated Asset Score Preview ranges calculated based on benchmarking data and the City’s estimate of number of building floors; and (2) storage of full Asset Score information. Leveraging the construction permit history and site visit information for the 26 office buildings, The Cadmus Group (Cadmus) analyzed benchmarking data and Asset Score information to identify opportunities and drivers to improve performance. Correlation and regression analysis of the combined dataset was completed to determine whether building operations or specific systems – mechanical, lighting or envelope – present the best opportunities to improve energy performance. The central outcome of this project was Cadmus’ detailed analysis of the richer dataset that was created by combining the City of Portland’s commercial building permit activity and building-specific energy performance information based on metrics from ESPM and Asset Score Tool. Some of their findings collaborate with well-known facts, such as buildings with less efficient lighting systems are good targets for efficiency improvements through lighting upgrades. This analysis also helped identify that individual heating, ventilation, and air conditioning (HVAC) equipment efficiency does not necessarily correlate with building performance. However, overall HVAC system performance, as measured with the Total System Performance Ratio (TSPR) metric, is a better indicator of building performance (Goel et al. 2014). As a metric, TSPR considers whole system performance that includes equipment efficiency and system controls, such as resets for static pressure and chilled water setpoints, which can be more effective and financially feasible options for improving building performance. Given the high cost associated with envelope retrofits, opportunities to upgrade envelope insulation are likely to be limited. However, buildings with single-pane and non-thermal break windows are good candidates for window replacements, such as upgrading to double-pane windows with thermal breaks. This research project determined that developing Asset Scores based on site visits is preferable to mining City building permit data. In addition, the use of benchmarking data to develop Asset Score Preview Scores is recommended as a first step in engaging building managers to develop full Asset Scores. Asset Score Tool outputs can help building managers quickly identify opportunities for improvement and then work with the utility incentive administrator and energy service providers to develop their retrofit plan. Where full Asset Scores are available, analysis using a quadrant matrix with ENERGY STAR scores is recommended to identify buildings that present the best opportunities for operational-behavioral improvements versus ones with opportunities for upgrades to physical building systems. This workflow and the tools will help cities, utility incentive administrators, building portfolio managers and energy service providers develop energy programs and strategies for screening and analysis of buildings with the greatest energy savings potential. This report describes the project approach and research results to inform future coordination of ESPM and Asset Score data, including the use of benchmarking data to generate Preview scores. Given better awareness of energy performance, building managers can make more informed decisions to reduce energy consumption and carbon emissions. With cities, states and jurisdictions adopting benchmarking and auditing ordinances, there are substantial amounts of data submitted to cities that could be used to inform investments in energy efficiency.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Managing well leakage risks at a geologic carbon storage site with many wells

Potential geologic carbon storage (GCS) sites with a history of oil and gas production have well-characterized injectivity and storage capacity, but the presence of legacy wells increases leakage risk. Thus, the success of GCS operations at sites with many wells will require a thorough leakage risk assessment and a robust strategy for managing well leakage risk over the lifetime of the project. In this study, we demonstrate a workflow that uses the National Risk Assessment Partnership’s open-source Integrated Assessment Model to quantify well leakage risks and test the performance of various leakage risk management strategies at a heavily drilled GCS site. Here, our model simulates a 50-year basin-scale injection of CO 2 at a hypothetical site based on the Kimberlina Project Site in the Southern San Joaquin Valley of California. Brine and CO 2 leakage through 1,000 legacy wells into a USDW are stochastically simulated as a proxy for risk. We consider multiple scenarios that explore the efficacy of various well leakage risk management strategies with changes in well leakage behavior, reservoir behavior, and post-injection site care (PISC) length. Predicted leakage at the site after 100 years was small with a maximum CO 2 leakage of 102.1 tonnes (4.08×10 -5 % of the 250 Mt injected) and a maximum brine leakage of 2.4 tonnes. Leakage risk management strategies based on accurate prior information about well leak probability reduced leakage risks more effectively at the modeled site and were more robust with respect to reservoir uncertainty than strategies based on the distance of the legacy well from the injector. The importance of the PISC period length was not clear as it had a negligible impact on CO 2 leakage risk but a sizable impact on brine leakage risk in our model.

58 GEOSCIENCES↗

Developing an Advanced Environment for Collaborative Computing

Knowledge management in general tries to organize and make available important know-how, whenever and where ever is needed. Today, organizations rely on decision-makers to produce "mission critical" decisions that am based on inputs from multiple domains. The ideal decision-maker has a profound understanding of specific domains that influence the decision-making process coupled with the experience that allows them to act quickly and decisively on the information. In addition, learning companies benefit by not repeating costly mistakes, and by reducing time-to-market in Research & Development projects. Group-decision making tools can help companies make better decisions by capturing the knowledge from groups of experts. Furthermore, companies that capture their customers preferences can improve their customer service, which translates to larger profits. Therefore collaborative computing provides a common communication space, improves sharing of knowledge, provides a mechanism for real-time feedback on the tasks being performed, helps to optimize processes, and results in a centralized knowledge warehouse. This paper presents the research directions. of a project which seeks to augment an advanced collaborative web-based environment called Postdoc, with workflow capabilities. Postdoc is a "government-off-the-shelf" document management software developed at NASA-Ames Research Center (ARC).

Becerra-Fernandez, Irma↗

Development and operational experience of the web based application to collect, manage, and release the alignment and calibration configurations for data processing at CMS

The alignment and calibration workflows at the Compact Muon Solenoid (CMS) experiment are fundamental to provide a high quality physics data and to maintain the design performance of the experiment. To facilitate the operational efforts required by the experiment, the alignment and calibration team has developed and deployed a set of web-based applications to search, navigate and prepare a consistent set of calibrations to be consumed in reconstruction of data for physics, accessible through the Condition DB Browser. The Condition DB Browser hosts also various data management tools, including a visualization tool that allows to easily inspect alignment an calibration contents, an user-defined notification agent for delivering updates on modification to the database, a logging service for the user and the automatic online-to-offline condition uploads. In this paper we report on the operational experience of this web application from 2017 data taking, with focus on new features and tools incorporated during this period.

Md, Hasbuddin↗

Highly interactive, steered scientific workflows on HPC systems: optimizing design solutions

Scientific workflows are becoming increasingly important in high performance computing (HPC) settings, as the feasibility and appeal of many simultaneous heterogeneous tasks increases with increasing hardware capabilities. Currently no HPC-based workflow platform supports a dynamically adaptable workflow with interactive steering and analysis at run-time. Furthermore, for most workflow programs, compute resources are fixed for a given instance, resulting in a possible waste of expensive allocation resources when tasks are spawned and killed. Here we describe the design and testing of a run-time-interactive, adaptable, steered workflow tool capable of executing thousands of parallel tasks without an MPI programming model, using a database management system to facilitate task management through multiple live connections. We find that on the Oak Ridge Leadership Computing Facility pre-exascale Summit supercomputer it is possible to launch and interactively steer workflows with thousands of simultaneous tasks with negligible latency. For the case of particle simulation and analysis tasks that run for minutes to hours, this paradigm offers the prospect of a robust and efficient means to perform simulation-space exploration with on-the-fly analysis and adaptation.

Ossyra, John R↗

Conquering Data Chaos: Research Data Management with Kubernetes

Managing massive volumes of data and effectively making it accessible to researchers poses significant challenges and is a barrier to scientific discovery. In many cases, critical data is locked up in unwieldy file formats or one-off databases and is too large to effectively process on a single machine. This talk explores the role of Kubernetes, an open-source container orchestration platform, in addressing research data management challenges. I will discuss how we are using a set of publicly available open-source and home-grown tools in the National Renewable Energy Lab (NREL) Data, Analysis, and Visualization (DAV) group to help researchers overcome data-related bottlenecks. The talk will begin by providing an overview of the data challenges faced in research data management, including data storage, processing, and analysis. I will highlight Kubernetes' ability to handle large-scale data by leveraging containerization and distributed computing, including distributed storage. Kubernetes allows researchers to encapsulate data processing infrastructure and workflows into portable containers, enabling reproducibility and ease of deployment. Kubernetes can then schedule and manage the resource allocation of these containers to enable efficient utilization of limited computing resources, leading to more efficient data processing and analysis. I will discuss some limitations of traditional, siloed approaches to dealing with data and emphasize the need for solutions which foster collaboration. I will highlight how we are using Kubernetes at NREL to facilitate data sharing and cooperation among research teams. Kubernetes' flexible architecture enables the deployment of shared computing environments, such as Apache Superset, where researchers can seamlessly access and analyze shared datasets. Providing the ability to have one research team easily consume data generated by another, utilizing Kubernetes' as a central data platform, is one of the major wins we've encountered by adopting the platform. Finally, I will showcase real-world use cases from NREL where we have used Kubernetes to solve some persistent data challenges involving large volumes of sensor and monitoring data. I will discuss the challenges we encountered when creating our cluster and making it available as a production-ready resource. I will also discuss the specific suite of tools, including Postgres and Apache Druid for columnar and timeseries data, and Redpanda Kafka for streaming data we have deployed in our infrastructure, and the process that went into the selection of these tools.

collaborative environment↗

Automatic log analysis with NLP for the CMS workflow handling

The central Monte-Carlo production of the CMS experiment utilizes the WLCG infrastructure and manages daily thousands of tasks, each up to thousands of jobs. The distributed computing system is bound to sustain a certain rate of failures of various types, which are currently handled by computing operators a posteriori. Within the context of computing operations, and operation intelligence, we propose a Machine Learning technique to learn from the operators with a view to reduce the operational workload and delays. This work is in continuation of CMS work on operation intelligence to try and reach accurate predictions with Machine Learning. We present an approach to consider the log files of the workflows as regular text to leverage modern techniques from Natural Language Processing (NLP). In general, log files contain a substantial amount of text that is not human language. Therefore, different log parsing approaches are studied in order to map the log files’ words to high dimensional vectors. These vectors are then exploited as feature space to train a model that predicts the action that the operator has to take. This approach has the advantage that the information of the log files is extracted automatically and the format of the logs can be arbitrary. In this work the performance of the log file analysis with NLP is presented and compared to previous approaches.

Layer, Lukas↗

A Co-Design Study Of Fusion Whole Device Modeling Using Code Coupling

Complex workflows consisting of multiple simulation and analysis codes running concurrently through in-memory coupling is becoming popular due to inherent advantages in online management of large-scale data, resilience, and the code development process. However, orchestrating such a multi-application workflow to efficiently utilize resources on a heterogeneous architecture is challenging. In this paper, we present our results with running the Fusion Whole Device Modeling benchmark workflow on Summit, a pre-exascale supercomputer at Oak Ridge National Laboratory. We explore various resource distribution and process placement mechanisms, including sharing compute nodes between processes from separate applications. We show that fine-grained process placement can have a significant impact towards efficient utilization of the compute power of a node on Summit, and conclude that sophisticated tools for performing co-design studies of multi-application workflows can play an important role towards efficient orchestration of such workflows.

Choi, Jong Youl↗