Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Job Task Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Use of DAGMan in CRAB3 to improve the splitting of CMS user jobs

CRAB3 is a workload management tool used by CMS physicists to analyze data acquired by the Compact Muon Solenoid (CMS) detector at the CERN Large Hadron Collider (LHC). Research in high energy physics often requires the analysis of large collections of files, referred to as datasets. The task is divided into jobs that are distributed among a large collection of worker nodes throughout the Worldwide LHC Computing Grid (WLCG). Splitting a large analysis task into optimally sized jobs is critical to efficient use of distributed computing resources. Jobs that are too big will have excessive runtimes and will not distribute the work across all of the available nodes. However, splitting the project into a large number of very small jobs is also inefficient, as each job creates additional overhead which increases load on infrastructure resources. Currently this splitting is done manually, using parameters provided by the user. However the resources needed for each job are difficult to predict because of frequent variations in the performance of the user code and the content of the input dataset. As a result, dividing a task into jobs by hand is difficult and often suboptimal. In this work we present a new feature called “automatic splitting” which removes the need for users to manually specify job splitting parameters. We discuss how HTCondor DAGMan can be used to build dynamic Directed Acyclic Graphs (DAGs) to optimize the performance of large CMS analysis jobs on the Grid. We use DAGMan to dynamically generate interconnected DAGs that estimate the processing time the user code will require to analyze each event. This is used to calculate an estimate of the total processing time per job, and a set of analysis jobs are run using this estimate as a specified time limit. Some jobs may not finish within the alloted time; they are terminated at the time limit, and the unfinished data is regrouped into smaller jobs and resubmitted.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Evolution of CMS Workload Management Towards Multicore Job Support

The successful exploitation of multicore processor architectures is a key element of the LHC distributed computing system in the coming era of the LHC Run 2. High-pileup complex-collision events represent a challenge for the traditional sequential programming in terms of memory and processing time budget. The CMS data production and processing framework is introducing the parallel execution of the reconstruction and simulation algorithms to overcome these limitations. CMS plans to execute multicore jobs while still supporting singlecore processing for other tasks difficult to parallelize, such as user analysis. The CMS strategy for job management thus aims at integrating single and multicore job scheduling across the Grid. This is accomplished by employing multicore pilots with internal dynamic partitioning of the allocated resources, capable of running payloads of various core counts simultaneously. An extensive test programme has been conducted to enable multicore scheduling with the various local batch systems available at CMS sites, with the focus on the Tier-0 and Tier-1s, responsible during 2015 of the prompt data reconstruction. Scale tests have been run to analyse the performance of this scheduling strategy and ensure an efficient use of the distributed resources. This paper presents the evolution of the CMS job management and resource provisioning systems in order to support this hybrid scheduling model, as well as its deployment and performance tests, which will enable CMS to transition to a multicore production model for the second LHC run.

Perez-Calero Yzquierdo, A.↗

CMS distributed data analysis with CRAB3

The CMS Remote Analysis Builder (CRAB) is a distributed workflow management tool which facilitates analysis tasks by isolating users from the technical details of the Grid infrastructure. Throughout LHC Run 1, CRAB has been successfully employed by an average of 350 distinct users each week executing about 200,000 jobs per day.CRAB has been significantly upgraded in order to face the new challenges posed by LHC Run 2. Components of the new system include 1) a lightweight client, 2) a central primary server which communicates with the clients through a REST interface, 3) secondary servers which manage user analysis tasks and submit jobs to the CMS resource provisioning system, and 4) a central service to asynchronously move user data from temporary storage in the execution site to the desired storage location. Furthermore, the new system improves the robustness, scalability and sustainability of the service.Here we provide an overview of the new system, operation, and user support, report on its current status, and identify lessons learned from the commissioning phase and production roll-out.

97 MATHEMATICS AND COMPUTING↗

SPSP Phase III Recruiting, Selecting, and Developing Secure Power Systems Professionals: Individual and Team Performance Guidelines

The pending retirement of significant numbers of power system experts and a continuing trend of increasing complexity in modern power systems prompted the U.S. Department of Energy (DOE) to establish the three-phase Secure Power Systems Professional (SPSP) project. The project developed resources for job definition and competency analysis along with tools to guide assessment and curriculum design and to help identify and develop required knowledge, skills, and abilities in technical and operational aspects of power systems cybersecurity. The Secure Power Systems Professional Phase III final report was released last year which an appendix of Individual and Team Performance Guidelines. This new report is that appendix broken out as a standalone document to assist utilities in recruiting and developing Secure Power Systems Professionals at their site. Individual/Team Performance Guidelines were developed for each of the four job roles: the secure power systems engineer, incident responder, intrusion analyst, and security operator. They enable an “at-a-glance” view of the multidisciplinary and collaborative nature of SPSP job roles, for which skill sets were studied and then mapped to stories of critical incidents or vignettes. This afforded an enumeration of tasks required of each job role to effectively perform responsibilities and to meet bestpractice goals for that operation. Assignment of these tasks among the job roles, either as an individual responsibility or a collaborative responsibility, was validated during a job performance model analysis. The analysis identified 13 master vignettes from the collection of critical incident stories, many of which shared a common set of job responsibilities and tasks that were grouped into four vignette categories according to the task requirement for effective response to each event.

24 POWER TRANSMISSION AND DISTRIBUTION↗

National Certification Standard for Ground Source Heat Pump Personnel

The National Certification Standard for the Geothermal Heat Pump Industry adds to the understanding of the barriers to rapid growth of the geothermal heat pump (GHP) industry by bringing together for the first time an analysis of the roles and responsibilities of each of the individual job tasks involved in the design and installation of GHP systems. The standard addresses applicable qualifications for all primary personnel involved in the design, installation, commissioning, operation and maintenance of GHP systems, including their knowledge, skills and abilities. The resulting standard serves as a foundation for subsequent development of curriculum, training and certification programs, which are not included in the scope of this project, but are briefly addressed in the standard to describe ways in which the standard developed in this project may form a foundation to support further progress in accomplishing those other efforts. Follow-on efforts may use the standard developed in this project to improve the technical effectiveness and economic feasibility of curriculum development and training programs for GHP industry personnel, by providing a more complete and objective assessment of the individual job tasks necessary for successful implementation of GHP systems. When incorporated into future certification programs for GHP personnel, the standard will facilitate increased consumer confidence in GHP technology, reduce the potential for improperly installed GHP systems, and assure GHP system quality and performance, all of which benefit the public through improved energy efficiency and mitigated environmental impacts of the heating and cooling of homes and businesses.

15 GEOTHERMAL ENERGY↗

Efficient monitoring of CRAB jobs at CMS

CRAB is a tool used for distributed analysis of CMS data. Users can submit sets of jobs with similar requirements (tasks) with a single request. CRAB uses a client-server architecture, where a lightweight client, a server, and ancillary services work together and are maintained by CMS operators at CERN. As with most complex software, good monitoring tools are crucial for efficient use and longterm maintainability. This work gives an overview of the monitoring tools developed to ensure the CRAB server and infrastructure are functional, help operators debug user problems, and minimize overhead and operating cost. This work also illustrates the design choices and gives a report on our experience with the tools we developed and the external ones we used.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Testing Job Submission to FermiGrid via Dask on the Fermilab Elastic Analysis Facility

Over the course of a summer internship, Jobsub Lite task submission and Dask operations were tested in development and production EAF at Fermilab, first via terminal and subsequently via Jupyter notebook cell. Testing allowed for development of a script available to EAF users that expands and makes easier access to parallel computing resources at Fermilab and associated sites.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Finding Tropical Cyclones on a Cloud Computing Cluster: Using Parallel Virtualization for Large-Scale Climate Simulation Analysis

Extensive computing power has been used to tackle issues such as climate changes, fusion energy, and other pressing scientific challenges. These computations produce a tremendous amount of data; however, many of the data analysis programs currently only run a single processor. In this work, we explore the possibility of using the emerging cloud computing platform to parallelize such sequential data analysis tasks. As a proof of concept, we wrap a program for analyzing trends of tropical cyclones in a set of virtual machines (VMs). This approach allows the user to keep their familiar data analysis environment in the VMs, while we provide the coordination and data transfer services to ensure the necessary input and output are directed to the desired locations. This work extensively exercises the networking capability of the cloud computing systems and has revealed a number of weaknesses in the current cloud system software. In our tests, we are able to scale the parallel data analysis job to a modest number of VMs and achieve a speedup that is comparable to running the same analysis task using MPI. However, compared to MPI based parallelization, the cloud-based approach has a number of advantages. The cloud-based approach is more flexible because the VMs can capture arbitrary software dependencies without requiring the user to rewrite their programs. The cloud-based approach is also more resilient to failure; as long as a single VM is running, it can make progress while as soon as one MPI node fails the whole analysis job fails. In short, this initial work demonstrates that a cloud computing system is a viable platform for distributed scientific data analyses traditionally conducted on dedicated supercomputing systems.

97 MATHEMATICS AND COMPUTING↗

Radiological Operations Support Specialist (ROSS) Pilot Course Summary and Recommendations

In support of the Department of Homeland Security / Science and Technology Directorate’s (DHS/S&T) creation of a new position called the Radiological Operations Support Specialist (ROSS), Lawrence Livermore National Laboratory (LLNL) in Sub-task 1.1 and 1.2 has assisted in the development of the ROSS skills, knowledge, and abilities (SKAs); identified potentially relevant training; cross-mapped the training to the SKAs; and identified gaps in the training related to the SKAs, as well as their respective level of training knowledge - current versus desired. In the follow on task, Sub-task 1.3, a 5 day ROSS Pilot Training course was developed to fill the priority gaps identified in Sub-Task 1.2. Additionally, in Sub-Task 1.5, LLNL has performed a gap analysis of electronic tools, handbooks, and job-aides currently available to the ROSS and developed recommendations for additional and next generation tools to ensure the operational effectiveness of the ROSS position. This document summarizes the feedback received from the instructors and pilot course observers on what worked in the course and what could be improved as well as an assessment of the Pre- and Post-Test administered to the students.

61 RADIATION PROTECTION AND DOSIMETRY↗

GeoVision Analysis: Exploration Task Force Report

The GeoVision study effort included ground-breaking, detailed research on current and future market conditions and geothermal technologies in order to forecast and quantify the electric and non-electric deployment potentials under a range of scenarios, in addition to their impacts on the Nation’s jobs, economy and environment. Coordinated by the U.S. Department of Energy’s (DOE’s) Geothermal Technologies Office (GTO), the GeoVision study development relied on the collection, modeling, and analysis of robust datasets through seven national laboratory partners, which were organized into eight technical Task Force groups. The purpose of this report is to provide a central repository for the research conducted by the Exploration Task Force. The Exploration Task Force consists of four individuals representing three national laboratories: Patrick Dobson (task lead) and Christine Doughty of Lawrence Berkeley National Laboratory, Anna Wall of National Renewable Energy Laboratory, Travis McLing of Idaho National Laboratory, and Chester Weiss of Sandia National Laboratories. As part of the GeoVision analysis, our team conducted extensive scientific and financial analyses on a number of topics related to current and future geothermal exploration methods. The GeoVision Exploration Task Force complements the drilling and resource technology investigations conducted as part of the Reservoir Maintenance and Development Task Force. The Exploration Task Force however has focused primarily on early stage R&D technologies in exploration and confirmation drilling, along with an evaluation of geothermal financing challenges and assumptions, and innovative “blue-sky” technologies. This research was used to develop geothermal resource supply curves (through the use of GETEM) for use in the ReEDS capacity expansion modeling that determines geothermal technology deployment potential. It also catalogues and explores the large array of early-stage R&D technologies with the potential to dramatically reduce exploration and geothermal development costs, forming the basis of the GeoVision Technology Improvement (TI) scenario. These modeling topics are covered in detail in Potential to Penetration task force report. Most of the research contained herein has been published in peer-reviewed papers or conference proceedings and are cited and referenced accordingly. The sections that follow provide a central repository for all of the research findings of the Exploration and Confirmation Task Force. In summary, it provides a comprehensive discussion of Engineered Geothermal Systems (EGS) and associated technology challenges, the risks and costs of conducting geothermal exploration, a review of existing government efforts to date in advancing early-stage R&D in both exploration and EGS technologies, as well as a discussion of promising and innovative technologies and implementation of blue-sky concepts that could significantly reduce costs, lower risks, and shorten the time needed to explore and develop geothermal resources of all types.

15 GEOTHERMAL ENERGY↗

Three practical workflow schedulers for easy maximum parallelism

Runtime scheduling and workflow systems are an increasingly popular algorithmic component in HPC because they allow full system utilization with relaxed synchronization requirements. There are so many special-purpose tools for task scheduling, one might wonder why more are needed. Use cases seen on the Summit supercomputer needed better integration with MPI and greater flexibility in job launch configurations. Preparation, execution, and analysis of computational chemistry simulations at the scale of tens of thousands of processors revealed three distinct workflow patterns. A separate job scheduler was implemented for each one using extremely simple and robust designs: file-based, task-list based, and bulk-synchronous. Comparing to existing methods shows unique benefits of this work, including simplicity of design, suitability for HPC centers, short startup time, and well-understood per-task overhead. All three new tools have been shown to scale to full utilization of Summit, and have been made publicly available with tests and documentation. This work presents a complete characterization of the minimum effective task granularity for efficient scheduler usage scenarios. Here, these schedulers have the same bottlenecks, and hence similar task granularities as those reported for existing tools following comparable paradigms.

97 MATHEMATICS AND COMPUTING↗

Usage Pattern Analysis for the Summit Login Nodes

High performance computing (HPC) users interact with Summit through dedicated gateways, also known as login nodes. The performance and stability of these login nodes can have a significant impact on the user experience. In this study, the performance and stability of Summit’s five login nodes are evaluated by analyzing the log data from 2020 and 2021. The analysis focuses on the computing capability (CPU average load, users and tasks) and the storage performance, along with the associated job scheduler activity. The outcome of this study can serve as the foundation of a predictive modeling framework that enables the system admin of an HPC system to preemptively deploy countermeasures before the onset of a system failure.

Eiffert, Brett↗

Personnel Certifications for the Design, Installation and Maintenance of Photovoltaic Systems. Final Technical Report

The North American Board of Certified Energy Practitioners (NABCEP) is the most respected, well-established, and widely recognized national certification organization for professionals in the field of renewable energy. NABCEP’s certifications were created to facilitate consumer protection, improve quality assurance, and help employers judge the skills of existing or potential employees. The certifications also allow individuals to differentiate themselves from others in their field and advance their careers while fostering increased recognition of the profession. Since NABCEP’s founding in 2002, all of its certification exams have been administered in paper-and-pencil format twice per year at 30 exam sites around the United States. The biannual test administration and relatively low number of exam sites make it challenging for many qualified renewable energy professionals to take NABCEP’s certification exams. The rigorous nature of the exam, which leads to many candidates failing the exam on their first and sometimes second attempts, combined with the low number of annual test administrations means that anyone who fails will need to wait another six months for the chance to retake the exam. These factors limited the potential growth of NABCEP’s certification programs and the resultant quality assurance within the renewable energy industry. Thanks to a generous grant by the Department of Energy’s Solar Training and Education for Professional (STEP) program, NABCEP recently began updating its Photovoltaic (PV) Installation Professional Exam and developing three new specialty certifications in PV design, PV installation, and PV commissioning & maintenance. One of the primary objectives of this STEP-funded project was to develop the exams using a computer-based testing (CBT) model to increase the number of exam administrations and available exam sites for NABCEP’s certification exams. To successfully complete this project, NABCEP partnered with Castle Worldwide, Inc. With more than 25 years of experience in test development, test administration, analysis of examinations, and has a staff of 70 experienced personnel, NABCEP was confident that Castle’s team was more than qualified to help NABCEP make the transition to CBT examinations. NABCEP and Castle Worldwide jointly developed a project management plan that included key deliverables, scope management, a work breakdown structure, roles and responsibilities, and a project schedule with milestones. The project schedule was carefully monitored to ensure that activities along the critical path were not delayed and, when deemed necessary, the project was crashed to add resources so that activities could occur in parallel. NABCEP and Castle Worldwide staff met weekly to discuss any potential schedule variances and activities which might delay the closing of the project. NABCEP also recruited 46 subject matter experts (SMEs) to work with Castle’s staff on this project. NABCEP staff, SME’s, and Castle staff worked together to develop the Job Task Analyses that form the basis for the certification exams, create new examination forms, conduct passing score studies, and complete a psychometrical analysis of potential exam items. NABCEP’s SMEs utilized Castle’s item development platform during face-to-face meetings, webinars, and remote item authoring to write, review, and approve exam items. Castle also used their software to psychometrically analyze existing exam items to confirm the reliability and validity of the NABCEP’s certification exams. On January 3rd, 2018, NABCEP’s began offering its PV Installation Professional certification exam via CBT. The exam was available at Castle’s nearly 600 exam sites in the United States. A large number of sites will help to ensure better access for solar professionals in rural and remote areas, which drives down the implicit costs of taking the certification exams. Thanks to the STEP grant and the robust partnership between NABCEP and Castle Worldwide, Inc., personnel certifications will be much more accessible and affordable for renewable energy professionals.

14 SOLAR ENERGY↗

Automatic log analysis with NLP for the CMS workflow handling

The central Monte-Carlo production of the CMS experiment utilizes the WLCG infrastructure and manages daily thousands of tasks, each up to thousands of jobs. The distributed computing system is bound to sustain a certain rate of failures of various types, which are currently handled by computing operators a posteriori. Within the context of computing operations, and operation intelligence, we propose a Machine Learning technique to learn from the operators with a view to reduce the operational workload and delays. This work is in continuation of CMS work on operation intelligence to try and reach accurate predictions with Machine Learning. We present an approach to consider the log files of the workflows as regular text to leverage modern techniques from Natural Language Processing (NLP). In general, log files contain a substantial amount of text that is not human language. Therefore, different log parsing approaches are studied in order to map the log files’ words to high dimensional vectors. These vectors are then exploited as feature space to train a model that predicts the action that the operator has to take. This approach has the advantage that the information of the log files is extracted automatically and the format of the logs can be arbitrary. In this work the performance of the log file analysis with NLP is presented and compared to previous approaches.

Layer, Lukas↗

Task parallel sensitivity analysis and parameter estimation of groundwater simulations through the SALSSA framework

The Support Architecture for Large-Scale Subsurface Analysis (SALSSA) provides an extensible framework, sophisticated graphical user interface, and underlying data management system that simplifies the process of running subsurface models, tracking provenance information, and analyzing the model results. Initially, SALSSA supported two styles of job control: user directed execution and monitoring of individual jobs, and load balancing of jobs across multiple machines taking advantage of many available workstations. Recent efforts in subsurface modelling have been directed at advancing simulators to take advantage of leadership class supercomputers. We describe two approaches, current progress, and plans toward enabling efficient application of the subsurface simulator codes via the SALSSA framework: automating sensitivity analysis problems through task parallelism, and task parallel parameter estimation using the PEST framework.

99 GENERAL AND MISCELLANEOUS↗

Operational Analytics Studies for ATLAS Distributed Computing: Data Popularity Forecast and Utilization of the WLCG Centers

Operational analytics is the direction of research related to the analysis of the current state of computing processes and the prediction of future states in order to anticipate imbalances and take timely measures to stabilize a complex system. There are two relevant areas in ATLAS Distributed Computing that are currently the focus of studies: user physics analysis including the forecast of popularity of data samples among users, and evaluating WLCG centers for their readiness to process user analysis payloads. Studying these areas is challenging due to the complexity involved, as it requires a comprehensive understanding of numerous boundary conditions typically found in large-scale distributed computing infrastructures. Forecasts of data popularity are problematic without the categorization of user tasks by their types (data transformation or physics analysis), which do not always appear on the surface but may induce noise, which introduces significant distortions for predictive analysis. Evaluating the WLCG resources by their analysis workloads is also a challenging task as it is necessary to find a balance between the workload of the resource, its performance, the waiting time for jobs on it, as well as the volume of jobs that it processes. This is especially difficult in a heterogeneous computing environment, where legacy resources are used along with modern high-performance machines. We will look at these areas of research in detail and discuss what tools and methods are used in our work, demonstrating results already obtained.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

GeoVision Analysis Supporting Task Force Report: Barriers-An Analysis of Non-Technical Barriers to Geothermal Deployment and Potential Improvement Scenarios

This report describes research and analysis performed in support of the U.S. Department of Energy Geothermal Technologies Office for its Geothermal Vision Study. A summary of the study is captured in DOE’s report, GeoVision: Harnessing the Heat Beneath Our Feet and included ground-breaking, detailed research on geothermal technologies. The study projects and quantifies the future electric and nonelectric deployment potentials of these geothermal technologies within a range of scenarios in addition to their impacts on U.S. jobs, the economy, and environment. Coordinated by the U.S. Department of Energy Geothermal Technologies Office, the Geothermal Vision Study development relied on collecting, modeling, and analyzing robust data sets through seven national laboratory partners that were organized into eight technical task force groups. These task forces and their respective principal leading national laboratory are listed in Table F-1. The table also provides a guide to the final research documents produced by each GeoVision task force. In most cases, these were prepared as laboratory reports, and they are referenced accordingly. Consult these external reports for detailed discussions of the topics contained within, which form the basis of the GeoVision analysis.

15 GEOTHERMAL ENERGY↗

GeoVision Analysis Supporting Task Force Report: Electric Sector Potential to Penetration

This report describes research and analysis performed in support of the U.S. Department of Energy Geothermal Technologies Office for its Geothermal Vision Study. A summary of the study is captured in DOE’s report, GeoVision: Harnessing the Heat Beneath Our Feet and included ground-breaking, detailed research on geothermal technologies. The study projects and quantifies the future electric and nonelectric deployment potentials of these geothermal technologies within a range of scenarios in addition to their impacts on U.S. jobs, the economy, and environment. Coordinated by the U.S. Department of Energy Geothermal Technologies Office, the Geothermal Vision Study development relied on collecting, modeling, and analyzing robust data sets through seven national laboratory partners that were organized into eight technical task force groups. These task forces and their respective principal leading national laboratory are listed in Table F-1. The table also provides a guide to the final research documents produced by each GeoVision task force. In most cases, these were prepared as laboratory reports, and they are referenced accordingly. Consult these external reports for detailed discussions of the topics contained within, which form the basis of the GeoVision analysis.

15 GEOTHERMAL ENERGY↗