SEARCH · Engineering Papers

Results for “cluster scheduling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

All sources NASA NTRS DOE OSTI

DOE OSTI2024

OctoFAS: A Two-Level Fair Scheduler That Increases Fairness in Network-Based Key-Value Storage

We identified a fairness problem in a network-based key-value storage system using Intel Storage Performance Development Kit (SPDK) in a multitenant environment. In such an environment, each tenant’s I/O service rate is not fairly guaranteed compared to that of other tenants. To address the fairness problem, we propose OctoFAS, a two-level fair scheduler designed to improve overall throughput and fairness among tenants. The two-level scheduler of OctoFAS consists of (i) inter-core scheduling and (ii) intra-core scheduling. Through inter-core scheduling, OctoFAS addresses the load imbalance problem that is inherent in SPDK on the storage server by dynamically migrating I/O requests from overloaded cores to underloaded cores, thereby increasing overall throughput. Intra-core scheduling prioritizes handling requests from starving tenants over well-fed tenants within core-specific event queues to ensure fair I/O services among multiple tenants. OctoFAS is deployed on a Linux cluster with SPDK. Through extensive evaluations, we found that OctoFAS ensures that the total system throughput remains high and balanced, while enhancing fairness by approximately 10% compared to the baseline, when both scheduling levels operate in a hybrid fashion.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

VC3: Virtual Clusters for Community Computation (Final Technical Report)

A traditional HPC computing facility provides a large amount of computing power but has a fixed environment designed to satisfy local needs. This makes it very challenging for users to deploy complex applications that span multiple sites and require specific application software, scheduling middleware, or sharing policies. This project addressed many of these challenges by making it possible for researchers to easily aggregate and share resources, install custom software environments, and deploy clustering frameworks across multiple HPC facilities through the concept of “virtual clusters”. We designed and implemented a prototype virtual cluster facility that enabled unprivileged users to create dynamic aggregations of computing power across multiple sites, deployed with custom middleware and complex software dependencies.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Using containers to speed up development, to run integration tests and to teach about distributed systems

GlideinWMS is a workload manager provisioning resources for many experiments including CMS and DUNE. The software is distributed both as native packages and specialized production containers. Following an approach used in other communities like web development we built our workspaces, system-like containers to ease development and testing. Developers can change the source tree or check out a different branch and quickly reconfigure the services to see the effect of their changes. In this paper, we’ll talk about what differentiates workspaces from other containers. We’ll describe our base system composed of three containers. A one-node cluster including a compute element and a batch system. A GlideinWMS Factory controlling pilot jobs. And a scheduler and Frontend, to submit jobs and provision resources. Additional containers can be used for optional components. This system can easily run on a laptop and we’ll share our evaluation of different container runtimes, with an eye for ease of use and performance. Finally, we’ll talk about our experience as developers and with students. The GlideinWMS workspaces are easily integrated with IDEs like VS Code, simplifying debugging and allowing development and testing of the system also when offline. They simplified the training and onboarding of new team members and Summer interns. And they were useful in workshops where students could have first-hand experience with the mechanisms and components that, in production, run millions of jobs.

Mambelli, Marco↗

DOE OSTI2025

Using Containers to Speed Up Development, to Run Integration Tests and to Teach About Distributed Systems

GlideinWMS is a workload manager provisioning resources for many experiments, including CMS and DUNE. The software is distributed both as native packages and specialized production containers. Following an approach used in other communities like web development, we built our workspaces, system-like containers to ease development and testing. Developers can change the source tree or check out a different branch and quickly reconfigure the services to see the effect of their changes. In this paper, we will talk about what differentiates workspaces from other containers. We will describe our base system, composed of three containers: a one-node cluster including a compute element and a batch system, a GlideinWMS Factory controlling pilot jobs, and a scheduler and Frontend to submit jobs and provision resources. Additional containers can be used for optional components. This system can easily run on a laptop, and we will share our evaluation of different container runtimes, with an eye for ease of use and performance. Finally, we will talk about our experience as developers and with students. The GlideinWMS workspaces are easily integrated with IDEs like VS Code, simplifying debugging and allowing development and testing of the system even when offline. They simplified the training and onboarding of new team members and summer interns. And they were useful in workshops where students could have first-hand experience with the mechanisms and components that, in production, run millions of jobs.

Mambelli, Marco [Fermilab] (ORCID:0000000294892681↗

NASA NTRSDate not supplied

Impact, a Tool Suite for Crew Health and Performance System Trade Analyses and Decision Support - Status of Development

Mission planners, systems engineers, and clinicians that support crew health and performance face very difficult choices on upcoming exploration missions. Given that there will be a heavily constrained mass and volume allocation for a medical system on these missions, what medical capability should be manifested to minimize both medical risk and mission risk? Given that not all promising research and technology proposals can be funded, how can proposals be prioritized so that those funded research investments produce the maximum benefit in reducing overall medical risk? The Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) project seeks to answer these kinds of questions and others to support upcoming exploration missions. IMPACT enables risk-informed and evidence-based trade space analysis for future space vehicles, missions, and systems. This presentation will discuss the long-term HRP and ExMC vision for the larger ecosystem of tools, which include an updated medical database (consisting of an Evidence Library for medical conditions and a medical item database (MedID) for medical resources), dynamic Probabilistic Risk Assessment (PRA) capabilities, System Modeling Language (SysML) models, and contextual data visualizations of output data. IMPACT is the result of a multi-center collaborative effort. The trade space analyses performed by IMPACT can directly inform mission, vehicle, and habitat development by quantifying medical risk, given a design reference mission, crew attributes and a set of medical capabilities. This presentation will update the audience on the development status of the tool suite, its constituent parts and the plans for when it will deploy as an operational product, currently scheduled for the end of FY22. Recent development successes on the IMPACT project include the ability to cluster medical resources into medical capabilities and mutually-dependent bundles, the ability to perform trade analyses on different medical sets, different DRMs, and different crew complements, and the ability to specify mission segments as periods of time assigned to one or more members of the crew during which unique mission events occur (e.g., extravehicular activities, surface operations, gravity well adaptations, etc.). IMPACT is gearing up for its verification and validation phase to compile the data package necessary for a successful Transition to Operations (TtO), per the guidance in NPR 8900.1B .

IMPACT↗

DOE OSTI2023

The Empirical Effect of Fleet Optimization on Synchronization and Rebound Effects in Heat Pump Water Heaters

Demand response is a growing concept in light of the internet of things and an increasing need for grid flexibility. Water heaters are one of the preferred devices for providing demand response for grid services and peak management due to their capability to store energy. The efficient use of water heaters for demand response requires consideration of the associated load effects such as synchronization of device schedules and rebound effect. These effects present a significant challenge. Despite the importance of the mentioned effects for water heater queuing and scheduling, there has been no effort to quantify and empirically validate their impact. This study attempts to address this gap by offering two methods - Ward clustering and Euclidean K-means - to evaluate the extent of synchronization in a fleet of 42 water heaters in Atlanta, GA. Using the aforementioned methods on the measured data, we find evidence of convergence of water heater loads as a result of optimization compared to an idle period and analyzed their impact.

demand response↗

DOE OSTI2023

Optimizing Performance on Trinity Utilizing Machine Learning, Proxy Applications and Scheduling Priorities

The sheer number of nodes continues to increase in today’s supercomputers, the first half of Trinity alone contains more than 9400 compute nodes. Since the speed of today’s clusters are limited by the slowest nodes, it more important than ever to identify slow nodes, improve their performance if it can be done, and assure minimal usage of slower nodes during performance critical runs. This is an ongoing maintenance task that occurs on a regular basis and, therefore, it is important to minimize the impact upon its users by assessing and addressing slow performing nodes and mitigating their consequences while minimizing down time. These issues can be solved, in large part, through a systematic application of fast running hardware assessment tests, the application of Machine Learning, and making use of performance data to increase efficiency of large clusters. Proxy applications utilizing both MPI and OpenMP were developed to produce data as a substitute for long runtime applications to evaluate node performance. Machine learning is applied to identify underperforming nodes, and policies are being discussed to both minimize the impact of underperforming nodes and increase the efficiency of the system. In this paper, I will describe the process used to produce quickly performing proxy tests, consider various methods to isolate the outliers, and produce ordered lists for use in scheduling to accomplish this task.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2014

Clustering Days with Similar Airport Weather Conditions

On any given day, traffic flow managers must often rely on past experience and intuition when developing traffic flow management initiatives that mitigate imbalances between the aircraft demand and the weather impacted airport capacity. The goal of this study was to build on recent efforts to apply data mining classification and clustering algorithms to vast archives of historical weather and air traffic data to identify patterns and past decisions that can ultimately inform day-of-operations decision-making. More specifically, this study identified similar weather impacted days at select U.S. airports, and analyzed the traffic management initiatives implemented on these representative days. The identification of the similar days was accomplished by applying a decision tree algorithm to the hourly Localized Aviation Model Output Statistics Program observations and the arrival delays for Newark Liberty International Airport. The branches from the trained decision tree were subsequently pruned to identify four weather conditions that resulted in medium to high delays for the arrivals scheduled to Newark in 2012. Using these weather conditions, four, daily airport-level Weather Impacted Traffic Index values were calculated using the Localized Aviation Model Output Statistics Program observations and the 2012 scheduled arrival counts from the FAAs Aviation System Performance Metric system. The four, daily Weather Impacted Traffic Index values for 2012 were subsequently clustered using an Expectation Maximization clustering algorithm, and nine unique types of weather days at Newark were identified. By far the most prominent type of day at Newark was a day associated with relatively good weather conditions, where there was little convective activity, winds were low, ceilings and visibility were high and there was little precipitation. Moderate levels of convective activity characterized the next most prominent type of day. Days with persistently high winds or low ceiling and visibility levels were relatively rare in 2012. Lastly, the frequency at which Ground Delay Programs, Ground Stops and Miles-in-Trail restrictions were implemented on each of the typical types of days at Newark were analyzed. Based on the results, it does appear as if the usage of Miles-in-Trail, Ground Delay Program and Ground Stop restrictions correlates well with the severity of the weather associated with each unique type of weather impacted day at Newark. Furthermore, the results demonstrate that it is feasible to use historical weather and air traffic archives to provide guidance on the types of traffic management restrictions to implement in response to the weather conditions impacting an airport.

traffic flow management↗

NASA NTRS2014

Clustering Days with Similar Airport Weather Conditions

weather↗

DOE OSTI2021

Renewable hydrogen and ammonia for combined heat and power systems in remote locations: Optimal design and scheduling

Abstract Using hydrogen (H ) and ammonia (NH ) for renewable energy storage has the potential to enable economical power and heat supply with high renewable penetrations, especially in remote locations which are characterized by high energy costs. In this work we assess the economic competitiveness of renewable combined heat and power (CHP) systems in Mahaka HI, Nantucket MA, and Northwest Arctic Borough (NWAB) AK by optimally designing these systems for scenarios in which power and heat can be purchased over a range of historical energy prices as well as when 100% renewable supply is required. We use a combined optimal design and scheduling model which minimizes annualized net present cost by determining optimal technology selection and size simultaneously with optimal schedules for each period of a system operating horizon aggregated from full year hourly resolution data via a consecutive temporal clustering algorithm. We find that renewable generation meets at least 85% of power demands and 75% of heat demands under the lowest energy prices investigated. Higher conventional energy prices lead to increased renewable penetration which is facilitated by renewable NH as a seasonal energy storage medium, as are 100% renewable CHP systems. NH is used for power generation with heat cogeneration in all three locations, as well as directly for heating in NWAB. On an annual cost basis, NH ‐enabled 100% renewable CHP is only 3% more expensive in Mahaka and NWAB than systems which can purchase energy at the lowest prices, while it is 15% more expensive in Nantucket.

Palys, Matthew J.↗

DOE OSTI2024

Condition-Based Maintenance of a Circulating Water System of a Canadian Nuclear Power Plant using Machine Learning and Statistical Tools

Canada Deuterium Uranium pressurized-heavy-water reactors (PHWR) are a type of nuclear power plant that generate clean and reliable energy. The scope of this work is to automate data analysis methodologies to inform a condition-based maintenance strategy of a circulating water system (CWS) of a PHWR. The multiunit CWS provides a continuous supply of water to cool steam condensers, even during transient scenarios, thereby improving the thermal efficiency. This work aims to develop a machine learning (ML) based approach to detect anomalies in heterogeneous data of a CWS in a PHWR to help inform a predictive maintenance strategy. The heterogeneous data include textual and numeric time series data for a PHWR. Natural-language-processing (NLP)-based models are used to analyze textual data contained in work orders and operator logs and an event-timeseries correlation detection method is applied to assist anomalies diagnoses for CWS. An ML model Robust Linear Model (RLM) is also used to remove the seasonal variations in the system variable distributions based on distributions of environmental variables. A machine learning model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), trained on both original data and data without any seasonal variations will then be used to detect if an anomaly exists. Thus, by moving to an automated methodology to detect, classify, and forecast anomalies, the maintenance strategy would be based on component condition instead of a time-based schedule.

97 - MATHEMATICS AND COMPUTING↗

NASA NTRSDate not supplied

IMPACT, a Tool Suite for Crew Health and Performance System Trade Analyses and Decision Support - Status of Development

Mission planners, systems engineers, and clinicians that support crew health and performance face very difficult choices on upcoming exploration missions. Given that there will be a heavily constrained mass and volume allocation for a medical system on these missions, what medical capability should be manifested to minimize both medical risk and mission risk? Given that not all promising research and technology proposals can be funded, how can proposals be prioritized so that those funded research investments produce the maximum benefit in reducing overall medical risk? The Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) project seeks to answer these kinds of questions and others to support upcoming exploration missions. IMPACT enables risk-informed and evidence-based trade space analysis for future space vehicles, missions, and systems. This presentation will discuss the long-term HRP and ExMC vision for the larger ecosystem of tools, which include an updated medical database, consisting of an Evidence Library for medical conditions and a medical item database (MedID) for medical resources, dynamic Probabilistic Risk Assessment (PRA) capabilities, System Modeling Language (SysML) models, and contextual data visualizations of output data. IMPACT is the result of a multi-center collaborative effort. The trade space analyses performed by IMPACT can directly inform mission, vehicle, and habitat development by quantifying medical risk, given a design reference mission, crew attributes and a set of medical capabilities. This presentation will update the audience on the development status of the tool suite as it nears its System Acceptance Review (SAR). It will review IMPACT’s constituent parts, briefly discuss typical outputs and outline the plans for transitioning to operations, currently scheduled for later in FY23. Recent development successes on the IMPACT project include the integration of the Medical Extensible Dynamic Probabilistic Risk Assessment Tool (MEDPRAT) v2.0 to accommodate segmented missions with multiple carriers and medical systems, full onboarding of the IMPACT Medical Database (IMPACT-MD), clustering medical resources and skills into medical capabilities and mutually-dependent bundles, and the ability to perform trade analyses on different medical sets, different design reference missions (DRM), with different crew complements and extra-vehicular activity (EVA) schedule.

IMPACT↗

NASA NTRS2005

Coordinated Science Campaign Scheduling for Sensor Webs

Future Earth observing missions will study different aspects and interacting pieces of the Earth's eco-system. Scientists are designing increasingly complex, interdisciplinary campaigns to exploit the diverse capabilities of multiple Earth sensing assets. In addition, spacecraft platforms are being configured into clusters, trains, or other distributed organizations in order to improve either the quality or the coverage of observations. These simultaneous advances in the design of science campaigns and in the missions that will provide the sensing resources to support them offer new challenges in the coordination of data and operations that are not addressed by current practice. For example, the scheduling of scientific observations for satellites in low Earth orbit is currently conducted independently by each mission operations center. An absence of an information infrastructure to enable the scheduling of coordinated observations involving multiple sensors makes it difficult to execute campaigns involving multiple assets. This paper proposes a software architecture and describes a prototype system called DESOPS (Distributed Earth Science Observation Planning and Scheduling) that will address this deficiency.

Edgington, Will↗

NASA NTRS1982

Mod-2 wind turbine system cluster research test program. Volume 1: Initial plan E-1290

Upon completion of the design and development of three Mod-2 wind turbines, a series of research experiments are planned to gather data on and evaluate the performance, environmental effects, and operation of a cluster as well as a single, large multimegawatt wind turbine. Information on the program objectives, a Mod-2 system description, a planned schedule, organizational roles, and responsibilities, is included.

Gordon, L. H.↗

NASA NTRSDate not supplied

Impact, A Tool Suite for Crew Health and Performance System Trade Analyses and Decision to Support - Transition to Operations

Mission planners, systems engineers, and clinicians that support crew health and performance face very difficult choices on upcoming exploration missions. Given that there will be a heavily constrained mass and volume allocation for a medical system on these missions, what medical capability should be manifested to minimize both medical risk and mission risk? Given that not all promising research and technology proposals can be funded, how can proposals be prioritized so that those funded research investments produce the maximum benefit in reducing overall medical risk? The Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) project seeks to answer these kinds of questions and others to support upcoming exploration missions. IMPACT enables risk-informed and evidence-based trade space analysis for future space vehicles, missions, and systems. This presentation will discuss the long-term HRP and ExMC vision for the larger ecosystem of tools, which include an updated medical database, consisting of an Evidence Library for medical conditions and a medical item database (MedID) for medical resources, dynamic Probabilistic Risk Assessment (PRA) capabilities, System Modeling Language (SysML) models, and contextual data visualizations of output data. IMPACT is the result of a multi-center collaborative effort. The trade space analyses performed by IMPACT can directly inform mission, vehicle, and habitat development by quantifying medical risk, given a design reference mission, crew attributes and a set of medical capabilities. This presentation will update the audience on the development status of the IMPACT tool suite as it comes out of its System Acceptance Review (SAR) and nears Transition to Operations (TTO). It will review IMPACT’s constituent parts, briefly discuss typical outputs, and outline the plans for transitioning to operations, currently scheduled for later in FY24. Recent development successes on the IMPACT project include the integration of the Medical Extensible Dynamic Probabilistic Risk Assessment Tool (MEDPRAT) v2.0 to accommodate segmented missions with multiple carriers and medical systems, full onboarding of the IMPACT Medical Database (IMPACT-MD), clustering medical resources and skills into medical capabilities and mutually-dependent bundles, verification of IMPACT-MD, and the ability to perform trade analyses on different medical sets, different design reference missions (DRM), with different crew complements and extra-vehicular activity (EVA) schedules.

IMPACT↗

NASA NTRSDate not supplied

IMPACT, A Tool Suite for Crew Health and Performance System Trade Analyses and Decision Support- Transition to Operations

Mission planners, systems engineers, and clinicians that support crew health and performance face very difficult choices on upcoming exploration missions. Given that there will be a heavily constrained mass and volume allocation for a medical system on these missions, what medical capability should be manifested to minimize both medical risk and mission risk? Given that not all promising research and technology proposals can be funded, how can proposals be prioritized so that those funded research investments produce the maximum benefit in reducing overall medical risk? The Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) project seeks to answer these kinds of questions and others to support upcoming exploration missions. IMPACT enables risk-informed and evidence-based trade space analysis for future space vehicles, missions, and systems. This presentation will discuss the long-term HRP and ExMC vision for the larger ecosystem of tools, which include an updated medical database, consisting of an Evidence Library for medical conditions and a medical item database (MedID) for medical resources, dynamic Probabilistic Risk Assessment (PRA) capabilities, System Modeling Language (SysML) models, and contextual data visualizations of output data. IMPACT is the result of a multi-center collaborative effort. The trade space analyses performed by IMPACT can directly inform mission, vehicle, and habitat development by quantifying medical risk, given a design reference mission, crew attributes and a set of medical capabilities. This presentation will update the audience on the development status of the IMPACT tool suite as it comes out of its System Acceptance Review (SAR) and nears Transition to Operations (TTO). It will review IMPACT’s constituent parts, briefly discuss typical outputs, and outline the plans for transitioning to operations, currently scheduled for later in FY24. Recent development successes on the IMPACT project include the integration of the Medical Extensible Dynamic Probabilistic Risk Assessment Tool (MEDPRAT) v2.0 to accommodate segmented missions with multiple carriers and medical systems, full onboarding of the IMPACT Medical Database (IMPACT-MD), clustering medical resources and skills into medical capabilities and mutually-dependent bundles, verification of IMPACT-MD, and the ability to perform trade analyses on different medical sets, different design reference missions (DRM), with different crew complements and extra-vehicular activity (EVA) schedules.

IMPACT↗

NASA NTRS2011

NASA Instrument Cost/Schedule Model

NASA's Office of Independent Program and Cost Evaluation (IPCE) has established a number of initiatives to improve its cost and schedule estimating capabilities. 12One of these initiatives has resulted in the JPL developed NASA Instrument Cost Model. NICM is a cost and schedule estimator that contains: A system level cost estimation tool; a subsystem level cost estimation tool; a database of cost and technical parameters of over 140 previously flown remote sensing and in-situ instruments; a schedule estimator; a set of rules to estimate cost and schedule by life cycle phases (B/C/D); and a novel tool for developing joint probability distributions for cost and schedule risk (Joint Confidence Level (JCL)). This paper describes the development and use of NICM, including the data normalization processes, data mining methods (cluster analysis, principal components analysis, regression analysis and bootstrap cross validation), the estimating equations themselves and a demonstration of the NICM tool suite.

JCL↗

DOE OSTI2021

Efficient prediction of concentrating solar power plant productivity using data clustering

Concentrating solar power (CSP) plants convert solar energy to electricity and can be deployed with a thermal storage capability to shift electricity generation from time periods with available solar resource to those with high electricity demand or electricity price. Rigorous optimization of plant design and operational strategies can improve the market-competitiveness and commercial viability; however, such optimization may require hundreds of annual performance simulations, each of which can be computationally expensive when including considerations such as optimization of dispatch scheduling, sub-hourly time resolution, and stochastic effects due to uncertain weather or electricity price forecasts. This paper proposes a methodology to reduce the computational burden associated with simulation of electricity yield and revenue for CSP plants over a single- or multi-year period. Data-clustering techniques are employed to select a small number of limited-duration time blocks for simulation that, when appropriately weighted, can reproduce generation and revenue over a single year or within each year of a multi-year period. After selection of appropriate data features and weighting factors defining similarity between time-series profiles, the methodology captured annual revenue within 2.3%, 1.7%, or 1.2% using simulation of 10, 30, or 50 three-day exemplar time blocks, respectively, for each of three single-year location/weather/market scenarios and five plant configurations ranging from low to high solar multiple and storage capacity. When applied to multi-year datasets, the proposed methodology can capture inter-year variability that is unavailable from typical meteorological year (TMY) datasets while simultaneously requiring simulation of less than a single year of data.

14 SOLAR ENERGY↗