High-Performance Spaceflight Computing (HPSC) Middleware Release 4 Update
Explore the source record for details and available documents.
J Lai↗
SEARCH · Engineering Papers
Results for “Middleware”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Robust Multicore Middleware (RMuM)
Explore the source record for details and available documents.
software↗
Air Traffic Management TestBed: Messaging Performance
The Air Traffic Management (ATM) TestBed is an air traffic management modeling and simulation platform and framework developed by the National Aeronautics and Space Administration (NASA) to help design, configure, integrate, run, and monitor air traffic simulations. The communication middleware, implemented in the TestBed framework layer, is a core feature for data message exchange. The feature provides an abstraction layer called Messaging Support to allow switching one middleware to another without a need to rebuild the simulation components. Messaging performance such as latencies, run durations, and throughputs are important factors. Low latencies can produce accurate results in high-fidelity and visualization models. Short run durations are preferred because better run efficiency can be achieved. High throughputs allow more runs to be executed concurrently. This technical memorandum studies and compares the messaging performance by running a full-day, fast-time simulation using three communication middleware as well as tweaking the default communication middleware settings used by the TestBed. Results indicate that the messaging performance could be improved by disabling either compression or persistence settings, while the run duration and throughput could be further improved by disabling both settings with a tradeoff of the message latencies increased by a factor of ten.
Chok Fung Lai↗
Adaptive Resource Management Technology for Satellite Constellations
This manuscript describes the Sensor Web Adaptive Resource Manager (SWARM) project. The primary focus of the project is on the design and prototyping of middleware for managing computing and network resources in a way that enables the information systems of satellite constellations to provide realtime performance within dynamic environments. The middleware has been prototyped, and it has been evaluated by employing it to manage a pool of distributed resources for the ITOS (Integrated Test and Operations System) satellite command and control software system. The design of the middleware is discussed and a summary of the evaluation effort is provided.
Welch, Lonnie↗
[Interview Questions]
The Goddard Mission Services Evolution Center, or GMSEC, was started in 2001 to create a new standard approach for managing GSFC missions. Standardized approaches in the past involved selecting and then integrating the most appropriate set of functional tools. Assumptions were made that "one size fits all" and that tool changes would not be necessary for many years. GMSEC took a very different approach and has proven to be very successful. The core of the GMSEC architecture consists of a publish/subscribe message bus, standardized message formats, and an Applications Programming Interface (API). The API supports multiple operating systems, programming languages and messaging middleware products. We use a GMSEC-developed free middleware for low-cost development. A high capacity, robust middleware is used for operations and a messaging system with a very small memory footprint is used for on-board flight software. Software components can use the standard message formats or develop adapters to convert from their native formats to the GMSEC formats. We do not want vendors to modify their core products. Over 50 software components are now available for use with the GMSEC architecture. Most available commercial telemetry and command systems, including the GMV hifly Satellite Control System, have been adapted to run in the GMSEC labs.
Smith, Dan↗
CCSDS Spacecraft Monitor and Control Mission Operations Interoperability Prototype
We are entering a new era in space exploration. Reduced operating budgets require innovative solutions to leverage existing systems to implement the capabilities of future missions. Custom solutions to fulfill mission objectives are no longer viable. Can NASA adopt international standards to reduce costs and increase interoperability with other space agencies? Can legacy systems be leveraged in a service oriented architecture (SOA) to further reduce operations costs? The Operations Technology Facility (OTF) at the Johnson Space Center (JSC) is collaborating with Deutsches Zentrum fur Luft- und Raumfahrt (DLR) to answer these very questions. The Mission Operations and Information Management Services Area (MOIMS) Spacecraft Monitor and Control (SM&C) Working Group within the Consultative Committee for Space Data Systems (CCSDS) is developing the Mission Operations standards to address this problem space. The set of proposed standards presents a service oriented architecture to increase the level of interoperability among space agencies. The OTF and DLR are developing independent implementations of the standards as part of an interoperability prototype. This prototype will address three key components: validation of the SM&C Mission Operations protocol, exploration of the Object Management Group (OMG) Data Distribution Service (DDS), and the incorporation of legacy systems in a SOA. The OTF will implement the service providers described in the SM&C Mission Operation standards to create a portal for interaction with a spacecraft simulator. DLR will implement the service consumers to perform the monitor and control of the spacecraft. The specifications insulate the applications from the underlying transport layer. We will gain experience with a DDS transport layer as we delegate responsibility to the middleware and explore transport bridges to connect disparate middleware products. A SOA facilitates the reuse of software components. The prototype will leverage the capabilities of existing legacy systems. Various custom applications and middleware solutions will be combined into one system providing the illusion of a set of homogenous services. This paper will document our journey as we implement the interoperability prototype. The team consists of software engineers with experience on the current command, telemetry and messaging systems that support the International Space Station (ISS) and Space Shuttle programs. Emphasis will be on the objectives, results and potential cost saving benefits.
Lucord, Steve↗
RAPID and DDS
This talk gives an overview of the the Robot Applications Programmers Interface Delegate (RAPID) as well as the distributed systems middleware Data Distribution Service (DDS). DDS is an open software standard, RAPID is cleared for open-source release under NOSA. RAPID specifies data-structures and semantics for high-level telemetry published by NASA robotic software. These data-structures are supported by multiple robotic platforms at Johnson Space Center (JSC), Jet Propulsion Laboratory (JPL) and Ames Research Center (ARC), providing high-level interoperability between those platforms. DDS is used as the middleware for data transfer. The feature set of the middleware heavily influences the design decision made in the RAPID specification. So it is appropriate to discuss both in this introductory talk.
Utz, Hans Heinrich↗
WLCG Token Usage and Discovery
Since 2017, the Worldwide LHC Computing Grid (WLCG) has been working towards enabling token based authentication and authorisation throughout its entire middleware stack. Following the publication of the WLCG Common JSON Web Token (JWT) Schema v1.0 [1] in 2019, middleware developers have been able to enhance their services to consume and validate the JWT-based [2] OAuth2.0 [3] tokens and process the authorization information they convey. Complex scenarios, involving multiple delegation steps and command line flows, are a key challenge to be addressed in order for the system to be fully operational. This paper expands on the anticipated token based workflows, with a particular focus on local storage of tokens and their discovery by services. The authors include a walk-through of this token flow in the RUCIO managed data-transfer scenario, including delegation to FTS and authorised access to storage elements. Next steps are presented, including the current target of submitting production jobs authorised by Tokens within 2021.
Bockelman, Brian↗
WLCG Transition from X.509 to Tokens. Status, Plans, and Timeline
Since 2017, the Worldwide LHC Computing Grid (WLCG) has been working towards enabling token-based authentication and authorization throughout its entire middleware stack. Following the initial publication of the WLCG Token Schema v1.0 in 2019, OAuth2.0 token workflows have been integrated across grid middleware. There are many complex challenges to be addressed before the WLCG can be end-to-end token-based, including not just technical hurdles but also interoperability with the wider authentication and authorization landscape. This paper presents the status of the WLCG coordination and deployment work, and how it relates to software providers and partner communities. The authors also detail how the WLCG token transition timeline has progressed, and how it has changed since its publication.
97 MATHEMATICS AND COMPUTING↗
WLCG transition from X.509 to Tokens: Progress and Outlook
Since 2017, the Worldwide LHC Computing Grid (WLCG) has been working towards enabling token-based authentication and authorization throughout its entire middleware stack.Taking guidance from the WLCG Token Transition Timeline, published in 2022, substantial progress has been achieved not only in making middleware compatible with the use of tokens, but also in understanding the limitations of the WLCG Common JWT Profiles, first published in 2019. Significant scalability experience has been gained from Data Challenge 2024, during which millions of files were transferred with tokens used as credentials - a significant percentage of the total transfers completed.Besides describing the state of affairs in the transition to tokens, revisions to the WLCG token profile, and the evolving road maps, this contribution also covers the corresponding transition from VOMS-Admin to INDIGO-IAM services, with continuing improvements in terms of functionality as well as deployment.
Dack, Thomas [Rutherford Appleton Laboratory]↗
Interpreting Write Performance of Supercomputer I/O Systems with Regression Models
This work seeks to advance the state of the art in HPC I/O performance analysis and interpretation. In particular, we demonstrate effective techniques to: (1) model output performance in the presence of I/O interference from production loads; (2) build features from write patterns and key parameters of the system architecture and configurations; (3) employ suitable machine learning algorithms to improve model accuracy. We train models with five popular regression algorithms and conduct experiments on two distinct production HPC platforms. We find that the lasso and random forest models predict output performance with high accuracy on both of the target systems. We also explore use of the models to guide adaptation in I/O middleware systems, and show potential for improvements of at least 15% from model-guided adaptation on 70% of samples, and improvements up to 10× on some samples for both of the target systems.
Xie, Bing↗
Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production Load
Scientific computing workloads at HPC facilities have been shifting from traditional numerical simulations to AI/ML applications for training and inference while processing and producing ever-increasing amounts of scientific data. To address the growing need for increased storage capacity, lower access latency, and higher bandwidth, emerging technologies such as non-volatile memory are integrated into supercomputer I/O subsystems. With these emerging trends, we need a better understanding of the multilayer supercomputer I/O systems and ways to use these subsystems efficiently. In this work, we study the I/O access patterns and performance characteristics of two representative supercomputer I/O subsystems. Through an extensive analysis of year-long I/O logs on each system, we report new observations in I/O reads and writes, unbalanced use of storage system layers, and new trends in user behaviors at the HPC I/O middleware stack.
Bez, JL↗
HPC I/O innovations in the exascale era
As high performance computing architecture evolves to deliver ever-increasing performance, the middleware tools also need to adapt in order for applications to better use these higher-performance features. Here, the Adaptable Input Output System (ADIOS), which provides scalable IO performance for exascale HPC applications is one such middleware. During the Exascale Computing Project (ECP), key portions of the ADIOS environment were adapted to respond to ongoing developments in exascale computing and the stresses and opportunities inherent in those changes. This paper examines those changes and where appropriate compares them to pre-exascale implementations.
ADIOS↗
VC3: Virtual Clusters for Community Computation (Final Technical Report)
A traditional HPC computing facility provides a large amount of computing power but has a fixed environment designed to satisfy local needs. This makes it very challenging for users to deploy complex applications that span multiple sites and require specific application software, scheduling middleware, or sharing policies. This project addressed many of these challenges by making it possible for researchers to easily aggregate and share resources, install custom software environments, and deploy clustering frameworks across multiple HPC facilities through the concept of “virtual clusters”. We designed and implemented a prototype virtual cluster facility that enabled unprivileged users to create dynamic aggregations of computing power across multiple sites, deployed with custom middleware and complex software dependencies.
97 MATHEMATICS AND COMPUTING↗
Hamilton: Flexible, Open Source $10 Wireless Sensor System for Energy Efficient Building Operation
Sensors for improving building performance are rapidly populating the market, driven in part by the drive to reduce greenhouse gas emissions resulting from energy production as well as improve the interior environment for healthy and more productive spaces. UC Berkeley has led wireless sensor development over the past 25 years (e.g., Telos mote), with the Hamilton (named after Alexander Hamilton on the US $10 bill) as the most recent. The Hamilton sensor was designed as a low-cost high-performance sensor that is modular and interoperable. The objective of the Hamilton project was to create, evaluate and establish the technological foundations for secure and easy to deploy building energy efficiency applications utilizing pervasive, low-cost wireless sensors integrated with traditional Building Management Systems (BMS), consumer-sector building components, and powerful data analytics. The project included iterative hardware design, incorporating a high-performance database (BTrDb, http://btrdb.io/), creating and iterating the development of secure data middleware (BOSSwave, WAVE/WAVEMQ), working with and pushing the development of an open-source tiny operating system RiotOS, and implementing and improving protocols such as Thread/OpenThread and TCP/IP. The hardware benefited from careful design to drive down the cost; the design included a System-on-a-Chip (SoC), chip antenna, single crystal and five passive components. Careful design of the operating system created a low-power design to enable a long life with small batteries. The hardware included several sensors: temperature, radiant temperature, relative humidity, magnetometer, accelerometer, and light, with an optional occupancy (Passive InfraRed) sensor. The project was the basis of several applications, both internal to the research team and other researchers and professionals at other institutions. Several applications used the sensor hardware as the basis for other complex devices. Other applications used the sensors to improve building performance through interoperating with the building Heating Ventilation and Air-Conditioning (HVAC) system, such as using occupancy and/or distributed temperature sensing to reduce HVAC zone energy while still providing thermal comfort and to reduce peak loads in small commercial buildings. We demonstrated cloud-based energy analytics, implemented a schedule and a Model Predictive Controller in a small commercial building to optimize HVAC energy, occupancy and electricity price. Initial integration of these technological innovations was performed through the creation of execution containers containing the WAVE agent and various driver, proxy, or building system function logic. The research added to the understanding of efficient sensor hardware, secure middleware, time-series data management (high performance database), efficient communication protocols, and interoperating with applications and building systems. The project showed the technical effectiveness and economic feasibility of creating a low-cost, modular, and easy-to-deploy sensor. Through conversations with multiple end users, the research team discovered that many customers wanted data management and services in addition to the sensors. HamiltonIOT developed packages of sensors, border router, and data services to provide a seamless “plug-and-play” sensor deployment. Some customers were willing to pay for higher quality sensors (such as light); some customers wanted a robust enclosure (waterproof).
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
VC3: Virtual Clusters for Community Computation
A traditional HPC computing facility provides a large amount of computing power but has a fixed environment designed to satisfy local needs. This makes it very challenging for users to deploy complex applications that span multiple sites and require specific application software, scheduling middleware, or sharing policies. This project addressed many of these challenges by making it possible for researchers to easily aggregate and share resources, install custom software environments, and deploy clustering frameworks across multiple HPC facilities through the concept of “virtual clusters”. We designed and implemented a prototype virtual cluster facility that enabled unprivileged users to create dynamic aggregations of computing power across multiple sites, deployed with custom middleware and complex software dependencies. This service is hosted at the University of Chicago and available through the site virtualclusters.org.
97 MATHEMATICS AND COMPUTING↗
Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O
In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.
Venkatesh, Ranjan Sarpangala↗
Adaptive Management of Computing and Network Resources for Spacecraft Systems
It is likely that NASA's future spacecraft systems will consist of distributed processes which will handle dynamically varying workloads in response to perceived scientific events, the spacecraft environment, spacecraft anomalies and user commands. Since all situations and possible uses of sensors cannot be anticipated during pre-deployment phases, an approach for dynamically adapting the allocation of distributed computational and communication resources is needed. To address this, we are evolving the DeSiDeRaTa adaptive resource management approach to enable reconfigurable ground and space information systems. The DeSiDeRaTa approach embodies a set of middleware mechanisms for adapting resource allocations, and a framework for reasoning about the real-time performance of distributed application systems. The framework and middleware will be extended to accommodate (1) the dynamic aspects of intra-constellation network topologies, and (2) the complete real-time path from the instrument to the user. We are developing a ground-based testbed that will enable NASA to perform early evaluation of adaptive resource management techniques without the expense of first deploying them in space. The benefits of the proposed effort are numerous, including the ability to use sensors in new ways not anticipated at design time; the production of information technology that ties the sensor web together; the accommodation of greater numbers of missions with fewer resources; and the opportunity to leverage the DeSiDeRaTa project's expertise, infrastructure and models for adaptive resource management for distributed real-time systems.
Pfarr, Barbara↗