SEARCH · Engineering Papers
Results for “Kafka”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Message Brokering Evaluation for Live Spacecraft Telemetry Monitoring, Recorded Playback, and Analysis
Live monitoring and post-flight analysis of telemetry data play a vital role in the development, diagnosis, and deployment of components of a space flight mission. Requirements for such a system include low end-to-end latency between data producers and visualizers, preserved ordering of messages, data stream archiving with random access playback, and real-time creation of derived data streams. We evaluate the RabbitMQ and Kafka message brokering systems, on how well they can enable a real-time, scalable, and robust telemetry framework that delivers telemetry data to multiple clients across heterogeneous platforms and flight projects. In our experiments using an actively developed robotic arm testbed, Kafka yielded a much higher message throughput rate and a consistent publishing rate across the number of topics and consumers. Consumer message rates were consistent across the number of topics but can exhibit bursty behavior with an increase in the contention for a single topic partition with increasing number of consumers.
Toward a persistent event-streaming system for high-performance computing applications
High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.
Arroyo Stream Processing Toolset (arroyopy) v0.1.0
Processing event or streaming data presents several technological challenges. A variety of technologies are often used by scientific user facilities. ZMQ is used to stream data and messages in a peer-to-peer fashion. Message brokers like Kafka, Redis Pubsub, EPICS PVA and RabbitMQ are often employed to route and pass messages from instruments to processing workflows. Arroyopy provides an API and structure to flexibly integrate with these tools and incorporate arbitrarily complex processing workflows, letting the hooks to the workflow code be independent of the connection code and hence reusable at a variety of instruments.
fermi-ad/rust-pubsub-lib
A Rust library for publisher-subscriber interactions. Used to hide the implementation of whatever pub-sub service is in use (e.g., Kafka, Redis, etc.).
Evaluating Function-as-a-Service (FaaS) frameworks for the Accelerator Control System
As particle accelerator control systems evolve in complexity and scale, the need for responsive, scalable, and cost-effective computational infrastructure becomes increasingly critical. Function-as-a-Service (FaaS) offers an alternative to traditional monolithic architecture by enabling event-driven execution, automatic scaling, and fine-grained resource utilization. This paper explores the applicability and performance of FaaS frameworks in the context of a modern particle accelerator control system, with the objective of evaluating their suitability for short lived and triggered workloads. In this paper, we evaluate prominent open-source FaaS platforms in executing functional logic, triggers, and diagnostics routines. Evaluation metrics consist of cold-start latency, scalability, performance, integration with other open-source tools like Kafka. Experimental workloads were designed to simulate real-world control tasks when implemented as stateless FaaS functions. These workloads were benchmarked under various invocation loads and network conditions. Self-hosted FaaS platforms, when deployed within accelerator networks, offer greater control over execution environment, better integration with legacy systems, and support for real-time guarantees when paired with message queues. Based on lessons learned and evaluation metrics, this paper describes reliability of the FaaS framework for the Accelerator Control Systems (ACS).
Rucio at LSST/Rubin
In this presentation, we will explore the Rucio experience with the Rubin Observatory experiment. Our discussion will cover several key areas: Scalability Tests: Insights into the performance and scalability evaluations of Rucio in the context of Rubin's data needs and what we have learned, especially with many small files. Role in Rubin's Data Curation: Rubin's Data Butler: An overview of how Rucio, along with with Rubin's Data Butler using Hermes-K, which involves message passing through Kafka, is integrated in the Rubin's data curation system. Monitoring and Support: Current status of Rucio and PostgreSQL monitoring and Rucio deployment and support within the Rubin environment. Tape RSE Implementation: Deal with the order of magnitude more files going to tape than HEP. Future Needs: An examination of Rubin's evolving requirements for Rucio services and how we plan to address them.
Achieving Fast Operational Intelligence in NASA's Deep Space Network Through Complex Event Processing
NASA’s Deep Space Network (DSN) is a complex, global project, in which the expertise of human operators remain crucial for its successful operation. To find ways to save costs in operations and to improve its services, a number of modernization efforts are underway in the DSN. One such effort is a research and technology development task at the Jet Propulsion Laboratory that is investigating the use of complex event processing (CEP) for intelligent assessment of situations, trend analysis, and advanced automation. The technology leverages the significant business intelligence (BI) and data science advancements made in the enterprise industries over the last several years. The open source big data processing engine Apache SparkTM and the high-throughput, distributed messaging system Apache Kafka form the core of the DSN Complex Event Processing (DCEP) framework. This paper discusses the system engineering perspective of why achieving efficient, lower-cost operations in the DSN is a challenging problem, how the DCEP system handles the use cases that help realize intelligent operations, and how this solution fits into the overall model of the planned DSN Follow-the- Sun Operations (FtSO).
Pulse-length dependence of the laser-induced damage behavior of a fused-silica antireflective metasurface
A broadband, antireflective metasurface optic on a silica substrate is subjected to laser-induced damage-threshold measurements to quantify its performance under exposure to high-intensity/fluence laser pulses in the near-infrared at four pulse durations, ranging from 20 fs to 1.4 ns. The performance of the metasurface is benchmarked against that obtained from an equivalent bare fused-silica substrate that did not receive reactive-ion-etching metasurface treatment. Results showed that the damage threshold of the antireflective metasurface was always lower than the input-surface damage threshold of the untreated substrate. The damage initiations with nanosecond and picosecond pulses resulted in localized modification and removal of the nanostructures, whereas the onset of laser-induced modification with 20-fs pulses in a vacuum environment manifested as changes in the optical and electronic properties without significant material removal. The broader goal of this work is to develop a preliminary understanding of the laser-induced failure mechanisms of silica-based metasurface optics.
Asymptotic inconsistency of the cumulative algorithm for laser-induced damage probability analysis
The “cumulative algorithm” is a data analysis method that has been proposed to provide an objective, nonparametric determination of laser-induced damage probability as a function of fluence from experimental data that contain both damaged sites and undamaged sites (i.e., 1-on-1 or S-on-1 testing protocols). In this work, the limitations of this approach are explored by considering the asymptotic limit of a large number of test sites. It is shown that the cumulative algorithm does not converge to the true probability distribution and significantly underestimates the damage probability near the damage onset. Here, based on the results of this work, the cumulative algorithm is not recommended for accurate estimation of damage probability.
Characterization of ablator dynamics initiated by picket-pulse conditions
Direct-drive laser-fusion targets may include low-cost plastic ablator material. Here, this work describes a tabletop experimental system designed to enable the investigation of the interaction of such ablator material as a function of the laser wavelength and the associated excitation mechanism. This experimental system offers time-resolved imaging of the material response with subpicosecond temporal resolution and ≈1-μm spatial resolution. The system involves synchronized femtosecond and picosecond lasers to initiate and characterize the dynamics of polystyrene planar targets irradiated by conditions similar to a “picket” prepulse from a direct-drive pulse shape. Ultrafast imaging of the initial excitation process followed by the resulting critical plasma formation, shock-wave propagation, and blowoff plasma expansion is used to compare the impact of excitation wavelength, 355 nm or 266 nm, on the ensuing dynamics. This work aims to investigate fundamental laser material interactions to help advance physics models that can inform the development of laser driver and target designs.
Experimental and numerical evaluation of laser shinethrough in polystyrene at 3 ω and 4 ω laser frequencies
In laser-direct-drive inertial confinement fusion (ICF), laser shinethrough is the process associated with laser propagation through the target plastic ablator, polystyrene is often used in ICF experiments, and leads to energy deposition in the target until the critical density is established at the ablator front surface. This process causes material modifications that may change the shock propagation and lead to density and pressure perturbations at the interface between the polystyrene and cryogenic deuterium-tritium (DT) layer. This work reports on experimental observations and numerical investigations of laser shinethrough at the third and the fourth harmonic of ICF class laser systems (1 ω corresponds to a wavelength of ∼1.06 μm ). The experiments evidence that, at 4 ω , the laser is absorbed over a distance of about a few microns and laser shinethrough is completely eliminated in contrast with the behavior at 3 ω . These experimental observations are reproduced by numerical simulations. The laser fluence required to reach the critical density at the ablator front surface is also found to be more than one order of magnitude higher at the third harmonic. Finally, modeling suggests that the back surface of the ablator exhibits ion temperatures greater than the melting temperature of cryogenic DT and electron pressure larger than the bulk and shear moduli of cryogenic DT for the case of using a 3 ω laser.
Scalable waveplate for ultraviolet applications using laser-imprinted birefringence in silica
A method for fabrication of large-aperture waveplate suitable for operation in the near-ultraviolet spectral region is demonstrated. The waveplate was fabricated using femtosecond direct laser writing of bulk fused silica. This method enables the precise control of the induced birefringence, which is introduced via the generation of nanopores with geometrical features that are controlled by the laser processing parameters. The resulting structures also introduce a small loss of transmission that is on the order of 4% at 351 nm. The laser-induced–damage threshold in the nanosecond regime was found to approach that of the native fused-silica substrate, which makes it a suitable candidate for polarization control in laser fusion research applications.
Laser-induced damage-growth threshold at 351 nm in the nanosecond regime in dielectric coatings employing different high-index materials
The laser-damage performance of HfO 2 /SiO 2 and Al 2 O 3 /SiO 2 mirrors was investigated at a laser wavelength of 351 nm with nanosecond pulse durations. A method was developed to quantify the laser-induced damage-growth threshold under conditions relevant to high-repetition-rate, large-aperture laser systems. The results suggest that the damage-growth performance cannot be inferred from the corresponding damage-initiation threshold that is commonly used to evaluate the performance of optical materials. Here, a systematic difference was also observed in the performance between these two groups of high-index materials, with the Al 2 O 3 /SiO 2 mirrors exhibiting about twofold improvement in the damage-growth threshold.
Investigation of a gallium alloy liquid metal mirror for application in inertial fusion energy systems
The final optic in future inertial fusion energy plants must maintain high performance and survivability under gigashot laser irradiation while exposed to fusion byproducts. Although no practical solution has been demonstrated, a self-healing final turning mirror based on a grazing-incidence liquid metal mirror has been theoretically explored. This work investigates the suitability of a gallium alloy. The experimental system simulates key conditions, including vacuum exposure and 10-Hz, 355-nm, and 6-ns laser pulses. Experiments address reflectivity, damage, and response as a function of pressure. Results highlight the potential of liquid metal gallium alloys for robust final optics in laser-based fusion power plants and inform non-traditional optics for intense laser systems.
Inequities Between Security Infrastructure and Performance Increases Presentation
Explore the source record for details and available documents.
JT9D engine diagnostics. Task 2: Feasibility study of measuring in-service flight loads
The feasibility of measuring JT9D propulsion system flight inertia loads on a 747 airplane is studied. Flight loads background is discussed including the current status of 747/JT9D loads knowledge. An instrumentation and test plan is formulated for an airline-owned in-service airplane and the Boeing-owned RA001 test airplane. Technical and cost comparisons are made between these two options. An overall technical feasibility evaluation is made and a cost summary presented. Conclusions and recommendations are presented in regard to using existing inertia loads data versus conducting a flight test to measure inertia loads.