Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Real Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science

ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.

59 BASIC BIOLOGICAL SCIENCES↗

Design and Testing of the Address in Real-Time Data Driver Card for the Micromegas Detector of the ATLAS New Small Wheel Upgrade

The Address in Real Time Data Driver Card (ADDC) is designed to transmit the trigger data in the Micro-Mesh Gaseous Structure (Micromegas) detector of the ATLAS New Small Wheel (NSW) upgrade. The Address in Real Time (ART) signals are generated by the front-end ASIC, named VMM chip, to indicate the address of the first above-threshold event. A custom ASIC (ART ASIC) is designed to receive the ART signals from the VMM chip and implement the hit-selection. The processed data from the ART ASIC will be transmitted out of the NSW through the GBTx serializer, the unidirectional Versatile Twin Transmitter (VTTx) and fiber optical links. The ART signal is critical for the ATLAS experiment trigger selection thus the performance and stability of the ADDC is very important. To ensure extensive testing of the ADDC, an FPGA Mezzanine Card (FMC) based testing platform and a special designed firmware/software are developed. We find that this test platform works with the commercial Xilinx VC707 FPGA development kit, independent of the other electronics of the NSW it can test all the functions of the ADDC and its long-term stability. This paper will introduce the design, testing procedure and results of the ADDC and the FMC testing platform.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Deep Learning At Depth: Estimating subsurface parameters from geophysical monitoring data

Geophysical imaging techniques are a non-invasive way to image the subsurface and understand both subsurface solid (rock/soil) and fluid property distributions and their evolution in time. Inversions of the geophysical data, such as Electrical Resistance Tomography (ERT) data, are solved to estimate the subsurface property distributions, such as conductivity, and many inversion techniques smooth out sharp gradients in rock or fluid property distributions. Sharp gradients in subsurface properties tend to be present in situations with complex subsurface structures, which are common in many subsurface applications. We have successfully demonstrated that it is possible to inform, or constrain, inversions with neural networks trained on synthetic data with complex subsurface structures. Initial results suggest this process may be optimizable to yield property distributions that better represent the true property distributions than the same inversion process without the neural network constraint. Future work would optimize the neural network performance for this application and then apply the synthetic-data trained neural network to real data to understand the utility and performance of this technique for real data sets.

47 OTHER INSTRUMENTATION↗

Real-time data analysis for medical diagnosis using FPGA-accelerated neural networks

Real-time analysis of patient data during medical procedures can provide vital diagnostic feedback that significantly improves chances of success. With sensors becoming increasingly fast, frameworks such as Deep Neural Networks are required to perform calculations within the strict timing constraints for real-time operation. However, traditional computing platforms responsible for running these algorithms incur a large overhead due to communication protocols, memory accesses, and static (often generic) architectures. In this work, we implement a low-latency Multi-Layer Perceptron (MLP) processor using Field Programmable Gate Arrays (FPGAs). Unlike CPUs and Graphics Processing Units (GPUs), our FPGA-based design can directly interface sensors, storage devices, display devices and even actuators, thus reducing the delays of data movement between ports and compute pipelines. Moreover, the compute pipelines themselves are tailored specifically to the application, improving resource utilization and reducing idle cycles. We demonstrate the effectiveness of our approach using mass-spectrometry data sets for real-time cancer detection. We demonstrate that correct parameter sizing, based on the application, can reduce latency by 20% on average. Furthermore, we show that in an application with tightly coupled data-path and latency constraints, having a large amount of computing resources can actually reduce performance. Using mass-spectrometry benchmarks, we show that our proposed FPGA design outperforms both CPU and GPU implementations, with an average speedup of 144x and 21x, respectively. In our work, we demonstrate the importance of application-specific optimizations in order to minimize latency and maximize resource utilization for MLP inference. By directly interfacing and processing sensor data with ultra-low latency, FPGAs can perform real-time analysis during procedures and provide diagnostic feedback that can be critical to achieving higher percentages of successful patient outcomes.

60 APPLIED LIFE SCIENCES↗

Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

Abstract We present Lightcurve Anomaly Identification and Similarity Search ( LAISS ), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1–5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes ( spectroscopic anomalies), of uncommon host galaxy environments ( contextual anomalies), and of peculiar or interaction-powered phenomena ( behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.

79 ASTRONOMY AND ASTROPHYSICS↗

Near Real-time Data Analysis of Core-collapse Supernova Simulations with Bellerophon

We present an overview of a software system, Bellerophon, built to support a production-level HPC application called CHIMERA, which simulates core-collapse supernova events at the petascale. Developed over the last four years, Bellerophon enables CHIMERA's geographically dispersed team of collaborators to perform data analysis in near real-time. Its n-tier architecture provides an encapsulated, end-to-end software solution that enables the CHIMERA team to quickly and easily access highly customizable animated and static views of results from anywhere in the world via a web- deliverable, cross-platform desktop application. In addition, Bellerophon addresses software engineering tasks for the CHIMERA team by providing an automated mechanism for performing regression testing on a variety of supercomputing platforms. Elements of the team's workflow management needs are met with software tools that dynamically generate code repository statistics, access important online resources, and monitor the current status of several supercomputing resources.

97 MATHEMATICS AND COMPUTING↗

Near Real-time Data Analysis of Core-Collapse Supernova Simulations With Bellerophon

We present an overview of a software system, Bellerophon, built to support a production-level HPC application called CHIMERA, which simulates core-collapse supernova events at the petascale. Developed over the last four years, Bellerophon enables CHIMERA s geographically dispersed team of collaborators to perform data analysis in near real-time. Its n-tier architecture provides an encapsulated, end-to-end software solution that enables the CHIMERA team to quickly and easily access highly customizable animated and static views of results from anywhere in the world via a web-deliverable, cross-platform desktop application. In addition, Bellerophon addresses software engineering tasks for the CHIMERA team by providing an automated mechanism for performing regression testing on a variety of supercomputing platforms. Elements of the team s workflow management needs are met with software tools that dynamically generate code repository statistics, access important online resources, and monitor the current status of several supercomputing resources.

Lingerfelt, Eric J↗

Get Non-Real: Randomized Sketching for High-Dimensional Non-Real Valued Data (Final Report)

In our final report for DE-C0022186, we describe the work we did on this grant towards the goals we proposed. Our first goal was characterizing fundamental limits for sketching of discrete high-dimensional matrices with low-dimensional structures. Our second main goal was designing algorithms for data reconstruction from sketches. We focus on approaches that are either specifically designed for non-real-valued data (binary, finite field) or that will translate more readily to that setting.

97 MATHEMATICS AND COMPUTING↗

LANL12-RS-108J Report on Device Modeler Testing of the Device Modeler Tool Kit. DMTK in FY14

This document covers the various testing and modifications of the Device Modeler Tool Kit (DMTK) for project LANL12-RS-108J in FY14. The testing has been comprised of different device modelers and trainees for device modeling using DMTK on the secure network for a few test problems. Most of these problems have been synthetic data problems. There has been a local secure network training drill where one of the trainees has used DMTK for real data. DMTK has also been used on a laptop for a deployed real data training drill. Once DMTK gets into the home team, it will be used for more training drills (TDs) which will contain real data in the future.

97 MATHEMATICS AND COMPUTING↗

Real-time data processing for serial crystallography experiments

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

47 OTHER INSTRUMENTATION↗

Medium Truck Duty Cycle Data from Real-World Driving Environments: Project Interim Report

Since the early part of the 20th century, the US trucking industry has provided a safe and economical means of moving commodities across the country. At the present time, nearly 80% of the US domestic freight movement involves the use of trucks. The US Department of Energy (DOE) is spearheading a number of research efforts to improve heavy vehicle fuel efficiencies. This includes research in engine technologies (including hybrid and fuel cell technologies), lightweight materials, advanced fuels, and parasitic loss reductions. In addition, DOE is developing advanced tools and models to support heavy vehicle truck research, and is leading the 21st Century Truck Partnership whose stretch goals involve a reduction by 50% of the fuel consumption of heavy vehicles on a ton-mile basis. This Medium Truck Duty Cycle (MTDC) Project is a critical element in DOE s vision for improved heavy vehicle energy efficiency and is unique in that there is no other national database of characteristic duty cycles for medium trucks. It involves the collection of real-world data for various situational characteristics (rural/urban, freeway/arterial, congested/free-flowing, good/bad weather, etc.) and looks at the unique nature of medium trucks drive cycles (stop-and-go delivery, power takeoff, idle time, short-radius trips), to provide a rich source of data that can contribute to the development of new tools for fuel efficiency and modeling, provide DOE a sound basis upon which to make technology investment decisions, and provide a national archive of real-world-based medium-truck operational data to support heavy vehicle energy efficiency research. The MTDC project involves a two-part field operational test (FOT). For the Part-1 FOT, three vehicles, each from two vocations (urban transit and dry-box delivery) were instrumented for one year of data collection. The Part-2 FOT will involve the towing/recovery and utility vocations. The vehicles participating in the MTDC project are doing so through gratis partnerships in return for early access to the results of this study. Partnerships such as these are critical to FOTs in which real-world data is being collected. In Part 1 of the project, Oak Ridge National Laboratory(ORNL) established partnerships with the H.T. Hackney Company, one of the largest wholesale distributors in the country, distributing products to 21 states; and with the Knoxville Area Transit (KAT), the City of Knoxville s transit system, operating services across the city of Knoxville and parts of Knox co. These partnerships and agreements provided ORNL access to three Class-7 2005/2007 International day-cab tractors, model 8600, which regularly haul 28 ft pup trailers (H.T. Hackney Co) and three Class-7 2005 Optima LF-34 buses (KAT), for collection of duty cycle data. In addition, ORNL has collaborated with the Federal Motor Carrier Safety Administration (FMCSA) to determine if there were possible synergies between this duty cycle data collection effort and FMCSA s need to learn more about the operation and duty cycles of the second-largest fuel consuming commercial vehicle category in the US. FMCSA s primary interest was in collecting safety data relative to the driver, carrier, and vehicle. In order to collect the duty cycle and safety-related data, ORNL developed a data acquisition and wireless communication system that was placed on each test vehicle. Each signal recorded in this FOT was collected by means of one of the instruments incorporated into each data acquisition system (DAS). Native signals were obtained directly from the vehicle s J1939 and J1708 data buses. A VBOX II Lite collected Global Positioning System related information including speed, acceleration, and spatial location information at a rate of 5 Hz, and communicated this data via the CAN (J1939) protocol. The Air-Weigh LoadMaxx, a self-weighing system which determines the vehicle s gross weight by means of pressure transducers and posts the weight to the vehicle s J1939 data bus, was used to collect vehicle payload information. A cellular modem, the Raven X EVDO V4221, facilitated the communication between the eDAQ-lite (the data collection engine of the system) and the user, via the internet. The modem functioned as a wireless gateway, allowing data retrievals and system checks to be performed remotely. Also, and in partnership with FMCSA, two additional safety sensors were installed on the combination vehicles: the MGM e-Stroke brake monitoring system and the Tire SafeGuard tire pressure monitoring system. Both of these systems posted data to the J1939 data bus, enabling these signals to be read without any additional DAS interface hardware. Seventy-three signals from the different deployed sensors and available vehicle systems were collected. Because of the differences in vehicle data buses (i.e., J1939 and J1708 data buses), not all desired signals were available for both types of vehicles.

duty cycles↗

Hypothesis-Agnostic Network-Based Analysis of Real-World Data Suggests Ondansetron is Associated with Lower COVID-19 Any Cause Mortality

Background: The COVID-19 pandemic generated a massive amount of clinical data, which potentially hold yet undiscovered answers related to COVID-19 morbidity, mortality, long-term effects, and therapeutic solutions.Objectives: The objectives of this study were (1) to identify novel predictors of COVID-19 any cause mortality by employing artificial intelligence analytics on real-world data through a hypothesis-agnostic approach and (2) to determine if these effects are maintained after adjusting for potential confounders and to what degree they are moderated by other variables.Methods: A Bayesian statistics-based artificial intelligence data analytics tool (bAIcis®) within the Interrogative Biology® platform was used for Bayesian network learning and hypothesis generation to analyze 16,277 PCR+ patients from a database of 279,281 inpatients and outpatients tested for SARS-CoV-2 infection by antigen, antibody, or PCR methods during the first pandemic year in Central Florida. This approach generated Bayesian networks that enabled unbiased identification of significant predictors of any cause mortality for specific COVID-19 patient populations. These findings were further analyzed by logistic regression, regression by least absolute shrinkage and selection operator, and bootstrapping.Results: We found that in the COVID-19 PCR+ patient cohort, early use of the antiemetic agent ondansetron was associated with decreased any cause mortality 30 days post-PCR+ testing in mechanically ventilated patients.Conclusions: The results demonstrate how a real-world COVID-19-focused data analysis using artificial intelligence can generate unexpected yet valid insights that could possibly support clinical decision making and minimize the future loss of lives and resources.

60 APPLIED LIFE SCIENCES↗

Local Weather Station Design and Development for Cost-Effective Environmental Monitoring and Real-Time Data Sharing

Current weather monitoring systems often remain out of reach for small-scale users and local communities due to their high costs and complexity. This paper addresses this significant issue by introducing a cost-effective, easy-to-use local weather station. Utilizing low-cost sensors, this weather station is a pivotal tool in making environmental monitoring more accessible and user-friendly, particularly for those with limited resources. It offers efficient in-site measurements of various environmental parameters, such as temperature, relative humidity, atmospheric pressure, carbon dioxide concentration, and particulate matter, including PM 1, PM 2.5, and PM 10. The findings demonstrate the station’s capability to monitor these variables remotely and provide forecasts with a high degree of accuracy, displaying an error margin of just 0.67%. Furthermore, the station’s use of the Autoregressive Integrated Moving Average (ARIMA) model enables short-term, reliable forecasts crucial for applications in agriculture, transportation, and air quality monitoring. Furthermore, the weather station’s open-source nature significantly enhances environmental monitoring accessibility for smaller users and encourages broader public data sharing. With this approach, crucial in addressing climate change challenges, the station empowers communities to make informed decisions based on real-time data. In designing and developing this low-cost, efficient monitoring system, this work provides a valuable blueprint for future advancements in environmental technologies, emphasizing sustainability. The proposed automatic weather station not only offers an economical solution for environmental monitoring but also features a user-friendly interface for seamless data communication between the sensor platform and end users. This system ensures the transmission of data through various web-based platforms, catering to users with diverse technical backgrounds. Furthermore, by leveraging historical data through the ARIMA model, the station enhances its utility in providing short-term forecasts and supporting critical decision-making processes across different sectors.

54 ENVIRONMENTAL SCIENCES↗

PvaPy streaming framework for real-time data processing

User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.

EPICS pvAccess↗

Improved tomographic reconstruction of large-scale real-world data by filter optimization

In advanced tomographic experiments, large detector sizes and large numbers of acquired datasets can make it difficult to process the data in a reasonable time. At the same time, the acquired projections are often limited in some way, for example having a low number of projections or a low signal-to-noise ratio. Direct analytical reconstruction methods are able to produce reconstructions in very little time, even for large-scale data, but the quality of these reconstructions can be insufficient for further analysis in cases with limited data. Iterative reconstruction methods typically produce more accurate reconstructions, but take significantly more time to compute, which limits their usefulness in practice. In this paper, we present the application of the SIRT-FBP method to large-scale real-world tomographic data. The SIRT-FBP method is able to accurately approximate the simultaneous iterative reconstruction technique (SIRT) method by the computationally efficient filtered backprojection (FBP) method, using precomputed experiment-specific filters. We specifically focus on the many implementation details that are important for application on large-scale real-world data, and give solutions to common problems that occur with experimental data. We show that SIRT-FBP filters can be computed in reasonable time, even for large problem sizes, and that precomputed filters can be reused for future experiments. Reconstruction results are given for three different experiments, and are compared with results of popular existing methods. The results show that the SIRT-FBP method is able to accurately approximate iterative reconstructions of experimental data. Furthermore, they show that, in practice, the SIRT-FBP method can produce more accurate reconstructions than standard direct analytical reconstructions with popular filters, without increasing the required computation time.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

A Digital Twin for an Inverter-Based Resource Power Plant: Real-time data streaming unlocks situation awareness

Here, this study presents the development and successful implementation of a digital twin specifically designed for a grid-connected IBR power plant. By integrating a reduced-order model of the IBR system and dynamically updating the grid impedance with real-time data, the digital twin effectively captures and replicates the behavior of the physical system. Its accuracy and reliability are validated through critical test scenarios, including a three-phase fault and a line-tripping event. The results confirm that the digital twin closely emulates its physical counterpart, demonstrating its strong potential for real-time analysis, system monitoring, and predictive decision making in modern power systems.

Digital twins↗

Real-time Data Center Energy Efficiency At Pacific Northwest National Laboratory

The escalating consumption of power in data centers worldwide has brought the issue of data center energy efficiency to the forefront. Data center owners and operators now regard detailed knowledge of the energy efficiencies of their data centers as a competitive advantage. With funding from the Department of Energy (NNSA), PNNL has undertaken an in-depth analysis of the real-time energy efficiency for its Energy Smart Data Center Test Bed(ESDC-TB), which is housed in the mixed-use EMSL. The analysis is centered around the real-time display of The Green Grid’s proposed DCiE metric. To calculate this metric, PNNL relies on a variety of sources of data. At the ESDC-TB level, the data center is instrumented to the 100% level (all power consumption, and water temperatures and flow rates are measured). Most of this data is monitored in real-time, but the exception to this is with the CRAHs, which rely on a one-time power consumption measurement for the blowers (these are single speed blowers, so a one-time measurement suffices.). Outside of the data center (EMSL facility level), PNNL relies on the following: • Real-time data from the entire chiller plant (five chillers), six chilled water pumps, and one of four cooling towers (blowers only). • One-time power measurements for a single fixed speed pump that is representative of each grouping of pumps (the other pumps are assumed to possess the same power consumption levels). • One-time power measurements for a single two-speed cooling tower blower. This same blower model is deployed in three of the four cooling towers, so is assumed to be representative for all these blowers. • One-time power measurements for a single fixed speed cooling tower pump. This same pump model is deployed in all four cooling towers, so is assumed to be representative for all these pumps. A software tool named FRED was developed by PNNL to acquire, reduce, display, and archive all the data acquired from the ESDC-TB and EMSL. FRED provides the ability to display various levels of real-time data starting at the ESDC-TB and EMSL levels, then to lower levels as desired. For example, for the ESDC-TB, graphical screens are provided at the data center level, the rack level, the server level, and even the component level. In the near-term, FRED will also display the real-time DCiE. One of the major challenges to doing this in a mixed-use facility has been to quantify the power consumption of each major mechanical or electrical subsystem that is attributable to the IT equipment housed within the ESDC-TB (i.e., NW-ICE). PNNL has tackled this issue for its five chiller plant, and the analysis is presented in the paper. The analysis for all the remaining mechanical and electrical subsystems is now underway and will be presented in future publications.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗