Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Using heterogeneous data to identify signatures of dengue outbreaks at fine spatio-temporal scales across Brazil

Dengue virus remains a significant public health challenge in Brazil, and seasonal preparation efforts are hindered by variable intra- and interseasonal dynamics. Here, we present a framework for characterizing weekly dengue activity at the Brazilian mesoregion level from 2010–2016 as time series properties that are relevant to forecasting efforts, focusing on outbreak shape, seasonal timing, and pairwise correlations in magnitude and onset. In addition, we use a combination of 18 satellite remote sensing imagery, weather, clinical, mobility, and census data streams and regression methods to identify a parsimonious set of covariates that explain each time series property. The models explained 54% of the variation in outbreak shape, 38% of seasonal onset, 34% of pairwise correlation in outbreak timing, and 11% of pairwise correlation in outbreak magnitude. Regions that have experienced longer periods of drought sensitivity, as captured by the “normalized burn ratio,” experienced less intense outbreaks, while regions with regular fluctuations in relative humidity had less regular seasonal outbreaks. Both the pairwise correlations in outbreak timing and outbreak trend between mesoresgions were best predicted by distance. Our analysis also revealed the presence of distinct geographic clusters where dengue properties tend to be spatially correlated. Forecasting models aimed at predicting the dynamics of dengue activity need to identify the most salient variables capable of contributing to accurate predictions. Our findings show that successful models may need to leverage distinct variables in different locations and be catered to a specific task, such as predicting outbreak magnitude or timing characteristics, to be useful. This advocates in favor of “adaptive models” rather than “one-size-fits-all” models. The results of this study can be applied to improving spatial hierarchical or target-focused forecasting models of dengue activity across Brazil.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 2 Sensor Data v2-1

This is the version v2-1 Level 2 (L2) data release for COMPASS-FME environmental sensors located at our Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in MD, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments. Level 2 (L2) data consist of sensor observations from the COMPASS-FME synoptic sites, TEMPEST, and DELUGE. Compared to the L1 data, these are more consistent (always 15-minute timestamps for the entire year); better QA/QC’d (out of bounds, out of service, and extreme outlier values are removed); and more complete, with a gap-filled time series available alongside the main observations, and additional derived (calculated) variables. L2 data are intended to be rapidly and easily usable in analyses and simulations. However, algorithmic outlier identification always carries the risk of removing valid data, and Level 1 data may be more suitable for analyses that focus on variability or extreme events. This dataset includes: - An overall dataset README file that describes the current version, gives citation and contact information, etc. - Site- and year-specific folders, each holding variable-specific Parquet (a high performance, space efficient format; see https://parquet.apache.org) data files for each site and plot in that year. - Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, detailed flood times, as well as a general description of the site. - Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are reported every 15 minutes. Please see v2-1 TEMPEST L2 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning. Data files are in Apache Parquet, a high performance, space efficient format for tabular data. These files can be read using R's `arrow` package (https://arrow.apache.org/docs/r/), with similar tools available in other languages. The TEMPEST flood events occurred on the following dates. They lasted for ~10 hours each day and delivered ~80,000 gallons to each plot; many data streams are available at 1 or 5 minute frequency during these periods. * Tests: Aug 25 (fresh plot) and Sep 9 (salt plot), 2021 * TEMPEST 1: June 22, 2022 * TEMPEST 2: June 6-7, 2023 * TEMPEST 3: June 11-13, 2024

EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATU↗
DOE OSTI2026

Machine learning at the Spallation Neutron Source accelerator and target

We describe the ongoing efforts to apply Machine Learning techniques to improve the performance of our accelerator and target. Specially, we are looking to minimize halo beam losses in the absence of a proper physics model, automatically detect and log anomalies in the target support systems such as cooling, and detect and prevent errant beam pulses in the linac. We also describe the infrastructure we use to acquire and stream data to the GPU cluster for training, our code development cycle, and edge computing for model inference. To minimize halo beam losses, we use a Reinforcement Learning technique tested on a virtual accelerator. The target anomaly detection is trained on archived data using incomplete physics models and is made part of the existing target reporting system. The errant beam prevention analyzes beam current and beam phase waveforms as well as accelerator configuration data to predict errant pulses. We also develop continual learning to adapt to changes in the accelerator.

Accelerator Physics↗
DOE OSTI2020

NRC Multiphysics Analysis Capability Deployment FY2020 - Part 4

This report details progress and activities of Idaho National Laboratory (INL) on the Nuclear Regulatory Commission (NRC) project “Development and Modeling Sup- port for Advanced Non-Light Water Reactors.” The tasks completed for this report are: • Task 3a: This task demonstrates the effectiveness of Griffin’s methods for interpolating cross sections and SPH correction factors as a function of control drum rotation. The Monte Carlo code Serpent is used to generate reference results. The accuracy of Griffin-calculated eigenvalues and reaction rates are studied as a function of control drum rotation for cross section and SPH factor libraries of varying fidelity. Higher-fidelity libraries require less interpolation and are therefore inherently more accurate; however, they require more computational resources to create. Overall, Griffin is able to accurately model eigenvalue and reaction rates over the full range of drum rotation using a library generated from three to five discrete drum rotations for a particular microreactor design. • Task 3c: This preliminary task successfully demonstrates the performance of a coupled neutron-photon transport calculation with Griffin using the spherical harmonics approximation. The Griffin solutions are compared to reference solutions from the continuous energy Monte Carlo codes MCNP and Serpent. Furthermore, this work also demonstrates how to prepare the neutron and photon libraries using NJOY, MCNP, and Serpent with the latest ENDF/B-VIII data for 235 U and graphite. The Griffin total energy deposition in the active core region is calculated within 0.3% of the reference, but it degrades in the reflector regions with a maximum difference of -1.5%. Overall, the results look promising and future analysis should use an improved data stream and focus on determining the potential cancellation of error within each energy group.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

Supporting Cyber Security of Power Distribution Systems by Detecting Differences Between Real-time Micro-Synchrophasor Measurements and Cyber-Reported SCADA (Final Report)

As modern power grids tend towards greater levels of automation and communication, the challenges of identifying and mitigating vulnerabilities to cyber-attacks are ones that are increasingly demanding attention. Today’s power system has evolved to form the foundational bedrock of modern society, and an attack on this infrastructure could prove disastrous. In this project we were tasked to investigate the use of distribution synchrophasors as an independent isolated sensor network with which we can corroborate, or flag potentially spoofed,Supervisory Control And Data Acquisition (SCADA) data. We adapted an approach to marry the underlying physical properties of power systems with the network communications used by power systems in order to offer insights unattainable by either data stream isolation. While the concept of intrusion detection systems (IDS) is well understood for monitoring network traffic and traditional IT computing systems, the approach discussed in this report is motivated by several key notions: first, current SCADA communications alone presents an incomplete view of the grid. Second, the power grid, and the equipment controlling it, is grounded by laws of physics. Given this, we leverage high-frequency physical grid measurements to understand the physical condition of the grid, and combine this with SCADA. While high-frequency physical grid measurements and SCADA communication over Internet Protocol (IP) networks are fundamentally disparate information sources, when collectively examined through appropriate lenses, they offer a much more nuanced depiction of the grid.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

The Grey Zone Test Range Integrated Urban Simulation Environment

Sandia National Laboratories is part of the government test and evaluation team for the Defense Advanced Research Projects Agency Collection and Monitoring via Planning for Active Situational Scenarios program. The program is designed to better understand competition in the area between peace and conventional conflict when adversary actions are subtle and difficult to detect. For the purposes of test and evaluation, Sandia conducted a range of activities for the program: creation of the Grey Zone Test Range; design of the data stream for a user experiment conducted with U.S. Indo-Pacific Command; design, implementation, and execution of the formal evaluation; and analysis and summary of the evaluation results. This report details Sandia's activities and provides additional information on the Grey Zone Test Range urban simulation environment developed to evaluate the performer technologies.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2021

A Grand Challenge "Uncertainty Project" to Accelerate Advances in Earth System Predictability: AI-Enabled Concepts and Applications

This proposal is emerging from GISS ModelE3 ESM development in the area of cloud physics, so we begin with an example of research needs/gaps from that work. Here, some of our greatest development concerns arise where we lack fundamental process-level understanding, as in ice formation. Namely, it is currently unclear what is the main process that is forming the majority of ice crystals in commonly occurring convection, apparently via secondary ice production at warm temperatures. We are keenly awaiting laboratory data for candidate mechanisms, which is not yet in hand to crucially establish their efficiency. Our progress is also hampered by a lack of uncertainty characterization in currently available measurements of ice crystal number size distributions. Furthermore, the same multiplication process may be responsible for a majority of ice crystals in many extratropical mixed-phase clouds, whose variable representation in CMIP6 ESMs may be a leading cause of differences in cloud phase feedback and ECS. Yet we have been required to deliver an ESM with the cloud physics knowledge at hand. The proposed grand challenge project is AI-enabled via application of machine learning (ML) to climate model and observational data streams (focal area 3), and applications include AI-guided observing system design and model/component/parameterization selection (areas 1 and 2). The project is structurally agnostic as to whether model or observing system components use AI approaches or not, but uncertainties must be estimated and propagatable in both.

58 GEOSCIENCES↗
DOE OSTI2021

A new era of observationally-infused E3SM: GANs for unifying imagery archives

This paper presents an idea to develop a “Rosetta Stone” for unifying observations from various satellite or remote sensors into a common format that would vastly advance our ability to exploit existing datasets for improving predictability within Earth System Models (ESMs). While the applications of such a unified archive are broad, we believe it will be a critical step toward ushering in a new generation of ESMs that are richly informed, guided by, and validated by extensive observational data. With the vast quantity of both remotely-sensed and in-situ data streams available and coming online, new approaches are needed that can harmonize and thus fully exploit these expensive datasets. While we present the broader idea, we point to examples of applications that impact the water cycle and its representation in ESMs.

58 GEOSCIENCES↗
DOE OSTI2021

Open Radiation Monitoring: Histogram Builder Module Design

The Open Radiation Monitoring Project seeks to develop and demonstrate a modular radiation detection architecture designed specifically for use in arms control treaty verification (ACTV) applications that will facilitate rapid development of trusted systems to meet the needs of potential future treaties. A modular architecture can be used to reduce more complex systems to a series of single purpose building blocks, thereby facilitating equipment inspection and in turn building trust in the equipment by all treaty parties. Furthermore, a modular architecture can be used to control data flow within the measurement system, reducing the risk of "hidden switches" and constraining the amount of sensitive information that could potentially be inadvertently leaked. This report details the first revision of a prototype circuit that will convert analog pulses directly into a histogrammed data set for further processing. The circuit was designed with both spectroscopy and multiplicity analysis in mind but can, in principle, be used to reduce any raw data stream into a histogram. The number of output channels is limited, and the histogram bin ranges are user configurable to allow for non-uniform and discontinuous bins, which makes it possible to restrict the information being passed down stream if desired. Pulse processing relies entirely on analog circuitry and non- programmable logic, which enables operation without the need for a central processor or other programmable control unit. The circuit remains untested under the Open Radiation Monitoring project due to the closure of the sponsoring program. However, further development and testing is scheduled to take place in support of a purpose-built trusted verification system development effort known as COGNIZANT, which demonstrates the potential benefit of developing a suite of modular trusted system components.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2021

The Building Adapter: Automatic Mapping of Commercial Buildings for Scalable Building Analytics

This project creates new solutions for the manual metadata mapping problem: the costly process of creating a match between a building’s sensor data streams and the inputs of a building analytics engine. This goal is achieved by creating and improving techniques for metadata inference: automatically constructing new contextual information for sensing and control points based on the sensor point names and the raw time series values. The objective is to enable vendors to apply building analytics to 90% of buildings with no manual mapping, and to 10% of buildings with a 90% reduction in manual mapping. These targets are set for all types of metadata required by current analytics engines, including type, location, equipment type, and other relationships. The outcome of this project is a suite of solutions to the manual mapping problem collectively called the Building Adapter that allows vendors to apply analytics engines to new buildings at a significantly reduced cost.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2022

Oak Ridge National Laboratory Pilot Demonstration of an Attestation and Anomaly Detection Framework using Distributed Ledger Technology for Power Grid Infrastructure

This report summarizes the design and pilot demonstration of a framework called Grid Guard that was created to provide increased data and device trustworthiness to electric grid devices by leveraging distributed ledger technology (DLT), specifically blockchain. Grid Guard contains a combination of core cryptographic methods such as the secure hash algorithm (SHA), and asymmetric cryptography, private permissioned blockchain, baselining configuration data, consensus algorithm (Raft) and the Hyperledger Fabric (HLF) framework. The system implements a low energy, fast, and robust enhancement to system trustworthiness within and across electric grid systems such as substations, control centers and metering infrastructures. Blockchain is a distributed database structured that provides a practically unalterable (immutable) timeline of stored transactions. By relying on hashing and the Raft consensus algorithm, if an entity tries to illegitimately alter a record at one instance of the database the other ledger nodes are not altered. They work to cross-reference each other and easily locate any incorrectly added data and remove it. The bulk raw data is stored in an off-chain storage (outside of the blockchain ledger) and a hash of this baseline data is stored in the Blockchain ledger via hashing windows of time-series and configuration data, after aggregation and filtering. The bulk off-chain data repository is then considered to be trust-anchored using the hashes stored in the blockchain. To secure the electric grid testbed devices and data, device configuration baselines were compared to those baselines that had been previously stored in the ledger. Statistical baselines for device configurations, network communication patterns, and high-speed sensor data are calculated and then stored off-chain and hashes stored in the ledger. Measurements such as three-phase voltage and current, frequency, breaker status, protection scheme settings, network configuration settings (and other device configuration artifacts) and network traffic features (packet interarrival times) are compared every minute or other selected time windows. During phase 1 of the Grid Guard DLT project different DLT technologies were studies, and an assessment was performed on DLT technology vulnerabilities, uses, and key characteristics. DLT consensus protocols were studies (e.g., RAFT, named after Reliable, Replicated, Redundant, And Fault-Tolerant). Also, cryptography, public, private and permissioned or permissionless systems were assessed. Grid Guard implements a permissioned private DLT. Consensus algorithm selection and choice of DLT implementation depended heavily on the use-case. For this use-case, parameters were selected to measure performance and existing tools for assessment. Benchmarking was performed theoretically and practically. During phase 2 hashed transactions/blocks were inserted into the ledger every second. During phase 2 of the Grid Guard DLT project, a prototype framework was developed and demonstrated for attestation of critical substation devices and data using precision timing systems that use PTP and IRIG-B protocols) on a testbed of operational devices that emulated a distribution substation, control center, and power metering infrastructure using real Operational Technology (OT). The testbed includes OT devices such as protective relays, human machine interfaces (HMI), and power meters. To determine when to collect and compare system and network baselines, an initial examination of an anomaly detection capability to identify malicious manipulation of data streams was conducted. The resulting anomaly detection was demonstrated in a set of experiments and leveraged to trigger device artifact attestation checks. Attestation checks occur against device configuration baselines when compared with the immutable blockchain-stored baselines, which provided a cryptographically supported means by which to store baselines. The electrical substation-grid testbed was created to test the Grid Guard framework. The testbed emulates the operations of a portion of a power grid and SCADA systems as closely as possible. The testbed integrates real protocols, mainly IEC 61850 standard protocols, such as the Sampled Value (SV) and the GOOSE protocols. The testbed also supports DNP3 and other layer 2 and layer 3 protocols such as Telnet, SSH, SFTP/FTP and other proprietary protocols needed to connect to industrial control system equipment. The testbed emulates real power conditions using the OpalRT hardware-in-the-loop (HIL) device which can create fault situations that cannot be easily tested on real systems. The electrical substation-grid testbed was created using real measurement, communication, and protection devices that electrical utilities commonly use.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Tools for Assessing Performance: FY23-Q1 Report [Slides]

LANL was tasked with working with NREL to implement QUIC within the TAP API by the end of Q1, FY2023. If QUICURB cannot be implemented in a way that it can be run outside of the QUIC platform, there will be no further R&D funding for QUIC development. QUIC includes a graphical user interface that imports and preprocesses the various input data streams (3D building databases, ambient wind, vegetation, etc.) and writes them in a format that the QUICURB Fortran executable requires. In order to interface with the TAP API, a Python script was developed that would replace the functionality previously only available within QUICGUI. LANL worked with NREL to test the Python script to ensure that it was operation and fulfilled the requirements.

58 GEOSCIENCES↗
DOE OSTI2022

Highly Segmented Silicon Strip Detectors for Radiation Detection (LDRD Final Report)

Large area highly segmented silicon strip detectors were investigated for use in applied radiation detection applications. Two ASIC-based readout electronics solutions were tested for compatibility. During the analysis, it was determined that significant computational resources and highly specialized firmware are required to reconstruct the data into a meaningful data stream for generalized spectroscopic performance. Future investment in the technology and careful choice of readout ASIC is expected to demonstrate relevance in measuring high activity samples.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

Sentinel Devices LLC (CRADA Final Report)

Industrial equipment is a critical component of virtually all at-scale manufacturing and infrastructure. Broadly speaking, modern equipment is predominantly controlled using hardened digital controllers – computers designed to be able to operate continuously for sometimes extremely long periods of time, with little to no maintenance. Due to the nature of how these digital controllers have evolved, they are solely optimized to execute a single task, and do not have the capabilities or resources to monitor or analyze their internal state beyond simple execution of their program. As a result, for many “common sense” situations where individual data points can be easily determined to be out-of-normal, the controllers are unable to identify these incorrect operational modes unless a human has explicitly programmed in detection of this degradation. This project seeks to develop an AI/ML system which can identify incorrect or anomalous trends in industrial data streams, of exactly the kind that would be produced and seen by these digital controllers, with a minimal amount of computing resources. The benefits produced by developing this system would ultimately be self-monitoring and self-reporting infrastructure, capable of identifying and alerting humans to issues as soon as they happen, potentially long before they have the chance to impact the industrial process.

42 ENGINEERING↗
DOE OSTI2023

How Dynamic Time Warping Can Assist Conventional Cross-correlation

Waveform cross-correlation is a sensitive phase-matched filtering technique that can detect seismic events for nuclear explosion monitoring. However, there are outstanding challenges with correlation detectors, most notably a direct dependence on the completeness of the waveform template library. To ameliorate these challenges, we investigate how dynamic time warping (DTW) may make waveform correlation more robust. DTW analyzes the differences between two time series and attempts to “warp” one time series relative to another in a recursive manner. We apply DTW to synthetic earthquake and recorded explosion templates to expand the capability of correlation detectors. We explore what conditions (e.g., source, station distance, frequency bands) and/or DTW algorithms generate stronger correlation scores. We show that DTW performs well on noisy signals and can dramatically improve the cross-correlation coefficient between a template and data-stream waveform. We conclude with recommendations on how to utilize DTW in nuclear monitoring detection.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2024

Diaspora: Resilience-enabling services for science from HPC to edge

Scientific applications of interest to DOE must increasingly engage distributed resources (e.g., instruments, remote computers, data stores, edge devices) and deliver more stringent levels of service (e.g., uninterrupted processing of experiment data streams). In such systems, state is distributed and components can fail in many ways, often silently, making application resilience a major concern. Addressing the resilience needs of such applications requires methods for gaining knowledge of resources and applications and for translating that knowledge into action. We are working on addressing these needs in the context of multi-messenger astronomy, where detecting and responding to unusual transient events in multiple cosmic messengers (gravitational wave, electromagnetic, high- energy particles) from different instruments leads to a federated learning problem.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Graph-Learning-Assisted State and Event Tracking for Solar-Penetrated Power Grids with Heterogeneous Data Sources

Unlike transmission systems, distribution systems do not typically contain sufficient metering to enable real-time state estimation. The lack of sufficient real-time measurements prohibits accurate and timely monitoring of the state of distribution systems. As a result, control and optimal operation of distribution systems, especially those containing large numbers of renewable generation units are not possible without proper data and information about the current state of the system. The main motivation of this project is to address this shortcoming by developing an approach which provides “predicted” real-time measurements so that they can be used to execute a distribution system state estimator. Thus, the objective of the project is to make the distribution systems fully observable, such that the hosting capacity for solar generation can be accurately estimated, and unnecessary solar curtailments can be avoided. In order to accomplish this goal, the project investigated the use of a grid-model-informed machine learning (ML) tool which integrates heterogeneous data streams obtained from AMI meters, SCADA as well as PMU measurements and created synchronous measurement snapshots for the state estimator (SE); and developed a hybrid robust SE which provides not only accurate state estimates but also real-time feedback for the ML model refinement.

14 SOLAR ENERGY↗
DOE OSTI2025

Developing and Managing Data Acquisition Software Using Spack

The Data Acquisition systems of particle physics experiments regularly push the boundaries of high-throughput computing, demanding low-latency collection of data from thousands of devices, collating data into time-sliced events, processing these events and making trigger decisions, and writing the selected data streams to disk. To accomplish these tasks, the DAQ Engineering and Operations department at Fermilab leverages multiple software libraries and builds reusable DAQ frameworks on top. These libraries must be delivered in well-defined bundles and are thoroughly tested for compatibility and functionality before being deployed to live detectors. We have several techniques used to ensure that a consistent set of dependencies can be delivered and re-created at need. We must also support active development of DAQ software components, ideally in an environment as close as possible to that of the detectors. This development often occurs across multiple packages which have to be built in concert and features tested in a consistent and reproducible manner. I will present our scheme for accomplishing these goals using Spack environments, bundle packages, and Github Actions-based CI.

Flumerfelt, Eric [Fermilab]↗