Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,297 records · Page 72

Effect Of Transition Density On Performance Of A DTTL

One estimates minimum transition density needed to limit symbol-error rate. Theoretical calculations and accompanying computer simulations provide numerical basis for specification of minimum data-transition density needed for adequate performance of radio receiver processing noisy signal containing non-return-to-zero (NRZ) stream of data symbols (bits). By use of digital-data-transition-tracking loop (DTTL), receiver strives to synchronize operation with data symbols. Performance of receiver in terms of ability to keep symbol-error rate (SER) in output of receiver below specified level depends on symbol signal-to-noise ratio (SSNR) of incoming signal and on degree of synchronization. Ability of DTTL to maintain synchronization depends partly on SSNR and partly on data-transition density.

Nguyen, Tien M.↗

Pre-commercial Demonstration of Direct Non-iterative State Estimator for Operational Use of Synchrophasor Management Systems

The objective of this project was to demonstrate functionality and performance of a Direct Non-iterative State Estimator (DNSE) integrated with NYPA’s Energy Management System (EMS) and with enhanced Real Time Dynamics Monitoring System (RTDMS) synchrophasor platform from Electric Power Group (EPG). DNSE is designed to overcome a major obstacle to operational use of Synchro-Phasor Management Systems (SPMS) by providing to synchrophasor management systems (SPMS) applications a consistent and a complete synchrophasor data foundation in the same way that a traditional EMS State Estimator (SE) provides to EMS applications. Specifically, DNSE is designed to use synchrophasor measurements collected by a central PDC, Supervisory Control and Data Acquisition (SCADA) measurements, and Energy Management System (EMS) network model, to obtain the complete state of the utility’s operating model at rates that are close to the synchrophasor data rates. In this way, the system is comprehensive in that it does not only cover the part of the network that is “visible” via synchrophasors, but also the part that is only “visible” through the SCADA measurements. Visualization needs associated with the use of DNSE results are fulfilled through suitably enhanced Real Time Dynamics Monitoring System (RTDMS), with the enhancements implemented by EPG. This project had the following goals in mind: To advance the deployment of commercial grade DNSE software application that relies on synchrophasor and SCADA data ; Apply DNSE at other utilities, to address a generic and fundamental need for “clean” operational data for synchrophasor applications; Provide means for “live” estimated data access by control system operators; Enhance potential for situational awareness through full system operational model coverage; Sub-second execution rate of the Direct Non-iterative State Estimator, eventually at a near-phasor data rate execution speed, i.e. < 0.1 sec. Anticipated benefits from this projects are: Enhanced reliability and improvements in the economic efficiency of bulk power system planning and operations; Providing “clean” data to other synchrophasor applications; Enhancement of situational awareness by providing the full operational model updated at near synchrophasor rate; A production-grade software tool that incorporate synchrophasor and SCADA data; Provides a basis for development of next generation monitoring and control applications, based on both SCADA and PMU data. Quanta Technology (QT) team worked in collaboration with Electric Power Group (EPG) whose team has enhanced its commercial Real Time Dynamics Monitoring System (RTDMS) to accommodate the requirements posed by DNSE application. EPG also provided its ePDC and Model-less Data Conditioning (PDVC) software for integration with DNSE+. QT developed the system requirements for DNSE; developed system architecture and defined interfaces between internal DNSE components. The core DNSE algorithm with all surrounding interfaces was named DNSE+. Since the DNSE development was done in a simulated system environment, QT used its PMU simulator that was enhanced during this project for development and factory acceptance testing (FAT). SCADA data in this stage was simulated by commercial PSS/e software. The output of DNSE are estimates of System states in C37.118-2 format, sent to RTDMS for further processing and display. As the number of these states is large, it was necessary to expand the C37.111-2 standard to accommodate large data sets. This enhancement was implemented in RTDMS. The demonstration of pre-production DNSE technology was done at NYPA using streaming field data from NYPA PMUs and from its RTUs through their SCADA system. NYPA provided ICCP interface as well as Common Information Model (CIM). The relevance of the DNSE+ application is that it provides state estimation of the power system based on hybrid set of data, consisting of both available PMU data and SCADA measurements. As this is a direct, non-iterative method of calculation of the system states, if does not suffer from convergence issues which is potential problem for conventional state estimators. Also, it can take any available PMU measurements, so it does not need to have a high percentage of PMU coverage needed in the case of Linear State Estimator. As the DNSE calculates synchrophasors of the system states (both phase and absolute value) as sub-second rate, this application can provide a basis for development of next generation of applications based both on SCADA and PMU data.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Hardware Architectures for Data-Intensive Computing Problems: A Case Study for String Matching

DNA analysis is an emerging application of high performance bioinformatic. Modern sequencing machinery are able to provide, in few hours, large input streams of data, which needs to be matched against exponentially growing databases of known fragments. The ability to recognize these patterns effectively and fastly may allow extending the scale and the reach of the investigations performed by biology scientists. Aho-Corasick is an exact, multiple pattern matching algorithm often at the base of this application. High performance systems are a promising platform to accelerate this algorithm, which is computationally intensive but also inherently parallel. Nowadays, high performance systems also include heterogeneous processing elements, such as Graphic Processing Units (GPUs), to further accelerate parallel algorithms. Unfortunately, the Aho-Corasick algorithm exhibits large performance variability, depending on the size of the input streams, on the number of patterns to search and on the number of matches, and poses significant challenges on current high performance software and hardware implementations. An adequate mapping of the algorithm on the target architecture, coping with the limit of the underlining hardware, is required to reach the desired high throughputs. In this paper, we discuss the implementation of the Aho-Corasick algorithm for GPU-accelerated high performance systems. We present an optimized implementation of Aho-Corasick for GPUs and discuss its tradeoffs on the Tesla T10 and he new Tesla T20 (codename Fermi) GPUs. We then integrate the optimized GPU code, respectively, in a MPI-based and in a pthreads-based load balancer to enable execution of the algorithm on clusters and large sharedmemory multiprocessors (SMPs) accelerated with multiple GPUs.

Tumeo, Antonino↗

System for detecting objects in streaming 3D images formed from data acquired with a medium penetrating sensor

A detection system that detects subsurface objects within a medium and estimates various features of the objects is provided. The detection system receives a streaming sequence of image frames of the medium at various along-medium locations. An image frame contains voxel values (intensities) representing characteristics of the medium across the medium and in the depth (range) direction. The detection system depth-compensates the intensities for determining which voxels are part of an object using an unsupervised binary classifier. The detection system then connects object voxels into distinct objects and recursively estimates the features of those objects as the image frames stream based on the locations and intensities of the object voxels.

Paglieroni, David W.↗

VLBI Data Interchange Format (VDIF)

One important outcome of the 7th International e-VLBI Workshop in Shanghai in June 2008 was the creation of a task force to study and recommend a universal VLBI data format that is suitable for both on-the-wire e-VLBI data transfer, as well as direct disk storage. This task force, called the VLBI Data Interchange Format (VDIF) Task Force, is the first part of a two-part effort, the second of which will address standardization of e-VLBI data-transmission-protocols. The formation of the VDIF Task Force was prompted particularly by increased e-VLBI activity and the difficulties encountered when data arrive at a correlator in different formats from various instruments in various parts of the world. The task force created a streaming packetized data format that may be used for real-time and non-realtime e-VLBI, as well as direct disk storage. The data may contain multiple channels of time-sampled data with an arbitrary number of channels, arbitrary #bits/sample up to 32, and real or complex data; data rates in excess of 100 Gbps are supported. Each data packet is completely self-identifying via a short header, and data may be decoded without reference to any external information. The VDIF task force has completed its work, and the VDIF standard was ratified at the 2009 e-VLBI workshop in Madrid.

Whitney, Alan↗

Data Federation Challenges in Remote Near-Real-Time Fusion Experiment Data Processing

Fusion energy experiments and simulations provide critical information needed to plan future fusion reactors. As next-generation devices like ITER move toward long-pulse experiments, analyses, including AI and ML, should be performed in a wide range of time and computing constraints, from near-real-time constraints, between-shot analysis, and to campaign-wide long-term analysis. However, the data volume, velocity, and variety make it extremely challenging for analyses using only local computational resources. Researchers need the ability to compose and execute workflows spanning edge resources to large-scale high-performance computing facilities.We present Delta, a system to address data analysis challenges, including AI/ML, in fusion science, by leveraging the ADIOS I/O library and middleware, to support executing science workflows over the wide area network for near-real-time streaming. We discuss the data federation challenges in performing remote workflows, focusing on on-going research work in (1) managing, reducing, and streaming data to minimize I/O and data movement overheads, (2) decompressing and reorganizing data for analysis, and (3) executing workflows for automated data analysis. We introduce examples for deep-learning based data analysis for the fusion domain and demonstrate how we use Delta to construct end-to-end workflows for a fusion device in Korea, connecting a remote DOE facility in the USA. The capability demonstrated by this project is the basis for improving the state of the art for near-real-time data federation amongst remote facilities.

Choi, Jong Youl↗

Evaluating LSM-Based Water Budgets Over a West African Basin Assisted with a River Routing Scheme

Within the framework of the African Monsoon Multidisciplinary Analysis (AMMA) Land Surface Model Intercomparison Project phase 2 (ALMIP-2), this study evaluates the water balance simulated by the Interactions between Soil, Biosphere, and Atmosphere (ISBA) over the upper Oum River basin, in Benin, using a mesoscale river routing scheme (RRS). The RRS is based on the nonlinear Muskingum Cunge method coupled with two linear reservoirs that simulate the time delay of both surface runoff and base flow that are produced by land surface models. On the basis of the evidence of a deep water-table recharge in that region,a reservoir representing the deep-water infiltration (DWI) is introduced. The hydrological processes of the basin are simulated for the 2005-08 AMMA field campaign period during which rainfall and stream flow data were intensively collected over the study area. Optimal RRS parameter sets were determined for three optimization experiments that were performed using daily stream flow at five gauges within the basin. Results demonstrate that the RRS simulates stream flow at all gauges with relative errors varying from -22% to 3% and Nash-Sutcliffe coefficients varying from 0.62 to 0.90. DWI varies from 24% to 67% of the base flow as a function of the sub-basin. The relatively simple reservoir DWI approach is quite robust, and further improvements would likely necessitate more complex solutions (e.g., considering seasonality and soil type in ISBA); thus, such modifications are recommended for future studies. Although the evaluation shows that the simulated stream flows are generally satisfactory, further field investigations are necessary to confirm some of the model assumptions.

Multidisciplinary↗

An Incremental Tensor Train Decomposition Algorithm

We present a new algorithm for incrementally updating the tensor train decomposition of a stream of tensor data. This new algorithm, called the tensor train incremental core expansion (TT-ICE) improves upon the current state-of-the-art algorithms for compressing in tensor train format by developing a new adaptive approach that incurs significantly slower rank growth and guarantees compression accuracy. This capability is achieved by limiting the number of new vectors appended to the TT-cores of an existing accumulation tensor after each data increment. These vectors represent directions orthogonal to the span of existing cores and are limited to those needed to represent a newly arrived tensor to a target accuracy. We provide two versions of the algorithm: TT-ICE and TT-ICE accelerated with heuristics (TT-ICE*). Here, we provide a proof of correctness for TT-ICE and empirically demonstrate the performance of the algorithms in compressing large-scale video and scientific simulation datasets. Compared to existing approaches that also use rank adaptation, TT-ICE* achieves 57× higher compression and up to 95% reduction in computational time.

97 MATHEMATICS AND COMPUTING↗

Synchronization tracking in pulse position modulation receiver

A clock pulse generator for decoding pulse position modulation in an optical communication receiver is synchronized by a delay tracking loop which multiplies impulses of a data pulse by the square wave clock pulses from the generator to produce positive impulses when the clock pulse is of one level, and negative impulses when the clock pulse is of another level. A delay tracking loop integrates the impulses and produces an error signal that adjusts the delay so the clock pulses will be synchronized with data pulses. A dead-time tau sub d is provided between data pulses of an interval tau sub p in the data pulse period tau. When synchronized, the average number of positive impulses integrated will equal the average number of negative impulses over the continuous stream of data pulses.

Vilnrotter, Victor A.↗

Remote sensing images, DEM, and point clouds associated with “Accuracy evaluation of cost-effective 3D reconstruction approaches for hydrobiogeochemical processes in non-perennial stream riverbeds”

This data package is associated with the publication “Accuracy evaluation of cost-effective 3D reconstruction approaches for hydrobiogeochemical processes in non-perennial stream riverbeds” published in Frontiers in Environmental Science, Environmental Informatics and Remote Sensing (Bao et al., 2026; doi: 10.3389/fenvs.2026.1725258). This data package includes the drone photos for a section of Umtanum Creek in Washington, Unted States. The photos were used to reconstruct the 3-dimensional (3D) digital elevation model (DEM) of the riverbed for the investigated stream section. The reconstruction results from four approaches are provided: (1) unoccupied aerial vehicle (UAV, colloquially known as drone) imagery-based Structure-from-Motion (SfM), (2) a machine learning-based 3D reconstruction model, Visual Geometry Grounded Deep Structure from Motion (VGGSfM), (3) Visual Geometry Grounded Transformer for long sequence of images (VGGT-Long), and (4) handheld smartphone LiDAR scanning. The ground truth measurements by tripod-mounted optical level kit and ground control points GPS locations for evaluating the accuracy of the four reconstruction approaches are also provided in this data package. A preliminary version of this data package was published in October 2025 at the time of manuscript submission. It was updated in March 2026, at the time of manuscript acceptance, to include additional metadata (this readme, data dictionary, and file level metadata). The data did not change. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. This dataset is comprised of (1) 8 folders; (2) the detailed flight configuration html files; (3) field metadata; (4) a readme; (5) a data dictionary; and (6) file-level metadata. The folders “2024_10_18_d01” and “2024_10_18_d02” contain the original drone photos for the two drone flights (d01 and d02) on October 18, 2024. The reconstruction results from each of the approaches are in the folders called “ODM_SfM”, “VGGSfM”, “VGGTLong”, and “LiDAR”. The ground truth measurements are in the folder called “optical_level_kit”. Lastly, results comparing the different approaches are in the folder called “comparisons”. All files are .csv, .html, .jpg, .obj, .txt, and .npy. For information on using the .obj and .npy files, see the readme files within the same folder as the files.

54 ENVIRONMENTAL SCIENCES↗

Data Architecture for Security Monitoring (Project Summary)

The research has 2 thrusts: 1. new data architectures and tech-transfer-ready prototype tools that use write optimized data structures (WODS) to track real-world events, provide value to analysts, and support our cyber missions; and 2. algorithm research to address infinite streams of data, including expiration and sustainability.

97 MATHEMATICS AND COMPUTING↗

Gila Water Resources II: Using Earth Observations to Identify Wildfire Impacts on Hydrologic Functions and Recovery in the Gila National Forest

Wildfires have the potential to cause devastating and long-lasting impacts on ecological systems. In the Gila NF, wildfire events have occurred with increasing frequency and severity over recent years. These disturbances such as the historic Whitewater-Baldy Complex Fire (2012) and Silver Fire (2013) have raised concerns over post-fire soil erosion, flooding, debris flows, and vegetation recovery. Understanding the connection between burn events and ecological functions is crucial for developing effective land management practices within the Gila NF that ensure the conservation of the watershed. Our Gila Water Resources II team worked in partnership with the US Department of Agriculture (USDA) US Forest Service’s (USFS) Gila National Forest and Region 3. This project was designed to provide insight into the influence of wildfires on increased flooding events and to determine if restoration efforts in the Gila NF are having a beneficial and noticeable impact on recovery. The goals of this project included generating data-supported end products to inform land management decisions, including the prioritization of specific regions and time scales for post-fire restoration efforts. To understand recovery trends and hydrologic impact in the Gila NF between 2000 and 2020, this project used NASA EO and ancillary data, including, but not limited to, Landsat 5 Thematic Mapper (TM), Landsat 7 Enhanced Thematic Mapper Plus (ETM+), Landsat 8 Operational Land Imager (OLI), USGS stream gauge data, and data provided by USDA USFS’s Gila National Forest and Region 3.

Water Resources↗

SPLENDAQ: A Detector-Agnostic Data Acquisition System for Small-Scale Physics Experiments

Many scientific applications from rare-event searches to condensed matter system characterization to high-rate nuclear experiments require time-domain triggering on a raw stream of data, where the triggering is generally threshold-based or randomly acquired. When carrying out detector R &D, there is a need for a general data acquisition (DAQ) system to quickly and efficiently process such data. In the SPLENDOR collaboration, we are developing the Python-based SPLENDAQ package for this exact purpose—it offers two main features for offline analysis of continuous data: a threshold triggering algorithm based on the time-domain optimal filter formalism and an algorithm for randomly choosing nonoverlapping segments for noise measurements. Further, combined with the commercially available Moku platform, developed by Liquid Instruments, we have a full pipeline of event building off raw data with minimal setup. Here, we review the underlying principles of this detector-agnostic DAQ package and give concrete examples of its utility in various applications.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Telemetry Boards Interpret Rocket, Airplane Engine Data

For all the data gathered by the space shuttle while in orbit, NASA engineers are just as concerned about the information it generates on the ground. From the moment the shuttle s wheels touch the runway to the break of its electrical umbilical cord at 0.4 seconds before its next launch, sensors feed streams of data about the status of the vehicle and its various systems to Kennedy Space Center s shuttle crews. Even while the shuttle orbiter is refitted in Kennedy s orbiter processing facility, engineers constantly monitor everything from power levels to the testing of the mechanical arm in the orbiter s payload bay. On the launch pad and up until liftoff, the Launch Control Center, attached to the large Vehicle Assembly Building, screens all of the shuttle s vital data. (Once the shuttle clears its launch tower, this responsibility shifts to Mission Control at Johnson Space Center, with Kennedy in a backup role.) Ground systems for satellite launches also generate significant amounts of data. At Cape Canaveral Air Force Station, across the Banana River from Kennedy s location on Merritt Island, Florida, NASA rockets carrying precious satellite payloads into space flood the Launch Vehicle Data Center with sensor information on temperature, speed, trajectory, and vibration. The remote measurement and transmission of systems data called telemetry is essential to ensuring the safe and successful launch of the Agency s space missions. When a launch is unsuccessful, as it was for this year s Orbiting Carbon Observatory satellite, telemetry data also provides valuable clues as to what went wrong and how to remedy any problems for future attempts. All of this information is streamed from sensors in the form of binary code: strings of ones and zeros. One small company has partnered with NASA to provide technology that renders raw telemetry data intelligible not only for Agency engineers, but also for those in the private sector.

Source record↗

A Pan-STARRS1 VIEW OF THE BIFURCATED SAGITTARIUS STREAM

We use data from the Pan-STARRS1 survey to present a panoramic view of the Sagittarius tidal stream in the southern Galactic hemisphere. As a result of the extensive sky coverage of Pan-STARRS1, the southern stream is visible along more than 60 Degree-Sign of its orbit, nearly double the length seen by the Sloan Digital Sky Survey. The recently discovered southern bifurcation of the stream is also apparent, with the fainter branch of the stream visible over at least 30 Degree-Sign . Using a combination of fitting both the main-sequence turnoff and the red clump, we measure the distance to both arms of the stream in the south. We find that the distances to the bright arm of the stream agree very well with the N-body models of Law and Majewski. We also find that the faint arm lies {approx}5 kpc closer to the Sun than the bright arm, similar to the behavior seen in the northern hemisphere.

79 ASTRONOMY AND ASTROPHYSICS↗

Repurposing of the Run 2 CMS High Level Trigger Infrastructure as a Cloud Resource for Offline Computing

The former CMS Run 2 High Level Trigger (HLT) farm is one of the largest contributors to CMS compute resources, providing about 25k job slots for offline computing. This CPU farm was initially employed as an opportunistic resource, exploited during inter-fill periods, in the LHC Run 2. Since then, it has become a nearly transparent extension of the CMS capacity at CERN, being located on-site at the LHC interaction point 5 (P5), where the CMS detector is installed. This resource has been configured to support the execution of critical CMS tasks, such as prompt detector data reconstruction. It can therefore be used in combination with the dedicated Tier 0 capacity at CERN, in order to process and absorb peaks in the stream of data coming from the CMS detector. The initial configuration for this resource, based on statically configured VMs, provided the required level of functionality. However, regular operations of this cluster revealed certain limitations compared to the resource provisioning and use model employed in the case of WLCG sites. A new configuration, based on a vacuum-like model, has been implemented for this resource in order to solve the detected shortcomings. This paper reports about this redeployment work on the permanent cloud for an enhanced support to CMS offline computing, comparing the former and new models’ respective functionalities, along with the commissioning effort for the new setup.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Online Event Detection in Synchrophasor Data with Graph Signal Processing

Online detection of anomalies is crucial to enhancing the reliability and resiliency of power systems. We propose a novel data-driven online event detection algorithm with synchrophasor data using graph signal processing. In addition to being extremely scalable, our proposed algorithm can accurately capture and leverage the spatio-temporal correlations of the streaming PMU data. This paper also develops a general technique to decouple spatial and temporal correlations in multiple time series. Finally, we develop a unique framework to construct a weighted adjacency matrix and graph Laplacian for product graph. Case studies with real-world, large-scale synchrophasor data demonstrate the scalability and accuracy of our proposed event detection algorithm. Compared to the state-of-the-art benchmark, the proposed method not only achieves higher detection accuracy but also yields higher computational efficiency.

Event detection↗