Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 451 records · Page 25

Classifying thermodynamic cloud phase using machine learning models

Vertically resolved thermodynamic cloud-phase classifications are essential for studies of atmospheric cloud and precipitation processes. The Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Thermodynamic Cloud Phase (THERMOCLDPHASE) value-added product (VAP) uses a multi-sensor approach to classify the thermodynamic cloud phase by combining lidar backscatter and depolarization, radar reflectivity, Doppler velocity, spectral width, microwave-radiometer-derived liquid water path, and radiosonde temperature measurements. The measured pixels are classified as ice, snow, mixed phase, liquid (cloud water), drizzle, rain, and liq_driz (liquid+drizzle). We use this product as the ground truth to train three machine learning (ML) models to predict the thermodynamic cloud phase from multi-sensor remote sensing measurements taken at the ARM North Slope of Alaska (NSA) observatory: a random forest (RF), a multi-layer perceptron (MLP), and a convolutional neural network (CNN) with a U-Net architecture. Evaluations against the outputs of the THERMOCLDPHASE VAP with 1 year of data show that the CNN outperforms the other two models, achieving the highest test accuracy, F1 score, and mean intersection over union (IOU). Analysis of ML confidence scores shows that ice, rain, and snow have higher confidence scores, followed by liquid, while mixed, drizzle, and liq_driz have lower scores. Feature importance analysis reveals that the mean Doppler velocity and vertically resolved temperature are the most influential data streams for ML thermodynamic cloud-phase predictions. Lidar measurements exhibit lower feature importance due to rapid signal attenuation caused by the frequent presence of persistent low-level clouds at the NSA site. The ML models' generalization capacity is further evaluated by applying them at another Arctic ARM site in Norway using data taken during the ARM Cold-Air Outbreaks in the Marine Boundary Layer Experiment (COMBLE) field campaign. The models demonstrated similar performance to that observed at the NSA site. Finally, we evaluate the ML models' response to simulated instrument outages and signal degradation and show that a CNN U-Net model trained with input channel dropouts performs better when input fields are missing.

ARM Aerial Facility↗

Events and Trends in Text Streams

"Text streams--collections of documents or messages that are generated and observed over time--are ubiquitous. Our research and development are targeted at developing algorithms to find and characterize changes in topic within text streams. To date, this research has demonstrated the ability to detect and describe 1) short duration, atypical events and 2) the emergence of longer-term shifts in topical content. This technology has been applied to predefined temporally ordered document collections but is also suitable for application to near-real-time textual data streams."

99 GENERAL AND MISCELLANEOUS↗

Data Access Performance Through Parallelization and Vectored Access: Some Results

High Energy Physics data processing and analysis applications typically deal with the problem of accessing and processing data at high speed. Recent studies, development and test work have shown that the latencies due to data access can often be hidden by parallelizing them with the data processing, thus giving the ability to have applications which process remote data with a high level of efficiency. Techniques and algorithms able to reach this result have been implemented in the client side of the Scalla/xrootd system, and in this contribution we describe the results of some tests done in order to compare their performance and characteristics. These techniques, if used together with multiple streams data access, can also be effective in allowing to efficiently and transparently deal with data repositories accessible via a Wide Area Network.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

100 Gb/s High Throughput Serial Protocol (HTSP) for data acquisition systems with interleaved streaming

Demands on Field-Programmable Gate Array (FPGA) data transport have been increasing over the years as frame sizes and refresh rates increase. As the bandwidths requirements increase the ability to implement data transport protocol layers using "soft" programmable logic becomes harder and start to require harden IP blocks implementation. To reduce the number of physical links and interconnects, it is common for data acquisition systems to require interleaving of streams on the same link (e.g. streaming data and streaming register access). Further, this paper presents a way to leverage existing FPGA harden IP blocks to achieve a robust, low latency 100 Gb/s point-to-point link with minimal programmable logic overhead geared towards the needs of data acquisition systems with interleaved streaming requirements.

47 OTHER INSTRUMENTATION↗

HopPyBar

HopPyBar is a python program to import, analyze, and export split-Hopkinson pressure bar (SHPB, also known as Kolsky bar) data. Traditional analysis offers a black box approach, where input data is converted to analyzed output by performing a series of calculations without user involvement. This program serves as a developmental platform to "white box" the data analysis process. Data streams can be captured (in-situ) to enable advanced or unconventional analyses, statistics, and comparisons. Additionally, the program is geared towards the standardized forms of input and output used at LANL to streamline analysis, but the open nature of the program makes additional input/output schemes straightforward to add. General workflow will import SHPB data in one of a number of formats, identify relevant portions of data signals, and convert to stress-strain-strain rate to show material behavior as a function of dynamic testing.

Morrow, Benjamin↗

A Co-Registered In-Situ and Ex-Situ Dataset of Electrical, Acoustic, and CT Characteristics from Wire Arc Additive Manufacturing Process

Recent progress in sensing techniques and data analytics tools have significantly accelerated the development of Wire Arc Additive Manufacturing (WAAM) systems. This data centric approach emphasizes leveraging available data throughout the production process to optimize performance. Integration of extensive data analysis provides the opportunity to improve precision, reduce waste, and enhance the quality of produced parts. This method relies on AI/ML models and optimization techniques, which are developed using the data collected from various sources, including in-situ sensors, ex-situ imaging, and manufacturing process parameters. The quality and diversity of this data, along with the alignment between different data streams (achieved through spatiotemporal registration) are critical for the successful development of AI/ML and optimization models. In this work, we present a spatiotemporally registered dataset generated during the WAAM process of deposition of a rectangular block. The dataset includes the comprehensive description of deposition process, process parameters, in-situ collected welding characteristics, acoustic data, and X-Ray Computed Tomography analysis data for the build. Dataset A Co-Registered In-Situ and Ex-Situ Dataset of Electrical, Acoustic, and CT Characteristics from Wire Arc Additive Manufacturing Process has arisen under UT-Battelle, LLC’s Prime Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy (DOE) to manage and operate the Oak Ridge National Laboratory. UT-Battelle, LLC will not assert any rights under United States law or under the Prime Contract it has in the dataset against any user of the dataset, including any copyrights or patent rights. UT-Battelle, LLC requests that attribution to the dataset is provided as academically appropriate.

42 ENGINEERING↗

Effectiveness of Privacy Techniques in Smart Metering Systems

Smart grid technologies enable timely energy billing for residential homes. The ability to react to energy demands during peak hours allows energy providers to conserve power and operate efficiently. However, these data streams are also susceptible to privacy attacks within the energy company and from outside hackers. We implemented four different privacy models: k-anonymous, l-diversity, t-closeness, and ε-differential privacy. We demonstrate the models’ effectiveness using a real-world dataset composed of 15 different residential households with energy consumption data spanning over a year.

Peralta-Peterson, Martin↗

Data Always Getting Bigger—A Scalable DOI Architecture for Big and Expanding Scientific Data

The Atmospheric Radiation Measurement (ARM) Data Archive established a data citation strategy based on Digital Object Identifiers (DOIs) for the ARM datasets in order to facilitate citing continuous and diverse ARM datasets in articles and other papers. This strategy eases the tracking of data provided as supplements to articles and papers. Additionally, it allows future data users and the ARM Climate Research Facility to easily locate the exact data used in various articles. Traditionally, DOIs are assigned to individual digital objects (a report or a data table), but for ARM datasets, these DOIs are assigned to an ARM data product. This eliminates the need for creating DOIs for numerous components of the ARM data product, in turn making it easier for users to manage and cite the ARM data with fewer DOIs. In addition, the ARM data infrastructure team, with input from scientific users, developed a citation format and an online data citation generation tool for continuous data streams. This citation format includes DOIs along with additional details such as spatial and temporal information.

97 MATHEMATICS AND COMPUTING↗

Online Distribution System State Estimation via Stochastic Gradient Algorithm

Distribution network operation is becoming more challenging because of the growing integration of intermittent and volatile distributed energy resources (DERs). This motivates the development of new distribution system state estimation (DSSE) paradigms that can operate at fast timescale based on real-time data stream of asynchronous measurements enabled by modern information and communications technology. To solve the real-time DSSE with asynchronous measurements effectively and accurately, this paper formulates a weighted least squares DSSE problem and proposes an online stochastic gradient algorithm to solve it. The performance of the proposed scheme is analytically guaranteed and is numerically corroborated with realistic data on IEEE 123-bus feeder.

distribution system state estimation↗

Text Stream Trend Analysis using Multiscale Visual Analytics with Applications to Social Media Systems

In this paper, we introduce a new visual analytics system, called Matisse, that allows exploration of global trends in textual information streams with specific application to social media platforms. Despite the potential for real-time situational awareness using these services, interactive analysis of such semi-structured textual information is a challenge due to the high-throughput and high-velocity properties. Matisse addresses these challenges through the following contributions: (1) robust stream data management, (2) automated sen- timent/emotion analytics, (3) inferential temporal, geospatial, and term-frequency visualizations, and (4) a flexible drill-down interaction scheme that progresses from macroscale to microscale views. In addition to describing these contributions, our work-in-progress paper concludes with a practical case study focused on the analysis of Twitter 1% sample stream information captured during the week of the Boston Marathon bombings.

interaction↗

MSE based drilling optimization using neural network simulaton

The method disclosed receives a data stream from an MWD system and determines the response of a specific energy (SE) relationship and a rate of penetration (ROP) relationship respectively to variables controllable by the operator, in order to enable operation at a lowest SE, or a highest Rate-of-Penetration (ROP) to SE ratio. The method utilizes artificial neural networks trained by MWD data to deduce a depth-of-cut and torque based on relationships manifesting between the various data points collected, and an SE equation and a predicted ROP is evaluated over a series of probable operating points. The method continuously gathers and analyzes MWD data during the drilling operation and allows an operator to manage the controllable parameters such that operation at the lowest SE or highest ROP or ROP to SE ratio can be achieved during the drilling operation.

Zhang, Wu↗

CORE (Common Operating Response Environment) Software Technology Suite

Agencies that oversee complex, multi-stakeholder programs need efficient, secure ways to link people and knowledge within and across organizations. The Common Operating Response Environment (CORE), a software suite developed by PNNL researchers does just that. The CORE tool—which is customizable for a multitude of uses—facilitates situational awareness by integrating diverse data streams without the need to reformat them, summarizing that information, and providing users with the information they need to rapidly understand and appropriately respond to situations. It is mobile device-ready, has a straightforward interface for ease of use across organizations and skill sets, and is incredibly configurable to the needs of each specific user, whether they require data summaries for high-level decision makers or tactical maps, operational data, or weather information for responders in the field. Information can be input into CORE and queried in a variety of ways—using customized forms, reports, visuals, or other organizational templates—according to the needs of each user’s organization, teams, and business processes. CORE data forms, for instance, could be accessed and used in real-time to capture information about vessels being inspected for nuclear material.

42 ENGINEERING↗

Situ

Rapidly discovering novel and sophisticated cyber attacks and providing situation aware-ness to analysts are unsolved problems in cyber security. We have developed a platform that scores events in real-time based on probabilistic models to define how typical an event is. This anomaly detection approach is based on unsupervised, probabilistic modeling of data at multiple scales. Scoring events based on multiple scales allows the system to 1) score anomalousness at different levels to detect important events that would otherwise be hidden, and 2) explain to users why an event is anomalous, not just that it is. The system was designed to address several challenges: 1) scaling to very high volume, heterogeneous, streaming data, and 2) minimizing the time from observation to discovery to understanding. The prototype has the real-time framework for pushing scored events to a web-based visualization.

Source record↗

Combined impact of lead, cadmium, polychlorinated biphenyls and non-chemical risk factors on blood pressure in NHANES

High blood pressure is associated with exposure to multiple chemical and non-chemical risk factors, but epidemiological analyses to date have not assessed the combined effects of both chemical and non-chemical stressors on human populations in the context of cumulative risk assessment. We developed a novel modeling approach to evaluate the combined impact of lead, cadmium, polychlorinated biphenyls (PCBs), and multiple non-chemical risk factors on four blood pressure measures using data for adults aged ≥20 years from the National Health and Nutrition Examination Survey (1999–2008). We developed predictive models for chemical and other stressors. Structural equation models were applied to account for complex associations among predictors of stressors as well as blood pressure. Models showed that blood lead, serum PCBs, and established non-chemical stressors were significantly associated with blood pressure. Lead was the chemical stressor most predictive of diastolic blood pressure and mean arterial pressure, while PCBs had a greater influence on systolic blood pressure and pulse pressure, and blood cadmium was not a significant predictor of blood pressure. The simultaneously fit exposure models explained 34%, 43% and 52% of the variance for lead, cadmium and PCBs, respectively. The structural equation models were developed using predictors available from public data streams (e.g., U.S. Census), which would allow the models to be applied to any U.S. population exposed to these multiple stressors in order to identify high risk subpopulations, direct intervention strategies, and inform public policy. - Highlights: • We evaluated joint impact of chemical and non-chemical stressors on blood pressure. • We built predictive models for lead, cadmium and polychlorinated biphenyls (PCBs). • Our approach allows joint evaluation of predictors from population-specific data. • Lead, PCBs and established non-chemical stressors were related to blood pressure. • Framework allows cumulative risk assessment in specific geographic settings.

54 ENVIRONMENTAL SCIENCES↗

Automatic Traffic Queue-End Identification using Location-Based Waze User Reports

Traffic queues, especially queues caused by non-recurrent events such as incidents, are unexpected to high-speed drivers approaching the end of queue (EOQ) and become safety concerns. Though the topic has been extensively studied, the identification of EOQ has been limited by the spatial-temporal resolution of traditional data sources. This study explores the potential of location-based crowdsourced data, specifically Waze user reports. It presents a dynamic clustering algorithm that can group the location-based reports in real time and identify the spatial-temporal extent of congestion as well as the EOQ. The algorithm is a spatial-temporal extension of the density-based spatial clustering of applications with noise (DBSCAN) algorithm for real-time streaming data with an adaptive threshold selection procedure. Here, the proposed method was tested with 34 traffic congestion cases in the Knoxville, Tennessee area of the United States. It is demonstrated that the algorithm can effectively detect spatial-temporal extent of congestion based on Waze report clusters and identify EOQ in real-time. The Waze report-based detection are compared to the detection based on roadside sensor data. The results are promising: The EOQ identification time of Waze is similar to the EOQ detection time of traffic sensor data, with only 1.1 min difference on average. In addition, Waze generates 1.9 EOQ detection points every mile, compared to 1.8 detection points generated by traffic sensor data, suggesting the two data sources are comparable in respect of reporting frequency. The results indicate that Waze is a valuable complementary source for EOQ detection where no traffic sensors are installed.

99 GENERAL AND MISCELLANEOUS↗

X-composer: enabling cross-environments in-situ workflows between HPC and cloud

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

Wang, Dali↗

SEVEN-YEAR WILKINSON MICROWAVE ANISOTROPY PROBE (WMAP ) OBSERVATIONS: GALACTIC FOREGROUND EMISSION

We present updated estimates of Galactic foreground emission using seven years of WMAP data. Using the power spectrum of differences between multi-frequency template-cleaned maps, we find no evidence for foreground contamination outside of the updated (KQ85y7) foreground mask. We place a 15 {mu}K upper bound on rms foreground contamination in the cleaned maps used for cosmological analysis. Further, the cleaning process requires only three power-law foregrounds outside of the mask. We find no evidence for polarized foregrounds beyond those from soft (steep-spectrum) synchrotron and thermal dust emission; in particular we find no indication in the polarization data of an extra 'haze' of hard synchrotron emission from energetic electrons near the Galactic center. We provide an updated map of the cosmic microwave background (CMB) using the internal linear combination method, updated foreground masks, and updates to point source catalogs using two different techniques. With additional years of data, we now detect 471 point sources using a five-band technique and 417 sources using a three-band CMB-free technique. In total there are 62 newly detected point sources, a 12% increase over the five-year release. Also new are tests of the Markov chain Monte Carlo foreground fitting procedure against systematics in the time-stream data, and tests against the observed beam asymmetry. Within a few degrees of the Galactic plane, the behavior in total intensity of low-frequency foregrounds is complicated and not completely understood. WMAP data show a rapidly steepening spectrum from 20 to 40 GHz, which may be due to emission from spinning dust grains, steepening synchrotron, or other effects. Comparisons are made to a 1 deg 408 MHz map (Haslam et al.) and the 11 deg ARCADE 2 data (Singal et al.). We find that spinning dust or steepening synchrotron models fit the combination of WMAP and 408 MHz data equally well. ARCADE data appear inconsistent with the steepening synchrotron model and consistent with the spinning dust model, though some discrepancies remain regarding the relative strength of spinning dust emission. More high-resolution data in the 10-40 GHz range would shed much light on these issues.

79 ASTRONOMY AND ASTROPHYSICS↗

Misuse Detection for a Generalized SFR Test Reactor

Sodium-cooled Fast Reactors (SFRs) present unique challenges for international safeguards. SFRs possess neutron physics characteristics that if configured appropriately could produce more fissile material than consumed. An adversary state may choose to build an SFR, justified by a lack of domestic natural uranium and limited access or interest to procuring uranium from international markets. Once constructed, the state may choose to misuse the SFR for the purpose of diverting fissile plutonium from declared operation. This work shows that a demonstration SFR does not need to be configured as a plutonium breeder to create one Significant Quantity (SQ) of plutonium in a short amount of time (e.g., one to few years). However, such an extreme case of misuse would change the core reactivity in such a way as to be easily indicated by deviations of control rod position compared to declared operation. In this work a contrived SFR demonstration reactor was modeled for the purpose of exploring proliferation scenarios and how such misuse could be detected using the SFR's Reactor Data Acquisition System (RDAS). Typically, the International Atomic Energy Agency (IAEA) does not have access to the control rod position, power, thermal, pressure sensing and indicating systems of nuclear power plants. However, this work shows that such data streams can be compared against a parallel detailed simulation model (a Digital Twin) to detect possible misuse.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗