Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “INGESTION”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 775 records · Page 43

pyXPCSviewer : an open-source interactive tool for X-ray photon correlation spectroscopy visualization and analysis

pyXPCSviewer , a Python-based graphical user interface that is deployed at beamline 8-ID-I of the Advanced Photon Source for interactive visualization of XPCS results, is introduced. pyXPCSviewer parses rich X-ray photon correlation spectroscopy (XPCS) results into independent PyQt widgets that are both interactive and easy to maintain. pyXPCSviewer is open-source and is open to customization by the XPCS community for ingestion of diversified data structures and inclusion of novel XPCS techniques, both of which are growing demands particularly with the dawn of near-diffraction-limited synchrotron sources and their dedicated XPCS beamlines.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Real-time Multi-granular Analytics Framework for HIT Systems

Streaming analytics is the process of ingesting and digesting live data from multiple data sources. In the healthcare domain, as the importance of extracting immediate insights while data are streaming into the system grows, the focus is shifting from batch processing to streaming analytics. With data increasing dramatically at high speeds, many informatics designs have been proposed to adapt healthcare domain into this new environment. In our previous work, we introduced a prototype of health informatics technology (HIT) framework that aims to address challenges in adopting state-of-the-art technologies to enable advanced healthcare analytic tasks in new streaming environments. We recently made major updates to the framework so that anomaly from multiple streaming data sources at different granularity levels can be detected in near real-time. In this paper, we detail the implementation and deployment of the framework in Kubernetes clusters and report its performances when tested on electronic health record (EHR) data of Veterans Affairs.

Park, Byung↗

Software Defined Cyberinfrastructure

Within and across thousands of science labs, researchers and students struggle to manage data produced in experiments, simulations, and analyses. Largely manual research data lifecycle management processes mean that much time is wasted, research results are often irreproducible, and data sharing and reuse remain rare. In response, we propose a new approach to data lifecycle management in which researchers are empowered to define the actions to be performed at individual storage systems when data are created or modified: actions such as analysis, transformation, copying, and publication. We term this approach software-defined cyberinfrastructure because users can implement powerful data management policies by deploying rules to local storage systems, much as software-defined networking allows users to configure networks by deploying rules to switches.We argue that this approach can enable a new class of responsive distributed storage infrastructure that will accelerate research innovation by allowing any researcher to associate data workflows with data sources, whether local or remote, for such purposes as data ingest, characterization, indexing, and sharing. We report on early experiments with this approach in the context of experimental science, in which a simple if-trigger-then-action (IFTA) notation is used to define rules.

Foster, Ian↗

Detecting outliers in streaming time series data from ARM distributed sensors

The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

Lu, Yuping↗

Descriptor: Infrastructure Perception and Control: Multi-Sensor Object Tracking Dataset (IPC-MSOT)

Traffic intersections are crucial and challenging nodes in transportation networks where multiple lanes of vehicles and pedestrians converge. Traffic accidents often occur at traffic intersections, including a large proportion of traffic fatalities and about one-half of all traffic injuries in the United States. Object detection data were collected in 2024 across three intersections in Colorado Springs, CO, USA, over the course of multiple days and various times to induce a heterogeneous mix of traffic conditions and behaviors. The purpose of the data collection exercises was to learn various attributes about infrastructure sensors and to build a repository of high-resolution, object-level data that can be used for research and development (e.g., to develop multisensor data fusion algorithms). The Infrastructure Perception and Control:Multi-Sensor Object tracking (IPC-MSOT) dataset was collected as part of the U.S. Department of Transportation's Strengthening Mobility and Revolutionizing Transportation (SMART) project, where the city of Colorado Springs, Colorado, and the National Renewable Energy Laboratory collaborated to collect object-level trajectory data from road users using multiple types of infrastructure sensors deployed at different intersections. This dataset allows for testing of late-stage sensor fusion algorithms and their ability to ingest multimodal sensor data, and it can be utilized by traffic engineers to design and evaluate trajectory-based signal control strategies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

An Alternative Timing and Synchronization Approach for Situational Awareness and Predictive Analytics

Accurate and synchronized timing information is required by power system operators for controlling the grid infrastructure (relays, Phasor Measurement Units (PMUs), etc.) and determining asset positions. Satellite-based global positioning system (GPS) is the primary source of timing information. However, GPS disruptions today (both intentional and unintentional) can significantly compromise the reliability and security of our electric grids. A robust alternate source for accurate timing is critical to serve both as a deterrent against malicious attacks and as a redundant system in enhancing the resilience against extreme events that could disrupt the GPS network. To achieve this, we rely on the highly accurate, terrestrial atomic clock-based network for alternative timing and synchronization. In this paper, we discuss an experimental setup for an alternative timing approach. The data obtained from this experimental setup is continuously monitored and analyzed using various time deviation metrics. We also use these metrics to compute deviations of our clock with respect to the National Institute of Standards and Technologys (NIST) GPS data. The results obtained from these metric computations are elaborately discussed. Finally, we discuss the integration of the procedures involved, like real-time data ingestion, metric computation, and result visualization, in a novel microservices-based architecture for situational awareness.

Chinthavali, Supriya↗

Traffic Control via Connected and Automated Vehicles (CAVs): An Open-Road Field Experiment with 100 CAVs

The CIRCLES project aims to reduce instabilities in traffic flow, which are naturally occurring phenomena due to human driving behavior. Also called “phantom jams” or “stop-and-go waves,” these instabilities are a significant source of wasted energy. Toward this goal, the CIRCLES project designed a control system, referred to as the MegaController by the CIRCLES team, that could be deployed in real traffic. Our field experiment, the MegaVanderTest (MVT), leveraged a heterogeneous fleet of 100 longitudinally controlled vehicles as Lagrangian traffic actuators, each of which ran a controller with the architecture described in this article. The MegaController is a hierarchical control architecture that consists of two main layers. The upper layer is called the Speed Planner and is a centralized optimal control algorithm. It assigns speed targets to the vehicles, conveyed through the LTE cellular network. The lower layer is a control layer, running on each vehicle. It performs local actuation by overriding the stock adaptive cruise controller, using the stock onboard sensors. The Speed Planner ingests live data feeds provided by third parties as well as data from our own control vehicles and uses both to perform the speed assignment. The architecture of the Speed Planner allows for the modular use of standard control techniques, such as optimal control, model predictive control (MPC), kernel methods, and others. The architecture of the local controller allows for the flexible implementation of local controllers. Corresponding techniques include deep reinforcement learning (RL), MPC, and explicit controllers. Depending on the vehicle architecture, all onboard sensing data can be accessed by the local controllers or only some. Likewise, control inputs vary across different automakers, with inputs ranging from torque or acceleration requests for some cars to electronic selection of adaptive cruise control (ACC) setpoints in others. The proposed architecture technically allows for the combination of all possible settings proposed previously, that is {Speed Planner algorithms} × {local Vehicle Controller algorithms} × {full or partial sensing} × {torque or speed control}. As a result, most configurations were tested throughout the ramp up to the MegaVandertest (MVT).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

PATHA: Performance Analysis Tool for HPC Applications

Large science projects rely on complex workflows to analyze terabytes or petabytes of data. These jobs are often running over thousands of CPU cores and simultaneously performing data accesses, data movements, and computation. It is difficult to identify bottlenecks or to debug the performance issues in these large workflows. In order to address these challenges, we have developed Performance Analysis Tool for HPC Applications (PATHA) using the state-of-art open source big data processing tools. Our framework can ingest system logs to extract key performance measures, and apply the most sophisticated statistical tools and data mining methods on the performance data. Furthermore, it utilizes an efficient data processing engine to allow users to interactively analyze a large amount of different types of logs and measurements. To illustrate the functionality of PATHA, we conduct a case study on the workflows from an astronomy project known as the Palomar Transient Factory (PTF). This study processed 1.6 TB of system logs collected on the NERSC supercomputer Edison. Using PATHA, we were able to identify performance bottlenecks, which reside in three tasks of PTF workflow with the dependency on the density of celestial objects.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Storage for Advanced Scientific Use-Cases and Beyond

The dCache project provides open-source storage software deployed internationally to satisfy ever more demanding scientific storage requirements. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, through wide access and easy integration with existing systems. In this paper, we describe some of the recent features that facilitate the use of storage to maximise the gain from stored data, including quality-of-service management, heterogeneous systems - both through integrated tertiary storage support and geographical locality - the parallel NFS (pNFS) extension, and innovative delegated authorisation schemes.

43 PARTICLE ACCELERATORS↗

Navigating Exascale Operational Data Analytics: From Inundation to Insight

In this paper, we address the challenges in achieving sustainable data-driven efficiency by providing a detailed exploration of the end-to-end operational data analytics (ODA) framework that evolved through two generations of supercomputer systems at the Oak Ridge Leadership Computing Facility (OLCF). This framework addresses large data streams ingested from heavily instrumented HPC environment that accumulates multi-terabytes per day. We outline the multifaceted data life cycle across HPC procurement, operations, and research & development, identifying key obstacles and design decisions that shape effective strategies in building and supporting data pipelines end-to-end. By sharing key insights and lessons learned from our experience, we offer recommendations for the HPC community on enabling sustainable operational data analytics and beyond. Our contributions aim to bridge the gap between potential and real benefits of operational data, guiding future efforts towards integrated and sustainable operational intelligence in high-performance computing environments.

Shin, Woong↗

OpenSAMPL: An Open Source Library for Timing and Synchronization Measurements and Analytics

Today's power grid operators are implementing timing and synchronization solutions that provide resilience to Global Navigation Satellite System (GNSS) vulnerabilities. These vendor-specific solutions often come with additional software applications that are designed to monitor that vendor's synchronization performance data. However, resilient timing architectures often resulting in multi-vendor solutions, including approaches that blend terrestrial clocks with space-based subscription services. In such an environment, collecting, analyzing, and visualizing data from a variety of sources within a single platform was heretofore not possible. To address this need, the US Department of Energy's Center for Alternative Synchronization and Timing (CAST) developed OpenSAMPL, the Open Synchronized Analytics and Monitoring Platform, an open-source Python framework for processing, loading, and observing clock measurement data from distributed devices. OpenSAMPL enables the ingestion of diverse clock-probe sources into a scalable time-series database and applies robust analytics. OpenSAMPL currently supports two vendor data pipelines, and will be extended to more in the near future, enabling seamless monitoring of a variety of timing and synchronization devices in a common environment.

Grant, Josh [ORNL] (ORCID:0000000163475060)↗

Evaluation and Demonstration of Blockchain Applicability Framework

Blockchain technology has been gaining great interest from a variety of industry sectors, including financial, food processing, and power and energy markets. Realizing the strength of blockchain technology beyond the successful application in the cryptocurrency arena, researchers have been evaluating and using blockchain for applications such as supply chain management, transactive industry (both financial and energy), system integrity, device cybersecurity, identity management, and much more. One of the unique elements of the blockchain technology that made it such a captivating technology to researchers is its plethora of features. Some of the features include smart contracts, cryptocurrency and tokenizing, immutable distributed ledger, cryptographic hashing, and digital signature. In addition, there are multiple types of blockchains, such as permissioned/private and permissionless/public, and various consensus models, such as proof-of-work, proof-of-authority, proof-of-burn, and proof-of-stake. Therefore, it is often non-trivial to determine if an application requires a blockchain. If so, what kind of blockchain and consensus is most appropriate? This paper discusses the blockchain applicability framework (BAF), which was specifically designed with the purpose to answer those questions. BAF is divided into five domains, 18 subdomains, and about 100 controls. It is designed to ingest detailed user requirements to perform a weighted evaluation that is built on mathematical constructs to determine the ideal combination of blockchain that is appropriate for an application. Along with the core logical formulation of BAF, this paper depicts the efficacy of BAF through two use cases

Gourisetti, Sri Nikhil G.↗

Human Factors in Streaming Data Analysis: Challenges and Opportunities for Information Visualization: Human Factors in Streaming Data Analysis

State-of-the-art visual analytics models and frameworks mostly assume a static snapshot of the data, while in many cases it is a stream with constant updates and changes. Exploration of streaming data poses unique challenges as machine-level computations and abstractions need to be synchronized with the visual representation of the data and the temporally evolving human insights. In the visual analytics literature, we lack a thorough characterization of streaming data and analysis of the challenges associated with task abstraction, visualization design, and adaptation of the role of human-in-the-loop for exploration of data streams. We aim to fill this gap by conducting a survey of the state-of-the-art in visual analytics of streaming data for systematically describing the contributions and shortcomings of current techniques and analyzing the research gaps that need to be addressed in the future. Our contributions are: i) problem characterization for identifying challenges that are unique to streaming data analysis tasks, ii) a survey and analysis of the state-of-the-art in streaming data visualization research with a focus on the visualization design space for dynamic data and the role of the human-in-the-loop, and iii) reflections on the design-trade-offs for streaming visual analytics techniques and their practical applicability in real-world application scenarios.

streaming data↗

Anti‐adhesive effects of sialic acid and Lactobacillus plantarum on Staphylococcus aureus in vitro

Abstract Staphylococcus aureus (S. aureus) is a common food‐borne pathogen that causes severe diseases after adhesion to epithelial cells. Lactobacillus inhibits pathogenic bacterial adhesion and infection. In addition, sialic acid (SA) is widely known for its beneficial biological functions. A new way of reducing the occurrence of diseases and curbing the overuse of antibiotics is ingesting prebiotics and probiotics that regulate the intestinal flora. In this study, we first evaluated the anti‐adhesive effects of several strains of Lactobacillus on S. aureus . The study revealed that the S. aureus adhesion was inhibited by all the strains of Lactobacillus . Besides, the rate of inhibition by L. plantarum Z‐4 was significantly higher than other Lactobacillus species . We then investigated the effects of different SA concentrations (40, 100, 150, 200, and 260 μg/ml) on the growth and adhesion characteristics of L. plantarum and S. aureus . The results showed that SA influences bacterial adhesion by regulating the bacteria's growth characteristics. Finally, the effects of SA combined with Lactobacillus on the adhesion of S. aureus were assessed by competition, exclusion and displacement methods. SA with a concentration of 260 μg/mL combined with L. plantarum had the highest inhibition effect on the competition assays. In addition, the expression of S. aureus adhesion‐related genes was reduced. This provides a new perspective on the application of SA and/or L. plantarum and its potential to resist adhesion of S. aureus .

Lu, Yingying↗

The effects of allelochemical transfer on the dynamics of hosts, parasitoids, and competing hyperparasitoids

Allelochemicals produced by plants may be ingested by herbivorous insects and transferred to higher trophic levels with potentially deleterious effects. We develop a system of differential equations to investigate the effect of the transfer of allelochemicals, such as nicotine, on the population dynamics of a system of hosts, parasitoids, and two competing hyperparasitoids that attack different life stages of the parasitoids. We find both somewhat deleterious effects of nicotine on the larvae-attacking hyperparasitoids and increased attack rates for the pupae-attacking hyperparasitoids can promote coexistence. We also use an evolutionary game-theoretic approach to determine the optimal distribution of hyperparasitoid attacks among nicotine-producing and nicotine-free plants. With strong deleterious effects of nicotine and increased attack rates for the pupae-attacking hyperparasitoid, we find both species attack parasitoids on the nicotine-free plant but only pupae-attacking hyperparasitoids attack parasitoids on the nicotine-producing plant.

59 BASIC BIOLOGICAL SCIENCES↗

Calibrating constitutive models with full‐field data via physics informed neural networks

Abstract The calibration of solid constitutive models with full‐field experimental data is a long‐standing challenge, especially in materials that undergo large deformations. In this paper, we propose a physics‐informed deep‐learning framework for the discovery of hyperelastic constitutive model parameterizations given full‐field surface displacement data and global force‐displacement data. Contrary to the majority of recent literature in this field, we work with the weak form of the governing equations rather than the strong form to impose physical constraints upon the neural network predictions. The approach presented in this paper is computationally efficient, suitable for irregular geometric domains, and readily ingests displacement data without the need for interpolation onto a computational grid. A selection of canonical hyperelastic material models suitable for different material classes is considered including the Neo–Hookean, Gent, and Blatz–Ko constitutive models as exemplars for general non‐linear elastic behaviour, elastomer behaviour with finite strain lock‐up, and compressible foam behaviour, respectively. We demonstrate that physics informed machine learning is an enabling technology and may shift the paradigm of how full‐field experimental data are utilized to calibrate constitutive models under finite deformations.

Hamel, Craig M.↗

Evaluating the Effects of Transient Purge Flow on Stator-Rotor Seal Performance

Abstract As modern engine designs target higher efficiencies through increased turbine inlet temperatures, critical turbine components are at increased risk of damage from conditions exceeding material melting temperatures. In particular, improperly designed underplatform hardware components are susceptible to damage when hot main gas path flow is ingested into the stator-rotor cavity. While all turbines inherently experience transients during operation, a majority of publicly available turbine studies have been executed using steady operating conditions or inherently transient “blowdown” rigs. For this reason, routine transient events are not well understood. To address this need, the present study utilized a combined experimental and computational approach. The test article is a continuous-duration, one-stage test turbine operating with true-scale engine hardware and seal geometries at engine-representative flow conditions. The nature of the continuous-duration facility uniquely supports the direct assessment of transient events through its ability to transition between steady-state operating conditions. The effects of a transient purge flow were investigated in this study to identify general trends for transient events in a full-scale engine. Results from multiple measurement techniques in the wheelspace region show an interdependence of transient purge flow with a thermal lag of the underplatform hardware. Through experiments conducted at different coolant-to-main gas path temperature ratios, the use of pressure measurements as an indicator of fully purged behavior was introduced, and a thermally driven influence on rim seal performance was quantified. The computational results show good agreement with experimental pressure measurements and provide insight into the physical mechanisms that drive the relationship between pressure and sealing effectiveness measurements observed for the tested geometry.

Engineering↗

SQL and NoSQL Databases for Cyber Physical Production Systems in Internet of Things for Manufacturing (IoTfM)

Abstract In this paper, the design and performance differences between Relational Database Management Systems (RDBMS) and NoSQL Database Systems are examined, with attention to their applicability for real-world Internet of Things for manufacturing (IoTfM) data. While previous work has extensively compared SQL and NoSQL for both generalized and IoT uses, this work specifically examines the tradeoffs and performance differences for manufacturing applications by using a high-fidelity data set collected from a large US manufacturing firm. Growing an IoT system beyond the pilot stage requires scalable data storage; this work seeks to determine the impact of selected database systems on data write performance at scale. Payload size and message frequency were used as the primary characteristics to maintain model fidelity in simulated clients. As the number of simulated asset clients grow, the data write latency was calculated to determine how both database systems’ performance were affected. To isolate the RDBMS and NoSQL differences, a cloud environment was created using Amazon Web Services (AWS) with two identical data ingestion pipelines: writing data to an RDMBS (1) using AWS Aurora MySQL, and (2) using AWS DynamoDB NoSQL. The findings may provide guidance for further experimentation in large-scale manufacturing IoT implementations.

Gamero, David↗