Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data processing and logging”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Are System Baselines within OT Environments Feasible?

Critical infrastructure stakeholders need to baseline their systems to understand expected protocol communications.Baseline behaviors may vary based on operational context.Expected operations during a maintenance window, for example, may be different from normal operations.Furthermore, constructing system baselines for Industrial Control Systems (ICS) is difficult and time-consuming.ICS processes generate artifacts expressed across heterogeneous data sources such as network and device logs. There needs to be a corpus of data in order to develop and compare methods that evaluate the feasibility, performance, and generality of approaches to construct baselines for ICS events. Standalone repositories of network packet captures are insufficient to develop methods to classify or recognize operational events expressed across multiple data sources. Moreover, static data corpora do not enable researchers to compare the impact of changing the underlying system for which a baseline is being constructed and this limits the ability to evaluate the performance of system baselines given system changes (e.g. patches, configuration, maintenance events). In order to address these limitations within the community, this talk intends to promote discussion about the state of the practice of constructing baselines. In this manner, we can continue to understand requirements within industry that are not being met by current approaches to baseline construction. This talk builds on two previous talks on the topic of system baselines for OT environments. First, Weaver co-presented at the RSA Conference ICS Sandbox with Dan Gunter. The talk confirmed the need within industry to construct baselines across multiple types of data sources relative to the semantics of specific business processes. Second, Weaver presented at IEEE Security and Privacy Workshop on Language-Theoretic Security.

02 PETROLEUM↗

On The Processing of Log Files for Monitoring Antenna Health

In order to improve the quality of geodetic results, we have developed an infrastructure for timely processing of telemetry from IVS observing stations. We check every hour for new log files with telemetry from both VLBI observing sessions, single dish experiments, and stow-in data collection and automatically process them. The telemetry data we use is the system temperature, phase calibration phases and amplitudes, system equivalent flux density, and the differences between formatter clock and GPS clock. For the system temperature and phase calibration, processing includes filtering out outliers and computing averages and rms of the scatter in each scan. Furthermore, for the phase calibration we also compute the group delay and detect spurious signals. Cleaned and post-processed telemetry is archived. Our process detects abnormalities, such as, anomalously high system temperature, unstable phase calibration phases, jumps in the GPS and formatter clock differences, and others. With our procedure, the latency of detection of station abnormalities is reduced to less than two hours. Early detection of abnormalities reduces the amount of affected data since station personnel get early alerts. We discuss our experience of running this system since 2022.

Phase Calibration↗

On The Processing of Log Files for Monitoring Antenna Health

In order to improve the quality of geodetic results, we have developed an infrastructure for timely processing of telemetry from IVS observing stations. We check every hour for new log files with telemetry from both VLBI observing sessions, single dish experiments, and stow-in data collection and automatically process them. The telemetry data we use is the system temperature, phase calibration phases and amplitudes, system equivalent flux density, and the differences between formatter clock and GPS clock. For the system temperature and phase calibration, processing includes filtering out outliers and computing averages and rms of the scatter in each scan. Furthermore, for the phase calibration we also compute the group delay and detect spurious signals. Cleaned and post-processed telemetry is archived. Our process detects abnormalities, such as, anomalously high system temperature, unstable phase calibration phases, jumps in the GPS and formatter clock differences, and others. With our procedure, the latency of detection of station abnormalities is reduced to less than two hours. Early detection of abnormalities reduces the amount of affected data since station personnel get early alerts. We discuss our experience of running this system since 2022.

VLBI↗

Defining a Metric Space of Host Logs and Operational Use Cases

Host logs, in particular, Windows Event Logs, are a valuable source of information often collected by security operation centers (SOCs). The semi-structured nature of host logs inhibits automated analytics, and while manual analysis is common, the sheer volume makes manual inspection of all logs impossible. Although many powerful algorithms for analyzing time-series and sequential data exist, utilization of such algorithms for most cyber security applications is either infeasible or requires tailored, research-intensive preparations. In particular, basic mathematic and algorithmic developments for providing a generalized, meaningful similarity metric on system logs is needed to bridge the gap between many existing sequential data mining methods and this currently available but under-utilized data source. In this paper, we provide a rigorous definition of a metric product space on Windows Event Logs, providing an embedding that allows for the application of established machine learning and time-series analysis methods. We then demonstrate the utility and flexibility of this embedding with multiple use-cases on real data: (1) comparing known infected to new host log streams for attack detection and forensics, (2) collapsing similar streams of logs into semantically-meaningful groups (by user, by role), thereby reducing the quantity of data but not the content, (3) clustering logs as well as short sequences of logs to identify and visualize user behaviors and background processes over time. Overall, we provide a metric space framework for general host logs and log sequences that respects semantic similarity and facilitates a wide variety of data science analytics to these logs without data-specific preparations for each.

Verma, Miki↗

Creative Analytics of Mission Ops Event Messages

Historically, tremendous effort has been put into processing and displaying mission health and safety telemetry data; and relatively little attention has been paid to extracting information from missions time-tagged event log messages. Todays missions may log tens of thousands of messages per day and the numbers are expected to dramatically increase as satellite fleets and constellations are launched, as security monitoring continues to evolve, and as the overall complexity of ground system operations increases. The logs may contain information about orbital events, scheduled and actual observations, device status and anomalies, when operators were logged on, when commands were resent, when there were data drop outs or system failures, and much much more. When dealing with distributed space missions or operational fleets, it becomes even more important to systematically analyze this data. Several advanced information systems technologies make it appropriate to now develop analytic capabilities which can increase mission situational awareness, reduce mission risk, enable better event-driven automation and cross-mission collaborations, and lead to improved operations strategies: Industry Standard for Log Messages. The Object Management Group (OMG) Space Domain Task Force (SDTF) standards organization is in the process of creating a formal standard for industry for event log messages. The format is based on work at NASA GSFC. Open System Architectures. The DoD, NASA, and others are moving towards common open system architectures for mission ground data systems based on work at NASA GSFC with the full support of the commercial product industry and major integration contractors. Text Analytics. A specific area of data analytics which applies statistical, linguistic, and structural techniques to extract and classify information from textual sources. This presentation describes work now underway at NASA to increase situational awareness through the collection of non-telemetry mission operations information into a common log format and then providing display and analytics tools to provide in-depth assessment of the log contents. The work includes: Common interface formats for acquiring time-tagged text messages Conversion of common files for schedules, orbital events, and stored commands to the common log format Innovative displays to depict thousands of messages on a single display Structured English text queries against the log message data store, extensible to a more mature natural language query capability Goal of speech-to-text and text-to-speech additions to create a personal mission operations assistant to aid on-console operations. A wide variety of planned uses identified by the mission operations teams will be discussed.

events↗

Data Link Summary for Peer Review.

New generations of high-temperature (HT) sensors and electronics are enabling increased measurement speed and accuracy allowing collection of more accurate and relevant data by downhole tools. Unfortunately, this increased capability is often not realized due to the bottleneck in the uplink data transmission rates due to poor signal characteristics of HT wireline. The objective of this project is to enable the high transmission rate of raw data from downhole tools such as acoustic logging tools and seismic measurement devices to minimize the need for downhole signal processing. To achieve this objective, Sandia has undertaken the effort to develop an asymmetric high-temperature (HT), highspeed data link system for downhole tools capable of operating at temperatures of 210°C while taking advantage of existing wireline transmission channels. Current data rates over HT single-conductor wireline are limited to approximately 200 kbps. The goal system will be capable of transmitting data from the tool to the surface (uplink) at rates of > 1Mbps over 5,000 feet of single-conductor wireline as well as automatically adapt the data rate to the longer wirelines by adapting modern telecommunications techniques to operate on high temperature electronics. The data rate from the surface to the tool (downlink) will be significantly smaller but sufficient for command and control functions. While 5,000 feet of cable is the benchmark for this effort, improvements apply to all lengths of cable.

47 OTHER INSTRUMENTATION↗

MO-F-CAMPUS-I-01: A System for Automatically Calculating Organ and Effective Dose for Fluoroscopically-Guided Procedures

Purpose: A system was developed that automatically calculates the organ and effective dose for individual fluoroscopically-guided procedures using a log of the clinical exposure parameters. Methods: We have previously developed a dose tracking system (DTS) to provide a real-time color-coded 3D- mapping of skin dose. This software produces a log file of all geometry and exposure parameters for every x-ray pulse during a procedure. The data in the log files is input into PCXMC, a Monte Carlo program that calculates organ and effective dose for projections and exposure parameters set by the user. We developed a MATLAB program to read data from the log files produced by the DTS and to automatically generate the definition files in the format used by PCXMC. The processing is done at the end of a procedure after all exposures are completed. Since there are thousands of exposure pulses with various parameters for fluoroscopy, DA and DSA and at various projections, the data for exposures with similar parameters is grouped prior to entry into PCXMC to reduce the number of Monte Carlo calculations that need to be performed. Results: The software developed automatically transfers data from the DTS log file to PCXMC and runs the program for each grouping of exposure pulses. When the dose from all exposure events are calculated, the doses for each organ and all effective doses are summed to obtain procedure totals. For a complicated interventional procedure, the calculations can be completed on a PC without manual intervention in less than 30 minutes depending on the level of data grouping. Conclusion: This system allows organ dose to be calculated for individual procedures for every patient without tedious calculations or data entry so that estimates of stochastic risk can be obtained in addition to the deterministic risk estimate provided by the DTS. Partial support from NIH grant R01EB002873 and Toshiba Medical Systems Corp.

60 APPLIED LIFE SCIENCES↗

Hindsight logging for model training

In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.

Computer Science↗

The CCD/Transit Instrument (CTI) data-analysis system

The automated software system for archiving, analyzing, and interrogating data from the CCD/Transit Instrument (CTI) is described. The CTI collects up to 450 Mbytes of image-data each clear night in the form of a narrow strip of sky observed in two colors. The large data-volumes and the scientific aims of the project make it imperative that the data are analyzed within the 24-hour period following the observations. To this end a fully automatic and self evaluating software system has been developed. The data are collected from the telescope in real-time and then transported to Tucson for analysis. Verification is performed by visual inspection of random subsets of the data and obvious cosmic rays are detected and removed before permanent archival is made to the optical disc. The analysis phase is performed by a pair of linked algorithms, one operating on the absolute pixel-values and the other on the spatial derivative of the data. In this way both isolated and merged images are reliably detected in a single pass. In order to isolate the latter algorithm from the effects of noise spikes a 3x3 Hanning filter is applied to the raw data before the analysis is run. The algorithms reduce the input pixel-data to a database of measured parameters for each image which has been found. A contrast filter is applied in order to assign a detection-probability to each image and then x-y calibration and intensity calibration are performed using known reference stars in the strip. These are added to as necessary by secondary standards boot-strapped from the CTI data itself. The final stages involve merging the new data into the CTI Master-list and History-list and the automatic comparison of each new detection with a set of pre-defined templates in parameter-space to find interesting objects such as supernovae, quasars and variable stars. Each stage of the processing from verification to interesting image selection is performed under a data-logging system which both controls the pipe-lining of data through the system and records key performance monitor parameters which are built into the software. Furthermore, the data from each stage are stored in databases to facilitate evaluation, and all stages offer the facility to enter keyword-indexed free-format text into the data-logging system. In this way a large measure of certification is built into the system to provide the necessary confidence in the end results.

Cawson, M. G. M.↗

SU-E-T-184: Clinical VMAT QA Practice Using LINAC Delivery Log Files

Purpose: To evaluate the accuracy of volumetric modulated arc therapy (VMAT) treatment delivery dose clouds by comparing linac log data to doses measured using an ionization chamber and film. Methods: A commercial IMRT quality assurance (QA) process utilizing a DICOM-RT framework was tested for clinical practice using 30 prostate and 30 head and neck VMAT plans. Delivered 3D VMAT dose distributions were independently checked using a PinPoint ionization chamber and radiographic film in a solid water phantom. DICOM RT coordinates were used to extract the corresponding point and planar doses from 3D log file dose distributions. Point doses were evaluated by computing the percent error between log file and chamber measured values. A planar dose evaluation was performed for each plan using a 2D gamma analysis with 3% global dose difference and 3 mm isodose point distance criteria. The same analysis was performed to compare treatment planning system (TPS) doses to measured values to establish a baseline assessment of agreement. Results: The mean percent error between log file and ionization chamber dose was 1.0%±2.1% for prostate VMAT plans and −0.2%±1.4% for head and neck plans. The corresponding TPS calculated and measured ionization chamber values agree within 1.7%±1.6%. The average 2D gamma passing rates for the log file comparison to film are 98.8%±1.0% and 96.2%±4.2% for the prostate and head and neck plans, respectively. The corresponding passing rates for the TPS comparison to film are 99.4%±0.5% and 93.9%±5.1%. Overall, the point dose and film data indicate that log file determined doses are in excellent agreement with measured values. Conclusion: Clinical VMAT QA practice using LINAC treatment log files is a fast and reliable method for patient-specific plan evaluation.

60 APPLIED LIFE SCIENCES↗

Pulsed Neutron Capture for Monitoring CO 2 Storage with Enhanced Oil Recovery in Northern Michigan

The Midwest Regional Carbon Sequestration Partnership (MRCSP) was founded in 2003 as part of the U.S. Department of Energy’s (DOE’s) Regional Carbon Sequestration Partnership initiative. Since its founding, MRCSP has made significant strides toward making CCUS a viable option for states in the region. The public/private consortium, funded through the DOE Regional Carbon Sequestration Initiative, brings together nearly 40 industry partners and 10 states. Battelle, as the project lead, oversees research, development and operations and coordinates activities among the partners. The incremental, phased approach has built a valuable knowledge base for the industry and paved the way for commercial-scale adoption of CCUS technologies. From 2008 to 2020, MRCSP Phase III focused on the development of large-scale injection projects. This report is part of a series of reports prepared under the Midwestern Regional Carbon Sequestration Partnership (MRCSP) Phase III (Development Phase). These reports summarize and detail the findings of the work conducted under the Phase III project. Pulsed neutron capture (PNC) logging has been used as part of the MRCSP monitoring of CO 2 injection and storage during assessment of enhanced oil recovery (EOR) in several Northern Niagara Pinnacle Reef Trend (NNPRT) reefs in Michigan. This technique and data processing cost less than traditional fluid sampling and presents a low risk to well operations by utilizing through-tubing logging capabilities. Additionally, monitoring CO 2 migration and break-through can be conducted from a single monitoring well in the reservoir. A total of four reefs were selected for these studies to monitor CO 2 migration and to test the viability and effectiveness of the PNC technology for saturation monitoring in a low porosity and highly diverse lithology environment. This study provides assessment of the technology and recommendations for utilization of saturation analysis.

01 COAL, LIGNITE, AND PEAT↗

Retrieving Historical Electrorefining Data

Pyrochemical Operations began at Los Alamos National Laboratory (LANL) during 1962 (1). Electrorefining (ER) has been implemented as a routine process since the 1980’s. The process data that went through the ER operation was recorded but had never been logged in an online database. Without a database new staff members are hindered in their work by the lack of information. To combat the issue a database in Access was created to collect the historical data. The years from 2000 onward were entered and queries were created to analyze trends. These trends will aid engineering and operations staff to reach optimal performance for the startup of the new lines.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗

Convergence of Emerging Technologies - EAGL Test Information

The Emergency Automatic Gunshot Detection and Lockdown (EAGL) system provides automatic, autonomous, and timely gunshot detection in both indoor and outdoor environments. This system uses both wired and wireless devices. Self-contained wireless EAGL sensors passively “listen” for gunshot events. These devices also perform a single, daily supervisory heartbeat (HB) function to include a device self-check with reporting capability. Transmissions are received by an assigned EAGL Gateway, which translates the RF sensor data to a PoE network format solely for use by the EAGL system server. The server then performs additional processes after data receipt, which include but are not limited to: event validation and logging, GUI presentation, notifications, and other independent operations.

47 OTHER INSTRUMENTATION↗

Improved Discrete Approximation of Laplacian of Gaussian

An improved method of computing a discrete approximation of the Laplacian of a Gaussian convolution of an image has been devised. The primary advantage of the method is that without substantially degrading the accuracy of the end result, it reduces the amount of information that must be processed and thus reduces the amount of circuitry needed to perform the Laplacian-of- Gaussian (LOG) operation. Some background information is necessary to place the method in context. The method is intended for application to the LOG part of a process of real-time digital filtering of digitized video data that represent brightnesses in pixels in a square array. The particular filtering process of interest is one that converts pixel brightnesses to binary form, thereby reducing the amount of information that must be performed in subsequent correlation processing (e.g., correlations between images in a stereoscopic pair for determining distances or correlations between successive frames of the same image for detecting motions). The Laplacian is often included in the filtering process because it emphasizes edges and textures, while the Gaussian is often included because it smooths out noise that might not be consistent between left and right images or between successive frames of the same image.

Shuler, Robert L., Jr.↗

Language-Theoretic Data Analysis to Support ICS Protocol Baselining

Critical infrastructure stakeholders need to baseline their systems to understand expected protocol communications. Baseline behaviors may vary based on operational context. Expected operations during a maintenance window, for example, may be different from normal operations. Furthermore, constructing system baselines for Industrial Control Systems (ICS) is difficult and time-consuming. ICS processes generate artifacts expressed across heterogeneous data sources such as network traffic and device logs. This paper explores the hypothesis that such ICS artifacts form a language in the language-theoretic sense. From a theoretical perspective, the variety of implementations of ICS protocols and constrained environment of OT networks provide a rich application domain for language-theoretic approaches. We present several use cases related to the practical construction of system baselines: grammars for data fusion, language dialects for device fingerprinting, and security automata for system baselining

24 POWER TRANSMISSION AND DISTRIBUTION↗

HERO WEC Belt Test Data

The following submission includes raw and processed data from the 2024 Hydraulic and Electric Reverse Osmosis Wave Energy Converter (HERO WEC) belt tests conducted using NREL's Large Amplitude Motion Platform (LAMP). A description of the motion profiles run during testing can be found in the run log document. Data was collected using NREL's Modular Ocean Data AcQuisition (MODAQ) system in the form of TDMS files. Data was then processed using Python and MATLAB and converted to MATLAB workspace, parquet, and csv file formats. During Data processing, a low pass filter was applied to each array and the arrays were then resampled to common 10Hz timestamps. A MATLAB data viewer script is provided to quickly visualize these data sets. The following arrays are contained in each test data file: - Time: Unix seconds timestamp - Test_Time: Time in seconds since beginning of test - POS_OS_1001: Encoder position in degrees (the encoder is located on the secondary shaft of the spring return and is driven by the winch after a 4.5:1 gear reduction) - LC_ST_1001: Anchor load cell data in lbf - PRESS_OS_2002: Air spring pressure in psi This data set has been developed by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy Water Power Technologies Office.

16 TIDAL AND WAVE POWER↗

Condition-Based Maintenance of a Circulating Water System of a Canadian Nuclear Power Plant using Machine Learning and Statistical Tools

Canada Deuterium Uranium pressurized-heavy-water reactors (PHWR) are a type of nuclear power plant that generate clean and reliable energy. The scope of this work is to automate data analysis methodologies to inform a condition-based maintenance strategy of a circulating water system (CWS) of a PHWR. The multiunit CWS provides a continuous supply of water to cool steam condensers, even during transient scenarios, thereby improving the thermal efficiency. This work aims to develop a machine learning (ML) based approach to detect anomalies in heterogeneous data of a CWS in a PHWR to help inform a predictive maintenance strategy. The heterogeneous data include textual and numeric time series data for a PHWR. Natural-language-processing (NLP)-based models are used to analyze textual data contained in work orders and operator logs and an event-timeseries correlation detection method is applied to assist anomalies diagnoses for CWS. An ML model Robust Linear Model (RLM) is also used to remove the seasonal variations in the system variable distributions based on distributions of environmental variables. A machine learning model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), trained on both original data and data without any seasonal variations will then be used to detect if an anomaly exists. Thus, by moving to an automated methodology to detect, classify, and forecast anomalies, the maintenance strategy would be based on component condition instead of a time-based schedule.

97 - MATHEMATICS AND COMPUTING↗

Midwest Consortium for Wind Turbine Reliability and Optimization

This report provides an overview of the efforts aimed to establish a student focused laboratory apparatus that will enhance Purdue's ability to recruit and train students in topics related to the dynamics, operations and economics of wind turbines. The project also aims to facilitate outreach to students at Purdue and in grades K-12 in the State of Indiana by sharing wind turbine operational data. For this project, a portable wind turbine test apparatus was developed and fabricated utilizing an AirX 400W wind energy converter. This turbine and test apparatus was outfitted with an array of sensors used to monitor wind speed, turbine rotor speed, power output and the tower structural dynamics. A major portion of this project included the development of a data logging program used to display real-time sensor data and the recording and creation of output files for data post-processing. The apparatus was tested in an open field to subject the turbine to typical operating conditions and the data acquisition system was adjusted to obtain desired functionality to facilitate use for student projects in existing courses offered at Purdue University and Indiana University. Data collected using the data logging program is analyzed and presented to demonstrate the usefulness of the test apparatus related to wind turbine dynamics and operations.

17 WIND ENERGY↗