Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data processing and logging”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Summit Darshan Archival Dataset

Summit Darshan Archival Dataset contains 2021 Summit Darshan log data for 25 applications and is grouped into science domains. The dataset is processed, and all the propriety fields are anonymized. The resultant data is converted into a tabular structure and saved in parquet file format. In this notebook, we demonstrate how to access the data. Data Organization: The data is organized into two directories: Darshan total (`darshan_total`): List all the high levels generated by the `darshan-parser --total` command on `.darshan` files. There is one parquet file for each application. Note: `uid` and `exe` field are masked Darshan detail (`darshan_detail`): This data contains detailed job level log information extracted by command `darshan-parser` on the raw `.darshan` files. The data is sorted by directory hierarchy in the order of `year/month/day (2021/12/07)`. For instance, to get the data for a `job_id` 3819766 of application `App11`, which was executed on `2021-12-07`can be accessed as follows. Note:`uid` and `filename` fields are masked

97 MATHEMATICS AND COMPUTING↗

Are System Baselines within OT Environments Feasible?

Critical infrastructure stakeholders need to baseline their systems to understand expected protocol communications.Baseline behaviors may vary based on operational context.Expected operations during a maintenance window, for example, may be different from normal operations.Furthermore, constructing system baselines for Industrial Control Systems (ICS) is difficult and time-consuming.ICS processes generate artifacts expressed across heterogeneous data sources such as network and device logs. There needs to be a corpus of data in order to develop and compare methods that evaluate the feasibility, performance, and generality of approaches to construct baselines for ICS events. Standalone repositories of network packet captures are insufficient to develop methods to classify or recognize operational events expressed across multiple data sources. Moreover, static data corpora do not enable researchers to compare the impact of changing the underlying system for which a baseline is being constructed and this limits the ability to evaluate the performance of system baselines given system changes (e.g. patches, configuration, maintenance events). In order to address these limitations within the community, this talk intends to promote discussion about the state of the practice of constructing baselines. In this manner, we can continue to understand requirements within industry that are not being met by current approaches to baseline construction. This talk builds on two previous talks on the topic of system baselines for OT environments. First, Weaver co-presented at the RSA Conference ICS Sandbox with Dan Gunter. The talk confirmed the need within industry to construct baselines across multiple types of data sources relative to the semantics of specific business processes. Second, Weaver presented at IEEE Security and Privacy Workshop on Language-Theoretic Security.

02 PETROLEUM↗

Defining a Metric Space of Host Logs and Operational Use Cases

Host logs, in particular, Windows Event Logs, are a valuable source of information often collected by security operation centers (SOCs). The semi-structured nature of host logs inhibits automated analytics, and while manual analysis is common, the sheer volume makes manual inspection of all logs impossible. Although many powerful algorithms for analyzing time-series and sequential data exist, utilization of such algorithms for most cyber security applications is either infeasible or requires tailored, research-intensive preparations. In particular, basic mathematic and algorithmic developments for providing a generalized, meaningful similarity metric on system logs is needed to bridge the gap between many existing sequential data mining methods and this currently available but under-utilized data source. In this paper, we provide a rigorous definition of a metric product space on Windows Event Logs, providing an embedding that allows for the application of established machine learning and time-series analysis methods. We then demonstrate the utility and flexibility of this embedding with multiple use-cases on real data: (1) comparing known infected to new host log streams for attack detection and forensics, (2) collapsing similar streams of logs into semantically-meaningful groups (by user, by role), thereby reducing the quantity of data but not the content, (3) clustering logs as well as short sequences of logs to identify and visualize user behaviors and background processes over time. Overall, we provide a metric space framework for general host logs and log sequences that respects semantic similarity and facilitates a wide variety of data science analytics to these logs without data-specific preparations for each.

Verma, Miki↗

Data Link Summary for Peer Review.

New generations of high-temperature (HT) sensors and electronics are enabling increased measurement speed and accuracy allowing collection of more accurate and relevant data by downhole tools. Unfortunately, this increased capability is often not realized due to the bottleneck in the uplink data transmission rates due to poor signal characteristics of HT wireline. The objective of this project is to enable the high transmission rate of raw data from downhole tools such as acoustic logging tools and seismic measurement devices to minimize the need for downhole signal processing. To achieve this objective, Sandia has undertaken the effort to develop an asymmetric high-temperature (HT), highspeed data link system for downhole tools capable of operating at temperatures of 210°C while taking advantage of existing wireline transmission channels. Current data rates over HT single-conductor wireline are limited to approximately 200 kbps. The goal system will be capable of transmitting data from the tool to the surface (uplink) at rates of > 1Mbps over 5,000 feet of single-conductor wireline as well as automatically adapt the data rate to the longer wirelines by adapting modern telecommunications techniques to operate on high temperature electronics. The data rate from the surface to the tool (downlink) will be significantly smaller but sufficient for command and control functions. While 5,000 feet of cable is the benchmark for this effort, improvements apply to all lengths of cable.

47 OTHER INSTRUMENTATION↗

MO-F-CAMPUS-I-01: A System for Automatically Calculating Organ and Effective Dose for Fluoroscopically-Guided Procedures

Purpose: A system was developed that automatically calculates the organ and effective dose for individual fluoroscopically-guided procedures using a log of the clinical exposure parameters. Methods: We have previously developed a dose tracking system (DTS) to provide a real-time color-coded 3D- mapping of skin dose. This software produces a log file of all geometry and exposure parameters for every x-ray pulse during a procedure. The data in the log files is input into PCXMC, a Monte Carlo program that calculates organ and effective dose for projections and exposure parameters set by the user. We developed a MATLAB program to read data from the log files produced by the DTS and to automatically generate the definition files in the format used by PCXMC. The processing is done at the end of a procedure after all exposures are completed. Since there are thousands of exposure pulses with various parameters for fluoroscopy, DA and DSA and at various projections, the data for exposures with similar parameters is grouped prior to entry into PCXMC to reduce the number of Monte Carlo calculations that need to be performed. Results: The software developed automatically transfers data from the DTS log file to PCXMC and runs the program for each grouping of exposure pulses. When the dose from all exposure events are calculated, the doses for each organ and all effective doses are summed to obtain procedure totals. For a complicated interventional procedure, the calculations can be completed on a PC without manual intervention in less than 30 minutes depending on the level of data grouping. Conclusion: This system allows organ dose to be calculated for individual procedures for every patient without tedious calculations or data entry so that estimates of stochastic risk can be obtained in addition to the deterministic risk estimate provided by the DTS. Partial support from NIH grant R01EB002873 and Toshiba Medical Systems Corp.

60 APPLIED LIFE SCIENCES↗

Hindsight logging for model training

In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.

Computer Science↗

SU-E-T-184: Clinical VMAT QA Practice Using LINAC Delivery Log Files

Purpose: To evaluate the accuracy of volumetric modulated arc therapy (VMAT) treatment delivery dose clouds by comparing linac log data to doses measured using an ionization chamber and film. Methods: A commercial IMRT quality assurance (QA) process utilizing a DICOM-RT framework was tested for clinical practice using 30 prostate and 30 head and neck VMAT plans. Delivered 3D VMAT dose distributions were independently checked using a PinPoint ionization chamber and radiographic film in a solid water phantom. DICOM RT coordinates were used to extract the corresponding point and planar doses from 3D log file dose distributions. Point doses were evaluated by computing the percent error between log file and chamber measured values. A planar dose evaluation was performed for each plan using a 2D gamma analysis with 3% global dose difference and 3 mm isodose point distance criteria. The same analysis was performed to compare treatment planning system (TPS) doses to measured values to establish a baseline assessment of agreement. Results: The mean percent error between log file and ionization chamber dose was 1.0%±2.1% for prostate VMAT plans and −0.2%±1.4% for head and neck plans. The corresponding TPS calculated and measured ionization chamber values agree within 1.7%±1.6%. The average 2D gamma passing rates for the log file comparison to film are 98.8%±1.0% and 96.2%±4.2% for the prostate and head and neck plans, respectively. The corresponding passing rates for the TPS comparison to film are 99.4%±0.5% and 93.9%±5.1%. Overall, the point dose and film data indicate that log file determined doses are in excellent agreement with measured values. Conclusion: Clinical VMAT QA practice using LINAC treatment log files is a fast and reliable method for patient-specific plan evaluation.

60 APPLIED LIFE SCIENCES↗

Pulsed Neutron Capture for Monitoring CO 2 Storage with Enhanced Oil Recovery in Northern Michigan

The Midwest Regional Carbon Sequestration Partnership (MRCSP) was founded in 2003 as part of the U.S. Department of Energy’s (DOE’s) Regional Carbon Sequestration Partnership initiative. Since its founding, MRCSP has made significant strides toward making CCUS a viable option for states in the region. The public/private consortium, funded through the DOE Regional Carbon Sequestration Initiative, brings together nearly 40 industry partners and 10 states. Battelle, as the project lead, oversees research, development and operations and coordinates activities among the partners. The incremental, phased approach has built a valuable knowledge base for the industry and paved the way for commercial-scale adoption of CCUS technologies. From 2008 to 2020, MRCSP Phase III focused on the development of large-scale injection projects. This report is part of a series of reports prepared under the Midwestern Regional Carbon Sequestration Partnership (MRCSP) Phase III (Development Phase). These reports summarize and detail the findings of the work conducted under the Phase III project. Pulsed neutron capture (PNC) logging has been used as part of the MRCSP monitoring of CO 2 injection and storage during assessment of enhanced oil recovery (EOR) in several Northern Niagara Pinnacle Reef Trend (NNPRT) reefs in Michigan. This technique and data processing cost less than traditional fluid sampling and presents a low risk to well operations by utilizing through-tubing logging capabilities. Additionally, monitoring CO 2 migration and break-through can be conducted from a single monitoring well in the reservoir. A total of four reefs were selected for these studies to monitor CO 2 migration and to test the viability and effectiveness of the PNC technology for saturation monitoring in a low porosity and highly diverse lithology environment. This study provides assessment of the technology and recommendations for utilization of saturation analysis.

01 COAL, LIGNITE, AND PEAT↗

Retrieving Historical Electrorefining Data

Pyrochemical Operations began at Los Alamos National Laboratory (LANL) during 1962 (1). Electrorefining (ER) has been implemented as a routine process since the 1980’s. The process data that went through the ER operation was recorded but had never been logged in an online database. Without a database new staff members are hindered in their work by the lack of information. To combat the issue a database in Access was created to collect the historical data. The years from 2000 onward were entered and queries were created to analyze trends. These trends will aid engineering and operations staff to reach optimal performance for the startup of the new lines.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗

Convergence of Emerging Technologies - EAGL Test Information

The Emergency Automatic Gunshot Detection and Lockdown (EAGL) system provides automatic, autonomous, and timely gunshot detection in both indoor and outdoor environments. This system uses both wired and wireless devices. Self-contained wireless EAGL sensors passively “listen” for gunshot events. These devices also perform a single, daily supervisory heartbeat (HB) function to include a device self-check with reporting capability. Transmissions are received by an assigned EAGL Gateway, which translates the RF sensor data to a PoE network format solely for use by the EAGL system server. The server then performs additional processes after data receipt, which include but are not limited to: event validation and logging, GUI presentation, notifications, and other independent operations.

47 OTHER INSTRUMENTATION↗

Language-Theoretic Data Analysis to Support ICS Protocol Baselining

Critical infrastructure stakeholders need to baseline their systems to understand expected protocol communications. Baseline behaviors may vary based on operational context. Expected operations during a maintenance window, for example, may be different from normal operations. Furthermore, constructing system baselines for Industrial Control Systems (ICS) is difficult and time-consuming. ICS processes generate artifacts expressed across heterogeneous data sources such as network traffic and device logs. This paper explores the hypothesis that such ICS artifacts form a language in the language-theoretic sense. From a theoretical perspective, the variety of implementations of ICS protocols and constrained environment of OT networks provide a rich application domain for language-theoretic approaches. We present several use cases related to the practical construction of system baselines: grammars for data fusion, language dialects for device fingerprinting, and security automata for system baselining

24 POWER TRANSMISSION AND DISTRIBUTION↗

HERO WEC Belt Test Data

The following submission includes raw and processed data from the 2024 Hydraulic and Electric Reverse Osmosis Wave Energy Converter (HERO WEC) belt tests conducted using NREL's Large Amplitude Motion Platform (LAMP). A description of the motion profiles run during testing can be found in the run log document. Data was collected using NREL's Modular Ocean Data AcQuisition (MODAQ) system in the form of TDMS files. Data was then processed using Python and MATLAB and converted to MATLAB workspace, parquet, and csv file formats. During Data processing, a low pass filter was applied to each array and the arrays were then resampled to common 10Hz timestamps. A MATLAB data viewer script is provided to quickly visualize these data sets. The following arrays are contained in each test data file: - Time: Unix seconds timestamp - Test_Time: Time in seconds since beginning of test - POS_OS_1001: Encoder position in degrees (the encoder is located on the secondary shaft of the spring return and is driven by the winch after a 4.5:1 gear reduction) - LC_ST_1001: Anchor load cell data in lbf - PRESS_OS_2002: Air spring pressure in psi This data set has been developed by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy Water Power Technologies Office.

16 TIDAL AND WAVE POWER↗

Condition-Based Maintenance of a Circulating Water System of a Canadian Nuclear Power Plant using Machine Learning and Statistical Tools

Canada Deuterium Uranium pressurized-heavy-water reactors (PHWR) are a type of nuclear power plant that generate clean and reliable energy. The scope of this work is to automate data analysis methodologies to inform a condition-based maintenance strategy of a circulating water system (CWS) of a PHWR. The multiunit CWS provides a continuous supply of water to cool steam condensers, even during transient scenarios, thereby improving the thermal efficiency. This work aims to develop a machine learning (ML) based approach to detect anomalies in heterogeneous data of a CWS in a PHWR to help inform a predictive maintenance strategy. The heterogeneous data include textual and numeric time series data for a PHWR. Natural-language-processing (NLP)-based models are used to analyze textual data contained in work orders and operator logs and an event-timeseries correlation detection method is applied to assist anomalies diagnoses for CWS. An ML model Robust Linear Model (RLM) is also used to remove the seasonal variations in the system variable distributions based on distributions of environmental variables. A machine learning model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), trained on both original data and data without any seasonal variations will then be used to detect if an anomaly exists. Thus, by moving to an automated methodology to detect, classify, and forecast anomalies, the maintenance strategy would be based on component condition instead of a time-based schedule.

97 - MATHEMATICS AND COMPUTING↗

Midwest Consortium for Wind Turbine Reliability and Optimization

This report provides an overview of the efforts aimed to establish a student focused laboratory apparatus that will enhance Purdue's ability to recruit and train students in topics related to the dynamics, operations and economics of wind turbines. The project also aims to facilitate outreach to students at Purdue and in grades K-12 in the State of Indiana by sharing wind turbine operational data. For this project, a portable wind turbine test apparatus was developed and fabricated utilizing an AirX 400W wind energy converter. This turbine and test apparatus was outfitted with an array of sensors used to monitor wind speed, turbine rotor speed, power output and the tower structural dynamics. A major portion of this project included the development of a data logging program used to display real-time sensor data and the recording and creation of output files for data post-processing. The apparatus was tested in an open field to subject the turbine to typical operating conditions and the data acquisition system was adjusted to obtain desired functionality to facilitate use for student projects in existing courses offered at Purdue University and Indiana University. Data collected using the data logging program is analyzed and presented to demonstrate the usefulness of the test apparatus related to wind turbine dynamics and operations.

17 WIND ENERGY↗

Nevada National Security Site (NNSS) Environmental Remediation Using EPA's Model Evaluation Guidance to Move to Closure - 15588

The National Nuclear Security Administration Nevada Field Office (NNSA/NFO) has adapted the Environmental Protection Agency's guidance on environmental model evaluation as a framework for evaluating corrective action investigation models. Model evaluation is an important confidence building activity, which is needed to support closure of the Nevada National Security Site (NNSS) Corrective Action Units (CAUs) associated with underground nuclear testing. Through this process, site-investigation models of groundwater flow and contaminant transport are reviewed and compared to new site-specific data to determine if the results can be used to support regulatory decisions required for closure and to achieve NNSA/NFO site management objectives. The regulatory decisions include identifying and establishing CAU regulatory boundaries; identifying institutional controls, including use-restriction boundaries; and developing a long-term closure monitoring program. A successful case study from Frenchman Flat, Nevada is used to illustrate the model evaluation process. Groundwater flow and transport from two of the 10 underground nuclear tests in Frenchman Flat had the largest potential for contaminating groundwater, and were selected for evaluation. Wells were drilled, logged, hydraulically tested, sampled, and analyzed for radionuclides and the data was evaluated with respect to previous site conceptualization and numerical model results. The approach and results of this process as applied to the PIN STRIPE test and how it helped build confidence to close the Frenchman Flat CAU is described in this case study. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

EGS Collab Experiment 1: Wireline Geophysical Well Logs

This is the full wireline geophysical datasets for characterization of the EGS Collab Experiment #1 testbed on the 4850 level. A metadata file is included within the dataset explaining the logs, fracture picks, etc. Eight boreholes were drilled for this testbed and each one was logged with north seeking gyro, optical televiewer, acoustic televiewer, fluid temperature conductivity, resistivity and gamma, and full waveform sonic. In these folders are the processed results as text, csv, and pdf files, along with the raw data which will need to be read using WellCAD software.

15 GEOTHERMAL ENERGY↗

A fast partial Fourier transform (FPFT) for data compression and filtering.

A discrete Fourier transform (DFT) or the closely related discrete cosine transform (DCT) is often employed as part of a data compression scheme. This paper presents a fast partial Fourier transform (FPFT) algorithm that is useful for calculating a subset of M Fourier transform coefficients for a data set comprised of N points (M < N). This algorithm reduces to the standard DFT when M = 1 and it reduces to the radix-2, decimation-in-time FFT when M = N and N is a power of 2. The DFT requires on the order of MN complex floating point multiplications to calculate M coefficients for N data points, a complete FFT requires on the order of (N/2)log{sub 2}N multiplications independent of M, and the new FPFT algorithm requires on the order of (N/2)log{sub 2}M + N multiplications. The FPFT algorithm introduced in this paper could be readily adapted to parallel processing. In addition to data compression, the FPFT algorithm described in this paper might be useful for very narrow band filter operations that pass only a small number of non-zero frequency coefficients such that M << N.

99 GENERAL AND MISCELLANEOUS↗

Equipment Testing Environment (ETE) Process Specification

This document is intended to be utilized with the Equipment Test Environment being developed to provide a standard process by which the ETE can be validated. The ETE is developed with the intent of establishing cyber intrusion, data collection and through automation provide objective goals that provide repeatability. This testing process is being developed to interface with the Technical Area V physical protection system. The document will overview the testing structure, interfaces, device and network logging and data capture. Additionally, it will cover the testing procedure, criteria and constraints necessary to properly capture data and logs and record them for experimental data capture and analysis.

97 MATHEMATICS AND COMPUTING↗