Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “log analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

Using parallel file systems efficiently is a tricky problem due to inter-dependencies among multiple layers of I/O software, including high-level I/O libraries (HDF5, netCDF, etc.), MPI-IO, POSIX, and file systems (GPFS, Lustre, etc.). Profiling tools such as Darshan collect traces to help understand the I/O performance behavior. However, there are significant gaps in analyzing the collected traces and then applying tuning options offered by various layers of I/O software. Seeking to connect the dots between I/O bottleneck detection and tuning, we propose DXT Explorer, an interactive log analysis tool. In this paper, we present a case study using our interactive log analysis tool to identify and apply various I/O optimizations. We report an evaluation of performance improvement achieved for four I/O kernels extracted from science applications.

Bez, Jean Luca↗
DOE OSTI2025

Machine Learning for Well Log Analysis in Uranium Mining

This project explores the use of Artificial Intelligence (AI) and Machine Learning (ML) techniques to automate well log analysis for uranium mining. Geophysical log data—spontaneous potential, resistivity, and gamma ray—were used to classify lithology, correlate well logs and identify roll front zonation patterns, which are critical for locating uranium ore bodies. Supervised ML algorithms such as eXtreme Gradient Boosting (XGBoost), Categorical Boosting (CatBoost), and Random Forest were trained to classify lithology with high accuracy. Gradient Boosting Machines (GBM), XGBoost, Random Forest, and Neural Networks were also used for role front zone identification. Moreover, a Fast Dynamic Time Warping (FastDTW) algorithm was employed for well log correlation. Additionally, sample lag was addressed using dynamic programming. Results demonstrate the potential of AI and ML to streamline well log analysis and enhance uranium exploration workflows.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
NASA NTRS2012

Requirements-Driven Log Analysis Extended Abstract

Imagine that you are tasked to help a project improve their testing effort. In a realistic scenario it will quickly become clear, that having an impact is diffcult. First of all, it will likely be a challenge to suggest an alternative approach which is significantly more automated and/or more effective than current practice. The reality is that an average software system has a complex input/output behavior. An automated testing approach will have to auto-generate test cases, each being a pair (i; o) consisting of a test input i and an oracle o. The test input i has to be somewhat meaningful, and the oracle o can be very complicated to compute. Second, even in case where some testing technology has been developed that might improve current practice, it is then likely difficult to completely change the current behavior of the testing team unless the technique is obviously superior and does everything already done by existing technology. So is there an easier way to incorporate formal methods-based approaches than the full edged test revolution? Fortunately the answer is affirmative. A relatively simple approach is to benefit from possibly already existing logging infrastructure, which after all is part of most systems put in production. A log is a sequence of events, generated by special log recording statements, most often manually inserted in the code by the programmers. An event can be considered as a data record: a mapping from field names to values. We can analyze such a log using formal methods, for example checking it against a formal specification. This separates running the system for analyzing its behavior. It is not meant as an alternative to testing since it does not address the important in- put generation problem. However, it offers a solution which testing teams might accept since it has low impact on the existing process. A single person might be assigned to perform such log analysis, compared to the entire testing team changing behavior.

log analysis↗
DOE OSTI2020

Automatic log analysis with NLP for the CMS workflow handling

The central Monte-Carlo production of the CMS experiment utilizes the WLCG infrastructure and manages daily thousands of tasks, each up to thousands of jobs. The distributed computing system is bound to sustain a certain rate of failures of various types, which are currently handled by computing operators a posteriori. Within the context of computing operations, and operation intelligence, we propose a Machine Learning technique to learn from the operators with a view to reduce the operational workload and delays. This work is in continuation of CMS work on operation intelligence to try and reach accurate predictions with Machine Learning. We present an approach to consider the log files of the workflows as regular text to leverage modern techniques from Natural Language Processing (NLP). In general, log files contain a substantial amount of text that is not human language. Therefore, different log parsing approaches are studied in order to map the log files’ words to high dimensional vectors. These vectors are then exploited as feature space to train a model that predicts the action that the operator has to take. This approach has the advantage that the information of the log files is extracted automatically and the format of the logs can be arbitrary. In this work the performance of the log file analysis with NLP is presented and compared to previous approaches.

Layer, Lukas↗
DOE OSTI2026

Database-Agnostic Log Analysis and Monitoring Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, US, IL; Fermilab]↗
DOE OSTI2023

Design and implementation of I/O performance prediction scheme on HPC systems through large-scale log analysis

Abstract Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units used by hundreds to thousands of users simultaneously. Applications from large numbers of users have diverse characteristics, such as varying computation, communication, memory, and I/O intensity. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, I/O performance is becoming increasingly important as data sizes rapidly increase and large-scale applications, such as simulation and model training, are widely adopted. However, predicting I/O performance is difficult because I/O systems are shared among all users and involve many layers of software and hardware stack, including the application, network interconnect, operating system, file system, and storage devices. Furthermore, updates to these layers and changes in system management policy can significantly alter the I/O behavior of applications and the entire system. To improve the prediction of the I/O performance on HPC systems, we propose integrating information from several different system logs and developing a regression-based approach to predict the I/O performance. Our proposed scheme can dynamically select the most relevant features from the log entries using various feature selection algorithms and scoring functions, and can automatically select the regression algorithm with the best accuracy for the prediction task. The evaluation results show that our proposed scheme can predict the write performance with up to 90% prediction accuracy and the read performance with up to 99% prediction accuracy using the real logs from the Cori supercomputer system at NERSC.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Strym: A Python Package for Real-time CAN Data Logging, Analysis and Visualization to Work with USB-CAN Interface

In this report, we describe a data analysis tool developed for decoding and analyzing vehicle data obtained from a passenger vehicle’s onboard controller area network (CAN) bus. The tool developed in this paper provides a timeseries framework to perform domain-specific analysis at scale when interpreting data from a vehicle or a collection of vehicles in light of how to design intelligent vehicle applications. The tool, called Strym, exploits the CAN bus mechanism of modern vehicles to capture data using commercially available CAN-to-USB hardware Comma.ai Panda devices, managed through open-source software Libpanda. Strym permits the decoding of vendor-specific CAN messages in a vehicle-agnostic manner. Through this, a researcher can characterize data throughput, assess data quality, and perform analyses. Such analyses are useful in a number of research such as studying human driving behavior in mixed-autonomy, new driver models, rare-event detection, traffic flow estimation, and custom control of vehicles.

Performance evaluation, Smart cities, Intelligent ↗
NASA NTRS2004

User Evaluation of the NASA Technical Report Server Recommendation Service

We present the user evaluation of two recommendation server methodologies implemented for the NASA Technical Report Server (NTRS). One methodology for generating recommendations uses log analysis to identify co-retrieval events on full-text documents. For comparison, we used the Vector Space Model (VSM) as the second methodology. We calculated cosine similarities and used the top 10 most similar documents (based on metadata) as 'recommendations'. We then ran an experiment with NASA Langley Research Center (LaRC) staff members to gather their feedback on which method produced the most 'quality' recommendations. We found that in most cases VSM outperformed log analysis of co-retrievals. However, analyzing the data revealed the evaluations may have been structurally biased in favor of the VSM generated recommendations. We explore some possible methods for combining log analysis and VSM generated recommendations and suggest areas of future work.

Nelson, Michael L.↗
NASA NTRS2004

User Evaluation of the NASA Technical Report Server Recommendation Service

We present the user evaluation of two recommendation server methodologies implemented for the NASA Technical Report Server (NTRS). One methodology for generating recommendations uses log analysis to identify co-retrieval events on full-text documents. For comparison, we used the Vector Space Model (VSM) as the second methodology. We calculated cosine similarities and used the top 10 most similar documents (based on metadata) as recommendations . We then ran an experiment with NASA Langley Research Center (LaRC) staff members to gather their feedback on which method produced the most quality recommendations. We found that in most cases VSM outperformed log analysis of co-retrievals. However, analyzing the data revealed the evaluations may have been structurally biased in favor of the VSM generated recommendations. We explore some possible methods for combining log analysis and VSM generated recommendations and suggest areas of future work.

Nelson, Michael L.↗
DOE OSTI2022

Intelligent Process Visualization through Nuclear Operation Process Modeling, Reasoning, and Object Detection from Field Videos (Final Report)

This report is a deliverable for the “Final Report” task of DOE NEET Project 19-16790, "Context-Aware Safety Information Display for Nuclear Field Workers." This project's overall goal is to test the hypothesis that integrating computer vision and process reasoning methods will enable proactive visualization of the safe operation and maintenance processes of Nuclear Power Plants (NPP) for field workers. Augmented Reality (AR) glasses adopting such proactive safety information visualization techniques can significantly increase personnel safety and reduce the NPP’s operating costs. The current practice of monitoring NPPs requires workers to switch between digital models, data, and physical workspaces in identifying relevant but potentially occluded objects and in assessing the risks of operation and maintenance processes. On the other hand, frequently changed field conditions require field workers to report to supervisors for real-time guidance. Such guidance is essential to ensure that changing conditions will not invalidate or endanger the work order and other ongoing processes that may jeopardize NPP operations. Additionally, incorrect recognition of equipment objects can result in communication errors and safety problems. AR techniques can assist engineers in viewing the physical workspaces with objects labeled with detailed operation procedures and safety reminders during field operations. The project team developed an “Intelligent Context-Aware Safety Information Display” (ICAD) for supporting Nuclear Power Plant (NPP) field workers in achieving safe and efficient execution of a series of operational tasks in uncertain and changing workspaces of an NPP. Before designing the ICAD-AR prototype, the project team synthesized NPP operational knowledge models through literature review studies, surveys, interviews with domain experts, and knowledge modeling. The project team conducted an extensive study of the operational procedures of various NPPs, and digital technologies that can support the safe and efficient execution of those procedures in different NPP operational contexts. This literature review helped the project team conduct surveys and interviews with nuclear engineers and field workers to identify three categories of information. The NPP knowledge modeling efforts reveal that the three categories of information identified have different levels of importance in a typical procedure of carrying out a series of tasks to achieve a specific NPP operation goal (e.g., shutdown, mode changes). These three categories of information include 1) Workspace dynamics – the changing spatial arrangements of workspaces, tools, protection equipment, and supporting materials, 2) Workflow prognostics – the dynamic dependencies between different parts of an NPP that functionally support and influence each other in terms of safety and efficiency, and 3) Hazards – objects and spaces that contain hazardous materials or physical conditions that can pose risks to workers or mechanical systems. The project team has profiled the importance levels of these categories of information into a knowledge model. This knowledge model specifies what types of information are more critical for a given task in a given workspace so that computers can automatically identify critical objects and sensors in a scene for delivering context-ware safety information to field workers through AR devices. Significant research development of this project results in technical research outcomes and a prototyping system that illustrates the technical feasibility of establishing an ICAD-AR system supporting the proactive safety information display for nuclear field workers. This final report summarizes the project team’s technological achievements in the past three years. Overall, the project team completed the development and integration of five techniques into a prototype ICAD Augmented Reality (ICAD-AR) system and demonstrated the developed system’s real-time execution in a mechanical room. The project team completed the analysis of using this prototype in other types of workspaces based on 3D image data and digital design models collected from two additional workspaces (a water treatment plant and a flow loop training facility). The integrated techniques include 1) Natural Language Processing (NLP) algorithms supporting the generation and updates of nuclear fieldwork process models based on text analysis of work packages and operation manuals; 2) sensor log analysis for predicting control actions in given sensor reading contexts; 3) computer vision algorithms for automatic localization and navigation of workers; 4) object detection algorithms for identifying task-related objects and correlated sensors for safety checking; 5) AR technique as a platform for supporting the integration. The testing results of these five techniques have shown that 1) the sensor log analysis model can predict the next control action with an accuracy of 0.883; 2) the trained natural language processing model can extract more than 80% of the critical information from paper-based procedures (PBPs); 3) the navigation algorithm with the integration of Visual Inertial Odometry (VIO) and Non-Recursive Bayesian Filter methods make operator’s trajectory estimation resilient to drift error; 4) the computer vision algorithm can detect task-specific and safety-critical objects with an average accuracy of 95.3%. The project team used work procedures collected from a flow loop training facility and two datasets collected from two mechanical rooms simulating the workspaces of NPPs to demonstrate the technical capabilities of the developed ICAD-AR prototype. The demonstration validated the technical feasibility of establishing the ICAD-AR system for nuclear field workers and identified the challenges in 1) automatic text analysis of work packages; 2) use of limited samples of sensor logs for predicting the proper timings of control actions; 3) reliably tracking workers and their task progress in mechanical rooms with many similar objects.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Assessing pore network heterogeneity across multiple scales to inform CO2 injection models

Geologic heterogeneity is a key feature that must be considered when translations of scaled data are performed. This paper presents the assessment of geologic heterogeneity using a multiscale workflow that includes image analysis-based methods coupled with well log analysis to provide data in which fractals and machine learning methods estimate the carbon dioxide (CO 2 ) storage resource potential of a reservoir. The heterogeneity of rock properties of the complex Bell Creek reservoir in Montana, USA, was explored at the pore scale (~nm to mm), core scale (~mm to m), and well scale (~cm to m). The data used in this study included advanced image analysis of micro-CT (computed tomography) images (pore scale), thin sections (pore scale), plugs and core images (core scale) and well logs (well scale). The micro-CT images were segmented using a U-net segmentation approach into objects of pores and grains. Further, the segmented images were reconstructed into subvolumes of different sizes. Physical properties (porosity and permeability) and fractal dimensions were calculated for the various subvolumes, and Lorenz coefficient (Lc) values, a single parameter to describe the degree of heterogeneity within a pay zone section, were calculated from thin-section images and well logs. Porosity and fractal dimension values were used to estimate the 188-µm threshold of representative elementary volume (REV) in this study. Both the Lc and fractal dimension values were found to be negatively correlated. When these two parameters are combined, it is possible to discern differences in the complex porous networks of the samples analyzed in this study.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

Log-N/Period Sensitivity Analysis

The Log-N/Period system utilizes a compensated ion chamber to measure neutron radiation in the reactor core. This analysis focuses on determining a more accurate value for the sensitivity of the installed compensated ion chambers for the Log-N/Period system at the ATR. Using calibration procedures a formula for indirectly calculating the sensitivity of the chambers was derived. Using historical data from approximately ten reactor cycles a new average sensitivity for both channels of the Log-N/Period system was determined.

42 ENGINEERING↗
DOE OSTI2020

Theoretical power-law relationship between permeability and formation factor

Permeability k and formation factor F are important petrophysical properties that have broad applications to reservoir engineering. Although well-log analysis has advanced reservoir characterizations, direct measurement of k is still not available. Accordingly, resistivity logs and more particularly formation factor have been frequently used to estimate k. In this study, we present a theoretical framework to relate k to F. We use power-law scaling from continuum percolation theory to develop a theoretic relationship between k and F, which is similar in form to a published power-law equation with empirical coefficients. Our result gives a physical interpretation to this empirical relationship. We also compare power-law scaling laws from percolation theory with data derived from experimental measurements and numerical simulations in the literature. Finally, using the Curve Fitting toolbox of MATLAB, the parameters of the power-law equations are determined, and the significance of the non-universal power laws are discussed.

42 ENGINEERING↗
DOE OSTI2026

Improving Production in the Emerging Paradox Oil Play (Final Technical Report)

This report documents the results of the DOE-funded project Improving Production in the Emerging Paradox Oil Play, focused on integrating geological characterization, petrophysical analysis, reservoir simulation, and field development planning to improve hydrocarbon recovery from the Paradox Basin. Project activities included core and log analysis, assessment of reservoir heterogeneity and compartmentalization, evaluation of completion and stimulation strategies, and development of an uncertainty-aware field development framework. Findings highlight the importance of selective completions, geology-driven well placement, and targeted stimulation in vertically heterogeneous carbonate reservoirs.

02 PETROLEUM↗
NASA NTRS2020

What Can We Learn from One Billion Ground System Log Messages?

Shortage of log-based data in a ground system they have traditionally been the under achievers in a satellite ground system. This is due to several factors: Once log messages scroll out of view on the TTC event console window they are soon forgotten. Application and system log files are scattered across directories within a system, across a multitude of servers, and across one or more databases making access cumbersome. Typical tools to perform log file content searching are generally crude and typically only employed as part of trouble-shooting exercises.As we move towards satellite constellations and fleets and add even more status information, the number of messages keeps growing. One mission now estimates that they could generate 3,000,000 messages per day 1 billion per year - for the life of their mission. What to do with those 1 billion messages? That is the challenge. With the recent technological advances in the management of large data sets, text-based processing, and data analytics, there are now capabilities that we can provide to the ground system engineers and satellite operators to address what we postulate are missed opportunities. Advanced real-time log analysis can allow us to be less reactionary in favor of being more proactive. Analytics goals include the ability to: Identify root cause of unexpected events, failures or error conditions enabled by correlating disparate data. Detect security breaches attempts before they are successful. Help admins ensure IT resources continue running optimally. Identify trends and patterns that may indicate impending failures or error conditions for valuable assets before they happen. Compare satellites in a fleet or constellation in terms of number of alarms, number of command sent to them, etc.. Answer questions like "Are the operations support needs increasing over the past year?" or "Have we seen this combination of alarm conditions before?" But really, once the tools are readily available the users will start realizing what can be done with their new powers. In this presentation we will show the results of analyzing millions of actual mission operations log messages, how the results can be displayed to the user, and how new products now available as open source can be applied to the challenges of large scale time-tagged text-based mission operations messages. Flight operations team members believe that this is a powerful new option for how they assess overall system and space asset health. Technical descriptions of the design, tools, and storage will be provided. One billion messages? Bring'em on!

Orsborne, Sharon↗
DOE OSTI2020

Wellbore injectivity response to step-rate CO 2 injection: Coupled thermo-poro-elastic analysis in a vertically heterogeneous formation

Safe and permanent carbon geological storages require an elaborate analysis of geomechanical stability. Subsurface injection of CO 2 changes the local temperature and pore pressure and, further, alters the stress due to thermo-poro-elastic responses. A permanent CO 2 storage testing was conducted in the water leg of Cranfield reservoir in Mississippi, USA, where hosted CO 2 enhanced-oil recovery activities. During CO 2 injection, the injection rate was ramped up twice but the bottom-hole pressure did not increase with the imposed injection rates as expected. This unexpected field observation suggests the possibility of an open-mode fracture development at the injector. However, the injector response and potential fracture development have not been rigorously interpreted with detailed near-wellbore temperature and pore pressure change upon the CO 2 injection. In this study, we performed history matching of CO 2 injection using coupled thermo-poro-elastic reservoir simulation to examine the possibility of fracturing. We built a reservoir model including vertical heterogeneity in both petrophysical and geomechanical properties estimated from well-logging analysis and laboratory experiments. The simulation results show that CO 2 injection changes stresses and support the hypothesis of development of an open-mode fracture at the injector during the Cranfield test. The near-injector region exhibits a large temperature reduction up to 55 °C with ensuing effective horizontal stress reduction up to 9.1 MPa. However, a caprock integrity issue is unlikely because of the horizontal stress contrast within layers and low hydraulic communication with the injection zone. This work indicates that the injectant temperature should be considered in the design of high-rate CO 2 injector.

58 GEOSCIENCES↗
DOE OSTI2020

Pressure falloff testing to characterize CO 2 plume and dry-out zone during CO 2 injection in saline aquifers

The fluid mobilities and the lateral extent of the CO 2 plume and the dry-out zone are among important unknowns required to effectively manage CO 2 storage in saline aquifers projects. These unknowns can be derived from pressure falloff testing which has been widely used to characterize the subsurface. This paper provides a three-region analytical solution for falloff pressure during the infinite-acting period of CO 2 injection in saline aquifers. The solution is derived using Laplace transformation technique of the governing diffusivity equations and the accompanying initial and boundary conditions. The exact solution is validated against numerical results obtained from a black-oil reservoir simulator for two cases. The solution is approximated into simple forms in real-time domain considering different time periods. Then, a graphical semi-log analysis technique based on the approximate solutions is provided as an interpretation tool of falloff tests where the fluid mobilities and the extents of dry-out zone and CO 2 plume can be inferred. The interpretation technique is applied to synthetic falloff test data to investigate its potential and reliability.

02 PETROLEUM↗