Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Analyst”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Assessment of HRA method predictions against operating crew performance: Part I: Study background, design and methodology

This is the first in a series of four papers document ing two large -scale human reliability analysis (HRA) empirical studies – the International HRA Empirical Study and the US HRA Empirical Study. The two studies are the first major efforts in recent years to benchmark HRA methods by comparing HRA method predictions against actual operator performance in accident scenarios simulated on nuclear power plant (NPP) simulators. When a met hod was applied by different analyst teams, the predictions of the different teams were also compared against each other. The purpose of the comparisons was to obtain insights concerning the strengths and weaknesses of the HRA methods addressed in the studies as well as the factors contributing to inter-analyst (or intra-method) variability. The background and motivation for the studies is first presented in the article. Then, the overall study design is described with a discussion on high-level tasks and participants. Next, the methodology to achieve the high-level tasks is discussed. Finally, concluding remarks are made concerning lessons learned on benchmarking HRA methods with crew performance of scenarios on NPP simulators.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Assessment of HRA method predictions against operating crew performance: Part II: Overall simulator data, HRA method predictions, and intra-method comparisons

This is the third in a series of four papers documenting two large-scale human reliability analysis (HRA) empirical studies – the International HRA Empirical Study and the US HRA Empirical Study. Here, the goal of the two studies was to develop an empirically-based understanding of the performance, strengths, and weaknesses of HRA methods by comparing HRA method predictions against actual operator performance in simulated accident scenarios on nuclear power plant (NPP) simulators. However, since in most cases only a single HRA team applied a given method in the International study, it was often difficult to separate analyst effects from variability in results related to the methods themselves. Since at least two HRA teams used each of the HRA methods in the US Study, intra-method comparisons were performed to identify method strengths and weaknesses independent of analyst specific effects where possible. This paper first summarizes the intra-method comparison results from the U.S. Study. Then, it discusses the reasons for the observed HRA predictive differences and the underlying methodological and guidance limitations that permitted the differences to arise. In the fourth paper, this information is combined wit h the results of the comparisons of method predictions to the actual crew data, from both the International and U.S. Studies, to develop the final conclusions about overall strengths and weaknesses of HRA methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Toward adaptive decision support for assessing infrastructure system resilience using hidden performance measures

The understanding of resilience is an emerging topic within the study of risks affecting distributed transportation infrastructure systems. Although recent studies have explored the quantification of system resilience, there has been limited research aimed at understanding the role of multiple performance measures, geographies, and modeling uncertainties within the assessment of resilience and associated decision-making. Under real-world conditions, there is an increased burden on analysts for translating observed system data (from human and electronic sensor observations) into the estimation of system performance that may not be directly observable. This paper addresses these issues using a scenario-based risk modeling approach to understand: 1) resilience of complex systems, often in cases of hidden (not readily observable) measures of performance, 2) resilience sensitivity to modeling uncertainties in event and system characteristics, and 3) resilience sensitivity to the measurement of performance across multiple dimensions. The methods in this paper integrate uncertainty-driven risk and probabilistic modeling within a multi-state Markov approach. This study contributes to the state-of-the-art by developing methodologies for assessing community perceptions of infrastructure system resilience using observable factors and inferring possibly hidden performance measures for facilitating adaptive decision-support. The methods are demonstrated with hypothetical spatiotemporal data for the Washington, District of Columbia and Baltimore, Maryland urban regions in the United States against evacuation-causing and shelter-in-place disruptive scenarios across multiple system performance dimensions. The analysis results may be useful for risk managers, supply chain analysts, emergency decision-makers, and the general public.

Infrastructure System Resilience, Risk Management,↗
DOE OSTI2026

A New Shutdown Dose Rate Benchmark Problem for Representative Fusion Applications

Here, this work introduces a new benchmark problem for calculating shutdown dose rates (SDDRs) aimed at fusion reactor applications. The model is designed to represent a simplified version of a typical ITER port plug. The responses of interest include neutron flux, gamma flux, and gamma SDDR at 12 different locations scattered throughout the port. This article outlines the geometry specifications of the problem, provides material definitions for the components, specifies the required responses to be calculated, and presents the source definition information. The need for this benchmark arises from the limited availability of publicly accessible references, with only one benchmark representing the typical dimensions and materials found in fusion systems. This existing benchmark has been cited extensively, reflecting the demand within the scientific community to test both established and novel workflows for SDDR calculations. However, since its presentation at a conference in 2011, the results have become increasingly well known. Moreover, the absence of formal publication and peer review has led to the details of this benchmark being extracted from secondary sources, such as subsequent studies that reference it. As a result, analysts are left with significant flexibility in interpreting the key parameters, which can be adjusted to account for unknown systematic errors, ultimately reproducing the already well-known responses. This new benchmark serves as an updated version of that earlier work, with the aim of providing a more reliable description of the materials and their impurities, which is crucial for assessing activation and subsequent gamma emission. Additionally, it seeks to provide a geometry that more closely represents an ITER port plug. The improvements in the problem definition will lead to a more reproducible benchmark problem, while also presenting the radiation transport community with a completely new challenge. The results will be published in a future article to allow analysts adequate time to analyze this problem independently.

Benchmark↗
DOE OSTI2018

Uncertainty quantification in the analyses of operational wind power plant performance

In the present work, we examine the variation introduced in the evaluation of an operating plant's wind power production as a result of the choices analysts make in the processing of the operational data. For this study, an idealized power production for individual turbines over an operational period was predicted by fitting power curves to the turbine production data collected during expected operation (that is, without curtailment or availability losses). A set of 240 possible methods were developed for (a) defining what data represented expected operation and (b) modeling the power curve. The spread in the idealized power production as predicted by the different methods was on average almost 3% for the 100 turbines considered. Such significant variation places a lower bound on the precision with which analysts may employ such data as benchmarks for calibration of their energy estimation processes and limits the potential for identification of refinements to the energy estimation models for improved accuracy.

17 WIND ENERGY↗
DOE OSTI2023

Evaluating Awkward Arrays, uproot, and coffea as a query platform for High Energy Physics Data

Query languages for High Energy Physics (HEP) are an ever present topic within the field. A query language that can efficiently represent the nested data structures that encode the statistical and physical meaning of HEP data will help analysts by ensuring their code is more clear and pertinent. As the result of a multi-year effort to develop an in-memory columnar representation of high energy physics data, the NumPy, Awkward Array, and uproot Python packages present a mature and efficient interface to HEP data. Atop that base, the coffea package adds functionality to launch queries at scale, manage and apply experiment-specific transformations to data, and present a rich object-oriented columnar data representation to the analyst. Recently, a set of Analysis Description Language (ADL) benchmarks has been established to compare HEP queries in multiple languages and frameworks. In this paper we present these benchmark queries implemented within the coffea framework and discuss their readability and performance characteristics. We find that the columnar queries perform as well or better than the implementations given in previous studies.

Gray, L.↗
DOE OSTI2017

Simulation Exploration through Immersive Parallel Planes

We present a visualization-driven simulation system that tightly couples systems dynamics simulations with an immersive virtual environment to allow analysts to rapidly develop and test hypotheses in a high-dimensional parameter space. To accomplish this, we generalize the two-dimensional parallel-coordinates statistical graphic as an immersive 'parallel-planes' visualization for multivariate time series emitted by simulations running in parallel with the visualization. In contrast to traditional parallel coordinate's mapping the multivariate dimensions onto coordinate axes represented by a series of parallel lines, we map pairs of the multivariate dimensions onto a series of parallel rectangles. As in the case of parallel coordinates, each individual observation in the dataset is mapped to a polyline whose vertices coincide with its coordinate values. Regions of the rectangles can be 'brushed' to highlight and select observations of interest: a 'slider' control allows the user to filter the observations by their time coordinate. In an immersive virtual environment, users interact with the parallel planes using a joystick that can select regions on the planes, manipulate selection, and filter time. The brushing and selection actions are used to both explore existing data as well as to launch additional simulations corresponding to the visually selected portions of the input parameter space. As soon as the new simulations complete, their resulting observations are displayed in the virtual environment. This tight feedback loop between simulation and immersive analytics accelerates users' realization of insights about the simulation and its output.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2011

Atypical Behavior Identification in Large Scale Network Traffic

Cyber analysts are faced with the daunting challenge of identifying exploits and threats within potentially billions of daily records of network traffic. Enterprise-wide cyber traffic involves hundreds of millions of distinct IP addresses and results in data sets ranging from terabytes to petabytes of raw data. Creating behavioral models and identifying trends based on those models requires data intensive architectures and techniques that can scale as data volume increases. Analysts need scalable visualization methods that foster interactive exploration of data and enable identification of behavioral anomalies. Developers must carefully consider application design, storage, processing, and display to provide usability and interactivity with large-scale data. We present an application that highlights atypical behavior in enterprise network flow records. This is accomplished by utilizing data intensive architectures to store the data, aggregation techniques to optimize data access, statistical techniques to characterize behavior, and a visual analytic environment to render the behavioral trends, highlight atypical activity, and allow for exploration.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2013

Trelliscope: A System for Detailed Visualization in Analysis of Large Complex Data

Visualization plays a critical role in the statistical model building and data analysis process. Data analysts, well-versed in statistical and machine learning methods, visualize data to hypothesize and validate models. These analysts need flexible, scalable visualization tools that are not decoupled from their analysis environment. In this paper we introduce Trelliscope, a visualization framework for statistical analysis of large complex data. Trelliscope extends Trellis, an effective visualization framework that divides data into subsets and applies a plotting method to each subset, arranging the results in rows and columns of panels. Trelliscope provides a way to create, arrange and interactively view panels for very large datasets, enabling flexible detailed visualization for data of any size. Scalability is achieved using distributed computing technologies coupled with . We discuss the underlying principles, design, and scalable architecture of Trelliscope, and illustrate its use on three analysis projects in the domains of proteomics, high energy physics, and power systems engineering.

Hafen, Ryan P.↗
DOE OSTI2016

SeekAView: An Intelligent Dimensionality Reduction Strategy for Navigating High-Dimensional Data Spaces

Dealing with the curse of dimensionality is a key challenge in high-dimensional data visualization. We present SeekAView to address three main gaps in the existing research literature. First, automated methods like dimensionality reduction or clustering suffer from a lack of transparency in letting analysts interact with their outputs in real-time to suit their exploration strategies. The results often suffer from a lack of interpretability, especially for domain experts not trained in statistics and machine learning. Second, exploratory visualization techniques like scatter plots or parallel coordinates suffer from a lack of visual scalability: it is difficult to present a coherent overview of interesting combinations of dimensions. Third, the existing techniques do not provide a flexible workflow that allows for multiple perspectives into the analysis process by automatically detecting and suggesting potentially interesting subspaces. In SeekAView we address these issues using suggestion based visual exploration of interesting patterns for building and refining multidimensional subspaces. Compared to the state-of-the-art in subspace search and visualization methods, we achieve higher transparency in showing not only the results of the algorithms, but also interesting dimensions calibrated against different metrics. We integrate a visually scalable design space with an iterative workflow guiding the analysts by choosing the starting points and letting them slice and dice through the data to find interesting subspaces and detect correlations, clusters, and outliers. We present two usage scenarios for demonstrating how SeekAView can be applied in real-world data analysis scenarios.

Krause, Josua↗
DOE OSTI2017

Rethinking Visual Analytics for Streaming Data Applications

In the age of data science, the use of interactive information visualization techniques has become increasingly ubiquitous. From online scientific journals to the New York Times graphics desk, the utility of interactive visualization for both storytelling and analysis has become ever more apparent. As these techniques have become more readily accessible, the appeal of combining interactive visualization with computational analysis continues to grow. Arising out of a need for scalable, human-driven analysis, primary objective of visual analytics systems is to capitalize on the complementary strengths of human and machine analysis, using interactive visualization as a medium for communication between the two. These systems leverage developments from the fields of information visualization, computer graphics, machine learning, and human-computer interaction to support insight generation in areas where purely computational analyses fall short. Over the past decade, visual analytics systems have generated remarkable advances in many historically challenging analytical contexts. These include areas such as modeling political systems [Crouser et al. 2012], detecting financial fraud [Chang et al. 2008], and cybersecurity [Harrison et al. 2012]. In each of these contexts, domain expertise and human intuition is a necessary component of the analysis. This intuition is essential to building trust in the analytical products, as well as supporting the translation of evidence into actionable insight. In addition, each of these examples also highlights the need for scalable analysis. In each case, it is infeasible for a human analyst to manually assess the raw information unaided, and the communication overhead to divide the task between a large number of analysts makes simple parallelism intractable. Regardless of the domain, visual analytics tools strive to optimize the allocation of human analytical resources, and to streamline the sensemaking process on data that is massive, complex, incomplete, and uncertain in scenarios requiring human judgment.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Characterizing cyber tools for monitoring power grid systems: what information is available and who needs it?

Abstract—The power grid is a complex, interconnected system that requires the coordination of multiple entities ensure the reliability of the system. System operators must be aware of the impacts of various natural and man-made contingencies through telemetry from their control systems, and they must also be aware of the integrity and availability of the cyber assets on their systems. A team of researchers from PNNL and INL have conducted a series of observations and interviews at power utilities to identify what information needs to be available for both control room dispatchers and cyber security analysts to maintain situation awareness of cyber security to effectively monitor the cyber assets on utility control systems. The researchers interviewed and observed control room dispatchers and spoke with regional control system engineers and cyber security analysts to define the context in which such information would be used. The results of this work will inform the design of a tool that will enhance cyber situation awareness and help facilitate effective communication between the various roles and organizations with utility and other regional authorities during contingencies. Index Terms—

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

A multi-modal scanning system to digitize CBRNE emergency response scenes

A handheld system developed to digitize a contextual understanding of the scene at a chemical, biological, radiological, nuclear and/or explosives (CBRNE) events is described. The system uses LiDAR and cameras to create a colorized 3D model of the environment, which helps domain experts that are supporting responders in the field. To generate the digitized model, a responder scans any suspicious objects and the surroundings by carrying the system through the scene. The scanning system provides a real-time user interface to inform the user about scanning progress and to indicate any areas that may have been missed either by the LiDAR sensors or the cameras. Currently, the collected data are post-processed on a different device, building a colorized triangular mesh of the encountered scene, with the intention of moving this pipeline to the scanner at a later point. The mesh is sufficiently compressed to be sent over a reduced bandwidth connection to a remote analyst. Furthermore, the system tracks fiducial markers attached to diagnostic equipment that is placed around the suspicious object. The resulting tracking information can be transmitted to remote analysts to further facilitate their supporting efforts. The paper will discuss the system's design, software components, the user interface used for scanning a scene, the necessary procedures for calibration of the sensors, and the processing steps of the resulting data. The discussion will close by evaluating the system's performance on 11 scenes.

Salathe, Marco↗
DOE OSTI2021

Beyond Visual Analytics: Human-Machine Teaming for AI-Driven Data Sensemaking

"Detect the expected, discover the unexpected" was the founding principle of the field of visual analytics. This mantra implies that human stakeholders, like a domain expert or data analyst, could leverage visual analytics techniques to seek answers to known unknowns and discover unknown unknowns in the course of the data sensemaking process. We argue that in the era of AI-driven automation, we need to recalibrate the roles of humans and machines (e.g., a machine learning model) as teammates. We posit that by realizing human-machine teams as a stakeholder unit, we can better achieve the best of both worlds: automation transparency and human reasoning efficacy. However, this also increases the burden on analysts and domain experts towards performing more cognitively demanding tasks than what they are used to. In this paper, we reflect on the complementary roles in a human-machine team through the lens of cognitive psychology and map them to existing and emerging research in the visual analytics community. We discuss open questions and challenges around the nature of human agency and analyze the shared responsibilities in human-machine teams.

Sensemaking, human-machine teaming, agency, artifi↗
DOE OSTI2018

Situ: Identifying and Explaining Suspicious Behavior in Networks

Despite the best efforts of cyber security analysts, networked computing assets are routinely compromised, resulting in the loss of intellectual property, the disclosure of state secrets, and major financial damages. Anomaly detection methods are beneficial for detecting new types of attacks and abnormal network activity, but such algorithms can be difficult to understand and trust. Network operators and cyber analysts need fast and scalable tools to help identify suspicious behavior that bypasses automated security systems, but operators do not want another automated tool with algorithms they do not trust. Experts need tools to augment their own domain expertise and to provide a contextual understanding of suspicious behavior to help them make decisions. In this paper we present Situ, a visual analytics system for discovering suspicious behavior in streaming network data. Situ provides a scalable solution that combines anomaly detection with information visualization. The system's visualizations enable operators to identify and investigate the most anomalous events and IP addresses, and the tool provides context to help operators understand why they are anomalous. Finally, operators need tools that can be integrated into their workflow and with their existing tools. This paper describes the Situ platform and its deployment in an operational network setting. Furthermore, we discuss how operators are currently using the tool in a large organization's security operations center and present the results of expert reviews with professionals.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

The Influence of Visual Provenance Representations on Strategies in a Collaborative Hand-off Data Analysis Scenario

Conducting data analysis tasks rarely occur in isolation. Especially in intelligence analysis scenarios where different experts contribute knowledge to a shared understanding, members must communicate how insights develop to establish common ground among collaborators. The use of provenance to communicate analytic sensemaking carries promise by describing the interactions and summarizing the steps taken to reach insights. Yet, no universal guidelines exist for communicating provenance in different settings. Our work here focuses on the presentation of provenance information and the resulting conclusions reached and strategies used by new analysts. In an open-ended, 30-minute, textual exploration scenario, we qualitatively compare how adding different types of provenance information (specifically data coverage and interaction history) affects analysts' confidence in conclusions developed, propensity to repeat work, filtering of data, identification of relevant information, and typical investigation strategies. We see that data coverage (i.e., what was interacted with) provides provenance information without limiting individual investigation freedom. On the other hand, while interaction history (i.e., when something was interacted with) does not significantly encourage more mimicry, it does take more time to comfortably understand, as represented by less confident conclusions and less relevant information-gathering behaviors. In conclusion, our results contribute empirical data towards understanding how provenance summarizations can influence analysis behaviors.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Trust Your Gut: Comparing Human and Machine Inference from Noisy Visualizations

People commonly utilize visualizations not only to examine a given dataset, but also to draw generalizable conclusions about the underlying models or phenomena. Prior research has compared human visual inference to that of an optimal Bayesian agent, with deviations from rational analysis viewed as problematic. However, human reliance on non-normative heuristics may prove advantageous in certain circumstances. We investigate scenarios where human intuition might surpass idealized statistical rationality. In two experiments, we examine individuals’ accuracy in characterizing the parameters of known data-generating models from bivariate visualizations. Our findings indicate that, although participants generally exhibited lower accuracy compared to statistical models, they frequently outperformed Bayesian agents, particularly when faced with extreme samples. Participants appeared to rely on their internal models to filter out noisy visualizations, thus improving their resilience against spurious data. However, participants displayed overconfidence and struggled with uncertainty estimation. They also exhibited higher variance than statistical machines. Our findings suggest that analyst gut reactions to visualizations may provide an advantage, even when departing from rationality. These results carry implications for designing visual analytics tools, offering new perspectives on how to integrate statistical models and analyst intuition for improved inference and decision-making. The data and materials for this paper are available at https://osf.io/qmfv6

human-machine collaboration↗