Engineering Papers⌕ Search

Engineering topics

Scholtz, Jean

Publications and source records attributed to Scholtz, Jean.

How Do Visual Explanations Foster End Users' Appropriate Trust in Machine Learning?

We investigated the effects of different visual explanations on users' trust in machine learning classification. We proposed three forms of visual explanations of a classification based on identifying relevant training instances. We conducted a user study to evaluate these visual explanations as well as a no explanation condition. We measured users' trust of a classifier, quantified the effects of these three forms of explanations, and assessed the changes in users' trust. We found that participants trust a classifier appropriately when an explanation is available. The combination of human, classification algorithm and understandable explanation makes better decisions than the classifier and human alone.This work advances the state-of-the-art closer to building trust-able machine learning models and informs the design and appropriate use of automated systems.

Yang, Fumeng↗

Characterizing cyber tools for monitoring power grid systems: what information is available and who needs it?

Abstract—The power grid is a complex, interconnected system that requires the coordination of multiple entities ensure the reliability of the system. System operators must be aware of the impacts of various natural and man-made contingencies through telemetry from their control systems, and they must also be aware of the integrity and availability of the cyber assets on their systems. A team of researchers from PNNL and INL have conducted a series of observations and interviews at power utilities to identify what information needs to be available for both control room dispatchers and cyber security analysts to maintain situation awareness of cyber security to effectively monitor the cyber assets on utility control systems. The researchers interviewed and observed control room dispatchers and spoke with regional control system engineers and cyber security analysts to define the context in which such information would be used. The results of this work will inform the design of a tool that will enhance cyber situation awareness and help facilitate effective communication between the various roles and organizations with utility and other regional authorities during contingencies. Index Terms—

24 POWER TRANSMISSION AND DISTRIBUTION↗

Cybersecurity Awareness in the Power Grid

We report on a series of interviews and observations conducted with control room dispatchers in a bulk electrical system. These dispatchers must react quickly to incidents as they happen in order to ensure the reliability and safe operation of the power grid. They do not have the time to evaluate incidents for signs of cyber-attack as part of their initial response. Cyber-attack detection involves multiple personnel from a variety of roles at both local and regional levels. Smart grid technology will improve detection and defense capabilities of the future grid, however, the current infrastructure remains a mixture of old and new equipment which will continue to operate for some time. Thus, research still needs to focus on strategies for the detection of malicious activity on current infrastructure as well as protection and remediation.

Scholtz, Jean↗

Visual Analytics 101

This course will introduce the field of Visual Analytics to HCI researchers and practitioners highlighting the contributions they can make to this field. Topics will include a definition of visual analytics along with examples of current systems, types of tasks and end users, issues in defining user requirements, design of visualizations and interactions, guidelines and heuristics, the current state of user-centered evaluations, and metrics for evaluation. We encourage designers, HCI researchers, and HCI practitioners to attend to learn how their skills can contribute to advancing the state of the art of visual analytics

design of visualizations and interactions↗

Human Research Program Space Human Factors Engineering (SHFE) Standing Review Panel (SRP)

The Space Human Factors Engineering (SHFE) Standing Review Panel (SRP) evaluated 22 gaps and 39 tasks in the three risk areas assigned to the SHFE Project. The area where tasks were best designed to close the gaps and the fewest gaps were left out was the Risk of Reduced Safety and Efficiency dire to Inadequate Design of Vehicle, Environment, Tools or Equipment. The areas where there were more issues with gaps and tasks, including poor or inadequate fit of tasks to gaps and missing gaps, were Risk of Errors due to Poor Task Design and Risk of Error due to Inadequate Information. One risk, the Risk of Errors due to Inappropriate Levels of Trust in Automation, should be added. If astronauts trust automation too much in areas where it should not be trusted, but rather tempered with human judgment and decision making, they will incur errors. Conversely, if they do not trust automation when it should be trusted, as in cases where it can sense aspects of the environment such as radiation levels or distances in space, they will also incur errors. This will be a larger risk when astronauts are less able to rely on human mission control experts and are out of touch, far away, and on their own. The SRP also identified 11 new gaps and five new tasks. Although the SRP had an extremely large quantity of reading material prior to and during the meeting, we still did not feel we had an overview of the activities and tasks the astronauts would be performing in exploration missions. Without a detailed task analysis and taxonomy of activities the humans would be engaged in, we felt it was impossible to know whether the gaps and tasks were really sufficient to insure human safety, performance, and comfort in the exploration missions. The SRP had difficulty evaluating many of the gaps and tasks that were not as quantitative as those related to concrete physical danger such as excessive noise and vibration. Often the research tasks for cognitive risks that accompany poor task or information design addressed only part, but not all, of the gaps they were programmed to fill. In fact the tasks outlined will not close the gap but only scratch the surface in many cases. In other cases, the gap was written too broadly, and really should be restated in a more constrained way that can be addressed by a well-organized and complementary set of tasks. In many cases, the research results should be turned into guidelines for design. However, it was not clear whether the researchers or another group would construct and deliver these guidelines.

Wichansky, Anna↗

Common Metrics for Human-Robot Interaction

This paper describes an effort to identify common metrics for task-oriented human-robot interaction (HRI). We begin by discussing the need for a toolkit of HRI metrics. We then describe the framework of our work and identify important biasing factors that must be taken into consideration. Finally, we present suggested common metrics for standardization and a case study. Preparation of a larger, more detailed toolkit is in progress.

Steinfeld, Aaron↗