Usability Evaluation of Cloud for HPC Applications
Explore the source record for details and available documents.
Sochat, V. [Lawrence Livermore National Laboratory↗
SEARCH · Engineering Papers
Results for “Usability”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
MALAMUTE Usability Improvements to Expand User Base
Proprietary “advanced materials analysis” tools are expensive (i.e COMSOL, ANSYS) MALAMUTE: open-source, modular, & equipped with multiphysics capabilities!
36 - MATERIALS SCIENCE↗
Characterizing the Performance and Usability of GPU JIT Compilation Interfaces using Proteus
Explore the source record for details and available documents.
Computer science↗
Usability/Sentiment for the Enterprise and ENTERPRISE
The purpose of the Sentiment of Search Study for NASA Johnson Space Center (JSC) is to gain insight into the intranet search environment. With an initial usability survey, the authors were able to determine a usability score based on the Systems Usability Scale (SUS). Created in 1986, the freely available, well cited, SUS is commonly used to determine user perceptions of a system (in this case the intranet search environment). As with any improvement initiative, one must first examine and document the current reality of the situation. In this scenario, a method was needed to determine the usability of a search interface in addition to the user's perception on how well the search system was providing results. The use of the SUS provided a mechanism to quickly ascertain information in both areas, by adding one additional open-ended question at the end. The first ten questions allowed us to examine the usability of the system, while the last questions informed us on how the users rated the performance of the search results. The final analysis provides us with a better understanding of the current situation and areas to focus on for improvement. The power of search applications to enhance knowledge transfer is indisputable. The performance impact for any user unable to find needed information undermines project lifecycle, resource and scheduling requirements. Ever-increasing complexity of content and the user interface make usability considerations for the intranet, especially for search, a necessity instead of a 'nice-to-have'. Despite these arguments, intranet usability is largely disregarded due to lack of attention beyond the functionality of the infrastructure (White, 2013). The data collected from users of the JSC search system revealed their overall sentiment by means of the widely-known System Usability Scale. Results of the scores suggest 75%, +/-0.04, of the population rank the search system below average. In terms of a grading scaled, this equated to D or lower. It is obvious JSC users are not satisfied with the current situation, however they are eager to provide information and assistance in improving the search system. A majority of the respondents provided feedback on the issues most troubling them. This information will be used to enrich the next phase, root cause analysis and solution creation.
Meza, David↗
Apex Reference Manual 3.0 Beta
Apex is a toolkit for constructing software that behaves intelligently and responsively in demanding task environments. Reflecting its origin at NASA where Apex continues to be developed, current applications include: a) Providing autonomous mission management and tactical control capabilities for unmanned aerial vehicles including an autonomous surveillance helicopter and a simulation prototype of an unmanned fixed-wing aircraft to be used for wildfire mapping; b) Simulating human air traffic controllers, pilots and astronauts to help predict how people might respond to changes in equipment or procedures; and c) Predicting the precise duration and sequence of routine human behaviors based on a human-computer interaction engineering technique called CPM-GOMS. Among Apex s components are a set of implemented reasoning services, such as those for reactive planning and temporal pattern recognition; a software architecture that embeds and integrates these services and allows additional reasoning elements to be added as extensions; a formal language for specifying agent knowledge; a simulation environment to facilitate prototyping and analysis; and Sherpa, a set of tools for visualizing autonomy logic and runtime behavior. In combination, these are meant to provide a flexible and usable framework for creating, testing, and deploying intelligent agent software. Overall, our goal in developing Apex is to lower economic barriers to developing intelligent software agents. New ideas about how to extend or modify the system are evaluated in terms of their impact in reducing the time, expertise, and inventiveness required to build and maintain applications. For example, potential enhancements to the AI reasoning capabilities in the system are reviewed not only for usefulness and distinctiveness, but also for their impact on the readability and general usability of Apex s behavior representation language (PDL) and on the transparency of resulting behavior. A second central part of our approach is to iteratively refine Apex based on lessons learned from as diverse a set of applications as possible. Many applications have been developed by users outside the core development team including engineers, researchers, and students. Usability is thus a central concern for every aspect of Apex visible to a user, including PDL, Sherpa, the Apex installation process, APIs, and user documentation. Apex users vary in their areas of expertise and in their familiarity with autonomy technology. Focusing on usability, a development philosophy summarized by the project motto "Usable Autonomy," has been important part of enabling diverse users to employ Apex successfully and to provide feedback needed to guide iterative, user-centered refinement.
Freed, Michael A.↗
Record High Hydrogen Storage Capacity in the Metal–Organic Framework Ni 2 ( m -dobdc) at Near-Ambient Temperatures
Hydrogen holds promise as a clean alternative automobile fuel, but its on-board storage presents significant challenges due to the low temperatures and/or high pressures required to achieve a sufficient energy density. The opportunity to significantly reduce the required pressure for high density H 2 storage persists for metal-organic frameworks due to their modular structures and large internal surface areas. The measurement of H 2 adsorption in such materials under conditions most relevant to on-board storage is crucial to understanding how these materials would perform in actual applications, although such data have to date been lacking. In the present work, the metal-organic frameworks M 2 (m-dobdc) (M = Co, Ni; m-dobdc 4- = 4,6-dioxido-1,3-benzenedicarboxylate) and the isomeric frameworks M 2 (dobdc) (M = Co, Ni; dobdc 4- = 1,4-dioxido-1,3-benzenedicarboxylate), which are known to have open metal cation sites that strongly interact with H 2 , were evaluated for their usable volumetric H 2 storage capacities over a range of near-ambient temperatures relevant to on-board storage. Based upon adsorption isotherm data, Ni 2 (m-dobdc) was found to be the top-performing physisorptive storage material with a usable volumetric capacity between 100 and 5 bar of 11.0 g/L at 25 °C and 23.0 g/L with a temperature swing between -75 and 25 °C. Additional neutron diffraction and infrared spectroscopy experiments performed with in situ dosing of D 2 or H 2 were used to probe the hydrogen storage properties of these materials under the relevant conditions. Finally, the results provide benchmark characteristics for comparison with future attempts to achieve improved adsorbents for mobile hydrogen storage applications.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Evaluating a Web-Based Interface for Internet Telemedicine
The objective is to introduce the usability engineering methodology, heuristic evaluation, to the design and development of a web-based telemedicine system. Using a set of usability criteria, or heuristics, one evaluator examined the Spacebridge to Russia web-site for usability problems. Thirty-four usability problems were found in this preliminary study and all were assigned a severity rating. The value of heuristic analysis in the iterative design of a system is shown because the problems can be fixed before deployment of a system and the problems are of a different nature than those found by actual users of the system. It was therefore determined that there is potential value of heuristic evaluation paired with user testing as a strategy for optimal system performance design.
Lathan, Corinna E.↗
Logistical Consideration in Computer-Based Screening of Astronaut Applicants
This presentation reviews the logistical, ergonomic, and psychometric issues and data related to the development and operational use of a computer-based system for the psychological screening of astronaut applicants. The Behavioral Health and Performance Group (BHPG) at the Johnson Space Center upgraded its astronaut psychological screening and selection procedures for the 1999 astronaut applicants and subsequent astronaut selection cycles. The questionnaires, tests, and inventories were upgraded from a paper-and-pencil system to a computer-based system. Members of the BHPG and a computer programmer designed and developed needed interfaces (screens, buttons, etc.) and programs for the astronaut psychological assessment system. This intranet-based system included the user-friendly computer-based administration of tests, test scoring, generation of reports, the integration of test administration and test output to a single system, and a complete database for past, present, and future selection data. Upon completion of the system development phase, four beta and usability tests were conducted with the newly developed system. The first three tests included 1 to 3 participants each. The final system test was conducted with 23 participants tested simultaneously. Usability and ergonomic data were collected from the system (beta) test participants and from 1999 astronaut applicants who volunteered the information in exchange for anonymity. Beta and usability test data were analyzed to examine operational, ergonomic, programming, test administration and scoring issues related to computer-based testing. Results showed a preference for computer-based testing over paper-and -pencil procedures. The data also reflected specific ergonomic, usability, psychometric, and logistical concerns that should be taken into account in future selection cycles. Conclusion. Psychological, psychometric, human and logistical factors must be examined and considered carefully when developing and using a computer-based system for psychological screening and selection.
Galarza, Laura↗
Integrating Ground System Tools From Multiple Technologies Into a Single System Environment
With rapid technology changes and new and improved development techniques, it becomes extremely difficult to try to add capabilities to existing ground systems without wanting to replace the entire system. Replacing entire systems is not usually cost effective so there is a need to be able to slowly improve systems without long development times that introduce risk due to large amounts of change. The Marshall Space Flight Center s (MSFC) Payload Operations Integration Center (POIC) ground system provides command, telemetry, and payload planning systems in support of the International Space Station. Our systems have continuously evolved with technology changes due to hardware end of life issues, and also due to user requirement changes. As changes have been implemented, we have tried to take advantage of some of the latest technologies while at the same time maintaining certain legacy capabilities that are not cost affective to replace. One of our biggest challenges is to integrate all of these implementations into a single system that is usable, maintainable, and scalable. Another challenge is to provide access to our tools in such a way that users are not aware of all the various implementation methods and tools being used. This approach not only makes our system much more usable, it allows us to continue to migrate capabilities and to add capabilities without impacting system usability. This paper will give an overview of the tools used for MSFC ISS payload operations and show an approach for integrating various technologies into a single environment that is maintainable, flexible, usable, cost effective, and that meets user needs.
Ritter, George H.↗
Integration Of Launch Vehicle Simulation/Analysis Tools And Lunar Cargo Lander Design
Part 2, which will be discussed in this report, will discuss the development of a Lunar Cargo Lander (unmanned launch vehicle) that will transport usable payload from Trans- Lunar Injection to the moon. The Delta IV-Heavy was originally used to transport the Lunar Cargo Lander to TLI, but other launch vehicles have been studied. In order to uncover how much payload is possible to land on the moon, research was needed in order to design the sub-systems of the spacecraft. The report will discuss and compare the use of a hypergolic and cryogenic system for its main propulsion system. The guidance, navigation, control, telecommunications, thermal, propulsion, structure, mechanisms, landing gear, command, data handling, and electrical power sub-systems were designed by scaling off other flown orbiters and moon landers. Once all data was collected, an excel spreadsheet was created to accurately calculate the usable payload that will land on the moon along with detailed mass and volume estimating relations. As designed, The Lunar Cargo Lander can plant 5,400 lbm of usable payload on the moon using a hypergolic system and 7,400 lbm of usable payload on the moon using a cryogenic system.
DeJean, George Brian↗
Reinventing Image Detective: An Evidence-Based Approach to Citizen Science Online
Usability studies demonstrate that web users are notoriously impatient, spending as little as 15 seconds on a home page. How do you get users to stay long enough to understand a citizen science project? How do you get users to complete complex citizen science tasks online? Image Detective, a citizen science project originally developed by scientists and science engagement specialists at the NASA Johnson Space center to engage the public in the analysis of images taken from space by astronauts to help enhance NASA's online database of astronaut imagery, partnered with the CosmoQuest citizen science platform to modernize, offering new and improved options for participation in Image Detective. The challenge: to create a web interface that builds users' skills and knowledge, creating engagement while learning complex concepts essential to the accurate completion of tasks. The project team turned to usability testing for an objective understanding of how users perceived Image Detective and the steps required to complete required tasks. A group of six users was recruited online for unmoderated and initial testing. The users followed a think-aloud protocol while attempting tasks, and were recorded on video and audio. The usability test examined users' perception of four broad areas: the purpose of and context for Image Detective; the steps required to successfully complete the analysis (differentiating images of Earth's surface from those showing outer space and identifying common surface features); locating the image center point on a map of Earth; and finally, naming geographic locations or natural events seen in the image. Usability test findings demonstrated that the following best practices can increase participation in Image Detective and can be applied to the successful implementation of any citizen science project: (1) Concise explanation of the project, its context, and its purpose; (2) Including a mention of the funding agency (in this case, NASA); (3) A preview of the specific tasks required of participants; (4) A dedicated user interface for the actual citizen science interaction. In addition, testing revealed that users may require additional context when a task is complex, difficult, or unusual (locating a specific image and its center point on a map of Earth). Video evidence will be made available with this presentation.
Romano, Cia↗
Knowledge Discovery for Early Failure Assessment of Complex Engineered Systems Using Natural Language Processing
Emerging complex engineered systems may have unexpected safety issues due to novel operational environments, increasing autonomy, human-machine interaction, and other factors. To prevent failures in operation or testing that necessitate costly redesign, it is desirable to predict likely failure modes early in the design process. Information about past engineering failures in natural language format presents one possible solution by enabling the retrieval of information that can inform new designs. However, identifying documents containing usable information and extracting the required information can be prohibitively time-consuming when implemented at scale. In this research, an automated natural language processing (NLP) framework is proposed to discover relevant knowledge from documents containing failure-related design information. The framework is applied to NASA’s Lessons Learned Information System (LLIS),which is publicly available. Documents containing usable information are filtered using two different NLP-based models. Next, from the identified usable documents, a failure taxonomy is extracted using a partitioned hierarchical topic modeling approach. Partitions of the document describe different sections of the failure taxonomy – i.e., failure, cause of failure, and recommendations – as indicated by the structure of the original document. The extracted failure taxonomy can be leveraged in early design failure assessment methods. Moreover, the framework can be used to identify documents containing usable failure-related design information from other databases and extract relevant information from these documents.
Documentation and Information Science↗
Knowledge Discovery for Early Failure Assessment of Complex Engineered Systems Using Natural Language Processing
Emerging complex engineered systems may have unexpected safety issues due to novel operational environments, increasing autonomy, human-machine interaction, and other factors. To prevent failures in operation or testing that necessitate costly redesign, it is desirable to predict likely failure modes early in the design process. Information about past engineering failures in natural language format presents one possible solution by enabling the retrieval of information that can inform new designs. However, identifying documents containing usable information and extracting the required information can be prohibitively time-consuming when implemented at scale. In this research, an automated natural language processing (NLP) framework is proposed to discover relevant knowledge from documents containing failure-related design information. The framework is applied to NASA’s Lessons Learned Information System (LLIS),which is publicly available. Documents containing usable information are filtered using two different NLP-based models. Next, from the identified usable documents, a failure taxonomy is extracted using a partitioned hierarchical topic modeling approach. Partitions of the document describe different sections of the failure taxonomy – i.e., failure, cause of failure, and recommendations – as indicated by the structure of the original document. The extracted failure taxonomy can be leveraged in early design failure assessment methods. Moreover, the framework can be used to identify documents containing usable failure-related design information from other databases and extract relevant information from these documents.
Documentation and Information Science↗
Systematization of Password Manager Use Cases and Design Paradigms
Despite efforts to replace them, passwords remain the primary form of authentication on the web. Password managers seek to address many of the problems with passwords by helping users generate, store, and fill strong and unique passwords. Even though experts frequently recommend password managers, there is limited information regarding their usability. To aid in designing such usability studies, we systematize password manager use cases, identifying ten essential use cases, three recommended use cases, and four extended use cases. We also systematize the system designs employed to satisfy these use cases, designs that should be examined in usability studies to understand their relative strengths and weaknesses. Finally, we describe observations from 136 cognitive walkthroughs exploring the identified essential use cases in eight popular managers. Ultimately, we expect that this work will serve as the foundation for an explosion of new research into the usability of password managers.
Simmons, James↗
Soft Robotic Glove System for Wearable Haptic Teleoperation - 18373
Managing nuclear waste is a difficult, dangerous, and costly task. Teleoperation of robotic operators is an important functionality for this application, allowing robotic devices to operate in dangerous environments while still leveraging the existing abilities and technical knowledge of human workers. One difficulty in implementing effective teleoperation systems is that human operators rely on more senses beside sight: in particular, touch. The sense of touch could provide significant benefits for teleoperation, preventing environmental mishaps that could cost millions of dollars. To help remedy this problem, we present a soft haptic feedback glove system. The primary functionality of this system is to provide tactile and force feedback on the user's hands. This was achieved using custom cylindrical pneumatic actuators. These actuators consist of plastic sheeting folded into a hollow-walled cylinder which is connected to a pressure source. When pressurized, the cylinder walls expand and the cylinder straightens, with the force being related to the pressure. The glove system has one of these actuators over each of the user's three main fingers (index, middle, and thumb), where pressurizing the actuator applies a reaction force to the finger of the user, mimicking the resistance of a grasped object. The expansion of these actuators also causes them to physically press against the user's fingers, giving them the feeling of contact. To demonstrate the usability of this system, we implemented a virtual environment in V-REP simulating the interactions between a robotic Jaco Arm and a floating ball. This simulation receives hand and figure positions from the haptic glove and used them to drive the state of the simulated Jaco Arm. In response to the interactions between the Jaco and the floating ball, the simulation returns commands to the glove, instructing it to pressurize or depressurize its actuators as appropriate, allowing the user to feel the ball in their hand. We tested users in their ability to prevent the ball from floating away using this system with and without the haptic feedback. Results show that the addition of haptic feedback significantly improved the ability of users to interact precisely with the environment. We also used the finger positions of the haptic glove to control a physical robotic hand, and demonstrated the system's usability for control of real robots, though the lack of sensors on the robot made providing the tactile feedback impractical. However, there is more to grasping and holding an object than just the feeling of contact and the resistance to closing one's hand. The weight of the object will apply forces to the user's other joints as they hold it. To this end, we added a haptic feedback device for the user's wrist, a technique that can be generalized to the user's other joints. This is made up of reverse Pneumatic Artificial Muscles (rPAMs) stretching across the wrist between mounts on the hand and forearm. These rPAM actuators are tubes of silicone wrapped in thread such that they extend when pressurized. This extension can apply torque on the user's wrist, mimicking the feeling of gravity, or allowing the simulation of additional contact forces (such as the user's hand impacting an object). We tested this wrist device independently using a simulated path-following experiment. Users showed a significant improvement in performance when aided by haptic feedback during sinusoidal path experiments, demonstrating the usability of this approach. We then integrated the two systems, further demonstrating the usability of this approach. This project represents the first step towards a more effective robotic teleoperation system, allowing skilled human workers to perform tasks more effectively while remaining out of danger. (authors)
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Improving the User Interface of the DeepLynx Data Warehouse
DeepLynx is an open-source ontology-based data warehouse created by INL to support the creation and life cycle of digital engineering projects, with a particular emphasis on digital twins [1]. Digital twins are systems that represent physical assets and process in a real-time digital environment [1]. Most well-known commercial data warehouses use Graphical User Interfaces (GUIs) for users to interact with their systems [3]. Limited publications have addressed the design of these interfaces and understanding of their target users. The current users and development team acknowledge the need to improve the current UI, not just for aesthetics but to improve functionality and workflow of DeepLynx. Traditional data warehouse users are developers, data scientists and business analysts [2]. DeepLynx users have a vast range of experience using data warehouses, and diverse roles, including engineers, scientists and management positions. Because there is a broader audience of target users for DeepLynx than a typical data warehouse, it is essential that DeepLynx has a useable and intuitive user interface. To achieve this the team performed human-computer interaction methods, including a Heuristic Evaluation of current UI using Neilsen’s Usability Heuristic, create personas based on current users by designing a user survey, data analysis and develop of personas. Followed by a redesign of the UI following using Neilsen’s Usability Heuristic and Norman’s Principles of Interactive Design in industry standard software Figma. Lastly a Heuristic Evaluation of new UI design, using Neilsen’s Usability Heuristic and User testing of redesign UI and have a group of users complete a Thinking Aloud Test of the new UI. Preliminary results of the Heuristic Evaluation of current UI arise issue with Consistency and Standards, Visibility of System Status, Match System and Real World and Recognition Rather than Recall. These issues were addressed in the proposed redesign by applying Neilsen’s Usability Heuristic and Norman’s Principles of Interactive Design. Next steps include formalized list of lessons learned and design implications for future publications.
97 MATHEMATICS AND COMPUTING↗
User-Centered Evaluation of Visual Analytics
Visual analytics systems are becoming very popular. More domains now use interactive visualizations to analyze the ever-increasing amount and heterogeneity of data. More novel visualizations are being developed for more tasks and users. We need to ensure that these systems can be evaluated to determine that they are both useful and usable. A user-centered evaluation for visual analytics needs to be developed for these systems. While many of the typical human-computer interaction (HCI) evaluation methodologies can be applied as is, others will need modification. Additionally, new functionality in visual analytics systems needs new evaluation methodologies. There is a difference between usability evaluations and user-centered evaluations. Usability looks at the efficiency, effectiveness, and user satisfaction of users carrying out tasks with software applications. User-centered evaluation looks more specifically at the utility provided to the users by the software. This is reflected in the evaluations done and in the metrics used. In the visual analytics domain this is very challenging as users are most likely experts in a particular domain, the tasks they do are often not well defined, the software they use needs to support large amounts of different kinds of data, and often the tasks last for months. These difficulties are discussed more in the section on User-centered Evaluation. Our goal is to provide a discussion of user-centered evaluation practices for visual analytics, including existing practices that can be carried out and new methodologies and metrics that need to be developed and agreed upon by the visual analytics community. The material provided here should be of use for both researchers and practitioners in the field of visual analytics. Researchers and practitioners in HCI and interested in visual analytics will find this information useful as well as a discussion on changes that need to be made to current HCI practices to make them more suitable to visual analytics. A history of analysis and analysis techniques and problems is provided as well as an introduction to user-centered evaluation and various evaluation techniques for readers from different disciplines. The understanding of these techniques is imperative if we wish to support analysis in the visual analytics software we develop. Currently the evaluations that are conducted and published for visual analytics software are very informal and consist mainly of comments from users or potential users. Our goal is to help researchers in visual analytics to conduct more formal user-centered evaluations. While these are time-consuming and expensive to carryout, the outcomes of these studies will have a defining impact on the field of visual analytics and help point the direction for future features and visualizations to incorporate. While many researchers view work in user-centered evaluation as a less-than-exciting area to work, the opposite is true. First of all, the goal is user-centered evaluation is to help visual analytics software developers, researchers, and designers improve their solutions and discover creative ways to better accommodate their users. Working with the users is extremely rewarding as well. While we use the term “users” in almost all situations there are a wide variety of users that all need to be accommodated. Moreover, the domains that use visual analytics are varied and expanding. Just understanding the complexities of a number of these domains is exciting. Researchers are trying out different visualizations and interactions as well. And of course, the size and variety of data are expanding rapidly. User-centered evaluation in this context is rapidly changing. There are no standard processes and metrics and thus those of us working on user-centered evaluation must be creative in our work with both the users and with the researchers and developers.
intelligence analysis↗
Evaluation of Visualization Heuristics
Heuristics have been used since the early 1990's in the Human Computer Interaction (HCI) domain, for evaluating user interfaces. Most notably, heuristics developed by Nielsen and Molich (1990) have been used by the HCI community. Forsell and Johansson (2010) used the same methodology as Nielsen and Molich and developed a set of 10 heuristics for information visualization. Currently, neither the information visualization or the visual analytics community has adopted these heuristics. However, several studies have been conducted using heuristics. Hearst et al. (2015) found that using the heuristics for visualizations along with asking questions about the data resulted in complementary results as the questions about the data could compensate for heuristics that were difficult to understand and apply. Heli Väätäjä et al. (2016) found that heuristics related to interaction, veracity, and aesthetics needed to be added to the Forsell and Johansson (2010) set. They also noted that a lack of domain knowledge caused the evaluators issues in conducting a heuristic evaluation. We conducted a controlled experiment to determine what factors might impact the expert evaluators' capability to conduct a heuristic evaluation of a static information visualization. In particular, we wanted to understand how their familiarity with the visualizations and their experience in conducting heuristic evaluations impacted their evaluations of visualizations. We recruited 10 participants who were domain experts in the visualization field each having experience with one of more of the following: user experience design or research, user interface development, and human computer interaction research. We used a heuristic set proposed by Forsell and Johansson. Participants were also asked to complete a demographic survey in which they rated their experience with each visualization type generally, their experience conducting heuristic evaluations, their experience using each visualization type to extract information, and their experience developing and creating each visualization type. The participants were shown a series of 5 common visualizations (Scatterplots, Sunbursts, Tree Maps, Parallel Sets, and Area Graphs) all displaying the VAST 2008 Mini Challenge Two- Migrant Boats data . Participants were given a suggested analytical question related to each visualization to keep in mind as they applied the 10 heuristics to each visualization. They were asked if they could find any usability problems in the visualizations and which heuristic(s) helped them to identify these problems. The participants were asked to rate how relevant each heuristic was for evaluating each visualization along with rating how confident they felt in their ability to apply the heuristic to evaluate each visualization. At the conclusion of the study, participants provided qualitative feedback on the usability issues of each visualization. Additionally, we collected eye tracking data on 6 of the 10 participants. We will be using this data to determine how long participants looked at the various visualizations and what parts of the visualizations they focused on. We had two hypotheses for this empirical evaluation: 1.The Forsell and Johansson heuristics of Orientation and help, Data set reduction, and Prompting are less useful for static visualizations because they require some interaction in order to test completely. 2.Participants that do not have experience with heuristic evaluation will provide lower evaluation scores for the heuristics compared to those that have experience. Related to hypothesis 1, as our study used only static visualizations, interaction-related usability challenges should be harder to identify. This should result in lower usefulness ratings for those heuristics. Preliminary results indicate that participants found it more difficult to apply heuristics to unfamiliar visualizations.
Williams, Ryan G.↗