Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “methodology”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2014

A Simple and Efficient Methodology To Improve Geometric Accuracy in Gamma Knife Radiation Surgery: Implementation in Multiple Brain Metastases

Purpose: To propose, verify, and implement a simple and efficient methodology for the improvement of total geometric accuracy in multiple brain metastases gamma knife (GK) radiation surgery. Methods and Materials: The proposed methodology exploits the directional dependence of magnetic resonance imaging (MRI)-related spatial distortions stemming from background field inhomogeneities, also known as sequence-dependent distortions, with respect to the read-gradient polarity during MRI acquisition. First, an extra MRI pulse sequence is acquired with the same imaging parameters as those used for routine patient imaging, aside from a reversal in the read-gradient polarity. Then, “average” image data are compounded from data acquired from the 2 MRI sequences and are used for treatment planning purposes. The method was applied and verified in a polymer gel phantom irradiated with multiple shots in an extended region of the GK stereotactic space. Its clinical impact in dose delivery accuracy was assessed in 15 patients with a total of 96 relatively small (<2 cm) metastases treated with GK radiation surgery. Results: Phantom study results showed that use of average MR images eliminates the effect of sequence-dependent distortions, leading to a total spatial uncertainty of less than 0.3 mm, attributed mainly to gradient nonlinearities. In brain metastases patients, non-eliminated sequence-dependent distortions lead to target localization uncertainties of up to 1.3 mm (mean: 0.51 ± 0.37 mm) with respect to the corresponding target locations in the “average” MRI series. Due to these uncertainties, a considerable underdosage (5%-32% of the prescription dose) was found in 33% of the studied targets. Conclusions: The proposed methodology is simple and straightforward in its implementation. Regarding multiple brain metastases applications, the suggested approach may substantially improve total GK dose delivery accuracy in smaller, outlying targets.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2026

Large language models for transportation research: Methodologies, state of the art, and future opportunities

The rapid rise of large language models (LLMs) is transforming transportation research, with significant advancements emerging between 2023 and 2025, a period marked by the inception and swift growth of adopting and adapting LLMs for various transportation applications. Despite these significant advancements, however, a systematic review and synthesis of the existing literature remains lacking. This paper aims to fill this gap by providing a comprehensive review of the methodologies and applications of LLMs in transportation. We explore key applications, including autonomous driving, travel behavior prediction, and general transportation-related queries, alongside LLM methodologies such as zero- or few-shot learning, prompt engineering, and fine-tuning. From the review, critical research gaps are identified. From the methodological perspective, many of the research limitations can be addressed by integrating LLMs with existing tools and refining LLM architectures. From the application perspective, research opportunities for LLMs to address various transportation challenges are also explored. By synthesizing these findings, this review not only presents the state-of-the-art LLM adoption and adaptation in transportation, but also proposes future research directions as well as insights and recommendations for policymakers and practitioners, paving the way for greater LLM-driven research innovations in transportation in the future.

42 ENGINEERING↗
DOE OSTI2026

Measuring Success: A Refined Methodology for Estimating Long-term Continuous Improvement

Successful resource management systems require current and detailed feedback on operational and corporate-level performance. As corporate accountability concerns intensify, the precision and reliability of these performance metrics have become crucial. Traditional savings estimation methods can be difficult to understand, particularly linear regression, and can provide varying results. This paper reviews common efficiency metrics and highlights underlying mathematical inconsistencies when estimating total and percent savings with current methods. A refined approach to calculating long-term utility savings is proposed that simplifies current methodologies utilizing ratios to define an adjusted baseline, allowing for consistent and fair aggregation of results across multiple scales from resources to corporate performance. A simplified example demonstrates how the proposed methodology improves upon existing methods, especially in intermediate years. This paper’s major contributions are the simplified approach for converting modeled utility usage into estimated savings and the consistent roll-up methodology enabling more comparable, aggregable, and actionable results across scales.

Price, Chris [ORNL] (ORCID:0000000202007906)↗
DOE OSTI2021

MULTI-MODAL global surveillance methodology for predictive and on-demand characterization of localized processes using cube satellite platforms and deep learning techniques

This paper presents the work completed towards the development of a multi-modal global surveillance methodology using cube satellite (CubeSat) platforms and novel data analysis techniques. A CubeSat system equipped with adequate sensors and data analytics capabilities can autonomously characterize various phenomena of interest on the Earth’s surface. CubeSats are advantageous over conventional satellites in certain remote monitoring applications because of their reduced construction costs (due to the availability of commercially-off-the-shelf components) and are easier to launch. The CubeSat surveillance system developed in this paper focused on phenomena of interest surrounding the nuclear fuel cycle in support of nuclear non-proliferation and emergency response. To observe the phenomena, a constellation of 3U and 6U CubeSats deployed from the ISS with adequate components was chosen. Four different sensor configurations were identified for remote sensing: panchromatic/multispectral in the visible and near-infrared spectrum, multispectral in infrared spectrum, hyperspectral in infrared spectrum, and multispectral in ultraviolet spectrum. While a panchromatic/multispectral sensor configuration has CubeSat flight heritage at the required spatial resolutions, the other three sensor types need future 3 development to meet signature and system requirements. Once each sensor onboard the CubeSat system collects data on a target of interest, the onboard computers would then apply the deep learning-based characterization methodology developed in this paper to identify phenomena. Four surrogate datasets containing representative simplified “images” were created for each sensor type to train the characterization methodology. A convolutional neural network was applied to each dataset and produced recall rates for the phenomena between 89.7% - 99.3% and precision rates between 92.3% - 99.9%. Each phenomenon’s presence probability from each network is then combined into a final characterization solution for a target area. This paper covers multiple interdisciplinary areas to develop the foundation for a CubeSat surveillance system focused on phenomena surrounding the nuclear fuel cycle.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2015

Municipal solid waste composition: Sampling methodology, statistical analyses, and case study evaluation

Highlights: • Tiered approach to waste sorting ensures flexibility and facilitates comparison of solid waste composition data. • Food and miscellaneous wastes are the main fractions contributing to the residual household waste. • Separation of food packaging from food leftovers during sorting is not critical for determination of the solid waste composition. - Abstract: Sound waste management and optimisation of resource recovery require reliable data on solid waste generation and composition. In the absence of standardised and commonly accepted waste characterisation methodologies, various approaches have been reported in literature. This limits both comparability and applicability of the results. In this study, a waste sampling and sorting methodology for efficient and statistically robust characterisation of solid waste was introduced. The methodology was applied to residual waste collected from 1442 households distributed among 10 individual sub-areas in three Danish municipalities (both single and multi-family house areas). In total 17 tonnes of waste were sorted into 10–50 waste fractions, organised according to a three-level (tiered approach) facilitating comparison of the waste data between individual sub-areas with different fractionation (waste from one municipality was sorted at “Level III”, e.g. detailed, while the two others were sorted only at “Level I”). The results showed that residual household waste mainly contained food waste (42 ± 5%, mass per wet basis) and miscellaneous combustibles (18 ± 3%, mass per wet basis). The residual household waste generation rate in the study areas was 3–4 kg per person per week. Statistical analyses revealed that the waste composition was independent of variations in the waste generation rate. Both, waste composition and waste generation rates were statistically similar for each of the three municipalities. While the waste generation rates were similar for each of the two housing types (single-family and multi-family house areas), the individual percentage composition of food waste, paper, and glass was significantly different between the housing types. This indicates that housing type is a critical stratification parameter. Separating food leftovers from food packaging during manual sorting of the sampled waste did not have significant influence on the proportions of food waste and packaging materials, indicating that this step may not be required.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2024

A Reactor Scale-Up Methodology from Lab-Scale to Pilot-Scale Operations: Numerical Modeling of THFA Dehydration to DHP in Packed-Bed Reactors

This manuscript discusses developing a model-based scale-up methodology for a successful technology transfer of gas-phase catalytic reactors from a lab-scale to a pilot-scale operation. The manuscript demonstrates the methodology for gas-phase dehydration of tetrahydrofurfuryl alcohol (THFA) to dihydropyran (DHP) process over commercial Al 2 O 3 catalysts. A two-dimensional reactor model was developed using COMSOL Multiphysics 6.1 software. The model solves heat and mass transport equations in bed-scale and particle scales simultaneously. This powerful feature enables accurate prediction of the heat and mass transfer limitations in pilot-scale reactors, if any exists. Further, the model uses isothermal lab-scale experimental data to derive and validate the reaction chemistry, flow fields and boundary conditions. The model was then scaled-up to project conversion, selectivity, yield and formation rate of DHP in a pilot-scale reactor. The results highlight the complex nature of chemistry, heat, and mass transfer effects in lab-scale and pilot-scale reactors. The model results inform the possible operational limitations of the pilot-scale reactor and design strategies to improve process efficiency. Although the scale-up approach is explained through the THFA dehydration process, the methodology is applicable to any catalytic packed-bed reactor models for a successful process scale-up.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Methodological challenges and new perspectives of shifting vegetation phenology in eddy covariance data

While numerous studies report shifts in vegetation phenology, in this regard eddy covariance (EC) data, despite its continuous high-frequency observations, still requires further exploration. Furthermore, there is no general consensus on optimal methodologies for data smoothing and extracting phenological transition dates (PTDs). Here, we revisit existing methodologies and present new prospects to investigate phenological changes in gross primary productivity (GPP) from EC measurements. First, we present a smoothing technique of GPP time series through the derivative of its smoothed annual cumulative sum. Second, we calculate PTDs and their trends from a commonly used threshold method that identifies days with a fixed percentage of the annual maximum GPP. A systematic analysis is performed for various thresholds ranging from 0.1 to 0.7. Lastly, we examine the relation of PTDs trends to trends in GPP across the years on a weekly basis. Results from 47 EC sites with long time series (> 10 years) show that advancing trends in start of season (SOS) are strongest at lower thresholds but for the end of season (EOS) at higher thresholds. Moreover, the trends are variable at different thresholds for individual vegetation types and individual sites, outlining reasonable concerns on using a single threshold value. Relationship of trends in PTDs and weekly GPP reveal association of advanced SOS and delayed EOS to increase in immediate primary productivity, but not to the trends in overall seasonal productivity. Drawing on these analyses, we emphasise on abstaining from subjective choices and investigating relationship of PTDs trend to finer temporal trends of GPP. Our study examines existing methodological challenges and presents approaches that optimize the use of EC data in identifying vegetation phenological changes and their relation to carbon uptake.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Test methodology for validation of multi-frequency models of renewable energy generators using small-signal perturbations

Providing trustworthy and accurate multi-frequency (or harmonic) models for renewable energy generators (REG) is an ongoing challenge for harmonic studies. There have been effective attempts to propose and design a test device to validate the harmonic models, mainly based on shunt current perturbations. However, using additional devices for perturbations is costly for converter-based test sites. This paper provides the test specifications to extend the application of the grid emulators for voltage perturbations and appropriate harmonic model validation. Besides, the effects of the sequence couplings, initial emissions, and power set-points on the test results have been overlooked in the literature. Considering these effects, this paper proposes a generic test methodology to obtain more accurate models in the sequence domain. The experimental verification of the proposed methodology is demonstrated using a 7 MVA grid emulator for testing of a 2 MVA photo-voltaic converter and a 2 MVA Type 3 wind turbine. This way, the test challenges, specifications, and recommendations are presented using the MW-scale experiments on different REGs. Furthermore, the effects of sequence couplings and initial emissions on the calculation results are investigated and compared. The proposed methodology is applicable for harmonic model validation as well as empirical modelling.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2015

Criticism of generally accepted fundamentals and methodologies of traffic and transportation theory

It is explained why the set of the fundamental empirical features of traffic breakdown (a transition from free flow to congested traffic) should be the empirical basis for any traffic and transportation theory that can be reliable used for control and optimization in traffic networks. It is shown that generally accepted fundamentals and methodologies of traffic and transportation theory are not consistent with the set of the fundamental empirical features of traffic breakdown at a highway bottleneck. To these fundamentals and methodologies of traffic and transportation theory belong (i) Lighthill-Whitham-Richards (LWR) theory, (ii) the General Motors (GM) model class (for example, Herman, Gazis et al. GM model, Gipps’s model, Payne’s model, Newell’s optimal velocity (OV) model, Wiedemann’s model, Bando et al. OV model, Treiber’s IDM, Krauß’s model), (iii) the understanding of highway capacity as a particular stochastic value, and (iv) principles for traffic and transportation network optimization and control (for example, Wardrop’s user equilibrium (UE) and system optimum (SO) principles). Alternatively to these generally accepted fundamentals and methodologies of traffic and transportation theory, we discuss three-phase traffic theory as the basis for traffic flow modeling as well as briefly consider the network breakdown minimization (BM) principle for the optimization of traffic and transportation networks with road bottlenecks.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Design methodologies and engineering applications for ecosystem biomimicry: an interdisciplinary review spanning cyber, physical, and cyber-physical systems

Abstract Ecosystem biomimicry is a promising pathway for sustainable development. However, while typical form- and process-level biomimicry is prevalent, system-level ecosystem biomimicry remains a nascent practice in numerous engineering fields. This critical review takes an interdisciplinary approach to synthesize trends across case studies, evaluate design methodologies, and identify future opportunities when applying ecosystem biomimicry to engineering practices, including cyber systems (CS), physical systems (PS), and cyber-physical systems (CPS). After systematically sourcing publications from major databases, the papers were first analyzed at a meta level for their bibliographic context and for statistical correlations among categorical variables. Then, we investigated deeper into the engineering applications and design methodologies. Results indicate that CPS most frequently mimic organisms and ecosystems, while CS and PS frequently mimic populations-communities and molecules-tissues-organ systems, respectively (statistically highly significant). An indirect approach is most often used for mimicry at organizational levels from populations to ecosystems, while a direct approach frequently suits levels from molecules to organisms (highly significant). Dominant themes across engineering applications include symbiotic organism search algorithms for CS and ecological network analysis for CPS, while PS are highly diverse. For design methodologies, this work summarizes and details ten well-documented biomimetic process models among literature, which addresses an outdated concern for a lack of systematic methods for ecosystem biomimicry. In addition to the Biomimetics Standard ISO 18458, these methods include the Natural Step and Techno-Ecological Synergy framework, among others. Further, the analyses revealed future opportunities from less utilized design methods (e.g. interdisciplinary teams tackling indirect, ecosystem-level projects) to well-established engineering concepts ready for technological advancement (e.g. implementing membrane computing for physical applications). For future studies, this review provides a comprehensive reference for ecosystem biomimetic design practices and application opportunities across multiple engineering domains.

42 ENGINEERING↗
DOE OSTI2016

PV Degradation Methodology Comparison – A Basis for a Standard

What is the best method to determine long-term PV system performance and degradation rates? Ideally, one universally applicable methodology would be desirable so that a single number could be derived. However, data sets vary in their attributes and evidence is presented that defining two methodologies may be preferable. Monte Carlo simulations of artificial performance data allowed investigation of different methodologies and their respective confidence intervals. Tradeoffs between different approaches were delineated, elucidating as to why two separate approaches may need to be included in a standard. Regression approaches tend to be preferable when data sets are less contaminated by seasonality, noise and occurrence of outliers although robust regression can significantly improve the accuracy when outliers are present. In the presence of outliers, marked seasonality, or strong soiling events, year-on-year approaches tend to outperform regression approaches.

14 SOLAR ENERGY↗
DOE OSTI2015

Design methodology: edgeless 3D ASICs with complex in-pixel processing for pixel detectors

The design methodology for the development of 3D integrated edgeless pixel detectors with in-pixel processing using Electronic Design Automation (EDA) tools is presented. A large area 3 tier 3D detector with one sensor layer and two ASIC layers containing one analog and one digital tier, is built for x-ray photon time of arrival measurement and imaging. A full custom analog pixel is 65μm x 65μm. It is connected to a sensor pixel of the same size on one side, and on the other side it has approximately 40 connections to the digital pixel. A 32 x 32 edgeless array without any peripheral functional blocks constitutes a sub-chip. The sub-chip is an indivisible unit, which is further arranged in a 6 x 6 array to create the entire 1.248cm x 1.248cm ASIC. Each chip has 720 bump-bond I/O connections, on the back of the digital tier to the ceramic PCB. All the analog tier power and biasing is conveyed through the digital tier from the PCB. The assembly has no peripheral functional blocks, and hence the active area extends to the edge of the detector. This was achieved by using a few flavors of almost identical analog pixels (minimal variation in layout) to allow for peripheral biasing blocks to be placed within pixels. The 1024 pixels within a digital sub-chip array have a variety of full custom, semi-custom and automated timing driven functional blocks placed together. The methodology uses a modified mixed-mode on-top digital implementation flow to not only harness the tool efficiency for timing and floor-planning but also to maintain designer control over compact parasitically aware layout. The methodology uses the Cadence design platform, however it is not limited to this tool.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2016

WE-H-BRC-04: Implement Lean Methodology to Make Our Current Process of CT Simulation to Treatment More Efficient

Purpose: Our goal is to implement lean methodology to make our current process of CT simulation to treatment more efficient. Methods: In this study, we implemented lean methodology and tools and employed flowchart in excel for process-mapping. We formed a group of physicians, physicists, dosimetrists, therapists and a clinical physics assistant and huddled bi-weekly to map current value streams. We performed GEMBA walks and observed current processes from scheduling patient CT Simulations to treatment plan approval. From this, the entire workflow was categorized into processes, sub-processes, and tasks. For each process we gathered data on touch time, first time quality, undesirable effects (UDEs), and wait-times from relevant members of each task. UDEs were binned per frequency of their occurrence. We huddled to map future state and to find solutions to high frequency UDEs. We implemented visual controls, hard stops, and documented issues found during chart checks prior to treatment plan approval. Results: We have identified approximately 64 UDEs in our current workflow that could cause delays, re-work, compromise the quality and safety of patient treatments, or cause wait times between 1 – 6 days. While some UDEs are unavoidable, such as re-planning due to patient weight loss, eliminating avoidable UDEs is our goal. In 2015, we found 399 issues with patient treatment plans, of which 261, 95 and 43 were low, medium and high severity, respectively. We also mapped patient-specific QA processes for IMRT/Rapid Arc and SRS/SBRT, involving 10 and 18 steps, respectively. From these, 13 UDEs were found and 5 were addressed that solved 20% of issues. Conclusion: We have successfully implemented lean methodology and tools. We are further mapping treatment site specific workflows to identify bottlenecks, potential breakdowns and personnel allocation and employ tools like failure mode effects analysis to mitigate risk factors to make this process efficient.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

TH-CD-202-07: A Methodology for Generating Numerical Phantoms for Radiation Therapy Using Geometric Attribute Distribution Models

Purpose: A persistent challenge for the quality assessment of radiation therapy treatments (e.g. contouring accuracy) is the absence of the known, ground truth for patient data. Moreover, assessment results are often patient-dependent. Computer simulation studies utilizing numerical phantoms can be performed for quality assessment with a known ground truth. However, previously reported numerical phantoms do not include the statistical properties of inter-patient variations, as their models are based on only one patient. In addition, these models do not incorporate tumor data. In this study, a methodology was developed for generating numerical phantoms which encapsulate the statistical variations of patients within radiation therapy, including tumors. Methods: Based on previous work in contouring assessment, geometric attribute distribution (GAD) models were employed to model both the deterministic and stochastic properties of individual organs via principle component analysis. Using pre-existing radiation therapy contour data, the GAD models are trained to model the shape and centroid distributions of each organ. Then, organs with different shapes and positions can be generated by assigning statistically sound weights to the GAD model parameters. Organ contour data from 20 retrospective prostate patient cases were manually extracted and utilized to train the GAD models. As a demonstration, computer-simulated CT images of generated numerical phantoms were calculated and assessed subjectively and objectively for realism. Results: A cohort of numerical phantoms of the male human pelvis was generated. CT images were deemed realistic both subjectively and objectively in terms of image noise power spectrum. Conclusion: A methodology has been developed to generate realistic numerical anthropomorphic phantoms using pre-existing radiation therapy data. The GAD models guarantee that generated organs span the statistical distribution of observed radiation therapy patients, according to the training dataset. The methodology enables radiation therapy treatment assessment with multi-modality imaging and a known ground truth, and without patient-dependent bias.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2017

Design Science Methodology Applied to a Chemical Surveillance Tool

Public health surveillance systems gain significant benefits from integrating existing early incident detection systems,supported by closed data sources, with open source data.However, identifying potential alerting incidents relies on finding accurate, reliable sources and presenting the high volume of data in a way that increases analysts work efficiency; a challenge for any system that leverages open source data. In this paper, we present the design concept and the applied design science research methodology of ChemVeillance, a chemical analyst surveillance system.Our work portrays a system design and approach that translates theoretical methodology into practice creating a powerful surveillance system built for specific use cases.Researchers, designers, developers, and related professionals in the health surveillance community can build upon the principles and methodology described here to enhance and broaden current surveillance systems leading to improved situational awareness based on a robust integrated early warning system.

Huang, Zhuanyi↗
DOE OSTI2018

Methodology for Building Granular Testing in Multicomponent Scientific Software

Computational science and engineering communities develop complex application software with multiple mathematical models that need to interact with one another. Partly due to complexity of verifying scientific software, and partly because of the way incentives work in science, there has been insufficient testing of these codes. With a spotlight on the results produced with scientific software, and increasing awareness of software testing and verification as a critical contributor to the reliability of these results, testing is gaining more attention by the developing teams. However, many science teams struggle to find a good solution for themselves due either to lack of training or lack of resources within the team. In this experience paper, we describe test development methodologies utilized in two different scenarios: one explains a methodology for building granular tests where none existed before, while the second demonstrates a methodology for selecting test cases that build confidence in the software through a process similar to scaffolding. The common insight from both the experiences is that testing should be a part of software design from the beginning for better software and scientific productivity.

test building, granular testing, Earth system mode↗
DOE OSTI2018

Trip Energy Estimation Methodology and Model Based on Real-World Driving Data for Green-Routing Applications

A data-informed model to predict energy use for a proposed vehicle trip has been developed in this paper. The methodology leverages roughly one million miles of real-world driving data to generate the estimation model. Driving is categorized at the sub-trip level by average speed, road gradient, and road network geometry, then aggregated by category. An average energy consumption rate is determined for each category, creating an energy rate look-up table. Proposed vehicle trips are then categorized in the same manner, and estimated energy rates are appended from the look-up table. The methodology is robust and applicable to a wide range of driving data. The model has been trained on vehicle travel profiles from the Transportation Secure Data Center at the National Renewable Energy Laboratory and validated against on-road fuel consumption data from testing in Phoenix, Arizona. When compared against the detailed on-road conventional vehicle fuel consumption test data, the energy estimation model accurately predicted which route would consume less fuel over a dozen different tests. When compared against a larger set of real-world origin-destination pairs, it is estimated that implementing the present methodology should accurately select the route that consumes the least fuel 90% of the time. The model results can be used to inform control strategies in routing tools, such as change in departure time, alternate routing, and alternate destinations to reduce energy consumption. This work provides a highly extensible framework that allows the model to be tuned to a specific driver or vehicle type.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2014

Evaluation of Visual Analytics Environments: The Road to the Visual Analytics Science and Technology Challenge Evaluation Methodology

The evaluation of visual analytics environments was a topic in Illuminating the Path [Thomas 2005] as a critical aspect of moving research into practice. For a thorough understanding of the utility of the systems available, evaluation not only involves assessing the visualizations, interactions or data processing algorithms themselves, but also the complex processes that a tool is meant to support (such as exploratory data analysis and reasoning, communication through visualization, or collaborative data analysis [Lam 2012; Carpendale 2007]). Researchers and practitioners in the field have long identified many of the challenges faced when planning, conducting, and executing an evaluation of a visualization tool or system [Plaisant 2004]. Evaluation is needed to verify that algorithms and software systems work correctly and that they represent improvements over the current infrastructure. Additionally to effectively transfer new software into a working environment, it is necessary to ensure that the software has utility for the end-users and that the software can be incorporated into the end-user’s infrastructure and work practices. Evaluation test beds require datasets, tasks, metrics and evaluation methodologies. As noted in [Thomas 2005] it is difficult and expensive for any one researcher to setup an evaluation test bed so in many cases evaluation is setup for communities of researchers or for various research projects or programs. Examples of successful community evaluations can be found [Chinchor 1993; Voorhees 2007; FRGC 2012]. As visual analytics environments are intended to facilitate the work of human analysts, one aspect of evaluation needs to focus on the utility of the software to the end-user. This requires representative users, representative tasks, and metrics that measure the utility to the end-user. This is even more difficult as now one aspect of the test methodology is access to representative end-users to participate in the evaluation. In many cases the sensitive nature of data and tasks and difficult access to busy analysts puts even more of a burden on researchers to complete this type of evaluation. User-centered design goes beyond evaluation and starts with the user [Beyer 1997, Shneiderman 2009]. Having some knowledge of the type of data, tasks, and work practices helps researchers and developers know the correct paths to pursue in their work. When access to the end-users is problematic at best and impossible at worst, user-centered design becomes difficult. Researchers are unlikely to go to work on the type of problems faced by inaccessible users. Commercial vendors have difficulties evaluating and improving their products when they cannot observe real users working with their products. In well-established fields such as web site design or office software design, user-interface guidelines have been developed based on the results of empirical studies or the experience of experts. Guidelines can speed up the design process and replace some of the need for observation of actual users [heuristics review references]. In 2006 when the visual analytics community was initially getting organized, no such guidelines existed. Therefore, we were faced with the problem of developing an evaluation framework for the field of visual analytics that would provide representative situations and datasets, representative tasks and utility metrics, and finally a test methodology which would include a surrogate for representative users, increase interest in conducting research in the field, and provide sufficient feedback to the researchers so that they could improve their systems.

evaluation↗