Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “ASSESSMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27

Save Energy Now Assessments Results 2008 Summary Report

In October 2005, U.S. Department of Energy Secretary Bodman launched his Easy Ways to Save Energy campaign with a promise to provide energy assessments to 200 of the largest U.S. manufacturing plants. DOE's Industrial Technologies Program (ITP) responded to the Secretary's campaign with its Save Energy Now initiative, featuring a new and highly cost-effective form of energy savings assessment. The approach for these assessments drew heavily on the existing resources of ITP's technology delivery component. Over the years, ITP Technology Delivery has worked with industry partners to assemble a suite of respected software tools, proven assessment protocols, training curricula, certified energy experts, and strong partnerships for deployment. The Save Energy Now assessments conducted in calendar year 2006 focused on natural gas savings and targeted many of the nation's largest manufacturing plants - those that consume at least 1 TBtu of energy annually. The 2006 Save Energy Now assessments focused primarily on assessments of steam and process heating systems, which account for an estimated 74% of all natural gas use by U.S. manufacturing plants. Because of the success of the Save Energy Now assessments conducted in 2006 and 2007, the program was expanded and enhanced in two major ways in 2008: (1) a new goal was set to perform at least 260 assessments; and (2) the assessment focus was expanded to include pumping, compressed air, and fan systems in addition to steam and process heating. DOE ITP also has developed software tools to assess energy efficiency improvement opportunities in pumping, compressed air, and fan systems. The Save Energy Now assessments integrate a strong training component designed to teach industrial plant personnel how to use DOE's opportunity assessment software tools. This approach has the advantages of promoting strong buy-in of plant personnel for the assessment and its outcomes and preparing them better to independently replicate the assessment process at the company's other facilities. Another important element of the Save Energy Now assessment process is the follow-up process used to identify how many of the recommended savings opportunities from individual assessments have been implemented in the industrial plants. Plant personnel involved with the Save Energy Now assessments are contacted 6 months, 12 months, and 24 months after individual assessments are completed to determine implementation results. A total of 260 Save Energy Now assessments were successfully completed in calendar year 2008. This means that a total of 718 assessments were completed in 2006, 2007, and 2008. As of July 2009, we have received a total of 239 summary reports from the ESAs that were conducted in year 2008. Hence, at the time that this report was prepared, 680 final assessment reports were completed (200 from year 2006, 241 from year 2007, and 239 from year 2008). The total identified potential cost savings from these 680 assessments is $1.1 billion per year, including natural gas savings of about 98 TBtu per year. These results, if fully implemented, could reduce CO{sub 2} emissions by about 8.9 million metric tons annually. When this report was prepared, data on implementation of recommended energy and cost savings measures from 488 Save Energy Now assessments were available. For these 488 plants, measures saving a total of $147 million per year have been implemented, measures that will save $169 million per year are in the process of being implemented, and plants are planning implementation of measures that will save another $239 million per year. The implemented recommendations are already achieving total CO{sub 2} reductions of about 1.8 million metric tons per year. This report provides a summary of the key results for the Save Energy Now assessments completed in 2008; details of the 6-month, 12-month, and 24-month implementation results obtained to date; and an evaluation of these implementation results. This report also summarizes key accomplishments, findings, and lessons learned from all the Save Energy Now assessments completed to date. A separate report (Wright et al. 2010) provides more detailed information on key results for all of the 2008 assessments of steam, process heating, pumping, compressed air, and fan systems. Two prior reports (Wright et al. 2007 and Wright et al. 2009) detail the results from the 2006 and 2007 assessments and discuss the major components of the assessment process and improvements in the process made in 2007.

03 NATURAL GAS↗

Save Energy Now Assessments Results 2008 Detailed Report

In October 2005, U.S. Department of Energy Secretary Bodman launched his Easy Ways to Save Energy campaign with a promise to provide energy assessments to 200 of the largest U.S. manufacturing plants. DOE's Industrial Technologies Program (ITP) responded to the Secretary's campaign with its Save Energy Now initiative, featuring a new and highly cost-effective form of energy savings assessment. The approach for these assessments drew heavily on the existing resources of ITP's technology delivery component. Over the years, ITP Technology Delivery has worked with industry partners to assemble a suite of respected software tools, proven assessment protocols, training curricula, certified energy experts, and strong partnerships for deployment. The Save Energy Now assessments conducted in calendar year 2006 focused on natural gas savings and targeted many of the nation's largest manufacturing plants - those that consume at least 1 TBtu of energy annually. The 2006 Save Energy Now assessments focused primarily on assessments of steam and process heating systems, which account for an estimated 74% of all natural gas use by U.S. manufacturing plants. Because of the success of the Save Energy Now assessments conducted in 2006 and 2007, the program was expanded and enhanced in two major ways in 2008: (1) a new goal was set to perform at least 260 assessments; and (2) the assessment focus was expanded to include pumping, compressed air, and fan systems in addition to steam and process heating. DOE ITP also has developed software tools to assess energy efficiency improvement opportunities in pumping, compressed air, and fan systems. The Save Energy Now assessments integrate a strong training component designed to teach industrial plant personnel how to use DOE's opportunity assessment software tools. This approach has the advantages of promoting strong buy-in of plant personnel for the assessment and its outcomes and preparing them better to independently replicate the assessment process at the company's other facilities. Another important element of the Save Energy Now assessment process is the follow-up process used to identify how many of the recommended savings opportunities from individual assessments have been implemented in the industrial plants. Plant personnel involved with the Save Energy Now assessments are contacted 6 months, 12 months, and 24 months after individual assessments are completed to determine implementation results. A total of 260 Save Energy Now assessments were successfully completed in calendar year 2008. This means that a total of 718 assessments were completed in 2006, 2007, and 2008. As of July 2009, we have received a total of 239 summary reports from the ESAs that were conducted in year 2008. Hence, at the time that this report was prepared, 680 final assessment reports were completed (200 from year 2006, 241 from year 2007, and 239 from year 2008). The total identified potential cost savings from these 680 assessments is $1.1 billion per year, including natural gas savings of about 98 TBtu per year. These results, if fully implemented, could reduce CO{sub 2} emissions by about 8.9 million metric tons annually. When this report was prepared, data on implementation of recommended energy and cost savings measures from 488 Save Energy Now assessments were available. For these 488 plants, measures saving a total of $147 million per year have been implemented, measures that will save $169 million per year are in the process of being implemented, and plants are planning implementation of measures that will save another $239 million per year. The implemented recommendations are already achieving total CO{sub 2} reductions of about 1.8 million metric tons per year. This report provides a summary of the key results for the Save Energy Now assessments completed in 2008; details of the 6-month, 12-month, and 24-month implementation results obtained to date; and an evaluation of these implementation results. This report also summarizes key accomplishments, findings, and lessons learned from all the Save Energy Now assessments completed to date. A separate report (Wright et al. 2010) provides more detailed information on key results for all of the 2008 assessments of steam, process heating, pumping, compressed air, and fan systems. Two prior reports (Wright et al. 2007 and Wright et al. 2009) detail the results from the 2006 and 2007 assessments and discuss the major components of the assessment process and improvements in the process made in 2007.

03 NATURAL GAS↗

Remote Home Energy Score Assessments (Feasibility Study)

The delivery of Home Energy Scores (HES) has been significantly impacted over the last year by the recent pandemic and ongoing limitations to person-to-person interactions. There are also historic challenges in providing HES to more remote geographies where assessor infrastructure is not in place. Alternative approaches to in-person, on-site home energy assessments could help address these current restrictions, while also potentially providing opportunities for the efficient delivery of HES in certain use cases after COVID-related restrictions are lifted. From the program administrator’s perspective, there needs to be confidence that the remote scores closely match what they would have been if the assessment had been performed on-site. The recommendations shown on a report generated from a remote score should closely match those of an onsite assessment. A remote assessment should be an option for all HES Partners and have direction on how to deliver those scores. From the HES partner’s and assessor’s perspective, there needs to be understanding of what level of experience is needed, what system infrastructure is required, the time expenditure of the remote assessment process, and the limitations of a remote assessment. From the homeowner’s perspective, there needs to be an awareness and comfort with the time commitment, the level of knowledge of the home, the equipment needed, and any physical requirements of the remote assessment process. During this study, Earth Advantage assessed methods for delivering Home Energy Score assessments remotely through web-based technology platforms and resident interaction in a diverse array of home types in different geographic locations. Earth Advantage developed remote assessment test protocols and integrated those testing protocols into existing Home Energy Score programs being overseen by USDOE Home Energy Score partners New York State Energy Research & Development Authority (NYSERDA), the Oregon Department of Energy (ODOE), and the City of Portland (PDX). Earth Advantage staff acted as the Remote assessor and gathered HES data remotely during the video sessions with participants. Authorized Home Energy Score assessors performed the onsite assessments that were then used to compare with the remote assessment data. This allowed the research to effectively test the efficacy of various remote assessment approaches in a diverse array of home-types. Earth advantage identified the key components of a remote assessment infrastructure which included Remote Home Energy Score Assessments Page vi processes, tools, services, methods and mechanisms that would better enable remote assessments. This report details methods, results, and findings of the HES remote assessment research. In addition, the report provides recommendations and best practices that could be used as guidance for the Home Energy Score administrator and the HES partners seeking to effectively conduct remote assessments.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Environment, Safety and Health Self-Assessment Report Fiscal Year 2010

The Lawrence Berkeley National Laboratory (LBNL) Environment, Safety, and Health (ES&H) Self-Assessment Program was established to ensure that Integrated Safety Management (ISM) is implemented institutionally and by all divisions. The ES&H Self-Assessment Program, managed by the Office of Contractor Assurance (OCA), provides for an internal evaluation of all ES&H programs and systems at LBNL. The primary objective of the program is to ensure that work is conducted safely and with minimal negative impact to workers, the public, and the environment. Self-assessment follows the five core functions and guiding principles of ISM. Self-assessment is the mechanism used to promote the continuous improvement of the Laboratory's ES&H programs. The process is described in the Environment, Safety, and Health Assurance Plan (PUB-5344) and is composed of three types of self-assessments: Division ES&H Self-Assessment, ES&H Technical Assurance Program Assessment, and Division ES&H Peer Review. The Division ES&H Self-Assessment Manual (PUB-3105) provides the framework by which divisions conduct formal ES&H self-assessments to systematically identify program deficiencies. Issue-specific assessments are designed and implemented by the divisions and focus on areas of interest to division management. They may be conducted by teams and involve advance planning to ensure that appropriate resources are available. The ES&H Technical Assurance Program Manual (PUB-913E) provides the framework for systematic reviews of ES&H programs and processes. The ES&H Technical Assurance Program Assessment is designed to evaluate whether ES&H programs and processes are compliant with guiding regulations, are effective, and are properly implemented by LBNL divisions. The Division ES&H Peer Review Manual provides the framework by which division ISM systems are evaluated and improved. Peer Reviews are conducted by teams under the direction of senior division management and focus on higher-level management issues. Peer Review teams are selected on the basis of members knowledge and experience in the issues of interest to the division director. LBNL periodically requests in-depth independent assessments of selected ES&H programs. Such assessments augment LBNL's established assessment processes and provide an objective view of ES&H program effectiveness. Institutional Findings, Observations, and Noteworthy Practices identified during independent assessments are specifically intended to help LBNL identify opportunities for program improvement. This report includes the results of the Division ES&H Self-Assessment, ES&H Technical Assurance Program Assessment, and Division ES&H Peer Review, respectively.

99 GENERAL AND MISCELLANEOUS↗

A Retrospective Tiered Environmental Assessment of the Mount Storm Wind Energy Facility, West Virginia, USA

Bird and bat fatalities from wind energy projects are an environmental and public concern, with post-construction fatalities sometimes differing from predictions. Siting facilities in this context can be a challenge. In March 2012 the U.S. Fish and Wildlife Service (USFWS) released Land-based Wind Energy Guidelines to assess collision fatalities and other potential impacts to species of concern and their habitats to aid in siting and management. The Guidelines recommend a tiered approach for assessing risk to wildlife, including a preliminary site evaluation that may evaluate alternative sites, a site characterization, field studies to document wildlife and habitat and to predict project impacts, post construction studies to estimate impacts, and other post construction studies. We applied the tiered assessment framework to a case study site, the Mount Storm Wind Energy Facility in Grant County, West Virginia, USA, to demonstrate the use of the USFWS assessment approach, to indicate how the use of a tiered assessment framework might have altered outputs of wildlife assessments previously undertaken for the case study site, and to assess benefits of a tiered ecological assessment framework for siting wind energy facilities. The conclusions of this tiered assessment for birds are similar to those of previous environmental assessments for Mount Storm. This assessment found risk to individual migratory tree-roosting bats that was not emphasized in previous preconstruction assessments. Differences compared to previous environmental assessments are more related to knowledge accrued in the past 10 years rather than to the tiered structure of the Guidelines. Benefits of the tiered assessment framework include good communication among stakeholders, clear decision points, a standard assessment trajectory, narrowing the list of species of concern, improving study protocols, promoting consideration of population-level effects, promoting adaptive management through post-construction assessment and mitigation, and sharing information that can be used in other assessments.

17 WIND ENERGY↗

OVERVIEW OF THE U.S. DEPARTMENT OF ENERGY AND NUCLEAR REGULATORY COMMISSION PERFORMANCE ASSESSMENT APPROACHES: CEMENTITIOUS BARRIERS PARTNERSHIP

Engineered barriers including cementitious barriers are used at sites disposing or contaminated with low-level radioactive waste to enhance performance of the natural environment with respect to controlling the potential spread of contaminants. Drivers for using cementitious barriers include: high radionuclide inventory, radionuclide characteristics (e.g., long half-live, high mobility due to chemical form/speciation, waste matrix properties, shallow water table, and humid climate that provides water for leaching the waste). This document comprises the first in a series of reports being prepared for the Cementitious Barriers Partnership. The document is divided into two parts which provide a summary of: (1) existing experience in the assessment of performance of cementitious materials used for radioactive waste management and disposal and (2) sensitivity and uncertainty analysis approaches that have been applied for assessments. Each chapter is organized into five parts: Introduction, Regulatory Considerations, Specific Examples, Summary of Modeling Approaches and Conclusions and Needs. The objective of the report is to provide perspective on the state of the practice for conducting assessments for facilities involving cementitious barriers and to identify opportunities for improvements to the existing approaches. Examples are provided in two contexts: (1) performance assessments conducted for waste disposal facilities and (2) performance assessment-like analyses (e.g., risk assessments) conducted under other regulatory regimes. The introductory sections of each section provide a perspective on the purpose of performance assessments and different roles of cementitious materials for radioactive waste management. Significant experience with assessments of cementitious materials associated with radioactive waste disposal concepts exists in the US Department of Energy Complex and the commercial nuclear sector. Recently, the desire to close legacy facilities has created a need to assess the behavior of cementitious materials for applications in environmental remediation and decontamination and decommissioning (D&D) applications. The ability to assess the use and benefits of cementitious materials for these applications can significantly affect decisions related to cleanup activities. For example the need for costly remedial actions may not be necessary if existing or new cementitious barriers were adequately represented. The sections dealing with regulatory considerations include summaries of the different regulations that are relevant for various applications involving cementitious materials. A summary of regulatory guidance and/or policies pertaining to performance assessment of cementitious materials and sensitivity and uncertainty analyses is also provided in the following chapters. Numerous examples of specific applications are provided in each report. The examples are organized into traditional waste disposal applications (performance assessments), applications related to environmental remediation and D&D, and reactor and spent fuel related assessments. Sections that discuss specific facilities or sites contain: (1) descriptions of the role of the cementitious barriers or sensitivity/uncertainty analysis, (2) parameter assumptions and conceptual models, and (3) a relative discussion of the significance in the context of the assessment. Examples from both the U.S. Department of Energy Sites and the U.S. Nuclear Regulatory Commission are provided to illustrate the variety of applications and approaches that have been used. In many cases, minimal credit was taken for cementitious barriers. However, in some of those cases, benefits of being able to take credit for barriers were identified. The examples included: (1) disposal facilities (vaults, trenches, tank closures, cementitious waste forms and containers, etc.), (2) environmental remediation (old disposal facilities), (3) reactor and large structure decommissioning, and (4) spent fuel pools. These examples were selected to provide a perspective on the various needs, capabilities to model cementitious barriers, and use of sensitivity and uncertainty analysis and were not intended to include all cementitious barriers used in all low-level waste related PAs. The summary section in each report/chapter provides an overview of important considerations for the examples and compares and contrasts the different approaches that have been used. For example, specific time dependent physical processes (changes in hydraulic conductivity) and chemical processes (partitioning coefficients, and solubility coefficients) are identified and compared. The summary section also identifies key needs for future assessments. The Cementitious Barriers Partnership was established to address the key needs related to the use of cementitious barriers - waste forms, containment structures, physical stabilization fill materials.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Dependability Assessment of Software for Safety Instrumentation and Control Systems

In its 2013 recommendations to the International Atomic Energy Agency (IAEA), the Technical Working Group on Nuclear Power Plant Instrumentation and Control (TWG-NPPIC) identified assessment and evaluation of software dependability as a topic of interest for IAEA Member States. To implement the recommendation of the TWG-NPPIC, the IAEA Scientific Secretary for the TWG initiated a consultancy activity in 2014 to develop a technical report describing a dependability assessment framework suitable for safety instrumentation and control (I and C) systems in nuclear power plants (NPPs). The development of the software dependability assessment report has involved three consultancy meetings at IAEA headquarters in Vienna, Austria. The conduct of these meetings allowed consultation among subject matter experts to generate consensus technical information that serves as the content of the report. The technical contributors consist of international experts from academia, industry, regulatory bodies and research organizations. In September 2014, a technical meeting was held in Daejeon, Republic of Korea, to present interim findings of the consultancy effort and to solicit input from representatives of IAEA Member States. The information and technical comments contributed during the technical meeting were addressed in subsequent development of the report. This paper gives an overview of key technical elements of the IAEA report. The IAEA report presents a framework by which a dependability assessment of safety software can be performed. The specified guiding principles provide the basis for establishing an assessment strategy and implementing the assessment process. The assessment framework provides a systematic, coherent structure within which to organize technical and process evidence to satisfy arguments in support of claims about software dependability properties. During the consultancy activity, the participating subject matter experts focused on high level principles, a structured approach, and identification of suitable methods and techniques. However, it was recognized that what remains is to concretize the framework through specific examples and usage experience. Thus, following publication of the report, it is recommended that case studies be performed to illustrate and more precisely define specific phases and elements of the assessment framework. For example, current national assessment approaches could be mapped into the assessment framework to serve as case studies and help to identify any aspects of the assessment framework that needs further definition or adjustment. Additionally, a challenge problem could be defined to engage stakeholders to apply the assessment framework with a common focus. This could help to better illustrate strategic options and approaches to the assessment phases. It also could serve to suggest other considerations and potential sources of evidence not fully covered in the report. Based on findings from case studies, challenge problems and country experience, it should be possible to develop generalized templates to support application of the assessment framework. These findings can be collected to serve as the basis for a User's Guide on dependability assessment of safety software. (authors)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Challenges to Evaluating Ecological Risks in Performance Assessments - 18424

Performance assessments (PAs), whether probabilistic or deterministic, have largely focused on evaluating human health effects of radionuclides. In the past, it was asserted that protecting human health would automatically protect the environment. This blanket assertion has been questioned, so tools and methodologies to directly estimate non-human biota dose have been developed and made available since the early 2000's. For example, the Biota Dose Assessment Committee (BDAC) guidance led to the development of the RESRAD-BIOTA (RESRAD stands for Residual Radioactive materials) software to implement this guidance. The BDAC was a US Department of Energy (DOE) led working group with members from academia and consultancies in addition to DOE technical staff. Parallel efforts have been in place in Europe with the development of the ERICA tool (Environmental Risk from Ionising Contaminants: Assessment and Management). More recently, interpretation of the requirements of US Department of Energy (DOE) Order O458.1 have led some waste management facilities to include ecological risks in addition to human health effects in their PAs. We have recently developed methodologies to assess ecological risks within probabilistic PAs (PPAs) for two Federal facilities. One of the challenges in applying the non-human biota assessment models to PPAs is that ecological risk assessments following US Environmental Protection Agency (EPA) or DOE guidance start with protective screening assessments before moving into more site-specific ecological risk assessments. Conservative assumptions bias the screening assessments and greatly overestimate the risks posed by contaminants. However, although the screens are biased they can be useful in ruling out model spatial domain areas or types of biota from requiring further assessment. In some cases, the results of the screen could entirely rule out ecological risks from further assessment. For a more refined and unbiased estimate of dose to non-human biota, one would typically conduct site-specific studies to better define exposure through the food web or adverse effects based on eco-toxicity studies of environmental, and generally weathered, contaminant mixtures. However, for PPAs that consider future site conditions after the loss of institutional control, such site-specific studies are not possible. Rather, site-specific future ecological risk models use probabilistic inputs that result in a distribution of population doses or exposures and calculate the distribution of possible adverse effects to ecological populations. This approach considers both radionuclide and chemical inventory. For both Federal facilities, the model spatial domains include riparian and aquatic environments located downstream of the waste management site. These riparian and aquatic environments represent linear exposure elements for some wildlife populations and require special consideration of exposures to individuals and to populations along such spatial domains. At a minimum, wildlife foraging should be focused along the axis of such water bodies. Another challenge is how to incorporate ecologically relevant spatial scales into models that have contaminant transport as the primary focus. Riparian and aquatic environments also have threatened and endangered (T and E) species, which necessitate greater levels of protection given their rarity. A final challenge for conducting ecological risk assessments is how to evaluate T and E species in the future. This might be addressed by evaluating potential for impacts on individuals of feeding guilds with species that are currently threatened or endangered. We also have to consider how ecological populations might change as a result of climate change. It is a more difficult problem to predict what species might become rare in the future, and how to best assess future populations of threatened or endangered species. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Practitioners, professional cultures, and perceptions of impact assessment

The very nature of impact assessment (IA) means that it often involves practitioners from a very wide range of disciplinary and professional backgrounds, which open the possibility that how IA is perceived and practised may vary according to the professional background of the practitioner. The purpose of this study is to investigate the extent to which a practitioner's professional background influences their perceptions of the adequacy of impact assessment in New Zealand under the Resource Management Act (RMA). Information gathered concerned professional affiliations, training, understanding of impact assessment practise, and perceptions of adequacy in relation to impact assessment. The results showed a dominance of a legalistic, operational perspective of impact assessment under the Resource Management Act, across all the main professions represented in the study. However, among preparers of impact assessments there was clear evidence of differences between the four main professional groups - surveyors, planners, engineers and natural scientists - in the way they see the nature and purpose of impact assessment, the practical steps involved, and what constitutes adequacy. Similarly, impact assessment reviewers - predominantly planners and lawyers - showed variations in their expectations of impact assessment depending on their respective professional affiliation. Although in many cases the differences seem to be more of a matter of emphasis, rather than major disputes on what constitutes a good process, even those differences can add up to rather distinct professional cultures of impact assessment. The following factors are seen as leading to the emergence of such professional cultures: different professions often contribute in different ways to an impact assessment, affecting their perception of the nature and purpose of the process; impact assessment training will usually be a secondary concern, compared with the core professional training, which will be reflected in the depth and length of such training; and any impact assessment training provided within a profession will often have the 'cultural' imprint of that profession.

54 ENVIRONMENTAL SCIENCES↗

Self-assessed performance improves statistical fusion of image labels

Purpose: Expert manual labeling is the gold standard for image segmentation, but this process is difficult, time-consuming, and prone to inter-individual differences. While fully automated methods have successfully targeted many anatomies, automated methods have not yet been developed for numerous essential structures (e.g., the internal structure of the spinal cord as seen on magnetic resonance imaging). Collaborative labeling is a new paradigm that offers a robust alternative that may realize both the throughput of automation and the guidance of experts. Yet, distributing manual labeling expertise across individuals and sites introduces potential human factors concerns (e.g., training, software usability) and statistical considerations (e.g., fusion of information, assessment of confidence, bias) that must be further explored. During the labeling process, it is simple to ask raters to self-assess the confidence of their labels, but this is rarely done and has not been previously quantitatively studied. Herein, the authors explore the utility of self-assessment in relation to automated assessment of rater performance in the context of statistical fusion. Methods: The authors conducted a study of 66 volumes manually labeled by 75 minimally trained human raters recruited from the university undergraduate population. Raters were given 15 min of training during which they were shown examples of correct segmentation, and the online segmentation tool was demonstrated. The volumes were labeled 2D slice-wise, and the slices were unordered. A self-assessed quality metric was produced by raters for each slice by marking a confidence bar superimposed on the slice. Volumes produced by both voting and statistical fusion algorithms were compared against a set of expert segmentations of the same volumes. Results: Labels for 8825 distinct slices were obtained. Simple majority voting resulted in statistically poorer performance than voting weighted by self-assessed performance. Statistical fusion resulted in statistically indistinguishable performance from self-assessed weighted voting. The authors developed a new theoretical basis for using self-assessed performance in the framework of statistical fusion and demonstrated that the combined sources of information (both statistical assessment and self-assessment) yielded statistically significant improvement over the methods considered separately. Conclusions: The authors present the first systematic characterization of self-assessed performance in manual labeling. The authors demonstrate that self-assessment and statistical fusion yield similar, but complementary, benefits for label fusion. Finally, the authors present a new theoretical basis for combining self-assessments with statistical label fusion.

62 RADIOLOGY AND NUCLEAR MEDICINE↗