Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “expert elicitation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

50 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS1976

Cost estimation: An expert-opinion approach

A methodology is outlined which can be used to estimate the costs of research and development projects. The approach uses the Delphi technique a method developed by the Rand Corporation for systematically eliciting and evaluating group judgments in an objective manner. The use of the Delphi allows for the integration of expert opinion into the cost-estimating process in a consistent and rigorous fashion. This approach can also signal potential cost-problem areas. This result can be a useful tool in planning additional cost analysis or in estimating contingency funds. A Monte Carlo approach is also examined.

Buffalano, C.↗
NASA NTRS1987

Program risk analysis handbook

NASA regulations specify that formal risk analysis be performed on a program at each of several milestones. Program risk analysis is discussed as a systems analysis approach, an iterative process (identification, assessment, management), and a collection of techniques. These techniques, which range from extremely simple to complex network-based simulation, are described in this handbook in order to provide both analyst and manager with a guide for selection of the most appropriate technique. All program risk assessment techniques are shown to be based on elicitation and encoding of subjective probability estimates from the various area experts on a program. Techniques to encode the five most common distribution types are given. Then, a total of twelve distinct approaches to risk assessment are given. Steps involved, good and bad points, time involved, and degree of computer support needed are listed. Why risk analysis should be used by all NASA program managers is discussed. Tools available at NASA-MSFC are identified, along with commercially available software. Bibliography (150 entries) and a program risk analysis check-list are provided.

Batson, R. G.↗
NASA NTRS1997

An Approach to Building a Traceability Tool for Software Development

It is difficult in a large, complex computer program to ensure that it meets the specified requirements. As the program evolves over time, a11 program constraints originally elicited during the requirements phase must be maintained. In addition, during the life cycle of the program, requirements typically change and the program must consistently reflect those changes. Imagine the following scenario. Company X wants to develop a system to automate its assembly line. With such a large system, there are many different stakeholders, e.g., managers, experts such as industrial and mechanical engineers, and end-users. Requirements would be elicited from all of the stake holders involved in the system with each stakeholder contributing their point of view to the requirements. For example, some of the requirements provided by an industrial engineer may concern the movement of parts through the assembly line. A point of view provided by the electrical engineer may be reflected in constraints concerning maximum power usage. End-users may be concerned with comfort and safety issues, whereas managers are concerned with the efficiency of the operation. With so many points of view affecting the requirements, it is difficult to manage them, communicate information to relevant stakeholders. and it is likely that conflicts in the requirements will arise. In the coding process, the implementors will make additional assumptions and interpretations on the design and the requirements of the system. During any stage of development, stakeholders may request that a requirement be added or changed. In such a dynamic environment, it is difficult to guarantee that the system will preserve the current set of requirements. Tracing, the mapping between objects in the artifacts of the system being developed, addresses this issue. Artifacts encompass documents such as the system definition, interview transcripts, memoranda, the software requirements specification, user's manuals, the functional specifications, design reports, and system code. Tracing helps 1) validate system features against, the requirement specification, 2) identify error sources and, most importantly, 3) manage change. With so many people involved in the development of the system, it becomes necessary to identify the reasons behind the design requirements or the implementation decisions. This paper is concerned with an approach that maps documents to constraints that capture properties of and relationships between the objects being modeled by the program. Section 2 provides the reader with a background on traceability tools. Section 3 gives a brief description of the context monitoring system on which the approach suggested in this paper is based. Section 4 presents an overview of our approach to providing traceability. The last section presents our future direction of research.

Delgado, Nelly↗
NASA NTRS1986

Cost Risk Analysis Based on Perception of the Engineering Process

In most cost estimating applications at the NASA Langley Research Center (LaRC), it is desirable to present predicted cost as a range of possible costs rather than a single predicted cost. A cost risk analysis generates a range of cost for a project and assigns a probability level to each cost value in the range. Constructing a cost risk curve requires a good estimate of the expected cost of a project. It must also include a good estimate of expected variance of the cost. Many cost risk analyses are based upon an expert's knowledge of the cost of similar projects in the past. In a common scenario, a manager or engineer, asked to estimate the cost of a project in his area of expertise, will gather historical cost data from a similar completed project. The cost of the completed project is adjusted using the perceived technical and economic differences between the two projects. This allows errors from at least three sources. The historical cost data may be in error by some unknown amount. The managers' evaluation of the new project and its similarity to the old project may be in error. The factors used to adjust the cost of the old project may not correctly reflect the differences. Some risk analyses are based on untested hypotheses about the form of the statistical distribution that underlies the distribution of possible cost. The usual problem is not just to come up with an estimate of the cost of a project, but to predict the range of values into which the cost may fall and with what level of confidence the prediction is made. Risk analysis techniques that assume the shape of the underlying cost distribution and derive the risk curve from a single estimate plus and minus some amount usually fail to take into account the actual magnitude of the uncertainty in cost due to technical factors in the project itself. This paper addresses a cost risk method that is based on parametric estimates of the technical factors involved in the project being costed. The engineering process parameters are elicited from the engineer/expert on the project and are based on that expert's technical knowledge. These are converted by a parametric cost model into a cost estimate. The method discussed makes no assumptions about the distribution underlying the distribution of possible costs, and is not tied to the analysis of previous projects, except through the expert calibrations performed by the parametric cost analyst.

Dean, Edwin B.↗
NASA NTRS2009

Review of FEWS NET Biophysical Monitoring Requirements

The Famine Early Warning System Network (FEWS NET) provides monitoring and early warning support to decision makers responsible for responding to famine and food insecurity. FEWS NET transforms satellite remote sensing data into rainfall and vegetation information that can be used by these decision makers. The National Aeronautics and Space Administration has recently funded activities to enhance remote sensing inputs to FEWS NET. To elicit Earth observation requirements, a professional review questionnaire was disseminated to FEWS NET expert end-users: it focused upon operational requirements to determine additional useful remote sensing data and; subsequently, beneficial FEWS NET biophysical supplementary inputs. The review was completed by over 40 experts from around the world, enabling a robust set of professional perspectives to be gathered and analyzed rapidly. Reviewers were asked to evaluate the relative importance of environmental variables and spatio-temporal requirements for Earth science data products, in particular for rainfall and vegetation products. The results showed that spatio-temporal resolution requirements are complex and need to vary according to place, time, and hazard: that high resolution remote sensing products continue to be in demand, and that rainfall and vegetation products were valued as data that provide actionable food security information.

Ross, K. W.↗
NASA NTRSDate not supplied

The Ten Lockheed Martin Cyber-Physical Challenges: Formalized, Analyzed, and Explained

Capturing and analyzing requirements of Cyber-Physical Systems (CPS) can be challenging, since CPS models typically involve time-varying and real-valued variables, physical system dynamics, or even adaptive behavior. MATLAB/Simulinkis a development and simulation framework that is widely used in industry to capture such systems. In this paper, we report on the application of NASA Ames tools to perform end-to-end analysis of the Ten Lockheed Martin Challenge Problems (LMCPS). LMCPS is a set of industrial Simulink model benchmarks and natural language requirements developed by domain experts. Our framework, which integrates the tools FRET and COCOSIM, is used to: 1) elicit, explain, and formalize the semantics of the given natural language requirements; 2) generate verification code and monitors that can be automatically attached to the Simulink models; 3) perform verification by using SMT-based model checkers. FRET and COCOSIM are open source, and can be used by other researchers and practitioners to replicate our case study. We provide a categorization of recurring patterns in the formalization of the requirements and discuss the strengths and weaknesses of our automated verification approach.

Anastasia Mavridou↗
NASA NTRS2024

Ensuring Safe Decision-Making on the Moon and Mars: Cognitive Performance Assessment for Exploration Class Mission EVA

Extravehicular activity (EVA) is one of the most dangerous and cognitively demanding actions that astronauts can execute, and the cognitive demands associated with future partial gravity EVA on the Moon and Mars are expected to be higher compared to microgravity EVA currently conducted from the International Space Station (ISS). Decrements in cognitive performance present an important risk to crew safety during exploration mission class EVA. Yet there is currently insufficient data to characterize cognitive performance prior to, during, and following EVA. Furthermore, it is still unclear which cognitive domains are most important for conducting mission critical decisions with crew safety implications. To address this gap, we conducted a cognitive task analysis (CTA) of EVA to characterize the procedures, the cognitive demands required, and the critical safety decisions associated with decrements in cognitive performance. We used the applied cognitive task analysis method as outlined by Militello and Hutton (1998) and conducted interviews with subject matter experts (SMEs) in EVA at NASA Johnson Space Center in summer 2024. Interviews focused on exploration EVA and elicited feedback from SMEs on the cognitive skills required for specific EVA tasks including cognitive strategies, critical cues, and decision-making strategies. A cognitive demands table was assembled to consolidate and synthesize the information from all interviews. The information from this cognitive task analysis will inform ongoing work to identify the key cognitive domains for safe decision-making during exploration mission class EVA on the Moon and Mars. This work will importantly provide a link between cognitive domains identified in the cognitive task analysis and operationally-relevant performance metrics collected during spaceflight and simulation EVA that can be tracked and assessed by NASA mission operations. Ultimately, this work will aid in the development and application of exploration class cognitive performance monitoring capabilities to improve safe decision-making and inform EVA planning on future spaceflight missions to the Moon and Mars.

Steven R Anderson↗
NASA NTRS1991

Expert system training and control based on the fuzzy relation matrix

Fuzzy knowledge, that for which the terms of reference are not crisp but overlapped, seems to characterize human expertise. This can be shown from the fact that an experienced human operator can control some complex plants better than a computer can. Proposed here is fuzzy theory to build a fuzzy expert relation matrix (FERM) from given rules or/and examples, either in linguistic terms or in numerical values to mimic human processes of perception and decision making. The knowledge base is codified in terms of many implicit fuzzy rules. Fuzzy knowledge thus codified may also be compared with explicit rules specified by a human expert. It can also provide a basis for modeling the human operator and allow comparison of what a human operator says to what he does in practice. Two experiments were performed. In the first, control of liquid in a tank, demonstrates how the FERM knowledge base is elicited and trained. The other shows how to use a FERM, build up from linguistic rules, and to control an inverted pendulum without a dynamic model.

Ren, Jie↗
NASA NTRSDate not supplied

Spacecraft Line-Of-Sight Jitter Mitigation and Management Lessons Learned and Engineering Best Practices

Predicting, managing, controlling, and testing spacecraft line-of-sight (LoS) jitter caused by micro-vibrations due to onboard internal disturbance sources is a formidable multidisciplinary engineering task. It is especially challenging for those missions hosting high-performance, vibration-sensitive optical sensor payloads with stringent pointing stability requirements. Both NASA and ESA are planning technically aggressive spaceflight missions that include ultra-high-performance optical payloads with delicate, highly vibration-sensitive scientific and observational instruments. The GN&C community of practice will need to leverage and build upon their collective experiences and lessons learned to better address future micro-vibration challenges. To identify lessons learned and best engineering practices the NASA Engineering & Safety Center (NESC) sponsored a two-day Spacecraft LoS Jitter Workshop in late 2019. The workshop’s goal was to provide a multidisciplinary forum to elicit deeper understanding of the issues related to solving the spacecraft LoS jitter/micro-vibration problem. A primary objective was to identify and share best practices, rules of thumb, and options for jitter-related activities. Representatives from NASA, JPL, ESA, along with NASA’s industrial partners, independent consultant subject matter experts, and members of academia participated in the workshop. This paper will describe the motivation for the NASA Spacecraft LoS Jitter Workshop and summarize the identified findings, observations, and recommendations.

Lessons Learned;↗
NASA NTRS2021

Spacecraft Line-of-Sight Jitter Management and Mitigation Lessons Learned and Engineering Best Practices

Predicting, managing, controlling, and testing spacecraft line-of-sight (LoS) jitter caused by micro-vibrations due to on-board internal disturbance sources is a formidable multidisciplinary engineering task. It is especially challenging for those missions hosting high-performance (e.g., nano-radian/milli-arcsecond class), vibration-sensitive optical sensor payloads with stringent pointing stability requirements. The Nation Aeronautics and Space Administration (NASA) and the European Space Agency (ESA) are planning technically aggressive spaceflight missions that include ultra-high-performance optical payloads with delicate, highly vibration-sensitive scientific and observational instruments. The guidance, navigation, and control community of practice will need to leverage collective experiences and document their best practices and lessons learned to address future micro-vibration challenges. To identify lessons learned and best practices the NASA Engineering & Safety Center sponsored a 2-day Spacecraft LoS Jitter Workshop in late 2019. The workshop’s goal was to provide a multidisciplinary forum to elicit deeper understanding of the issues related to addressing the spacecraft LoS jitter/micro-vibration problem. The primary objective was to identify, document, and share lessons learned, best practices, and preferred options for jitter-related analysis and test activities. Representatives from NASA, ESA, along with NASA’s industrial partners, independent consultant subject matter experts, and members of academia participated in the workshop. This paper describes the motivation for the workshop and summarize the identified findings and recommendations.

Lessons Learned↗
NASA NTRS2021

Spacecraft Line-Of-Sight Jitter Management and Mitigation Lessons Learned And Engineering Best Practices

Predicting, managing, controlling, and testing spacecraft line-of-sight (LoS) jitter caused by micro-vibrations due to on-board internal disturbance sources is a formidable multidisciplinary engineering task. It is especially challenging for those missions hosting high-performance (e.g., nano-radian/milli-arcsecond class), vibration-sensitive optical sensor payloads with stringent pointing stability requirements. The Nation Aeronautics and Space Administration (NASA) and the European Space Agency (ESA) are planning technically aggressive spaceflight missions that include ultra-high-performance optical payloads with delicate, highly vibration-sensitive scientific and observational instruments. The guidance, navigation, and control community of practice will need to leverage collective experiences and document their best practices and lessons learned to address future micro-vibration challenges. To identify lessons learned and best practices the NASA Engineering & Safety Center sponsored a 2-day Spacecraft LoS Jitter Workshop in late 2019. The workshop’s goal was to provide a multidisciplinary forum to elicit deeper understanding of the issues related to addressing the spacecraft LoS jitter/micro-vibration problem. The primary objective was to identify, document, and share lessons learned, best practices, and preferred options for jitter-related analysis and test activities. Representatives from NASA, ESA, along with NASA’s industrial partners, independent consultant subject matter experts, and members of academia participated in the workshop. This paper will describe the motivation for the workshop and summarize the identified findings and recommendations.

Swanson, Davin K.↗
NASA NTRS2016

Engaging Students, Teachers, and the Public with NASA Astromaterials Research and Exploration Science (ARES) Assets

Engaging students, teachers, and the public with NASA Astromaterials Research and Exploration Science (ARES) assets, including Science, Technology, Engineering and Mathematics (STEM) experts and NASA curation astromaterial samples, provides an extraordinary opportunity to connect citizens with authentic aspects unique to our nation's space program. Effective engagement can occur through both virtual connections such as webcasts and in-person connections at educator workshops and public outreach events. Access to NASA ARES assets combined with adaptable resources and techniques that engage and promote scientific thinking helps translate the science and research being facilitated through NASA exploration, elicits a curiosity that aims to carry over even after a given engagement, and prepares our next generation of scientific explorers.

Graff, P. V.↗
NASA NTRS1988

Knowledge elicitation for an operator assistant system in process control tasks

A knowledge based system (KBS) methodology designed to study human machine interactions and levels of autonomy in allocation of process control tasks is presented. Users are provided with operation manuals to assist them in normal and abnormal situations. Unfortunately, operation manuals usually represent only the functioning logic of the system to be controlled. The user logic is often totally different. A method is focused on which illicits user logic to refine a KBS shell called an Operator Assistant (OA). If the OA is to help the user, it is necessary to know what level of autonomy gives the optimal performance of the overall man-machine system. For example, for diagnoses that must be carried out carefully by both the user and the OA, interactions are frequent, and processing is mostly sequential. Other diagnoses can be automated, in which the case the OA must be able to explain its reasoning in an appropriate level of detail. OA structure was used to design a working KBS called HORSES (Human Orbital Refueling System Expert System). Protocol analysis of pilots interacting with this system reveals that the a-priori analytical knowledge becomes more structured with training and the situation patterns more complex and dynamic. This approach can improve the a-priori understanding of human and automatic reasoning.

Boy, Guy A.↗
NASA NTRSDate not supplied

HAT m:N Cognitive Task Analysis (CTA)

This Cognitive Task Analysis (CTA) study was designed to understand the capability of the m:N Tactical Operator (TO) interfaces developed by the Human-Autonomy Teaming Laboratory at NASA Ames to support operators responsible for simplified pilot operations of 100 independently operated small UAS (sUAS) in a constrained geographic area. The m:N sUAS TO interface includes a central Tactical Situation Display (TSD) digital map with moving icons reflecting the sUAS location and planned flight route. The interface also has two side panels. The left panel includes a tabular list of UAS assets and mission tasking, a list of recently viewed assets, and a list of events and alerts. The right panel includes a tabular list of UAS assets and their associated telemetry, text-based chat communication window, and a tabbed checklist window. This CTA was adapted from the incident-based applied cognitive task analysis (Militello & Hutton, 1998) and included demographics questions, scenario-based simulations, a task diagram and knowledge audit methods. In addition to examining the support provided by this m:N sUAS TO interface, this CTA study, conducted with aviation subject matter experts in analogous roles to the future tactical operator, was designed to illuminate and project likely cognitive requirements of the tactical operator. Interviewees participated in two scenario-based simulations using the m:N sUAS TO interfaces. In the first simulation, the interviewees supervised 12 sUAS operating in downtown San Diego, California transiting to and from a central sUAS Hive, restaurant locations, and customer drop off locations. Interviewees were asked to react to a UAS Volume Reservation (UVR) event with a two-phase impact on food delivery operations. In the second scenario, the interviewees supervised 100 sUAS operating in the same airspace and with the same mission. Interviewees used the interfaces to recognize and react to two sUAS air vehicle problems. After each scenario, we asked the interviewees a semi-structured list of questions to elicit their reflections about using the interfaces. Interviewees were confident in their ability to respond to two off-nominal situations in each simulation. Interviewees felt that, given high levels of automation on the sUAS, they would be able to manage the events without requiring additional support or handing off the sUAS to a colleague or supervisor in both the n=12 or n=100 sUAS settings. In the n=12 sUAS condition, interviewees used the center map to understand the asset location and progress along with mission tasking. An additional display window, Asset Telemetry, helped interviewees understand battery state and sUAS altitude. When the number of sUAS increased, interviewees altered their behavior. Rather than maintaining awareness of individual assets, interviewees appeared to become more reactive, managing exceptions. Interviewees reported that they spent less time looking at the nominal aircraft, and focused their attention primarily on the off-nominal aircraft. In addition, in the second simulation with n=100 sUAS, interviewees reported that they relied more on the side panels (Mission Timeline and Asset Telemetry) to gather information.

human-autonomy teaming↗