Open Source Core Flight System (cFS) Flight Software (FSW) Verification & Validation (V&V) Final Summary: IV&V Analysis Technical Report
Explore the source record for details and available documents.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Loss of control (LOC) remains one of the largest contributors to aircraft fatal accidents worldwide. Aircraft LOC accidents are highly complex in that they can result from numerous causal and contributing factors acting alone or more often in combination. Hence, there is no single intervention strategy to prevent these accidents. Research is underway at the National Aeronautics and Space Administration (NASA) in the development of advanced onboard system technologies for preventing or recovering from loss of vehicle control and for assuring safe operation under off-nominal conditions associated with aircraft LOC accidents. The transition of these technologies into the commercial fleet will require their extensive validation and verification (V&V) and ultimate certification. The V&V of complex integrated systems poses highly significant technical challenges and is the subject of a parallel research effort at NASA. This chapter summarizes the V&V problem and presents a proposed process that could be applied to complex integrated safety-critical systems developed for preventing aircraft LOC accidents. A summary of recent research accomplishments in this effort is referenced.
No abstract available
The Orbital Debris Engineering Model (ORDEM) developed by the NASA Orbital Debris Program Office (ODPO) is a data-driven model — extensive radar, optical, laboratory, and in situ measurement data sets have been used to build the model since its earliest versions. A salient aspect of professional software development is the verification and validation (V&V) process. Verification answers the question “Is the model built correctly?” while validation addresses the question “Did we build the correct model?” Less extensive, reserved, or independent data sets serve the validation requirement. Due to the dynamic nature of the orbital debris environment, it is critical to use contemporaneous data sources that represent the current environment to support ORDEM development and validation. ORDEM has utilized in situ data collected from Space Shuttle and Hubble Space Telescope surface inspections, now over a decade old. This historical dataset is fundamental for providing baseline in situ measurement data for sizes between 10 to 300 microns, but new data sources are being evaluated using returned surfaces from or near the International Space Station (ISS). This paper reviews a general microscopic survey of ISS soft goods, the Pressurized Mating Adapter 2 (PMA-2) blanket, and a limited-scope feasibility study conducted on the Space Exploration Technologies Corporation (SpaceX) Dragon capsule’s Thermal Protection System (TPS) material. The PMA-2 blanket, exposed to the space environment between 09 July 2013 and 25 February 2015, is an approximately 3.7 m2-area blanket composed of a betacloth outer layer and multiple ballistic fabric inner layers. The SpaceX Cargo Dragon capsule regularly visited the ISS from 2012 through 2020 and potentially provides a timely and well-characterized source of data for modeling purposes. The capsule’s lateral surfaces use SpaceX Proprietary Ablative Material (SPAM) TPS material, a syntactic foam, for thermal management during all mission phases. Seven SPAM extracted samples have been analyzed to date. This paper will provide an overview of the characterization completed for impact features by size, depth, impactor diameter, and the impactor residues chemical analyses, allowing a differentiation between micrometeoroids and orbital debris and a categorization by mass density and density class. Impactor diameter is estimated using damage equations generated from ground-based hypervelocity impact testing. The orbital debris impactors are compared to the current ORDEM 3.2 model of the environment at ISS altitudes. We briefly discuss the meteoroid impactors, including constituents and mass densities, in the general context of current models.
The Orbital Debris Engineering Model (ORDEM) developed by the NASA Orbital Debris Program Office (ODPO) is a data-driven model — extensive radar, optical, laboratory, and in situ measurement data sets have been used to build the model since its earliest versions. A salient aspect of professional software development is the verification and validation (V&V) process. Verification answers the question “Is the model built correctly?” while validation addresses the question “Did we build the correct model?” Less extensive, reserved, or independent data sets serve the validation requirement. Due to the dynamic nature of the orbital debris environment, it is critical to use contemporaneous data sources that represent the current environment to support ORDEM development and validation. ORDEM has utilized in situ data collected from Space Shuttle and Hubble Space Telescope surface inspections, now over a decade old. This historical dataset is fundamental for providing baseline in situ measurement data for sizes between 10 to 300 microns, but new data sources are being evaluated using returned surfaces from or near the International Space Station (ISS). This paper reviews a general microscopic survey of ISS soft goods, the Pressurized Mating Adapter 2 (PMA-2) blanket, and a limited-scope feasibility study conducted on the Space Exploration Technologies Corporation (SpaceX) Dragon capsule’s Thermal Protection System (TPS) material. The PMA-2 blanket, exposed to the space environment between 09 July 2013 and 25 February 2015, is an approximately 3.7 m2-area blanket composed of a betacloth outer layer and multiple ballistic fabric inner layers. The SpaceX Cargo Dragon capsule regularly visited the ISS from 2012 through 2020 and potentially provides a timely and well-characterized source of data for modeling purposes. The capsule’s lateral surfaces use SpaceX Proprietary Ablative Material (SPAM) TPS material, a syntactic foam, for thermal management during all mission phases. Seven SPAM extracted samples have been analyzed to date. This paper will provide an overview of the characterization completed for impact features by size, depth, impactor diameter, and the impactor residues chemical analyses, allowing a differentiation between micrometeoroids and orbital debris and a categorization by mass density and density class. Impactor diameter is estimated using damage equations generated from ground-based hypervelocity impact testing. The orbital debris impactors are compared to the current ORDEM 3.2 model of the environment at ISS altitudes. We briefly discuss the meteoroid impactors, including constituents and mass densities, in the general context of current models.
The present talk will provide an overview of the work performed during the Advanced Composites Project (ACP) on the development, and verification and validation of the Floating Node Method (FNM) as well as the Virtual Crack Closure Technique (VCCT) and cohesive zone modeling (CZM). The FNM is a finite element-based technique to represent crack networks. The complex nature of matrix crack-delamination interactions observed in unidirectional (UD) tape laminates suggests that a methodology such as the FNM method may be required to be able to accurately simulate damage progression in these laminates. Simulating crack onset and growth within the context of the FNM relies on techniques such as VCCT and CZM. The talk is organized to provide, via select examples, an overview of the breadth of the Verification & Validation (V&V) exercises performed during the ACP, and how these challenged the state-of-the art and guided further developments in discrete crack modeling, while helping to establish confidence in the progress made and map the challenges ahead. The performance of the VCCT and CZM individually, and in combination with the FNM can be assessed through verification exercises. These exercises typically consist of a comparison of simulation results to known numerical or analytical solutions. Verification is key to identify implementation issues and limitations that, otherwise, may remain undetected and cloud any subsequent validation efforts. Indeed, a subset of these numerical exercises led to further developments of the VCCT and the FNM method as will be illustrated. Before embarking on the subsequent validation of the framework, it is critical to have adequate characterization data. However, the testing campaign conducted revealed material responses that challenged the state-of-the-art and required further developments. The developments in CZM technology associated with the modeling of the responses of hybrid interfaces (fabric/UD) will be given as an example. Finally, the talk will conclude with a summary of the validation exercises performed under quasi-static and fatigue loadings, highlighting some of the key achievements, outstanding challenges and lessons learned.
Recent activities at the Remote Sensing Program at Stennis Space Center have identified the need to properly verify and validate data provided by the remote sensing community. One important variable, which effects remote sensing data is bi-directional reflectance distribution (BRDF). In order to quantify the effects of BRDF on man-made and natural ground targets, the Stennis Verification & Validation (V&V) team commissioned the Systems Engineering Division at NASA Ames Research Center to develop a Field Goniometer for use at the V&V Large Target Range and for various ground truthing missions. The Swiss Field Goniometer (FIGOS) was used as a benchmark instrument to design the new state of the art Sandmeier Field Goniometer (SGF), named after Stefan Sandmeier, developer of FIGOS. After establishing requirements for the SFG, design efforts began in early May 1998. The design of the SFG was completed in September 1998. Manufacturing, construction, and testing was completed in May 1999. The SFG was shipped to NASA SSC and fully operational by June 1999.
Scientists with NASA's Earth Science Applications Directorate are creating a well-characterized Verification & Validation (V&V) site at the Stennis Space Center (SSC). This site enables the in-flight characterization of remote sensing systems and the data that they require. The data are predominantly acquired by commercial, high-spatial resolution satellite systems, such as IKONOS and QuickBird 2, and airborne systems. The smaller scale of these newer high-resolution remote sensing systems allows scientists to characterize the geometric, spatial, and radiometric data properties using a single V&V site. The targets and techniques used to characterize data from these newer systems can differ significantly from the earlier, coarser spatial resolution systems. Scientists are also using the SSC V&V site to characterize thermal infrared systems and active Light Detection and Ranging (LIDAR) systems. SSC employs geodetic targets, edge targets, radiometric tarps, and thermal calibration ponds to characterize remote sensing data products. This paper presents a proposed set of required measurements for visible-through-longwave infrared remote sensing systems, and a description of the Stennis characterization. Other topics discussed inslude: 1) use of ancillary atmospheric and solar measurements taken at SSC that support various characterizations, 2) other sites used for radiometric, geometric, and spatial characterization in the continental United States,a nd 3) the need for a standardized technique to be adopted by the Committee on Earth Observation Satellites (CEOS) and other organizations.
Scientists at NASA's Earth Science Applications Directorate are creating a well-characterized Verification & Validation (V&V) site at the Stennis Space Center. This site enables the in-flight characterization of remote sensing systems and the data they acquire. The data are predominantly acquired by commercial, high spatial resolution satellite systems, such as IKONOS and QuickBird 2, and airborne systems. The smaller scale of these newer high resolution remote sensing systems allows scientists to characterize the geometric, spatial, and radiometric data properties using a single V&V site. The targets and techniques used to characterize data from these newer systems can differ significantly from the techniques used to characterize data from the earlier, coarser spatial resolution systems. Scientists are also using the SSC V&V site to characterize thermal infrared systems and active LIDAR systems. SSC employs geodetic targets, edge targets, radiometric tarps, and thermal calibration ponds to characterize remote sensing data products. This paper presents a proposed set of required measurements for visible through long-wave infrared remote sensing systems and a description of the Stennis characterization. Other topics discussed include: 1) The use of ancillary atmospheric and solar measurements taken at SSC that support various characterizations; 2) Additional sites used for radiometric, geometric, and spatial characterization in the continental United States; 3) The need for a standardized technique to be adopted by CEOS and other organizations.
Modern exploration missions require modern control systems-control systems that can handle catastrophic changes in the system's behavior, compensate for slow deterioration in sustained operations, and support fast system ID. Adaptive controllers, based upon Neural Networks have these capabilities, but they can only be used safely if proper verification & validation (V&V) can be done. In this paper we present our V & V approach and simulation result within NASA's Intelligent Flight Control Systems (IFCS).
Scientists within NASA's Applied Sciences Directorate have developed a well-characterized remote sensing Verification & Validation (V&V) site at the John C. Stennis Space Center (SSC). This site has enabled the in-flight characterization of satellite high spatial resolution remote sensing system products form Space Imaging IKONOS, Digital Globe QuickBird, and ORBIMAGE OrbView, as well as advanced multispectral airborne digital camera products. SSC utilizes engineered geodetic targets, edge targets, radiometric tarps, atmospheric monitoring equipment and their Instrument Validation Laboratory to characterize high spatial resolution remote sensing data products. This presentation describes the SSC characterization capabilities and techniques in the visible through near infrared spectrum and examples of calibration results.
Scientists within NASA s Applied Sciences Directorate have developed a well-characterized remote sensing Verification & Validation (V&V) site at the John C. Stennis Space Center (SSC). This site enables the in-flight characterization of satellite and airborne high spatial and moderate resolution remote sensing systems and their products. The smaller scale of the newer high resolution remote sensing systems allows scientists to characterize geometric, spatial, and radiometric data properties using a single V&V site. The targets and techniques used to characterize data from these newer systems can differ significantly from the techniques used to characterize data from the earlier, coarser spatial resolution systems. Scientists are also using the SSC V&V site to characterize thermal infrared systems and active lidar systems. SSC employs geodetic targets, edge targets, radiometric tarps, atmospheric monitoring equipment, and thermal calibration ponds to characterize remote sensing data products. The SSC Instrument Validation Lab is a key component of the V&V capability and is used to calibrate field instrumentation and to provide National Institute of Standards and Technology traceability. This poster presents a description of the SSC characterization capabilities and examples of calibration data.
Scientists within NASA#s Applied Research & Technology Project Office (formerly the Applied Sciences Directorate) have developed a well-characterized remote sensing Verification & Validation (V&V) site at the John C. Stennis Space Center (SSC). This site enables the in-flight characterization of satellite and airborne high spatial resolution remote sensing systems and their products. The smaller scale of the newer high resolution remote sensing systems allows scientists to characterize geometric, spatial, and radiometric data properties using a single V&V site. The targets and techniques used to characterize data from these newer systems can differ significantly from the techniques used to characterize data from the earlier, coarser spatial resolution systems. Scientists have used the SSC V&V site to characterize thermal infrared systems. Enhancements are being considered to characterize active lidar systems. SSC employs geodetic targets, edge targets, radiometric tarps, atmospheric monitoring equipment, and thermal calibration ponds to characterize remote sensing data products. Similar techniques are used to characterize moderate spatial resolution sensing systems at selected nearby locations. The SSC Instrument Validation Lab is a key component of the V&V capability and is used to calibrate field instrumentation and to provide National Institute of Standards and Technology traceability. This poster presents a description of the SSC characterization capabilities and examples of calibration data.
The NASA Protoflight Research Initiative is an internal NASA study conducted within the Office of the Chief Engineer to better understand the use of Protoflight within NASA. Extensive literature reviews and interviews with key NASA members with experience in both robotic and human spaceflight missions has resulted in three main conclusions and two observations. The first conclusion is that NASA's Protoflight method is not considered to be "prescriptive." The current policies and guidance allows each Program/Project to tailor the Protoflight approach to better meet their needs, goals and objectives. Second, Risk Management plays a key role in implementation of the Protoflight approach. Any deviations from full qualification will be based on the level of acceptable risk with guidance found in NPR 8705.4. Finally, over the past decade (2004 - 2014) only 6% of NASA's Protoflight missions and 6% of NASA's Full qualification missions experienced a publicly disclosed mission failure. In other words, the data indicates that the Protoflight approach, in and of it itself, does not increase the mission risk of in-flight failure. The first observation is that it would be beneficial to document the decision making process on the implementation and use of Protoflight. The second observation is that If a Project/Program chooses to use the Protoflight approach with relevant heritage, it is extremely important that the Program/Project Manager ensures that the current project's requirements falls within the heritage design, component, instrument and/or subsystem's requirements for both the planned and operational use, and that the documentation of the relevant heritage is comprehensive, sufficient and the decision well documented. To further benefit/inform this study, a recommendation to perform a deep dive into 30 missions with accessible data on their testing/verification methodology and decision process to research the differences between Protoflight and Full Qualification missions' Design Requirements and Verification & Validation (V&V) (without any impact or special request directly to the project).
While NASA’s Mars rover Perseverance continues to make groundbreaking achievements on the Red Planet, its twin is hard at work here on Earth. The Operational Perseverance Twin for the Integration of Mechanisms and Instruments Sent to Mars, or OPTIMISM, is the Mars 2020 Vehicle System Testbed (VSTB) rover operated by NASA Jet Propulsion Laboratory (JPL) in Pasadena, California. OPTIMISM’s home is the JPL Mars Yard; an outdoor field with red soil that simulates the terrain encountered by Perseverance. The VSTB is a full-scale engineering model of the flight rover, serving a number of functions to ensure mission operations can continue smoothly and on schedule. The VSTB possesses instrumentation, computers, mechanisms, cameras, and a Mobility subsystem that are nearly identical to its extraterrestrial twin. Its high fidelity allows the rover to be a highly effective tool to fully test system functionality and performance prior to commanding the flight rover. The early stages of building OPTIMISM began a few months prior to Perseverance departing JPL for Cape Canaveral, FL in early 2020. Electrical integration of the flight system avionics, and compatibility checkouts of the electrical ground support equipment ensured that the foundation of the electrical system was operational and in place. Next, the internal harnessing was installed and compatibility checks of the rover instrumentation and mechanisms were performed to confirm the system was prepared for full buildup. Finally, mechanical assembly of the rover chassis with its external components completed the integration of the system before it was moved to the Mars Yard for its initial phase of testing to perform verification & validation (V&V) of the Mobility subsystem requirements. By the time Perseverance landed at Jezero Crater in February 2021, the first phase of VSTB operations was underway. Surface guidance, navigation, and control (SGNC) testing for the Mobility subsystem ensured functionality and performance requirements were met for various capabilities such as visual odometry (VO), mapping, and automatic navigation (AutoNav). Subsequent integration of the robotic arm (RA) onto the VSTB enabled the V&V campaign for surface sampling operations (SSO) to commence. As the mission’s engineering operations (EO) have gotten underway, the VSTB has been utilized for an array of purposes including troubleshooting software anomalies, and performing dry-runs for first time activities (FTAs) prior to sending the commands to Perseverance. OPTIMISM will continue to serve mission critical functions as long as Perseverance is roving the Red Planet.
The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.
The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.
Software independent verification and validation (IV and V) implementation methods, specific approaches and accomplishments in the application of IV and V to NASA programs are discussed. Because software is mission critical, software assurance is a critical aspect of NASA's safety and mission assurance program. An overview of NASA's software IV and V facility is given.