Software development cost estimation
Explore the source record for details and available documents.
cost risk↗
SEARCH · Engineering Papers
Results for “software risk”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Risk Assessment System
FEAT, a software system for evaluating risks, was developed by Lockheed and later enhanced under NASA funding. FEAT uses directed graph - or digraph - models to provide information on cause and effect if a set of failure events occurs. James Miller, the program designer at Lockheed, formed DiGraphics, Inc. to market the software that has evolved from FEAT. The Diquest Analyzer, the company's flagship product, assists product designers in identifying the redundancies and weaknesses of a system. The software has applications in the chemical industry for risk assessment, design evaluation, and change management. Additional markets have been found in operations monitoring diagnostics and training of new personnel.
Source record↗
Obtaining Valid Safety Data for Software Safety Measurement and Process Improvement
We report on a preliminary case study to examine software safety risk in the early design phase of the NASA Constellation spaceflight program. Our goal is to provide NASA quality assurance managers with information regarding the ongoing state of software safety across the program. We examined 154 hazard reports created during the preliminary design phase of three major flight hardware systems within the Constellation program. Our purpose was two-fold: 1) to quantify the relative importance of software with respect to system safety; and 2) to identify potential risks due to incorrect application of the safety process, deficiencies in the safety process, or the lack of a defined process. One early outcome of this work was to show that there are structural deficiencies in collecting valid safety data that make software safety different from hardware safety. In our conclusions we present some of these deficiencies.
Basili, Victor r.↗
Artificial intelligence approaches to software engineering
Artificial intelligence approaches to software engineering are examined. The software development life cycle is a sequence of not so well-defined phases. Improved techniques for developing systems have been formulated over the past 15 years, but pressure continues to attempt to reduce current costs. Software development technology seems to be standing still. The primary objective of the knowledge-based approach to software development presented in this paper is to avoid problem areas that lead to schedule slippages, cost overruns, or software products that fall short of their desired goals. Identifying and resolving software problems early, often in the phase in which they first occur, has been shown to contribute significantly to reducing risks in software development. Software development is not a mechanical process but a basic human activity. It requires clear thinking, work, and rework to be successful. The artificial intelligence approaches to software engineering presented support the software development life cycle through the use of software development techniques and methodologies in terms of changing current practices and methods. These should be replaced by better techniques that that improve the process of of software development and the quality of the resulting products. The software development process can be structured into well-defined steps, of which the interfaces are standardized, supported and checked by automated procedures that provide error detection, production of the documentation and ultimately support the actual design of complex programs.
Johannes, James D.↗
Using Dissimilarity Metrics to Identify Interesting Designs
A computer program helps to blend the power of automated-search software, which is able to generate large numbers of design solutions, with the insight of expert designers, who are able to identify preferred designs but do not have time to examine all the solutions. From among the many automated solutions to a given design problem, the program selects a smaller number of solutions that are worthy of scrutiny by the experts in the sense that they are sufficiently dissimilar from each other. The program makes the selection in an interactive process that involves a sequence of data-mining steps interspersed with visual displays of results of these steps to the experts. At crucial points between steps, the experts provide directives to guide the process. The program uses heuristic search techniques to identify nearly optimal design solutions and uses dissimilarity metrics defined by the experts to characterize the degree to which solutions are interestingly different. The search, data-mining, and visualization features of the program were derived from previously developed risk-management software used to support a risk-centric design methodology
Feather, Martin↗
Enigma Version 12
Enigma Version 12 software combines model building, animation, and engineering visualization into one concise software package. Enigma employs a versatile user interface to allow average users access to even the most complex pieces of the application. Using Enigma eliminates the need to buy and learn several software packages to create an engineering visualization. Models can be created and/or modified within Enigma down to the polygon level. Textures and materials can be applied for additional realism. Within Enigma, these models can be combined to create systems of models that have a hierarchical relationship to one another, such as a robotic arm. Then these systems can be animated within the program or controlled by an external application programming interface (API). In addition, Enigma provides the ability to use plug-ins. Plugins allow the user to create custom code for a specific application and access the Enigma model and system data, but still use the Enigma drawing functionality. CAD files can be imported into Enigma and combined to create systems of computer graphics models that can be manipulated with constraints. An API is available so that an engineer can write a simulation and drive the computer graphics models with no knowledge of computer graphics. An animation editor allows an engineer to set up sequences of animations generated by simulations or by conceptual trajectories in order to record these to highquality media for presentation. Enigma Version 12 Lyndon B. Johnson Space Center, Houston, Texas 28 NASA Tech Briefs, September 2013 Planetary Protection Bioburden Analysis Program NASA's Jet Propulsion Laboratory, Pasadena, California This program is a Microsoft Access program that performed statistical analysis of the colony counts from assays performed on the Mars Science Laboratory (MSL) spacecraft to determine the bioburden density, 3-sigma biodensity, and the total bioburdens required for the MSL prelaunch reports. It also contains numerous tools that report the data in various ways to simplify the reports required. The program performs all the calculations directly in the MS Access program. Prior to this development, the data was exported to large Excel files that had to be cut and pasted to provide the desired results. The program contains a main menu and a number of submenus. Analyses can be performed by using either all the assays, or only the accountable assays that will be used in the final analysis. There are three options on the first menu: either calculate using (1) the old MER (Mars Exploration Rover) statistics, (2) the MSL statistics for all the assays, or This software implements penetration limit equations for common micrometeoroid and orbital debris (MMOD) shield configurations, windows, and thermal protection systems. Allowable MMOD risk is formulated in terms of the probability of penetration (PNP) of the spacecraft pressure hull. For calculating the risk, spacecraft geometry models, mission profiles, debris environment models, and penetration limit equations for installed shielding configurations are required. Risk assessment software such as NASA's BUMPERII is used to calculate mission PNP; however, they are unsuitable for use in shield design and preliminary analysis studies. The software defines a single equation for the design and performance evaluation of common MMOD shielding configurations, windows, and thermal protection systems, along with a description of their validity range and guidelines for their application. Recommendations are based on preliminary reviews of fundamental assumptions, and accuracy in predicting experimental impact test results. The software is programmed in Visual Basic for Applications for installation as a simple add-in for Microsoft Excel. The user is directed to a graphical user interface (GUI) that requires user inputs and provides solutions directly in Microsoft Excel workbooks. This work was done by Shannon Ryan of the USRA Lunar and Planetary Institute for Johnson Space Center. Further information is contained in a TSP (see page 1). MSC- 24582-1 Micrometeoroid and Orbital Debris (MMOD) Shield Ballistic Limit Analysis Program Lyndon B. Johnson Space Center, Houston, Texas Commercially, because it is so generic, Enigma can be used for almost any project that requires engineering visualization, model building, or animation. Models in Enigma can be exported to many other formats for use in other applications as well. Educationally, Enigma is being used to allow university students to visualize robotic algorithms in a simulation mode before using them with actual hardware.
Shores, David↗
Micrometeoroid and Orbital Debris (MMOD) Shield Ballistic Limit Analysis Program
This software implements penetration limit equations for common micrometeoroid and orbital debris (MMOD) shield configurations, windows, and thermal protection systems. Allowable MMOD risk is formulated in terms of the probability of penetration (PNP) of the spacecraft pressure hull. For calculating the risk, spacecraft geometry models, mission profiles, debris environment models, and penetration limit equations for installed shielding configurations are required. Risk assessment software such as NASA's BUMPERII is used to calculate mission PNP; however, they are unsuitable for use in shield design and preliminary analysis studies. The software defines a single equation for the design and performance evaluation of common MMOD shielding configurations, windows, and thermal protection systems, along with a description of their validity range and guidelines for their application. Recommendations are based on preliminary reviews of fundamental assumptions, and accuracy in predicting experimental impact test results. The software is programmed in Visual Basic for Applications for installation as a simple add-in for Microsoft Excel. The user is directed to a graphical user interface (GUI) that requires user inputs and provides solutions directly in Microsoft Excel workbooks.
Ryan, Shannon↗
Planetary Protection Bioburden Analysis Program
This program is a Microsoft Access program that performed statistical analysis of the colony counts from assays performed on the Mars Science Laboratory (MSL) spacecraft to determine the bioburden density, 3-sigma biodensity, and the total bioburdens required for the MSL prelaunch reports. It also contains numerous tools that report the data in various ways to simplify the reports required. The program performs all the calculations directly in the MS Access program. Prior to this development, the data was exported to large Excel files that had to be cut and pasted to provide the desired results. The program contains a main menu and a number of submenus. Analyses can be performed by using either all the assays, or only the accountable assays that will be used in the final analysis. There are three options on the first menu: either calculate using (1) the old MER (Mars Exploration Rover) statistics, (2) the MSL statistics for all the assays, or This software implements penetration limit equations for common micrometeoroid and orbital debris (MMOD) shield configurations, windows, and thermal protection systems. Allowable MMOD risk is formulated in terms of the probability of penetration (PNP) of the spacecraft pressure hull. For calculating the risk, spacecraft geometry models, mission profiles, debris environment models, and penetration limit equations for installed shielding configurations are required. Risk assessment software such as NASA's BUMPERII is used to calculate mission PNP; however, they are unsuitable for use in shield design and preliminary analysis studies. The software defines a single equation for the design and performance evaluation of common MMOD shielding configurations, windows, and thermal protection systems, along with a description of their validity range and guidelines for their application. Recommendations are based on preliminary reviews of fundamental assumptions, and accuracy in predicting experimental impact test results. The software is programmed in Visual Basic for Applications for installation as a simple add-in for Microsoft Excel. The user is directed to a graphical user interface (GUI) that requires user inputs and provides solutions directly in Microsoft Excel workbooks. This work was done by Shannon Ryan of the USRA Lunar and Planetary Institute for Johnson Space Center. Further information is contained in a TSP (see page 1). MSC- 24582-1 Micrometeoroid and Orbital Debris (MMOD) Shield Ballistic Limit Analysis Program Lyndon B. Johnson Space Center, Houston, Texas Commercially, because it is so generic, Enigma can be used for almost any project that requires engineering visualization, model building, or animation. Models in Enigma can be exported to many other formats for use in other applications as well. Educationally, Enigma is being used to allow university students to visualize robotic algorithms in a simulation mode before using them with actual hardware. This work was done by David Shores and Sharon P. Goza of Johnson Space Center; Cheyenne McKeegan, Rick Easley, Janet Way, and Shonn Everett of MEI Technologies; Mark Manning of PTI; and Mark Guerra, Ray Kraesig, and William Leu of Tietronix Software, Inc. For further information, contact the JSC Innovation Partnerships Office at (281) 483-3809. MSC-24211-1 Spitzer Telemetry Processing System NASA's Jet Propulsion Laboratory, Pasadena, California The Spitzer Telemetry Processing System (SirtfTlmProc) was designed to address objectives of JPL's Multi-mission Image Processing Lab (MIPL) in processing spacecraft telemetry and distributing the resulting data to the science community. To minimize costs and maximize operability, the software design focused on automated error recovery, performance, and information management. The system processes telemetry from the Spitzer spacecraft and delivers Level 0 products to the Spitzer Science Center. SirtfTlmProc is a unique system with automated error notification and recovery, with a real-time continuous service that can go quiescent after periods of inactivity. The software can process 2 GB of telemetry and deliver Level 0 science products to the end user in four hours. It provides analysis tools so the operator can manage the system and troubleshoot problems. It automates telemetry processing in order to reduce staffing costs. This work was done by Alice Stanboli, Elmain M. Martinez, and James M. McAuley of Caltech for NASA's Jet Propulsion Laboratory. For more information, contact iaoffice @jpl.nasa.gov. This software is available for commercial licensing. Please contact Dan Broderick at Daniel.F. Broderick@jpl.nasa.gov. Refer to NPO-47803. NASA Tech Briefs, September 2013 29 This rapid response computer program predicts Orbiter Wing Leading Edge (WLE) damage caused by ice or foam impact during a Space Shuttle launch (Program "IMPACT2"). The program was developed after the Columbia accident in order to assess quickly WLE damage due to ice, foam, or metal impact (if any) during a Shuttle launch. IMPACT2 simulates an impact event in a few minutes for foam impactors, and in seconds for ice and metal impactors. The damage criterion is derived from results obtained from one sophisticated commercial program, which requires hours to carry out simulations of the same impact events. The program was designed to run much faster than the commercial program with prediction of projectile threshold velocities within 10 to 15% of commercial-program values. The mathematical model involves coupling of Orbiter wing normal modes of vibration to nonlinear or linear springmass models. IMPACT2 solves nonlinear or linear impact problems using classical normal modes of vibration of a target, and nonlinear/ linear time-domain equations for the projectile. Impact loads and stresses developed in the target are computed as functions of time. This model is novel because of its speed of execution. A typical model of foam, or other projectile characterized by material nonlinearities, impacting an RCC panel is executed in minutes instead of hours needed by the commercial programs. Target damage due to impact can be assessed quickly, provided that target vibration modes and allowable stress are known. This work was done by Robert Clark, Jr., Paul Cotter, and Constantine Michalopoulos of The Boeing Company for Johnson Space Center. For further information, contact the JSC Innovation Partnerships Office at (281) 483-3809. MSC-24988-1 Wing Leading Edge RCC Rapid Response Damage Prediction Tool (IMPACT2) Lyndon B. Johnson Space Center, Houston, Texas (3) the MSL statistics for only the accountable assays. Other options on the main menu include a data editing form and utility programs that produce various reports requested by the microbiologists and the project, and tools to generate the groupings for the final analyses. The analyses can be carried out in three ways: Each assay can be treated separately, the assays can be collectively treated for the whole zone as a group, or the assays can be collected in groups designated by the JPL Planetary Protection Manager. The latter approach was used to generate the final report because assays on the same equipment or similar equipment can be assumed to have been exposed to the same environment and cleaning. Thus, the statistics are improved by having a larger population, thereby reducing the standard deviation by the square root of N. For each method mentioned above, three reports are available. The first is a detailed report including all the data. This version was very useful in verifying the calculations. The second is a brief report that is similar to the full detailed report, but does not print out the data. The third is a grand total and summary report in which each assay requires only one line. For the first and second reports, most of the calculations are performed in the report section itself. For the third, all the calculations are performed directly in the query bound to the report. All the numerical results were verified by comparing them with Excel templates, then exporting the data from the Planetary Protection Analysis program to Excel.
Beaudet, Robert A.↗
Planning for the V&V of infused software technologies for the Mars Science Laboratory Mission
NASA's Mars Science Laboratory (MSL) rover mission is planning to make use of advanced software technologies in order to support fulfillment of its ambitious science objectives. The mission plans to adopt the Mission Data System (MDS) as the mission software architecture, and plans to make significant use of on-board autonomous capabilities for the rover software.
risk↗
Towards Test Driven Development for Computational Science with pFUnit
Developers working in Computational Science & Engineering (CSE)/High Performance Computing (HPC) must contend with constant change due to advances in computing technology and science. Test Driven Development (TDD) is a methodology that mitigates software development risks due to change at the cost of adding comprehensive and continuous testing to the development process. Testing frameworks tailored for CSE/HPC, like pFUnit, can lower the barriers to such testing, yet CSE software faces unique constraints foreign to the broader software engineering community. Effective testing of numerical software requires a comprehensive suite of oracles, i.e., use cases with known answers, as well as robust estimates for the unavoidable numerical errors associated with implementation with finite-precision arithmetic. At first glance these concerns often seem exceedingly challenging or even insurmountable for real-world scientific applications. However, we argue that this common perception is incorrect and driven by (1) a conflation between model validation and software verification and (2) the general tendency in the scientific community to develop relatively coarse-grained, large procedures that compound numerous algorithmic steps.We believe TDD can be applied routinely to numerical software if developers pursue fine-grained implementations that permit testing, neatly side-stepping concerns about needing nontrivial oracles as well as the accumulation of errors. We present an example of a successful, complex legacy CSE/HPC code whose development process shares some aspects with TDD, which we contrast with current and potential capabilities. A mix of our proposed methodology and framework support should enable everyday use of TDD by CSE-expert developers.
pFUnit↗
Using software security analysis to verify the secure socket layer (SSL) protocol
nal Aeronautics and Space Administration (NASA) have tens of thousands of networked computer systems and applications. Software Security vulnerabilities present risks such as lost or corrupted data, information the3, and unavailability of critical systems. These risks represent potentially enormous costs to NASA. The NASA Code Q research initiative 'Reducing Software Security Risk (RSSR) Trough an Integrated Approach '' offers, among its capabilities, formal verification of software security properties, through the use of model based verification (MBV) to address software security risks. [1,2,3,4,5,6] MBV is a formal approach to software assurance that combines analysis of software, via abstract models, with technology, such as model checkers, that provide automation of the mechanical portions of the analysis process. This paper will discuss: The need for formal analysis to assure software systems with respect to software and why testing alone cannot provide it. The means by which MBV with a Flexible Modeling Framework (FMF) accomplishes the necessary analysis task. An example of FMF style MBV in the verification of properties over the Secure Socket Layer (SSL) communication protocol as a demonstration.
software↗
NASA Conjunction Assessment Risk Analysis (CARA) Ground Systems, Software and Operations Infrastructure Cloud Implementation
The growing number of large-scale constellations and the improvements made in the tracking and detection of resident space objects have contributed to the increase of satellite conjunction events. As a result of the increase of objects in Earth orbit, the timeline for risk analysis computations for satellite collision avoidance is more urgent when conjunctions occur. It is imperative NASA maintains a space environment safety functionality for the agency that allows CARA remote access to the ground system while maintaining redundancy. CARA’s On-premises (On-prem) ground system would no longer be able to meet these access and redundancy requirements, hence the investigation and eventual implementation of a Cloud-based ground station infrastructure as a viable solution. This paper will provide an overview of the principal parts of the Conjunction Assessment (CA) risk analysis process used at CARA, summarize the key drivers towards the Cloud solution, and share key lessons learned that are significant for any operational entity considering a cloud solution for real-time satellite operations to ensure a sustainable space traffic environment for all.
Sustainability↗
Model based verification of the Secure Socket Layer (SSL) Protocol for NASA systems
The National Aeronautics and Space Administration (NASA) has tens of thousands of networked computer systems and applications. Software Security vulnerabilities present risks such as lost or corrupted data, information theft, and unavailability of critical systems. These risks represent potentially enormous costs to NASA. The NASA Code Q research initiative 'Reducing Software Security Risk (RSSR) Trough an Integrated Approach' offers formal verification of information technology (IT), through the creation of a Software Security Assessment Instrument (SSAI), to address software security risks.
software security↗
An improved approach for flight readiness certification: Methodology for failure risk assessment and application examples. Volume 2: Software documentation
An improved methodology for quantitatively evaluating failure risk of spaceflight systems to assess flight readiness and identify risk control measures is presented. This methodology, called Probabilistic Failure Assessment (PFA), combines operating experience from tests and flights with engineering analysis to estimate failure risk. The PFA methodology is of particular value when information on which to base an assessment of failure risk, including test experience and knowledge of parameters used in engineering analyses of failure phenomena, is expensive or difficult to acquire. The PFA methodology is a prescribed statistical structure in which engineering analysis models that characterize failure phenomena are used conjointly with uncertainties about analysis parameters and/or modeling accuracy to estimate failure probability distributions for specific failure modes, These distributions can then be modified, by means of statistical procedures of the PFA methodology, to reflect any test or flight experience. Conventional engineering analysis models currently employed for design of failure prediction are used in this methodology. The PFA methodology is described and examples of its application are presented. Conventional approaches to failure risk evaluation for spaceflight systems are discussed, and the rationale for the approach taken in the PFA methodology is presented. The statistical methods, engineering models, and computer software used in fatigue failure mode applications are thoroughly documented.
Moore, N. R.↗
Characteristics of Whipple Shield Performance in the Shatter Regime
Between the onset of projectile fragmentation and the assumption of rear wall failure due to an impulsive load, multi-wall ballistic limit equations are linearly interpolated to provide reasonable yet conservative predictions of perforation thresholds with conveniently simple mathematics. Although low velocity and hypervelocity regime predictions are based on analytical expressions, there is no such scientific foundation for predictions in the intermediate (or shatter) regime. As the debris flux in low earth orbit (LEO) becomes increasingly dominated by manmade pollution, the profile of micrometeoroid and orbital debris (MMOD) risk shifts continually towards lower velocities. For the International Space Station (ISS), encounter velocities below 7 km/s now constitute approximately 50% of the penetration risk. Considering that the transition velocity from shatter to hypervelocity impact regimes described by common ballistic limit equations (e.g. new non-optimum Whipple shield equation [1]) occurs at 7 km/s, 50% of station risk is now calculated based on failure limit equations with little analytical foundation. To investigate projectile and shield behavior for impact conditions leading to projectile fragmentation and melt, a series of hypervelocity impact tests have been performed on aluminum Whipple shields. In the experiments projectile diameter, bumper thickness, and shield spacing were kept constant, while rear wall thickness was adjusted to determine spallation and perforation limits at various impact velocities and angles. The results, shown in Figure 1 for normal and 45 impacts, demonstrated behavior that was not sufficiently described by the simplified linear interpolation of the NNO equation (also shown in Figure 1). Hopkins et al. [2] investigated the performance of a nominally-identical aluminum Whipple shield, identifying the effects of phase change in the shatter regime. The results (conceptually represented in Figure 2) were found to agree well with those obtained in this study at normal incidence, suggesting that shielding performance in the shatter regime could be well described by considering more complex phase conditions than currently implemented in most BLEs. Furthermore, evidence of these phase effects were found in the oblique test results, providing the basis for an empirical description of these effects that can be applied in MMOD risk assessment software. In this paper, results of the impact experiments are presented, and characteristics of target damage are evaluated. A comparison of intermediate velocity impact failure mechanisms in current BLEs are discussed and compared to the findings of the experimental study. Risk assessment calculations have been made on a simplified structure using currently implemented penetration equations and predicted limits from the experimental program, and the variation in perceived mission risk is discussed. It was found that ballistic limit curves that explicitly incorporated phase change effects within the intermediate regime lead to a decrease in predicted MMOD risk for ISS-representative orbits. When considered for all Whipple-based shielding configurations onboard the ISS, intermediate phase change effects could lead to significant variations in predicted mission risk.
Ryan, Shannon↗
Infusing and selecting V&V activities
The evolving nature of software development poses a continuing series of challenges for V&V. In response, the V&V community selectively adapts the use of existing V&V activities, and introduces new and improved ones.
verification validation risk software quality assu↗
SHINE Virtual Machine Model for In-flight Updates of Critical Mission Software
This software is a new target for the Spacecraft Health Inference Engine (SHINE) knowledge base that compiles a knowledge base to a language called Tiny C - an interpreted version of C that can be embedded on flight processors. This new target allows portions of a running SHINE knowledge base to be updated on a "live" system without needing to halt and restart the containing SHINE application. This enhancement will directly provide this capability without the risk of software validation problems and can also enable complete integration of BEAM and SHINE into a single application. This innovation enables SHINE deployment in domains where autonomy is used during flight-critical applications that require updates. This capability eliminates the need for halting the application and performing potentially serious total system uploads before resuming the application with the loss of system integrity. This software enables additional applications at JPL (microsensors, embedded mission hardware) and increases the marketability of these applications outside of JPL.
Plesea, Lucian↗
Estimating Software Reliability for Space Launch Vehicles in Probabilistic Risk Assessment (PRA)
It is acutely recognized in the Probabilistic Risk assessment (PRA) field that software plays a defining role in overall system reliability for all modern systems across a wide variety of industries. Regardless if the software is embedded firmware for working components or elements, part of a Human-Machine-Interface, or automated command and control logic, the success of the software to fulfill its function under nominal and off-nominal environments will be a dominant contributor to system reliability. It is also recognized that software reliability prediction and estimation is one of the more challenging and questionable aspects of any PRA or system analyses due to the nature of software and its integration with physics based systems. Irrespective of this dichotomy, any incorporation of software reliability methods requires that the contributions are accountable, quantitative, and tractable. This paper provides a brief overview of software reliability methods, establishes some minimum requirements that the methods should incorporate for completeness, and provides a logic structure for applying software reliability. Model resolution will be discussed that supports current testing plans and trade studies. We will provide initial recommendations for use in the NASA PRA and present a future dynamic option for software and PRA. Space Launch Vehicle Software is recognized to be reliable in static conditions, yet relatively vulnerable to a set of failure modes in changing environments/flight phases. Two quantitative methods were chosen to incorporate software reliability into a Space Launch Vehicle PRA accounting for phase adjustments. One method predicts latent software failure using statistical methods, and the second provides estimates of coding errors and software operating system failures based on test and historical data, respectively. Software uncertainty will also be discussed. We determined that recommendations for PRA software reliability should be modeled at the software module level where multiple software components compose a module and combinations of the software architecture can lead to a functional failure.
Novack, Steven↗