A practical software fault measurement and estimation framework
In this paper we present a practical software fault measurement and estimation framework.
software reliability software faults software meas↗
SEARCH · Engineering Papers
Results for “software reliability software faults software measurement software quality”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Practical Issues in Estimating Fault Content and Location in Software Systems
Over the past several years, techniques have been developed to discriminate between fault-prone software modules and those that are not, and to estimate a software system's residual fault content.
software↗
Software Fault Tolerance: A Tutorial
Because of our present inability to produce error-free software, software fault tolerance is and will continue to be an important consideration in software systems. The root cause of software design errors is the complexity of the systems. Compounding the problems in building correct software is the difficulty in assessing the correctness of software for highly complex systems. After a brief overview of the software development processes, we note how hard-to-detect design faults are likely to be introduced during development and how software faults tend to be state-dependent and activated by particular input sequences. Although component reliability is an important quality measure for system level analysis, software reliability is hard to characterize and the use of post-verification reliability estimates remains a controversial issue. For some applications software safety is more important than reliability, and fault tolerance techniques used in those applications are aimed at preventing catastrophes. Single version software fault tolerance techniques discussed include system structuring and closure, atomic actions, inline fault detection, exception handling, and others. Multiversion techniques are based on the assumption that software built differently should fail differently and thus, if one of the redundant versions fails, it is expected that at least one of the other versions will provide an acceptable output. Recovery blocks, N-version programming, and other multiversion techniques are reviewed.
Torres-Pomales, Wilfredo↗
Evaluating software development characteristics: Assessment of software measures in the Software Engineering Laboratory
Work on metrics is discussed. Factors that affect software quality are reviewed. Metrics is discussed in terms of criteria achievements, reliability, and fault tolerance. Subjective and objective metrics are distinguished. Product/process and cost/quality metrics are characterized and discussed.
Basili, V. R.↗
Software Requirements Analysis as Fault Predictor
Waiting until the integration and system test phase to discover errors leads to more costly rework than resolving those same errors earlier in the lifecycle. Costs increase even more significantly once a software system has become operational. WE can assess the quality of system requirements, but do little to correlate this information either to system assurance activities or long-term reliability projections - both of which remain unclear and anecdotal. Extending earlier work on requirements accomplished by the ARM tool, measuring requirements quality information against code complexity and test data for the same system may be used to predict specific software modules containing high impact or deeply embedded faults now escaping in operational systems. Such knowledge would lead to more effective and efficient test programs. It may enable insight into whether a program should be maintained or started over.
Wallace, Dolores↗
The cleanroom case study in the Software Engineering Laboratory: Project description and early analysis
This case study analyzes the application of the cleanroom software development methodology to the development of production software at the NASA/Goddard Space Flight Center. The cleanroom methodology emphasizes human discipline in program verification to produce reliable software products that are right the first time. Preliminary analysis of the cleanroom case study shows that the method can be applied successfully in the FDD environment and may increase staff productivity and product quality. Compared to typical Software Engineering Laboratory (SEL) activities, there is evidence of lower failure rates, a more complete and consistent set of inline code documentation, a different distribution of phase effort activity, and a different growth profile in terms of lines of code developed. The major goals of the study were to: (1) assess the process used in the SEL cleanroom model with respect to team structure, team activities, and effort distribution; (2) analyze the products of the SEL cleanroom model and determine the impact on measures of interest, including reliability, productivity, overall life-cycle cost, and software quality; and (3) analyze the residual products in the application of the SEL cleanroom model, such as fault distribution, error characteristics, system growth, and computer usage.
Green, Scott↗
The effects of fault counting methods on fault model quality
Over the past several years, we have been developing methods of predicting the fault content of software systems based on measured characteristics of their structural evolution.
detection↗
Quantitative Measures for Software Independent Verification and Validation
As software is maintained or reused, it undergoes an evolution which tends to increase the overall complexity of the code. To understand the effects of this, we brought in statistics experts and leading researchers in software complexity, reliability, and their interrelationships. These experts' project has resulted in our ability to statistically correlate specific code complexity attributes, in orthogonal domains, to errors found over time in the HAL/S flight software which flies in the Space Shuttle. Although only a prototype-tools experiment, the result of this research appears to be extendable to all other NASA software, given appropriate data similar to that logged for the Shuttle onboard software. Our research has demonstrated that a more complete domain coverage can be mathematically demonstrated with the approach we have applied, thereby ensuring full insight into the cause-and-effects relationship between the complexity of a software system and the fault density of that system. By applying the operational profile we can characterize the dynamic effects of software path complexity under this same approach We now have the ability to measure specific attributes which have been statistically demonstrated to correlate to increased error probability, and to know which actions to take, for each complexity domain. Shuttle software verifiers can now monitor the changes in the software complexity, assess the added or decreased risk of software faults in modified code, and determine necessary corrections. The reports, tool documentation, user's guides, and new approach that have resulted from this research effort represent advances in the state of the art of software quality and reliability assurance. Details describing how to apply this technique to other NASA code are contained in this document.
Lee, Alice↗
The Effects of Fault Counting Methods on Fault Model Quality
In this paper, we describe three other fault-counting techniques and compare the models resulting from the application of two of those methods to the models obtained from the application of our proposed definition.
software reliability↗
The Anatomy of Software Changes and Bugs in Autonomous Operating System
Cyberphysical systems with autonomous functions are complex pieces of software, consisting of many components, some of which implement autonomous functionality and some may use AI or machine learning algorithms. Software bugs in an autonomous system are of particular concern, as they can have catastrophic consequences. However, detailed studies based on empirical data are rare and therefore these bugs are not well understood. This paper aims to contribute towards filling that gap by investigating the software changes and bugs in Autonomy Operating System (AOS) for Unmanned Aircraft Systems (UAS), which consist of 26 components containing about 103,000 lines of code and having a total of 772 bugfixes. Based on the data extracted from the code repository and semi-structured interviews with the developers of AOS, we explore the differences among autonomous software components, components developed using Model-based Software Engineering, and reuse with respect to change proneness, fault proneness, distribution of bugfixes among AOS components and files of these components, and characteristics of bugs of different AOS components. Our results show that the autonomous components were significantly more change prone (measured in number of commits and code churn) and fault prone (measured in bugfixes per KLoC) than non-autonomous components. The distribution of the locations of bugfixes was skewed, both at component and file level (i.e., a small number of components / files contained the majority of bugs). These evidence-based findings provide important insights to researchers and practitioners alike and can be used to efficiently improve the quality and reliability of autonomous systems.
Katerina Goseva-Popstojanova↗
Understanding software faults and their role in software reliability modeling
This study is a direct result of an on-going project to model the reliability of a large real-time control avionics system. In previous modeling efforts with this system, hardware reliability models were applied in modeling the reliability behavior of this system. In an attempt to enhance the performance of the adapted reliability models, certain software attributes were introduced in these models to control for differences between programs and also sequential executions of the same program. As the basic nature of the software attributes that affect software reliability become better understood in the modeling process, this information begins to have important implications on the software development process. A significant problem arises when raw attribute measures are to be used in statistical models as predictors, for example, of measures of software quality. This is because many of the metrics are highly correlated. Consider the two attributes: lines of code, LOC, and number of program statements, Stmts. In this case, it is quite obvious that a program with a high value of LOC probably will also have a relatively high value of Stmts. In the case of low level languages, such as assembly language programs, there might be a one-to-one relationship between the statement count and the lines of code. When there is a complete absence of linear relationship among the metrics, they are said to be orthogonal or uncorrelated. Usually the lack of orthogonality is not serious enough to affect a statistical analysis. However, for the purposes of some statistical analysis such as multiple regression, the software metrics are so strongly interrelated that the regression results may be ambiguous and possibly even misleading. Typically, it is difficult to estimate the unique effects of individual software metrics in the regression equation. The estimated values of the coefficients are very sensitive to slight changes in the data and to the addition or deletion of variables in the regression equation. Since most of the existing metrics have common elements and are linear combinations of these common elements, it seems reasonable to investigate the structure of the underlying common factors or components that make up the raw metrics. The technique we have chosen to use to explore this structure is a procedure called principal components analysis. Principal components analysis is a decomposition technique that may be used to detect and analyze collinearity in software metrics. When confronted with a large number of metrics measuring a single construct, it may be desirable to represent the set by some smaller number of variables that convey all, or most, of the information in the original set. Principal components are linear transformations of a set of random variables that summarize the information contained in the variables. The transformations are chosen so that the first component accounts for the maximal amount of variation of the measures of any possible linear transform; the second component accounts for the maximal amount of residual variation; and so on. The principal components are constructed so that they represent transformed scores on dimensions that are orthogonal. Through the use of principal components analysis, it is possible to have a set of highly related software attributes mapped into a small number of uncorrelated attribute domains. This definitively solves the problem of multi-collinearity in subsequent regression analysis. There are many software metrics in the literature, but principal component analysis reveals that there are few distinct sources of variation, i.e. dimensions, in this set of metrics. It would appear perfectly reasonable to characterize the measurable attributes of a program with a simple function of a small number of orthogonal metrics each of which represents a distinct software attribute domain.
Munson, John C.↗
Failure Assessment
Three questions to which software developers want accurate, precise answers are "How can the software system fail?", "mat bad things will happen if the software fails?t', and "How many failures will the software experience?". Numerous techniques have been devised to answer these questions; three of the best known are: 1) Software Fault Tree Analysis (SFTA) 2) Software Failure Modes, Effects, and Criticality Analysis (SFMECA 3) Software Fault/Failure Modeling. SFTA and SFMECA have been successfully used to analyze the flight software for a number of robotic planetary exploration missions, including Galileo, Cassini, and Deep Space 1. Given the increasing interest in reusing software components from mission to mission, one of us has developed techniques for reusing the corresponding portions of the SFTA and SFMECA, reducing the effort required to conduct these analyses. SFTA has also been shown to be effective in analyzing the security aspects of software systems; intrusion mechanisms and effects can easily be modeled using these techniques. The Bi- Directional Safety Analysis (BDSA) method combines a forward search (similar to SFMECA) from potential failure modes to their effects, with a backward search (similar to SFTA) from feasible hazards to the contributing causes of each hazard. BDSA offers an efficient way to identify latent failures. Recent work has extended BDSA to product-line applications such as flight-instrumentation displays and developed tool support for the reuse of the failure-analysis artifacts within a product line. BDSA has also been streamlined to support those projects having tight cost and/or schedule constraints for their failure analysis efforts. We discuss lessons learned from practice, describe available tools, and identi@ some future directions for the topic. A substantial amount of research has been devoted to estimating the number of failures that a software system will experience during test and operations, as well as the number of faults that have been inserted into that system during its development. One of us has found that the amount of structural change to a system during its development is strongly related to the number of faults inserted into it. Using techniques requiring no additional effort on the part of the development organization, the required measurements of structural evolution can be easily obtained from a development effort's configuration management system and readily transformed into an estimate of fault content. So far, structure-fault relationships have been identified for source code; current work seeks to examine artifacts available earlier in the lifecycle to determine if similar relationships between structure and fault content can be found. In particular, relationships between requirements change requests and the number of faults inserted into the implemented system would provide a significant improvement in our ability to control software quality during the early development phases.
fault tree↗
Experiments in fault tolerant software reliability
Twenty functionally equivalent programs were built and tested in a multiversion software experiment. Following unit testing, all programs were subjected to an extensive system test. In the process sixty-one distinct faults were identified among the versions. Less than 12 percent of the faults exhibited varying degrees of positive correlation. The common-cause (or similar) faults spanned as many as 14 components. However, a majority of these faults were trivial, and easily detected by proper unit and/or system testing. Only two of the seven similar faults were difficult faults, and both were caused by specification ambiguities. One of these faults exhibited variable identical-and-wrong response span, i.e. response span which varied with the testing conditions and input data. Techniques that could have been used to avoid the faults are discussed. For example, it was determined that back-to-back testing of 2-tuples could have been used to eliminate about 90 percent of the faults. In addition, four of the seven similar faults could have been detected by using back-to-back testing of 5-tuples. It is believed that most, if not all, similar faults could have been avoided had the specifications been written using more formal notation, the unit testing phase was subject to more stringent standards and controls, and better tools for measuring the quality and adequacy of the test data (e.g. coverage) were used.
Mcallister, David F.↗