Code coverage

From Wikipedia, the free encyclopedia

Jump to: navigation, search

Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing. Currently, the use of code coverage is extended to the field of digital hardware, the contemporary design methodology of which relies on Hardware description languages (HDLs).

Code coverage techniques were amongst the first techniques invented for systematic software testing. The first published reference was by Miller and Maloney in Communications of the ACM in 1963.

Code coverage is one consideration in the safety certification of avionics equipment. The standard by which avionics gear is certified by the Federal Aviation Administration (FAA) is documented in DO-178(b) [1]

Contents

[edit] Coverage criteria

To measure how well the program is exercised by a test suite, one or more coverage criteria are used. There are a number of coverage criteria, the main ones being:

  • Function coverage - Has each function in the program been executed?
  • Statement coverage - Has each line of the source code been executed?
  • Decision coverage (also known as Branch coverage) - Has each control structure (such as an if statement) evaluated both to true and false?
  • Condition coverage - Has each boolean sub-expression evaluated both to true and false (this does not necessarily imply decision coverage)?
  • Modified Condition/Decision Coverage (MC/DC) - Has every condition in a decision taken on all possible outcomes at least once? Has each condition been shown to affect that decision outcome independently?
  • Path coverage - Has every possible route through a given part of the code been executed?
  • Entry/exit coverage - Has every possible call and return of the function been executed?

Safety-critical applications are often required to demonstrate that testing achieves 100% of some form of code coverage.

Some of the coverage criteria above are connected. For instance, path coverage implies decision, statement and entry/exit coverage. Decision coverage implies statement coverage, because every statement is part of a branch.

Full path coverage, of the type described above, is usually impractical or impossible. Any module with a succession of n decisions in it can have up to 2n paths within it; loop constructs can result in an infinite number of paths. Many paths may also be infeasible, in that there is no input to the program under test that can cause that particular path to be executed. However, a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem)[2]. Techniques for practical path coverage testing instead attempt to identify classes of code paths that differ only in the number of loop executions, and to achieve "basis path" coverage the tester must cover all the path classes.

[edit] Code coverage in practice

The target software is built with special options or libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) is mapped back to the function points in the source code. This process allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested. The resulting output is then analysed to see what areas of code have not been exercised and the tests are updated to include these areas as necessary. Combined with other code coverage methods, the aim is to develop a rigorous, yet manageable, set of regression tests.

Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches or code decision points were executed to complete the test. They both report a coverage metric, measured as a percentage. The meaning of this depends on what form(s) of code coverage have been used, as 67% path coverage is more comprehensive than 67% statement coverage.

Generally, code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus, they are only used in the lab. As one might expect, there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.

There are also some sorts of defects which are affected by such tools. In particular, some race conditions or similar real time sensitive operations can be masked when run under code coverage environments; and conversely, some of these defects may become easier to find as a result of the additional overhead of the testing code.

Code coverage may be regarded as a more up-to-date incarnation of debugging in that the automated tools used to achieve statement and path coverage are often referred to as “debugging utilities”. These tools allow the program code under test to be observed on screen while the program is executing; additionally, commands and keyboard function keys are available to allow the code to be “stepped” through literally line by line. Alternatively, it is possible to define pinpointed lines of code as “breakpoints” which will allow a large section of the code to be executed, then stopping at that point and displaying that part of the program on screen. Judging where to put breakpoints is based on a reasonable understanding of the program indicating that a particular defect is thought to exist around that point. The data values held in program variables can also be examined and, in some instances, altered (with care) to try out “what if” scenarios. Clearly, use of a debugging tool is more the domain of the software engineer at a unit test level and it is more likely that the software tester will ask the software engineer to perform this[citation needed]. However, it is useful for the tester to understand the concept of a debugging tool.

[edit] Software code coverage tools

  • Insure++ - a coverage of source code of application tested with functional tests.
  • iSYSTEM winIDEA - measures coverage on a wide variety of embedded processors. It works by recoding execution directly on hardware, without instrumenting code or modifying the program and in real-time.
  • EMMA - a free Java code coverage tool
  • Atlassian Clover - a Java code coverage and test visualization tool which also displays coverage per-test. Free for open source projects
  • Jtest, C++test, .Test - calculate percentage of code covered by tests.
  • LDRA Testbed measures statement coverage, branch/decision coverage, LCSAJ Coverage, procedure/function call coverage, branch condition coverage, branch condition combination coverage and modified condition decision coverage (MC/DC) for DO-178B Level A.
  • Cobertura - a free Java tool that calculates the percentage of code accessed by tests
  • Devel::Cover - Code coverage metrics for Perl
  • gcov - Code coverage test for GCC compiler
  • ggcov - GTK+ GUI for gcov
  • lcov - Web UI for gcov
  • gcov-kernel - gcov support for the Linux kernel
  • VB Watch - Visual Basic code coverage and performance analysis tool
  • rcov - Code coverage for Ruby
  • PartCover - .NET 2.0 code coverage tool
  • BullseyeCoverage - C and C++ code coverage tool
  • python-coverage - Code coverage for Python
  • XDebug - PHP debugging tool, including code coverage
  • CodeCover - Java / Cobol code coverage tool for use by shell in win / linux, apache ant script, or as Eclipse-Plugin (including boolean analyzer and correlation matrix), XML Reports
  • Sonar - Sonar collects, analyzes and reports metrics on source code. Includes consolidated reporting on and across projects throughout time.

[edit] Hardware code coverage tools

[edit] See also

[edit] Notes

  1. ^ RTCA/DO-178(b), Software Considerations in Airborne Systems and Equipment Certification, Radio Technical Commission for Aeronautics, December 1, 1992.
  2. ^ Dorf, Richard C.: Computers, Software Engineering, and Digital Devices, Chapter 12, pg. 15. CRC Press, 2006. ISBN 0849373409, 9780849373404; via Google Book Search

[edit] External links

Personal tools