Model-based testing
From Wikipedia, the free encyclopedia
Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT).
The model is usually an abstract, partial presentation of the system under test's desired behavior. The test cases derived from this model are functional tests on the same level of abstraction as the model. These test cases are collectively known as the abstract test suite. The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. This is done by mapping the abstract test cases to concrete test cases suitable for execution. In the case of online testing (see below), the abstract test suite exists only as a concept but not as an explicit artifact.
There are many different ways to "derive" tests from a model. Because testing is usually experimental and based on heuristics, there is no one best way to do this. It is common to consolidate all test derivation related design decisions into a package that is often known as "test requirements", "test purpose" or even "use case". This package can contain e.g. information about the part of the model that should be the focus for testing, or about the conditions where it is correct to stop testing (test stopping criteria).
Because test suites are derived from models and not from source code, model-based testing is usually seen as one form of black-box testing. In some aspects, this is not completely accurate. Model-based testing can be combined with source-code level test coverage measurement, and functional models can be based on existing source code in the first place.
Model-based testing for complex software systems is still an evolving field.
[edit] Models
Especially in Model Driven Engineering or in OMG's model-driven architecture the model is built before or parallel to the development process of the system under test. The model can also be constructed from the completed system. Recently the model is created mostly manually, but there are also attempts to create the model automatically, for instance out of the source code. One important way to create new models is by model transformation, using languages like ATL, a QVT-like Domain Specific Language.
Model-based testing inherits the complexity of the domain or, more particularly, of the related domain models. In this paper [1] the authors discuss the requirements related to model-based testing in the domain of Information Systems.
[edit] Deploying model-based testing
There are various known ways to deploy model-based testing, which include online testing, offline generation of executable tests, and offline generation of manually deployable tests.
Online testing means that a model-based testing tool connects “directly” to a system under test and tests it dynamically.
Offline generation of executable tests means that a model-based testing tool generates test cases as a computer-readable asset that can be later deployed automatically. This asset can be, for instance, a collection of Python classes that embodies the generated testing logic.
Offline generation of manually deployable tests means that a model-based testing tool generates test cases as a human-readable asset that can be later deployed manually. This asset can be, for instance, a PDF document in English that describes the generated test steps.
[edit] Deriving tests algorithmically
The effectiveness of model-based testing is primarily due to the potential for automation it offers. If the model is machine-readable and formal to the extent that it has a well-defined behavioral interpretation, test cases can in principle be derived mechanically.
Often the model is translated to or interpreted as a finite state automaton or a state transition system. This automaton represents the possible configurations of the system under test. To find test cases, the automaton is searched for executable paths. A possible execution path can serve as a test case. This method works if the model is deterministic or can be transformed into a deterministic one. Valuable off-nominal test cases may be obtained by leveraging un-specified transitions in these models.
Depending on the complexity of the system under test and the corresponding model the number of paths can be very large, because of the huge amount of possible configurations of the system. For finding appropriate test cases, i.e. paths that refer to a certain requirement to proof, the search of the paths has to be guided. For the test case selection multiple techniques are applied.
[edit] Test case generation by theorem proving
Theorem proving has been originally used for automated proving of logical formulas. For model-based testing approaches the system is modeled by a set of logical expressions (predicates) specifying the system's behavior. For selecting test cases the model is partitioned into equivalence classes over the valid interpretation of the set of the logical expressions describing the system under test. Each class is representing a certain system behavior and can therefore serve as a test case.
The simplest partitioning is done by the disjunctive normal form approach. The logical expressions describing the system's behavior are transformed into the disjunctive normal form.
The classification tree-method provides a more sophisticated hierarchical partitioning. Also partitioning heuristics are used supporting the partitioning algorithms, e.g. heuristics based on boundary value analysis.
[edit] Test case generation by constraint logic programming
Constraint programming can be used to select test cases satisfying specific constraints by solving a set of constraints over a set of variables. The system is described by the means of constraints. Solving the set of constraints can be done by Boolean solvers (e.g. SAT-solvers based on the Boolean satisfiability problem) or by numerical analysis, like the Gaussian elimination. A solution found by solving the set of constraints formulas can serve as a test cases for the corresponding system.
[edit] Test case generation by model checking
Originally model checking was developed as a technique to check if a property of a specification is valid in a model. We provide a model of the system under test and a property we want to test to the model checker. Within the procedure of proofing, if this property is valid in the model, the model checker detects witnesses and counterexamples. A witness is a path, where the property is satisfied, whereas a counterexample is a path in the execution of the model, where the property is violated. These paths can again be used as test cases.
[edit] Test case generation by symbolic execution
Symbolic execution is often used in frameworks for model-based testing. It can be a means in searching for execution traces in an abstract model. In principle the program execution is simulated using symbols for variables rather than actual values. Then the program can be executed in a symbolic way. Each execution path represents one possible program execution and can be used as a test case. For that, the symbols have to be instantiated by assigning values to the symbols.
[edit] Test case generation by using an event-flow model
A popular model that has recently been used extensively for testing software with a graphical user-interface (GUI) front-end is called the event-flow model that represents events and event interactions. In much the same way as a control-flow model represents all possible execution paths in a program, and a data-flow model represents all possible definitions and uses of a memory location, the event-flow model represents all possible sequences of events that can be executed on the GUI. More specifically, a GUI is decomposed into a hierarchy of modal dialogs; this hierarchy is represented as an integration tree; each modal dialog is represented as an event-flow graph that shows all possible event execution paths in the dialog; individual events are represented using their preconditions and effects. An overview of the event-flow model with associated algorithms to semi-automatically reverse engineer the model from an executing GUI software is presented in this 2007 paper. Because the event-flow model is not tied to a specific aspect of the GUI testing process, it may be used to perform a wide variety of testing tasks by defining specialized model-based techniques called event-space exploration strategies (ESES). These ESES use the event-flow model in a number of ways to develop an end-to-end GUI testing process, namely by checking the model, test-case generation, and test oracle creation.
[edit] Test case generation by using a Markov chains model
Markov chains are an efficient way to handle MBT, here call Statistical Model Based Testing. Markov chains can be understood as Finite State Machine weighted by probability on transitions. Collection of probability builds a profile. Model Based Testing from Markov chains uses a random process to generate test case conformed to the targeted profile. We say "usage model" and "usage profile”, when the probability pool defines the usage of the system. For example, the "Bing Bang" approach of Integration testing is defined as Usage Model Testing. In that case, test case results analysis provides system under test reliability.
Statistical Model Based Testing is also powerful for test process improvement. Because of its probability, random test case generation is controlled and driven to converge to the target. For example, by using a "usage profile", generated test cases will, in a statistical way, test the most used SUT functions before the others.
[edit] See also
- ATLAS Transformation Language (ATL)
- Domain Specific Language (DSL)
- Domain-specific modeling (DSM)
- Generic Modeling Environment (GME)
- Keyword-driven testing
- Meta-Object Facility (MOF)
- Model-driven architecture (MDA is an OMG Trademark)
- Model Driven Engineering (MDE is not an OMG Trademark)
- Model-based testing tools
- Object-oriented analysis and design (OOAD)
- OCL
- Meta-modeling
- XMI
- VIATRA
[edit] References
 |
This article includes a list of references or external links, but its sources remain unclear because it lacks inline citations. Please improve this article by introducing more precise citations where appropriate. (February 2008) |
[edit] Further reading
- Practical Model-Based Testing: A Tools Approach, Mark Utting and Bruno Legeard, ISBN-13: 978-0-12-372501-1, Morgan-Kaufmann 2007.
- Model-Based Software Testing and Analysis with C#, Jonathan Jacky, Margus Veanes, Colin Campbell, and Wolfram Schulte, ISBN-13: 978-0-521-68761-4, Cambridge University Press 2008.
- Model-Based Testing of Reactive Systems Advanced Lecture Series, LNCS 3472, Springer-Verlag, 2005.
- Hong Zhu et al. (2008). AST '08: Proceedings of the 3rd International Workshop on Automation of Software Test. ACM Press. ISBN 978-1-60558-030-2.
