Erlang -- Dependencies between Test Cases and Suites

Common Test

User's Guide

Version 1.16.1

- User's Guide
- Reference Manual
- Release Notes
- PDF
- Top

- Expand All
- Contract All

### Chapters

- Introduction
  - Top of chapter
  - Scope
  - Prerequisites
- Common Test Basics
  - Top of chapter
  - General
  - Test Suite Organisation
  - Support Libraries
  - Suites and Test Cases
  - External Interfaces
- Getting Started
  - Top of chapter
  - Introduction for Newcomers
  - Test Case Execution
  - A Simple Test Suite
  - A Test Suite with Configuration Functions
  - Questions and Answers
- Installation
  - Top of chapter
  - General Information
- Writing Test Suites
  - Top of chapter
  - Support for Test Suite Authors
  - Test Suites
  - Init and End per Suite
  - Init and End per Test Case
  - Test Cases
  - Test Case Information Function
  - Test Suite Information Function
  - Test Case Groups
  - Parallel Property and Nested Groups
  - Parallel Test Cases and I/O
  - Repeated Groups
  - Shuffled Test Case Order
  - Group Information Function
  - Information Functions for Init- and End-Configuration
  - Data and Private Directories
  - Execution Environment
  - Timetrap Time-Outs
  - Logging - Categories and Verbosity Levels
  - Illegal Dependencies
- Test Structure
  - Top of chapter
  - General
  - Skipping Test Cases
  - Definition of Terms
- Examples and Templates
  - Top of chapter
  - Test Suite Example
  - Test Suite Templates
- Running Tests and Analyzing Results
  - Top of chapter
  - Using the Common Test Framework
  - Automatic Compilation of Test Suites and Help Modules
  - Running Tests from the OS Command Line
  - Running Tests from the Erlang Shell or from an Erlang Program
  - Test Case Group Execution
  - Running the Interactive Shell Mode
  - Step-by-Step Execution of Test Cases with the Erlang Debugger
  - Test Specifications
  - Running Tests from the Web-Based GUI
  - Log Files
  - HTML Style Sheets
  - Repeating Tests
  - Silent Connections
- External Configuration Data
  - Top of chapter
  - General
  - Syntax
  - Requiring and Reading Configuration Data
  - Using Configuration Variables Defined in Multiple Files
  - Encrypted Configuration Files
  - Opening Connections Using Configuration Data
  - User-Specific Configuration Data Formats
  - Examples of Configuration Data Handling
  - Example of User-Specific Configuration Handler
- Code Coverage Analysis
  - Top of chapter
  - General
  - Use
  - Stopping the Cover Tool When Tests Are Completed
  - The Cover Specification File
  - Cross Cover Analysis
  - Logging
- Using Common Test for Large-Scale Testing
  - Top of chapter
  - General
  - Use
  - Test Specifications
  - Automatic Startup of Test Target Nodes
- Event Handling
  - Top of chapter
  - General
  - Use
- Dependencies between Test Cases and Suites
  - Top of chapter
  - General
  - Saving Configuration Data
  - Sequences
- Common Test Hooks
  - Top of chapter
  - General
  - Installing a CTH
  - CTH Scope
  - Manipulating Tests
  - Synchronizing External User Applications with Common Test
  - Example CTH
  - Built-In CTHs
- Some Thoughts about Testing
  - Top of chapter
  - Goals
  - What to Test

# 13 Dependencies between Test Cases and Suites

### 13.1 General

When creating test suites, it is strongly recommended to not create dependencies between test cases, that is, letting test cases depend on the result of previous test cases. There are various reasons for this, such as, the following:

- It makes it impossible to run test cases individually.
- It makes it impossible to run test cases in a different order.
- It makes debugging difficult (as a fault can be the result of a problem in a different test case than the one failing).
- There are no good and explicit ways to declare dependencies, so it can be difficult to see and understand these in test suite code and in test logs.
- Extending, restructuring, and maintaining test suites with test case dependencies is difficult.

There are often sufficient means to work around the need for test case dependencies. Generally, the problem is related to the state of the System Under Test (SUT). The action of one test case can change the system state. For some other test case to run properly, this new state must be known.

Instead of passing data between test cases, it is recommended that the test cases read the state from the SUT and perform assertions (that is, let the test case run if the state is as expected, otherwise reset or fail). It is also recommended to use the state to set variables necessary for the test case to execute properly. Common actions can often be implemented as library functions for test cases to call to set the SUT in a required state. (Such common actions can also be separately tested, if necessary, to ensure that they work as expected). It is sometimes also possible, but not always desirable, to group tests together in one test case, that is, let a test case perform a "scenario" test (a test consisting of subtests).

Consider, for example, a server application under test. The following functionality is to be tested:

- Starting the server
- Configuring the server
- Connecting a client to the server
- Disconnecting a client from the server
- Stopping the server

There are obvious dependencies between the listed functions. The server cannot be configured if it has not first been started, a client connot be connectd until the server is properly configured, and so on. If we want to have one test case for each function, we might be tempted to try to always run the test cases in the stated order and carry possible data (identities, handles, and so on) between the cases and therefore introduce dependencies between them.

To avoid this, we can consider starting and stopping the server for every test. We can thus implement the start and stop action as common functions to be called from init_per_testcase and end_per_testcase. (Remember to test the start and stop functionality separately.) The configuration can also be implemented as a common function, maybe grouped with the start function. Finally, the testing of connecting and disconnecting a client can be grouped into one test case. The resulting suite can look as follows:

     -module(my_server_SUITE).
     -compile(export_all).
     -include_lib("ct.hrl").

     %%% init and end functions...

     suite() -> [{require,my_server_cfg}].

     init_per_testcase(start_and_stop, Config) ->
         Config;

     init_per_testcase(config, Config) ->
         [{server_pid,start_server()} | Config];

     init_per_testcase(_, Config) ->
         ServerPid = start_server(),
         configure_server(),
         [{server_pid,ServerPid} | Config].

     end_per_testcase(start_and_stop, _) ->
         ok;

     end_per_testcase(_, _) ->
         ServerPid = ?config(server_pid),
         stop_server(ServerPid).

     %%% test cases...

     all() -> [start_and_stop, config, connect_and_disconnect].

     %% test that starting and stopping works
     start_and_stop(_) ->
         ServerPid = start_server(),
         stop_server(ServerPid).

     %% configuration test
     config(Config) ->
         ServerPid = ?config(server_pid, Config),
         configure_server(ServerPid).

     %% test connecting and disconnecting client
     connect_and_disconnect(Config) ->
         ServerPid = ?config(server_pid, Config),
         {ok,SessionId} = my_server:connect(ServerPid),
         ok = my_server:disconnect(ServerPid, SessionId).

     %%% common functions...

     start_server() ->
         {ok,ServerPid} = my_server:start(),
         ServerPid.

     stop_server(ServerPid) ->
         ok = my_server:stop(),
         ok.

     configure_server(ServerPid) ->
         ServerCfgData = ct:get_config(my_server_cfg),
         ok = my_server:configure(ServerPid, ServerCfgData),
         ok.

### 13.2 Saving Configuration Data

Sometimes it is impossible, or infeasible, to implement independent test cases. Maybe it is not possible to read the SUT state. Maybe resetting the SUT is impossible and it takes too long time to restart the system. In situations where test case dependency is necessary, CT offers a structured way to carry data from one test case to the next. The same mechanism can also be used to carry data from one test suite to the next.

The mechanism for passing data is called save_config. The idea is that one test case (or suite) can save the current value of Config, or any list of key-value tuples, so that the next executing test case (or test suite) can read it. The configuration data is not saved permanently but can only be passed from one case (or suite) to the next.

To save Config data, return tuple {save_config,ConfigList} from end_per_testcase or from the main test case function.

To read data saved by a previous test case, use macro config with a saved_config key as follows:

{Saver,ConfigList} = ?config(saved_config, Config)

Saver (atom()) is the name of the previous test case (where the data was saved). The config macro can be used to extract particular data also from the recalled ConfigList. It is strongly recommended that Saver is always matched to the expected name of the saving test case. This way, problems because of restructuring of the test suite can be avoided. Also, it makes the dependency more explicit and the test suite easier to read and maintain.

To pass data from one test suite to another, the same mechanism is used. The data is to be saved by finction end_per_suite and read by function init_per_suite in the suite that follows. When passing data between suites, Saver carries the name of the test suite.

Example:

     -module(server_b_SUITE).
     -compile(export_all).
     -include_lib("ct.hrl").

     %%% init and end functions...

     init_per_suite(Config) ->
         %% read config saved by previous test suite
         {server_a_SUITE,OldConfig} = ?config(saved_config, Config),
         %% extract server identity (comes from server_a_SUITE)
         ServerId = ?config(server_id, OldConfig),
         SessionId = connect_to_server(ServerId),
         [{ids,{ServerId,SessionId}} | Config].

     end_per_suite(Config) ->
         %% save config for server_c_SUITE (session_id and server_id)
         {save_config,Config}

     %%% test cases...

     all() -> [allocate, deallocate].

     allocate(Config) ->
         {ServerId,SessionId} = ?config(ids, Config),
         {ok,Handle} = allocate_resource(ServerId, SessionId),
         %% save handle for deallocation test
         NewConfig = [{handle,Handle}],
         {save_config,NewConfig}.

     deallocate(Config) ->
         {ServerId,SessionId} = ?config(ids, Config),
         {allocate,OldConfig} = ?config(saved_config, Config),
         Handle = ?config(handle, OldConfig),
         ok = deallocate_resource(ServerId, SessionId, Handle).

To save Config data from a test case that is to be skipped, return tuple {skip_and_save,Reason,ConfigList}.

The result is that the test case is skipped with Reason printed to the log file (as described earlier) and ConfigList is saved for the next test case. ConfigList can be read using ?config(saved_config, Config), as described earlier. skip_and_save can also be returned from init_per_suite. In this case, the saved data can be read by init_per_suite in the suite that follows.

### 13.3 Sequences

Sometimes test cases depend on each other so that if one case fails, the following tests are not to be executed. Typically, if the save_config facility is used and a test case that is expected to save data crashes, the following case cannot run. Common Test offers a way to declare such dependencies, called sequences.

A sequence of test cases is defined as a test case group with a sequence property. Test case groups are defined through function groups/0 in the test suite (for details, see section Test Case Groups.

For example, to ensure that if allocate in server_b_SUITE crashes, deallocate is skipped, the following sequence can be defined:

     groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].

Assume that the suite contains the test case get_resource_status that is independent of the other two cases, then function all can look as follows:

     all() -> [{group,alloc_and_dealloc}, get_resource_status].

If alloc succeeds, dealloc is also executed. If alloc fails however, dealloc is not executed but marked as SKIPPED in the HTML log. get_resource_status runs no matter what happens to the alloc_and_dealloc cases.

Test cases in a sequence are executed in order until all succeed or one fails. If one fails, all following cases in the sequence are skipped. The cases in the sequence that have succeeded up to that point are reported as successful in the log. Any number of sequences can be specified.

Example:

     groups() -> [{scenarioA, [sequence], [testA1, testA2]},
                  {scenarioB, [sequence], [testB1, testB2, testB3]}].

     all() -> [test1,
               test2,
               {group,scenarioA},
    	   test3,
               {group,scenarioB},
               test4].

A sequence group can have subgroups. Such subgroups can have any property, that is, they are not required to also be sequences. If you want the status of the subgroup to affect the sequence on the level above, return {return_group_result,Status} from end_per_group/2, as described in section Repeated Groups in Writing Test Suites. A failed subgroup (Status == failed) causes the execution of a sequence to fail in the same way a test case does.