Erlang -- Test Structure

Common Test

User's Guide

Version 1.15

- User's Guide
- Reference Manual
- Release Notes
- PDF
- Top

- Expand All
- Contract All

### Chapters

- Introduction
  - Top of chapter
  - Scope
  - Prerequisites
- Common Test Basics
  - Top of chapter
  - General
  - Test Suite Organisation
  - Support Libraries
  - Suites and Test Cases
  - External Interfaces
- Getting Started
  - Top of chapter
  - Introduction for Newcomers
  - Test Case Execution
  - A Simple Test Suite
  - A Test Suite with Configuration Functions
  - Questions and Answers
- Installation
  - Top of chapter
  - General Information
- Writing Test Suites
  - Top of chapter
  - Support for Test Suite Authors
  - Test Suites
  - Init and End per Suite
  - Init and End per Test Case
  - Test Cases
  - Test Case Information Function
  - Test Suite Information Function
  - Test Case Groups
  - Parallel Property and Nested Groups
  - Parallel Test Cases and I/O
  - Repeated Groups
  - Shuffled Test Case Order
  - Group Information Function
  - Information Functions for Init- and End-Configuration
  - Data and Private Directories
  - Execution Environment
  - Timetrap Time-Outs
  - Logging - Categories and Verbosity Levels
  - Illegal Dependencies
- Test Structure
  - Top of chapter
  - General
  - Skipping Test Cases
  - Definition of Terms
- Examples and Templates
  - Top of chapter
  - Test Suite Example
  - Test Suite Templates
- Running Tests and Analyzing Results
  - Top of chapter
  - Using the Common Test Framework
  - Automatic Compilation of Test Suites and Help Modules
  - Running Tests from the OS Command Line
  - Running Tests from the Erlang Shell or from an Erlang Program
  - Test Case Group Execution
  - Running the Interactive Shell Mode
  - Step-by-Step Execution of Test Cases with the Erlang Debugger
  - Test Specifications
  - Running Tests from the Web-Based GUI
  - Log Files
  - HTML Style Sheets
  - Repeating Tests
  - Silent Connections
- External Configuration Data
  - Top of chapter
  - General
  - Syntax
  - Requiring and Reading Configuration Data
  - Using Configuration Variables Defined in Multiple Files
  - Encrypted Configuration Files
  - Opening Connections Using Configuration Data
  - User-Specific Configuration Data Formats
  - Examples of Configuration Data Handling
  - Example of User-Specific Configuration Handler
- Code Coverage Analysis
  - Top of chapter
  - General
  - Use
  - Stopping the Cover Tool When Tests Are Completed
  - The Cover Specification File
  - Cross Cover Analysis
  - Logging
- Using Common Test for Large-Scale Testing
  - Top of chapter
  - General
  - Use
  - Test Specifications
  - Automatic Startup of Test Target Nodes
- Event Handling
  - Top of chapter
  - General
  - Use
- Dependencies between Test Cases and Suites
  - Top of chapter
  - General
  - Saving Configuration Data
  - Sequences
- Common Test Hooks
  - Top of chapter
  - General
  - Installing a CTH
  - CTH Scope
  - Manipulating Tests
  - Synchronizing External User Applications with Common Test
  - Example CTH
  - Built-In CTHs
- Some Thoughts about Testing
  - Top of chapter
  - Goals
  - What to Test

# 6 Test Structure

### 6.1 General

A test is performed by running one or more test suites. A test suite consists of test cases, configuration functions, and information functions. Test cases can be grouped in so called test case groups. A test suite is an Erlang module and test cases are implemented as Erlang functions. Test suites are stored in test directories.

### 6.2 Skipping Test Cases

Certain test cases can be skipped, for example, if you know beforehand that a specific test case fails. The reason can be functionality that is not yet implemented, a bug that is known but not yet fixed, or some functionality that does not work or is not applicable on a specific platform.

Test cases can be skipped in the following ways:

- Using skip_suites and skip_cases terms in test specifications.
- Returning {skip,Reason} from function init_per_testcase/2 or init_per_suite/1.
- Returning {skip,Reason} from the execution clause of the test case. The execution clause is called, so the author must ensure that the test case does not run.

When a test case is skipped, it is noted as SKIPPED in the HTML log.

### 6.3 Definition of Terms

Auto-skipped test case

  When a configuration function fails (that is, terminates unexpectedly), the test cases depending on the configuration function are skipped automatically by Common Test. The status of the test cases is then "auto-skipped". Test cases are also "auto-skipped" by Common Test if the required configuration data is unavailable at runtime.

Configuration function

  A function in a test suite that is meant to be used for setting up, cleaning up, and/or verifying the state and environment on the System Under Test (SUT) and/or the Common Test host node, so that a test case (or a set of test cases) can execute correctly.

Configuration file

  A file containing data related to a test and/or an SUT, for example, protocol server addresses, client login details, and hardware interface addresses. That is, any data that is to be handled as variable in the suite and not be hard-coded.

Configuration variable

  A name (an Erlang atom) associated with a data value read from a configuration file.

data_dir

  Data directory for a test suite. This directory contains any files used by the test suite, for example, extra Erlang modules, binaries, or data files.

Information function

  A function in a test suite that returns a list of properties (read by the Common Test server) that describes the conditions for executing the test cases in the suite.

Major log file

  An overview and summary log file for one or more test suites.

Minor log file

  A log file for one particular test case. Also called the test case log file.

priv_dir

  Private directory for a test suite. This directory is to be used when the test suite needs to write to files.

ct_run

  The name of an executable program that can be used as an interface for specifying and running tests with Common Test.

Test case

  A single test included in a test suite. A test case is implemented as a function in a test suite module.

Test case group

  A set of test cases sharing configuration functions and execution properties. The execution properties specify if the test cases in the group are to be executed in random order, in parallel, or in sequence, and if the execution of the group is be repeated. Test case groups can also be nested. That is, a group can, besides test cases, contain subgroups.

Test suite

  An Erlang module containing a collection of test cases for a specific functional area.

Test directory

  A directory containing one or more test suite modules, that is, a group of test suites.

Argument Config

  A list of key-value tuples (that is, a property list) containing runtime configuration data passed from the configuration functions to the test cases.

User-skipped test case

  The status of a test case explicitly skipped in any of the ways described in section Skipping Test Cases.