## 25.9. Stencil Test

The stencil test conditionally disables coverage of a sample based on the outcome of a comparison between the stencil value in the depth/stencil attachment at location (xf,yf) (for the appropriate sample) and a reference value. The stencil test also updates the value in the stencil attachment, depending on the test state, the stencil value and the stencil write masks. The test is enabled or disabled by the `stencilTestEnable` member of `VkPipelineDepthStencilStateCreateInfo`.

When disabled, the stencil test and associated modifications are not made, and the sample’s coverage is not modified.

The stencil test is controlled with the `front` and `back` members of `VkPipelineDepthStencilStateCreateInfo` which are of type `VkStencilOpState`.

The `VkStencilOpState` structure is defined as:

    typedef struct VkStencilOpState {
        VkStencilOp    failOp;
        VkStencilOp    passOp;
        VkStencilOp    depthFailOp;
        VkCompareOp    compareOp;
        uint32_t       compareMask;
        uint32_t       writeMask;
        uint32_t       reference;
    } VkStencilOpState;

- `failOp` is the action performed on samples that fail the stencil test.
- `passOp` is the action performed on samples that pass both the depth and stencil tests.
- `depthFailOp` is the action performed on samples that pass the stencil test and fail the depth test.
- `compareOp` is the comparison operator used in the stencil test.
- `compareMask` selects the bits of the unsigned integer stencil values participating in the stencil test.
- `writeMask` selects the bits of the unsigned integer stencil values updated by the stencil test in the stencil framebuffer attachment.
- `reference` is an integer reference value that is used in the unsigned stencil comparison.

There are two sets of stencil-related state, the front stencil state set and the back stencil state set. Stencil tests and writes use the front set of stencil state when processing fragments rasterized from non-polygon primitives (points and lines) and front-facing polygon primitives while the back set of stencil state is used when processing fragments rasterized from back-facing polygon primitives. For the purposes of stencil testing, a primitive is still considered a polygon even if the polygon is to be rasterized as points or lines due to the current `VkPolygonMode`. Whether a polygon is front- or back-facing is determined in the same manner used for face culling (see Basic Polygon Rasterization).

The operation of the stencil test is also affected by the `compareMask`, `writeMask`, and `reference` members of `VkStencilOpState` set in the pipeline state object if the pipeline state object is created without the `VK_DYNAMIC_STATE_STENCIL_COMPARE_MASK`, `VK_DYNAMIC_STATE_STENCIL_WRITE_MASK`, and `VK_DYNAMIC_STATE_STENCIL_REFERENCE` dynamic states enabled, respectively.

If the pipeline state object is created with the `VK_DYNAMIC_STATE_STENCIL_COMPARE_MASK` dynamic state enabled, then to dynamically set the stencil compare mask call:

    void vkCmdSetStencilCompareMask(
        VkCommandBuffer                             commandBuffer,
        VkStencilFaceFlags                          faceMask,
        uint32_t                                    compareMask);

- `commandBuffer` is the command buffer into which the command will be recorded.

- `faceMask` is a bitmask specifying the set of stencil state for which to update the compare mask. Bits which can be set include:

      typedef enum VkStencilFaceFlagBits {
          VK_STENCIL_FACE_FRONT_BIT = 0x00000001,
          VK_STENCIL_FACE_BACK_BIT = 0x00000002,
          VK_STENCIL_FRONT_AND_BACK = 0x00000003,
      } VkStencilFaceFlagBits;

  - `VK_STENCIL_FACE_FRONT_BIT` indicates that only the front set of stencil state is updated.
  - `VK_STENCIL_FACE_BACK_BIT` indicates that only the back set of stencil state is updated.
  - `VK_STENCIL_FRONT_AND_BACK` is the combination of `VK_STENCIL_FACE_FRONT_BIT` and `VK_STENCIL_FACE_BACK_BIT` and indicates that both sets of stencil state are updated.
- `compareMask` is the new value to use as the stencil compare mask.

If the pipeline state object is created with the `VK_DYNAMIC_STATE_STENCIL_WRITE_MASK` dynamic state enabled, then to dynamically set the stencil write mask call:

    void vkCmdSetStencilWriteMask(
        VkCommandBuffer                             commandBuffer,
        VkStencilFaceFlags                          faceMask,
        uint32_t                                    writeMask);

- `commandBuffer` is the command buffer into which the command will be recorded.
- `faceMask` is a bitmask of `VkStencilFaceFlagBits` specifying the set of stencil state for which to update the write mask, as described above for `vkCmdSetStencilCompareMask`.
- `writeMask` is the new value to use as the stencil write mask.

If the pipeline state object is created with the `VK_DYNAMIC_STATE_STENCIL_REFERENCE` dynamic state enabled, then to dynamically set the stencil reference value call:

    void vkCmdSetStencilReference(
        VkCommandBuffer                             commandBuffer,
        VkStencilFaceFlags                          faceMask,
        uint32_t                                    reference);

- `commandBuffer` is the command buffer into which the command will be recorded.
- `faceMask` is a bitmask of `VkStencilFaceFlagBits` specifying the set of stencil state for which to update the reference value, as described above for `vkCmdSetStencilCompareMask`.
- `reference` is the new value to use as the stencil reference value.

`reference` is an integer reference value that is used in the unsigned stencil comparison. Stencil comparison clamps the reference value to [0,2s-1], where s is the number of bits in the stencil framebuffer attachment. The s least significant bits of `compareMask` are bitwise `ANDed` with both the reference and the stored stencil value, and the resulting masked values are those that participate in the comparison controlled by `compareOp`. Let R be the masked reference value and S be the masked stored stencil value.

`compareOp` is a symbolic constant that determines the stencil comparison function:

    typedef enum VkCompareOp {
        VK_COMPARE_OP_NEVER = 0,
        VK_COMPARE_OP_LESS = 1,
        VK_COMPARE_OP_EQUAL = 2,
        VK_COMPARE_OP_LESS_OR_EQUAL = 3,
        VK_COMPARE_OP_GREATER = 4,
        VK_COMPARE_OP_NOT_EQUAL = 5,
        VK_COMPARE_OP_GREATER_OR_EQUAL = 6,
        VK_COMPARE_OP_ALWAYS = 7,
    } VkCompareOp;

- `VK_COMPARE_OP_NEVER`: the test never passes.
- `VK_COMPARE_OP_LESS`: the test passes when R < S.
- `VK_COMPARE_OP_EQUAL`: the test passes when R = S.
- `VK_COMPARE_OP_LESS_OR_EQUAL`: the test passes when R ≤ S.
- `VK_COMPARE_OP_GREATER`: the test passes when R > S.
- `VK_COMPARE_OP_NOT_EQUAL`: the test passes when R ≠ S.
- `VK_COMPARE_OP_GREATER_OR_EQUAL`: the test passes when R ≥ S.
- `VK_COMPARE_OP_ALWAYS`: the test always passes.

As described earlier, the `failOp`, `passOp`, and `depthFailOp` members of `VkStencilOpState` indicate what happens to the stored stencil value if this or certain subsequent tests fail or pass. Each enum is of type `VkStencilOp`, which is defined as:

    typedef enum VkStencilOp {
        VK_STENCIL_OP_KEEP = 0,
        VK_STENCIL_OP_ZERO = 1,
        VK_STENCIL_OP_REPLACE = 2,
        VK_STENCIL_OP_INCREMENT_AND_CLAMP = 3,
        VK_STENCIL_OP_DECREMENT_AND_CLAMP = 4,
        VK_STENCIL_OP_INVERT = 5,
        VK_STENCIL_OP_INCREMENT_AND_WRAP = 6,
        VK_STENCIL_OP_DECREMENT_AND_WRAP = 7,
    } VkStencilOp;

The possible values are:

- `VK_STENCIL_OP_KEEP` keeps the current value.
- `VK_STENCIL_OP_ZERO` sets the value to 0.
- `VK_STENCIL_OP_REPLACE` sets the value to `reference`.
- `VK_STENCIL_OP_INCREMENT_AND_CLAMP` increments the current value and clamps to the maximum representable unsigned value.
- `VK_STENCIL_OP_DECREMENT_AND_CLAMP` decrements the current value and clamps to 0.
- `VK_STENCIL_OP_INVERT` bitwise-inverts the current value.
- `VK_STENCIL_OP_INCREMENT_AND_WRAP` increments the current value and wraps to 0 when the maximum value would have been exceeded.
- `VK_STENCIL_OP_DECREMENT_AND_WRAP` decrements the current value and wraps to the maximum possible value when the value would go below 0.

For purposes of increment and decrement, the stencil bits are considered as an unsigned integer.

If the stencil test fails, the sample’s coverage bit is cleared in the fragment. If there is no stencil framebuffer attachment, stencil modification cannot occur, and it is as if the stencil tests always pass.

If the stencil test passes, the `writeMask` member of the `VkStencilOpState` structures controls how the updated stencil value is written to the stencil framebuffer attachment.

The least significant s bits of `writeMask`, where s is the number of bits in the stencil framebuffer attachment, specify an integer mask. Where a 1 appears in this mask, the corresponding bit in the stencil value in the depth/stencil attachment is written; where a 0 appears, the bit is not written. The `writeMask` value uses either the front-facing or back-facing state based on the facing-ness of the fragment. Fragments generated by front-facing primitives use the front mask and fragments generated by back-facing primitives use the back mask.