The stencil test conditionally disables coverage of a sample based on the
outcome of a comparison between the stencil value in the depth/stencil
attachment at location
$(x_f,y_f)$
(for the appropriate sample)
and a reference value. The stencil test also updates the value in the
stencil attachment, depending on the test state, the stencil value and the
stencil write masks. The test is enabled or disabled by the
stencilTestEnable
member of
VkPipelineDepthStencilStateCreateInfo
.
When disabled, the stencil test and associated modifications are not made, and the sample’s coverage is not modified.
The stencil test is controlled with the front
and back
members
of VkPipelineDepthStencilStateCreateInfo
which are of type
VkStencilOpState
.
The VkStencilOpState
structure is defined as:
typedef struct VkStencilOpState { VkStencilOp failOp; VkStencilOp passOp; VkStencilOp depthFailOp; VkCompareOp compareOp; uint32_t compareMask; uint32_t writeMask; uint32_t reference; } VkStencilOpState;
failOp
is the action performed on samples that fail the stencil
test.
passOp
is the action performed on samples that pass both the depth
and stencil tests.
depthFailOp
is the action performed on samples that pass the
stencil test and fail the depth test.
compareOp
is the comparison operator used in the stencil test.
compareMask
selects the bits of the unsigned integer stencil
values participating in the stencil test.
writeMask
selects the bits of the unsigned integer stencil values
updated by the stencil test in the stencil framebuffer attachment.
reference
is an integer reference value that is used in the
unsigned stencil comparison.
There are two sets of stencil-related state, the front stencil state set and
the back stencil state set. Stencil tests and writes use the front set of
stencil state when processing fragments rasterized from non-polygon
primitives (points and lines) and front-facing polygon primitives while the
back set of stencil state is used when processing fragments rasterized from
back-facing polygon primitives. For the purposes of stencil testing, a
primitive is still considered a polygon even if the polygon is to be
rasterized as points or lines due to the current VkPolygonMode
.
Whether a polygon is front- or back-facing is determined in the same manner
used for face culling (see Basic Polygon Rasterization).
The operation of the stencil test is also affected by the
compareMask
, writeMask
, and reference
members of VkStencilOpState
set in the pipeline state object if the
pipeline state object is created without the
VK_DYNAMIC_STATE_STENCIL_COMPARE_MASK
,
VK_DYNAMIC_STATE_STENCIL_WRITE_MASK
, and
VK_DYNAMIC_STATE_STENCIL_REFERENCE
dynamic states enabled,
respectively.
If the pipeline state object is created with the
VK_DYNAMIC_STATE_STENCIL_COMPARE_MASK
dynamic state enabled, then to
dynamically set the stencil compare mask call:
void vkCmdSetStencilCompareMask( VkCommandBuffer commandBuffer, VkStencilFaceFlags faceMask, uint32_t compareMask);
commandBuffer
is the command buffer into which the command will be
recorded.
faceMask
is a bitmask specifying the set of stencil state for
which to update the compare mask. Bits which can be set include:
typedef enum VkStencilFaceFlagBits { VK_STENCIL_FACE_FRONT_BIT = 0x00000001, VK_STENCIL_FACE_BACK_BIT = 0x00000002, VK_STENCIL_FRONT_AND_BACK = 0x00000003, } VkStencilFaceFlagBits;
VK_STENCIL_FACE_FRONT_BIT
indicates that only the front set of
stencil state is updated.
VK_STENCIL_FACE_BACK_BIT
indicates that only the back set of
stencil state is updated.
VK_STENCIL_FRONT_AND_BACK
is the combination of
VK_STENCIL_FACE_FRONT_BIT
and VK_STENCIL_FACE_BACK_BIT
and
indicates that both sets of stencil state are updated.
compareMask
is the new value to use as the stencil compare mask.
If the pipeline state object is created with the
VK_DYNAMIC_STATE_STENCIL_WRITE_MASK
dynamic state enabled, then to
dynamically set the stencil write mask call:
void vkCmdSetStencilWriteMask( VkCommandBuffer commandBuffer, VkStencilFaceFlags faceMask, uint32_t writeMask);
commandBuffer
is the command buffer into which the command will be
recorded.
faceMask
is a bitmask of VkStencilFaceFlagBits
specifying
the set of stencil state for which to update the write mask, as
described above for vkCmdSetStencilCompareMask
.
writeMask
is the new value to use as the stencil write mask.
If the pipeline state object is created with the
VK_DYNAMIC_STATE_STENCIL_REFERENCE
dynamic state enabled, then to
dynamically set the stencil reference value call:
void vkCmdSetStencilReference( VkCommandBuffer commandBuffer, VkStencilFaceFlags faceMask, uint32_t reference);
commandBuffer
is the command buffer into which the command will be
recorded.
faceMask
is a bitmask of VkStencilFaceFlagBits
specifying
the set of stencil state for which to update the reference value, as
described above for vkCmdSetStencilCompareMask
.
reference
is the new value to use as the stencil reference value.
reference
is an integer reference value that is used in the
unsigned stencil comparison. Stencil comparison clamps the reference value
to
$[0,2^s-1]$
, where
$s$
is the number
of bits in the stencil framebuffer attachment. The
$s$
least
significant bits of compareMask
are bitwise ANDed
with
both the reference and the stored stencil value, and the resulting masked
values are those that participate in the comparison controlled by
compareOp
. Let
$R$
be the masked reference value
and
$S$
be the masked stored stencil value.
compareOp
is a symbolic constant that determines the stencil
comparison function:
typedef enum VkCompareOp { VK_COMPARE_OP_NEVER = 0, VK_COMPARE_OP_LESS = 1, VK_COMPARE_OP_EQUAL = 2, VK_COMPARE_OP_LESS_OR_EQUAL = 3, VK_COMPARE_OP_GREATER = 4, VK_COMPARE_OP_NOT_EQUAL = 5, VK_COMPARE_OP_GREATER_OR_EQUAL = 6, VK_COMPARE_OP_ALWAYS = 7, } VkCompareOp;
VK_COMPARE_OP_NEVER
: the test never passes.
VK_COMPARE_OP_LESS
: the test passes when
$R \lt S$
.
VK_COMPARE_OP_EQUAL
: the test passes when
$R = S$
.
VK_COMPARE_OP_LESS_OR_EQUAL
: the test passes when
$R
\leq S$
.
VK_COMPARE_OP_GREATER
: the test passes when
$R \gt S$
.
VK_COMPARE_OP_NOT_EQUAL
: the test passes when
$R \neq
S$
.
VK_COMPARE_OP_GREATER_OR_EQUAL
: the test passes when
$R
\geq S$
.
VK_COMPARE_OP_ALWAYS
: the test always passes.
As described earlier, the failOp
, passOp
, and depthFailOp
members of VkStencilOpState
indicate what happens to the stored
stencil value if this or certain subsequent tests fail or pass. Each enum is
of type VkStencilOp
, which is defined as:
typedef enum VkStencilOp { VK_STENCIL_OP_KEEP = 0, VK_STENCIL_OP_ZERO = 1, VK_STENCIL_OP_REPLACE = 2, VK_STENCIL_OP_INCREMENT_AND_CLAMP = 3, VK_STENCIL_OP_DECREMENT_AND_CLAMP = 4, VK_STENCIL_OP_INVERT = 5, VK_STENCIL_OP_INCREMENT_AND_WRAP = 6, VK_STENCIL_OP_DECREMENT_AND_WRAP = 7, } VkStencilOp;
The possible values are:
VK_STENCIL_OP_KEEP
keeps the current value.
VK_STENCIL_OP_ZERO
sets the value to 0.
VK_STENCIL_OP_REPLACE
sets the value to reference
.
VK_STENCIL_OP_INCREMENT_AND_CLAMP
increments the current value and
clamps to the maximum representable unsigned value.
VK_STENCIL_OP_DECREMENT_AND_CLAMP
decrements the current value and
clamps to 0.
VK_STENCIL_OP_INVERT
bitwise-inverts the current value.
VK_STENCIL_OP_INCREMENT_AND_WRAP
increments the current value and
wraps to 0 when the maximum value would have been exceeded.
VK_STENCIL_OP_DECREMENT_AND_WRAP
decrements the current value and
wraps to the maximum possible value when the value would go below 0.
For purposes of increment and decrement, the stencil bits are considered as an unsigned integer.
If the stencil test fails, the sample’s coverage bit is cleared in the fragment. If there is no stencil framebuffer attachment, stencil modification cannot occur, and it is as if the stencil tests always pass.
If the stencil test passes, the writeMask
member of the
VkStencilOpState
structures controls how the updated stencil value is
written to the stencil framebuffer attachment.
The least significant
$s$
bits of writeMask
, where
$s$
is the number of bits in the stencil framebuffer attachment,
specify an integer mask. Where a
$1$
appears in this mask, the
corresponding bit in the stencil value in the depth/stencil attachment is
written; where a
$0$
appears, the bit is not written. The
writeMask
value uses either the front-facing or back-facing state
based on the facing-ness of the fragment. Fragments generated by
front-facing primitives use the front mask and fragments generated by
back-facing primitives use the back mask.