VK_EXT_image_drm_format_modifier(3)

# VK_EXT_image_drm_format_modifier(3) Manual Page

## Name

VK_EXT_image_drm_format_modifier - device extension

Loading… please wait.

## Registered Extension Number

159

## Revision

1

## Extension and Version Dependencies

- Requires Vulkan 1.0

- Requires `VK_KHR_bind_memory2`

- Requires `VK_KHR_get_physical_device_properties2`

- Requires `VK_KHR_image_format_list`

- Requires `VK_KHR_sampler_ycbcr_conversion`

### Other Extension Metadata

Last Modified Date

  2018-08-29

IP Status

  No known IP claims.

Contributors

  - Antoine Labour, Google

  - Bas Nieuwenhuizen, Google

  - Chad Versace, Google

  - James Jones, NVIDIA

  - Jason Ekstrand, Intel

  - Jőrg Wagner, ARM

  - Kristian Høgsberg Kristensen, Google

  - Ray Smith, ARM

### Description

This extension provides the ability to use DRM format modifiers with images, enabling Vulkan to better integrate with the Linux ecosystem of graphics, video, and display APIs.

Its functionality closely overlaps with `EGL_EXT_image_dma_buf_import_modifiers`2^ and `EGL_MESA_image_dma_buf_export`3^. Unlike the EGL extensions, this extension does not require the use of a specific handle type (such as a dma_buf) for external memory and provides more explicit control of image creation.

### Introduction to DRM Format Modifiers

A DRM format modifier is a 64-bit, vendor-prefixed, semi-opaque unsigned integer. Most modifiers represent a concrete, vendor-specific tiling format for images. Some exceptions are `DRM_FORMAT_MOD_LINEAR` (which is not vendor-specific); `DRM_FORMAT_MOD_NONE` (which is an alias of `DRM_FORMAT_MOD_LINEAR` due to historical accident); and `DRM_FORMAT_MOD_INVALID` (which does not represent a tiling format). The modifier’s vendor prefix consists of the 8 most significant bits. The canonical list of modifiers and vendor prefixes is found in `drm_fourcc.h` in the Linux kernel source. The other dominant source of modifiers are vendor kernel trees.

One goal of modifiers in the Linux ecosystem is to enumerate for each vendor a reasonably sized set of tiling formats that are appropriate for images shared across processes, APIs, and/or devices, where each participating component may possibly be from different vendors. A non-goal is to enumerate all tiling formats supported by all vendors. Some tiling formats used internally by vendors are inappropriate for sharing; no modifiers should be assigned to such tiling formats.

Modifier values typically do not describe memory layouts. More precisely, a modifier's lower 56 bits usually have no structure. Instead, modifiers name memory layouts; they name a small set of vendor-preferred layouts for image sharing. As a consequence, in each vendor namespace the modifier values are often sequentially allocated starting at 1.

Each modifier is usually supported by a single vendor and its name matches the pattern `{VENDOR}_FORMAT_MOD_*` or `DRM_FORMAT_MOD_{VENDOR}_*`. Examples are `I915_FORMAT_MOD_X_TILED` and `DRM_FORMAT_MOD_BROADCOM_VC4_T_TILED`. An exception is `DRM_FORMAT_MOD_LINEAR`, which is supported by most vendors.

Many APIs in Linux use modifiers to negotiate and specify the memory layout of shared images. For example, a Wayland compositor and Wayland client may, by relaying modifiers over the Wayland protocol `zwp_linux_dmabuf_v1`, negotiate a vendor-specific tiling format for a shared `wl_buffer`. The client may allocate the underlying memory for the `wl_buffer` with GBM, providing the chosen modifier to `gbm_bo_create_with_modifiers`. The client may then import the `wl_buffer` into Vulkan for producing image content, providing the resource’s dma_buf to VkImportMemoryFdInfoKHR and its modifier to VkImageDrmFormatModifierExplicitCreateInfoEXT. The compositor may then import the `wl_buffer` into OpenGL for sampling, providing the resource’s dma_buf and modifier to `eglCreateImage`. The compositor may also bypass OpenGL and submit the `wl_buffer` directly to the kernel’s display API, providing the dma_buf and modifier through `drm_mode_fb_cmd2`.

### Format Translation

Modifier-capable APIs often pair modifiers with DRM formats, which are defined in `drm_fourcc.h`. However, `VK_EXT_image_drm_format_modifier` uses VkFormat instead of DRM formats. The application must convert between VkFormat and DRM format when it sends or receives a DRM format to or from an external API.

The mapping from VkFormat to DRM format is lossy. Therefore, when receiving a DRM format from an external API, often the application must use information from the external API to accurately map the DRM format to a VkFormat. For example, DRM formats do not distinguish between RGB and sRGB (as of 2018-03-28); external information is required to identify the image’s colorspace.

The mapping between VkFormat and DRM format is also incomplete. For some DRM formats there exist no corresponding Vulkan format, and for some Vulkan formats there exist no corresponding DRM format.

### Usage Patterns

Three primary usage patterns are intended for this extension:

- Negotiation. The application negotiates with modifier-aware, external components to determine sets of image creation parameters supported among all components.

  In the Linux ecosystem, the negotiation usually assumes the image is a 2D, single-sampled, non-mipmapped, non-array image; this extension permits that assumption but does not require it. The result of the negotiation usually resembles a set of tuples such as (drmFormat, drmFormatModifier), where each participating component supports all tuples in the set.

  Many details of this negotiation—such as the protocol used during negotiation, the set of image creation parameters expressable in the protocol, and how the protocol chooses which process and which API will create the image—are outside the scope of this specification.

  In this extension, vkGetPhysicalDeviceFormatProperties2 with VkDrmFormatModifierPropertiesListEXT serves a primary role during the negotiation, and vkGetPhysicalDeviceImageFormatProperties2 with VkPhysicalDeviceImageDrmFormatModifierInfoEXT serves a secondary role.

- Import. The application imports an image with a modifier.

  In this pattern, the application receives from an external source the image’s memory and its creation parameters, which are often the result of the negotiation described above. Some image creation parameters are implicitly defined by the external source; for example, `VK_IMAGE_TYPE_2D` is often assumed. Some image creation parameters are usually explicit, such as the image’s `format`, `drmFormatModifier`, and `extent`; and each plane’s `offset` and `rowPitch`.

  Before creating the image, the application first verifies that the physical device supports the received creation parameters by querying vkGetPhysicalDeviceFormatProperties2 with VkDrmFormatModifierPropertiesListEXT and vkGetPhysicalDeviceImageFormatProperties2 with VkPhysicalDeviceImageDrmFormatModifierInfoEXT. Then the application creates the image by chaining VkImageDrmFormatModifierExplicitCreateInfoEXT and VkExternalMemoryImageCreateInfo onto VkImageCreateInfo.

- Export. The application creates an image and allocates its memory. Then the application exports to modifier-aware consumers the image’s memory handles; its creation parameters; its modifier; and the `offset`, `size`, and `rowPitch` of each memory plane.

  In this pattern, the Vulkan device is the authority for the image; it is the allocator of the image’s memory and the decider of the image’s creation parameters. When choosing the image’s creation parameters, the application usually chooses a tuple (format, drmFormatModifier) from the result of the negotiation described above. The negotiation’s result often contains multiple tuples that share the same format but differ in their modifier. In this case, the application should defer the choice of the image’s modifier to the Vulkan implementation by providing all such modifiers to VkImageDrmFormatModifierListCreateInfoEXT::`pDrmFormatModifiers`; and the implementation should choose from `pDrmFormatModifiers` the optimal modifier in consideration with the other image parameters.

  The application creates the image by chaining VkImageDrmFormatModifierListCreateInfoEXT and VkExternalMemoryImageCreateInfo onto VkImageCreateInfo. The protocol and APIs by which the application will share the image with external consumers will likely determine the value of VkExternalMemoryImageCreateInfo::`handleTypes`. The implementation chooses for the image an optimal modifier from VkImageDrmFormatModifierListCreateInfoEXT::`pDrmFormatModifiers`. The application then queries the implementation-chosen modifier with vkGetImageDrmFormatModifierPropertiesEXT, and queries the memory layout of each plane with vkGetImageSubresourceLayout.

  The application then allocates the image’s memory with VkMemoryAllocateInfo, adding chained extending structures for external memory; binds it to the image; and exports the memory, for example, with vkGetMemoryFdKHR.

  Finally, the application sends the image’s creation parameters, its modifier, its per-plane memory layout, and the exported memory handle to the external consumers. The details of how the application transmits this information to external consumers is outside the scope of this specification.

### Prior Art

Extension `EGL_EXT_image_dma_buf_import`1^ introduced the ability to create an `EGLImage` by importing for each plane a dma_buf, offset, and row pitch.

Later, extension `EGL_EXT_image_dma_buf_import_modifiers`2^ introduced the ability to query which combination of formats and modifiers the implementation supports and to specify modifiers during creation of the `EGLImage`.

Extension `EGL_MESA_image_dma_buf_export`3^ is the inverse of `EGL_EXT_image_dma_buf_import_modifiers`.

The Linux kernel modesetting API (KMS), when configuring the display’s framebuffer with `struct drm_mode_fb_cmd2`4^, allows one to specify the frambuffer’s modifier as well as a per-plane memory handle, offset, and row pitch.

GBM, a graphics buffer manager for Linux, allows creation of a `gbm_bo` (that is, a graphics buffer object) by importing data similar to that in `EGL_EXT_image_dma_buf_import_modifiers`1^; and symmetrically allows exporting the same data from the `gbm_bo`. See the references to modifier and plane in `gbm.h`5^.

### New Commands

- vkGetImageDrmFormatModifierPropertiesEXT

### New Structures

- VkDrmFormatModifierPropertiesEXT

- VkImageDrmFormatModifierPropertiesEXT

- Extending VkFormatProperties2:

  - VkDrmFormatModifierPropertiesListEXT

- Extending VkImageCreateInfo:

  - VkImageDrmFormatModifierExplicitCreateInfoEXT

  - VkImageDrmFormatModifierListCreateInfoEXT

- Extending VkPhysicalDeviceImageFormatInfo2:

  - VkPhysicalDeviceImageDrmFormatModifierInfoEXT

### New Enum Constants

- `VK_EXT_IMAGE_DRM_FORMAT_MODIFIER_EXTENSION_NAME`

- `VK_EXT_IMAGE_DRM_FORMAT_MODIFIER_SPEC_VERSION`

- Extending VkImageAspectFlagBits:

  - `VK_IMAGE_ASPECT_MEMORY_PLANE_0_BIT_EXT`

  - `VK_IMAGE_ASPECT_MEMORY_PLANE_1_BIT_EXT`

  - `VK_IMAGE_ASPECT_MEMORY_PLANE_2_BIT_EXT`

  - `VK_IMAGE_ASPECT_MEMORY_PLANE_3_BIT_EXT`

- Extending VkImageTiling:

  - `VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT`

- Extending VkResult:

  - `VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT`

- Extending VkStructureType:

  - `VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT`

  - `VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_EXPLICIT_CREATE_INFO_EXT`

  - `VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT`

  - `VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_PROPERTIES_EXT`

  - `VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT`

### Issues

1) Should this extension define a single DRM format modifier per `VkImage`? Or define one per plane?

+

RESOLVED: There exists a single DRM format modifier per `VkImage`.

DISCUSSION: Prior art, such as `EGL_EXT_image_dma_buf_import_modifiers`2^, `struct drm_mode_fb_cmd2`4^, and `struct gbm_import_fd_modifier_data`5^, allows defining one modifier per plane. However, developers of the GBM and kernel APIs concede it was a mistake. Beginning in Linux 4.10, the kernel requires that the application provide the same DRM format modifier for each plane. (See Linux commit bae781b259269590109e8a4a8227331362b88212). And GBM provides an entry point, `gbm_bo_get_modifier`, for querying the modifier of the image but does not provide one to query the modifier of individual planes.

2) When creating an image with VkImageDrmFormatModifierExplicitCreateInfoEXT, which is typically used when importing an image, should the application explicitly provide the size of each plane?

+

RESOLVED: No. The application must not provide the size. To enforce this, the API requires that VkImageDrmFormatModifierExplicitCreateInfoEXT::`pPlaneLayouts->size` must be 0.

DISCUSSION: Prior art, such as `EGL_EXT_image_dma_buf_import_modifiers`2^, `struct drm_mode_fb_cmd2`4^, and `struct gbm_import_fd_modifier_data`5^, omits from the API the size of each plane. Instead, the APIs infer each plane’s size from the import parameters, which include the image’s pixel format and a dma_buf, offset, and row pitch for each plane.

However, Vulkan differs from EGL and GBM with regards to image creation in the following ways:

Differences in Image Creation

- Undedicated allocation by default. When importing or exporting a set of dma_bufs as an `EGLImage` or `gbm_bo`, common practice mandates that each dma_buf’s memory be dedicated (in the sense of `VK_KHR_dedicated_allocation`) to the image (though not necessarily dedicated to a single plane). In particular, neither the GBM documentation nor the EGL extension specifications explicitly state this requirement, but in light of common practice this is likely due to under-specification rather than intentional omission. In contrast, `VK_EXT_image_drm_format_modifier` permits, but does not require, the implementation to require dedicated allocations for images created with `VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT`.

- Separation of image creation and memory allocation. When importing a set of dma_bufs as an `EGLImage` or `gbm_bo`, EGL and GBM create the image resource and bind it to memory (the dma_bufs) simultaneously. This allows EGL and GBM to query each dma_buf’s size during image creation. In Vulkan, image creation and memory allocation are independent unless a dedicated allocation is used (as in `VK_KHR_dedicated_allocation`). Therefore, without requiring dedicated allocation, Vulkan cannot query the size of each dma_buf (or other external handle) when calculating the image’s memory layout. Even if dedication allocation were required, Vulkan cannot calculate the image’s memory layout until after the image is bound to its dma_ufs.

The above differences complicate the potential inference of plane size in Vulkan. Consider the following problematic cases:

Problematic Plane Size Calculations

- Padding. Some plane of the image may require implementation-dependent padding.

- Metadata. For some modifiers, the image may have a metadata plane which requires a non-trivial calculation to determine its size.

- Mipmapped, array, and 3D images. The implementation may support `VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT` for images whose `mipLevels`, `arrayLayers`, or `depth` is greater than 1. For such images with certain modifiers, the calculation of each plane’s size may be non-trivial.

However, an application-provided plane size solves none of the above problems.

For simplicity, consider an external image with a single memory plane. The implementation is obviously capable calculating the image’s size when its tiling is `VK_IMAGE_TILING_OPTIMAL`. Likewise, any reasonable implementation is capable of calculating the image’s size when its tiling uses a supported modifier.

Suppose that the external image’s size is smaller than the implementation-calculated size. If the application provided the external image’s size to vkCreateImage, the implementation would observe the mismatched size and recognize its inability to comprehend the external image’s layout (unless the implementation used the application-provided size to select a refinement of the tiling layout indicated by the modifier, which is strongly discouraged). The implementation would observe the conflict, and reject image creation with `VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT`. On the other hand, if the application did not provide the external image’s size to vkCreateImage, then the application would observe after calling vkGetImageMemoryRequirements that the external image’s size is less than the size required by the implementation. The application would observe the conflict and refuse to bind the `VkImage` to the external memory. In both cases, the result is explicit failure.

Suppose that the external image’s size is larger than the implementation-calculated size. If the application provided the external image’s size to vkCreateImage, for reasons similar to above the implementation would observe the mismatched size and recognize its inability to comprehend the image data residing in the extra size. The implementation, however, must assume that image data resides in the entire size provided by the application. The implementation would observe the conflict and reject image creation with `VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT`. On the other hand, if the application did not provide the external image’s size to vkCreateImage, then the application would observe after calling vkGetImageMemoryRequirements that the external image’s size is larger than the implementation-usable size. The application would observe the conflict and refuse to bind the `VkImage` to the external memory. In both cases, the result is explicit failure.

Therefore, an application-provided size provides no benefit, and this extension should not require it. This decision renders VkSubresourceLayout::`size` an unused field during image creation, and thus introduces a risk that implementations may require applications to submit sideband creation parameters in the unused field. To prevent implementations from relying on sideband data, this extension requires the application to set `size` to 0.

#### References

1. `EGL_EXT_image_dma_buf_import`

2. `EGL_EXT_image_dma_buf_import_modifiers`

3. `EGL_MESA_image_dma_buf_export`

4. `struct drm_mode_fb_cmd2`

5. `gbm.h`

#### Version History

- Revision 1, 2018-08-29 (Chad Versace)

  - First stable revision

## See Also

VkDrmFormatModifierPropertiesEXT, VkDrmFormatModifierPropertiesListEXT, VkImageDrmFormatModifierExplicitCreateInfoEXT, VkImageDrmFormatModifierListCreateInfoEXT, VkImageDrmFormatModifierPropertiesEXT, VkPhysicalDeviceImageDrmFormatModifierInfoEXT, vkGetImageDrmFormatModifierPropertiesEXT

## Document Notes

For more information, see the Vulkan Specification

This page is a generated document. Fixes and changes should be made to the generator scripts, not directly.

## Copyright

Copyright (c) 2014-2020 The Khronos Group Inc.

SPDX-License-Identifier: CC-BY-4.0