AOMedia AV1 Codec: High-level Algorithm

| AOMedia AV1 Codec

Modules | Functions
High-level Algorithm
Encoder Algorithm

## Modules
| Speed vs Quality Trade Off
| Source Frame Processing
| Rate Control
| Temporal Dependency Modelling
| Two Pass Mode
| The Look-Ahead Buffer
| Golden Frame Group

## Functions
int | av1_pack_bitstream (AV1_COMP *const cpi, uint8_t *dst, size_t *size, int *const largest_tile_id)
| Pack the bitstream for one frame.
int | av1_encode_strategy (AV1_COMP *const cpi, size_t *const size, uint8_t *const dest, unsigned int *frame_flags, int64_t *const time_stamp, int64_t *const time_end, const aom_rational64_t *const timestamp_ratio, int flush)
| Implement high-level encode strategy. More...
static void | encode_frame_internal (AV1_COMP *cpi)
| Encoder setup(only for the current frame), encoding, and recontruction for a single frame.
void | av1_encode_frame (AV1_COMP *cpi)
| Setup reference frame buffers and encode a frame. More...
static void | cdef_restoration_frame (AV1_COMP *cpi, AV1_COMMON *cm, MACROBLOCKD *xd, int use_restoration, int use_cdef)
| Select and apply cdef filters and switchable restoration filters.
static void | loopfilter_frame (AV1_COMP *cpi, AV1_COMMON *cm)
| Select and apply in-loop deblocking filters, cdef filters, and restoration filters.
static int | encode_without_recode (AV1_COMP *cpi)
| Encode a frame without the recode loop, usually used in one-pass encoding and realtime coding. More...
static int | encode_with_recode_loop (AV1_COMP *cpi, size_t *size, uint8_t *dest)
| Recode loop for encoding one frame. the purpose of encoding one frame for multiple times can be approaching a target bitrate or adjusting the usage of global motions. More...
static int | encode_with_recode_loop_and_filter (AV1_COMP *cpi, size_t *size, uint8_t *dest, int64_t *sse, int64_t *rate, int *largest_tile_id)
| Recode loop or a single loop for encoding one frame, followed by in-loop deblocking filters, CDEF filters, and restoration filters. More...
static int | encode_frame_to_data_rate (AV1_COMP *cpi, size_t *size, uint8_t *dest)
| Run the final pass encoding for 1-pass/2-pass encoding mode, and pack the bitstream. More...
int | av1_receive_raw_frame (AV1_COMP *cpi, aom_enc_frame_flags_t frame_flags, YV12_BUFFER_CONFIG *sd, int64_t time_stamp, int64_t end_time_stamp)
| Obtain the raw frame data. More...
int | av1_get_compressed_data (AV1_COMP *cpi, unsigned int *frame_flags, size_t *size, uint8_t *dest, int64_t *time_stamp, int64_t *time_end, int flush, const aom_rational64_t *timebase)
| Encode a frame. More...
int | av1_encode (AV1_COMP *const cpi, uint8_t *const dest, const EncodeFrameInput *const frame_input, const EncodeFrameParams *const frame_params, EncodeFrameResults *const frame_results)
| Run 1-pass/2-pass encoding.

## Detailed Description

This module describes sequence level/frame level algorithm in AV1. More details will be added.

## Function Documentation

## ◆ av1_encode_strategy()

int av1_encode_strategy | ( | AV1_COMP *const | cpi,
|  | size_t *const | size,
|  | uint8_t *const | dest,
|  | unsigned int * | frame_flags,
|  | int64_t *const | time_stamp,
|  | int64_t *const | time_end,
|  | const aom_rational64_t *const | timestamp_ratio,
|  | int | flush
| ) |  |

Implement high-level encode strategy.

This function will implement high-level encode strategy, choosing frame type, frame placement, etc. It populates an EncodeFrameParams struct with the results of these decisions and then encodes the frame. The caller should use the output parameters *time_stamp and *time_end only when this function returns AOM_CODEC_OK.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | size | Bitstream size
  [in] | dest | Bitstream output
  [in] | frame_flags | Flags to decide how to encoding the frame
  [out] | time_stamp | Time stamp of the frame
  [out] | time_end | Time end
  [in] | timestamp_ratio | Time base
  [in] | flush | Decide to encode one frame or the rest of frames

Returns
  Returns a value to indicate if the encoding is done successfully.

Return values

  AOM_CODEC_OK |
  -1 |
  AOM_CODEC_ERROR |

References AOM_CODEC_ERROR, AOM_CODEC_OK, AOM_EFLAG_FORCE_KF, av1_cyclic_refresh_update_parameters(), av1_encode(), av1_get_one_pass_rt_params(), av1_get_second_pass_params(), av1_lookahead_depth(), av1_lookahead_peek(), av1_lookahead_pop_sz(), av1_set_reference_structure_one_pass_rt(), av1_twopass_postencode_update(), AV1_COMP::common, AV1_COMP::compressor_stage, AV1Common::cur_frame, AV1Common::current_frame, AV1_COMP::ext_flags, AV1Common::film_grain_params, AV1_COMP::film_grain_table, TimeStamps::first_ts_start, AV1Common::frame_presentation_time, EncodeFrameParams::frame_type, RATE_CONTROL::frames_to_key, KeyFrameCfg::fwd_kf_enabled, AV1_COMP::gf_frame_index, AV1_COMP::gf_group, has_no_stats_stage(), KeyFrameCfg::key_freq_max, AV1EncoderConfig::kf_cfg, AV1_PRIMARY::lookahead, AV1Common::number_spatial_layers, AV1Common::number_temporal_layers, AV1_COMP::oxcf, AV1EncoderConfig::pass, AV1_COMP::ppi, TimeStamps::prev_ts_end, AV1Common::quant_params, AV1_COMP::rc, AV1EncoderConfig::rc_cfg, AV1_COMP::ref_buffer_stack, ExternalFlags::ref_frame_flags, EncodeFrameParams::ref_frame_flags, ExternalFlags::refresh_frame, AV1_COMP::refresh_frame, EncodeFrameParams::refresh_frame, AV1Common::remapped_ref_idx, EncodeFrameParams::remapped_ref_idx, AV1Common::seq_params, AV1_COMP::sf, AV1_COMP::show_existing_alt_ref, AV1Common::show_existing_frame, EncodeFrameParams::show_frame, TplParams::skip_tpl_setup_stats, EncodeFrameParams::speed, AV1_COMP::svc, AV1_COMP::td, AV1_COMP::time_stamps, AV1_COMP::tpl_data, AV1_COMP::twopass, AV1_COMP::unfiltered_source, ExtRefreshFrameFlagsInfo::update_pending, and CommonQuantParams::using_qmatrix.

Referenced by av1_get_compressed_data().

## ◆ av1_encode_frame()

void av1_encode_frame | ( | AV1_COMP * | cpi | ) |

Setup reference frame buffers and encode a frame.

Parameters

  [in] | cpi | Top-level encoder structure

References AV1_COMP::common, AV1Common::cur_frame, AV1Common::current_frame, AV1_COMP::enc_seg, encode_frame_internal(), AV1Common::features, SPEED_FEATURES::hl_sf, FeatureFlags::interp_filter, CommonTileParams::large_scale, EncSegmentationInfo::map, CommonModeInfoParams::mi_cols, AV1Common::mi_params, CommonModeInfoParams::mi_rows, AV1_COMP::oxcf, FeatureFlags::reduced_tx_set_used, AV1_COMP::ref_frame_dist_info, AV1_COMP::ref_frame_flags, AV1Common::seg, AV1_COMP::sf, FeatureFlags::switchable_motion_mode, AV1_COMP::td, AV1Common::tiles, and FeatureFlags::tx_mode.

## ◆ encode_without_recode()

static int encode_without_recode | ( | AV1_COMP * | cpi | ) |
| static

Encode a frame without the recode loop, usually used in one-pass encoding and realtime coding.

Parameters

  [in] | cpi | Top-level encoder structure

Returns
  Returns a value to indicate if the encoding is done successfully.

Return values

  AOM_CODEC_OK |
  AOM_CODEC_ERROR |

References AV1_COMP::common, AV1Common::current_frame, SVC::downsample_filter_phase, SVC::downsample_filter_type, AV1Common::height, ResizePendingParams::height, AV1_COMP::oxcf, SPEED_FEATURES::part_sf, AV1_COMP::resize_pending_params, AV1_COMP::sf, AV1_COMP::svc, AV1_COMP::unscaled_source, AV1_COMP::use_svc, AV1Common::width, and ResizePendingParams::width.

Referenced by encode_with_recode_loop_and_filter().

## ◆ encode_with_recode_loop()

static int encode_with_recode_loop | ( | AV1_COMP * | cpi,
|  | size_t * | size,
|  | uint8_t * | dest
| ) |  |
| static

Recode loop for encoding one frame. the purpose of encoding one frame for multiple times can be approaching a target bitrate or adjusting the usage of global motions.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | size | Bitstream size
  [in] | dest | Bitstream output

Returns
  Returns a value to indicate if the encoding is done successfully.

Return values

  AOM_CODEC_OK |
  -1 |
  AOM_CODEC_ERROR |

References AOM_SUPERRES_NONE, AV1_COMP::common, AV1Common::current_frame, HIGH_LEVEL_SPEED_FEATURES::disable_extra_sc_testing, AV1_COMP::gm_info, SPEED_FEATURES::hl_sf, InterpSearchFlags::interp_filter_search_mask, AV1_COMP::interp_search_flags, SPEED_FEATURES::interp_sf, RateControlCfg::min_cr, AV1_COMP::oxcf, SPEED_FEATURES::part_sf, AV1_COMP::rc, AV1EncoderConfig::rc_cfg, HIGH_LEVEL_SPEED_FEATURES::recode_loop, AV1_COMP::sf, AV1_COMP::source, AV1_COMP::superres_mode, and AV1Common::superres_scale_denominator.

Referenced by encode_with_recode_loop_and_filter().

## ◆ encode_with_recode_loop_and_filter()

static int encode_with_recode_loop_and_filter | ( | AV1_COMP * | cpi,
|  | size_t * | size,
|  | uint8_t * | dest,
|  | int64_t * | sse,
|  | int64_t * | rate,
|  | int * | largest_tile_id
| ) |  |
| static

Recode loop or a single loop for encoding one frame, followed by in-loop deblocking filters, CDEF filters, and restoration filters.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | size | Bitstream size
  [in] | dest | Bitstream output
  [in] | sse | Total distortion of the frame
  [in] | rate | Total rate of the frame
  [in] | largest_tile_id | Tile id of the last tile

Returns
  Returns a value to indicate if the encoding is done successfully.

Return values

  AOM_CODEC_OK |
  AOM_CODEC_ERROR |

References FeatureFlags::allow_intrabc, AV1_COMP::ambient_err, AOM_CODEC_OK, CdefInfo::cdef_bits, AV1Common::cdef_info, CdefInfo::cdef_strengths, CdefInfo::cdef_uv_strengths, AV1_COMP::common, AV1Common::cur_frame, encode_with_recode_loop(), encode_without_recode(), AV1Common::features, RestorationInfo::frame_restoration_type, RATE_CONTROL::frames_to_key, SPEED_FEATURES::hl_sf, AV1Common::lf, loopfilter_frame(), CdefInfo::nb_cdef_strengths, AV1_COMP::oxcf, AV1_COMP::rc, HIGH_LEVEL_SPEED_FEATURES::recode_loop, AV1Common::render_height, AV1Common::render_width, RESTORE_NONE, AV1Common::rst_info, AV1Common::seq_params, AV1_COMP::sf, and AV1_COMP::source.

## ◆ encode_frame_to_data_rate()

static int encode_frame_to_data_rate | ( | AV1_COMP * | cpi,
|  | size_t * | size,
|  | uint8_t * | dest
| ) |  |
| static

Run the final pass encoding for 1-pass/2-pass encoding mode, and pack the bitstream.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | size | Bitstream size
  [in] | dest | Bitstream output

Returns
  Returns a value to indicate if the encoding is done successfully.

Return values

  AOM_CODEC_OK |
  AOM_CODEC_ERROR |

References FeatureFlags::allow_ref_frame_mvs, FeatureFlags::allow_screen_content_tools, FeatureFlags::allow_warped_motion, AOM_CBR, AOM_CODEC_ERROR, AOM_CODEC_OK, av1_pack_bitstream(), RATE_CONTROL::coefficient_size, AV1_COMP::common, AV1Common::cur_frame, FeatureFlags::cur_frame_force_integer_mv, AV1Common::current_frame, AV1_COMP::existing_fb_idx_to_show, AV1Common::features, AV1_COMP::force_intpel_info, AV1_COMP::frame_index_set, AV1_COMP::gf_group, has_no_stats_stage(), AV1Common::height, CommonTileParams::large_scale, AV1_COMP::last_frame_type, AV1_COMP::last_show_frame_buf, AV1Common::lf, RateControlCfg::mode, AV1_COMP::num_tg, AV1_COMP::oxcf, AV1_COMP::ppi, AV1_COMP::rc, AV1EncoderConfig::rc_cfg, AV1Common::ref_frame_id, AV1Common::seg, AV1Common::seq_params, AV1_PRIMARY::seq_params_locked, AV1Common::show_frame, CommonTileParams::single_tile_decoding, AV1_COMP::source, AV1Common::tiles, AV1_COMP::unscaled_last_source, and AV1Common::width.

Referenced by av1_encode().

## ◆ av1_receive_raw_frame()

int av1_receive_raw_frame | ( | AV1_COMP * | cpi,
|  | aom_enc_frame_flags_t | frame_flags,
|  | YV12_BUFFER_CONFIG * | sd,
|  | int64_t | time_stamp,
|  | int64_t | end_time_stamp
| ) |  |

Obtain the raw frame data.

This function receives the raw frame data from input.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | frame_flags | Flags to decide how to encoding the frame
  [in] | sd | Contain raw frame data
  [in] | time_stamp | Time stamp of the frame
  [in] | end_time_stamp | End time stamp

Returns
  Returns a value to indicate if the frame data is received successfully.

Note
  The caller can assume that a copy of this frame is made and not just a copy of the pointer.

References AV1_COMP::common, AV1_COMP::oxcf, and AV1Common::seq_params.

## ◆ av1_get_compressed_data()

int av1_get_compressed_data | ( | AV1_COMP * | cpi,
|  | unsigned int * | frame_flags,
|  | size_t * | size,
|  | uint8_t * | dest,
|  | int64_t * | time_stamp,
|  | int64_t * | time_end,
|  | int | flush,
|  | const aom_rational64_t * | timebase
| ) |  |

Encode a frame.

This function encodes the raw frame data, and outputs the frame bit stream to the designated buffer. The caller should use the output parameters *time_stamp and *time_end only when this function returns AOM_CODEC_OK.

Parameters

  [in] | cpi | Top-level encoder structure
  [in] | frame_flags | Flags to decide how to encoding the frame
  [in] | size | Bitstream size
  [in] | dest | Bitstream output
  [out] | time_stamp | Time stamp of the frame
  [out] | time_end | Time end
  [in] | flush | Decide to encode one frame or the rest of frames
  [in] | timebase | Time base used

Returns
  Returns a value to indicate if the encoding is done successfully.

Return values

  AOM_CODEC_OK |
  -1 | No frame encoded; more input is required.
  AOM_CODEC_ERROR |

References AOM_CODEC_ERROR, AOM_CODEC_OK, av1_encode_strategy(), av1_one_pass_cbr_svc_start_layer(), AV1_COMP::b_calculate_psnr, AV1_COMP::common, AV1Common::current_frame, AV1Common::features, AV1_COMP::level_params, AV1_COMP::no_show_fwd_kf, AV1Common::number_spatial_layers, AV1_COMP::oxcf, AV1EncoderConfig::pass, FeatureFlags::refresh_frame_context, AV1Common::show_existing_frame, AV1Common::show_frame, AV1Common::showable_frame, AV1_COMP::td, and AV1_COMP::use_svc.