Video Area Home >> Video Codecs Comparisons >> MSU Video Codecs Comparison 2023-2024

MSU Video Codecs Comparisons 2023-2024:
Call for Codecs

#1 codecs comparisons in the world
Video group head: Dr. Dmitriy Vatolin
Project head: Dr. Dmitriy Kulikov
Measurements, analysis: Nickolay Safonov,
Aleksandr Kostychev,
Anna Bigler
compression.ru Lomonosov
Moscow State University (MSU)
Graphics and Media Lab
Dubna International
State University
MSU Institute of Advanced
Studies of Artificial Intelligence
and Intelligent Systems


Important Dates

September, 20 4K comparison application deadline
(The submit system will be opened closer to the deadline)
February, 26 FPGA/Hardware comparison application deadline
December, 25 FullHD comparison application deadline

You can apply for participation using this form:
Or contact us

Navigation


Important updates


  • Only Linux compatible encoders this year
  • Subjective and Objective FullHD parts share
Speed alignment:
  • Only one submission iteration
  • We align the encoders' speed by bitrate on 7-10 publicly available videos that represent the complexity of the set. We will use 5 main objective quality metrics for speed-quality alignment
  • Each encoder should have relative speed less than 100% to reference encoder by all main metrics
Clarity and fairness of the publication
  • For each participant, the following information is fixed before the measurement begins and cannot be updated afterwards:
    • encoder name
    • encoder developer
    • developer contacts
    • publication conditions
  • Each team can only publish one encoder (preset) for each standard
  • We will share for each metric published on the site the place of the encoder (including private ones, without names)

About Annual MSU Video Codecs Comparisons


We perform comparative unbiased analysis for

  • software implementations
  • hardware (GPU-based) implementations
  • cloud-based implementations
of different video coding standards (H.265/HEVC, H.264/MPEG-4 AVC, AV1, VP9, VP8, MPEG-4 and other) using objective metrics (SSIM, VMAF, PSNR and other).

With MSU Codecs Comparison developers can verify the performance of their codec. We share test sequences, encoding parameters and codecs versions so all developers can reproduce the results of the comparison. Participation with publishing of all results is for FREE. Private participation is for fee (contact us for the details).

MSU codecs analysis team has been conducting video codec analysis, testing and optimization since 2004. Some facts about previous MSU Video Codecs Comparisons:
  • There were more than 400.000 downloads of previous H.264/MPEG-4 AVC and H.265/HEVC video codec comparisons reports
  • Many codecs' bugs were found and reported to developers
  • 240+ encoders were tested
  • More than 100 private reports for codec developers (description of codec's weak and strong points) after public report versions
Summary report topics:
  • Objective measurements
  • Subjective analysis
  • Encoding time
  • Speed/Quality trade-off analysis
  • Averaged objective results analysis
  • Leaders in different use cases

2023-2024 Comparison parts and requirements


FullHD Comparison Subjective Comparison 4K Comparison FPGA/Hardware-accelerated/
Ultra-fast Comparison
Use Cases Real-time
VoD
Offline
Real-time
VoD:
Offline
Real-time
VoD
Offline
Ultra-fast
Videos 50 FullHD videos 10 FullHD videos 20 4K videos 50 FullHD videos
Objective metrics Objective metrics: SSIM, PSNR, VMAF
3 color-planes - Y,U,V with different proportions: 4:11, 6:1:1, 8:11, 10:11
Powered by VQMT
Subjective metrics No Yes, powered by: Subjectify.us No No
Target Bitrate FullHD H.264: 1, 2, 4, 6, 8, 10, 12, 15 Mbps
FullHD H.265: 0.7, 1, 2, 3, 4, 6, 8, 10 Mbps
FullHD AV1, VVC: 0.25, 0.5, 1, 2, 3, 4, 6 Mbps
4K H.264: 2, 4, 6, 8, 10, 14, 20 Mbps
4K H.265: 2, 3, 4, 6, 8, 12 Mbps
4K AV1, VVC: 0.5, 1, 2, 3, 4, 5, 6, 8 Mbps
Hardware CPU: Intel Core i7 12700K (Alder Lake)
SSD: 1Tb
RAM: 4x16GB (64GB)
OS: Windows 11 x64, Ubuntu 22.04 LTS
FPGA Xilinx Alveo U250
GPU: NVIDIA Titan RTX,
Intel Arc
Download requirements

New features in MSU Video Codecs Comparison


Since 2022
  • Updated hardware to Alder Lake
  • Quality-alignment instead of speed-alignment
  • Enlarging of video collection
Since 2020
  • Automatic codecs submission system
  • Open to compare Linux-based solutions
  • Open to compare cloud-based solutions
  • Enlarging of video collection
Since 2019
  • Increasing the number of test videos up to 100+, adding UGC (user-generated) videos (vlogs, noise videos, etc.) and HDR videos
  • Open to compare cloud-based solutions
  • Enlarging of video collection
  • Going to use new no-reference metrics
Since 2018
  • Increased target bitrates for comparison on 4K videos
  • Started to use VMAF objective metric
  • Upgraded testing hardware to Coffee Lake
  • Enlarged video collection (15833 videos)

Comparison Rules and Methodology



2023-2024 Comparison methodology in presentation

Please pay attention that we will use multi-core CPU for encoding, so you can use multi-threading
  • Decoding is performed with reference decoder (H.264/AVC, H.265/HEVC, VP9, AV1). For other standards participants should provide decoder
  • We don't limit GOP size and intra-period
  • We don't limit number of passes in the encoding. Total encoding time should fulfill use case speed requirements
Comparison Rules

Applying for participating, you agree with comparison rules.

Codec Requirements
  • Presets for different speed requirements should be provided by the developers
  • Codec should allow to set arbitrary bitrate of resulting stream in constant quality mode
  • Preferable codec interface - console codec version (with batch processing support - bitrate and file names must be possible to assign from the command line)
  • Encoder should be compatible with reference decoder, for encoders of custom standards a decoder should be provided
Developers Deliverables

The following deliverables should be provided by each developer:
  • Codec files (CLI executable file is preferable)
  • Codec's presets
Frequently asked questions (FAQ)

If you have any more questions about rules or methodology please refer to FAQ or contact us at

Take part in 2023-2024 Video Codecs Comparison!

Deepest codecs review: 5 reports, including subjective with 1000+ viewers and 7000+ charts

Our pipeline of codecs testing:
  1. Prepare encoder. After your application, we send you instructions on how to prepare encoding presets for this year testing use cases (there are limitations for speed) and login for a submission system.
  2. Submission. You send us encoder and presets (our hardware characteristics can be found on the comparison page). Testing methodology is described in presentation.
  3. Speed and quality pre-check. We test the speed of your encoder. We let you know if some speedup is needed or any bugs are found. If the requirements are not satisfied, you submit an updated version and we test it again. Finally, when all requirements are satisfied, you confirm the usage of presets and encoder.
  4. Measurements. We perform the main measurements on this year test video set and prepare a draft of the report. This will take about 1-3 months depending on the number of participants.
  5. Draft report and verification. We send you a draft report with your results and the results of other public participants. You can double-check the results. If you participate privately, in this step you will need to decide whether you will publish your results.
  6. Report release. After your confirmation, we prepare and publish the final report on the comparison page. We send you the Enterprise version of the report (with all sequences, charts, etc.).

Thanks


Special thanks to the following contributors of our previous comparisons
Apple Google Intel NVidia
Huawei AMD Adobe Tencent
Zoom video communications VIMEO
VIMEO
Netflix Alibaba
KDDI R&D labs Dolby Tata Elxsi Octasic
Qualcomm Voceweb Elgato Telecast
MainConcept Vitec dicas

Leave a feedback



Contact Information

See all MSU Video Codecs Comparisons

MSU video codecs comparisons resources:


Other Materials

Video resources:

Last updated: 12-May-2022


Server size: 8069 files, 1215Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab