Tasks

There are three challenge tracks:

Image compression
Task

The goal is to compress a corpus of images to three bitrates:

  • 0.3bpp (bits per pixel)
  • 0.15bpp
  • 0.075bpp

The winners will be chosen based on a human rating task. The raters compare the images to the uncompressed image and chose the preferred codec.

For guidance, several objective metrics will be shown on the leaderboard but not considered for prizes.

Data

We provide a validation set that reflects the (hidden) test data. Validation set statistics:

  • 30 images, collected from Unsplash.
  • Provided as png files.
  • Images resized such that the longer side is 2048.
  • Download Here (250MB). Note: Updated March 3rd to reduce from 300 to 30 images.
Test set:

The images are released using the Unsplash license. Please contact the organiziers if you have objections to any of the released pictures.

Submission Guidelines
  • The decoder size should not exceed 250MB.
Video compression
Task

The goal is to compress a corpus of videos to an average bitrate of

  • 1 mbps (mega bits per second, i.e., 1 000 000 bits per second, so, e.g., for a corpus of 100 10second videos, you would use 1 000 000 000 bits.)
  • 0.1 mbps

The winners will be chosen based on a human rating task. The raters compare the compressed videos to the uncompressed videos and chose the preferred codec.

For guidance, several objective metrics will be shown on the leaderboard but not considered for prizes.

Data

We provide a validation set that reflects the (hidden) test data. Validation set statistics:

Test set:

Please contact the organiziers if you have objections to any of the released videos.

Submission Guidelines
  • The decoder size should not exceed 1GB.
  • IMPORTANT: Submissions only need to compress 30 videos of the validation set, see above.
  • We provide the videos as mp4, but we will expect submissions to output three PNGs per frame: one for Y, U, and V channels. Please use the devkit for converting to PNGs, if you want to train on PNGs.
    • Or look at the Example Submission below.
Example Submission
The devkit contains an example submission based on H264. It showcases the format expected by the server.
Perceptual metrics

In the perceptual metric track you will need to design a metric to rank the participants of the image compression task. Given a pair of images (one being the original and the other being a distorted image) your metric will need to generate a score. We can compare methods A and B by generating scores two scores d(O, A) and d(O, B), where O is the original image. If d(O, A) < d(O, B), the metric prefers method A over method B.

To evaluate your metric, we will compare the metric's preferences with the preferences of human raters. In our image compression evaluation, human raters are presented with three images (O, A, B) and asked to pick one of A or B. The winner of the perceptual metric challenge is chosen based on who predicted the correct preference the largest number of times.

Development Kit

The development kit is provided as an example of what is expected from participants. It is in no way intended to contain data representative of the final challenge simply because that is not possible. The final test set will be created from the files uploaded by the participants in the compression challenge, and as a result it’s simply impossible for us to provide data which will match that distribution in the validation set.

To get the devkit, use:

git clone https://github.com/fab-jul/clic2021-devkit.git

Data

The full details of the tasks are contained in the README file of the repository.

The 2021 test data may be used to train new models for this year's competition. Please note that the link above contains pointers to the raw data (unfiltered) which we collected from raters. Therefore, it is possible that certain pairs of images might have been rated both ways (i.e., one rater might have disagreed with the other). It is up to the participants to do any data cleanup that might be necessary.

Sponsored by