X264 TODO

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: Dark Shikari's pile of junk, Pengvado's pile of junk.

Motion Estimation

• Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
• (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
• Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
• Somehow take into account the effect of motion vector decision on future blocks.
  • Hierarchical motion estimation
  • Approximations from lookahead MVs
  • Iterative ME (as per Snow)
  • Trellis motion estimation

Intra Analysis

• Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
• SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

Mode Decision

• Can we find more ways to skip more motion searches in multiref?
• On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
• Chroma-aware mode decision for B-frames?
• See the TODOs for deblock-aware RD in common/deblock.c.

Psy

• Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
• Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
• Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
• Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
• RD should take into account local variance.
• Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
• Lambda should be picked independent of quantizer (i.e. with greater precision).
• Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
  • If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).

Lookahead

• Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
• Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
• Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

Quantization

• CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
• There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
• Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
• Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.

Interlacing

• Finish adaptive MBAFF. Talk to horlicks about this one.
• Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.

Weighted Prediction

• Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
• Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
• Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
• Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

Ratecontrol

• VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
  • Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
• Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
• Make the frame size predictors better. They currently are kind of crappy.
• Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
• 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.

GPU

• Motion estimation?
  • Methods
    • Hierarchical?
    • 2D Wave?
    • Something else?
  • "Easy": lookahead motion estimation
    • Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
  • "Hard": main motion estimation
    • Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
    • But potentially more useful...
• Other things?

x264CLI

• Finish audio support. Talk to Kovensky about this one.
• Add more filters.
  • Deinterlacers (YADIF).
  • Denoisers (HQDN3D?).
  • IVTC, decomb?