VideoLAN Wiki - User contributions [en]

X264 TODO

2012-06-25T20:51:29Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

=== Motion Estimation ===

*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
**I have a patch for this in the lookahead, but it didn't help much, since it only added predictors.
*Somehow take into account the effect of motion vector decision on future blocks.
**Hierarchical motion estimation
**Approximations from lookahead MVs
**Iterative ME (as per Snow)
**Trellis motion estimation
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
**But how do we know which ones we can afford to skip, and when?
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
**This seems to be awful from my testing, but maybe there's something we can do?
*Try sub-8x8 partitions in B-frames. Is it at all useful?
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
*Fullpel chroma ME?
**For TESA?

=== Intra Analysis ===

*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
**With the SSSE3-based fast intra analysis, we no longer do any early terminations for different modes, at least in SAD/SATD analysis. But there might still be improvements to be made.
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

=== Mode Decision ===

*Can we find more ways to skip more motion searches in multiref?
**A while back, I tried using weaker motion searches on older refs. This helped a bit for speed-vs-compression, but is ironically the opposite of what one wants; older refs will be harder to find good MVs in, and therefore really need better searches.
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
**Can we do something smart by analyzing fenc? It's impossible to tell whether a block is motionless by looking at fdec, but looking at the source pixels is useful. There's still complexity such as lower-QP-than-reference though.
*See the TODOs for deblock-aware RD in common/deblock.c.
**I tried correcting weightp references for deblock RDO, but it didn't help.
**I tried chroma, too, and again, it didn't help measurably.
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
**Doing a merged 4x4/8x8 SATD would help here, but would require new asm.
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
*How about saving CABAC state between each trellis call, rather than basing them all on the CABAC state at the start of the macroblock?
*Make subme=11 not do thresholding in qpel RD and bidir RD.

=== Psy ===

*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
*RD should take into account local variance.
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
*Lambda should be picked independent of quantizer (i.e. with greater precision).
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

=== Lookahead ===

*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?
*B-adapt 1 could be made quite a bit better -- it's important because it's used on all the fast speed modes (and even the defaults). "Harbour 4CIF" is a good example of a clip where it does noticeably badly.

=== Quantization ===

*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
**This is doubly important now, as CABAC trellis has been made way faster, but CAVLC hasn't. Many of the CABAC trellis improvements can be backported.
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
**How useful is this with an entropy coder that doesn't really bias towards zero-runs, as in CABAC?
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

=== Transform ===

*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

=== Interlacing ===

*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
*Constrained intra + adaptive MBAFF. Does anyone care about this?
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

=== Weighted Prediction ===

*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

=== Ratecontrol ===

*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
*Make the frame size and row size predictors better. They currently are kind of crappy.
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
*1-pass ratecontrol often can't adapt fast enough when there are lots of threads (12, 16, 24, etc), especially with smallish VBV buffers. Improve this?
*2-pass VBV is actually a bit more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
**2-pass is still better in the case of many threads, due to the above.
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

=== GPU ===

*Motion estimation?
**Methods
***Hierarchical?
***2D Wave?
***Something else?
**"Easy": lookahead motion estimation
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
**"Hard": main motion estimation
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
***But potentially more useful...
*Other things?
*MultiCoreWare is currently looking on a GPU-based lookahead.

=== x86 assembly ===

*Find holger, lock him in a closet, and make him commit his local patches. It's been years now!
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

=== Other assembly ===

*There's still a lot of unreviewed NEON assembly. Can we please get this in?
*Altivec assembly is very lacking.
*SPARC VIS assembly is only available when high bit-depth is disabled. But nobody cares about SPARC anyways.

=== Other CPU optimizations ===

*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

=== Other features ===

*MPEG-2 encoding support
**[https://github.com/kierank/x262/wiki/TODO x262]
*VP8 encoding support
**[https://github.com/DarkShikari/xvp8 xvp8]
*Support for SMPTE timecodes
*Merge speedcontrol
*Mixed lossless/lossy encoding.
*Segment re-encoding
*Spec compliant AVC-Intra encoding (ask kierank for spec)
**I tried this, but the problem is that the encoders/decoders in reality don't actually abide by their own spec. This is completely insane.

=== x264CLI ===

*Finish audio support. Talk to Kovensky about this one.
*Make the filtering system aware of BT.601 vs BT.709.
*Add more filters.
**Deinterlacers (YADIF).
**Denoisers (HQDN3D?).
**IVTC, decomb?
*Merge L-SMASH mp4 muxer.
*Add TS muxing support using HRD. Talk to kierank about this one.
*Add --device support.
*Add automatic --level restriction support.

X264 TODO

2012-03-22T22:35:31Z

Dark Shikari: /* Mode Decision */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

=== Motion Estimation ===

*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
**I have a patch for this in the lookahead, but it didn't help much, since it only added predictors.
*Somehow take into account the effect of motion vector decision on future blocks.
**Hierarchical motion estimation
**Approximations from lookahead MVs
**Iterative ME (as per Snow)
**Trellis motion estimation
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
**But how do we know which ones we can afford to skip, and when?
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
**This seems to be awful from my testing, but maybe there's something we can do?
*Try sub-8x8 partitions in B-frames. Is it at all useful?
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
*Fullpel chroma ME?
**For TESA?

=== Intra Analysis ===

*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
**With the SSSE3-based fast intra analysis, we no longer do any early terminations for different modes, at least in SAD/SATD analysis. But there might still be improvements to be made.
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

=== Mode Decision ===

*Can we find more ways to skip more motion searches in multiref?
**A while back, I tried using weaker motion searches on older refs. This helped a bit for speed-vs-compression, but is ironically the opposite of what one wants; older refs will be harder to find good MVs in, and therefore really need better searches.
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
**Can we do something smart by analyzing fenc? It's impossible to tell whether a block is motionless by looking at fdec, but looking at the source pixels is useful. There's still complexity such as lower-QP-than-reference though.
*See the TODOs for deblock-aware RD in common/deblock.c.
**I tried correcting weightp references for deblock RDO, but it didn't help.
**I tried chroma, too, and again, it didn't help measurably.
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
**Doing a merged 4x4/8x8 SATD would help here, but would require new asm.
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
*How about saving CABAC state between each trellis call, rather than basing them all on the CABAC state at the start of the macroblock?
*Make subme=11 not do thresholding in qpel RD and bidir RD.

=== Psy ===

*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
*RD should take into account local variance.
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
*Lambda should be picked independent of quantizer (i.e. with greater precision).
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

=== Lookahead ===

*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

=== Quantization ===

*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
**This is doubly important now, as CABAC trellis has been made way faster, but CAVLC hasn't. Many of the CABAC trellis improvements can be backported.
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
**How useful is this with an entropy coder that doesn't really bias towards zero-runs, as in CABAC?
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

=== Transform ===

*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

=== Interlacing ===

*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
*Constrained intra + adaptive MBAFF. Does anyone care about this?
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

=== Weighted Prediction ===

*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

=== Ratecontrol ===

*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
*Make the frame size and row size predictors better. They currently are kind of crappy.
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
*1-pass ratecontrol often can't adapt fast enough when there are lots of threads (12, 16, 24, etc), especially with smallish VBV buffers. Improve this?
*2-pass VBV is actually a bit more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
**2-pass is still better in the case of many threads, due to the above.
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

=== GPU ===

*Motion estimation?
**Methods
***Hierarchical?
***2D Wave?
***Something else?
**"Easy": lookahead motion estimation
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
**"Hard": main motion estimation
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
***But potentially more useful...
*Other things?
*MultiCoreWare is currently looking on a GPU-based lookahead.

=== x86 assembly ===

*Find holger, lock him in a closet, and make him commit his local patches. It's been years now!
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

=== Other assembly ===

*There's still a lot of unreviewed NEON assembly. Can we please get this in?
*Altivec assembly is very lacking.
*SPARC VIS assembly is only available when high bit-depth is disabled. But nobody cares about SPARC anyways.

=== Other CPU optimizations ===

*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

=== Other features ===

*MPEG-2 encoding support
**[https://github.com/kierank/x262/wiki/TODO x262]
*VP8 encoding support
**[https://github.com/DarkShikari/xvp8 xvp8]
*Support for SMPTE timecodes
*Merge speedcontrol
*Mixed lossless/lossy encoding.
*Segment re-encoding
*Spec compliant AVC-Intra encoding (ask kierank for spec)
**I tried this, but the problem is that the encoders/decoders in reality don't actually abide by their own spec. This is completely insane.

=== x264CLI ===

*Finish audio support. Talk to Kovensky about this one.
*Make the filtering system aware of BT.601 vs BT.709.
*Add more filters.
**Deinterlacers (YADIF).
**Denoisers (HQDN3D?).
**IVTC, decomb?
*Merge L-SMASH mp4 muxer.
*Add TS muxing support using HRD. Talk to kierank about this one.
*Add --device support.
*Add automatic --level restriction support.

X264 TODO

2012-03-22T21:17:57Z

Dark Shikari: /* Motion Estimation */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

=== Motion Estimation ===

*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
**I have a patch for this in the lookahead, but it didn't help much, since it only added predictors.
*Somehow take into account the effect of motion vector decision on future blocks.
**Hierarchical motion estimation
**Approximations from lookahead MVs
**Iterative ME (as per Snow)
**Trellis motion estimation
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
**But how do we know which ones we can afford to skip, and when?
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
**This seems to be awful from my testing, but maybe there's something we can do?
*Try sub-8x8 partitions in B-frames. Is it at all useful?
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
*Fullpel chroma ME?
**For TESA?

=== Intra Analysis ===

*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
**With the SSSE3-based fast intra analysis, we no longer do any early terminations for different modes, at least in SAD/SATD analysis. But there might still be improvements to be made.
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

=== Mode Decision ===

*Can we find more ways to skip more motion searches in multiref?
**A while back, I tried using weaker motion searches on older refs. This helped a bit for speed-vs-compression, but is ironically the opposite of what one wants; older refs will be harder to find good MVs in, and therefore really need better searches.
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
**Can we do something smart by analyzing fenc? It's impossible to tell whether a block is motionless by looking at fdec, but looking at the source pixels is useful. There's still complexity such as lower-QP-than-reference though.
*See the TODOs for deblock-aware RD in common/deblock.c.
**I tried correcting weightp references for deblock RDO, but it didn't help.
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
**Doing a merged 4x4/8x8 SATD would help here, but would require new asm.
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
*How about saving CABAC state between each trellis call, rather than basing them all on the CABAC state at the start of the macroblock?
*Make subme=11 not do thresholding in qpel RD and bidir RD.

=== Psy ===

*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
*RD should take into account local variance.
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
*Lambda should be picked independent of quantizer (i.e. with greater precision).
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

=== Lookahead ===

*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

=== Quantization ===

*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
**This is doubly important now, as CABAC trellis has been made way faster, but CAVLC hasn't. Many of the CABAC trellis improvements can be backported.
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
**How useful is this with an entropy coder that doesn't really bias towards zero-runs, as in CABAC?
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

=== Transform ===

*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

=== Interlacing ===

*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
*Constrained intra + adaptive MBAFF. Does anyone care about this?
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

=== Weighted Prediction ===

*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

=== Ratecontrol ===

*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
*Make the frame size and row size predictors better. They currently are kind of crappy.
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
*1-pass ratecontrol often can't adapt fast enough when there are lots of threads (12, 16, 24, etc), especially with smallish VBV buffers. Improve this?
*2-pass VBV is actually a bit more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
**2-pass is still better in the case of many threads, due to the above.
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

=== GPU ===

*Motion estimation?
**Methods
***Hierarchical?
***2D Wave?
***Something else?
**"Easy": lookahead motion estimation
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
**"Hard": main motion estimation
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
***But potentially more useful...
*Other things?
*MultiCoreWare is currently looking on a GPU-based lookahead.

=== x86 assembly ===

*Find holger, lock him in a closet, and make him commit his local patches. It's been years now!
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

=== Other assembly ===

*There's still a lot of unreviewed NEON assembly. Can we please get this in?
*Altivec assembly is very lacking.
*SPARC VIS assembly is only available when high bit-depth is disabled. But nobody cares about SPARC anyways.

=== Other CPU optimizations ===

*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

=== Other features ===

*MPEG-2 encoding support
**[https://github.com/kierank/x262/wiki/TODO x262]
*VP8 encoding support
**[https://github.com/DarkShikari/xvp8 xvp8]
*Support for SMPTE timecodes
*Merge speedcontrol
*Mixed lossless/lossy encoding.
*Segment re-encoding
*Spec compliant AVC-Intra encoding (ask kierank for spec)
**I tried this, but the problem is that the encoders/decoders in reality don't actually abide by their own spec. This is completely insane.

=== x264CLI ===

*Finish audio support. Talk to Kovensky about this one.
*Make the filtering system aware of BT.601 vs BT.709.
*Add more filters.
**Deinterlacers (YADIF).
**Denoisers (HQDN3D?).
**IVTC, decomb?
*Merge L-SMASH mp4 muxer.
*Add TS muxing support using HRD. Talk to kierank about this one.
*Add --device support.
*Add automatic --level restriction support.

X264 TODO

2012-03-22T21:16:51Z

Dark Shikari: More updates

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

=== Motion Estimation ===

*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
*Somehow take into account the effect of motion vector decision on future blocks.
**Hierarchical motion estimation
**Approximations from lookahead MVs
**Iterative ME (as per Snow)
**Trellis motion estimation
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
**But how do we know which ones we can afford to skip, and when?
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
*Try sub-8x8 partitions in B-frames. Is it at all useful?
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
*Fullpel chroma ME?
**For TESA?

=== Intra Analysis ===

*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
**With the SSSE3-based fast intra analysis, we no longer do any early terminations for different modes, at least in SAD/SATD analysis. But there might still be improvements to be made.
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

=== Mode Decision ===

*Can we find more ways to skip more motion searches in multiref?
**A while back, I tried using weaker motion searches on older refs. This helped a bit for speed-vs-compression, but is ironically the opposite of what one wants; older refs will be harder to find good MVs in, and therefore really need better searches.
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
**Can we do something smart by analyzing fenc? It's impossible to tell whether a block is motionless by looking at fdec, but looking at the source pixels is useful. There's still complexity such as lower-QP-than-reference though.
*See the TODOs for deblock-aware RD in common/deblock.c.
**I tried correcting weightp references for deblock RDO, but it didn't help.
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
**Doing a merged 4x4/8x8 SATD would help here, but would require new asm.
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
*How about saving CABAC state between each trellis call, rather than basing them all on the CABAC state at the start of the macroblock?
*Make subme=11 not do thresholding in qpel RD and bidir RD.

=== Psy ===

*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
*RD should take into account local variance.
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
*Lambda should be picked independent of quantizer (i.e. with greater precision).
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

=== Lookahead ===

*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

=== Quantization ===

*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
**This is doubly important now, as CABAC trellis has been made way faster, but CAVLC hasn't. Many of the CABAC trellis improvements can be backported.
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
**How useful is this with an entropy coder that doesn't really bias towards zero-runs, as in CABAC?
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

=== Transform ===

*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

=== Interlacing ===

*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
*Constrained intra + adaptive MBAFF. Does anyone care about this?
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

=== Weighted Prediction ===

*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

=== Ratecontrol ===

*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
*Make the frame size and row size predictors better. They currently are kind of crappy.
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
*1-pass ratecontrol often can't adapt fast enough when there are lots of threads (12, 16, 24, etc), especially with smallish VBV buffers. Improve this?
*2-pass VBV is actually a bit more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
**2-pass is still better in the case of many threads, due to the above.
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

=== GPU ===

*Motion estimation?
**Methods
***Hierarchical?
***2D Wave?
***Something else?
**"Easy": lookahead motion estimation
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
**"Hard": main motion estimation
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
***But potentially more useful...
*Other things?
*MultiCoreWare is currently looking on a GPU-based lookahead.

=== x86 assembly ===

*Find holger, lock him in a closet, and make him commit his local patches. It's been years now!
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

=== Other assembly ===

*There's still a lot of unreviewed NEON assembly. Can we please get this in?
*Altivec assembly is very lacking.
*SPARC VIS assembly is only available when high bit-depth is disabled. But nobody cares about SPARC anyways.

=== Other CPU optimizations ===

*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

=== Other features ===

*MPEG-2 encoding support
**[https://github.com/kierank/x262/wiki/TODO x262]
*VP8 encoding support
**[https://github.com/DarkShikari/xvp8 xvp8]
*Support for SMPTE timecodes
*Merge speedcontrol
*Mixed lossless/lossy encoding.
*Segment re-encoding
*Spec compliant AVC-Intra encoding (ask kierank for spec)
**I tried this, but the problem is that the encoders/decoders in reality don't actually abide by their own spec. This is completely insane.

=== x264CLI ===

*Finish audio support. Talk to Kovensky about this one.
*Make the filtering system aware of BT.601 vs BT.709.
*Add more filters.
**Deinterlacers (YADIF).
**Denoisers (HQDN3D?).
**IVTC, decomb?
*Merge L-SMASH mp4 muxer.
*Add TS muxing support using HRD. Talk to kierank about this one.
*Add --device support.
*Add automatic --level restriction support.

X264 TODO

2012-03-22T21:11:04Z

Dark Shikari: Some updates

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

=== Motion Estimation ===

*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
*Somehow take into account the effect of motion vector decision on future blocks.
**Hierarchical motion estimation
**Approximations from lookahead MVs
**Iterative ME (as per Snow)
**Trellis motion estimation
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
**But how do we know which ones we can afford to skip, and when?
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
*Try sub-8x8 partitions in B-frames. Is it at all useful?
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
*Fullpel chroma ME?
**For TESA?

=== Intra Analysis ===

*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
**With the SSSE3-based fast intra analysis, we no longer do any early terminations for different modes, at least in SAD/SATD analysis. But there might still be improvements to be made.
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

=== Mode Decision ===

*Can we find more ways to skip more motion searches in multiref?
**A while back, I tried using weaker motion searches on older refs. This helped a bit for speed-vs-compression, but is ironically the opposite of what one wants; older refs will be harder to find good MVs in, and therefore really need better searches.
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
**Can we do something smart by analyzing fenc? It's impossible to tell whether a block is motionless by looking at fdec, but looking at the source pixels is useful. There's still complexity such as lower-QP-than-reference though.
*See the TODOs for deblock-aware RD in common/deblock.c.
**I tried correcting weightp references for deblock RDO, but it didn't help.
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
**Doing a merged 4x4/8x8 SATD would help here, but would require new asm.
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
*How about saving CABAC state between each trellis call, rather than basing them all on the CABAC state at the start of the macroblock?
*Make subme=11 not do thresholding in qpel RD and bidir RD.

=== Psy ===

*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
*RD should take into account local variance.
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
*Lambda should be picked independent of quantizer (i.e. with greater precision).
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

=== Lookahead ===

*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

=== Quantization ===

*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

=== Transform ===

*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

=== Interlacing ===

*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
*Constrained intra + adaptive MBAFF. Does anyone care about this?
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

=== Weighted Prediction ===

*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

=== Ratecontrol ===

*VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
**Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
*Make the frame size and row size predictors better. They currently are kind of crappy.
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
*2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

=== GPU ===

*Motion estimation?
**Methods
***Hierarchical?
***2D Wave?
***Something else?
**"Easy": lookahead motion estimation
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
**"Hard": main motion estimation
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
***But potentially more useful...
*Other things?

=== x86 assembly ===

*Optimize more for the Phenom.
*Yell at holger to commit his local patches.
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

=== Other assembly ===

*NEON assembly is nowhere near complete.
**Chroma MC needs to be rewritten for NV12 support.
*Altivec assembly is very lacking.
*SPARC VIS assembly is only available when high bit-depth is disabled.

=== Other CPU optimizations ===

*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

=== Other features ===

*MPEG-2 encoding support
*VP8 encoding support
*Support for SMPTE timecodes
*Merge speedcontrol
*Mixed lossless/lossy encoding.
*Segment re-encoding
*Spec compliant AVC-Intra encoding (ask kierank for spec)

=== x264CLI ===

*Finish audio support. Talk to Kovensky about this one.
*Make the filtering system aware of fullrange vs TV range.
*Make the filtering system aware of BT.601 vs BT.709.
*Add more filters.
**Deinterlacers (YADIF).
**Denoisers (HQDN3D?).
**IVTC, decomb?
*Merge L-SMASH mp4 muxer.
*Add TS muxing support using HRD. Talk to kierank about this one.
*Add --device support.
*Add automatic --level restriction support.

X264 TODO

2011-07-16T00:01:00Z

Dark Shikari: /* x86 assembly */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
* Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Optimize more for the Phenom.
* Yell at holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.
* SPARC VIS assembly is only available when high bit-depth is disabled.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 colorspace support'''
* Support for SMPTE timecodes
* Merge speedcontrol
* Mixed lossless/lossy encoding.

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

X264 TODO

2011-07-15T23:54:43Z

Dark Shikari: /* Weighted Prediction */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
* Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Optimize more for the Phenom.
* Optimize for Sandy Bridge.
* Work more on 10-bit asm.
* If you read this, join #x264dev on irc://irc.freenode.net and ping holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.
* SPARC VIS assembly is only available when high bit-depth is disabled.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 colorspace support'''
* Support for SMPTE timecodes
* Merge speedcontrol
* Mixed lossless/lossy encoding.

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

SoC x264 2011

2011-03-29T23:02:15Z

Dark Shikari: /* Introduction to x264 and Summer of Code */

==Introduction to x264 and Summer of Code==
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.

But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.

x264 is part of the VideoLAN candidature for Google Summer of Code 2011.

*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)

An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.

On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].

==Guide to getting involved==
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.

==Skills needed==
These are required for all listed projects and probably anything not listed, too.
*Basic C programming.
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).

==Projects==
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.

===Optimization (ARM)===
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.

x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.

NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.

===4:4:4 and 4:2:2 colorspaces===
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.

The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.

This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.

===GPU motion estimation===
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:

*High-complexity motion search designed to get useful predictors to be used by the main motion search.
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.

The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.

This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.

===Non-local RD optimization===
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).

This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.

===H.264/MVC===
This project would be to implement the multiview coding extension (read: 3D video) of H.264. This would mostly be a high-level project: affecting headers, reference handling, and so forth, as opposed to low level compression steps. This would let x264 be used to author 3D Blu-rays and encode 3D content more efficiently. (kierank can get Blu-ray 3D streams verified)

This project would involve reading a lot of spec and implementing it in the cleanest way possible. It is probably not inordinately difficult.

==Qualification tasks==
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!

Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.

None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''

===Optimization (ARM)===
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.

===4:4:4 and 4:2:2 Colorspaces===
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.

A partially done qualification task for reference can be found here:
https://github.com/Andreas-HH/x264-colorspaces

===GPU Motion Estimation===
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:

* Set N equal to 2^M, where M is an integer. A common M is 4.
* WHILE N is greater than 1:
:* Downscale the image (from the original) by a factor of N.
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.
:* N = N/2
* Do a final refine at no downscale at all.

===Non-local RD optimization===
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.

===H.264/MVC===
Read the spec and write an outline of what changes you'll have to make to x264 (specifically, with x264 in mind) in order to implement MVC. In other words, a design document. It doesn't have to be very formal, but it needs to clearly demonstrate that you have put some amount of thought into this.

==Contact info==
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.

You should also contact the admin [[User:J-b|jb]].

SoC x264 2011

2011-03-15T20:58:59Z

Dark Shikari: /* 4:4:4 and 4:2:2 Colorspaces */

==Introduction to x264 and Summer of Code==
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.

But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in.

x264 is part of the VideoLAN candidature for Google Summer of Code 2011.

*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)

An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.

On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].

==Guide to getting involved==
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.

==Skills needed==
These are required for all listed projects and probably anything not listed, too.
*Basic C programming.
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).

==Projects==
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.

===Optimization (ARM)===
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.

x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.

NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.

===4:4:4 and 4:2:2 colorspaces===
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.

The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.

This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.

===GPU motion estimation===
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:

*High-complexity motion search designed to get useful predictors to be used by the main motion search.
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.

The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.

This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.

===Non-local RD optimization===
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).

This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.

===H.264/MVC===
This project would be to implement the multiview coding extension (read: 3D video) of H.264. This would mostly be a high-level project: affecting headers, reference handling, and so forth, as opposed to low level compression steps. This would let x264 be used to author 3D Blu-rays and encode 3D content more efficiently. (kierank can get Blu-ray 3D streams verified)

This project would involve reading a lot of spec and implementing it in the cleanest way possible. It is probably not inordinately difficult.

==Qualification tasks==
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!

Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.

None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''

===Optimization (ARM)===
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.

===4:4:4 and 4:2:2 Colorspaces===
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.

A partially done qualification task for reference can be found here:
https://github.com/Andreas-HH/x264-colorspaces

===GPU Motion Estimation===
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:

* Set N equal to 2^M, where M is an integer. A common M is 4.
* WHILE N is greater than 1:
:* Downscale the image (from the original) by a factor of N.
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.
:* N = N/2
* Do a final refine at no downscale at all.

===Non-local RD optimization===
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.

===H.264/MVC===
Read the spec and write an outline of what changes you'll have to make to x264 (specifically, with x264 in mind) in order to implement MVC. In other words, a design document. It doesn't have to be very formal, but it needs to clearly demonstrate that you have put some amount of thought into this.

==Contact info==
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.

You should also contact the admin [[User:J-b|jb]].

SoC x264 2011

2011-03-03T12:13:09Z

Dark Shikari: /* Qualification tasks */

==Introduction to x264 and Summer of Code==
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.

But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in.

x264 is part of the VideoLAN candidature for Google Summer of Code 2011.

*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)
*Possible other mentors: Ronald Bultje (BBB)?

An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.

On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].

==Guide to getting involved==
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.

==Skills needed==
These are required for all listed projects and probably anything not listed, too.
*Basic C programming.
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).

==Projects==
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.

===Optimization (ARM)===
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.

x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.

NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.

===4:4:4 and 4:2:2 colorspaces===
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.

The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.

This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.

===GPU motion estimation===
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:

*High-complexity motion search designed to get useful predictors to be used by the main motion search.
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.

The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.

This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.

===Non-local RD optimization===
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).

This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.

===H.264/MVC===
This project would be to implement the multiview coding extension (read: 3D video) of H.264. This would mostly be a high-level project: affecting headers, reference handling, and so forth, as opposed to low level compression steps. This would let x264 be used to author 3D Blu-rays and encode 3D content more efficiently.

This project would involve reading a lot of spec and implementing it in the cleanest way possible. It is probably not inordinately difficult.

==Qualification tasks==
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!

Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.

None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''

===Optimization (ARM)===
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.

===4:4:4 and 4:2:2 Colorspaces===
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.

===GPU Motion Estimation===
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:

* Set N equal to 2^M, where M is an integer. A common M is 4.
* WHILE N is greater than 1:
:* Downscale the image (from the original) by a factor of N.
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.
:* N = N/2
* Do a final refine at no downscale at all.

===Non-local RD optimization===
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.

===H.264/MVC===
Read the spec and write an outline of what changes you'll have to make to x264 (specifically, with x264 in mind) in order to implement MVC. In other words, a design document. It doesn't have to be very formal, but it needs to clearly demonstrate that you have put some amount of thought into this.

==Contact info==
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.

You should also contact the admin [[User:J-b|jb]].

SoC x264 2011

2011-03-03T12:11:23Z

Dark Shikari: /* Projects */

==Introduction to x264 and Summer of Code==
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.

But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in.

x264 is part of the VideoLAN candidature for Google Summer of Code 2011.

*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)
*Possible other mentors: Ronald Bultje (BBB)?

An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.

On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].

==Guide to getting involved==
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.

==Skills needed==
These are required for all listed projects and probably anything not listed, too.
*Basic C programming.
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).

==Projects==
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.

===Optimization (ARM)===
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.

x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.

NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.

===4:4:4 and 4:2:2 colorspaces===
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.

The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.

This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.

===GPU motion estimation===
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:

*High-complexity motion search designed to get useful predictors to be used by the main motion search.
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.

The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.

This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.

===Non-local RD optimization===
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).

This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.

===H.264/MVC===
This project would be to implement the multiview coding extension (read: 3D video) of H.264. This would mostly be a high-level project: affecting headers, reference handling, and so forth, as opposed to low level compression steps. This would let x264 be used to author 3D Blu-rays and encode 3D content more efficiently.

This project would involve reading a lot of spec and implementing it in the cleanest way possible. It is probably not inordinately difficult.

==Qualification tasks==
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!

Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.

None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''

===Optimization (ARM)===
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.

===4:4:4 and 4:2:2 Colorspaces===
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.

===GPU Motion Estimation===
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:

* Set N equal to 2^M, where M is an integer. A common M is 4.
* WHILE N is greater than 1:
:* Downscale the image (from the original) by a factor of N.
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.
:* N = N/2
* Do a final refine at no downscale at all.

===Non-local RD optimization===
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.

==Contact info==
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.

You should also contact the admin [[User:J-b|jb]].

SoC x264 2011

2011-03-03T12:05:29Z

Dark Shikari: Initial SoC 2011 page

==Introduction to x264 and Summer of Code==
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.

But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in.

x264 is part of the VideoLAN candidature for Google Summer of Code 2011.

*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)
*Possible other mentors: Ronald Bultje (BBB)?

An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.

On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].

==Guide to getting involved==
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.

==Skills needed==
These are required for all listed projects and probably anything not listed, too.
*Basic C programming.
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).

==Projects==
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.

===Optimization (ARM)===
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.

x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.

NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.

===4:4:4 and 4:2:2 colorspaces===
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.

The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.

This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.

===GPU motion estimation===
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:

*High-complexity motion search designed to get useful predictors to be used by the main motion search.
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.

The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.

This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.

===Non-local RD optimization===
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).

This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.

==Qualification tasks==
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!

Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.

None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''

===Optimization (ARM)===
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.

===4:4:4 and 4:2:2 Colorspaces===
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.

===GPU Motion Estimation===
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:

* Set N equal to 2^M, where M is an integer. A common M is 4.
* WHILE N is greater than 1:
:* Downscale the image (from the original) by a factor of N.
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.
:* N = N/2
* Do a final refine at no downscale at all.

===Non-local RD optimization===
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.

==Contact info==
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.

You should also contact the admin [[User:J-b|jb]].

SoC x264 2009

2011-01-11T03:49:15Z

Dark Shikari:

[[x264]] has loads of possibilities for [[SoC 2009]] projects. Those listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC.

This is part of the VideoLAN candidature for Google Summer of Code 2009.

*Mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)
*Possible backup mentor: Holger Lubitz

==Introduction to x264==
x264 is probably the most efficient, compression-wise, open source video encoder there is. It is quite competitive with commercial encoders, outclassing a large number of them.

While not actually part of VLC or ffmpeg (it has its own codebase), it is a major library used by both, licensed under the GPL, in addition to being a standalone encoder. As the only major open-source H.264 encoder, x264 has a near-complete monopoly on H.264 encoding in the consumer world, along with being used by many major corporations, including Youtube and Facebook. Some companies, such as Avail Media, have in the past offered bounties on improvements to the encoder.

An overview of x264's algorithm can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here].
==Skills needed==
These are required for all listed projects and probably anything not listed, too.
*Basic C programming.
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins.
*Confidence in the ability to learn the basics of following and similar topics (though not all projects will require such information):
:*Discrete cosine transform and similar frequency transforms
:*Motion estimation and compensation
:*Quantization and entropy encoding
A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here] (new version [http://dl.dropbox.com/u/2701213/pdfs/0470516925Video.pdf here]). It also has some more specific chapters on MPEG-4 Part 2 and Part 10.

==Projects==
We are probably only accepting one or two students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks.

===Optimization (x86)===
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.

This is probably the hardest task, as x264 is already so absurdly optimized that going significantly further is going to be very difficult. As such, the qualification task for this project (see below) requires you to prove that you can get at least somewhere. The goal here is at least a 5-10% performance increase in x264 on x86 (and ideally, other CPUs too, but processor-specific optimizations are allowed here). Some unconventional ideas in addition to the obvious tasks of writing loads of assembly and finding cases where the C code can be further optimized:

*Cache profiling: try to minimize cache misses.
*Code size profiling: find the inherent "value" in clock cycles of a line of cache and use this to try to optimize the code size of existing assembly.
*Aliasing optimizations: find places where aliasing is hurting the compiler's ability to optimize.

Not recommended for anyone other than hardcore assembly gurus.

===ARM Support===
x264 currently focuses on x86--and to a lesser extent--PowerPC. ARM is becoming more and more important every year--yet currently x264 cannot even run on some ARMs, and there's no configure support. Let alone the fact that there's no ARM assembly. This task would be to set up ARM support in configure, fix all unaligned accesses in the C code (there's probably only one), and then writing ARM assembly for all major DSP functions. There are already some similar ARM functions in ffmpeg for H.264 decoding which could be useful to you in this task. The goal here is to make x264 at least 4-5 times faster on ARM by implementing all the major assembly functions on ARM.

Also not recommended for anyone other than hardcore assembly gurus.

===4:4:4 and 4:2:2 Colorspaces===
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous two, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will probably be willing to do it for you if you aren't an assembly guru.

The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.

This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.

===GPU Motion Estimation===
In practice, this probably means CUDA, as OpenCL is not supported by anything yet.

While porting x264 entirely to CUDA is an insane task, putting a lookahead motion estimation on the GPU could be useful both for quality and performance. Tricky catches here include:

*The motion estimation method has to support x264's threading model. The easiest way to do this might be, as suggested, to make it into a lookahead function that is run on entire frames before the main encoding begins.
*The algorithm has to be able to make some sorts of basic mode decisions well enough to to be comparable with x264's basic SATD mode decision. This means you will have to implement the hpel and qpel interpolation algorithms, along with the SATD (Sum of Hadamard Transformed Differences) comparison method.
*Interlaced mode.

The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. More description of this method is in the Qualification Tasks section.

This project is not recommended unless you have a very significant amount of experience with CUDA.

===Weighted P-frame Prediction===
Of the projects listed, this is the only one with the potential to significantly improve encoding quality. Weighted P-frame prediction lets you assign weights to the frames in a reference list for the current frame, values to multiply all the pixels by. This is incredibly useful in dealing with fades, camera flashes, etc. However, it would require both a good enough algorithm to find optimal weighting factors and an efficient enough algorithm to be useful in practice.

A [http://akuvian.org/src/x264/x264_wpredp.0.diff patch] already exists for this--but it is so old (from the early days of x264) that it is practically useless except as a guide to how to start implementing it now.

Note that the challenge here is twofold: add support for weighted P-frame prediction and make an algorithm good enough and fast enough to make the feature useful.

==Qualification tasks==
This year, the qualification tasks will represent the start of your summer project. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!

Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.

===Optimization (x86)===
If you're interested in the optimization task, the qualification task is to speed up x264 on x86 (32 or 64-bit) by 1-2% on "normal settings" without changing the output. This is '''much''' harder than it sounds.

===ARM Support===
If you're interested in the ARM task, your qualification task will be to:

* Fix the unaligned access bug in the bitstream writer.
* Write NEON SIMD assembly for at least a few of the simpler significant DSP functions (SAD, SATD, etc).

===4:4:4 and 4:2:2 Colorspaces===
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be remotely viewable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.

===GPU Motion Estimation===
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. The hierarchical search works via the following algorithm.

* Set N equal to 2^M, where M is an integer. A common M is 4.
* WHILE N is greater than 1:
:* Downscale the image (from the original) by a factor of N.
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.
:* N = N/2
* Do a final refine at no downscale at all.

===Weighted P-frame Prediction===
The task for this project is to add support for Weighted P-frame Prediction--but not to write any sort of handling of it from an algorithmic standpoint, so no algorithm to decide the weights. That is, all the weights are just set to some constant value until we come up with a better way to do things. As with the other tasks, you can make many simplifications here to avoid corner cases, such as ignoring multithreading. Another simplification you can make is to only allow a weight on the first reference frame, or even have the program be completely ignorant of the weighted version of the frame until the final encode. All that matters is you get the basic framework working.

==Contact info==
If you are interested, drop by #x264dev or #x264 on Freenode.

You should also contact the admin [[User:J-b|jb]].

X264 asm intro

2010-12-02T13:16:29Z

Dark Shikari:

Open common/x86/predict-a.asm, go to predict_4x4_dc_mmxext, [http://git.videolan.org/?p=x264.git;a=blob;f=common/x86/predict-a.asm;h=a05e91beda809b5e113df8ba122496e756442df8;hb=HEAD#l292 git link] 
This function does the following: 
<pre> A B C D
E X X X X
F X X X X
G X X X X
H X X X X</pre>
It calculates (A+B+C+D+E+F+G+H+4)/8, and sets all the Xs equal to that value where those are 8-bit pixels in a 2D array with a stride of FDEC_STRIDE.
:<Q> What is FDEC_STRIDE? 
:<A> x264 does all its pixel operations on the current macroblock in a temporary buffer of constant stride. It's faster that way, and better on cache. So for example, motion compensation will write to this buffer (or intra prediction). 
:<Q> What's a stride? 
:<A> Stride is the distance between (x,y) and (x,y+1), so to get from one row to the next. 

Now that you understand what the function does, let's look at the asm. 
<pre>cglobal predict_4x4_dc_mmxext, 1,4</pre>
cglobal: declares a function accessible from outside of asm 
The function's name is <tt>x264_predict_4x4_dc_mmxext</tt> (the <tt>x264_</tt> is auto-added). 
The "1" means "we have one argument. Put it in <tt>r0</tt>.", that argument is uint8_t *src 
If we had a second argument, we'd say <tt>2</tt> and the second one would go in <tt>r1</tt> and if we had a third, it'd go in <tt>r2</tt>, etc. So at the start of the function, <tt>r0</tt> contains uint8_t *src. 
:<Q> "that argument is uint8_t *src", what does this mean? 
:<A> See the comment above: <tt>void predict_4x4_dc( uint8_t *src )</tt> 
:<Q> what tells the function that it's uint8_t? 
:<A> Nothing. It doesn't need to know. Types are a C-ism. 
The "4" means we want x264 to give us 4 registers to use. <tt>r0</tt>, <tt>r1</tt>, <tt>r2</tt>, <tt>r3</tt>. This, of course, includes the <tt>r0</tt> used for the parameter. So in short, after the first line: 
<tt>r0</tt> = src 
<tt>r1</tt>/<tt>r2</tt>/<tt>r3</tt> = free 
<tt>r4</tt> and up: can't use. 
:<Q> That's x86inc.asm's doing right? 
:<A> Yes, but we aren't going into that. 
:<Q> I assume it means we can use them, but if you do, it'll screw around with something you don't want to? 
:<A> Yes, which is why you can't use it. 

So now, this function as you can see has 4 real steps: 
# Sum up A through D 
# Sum up E through H 
# Do the math to get our final value 
# Store it into the 16 output Xs 
So let's see how this asm implements these. 

First, we'll look at step 1 
<pre>pxor mm7, mm7</pre>
<tt>mm7</tt> is a 64-bit register. 
<tt>xor</tt>, as you might know, is a nice way to zero things. 
:<Q> How do you tell how large a register is? 
:<A> <tt>mm*</tt> is 64-bit, <tt>xmm*</tt> is 128-bit. The mm registers have a fixed size, only the general purpose registers are wordsize-dependant on x86 
So, now <tt>mm7</tt> is zero. 
<pre>movd mm0, [r0-FDEC_STRIDE]</pre>
This sets <tt>mm0</tt> equal to {A,B,C,D,0,0,0,0} 
:<Q> Oh, and how do we know the mm* registers are free? 
:<A> They always are. 
In x86, b = byte, w = word (16-bit), d = doubleword (32-bit), q = quadword (64-bit), dq = double quadword (128-bit) 
So <tt>movd</tt> = move doubleword = move 32 bits 
So <tt>movd</tt> to <tt>mm0</tt> will load data to the first 4 bytes and zero the rest. Thus <tt>mm0</tt> is now ABCD0000 
<tt>[r0-FDEC_STRIDE]</tt> is equivalent to <tt>*(src-FDEC_STRIDE)</tt> in Cstyle. Hence why it points to ABCD. 
:<Q> Are the "A B C D" on top of the "X X X X" or do they start on top of the "E"? 
:<A> Former 
<pre>psadbw mm0, mm7</pre>
This is what <tt>psadbw</tt> does: 
<pre>uint16_t psadbw( uint8_t in[8], uint8_t out[8] )
{
uint16_t sum = 0;
for(int i = 0; i < 8; i++)
sum += abs(in[i]-out[i]);
return sum;
}</pre>
Parse that for a moment 
:<Q> where is the sum stored? 
:<A> <tt>psadbw X, Y</tt>, <tt>X</tt> is where the output is stored. So <tt>X</tt> is overwritten. So it's stored in the low 16 bits of <tt>X</tt>. 
Now, of course, <tt>mm7</tt> is zero! So we get abs(A-0) + abs(B-0) + abs(C-0) + abs(D-0) + abs(0-0) ... Or A+B+C+D. 
So after <tt>psadbw</tt>, <tt>mm0</tt> is A+B+C+D and <tt>mm7</tt> is still zero. 
<pre>movd r3d, mm0</pre>
Now, we move the result to <tt>r3d</tt>, a general purpose register 
:<Q> is <tt>r3d</tt> one of the things that come with the 4 registers that are free? 
:<A> This is an optimization: on 64-bit, using 32-bit versions of registers results in smaller instruction opcode sizes. So it's really just <tt>r3</tt>. <tt>r0</tt>, <tt>r1</tt>, <tt>r2</tt>, <tt>r3</tt> are the 4 that are free. So we're using <tt>r3</tt>. Note: the suffix 'd' means the 32-bit version, as opposed to the native-size version. 

Get moving with part 2 of the algorithm. 
Now <tt>r0</tt> has our source pointer, and <tt>r3</tt> has A+B+C+D. While the CPU is busy doing that, we'll go and do part 2, the E+F+G+H. 
Unfortunately, these bytes aren't in a straight line ("straight line" meaning "adjacent in memory"). So we can't just load EFGH and sad them. We'll have to do it the naive/slow way. So, now we're going to load E, F, G, H. Now you might notice some preprocessor commands here. <tt>%assign</tt>, <tt>%rep</tt>, etc are preprocessor commands. 
So, first step: load E into <tt>r1d</tt> 
<pre>movzx r1d, byte [r0-1]</pre>
<tt>movzx</tt> means "move, with zero extend". In C this would be: int r1d = r0[-1]; 
:<Q> My C is a bit rusty, what does that do? does it just take the location in memory before r0[0]? 
:<A> Yes, [] is just a dereference of a pointer. *(r0-1) = r0[-1] = (r0-1)[0] 
:<Q> what is r0-1 in that ascii matrix? 
:<A> E. 
So, here's what these 7 lines look like after the macro runs 
<pre>movzx r1d, byte [r0-1]
movzx r2d, byte [r0+FDEC_STRIDE*1-1]
add r1d, r2d
movzx r2d, byte [r0+FDEC_STRIDE*2-1]
add r1d, r2d
movzx r2d, byte [r0+FDEC_STRIDE*3-1]
add r1d, r2d</pre>
in order: load E, load F, add F to E, load G, add G to E, load H, add H to E 
:<Q> Where is n stored? 
:<A> It isn't. It's a preprocessor variable. 
:<Q> Oh, so it's like a macro? 
:<A> It is a macro. Note the pre-processed code above. Everything starting with % in yasm syntax is a macro 

Ok, now we have to do step 3: calculating A+B+C+D+E+F+G+H+4 / 8 
<pre>lea r1d, [r1+r3+4]</pre>
First, let's go over x86 addressing. What you can put inside the brackets is not infinite. Here's the capabilities, specifically: [REG1 + REG2 * {1,2,4,8} + CONST] 
A register, plus another register * 1/2/4/8, plus a constant (positive or negative). As you might note, this is pretty useful for accessing things like arrays. E.g. array[n+5], where array is an int array, would be: [array + n*4 + 20] 
:<Q> I suppose the [r0+FDEC_STRIDE*n-1] bit gets simplified on assembly to [register + const]? 
:<A> Yes, yasm sums up constants for you. 
So, as you might note, that's a pretty powerful addressing system. That's more powerful than, say... "add". So why not expose it in an instruction to let us use it for math? So Intel did. 
<tt>lea X, [expr]</tt> sets X equal to the value of expr just as fast as add. So that <tt>lea</tt> does r1d = r1 + r3 + 4 
:<Q> Wait, how does that work? 
:<A> <tt>lea</tt> runs the [REG1 + REG2 * {1,2,4,8} + CONST] math on its second argument and adds to the first. <tt>lea</tt> doesn't actually address it. It just calculates the result and stores it instead of going to memory. 
:<Q> And it's faster than add? 
:<A> It's just as fast except that you can do more with it. 
Now, technically, you can do more adds per cycle than <tt>lea</tt>, so you shouldn't go replacing all your adds with <tt>lea</tt>. But if you can use it to do more than one thing at a time, it's a big win. So this lets us add <tt>r3</tt>, and add 4, in one op.
<pre>shr r1d, 3</pre>
There's one that you can probably figure out yourself - shift right. 
:<Q> Why are we shifting right? 
:<A> +4 for correct rounding, >> 3 to divide (>>3 = /(2^3) = /8). 

Now for the final part: storing the results. 
<pre>imul r1d, 0x01010101</pre>
This is called a "splat" and you may have seen it in C as well. We're turning an 8-bit value into 4x that value, e.g. A -> AAAA 
:<Q> how does this work? 
:<A> A * 0x01010101 = A A A A 

So now we have a 32-bit register, <tt>r1d</tt> with one copy of A in each 8-bit nibble of that register. Now we go ahead and store this 4 times and we're done. 

Finally, we RET: x264 will automatically clean up after us.

X264 TODO

2010-11-22T12:59:34Z

Dark Shikari: /* x86 assembly */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work more on 10-bit asm.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

GCodeIn Ideas

2010-11-21T20:01:40Z

Dark Shikari: /* Google Code In */

= Google Code In =

This page is about gathering '''ideas''' for the VideoLAN project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

VideoLAN has been part of [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2007|2007]], [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

[[x264 GCodeIn Ideas|x264]] is also participating in Videolan's Code-In.

= Ideas for VideoLAN =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== VideoLAN Communication ==

=== Wiki Orphans and Double tracking ===
'''Category''': Documentation 
'''Description''': The wiki has way too many [[Special:LonelyPages|orphaned pages]] and [[Special:DoubleRedirects| double redirected pages]], they should be linked by other pages or marked for deletion, and redirects should be fixed 
'''Outcome''': A better wiki with less orphaned pages or redirects 
'''Difficulty''': easy 
'''Tools''': a wiki account 
'''Time''': 4hours 
'''Mentor''': [[User:Xtophe|xtophe]] 

=== Wiki short pages tracking ===
'''Category''': Documentation 
'''Description''': The wiki has too many [[Special:ShortPages|short pages]], they should be improved, marked for deletion or merged with other pages 
'''Outcome''': A better wiki with less short pages 
'''Difficulty''': easy 
'''Tools''': a wiki account 
'''Time''': 4hours 
'''Mentor''': [[User:Xtophe|xtophe]] 

=== VideoLAN flyer/poster ===
'''Category''': Outreach 
'''Description''': The VideoLAN project needs a flyer for promotional matters 
'''Outcome''': A cool A5-sized flyer presenting VideoLAN 
'''Difficulty''': medium 
'''Tools''': Image-Editing software 
'''Time''': 3days 
'''Mentor''': [[User:Jpsaman|Jean-Paul Saman]] 

=== VideoLAN Forum improvements ===
'''Category''': Research and Outreach 
'''Description''': The VideoLAN [http://forum.videolan.org forums] have many shortcomings, especially regarding spam and "Solved topics" 
We need research on solutions and advise us how we can improve the forums 
'''Outcome''': small report on ideas, advice and solution 
'''Difficulty''': medium 
'''Tools''': Browser and Text editor 
'''Time''': 3days 
'''Mentor''': VLC_help 

=== VideoLAN PHP webpage for file uploading for bugreports ===
'''Category''': Code 
'''Description''': The VideoLAN project needs a small WebPage in PHP to be able to upload the files for the bugreports 
As some of those files are big, some progression bar should be done in Javascript too 
'''Outcome''': a working deployed PHP script 
'''Difficulty''': hard 
'''Tools''': PHP development environment 
'''Time''': 5days 
'''Mentor''': etix 

== VLC ==
=== Create VLC videos for training ===
'''Category''': Training 
'''Description''': Creation of youtube Videos of screencasts of VLC usage 
This task can be divided in chunks of 5 videos 
'''Outcome''': VLC Youtube channels 
'''Difficulty''': easy 
'''Tools''': VLC, screencast recorders 
'''Time''': 2 days 
'''Mentor''': [[User:linkfanel|linkfanel]] 

=== VLC documentation illustration ===
'''Category''': Documentation 
'''Description''': Creation of VLC screenshots and small diagrams to improve the VLC documentation on the wiki 
'''Outcome''': VLC illustrations on the documentation 
'''Difficulty''': easy 
'''Tools''': VLC, Image Editing software 
'''Time''': 5 days 
'''Mentor''': [[User:ivoire|Rémi Duraffort]] 

=== VLC users survey creation ===
'''Category''': Outreach 
'''Description''': Creation of a survey for VLC users, about their usage of VLC, that we will put on the website 
'''Outcome''': Survey ready to be sent to the VLC users 
'''Difficulty''': medium 
'''Tools''': text editor and web browser 
'''Time''': 5 days 
'''Mentor''': [[User:J-b|jb]] 

=== VLC fullscreen controller redesign ===
'''Category''': User Interface 
'''Description''': Find ideas to improve and redesign the fullscreen controller of the VLC version on Windows/Linux 
'''Outcome''': Sketchs and ideas for the fullscreen controller 
'''Difficulty''': medium 
'''Tools''': web browser and image editor 
'''Time''': 5 days 
'''Mentor''': [[User:J-b|jb]] 

=== VLC volume controller redesign ===
'''Category''': User Interface 
'''Description''': Find ideas to improve and redesign the volume controller of the VLC version on Windows/Linux 
'''Outcome''': Sketchs and ideas for the volume controller 
'''Difficulty''': hard 
'''Tools''': web browser and image editor 
'''Time''': 3 weeks 
'''Mentor''': [[User:J-b|jb]] 

=== VLC Lyrics extension ===
'''Category''': Code 
'''Description''': Creation of one extension in lua that can be able to fetch and display Lyrics from one website API 
'''Outcome''': Working Lua Lyrics extension script 
'''Difficulty''': hard 
'''Tools''': text editor and VLC 
'''Time''': 10 days 
'''Mentor''': [[User:Jpeg|jpeg]] 

=== VLC Songkick extension ===
'''Category''': Code 
'''Description''': Creating one extension in lua that can be able to fetch and display Lyrics from Songkick API 
'''Outcome''': Working Lua Songkick extension script 
'''Difficulty''': hard 
'''Tools''': text editor and VLC 
'''Time''': 10 days 
'''Mentor''': [[User:Jpeg|jpeg]] 

=== VLC webplugin testpages ===
'''Category''': Code 
'''Description''': This task is about updating the Html/CSS/JS scripting test pages for the [[Documentation:WebPlugin|Webplugins]]. 
'''Outcome''': Usable pages for testing the VLC webplugins 
'''Difficulty''': hard 
'''Tools''': text editor and a browser 
'''Time''': 10 days 
'''Mentor''': [[User:Jpsaman|Jean-Paul Saman]] 

=== VLC warnings cleanup ===
'''Category''': Code 
'''Description''': This has for objective to delete a lot of warnings in C and C++ code when doing VLC compilation for Linux and Windows. 
'''Outcome''': Less warnings in C and C++ code 
'''Difficulty''': hard 
'''Tools''': text editor and compilation toolchain 
'''Time''': 10 days 
'''Mentor''': [[User:ivoire|Rémi Duraffort]] 

=== libVLC Qt example media player ===
'''Category''': Code 
'''Description''': Creating a small example of how to create a media player based on libVLC and Qt on Windows/Linux. 
'''Outcome''': a cool media player to demonstrate the libVLC API in Qt 
'''Difficulty''': hard 
'''Tools''': complete compilation toolchain 
'''Time''': 15 days 
'''Mentor''': pdherbemont 

=== libVLC Gtk example media player ===
'''Category''': Code 
'''Description''': Creating a small example of how to create a media player based on libVLC and Gtk on Windows/Linux. 
'''Outcome''': a cool media player to demonstrate the libVLC API in Gtk 
'''Difficulty''': hard 
'''Tools''': complete compilation toolchain 
'''Time''': 15 days 
'''Mentor''': pdherbemont 

=== libVLC wxWidgets example media player ===
'''Category''': Code 
'''Description''': Creating a small example of how to create a media player based on libVLC and wxWidgets on Windows/Linux. 
'''Outcome''': a cool media player to demonstrate the libVLC API in wxWidgets 
'''Difficulty''': hard 
'''Tools''': complete compilation toolchain 
'''Time''': 15 days 
'''Mentor''': pdherbemont 

== VLMC ==

=== Create VLMC videos for training ===
'''Category''': Training 
'''Description''': Creation of youtube Videos of screencasts of VLMC usage 
This task can be divided in chunks of 3 videos 
'''Outcome''': VLMC Youtube channels 
'''Difficulty''': easy 
'''Tools''': VLMC, screencasting tools 
'''Time''': 3 days 
'''Mentor''': [[User:etix|etix]] 

=== VLMC UI testing ===
'''Category''': Quality Assurance 
'''Description''': Testing VLMC Interface and testing all dialogs and options to find bugs 
'''Outcome''': Bugreports 
'''Difficulty''': easy 
'''Tools''': VLMC 
'''Time''': 3 days 
'''Mentor''': Hugo 

=== VLMC files testing ===
'''Category''': Quality Assurance 
'''Description''': Testing VLMC for Windows or Linux with many file formats 
'''Outcome''': Bug reports on the forum that don't work 
'''Difficulty''': medium 
'''Tools''': VLMC, mediainfo, Windows/Linux 
'''Time''': 7 days 
'''Mentor''': Hugo 

== Contact ==
For ANY question, contact [[User:J-b|jb]] or [[User:xtophe|xtophe]]

IRC channel: #videolan or irc://irc.freenode.net

X264 TODO

2010-11-21T01:28:18Z

Dark Shikari: /* x264CLI */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

X264 TODO

2010-11-20T10:40:43Z

Dark Shikari: /* Weighted Prediction */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 GCodeIn Ideas

2010-11-06T21:42:11Z

Dark Shikari: /* CLI Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

'''Category''': Add a --device option 
'''Description''': Add a new option to x264cli so users can specify a particular device 
'''Details''': Edit x264.c to change the input settings for a particular device in a similar way to --preset and --tune 

Most of the settings for specific devices (e.g iPod, Xbox, AppleTV) can be found online. x264 developers can help with more complex devices such as Blu-Ray.

'''Outcome''': Users can specify a specific device to x264. 
'''Difficulty''': medium 
'''Tools''': x264 
'''Time''': 3-5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9), Kieran Kunhya (kierank) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of C functions with assembly versions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. In many cases, some (but not all) versions of a function are already written in assembly, providing good examples for future versions. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you--Jason has taught many many students before. It usually takes a couple hours to learn the basics. 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm (for x86) or gas (for ARM) 
'''Time''': 0.5-7 days, depends heavily on function complexity 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 TODO

2010-11-06T20:01:04Z

Dark Shikari: /* Other features */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 GCodeIn Ideas

2010-11-06T19:59:50Z

Dark Shikari: /* libx264 Core Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of C functions with assembly versions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. In many cases, some (but not all) versions of a function are already written in assembly, providing good examples for future versions. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you--Jason has taught many many students before. It usually takes a couple hours to learn the basics. 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm (for x86) or gas (for ARM) 
'''Time''': 0.5-7 days, depends heavily on function complexity 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:56:01Z

Dark Shikari: /* libx264 Core Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of C functions with assembly versions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. In many cases, some (but not all) versions of a function are already written in assembly, providing good examples for future versions. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but it will take much longer. 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm (for x86) or gas (for ARM) 
'''Time''': 0.5-7 days, depends heavily on function complexity 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:55:25Z

Dark Shikari: /* libx264 Core Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of C functions with assembly versions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. In many cases, some (but not all) versions of a function are already written in assembly, providing good examples for future versions. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but don't complain when it's harder than you expected! 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm (for x86) or gas (for ARM) 
'''Time''': 0.5-7 days, depends heavily on function complexity 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:50:24Z

Dark Shikari: /* libx264 Core Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of C functions with assembly versions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but don't complain when it's harder than you expected! 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm (for x86) or gas (for ARM) 
'''Time''': 0.5-7 days, depends heavily on function complexity 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:49:58Z

Dark Shikari: /* libx264 Core Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of assembly functions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but don't complain when it's harder than you expected! 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm (for x86) or gas (for ARM) 
'''Time''': 0.5-7 days, depends heavily on function complexity 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:49:37Z

Dark Shikari: /* libx264 Core Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of assembly functions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but don't complain when it's harder than you expected! 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm 
'''Time''': 0.5-7 days, depends heavily on function complexity 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:48:33Z

Dark Shikari: /* CLI Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of assembly functions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but don't complain when it's harder than you expected! 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm 
'''Time''': 1-7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:46:03Z

Dark Shikari: /* CLI Development */

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264! 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development ==

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of assembly functions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but don't complain when it's harder than you expected! 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm 
'''Time''': 1-7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:45:48Z

Dark Shikari:

= Google Code In =

This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.

x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].

= Ideas for x264 =

== Warning ==
This is a temporary page for listing ideas for Google Code-in tasks.

The final tasks will be moved to melange, when needed.

== Tools ==
'''Category''': Quality Assurance 
'''Description''': Automated regression test tool 
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:

* Test any two revisions against each other.
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).

'''Outcome''': A regression test script (in Bash, Perl, Python, or similar) 
'''Difficulty''': medium-hard 
'''Tools''': x264, JM, ffmpeg, Linux, git 
'''Time''': 5 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari) 

== CLI Development ==

'''Category''': Video filtering 
'''Description''': Port ''any'' video filter to x264! 
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth. 

You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264. 

Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc. 

'''Outcome''': A new filter in x264. 
'''Difficulty''': hard 
'''Tools''': x264 
'''Time''': 7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9) 

== libx264 Core Development

'''Category''': Assembly Optimization 
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON) 
'''Details''': x264 has boatloads of assembly functions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. 

If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you, but don't complain when it's harder than you expected! 
'''Outcome''': A new assembly function in x264. 
'''Difficulty''': hard 
'''Tools''': x264, yasm 
'''Time''': 1-7 days 
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado) 

== Contact ==
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.

IRC channel: #x264dev on irc://irc.freenode.net

X264 GCodeIn Ideas

2010-11-06T19:36:28Z

Dark Shikari:

X264 GCodeIn Ideas

2010-11-06T19:23:34Z

Dark Shikari: /* Tools */

X264 GCodeIn Ideas

2010-11-06T19:22:18Z

Dark Shikari: /* Tools */

X264 GCodeIn Ideas

2010-11-06T19:20:56Z

Dark Shikari:

X264 GCodeIn Ideas

2010-11-06T19:14:52Z

Dark Shikari: initial gcodein page

X264 TODO

2010-09-19T12:11:26Z

Dark Shikari: /* Motion Estimation */ fullpel chroma me

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-19T12:11:00Z

Dark Shikari: /* Mode Decision */ chroma for b-frames is done

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-17T05:10:12Z

Dark Shikari: /* Psy */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-14T22:12:46Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-14T00:18:37Z

Dark Shikari: /* Ratecontrol */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* VP8 encoding support

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-13T23:36:15Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* VP8 encoding support

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-13T23:35:42Z

Dark Shikari: /* Ratecontrol */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-13T22:43:46Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-13T22:29:41Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-13T22:28:08Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.

X264 TODO

2010-09-13T22:26:52Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?

X264 TODO

2010-09-13T22:21:47Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?

X264 TODO

2010-09-13T22:18:07Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?

X264 TODO

2010-09-13T22:16:48Z

Dark Shikari:

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* Chroma-aware mode decision for B-frames?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...

===Lookahead===
* Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* Finish adaptive MBAFF. Talk to horlicks about this one.
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?

===Weighted Prediction===
* Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qcomp.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work on 10-bit asm. Talk to irock about this.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?

X264 TODO

2010-09-13T22:12:48Z

Dark Shikari:

X264 TODO

2010-09-13T22:08:41Z

Dark Shikari: