VideoLAN Wiki - User contributions [en]

X264 TODO

2012-03-04T05:09:25Z

Saintdev: /* Quantization */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

=== Motion Estimation ===

*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
*Somehow take into account the effect of motion vector decision on future blocks.
**Hierarchical motion estimation
**Approximations from lookahead MVs
**Iterative ME (as per Snow)
**Trellis motion estimation
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
**But how do we know which ones we can afford to skip, and when?
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
*Try sub-8x8 partitions in B-frames. Is it at all useful?
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
*Fullpel chroma ME?
**For TESA?

=== Intra Analysis ===

*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

=== Mode Decision ===

*Can we find more ways to skip more motion searches in multiref?
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
*See the TODOs for deblock-aware RD in common/deblock.c.
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
*How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

=== Psy ===

*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
*RD should take into account local variance.
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
*Lambda should be picked independent of quantizer (i.e. with greater precision).
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

=== Lookahead ===

*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

=== Quantization ===

*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

=== Transform ===

*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

=== Interlacing ===

*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
*Constrained intra + adaptive MBAFF. Does anyone care about this?
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

=== Weighted Prediction ===

*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.

=== Ratecontrol ===

*VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
**Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
*Make the frame size and row size predictors better. They currently are kind of crappy.
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
*2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

=== GPU ===

*Motion estimation?
**Methods
***Hierarchical?
***2D Wave?
***Something else?
**"Easy": lookahead motion estimation
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
**"Hard": main motion estimation
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
***But potentially more useful...
*Other things?

=== x86 assembly ===

*Optimize more for the Phenom.
*Yell at holger to commit his local patches.
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

=== Other assembly ===

*NEON assembly is nowhere near complete.
**Chroma MC needs to be rewritten for NV12 support.
*Altivec assembly is very lacking.
*SPARC VIS assembly is only available when high bit-depth is disabled.

=== Other CPU optimizations ===

*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

=== Other features ===

*MPEG-2 encoding support
*VP8 encoding support
*Support for SMPTE timecodes
*Merge speedcontrol
*Mixed lossless/lossy encoding.
*Segment re-encoding
*Spec compliant AVC-Intra encoding (ask kierank for spec)

=== x264CLI ===

*Finish audio support. Talk to Kovensky about this one.
*Make the filtering system aware of fullrange vs TV range.
*Make the filtering system aware of BT.601 vs BT.709.
*Add more filters.
**Deinterlacers (YADIF).
**Denoisers (HQDN3D?).
**IVTC, decomb?
*Merge L-SMASH mp4 muxer.
*Add TS muxing support using HRD. Talk to kierank about this one.
*Add --device support.
*Add automatic --level restriction support.

X264 asm intro

2011-12-13T03:05:28Z

Saintdev:

The discussions on this page have been edited for clarity, ease of reading, collation of later questions, and to trim out some noise. The original transcript can be found here: [[X264asm]]

__TOC__

== Example 1: predict_4x4_dc ==

Open common/x86/predict-a.asm, go to predict_4x4_dc_mmxext ([http://git.videolan.org/?p=x264.git;a=blob;f=common/x86/predict-a.asm;h=a05e91beda809b5e113df8ba122496e756442df8;hb=HEAD#l292 git link]). This function does the following:
<pre> A B C D
E X X X X
F X X X X
G X X X X
H X X X X</pre>
It calculates (A+B+C+D+E+F+G+H+4)/8, and sets all the Xs equal to that value where those are 8-bit pixels in a 2D array with a stride of FDEC_STRIDE.

:<Q> What is FDEC_STRIDE?
:<A> x264 does all its pixel operations on the current macroblock in a temporary buffer of constant stride. It's faster that way, and better on cache. So for example, motion compensation will write to this buffer (or intra prediction).
:<Q> What's a stride?
:<A> Stride is the distance between (x,y) and (x,y+1), so to get from one row to the next.

 Now that you understand what the function does, let's look at the asm.
<pre>cglobal predict_4x4_dc_mmxext, 1,4</pre>
"cglobal" declares a function accessible from outside of asm. The function's name is <tt>x264_predict_4x4_dc_mmxext</tt> (the <tt>x264_</tt> is auto-added). The "1" means "we have one argument. Put it in <tt>r0</tt>.", (that argument is uint8_t *src). If we had a second argument, we'd say "2" and the second one would go in <tt>r1</tt> and if we had a third, it'd go in <tt>r2</tt>, etc. So at the start of the function, <tt>r0</tt> contains uint8_t *src.

:<Q> "that argument is uint8_t *src", what does this mean?
:<A> See the comment above: <tt>void predict_4x4_dc( uint8_t *src )</tt>
:<Q> What tells the function that it's uint8_t?
:<A> Nothing. It doesn't need to know. Types are a C-ism.

The "4" means we want x264 to give us 4 registers to use. <tt>r0</tt>, <tt>r1</tt>, <tt>r2</tt>, <tt>r3</tt>. This, of course, includes the <tt>r0</tt> used for the parameter. So in short, after the first line: <tt>r0</tt> = src, <tt>r1</tt>/<tt>r2</tt>/<tt>r3</tt> = free, <tt>r4</tt> and up: can't use.

:<Q> That's x86inc.asm's doing right?
:<A> Yes, but we aren't going into that.
:<Q> I assume it means we can use them, but if you do, it'll screw around with something you don't want to?
:<A> Yes, which is why you can't use it.

 So now, this function as you can see has 4 real steps:

#Sum up A through D
#Sum up E through H
#Do the math to get our final value
#Store it into the 16 output Xs

So let's see how this asm implements these.

 First, we'll look at step 1
<pre>pxor mm7, mm7</pre>
<tt>mm7</tt> is a 64-bit register. <tt>xor</tt>, as you might know, is a nice way to zero things.

:<Q> How do you tell how large a register is?
:<A> <tt>mm*</tt> is 64-bit, <tt>xmm*</tt> is 128-bit. The mm registers have a fixed size, only the general purpose registers are wordsize-dependant on x86

So, now <tt>mm7</tt> is zero. 
<pre>movd mm0, [r0-FDEC_STRIDE]</pre>
This sets <tt>mm0</tt> equal to {A,B,C,D,0,0,0,0}

:<Q> Oh, and how do we know the mm* registers are free?
:<A> They always are.

In x86, b = byte, w = word (16-bit), d = doubleword (32-bit), q = quadword (64-bit), dq = double quadword (128-bit). So <tt>movd</tt> = move doubleword = move 32 bits. So <tt>movd</tt> to <tt>mm0</tt> will load data to the first 4 bytes and zero the rest. Thus <tt>mm0</tt> is now ABCD0000. <tt>[r0-FDEC_STRIDE]</tt> is equivalent to <tt>*(src-FDEC_STRIDE)</tt> in C-style. Hence why it points to ABCD.

:<Q> Are the "A B C D" on top of the "X X X X" or do they start on top of the "E"?
:<A> Former
<pre>psadbw mm0, mm7</pre>
This is what <tt>psadbw</tt> does:
<pre>uint16_t psadbw( uint8_t in[8], uint8_t out[8] )
{
uint16_t sum = 0;
for(int i = 0; i < 8; i++)
sum += abs(in[i]-out[i]);
return sum;
}</pre>
Parse that for a moment

:<Q> where is the sum stored?
:<A> <tt>psadbw X, Y</tt>, <tt>X</tt> is where the output is stored. So <tt>X</tt> is overwritten. So it's stored in the low 16 bits of <tt>X</tt>.

Now, of course, <tt>mm7</tt> is zero! So we get abs(A-0) + abs(B-0) + abs(C-0) + abs(D-0) + abs(0-0) ... Or A+B+C+D. So after <tt>psadbw</tt>, <tt>mm0</tt> is A+B+C+D and <tt>mm7</tt> is still zero.
<pre>movd r3d, mm0</pre>
Now, we move the result to <tt>r3d</tt>, a general purpose register.

:<Q> Is <tt>r3d</tt> one of the things that come with the 4 registers that are free?
:<A> This is an optimization: on 64-bit, using 32-bit versions of registers results in smaller instruction opcode sizes. So it's really just <tt>r3</tt>. <tt>r0</tt>, <tt>r1</tt>, <tt>r2</tt>, <tt>r3</tt> are the 4 that are free. So we're using <tt>r3</tt>. Note: the suffix 'd' means the 32-bit version, as opposed to the native-size version.

 Get moving with part 2 of the algorithm.

Now <tt>r0</tt> has our source pointer, and <tt>r3</tt> has A+B+C+D. While the CPU is busy doing that, we'll go and do part 2, the E+F+G+H. Unfortunately, these bytes aren't in a straight line ("straight line" meaning "adjacent in memory"). So we can't just load EFGH and sad them. We'll have to do it the naive/slow way. So, now we're going to load E, F, G, H. Now you might notice some preprocessor commands here. <tt>%assign</tt>, <tt>%rep</tt>, etc are preprocessor commands.

So, first step: load E into <tt>r1d</tt> 
<pre>movzx r1d, byte [r0-1]</pre>
<tt>movzx</tt> means "move, with zero extend". In C this would be: int r1d = r0[-1];

:<Q> My C is a bit rusty, what does that do? does it just take the location in memory before r0[0]?
:<A> Yes, [] is just a dereference of a pointer. *(r0-1) = r0[-1] = (r0-1)[0]
:<Q> what is r0-1 in that ascii matrix?
:<A> E.

So, here's what these 7 lines look like after the macro runs:
<pre>movzx r1d, byte [r0-1]
movzx r2d, byte [r0+FDEC_STRIDE*1-1]
add r1d, r2d
movzx r2d, byte [r0+FDEC_STRIDE*2-1]
add r1d, r2d
movzx r2d, byte [r0+FDEC_STRIDE*3-1]
add r1d, r2d</pre>
in order: load E, load F, add F to E, load G, add G to E, load H, add H to E

:<Q> Where is n stored?
:<A> It isn't. It's a preprocessor variable.
:<Q> Oh, so it's like a macro?
:<A> It is a macro. Note the pre-processed code above. Everything starting with % in yasm syntax is a macro.

 Ok, now we have to do step 3: calculating A+B+C+D+E+F+G+H+4 / 8
<pre>lea r1d, [r1+r3+4]</pre>
First, let's go over x86 addressing. What you can put inside the brackets is not infinite. Here's the capabilities, specifically: [REG1 + REG2 * {1,2,4,8} + CONST]. A register, plus another register * 1/2/4/8, plus a constant (positive or negative). As you might note, this is pretty useful for accessing things like arrays. E.g. array[n+5], where array is an int array, would be: [array + n*4 + 20]

:<Q> I suppose the [r0+FDEC_STRIDE*n-1] bit gets simplified on assembly to [register + const]?
:<A> Yes, yasm sums up constants for you.

So, as you might note, that's a pretty powerful addressing system. That's more powerful than, say... "add". So why not expose it in an instruction to let us use it for math? So Intel did. <tt>lea X, [expr]</tt> sets X equal to the value of expr just as fast as add. So that <tt>lea</tt> does r1d = r1 + r3 + 4

:<Q> Wait, how does that work?
:<A> <tt>lea</tt> runs the [REG1 + REG2 * {1,2,4,8} + CONST] math on its second argument and adds to the first. <tt>lea</tt> doesn't actually address it. It just calculates the result and stores it instead of going to memory.
:<Q> And it's faster than add?
:<A> It's just as fast except that you can do more with it.

Now, technically, you can do more adds per cycle than <tt>lea</tt>, so you shouldn't go replacing all your adds with <tt>lea</tt>. But if you can use it to do more than one thing at a time, it's a big win. So this lets us add <tt>r3</tt>, and add 4, in one op.
<pre>shr r1d, 3</pre>
There's one that you can probably figure out yourself - shift right.

:<Q> Why are we shifting right?
:<A> +4 for correct rounding, >> 3 to divide (>>3 = /(2^3) = /8).

 Now for the final part: storing the results.
<pre>imul r1d, 0x01010101</pre>
This is called a "splat" and you may have seen it in C as well. We're turning an 8-bit value into 4x that value, e.g. A -> AAAA

:<Q> how does this work?
:<A> A * 0x01010101 = A A A A

So now we have a 32-bit register, <tt>r1d</tt> with one copy of A in each 8-bit nibble of that register. Now we go ahead and store this 4 times and we're done.

Finally, we RET: x264 will automatically clean up after us.

== Example 2: pixel_sad ==

Ok, next. You may have noticed that <tt>psadbw</tt> is awesome. It does like 8 things in one. Whereas abs() is typically 4 instructions on x86. <tt>psadbw</tt> does 8 subtracts, 8 absolute values on those results and then adds them up. That's 8 + 32 + 7 = 47 instructions in one (at least, 47 equivalent).

:<Q> why is abs() so slow?
:<A> abs() isn't slow, there's just no instruction for it. The typical algorithm is:
<pre>int sign = x >> 31;
(x ^ sign) - sign;</pre>
:This needs a mov on x86, so that's 4 instructions.

 So <tt>psadbw</tt> is pretty awesome. It's very awesome for doing what its name implies you should do with it, that is -- SADs -- sum of absolute differences. So let's open sad-a.asm and hop down to line 95 ([http://git.videolan.org/?p=x264.git;a=blob;f=common/x86/sad-a.asm;h=0a96837aaa4c54f624b63c020872b74a790f1ffd;hb=HEAD#l95 git link]). Also open common/pixel.c and look at the first function: SAD ([http://git.videolan.org/?p=x264.git;a=blob;f=common/pixel.c;h=7fa497c7cf151df5795bb105895760fde89facb6;hb=HEAD#l44 git link]). This function is pretty simple. You should be able to see how it works. Look only at the C for now.

So as you'll notice, the C SAD has 7 different versions for 16x16, 16x8, 8x16, etc and it's instantiated via a macro. So, for our asm, we also need 7 versions and we also don't want to write the function 7 times, just like in the case of C we didn't.

So in the asm, we define a macro: <tt>%macro SAD 2</tt> that means this macro has two parameters. They are accessed as %1 and %2. We call the macro 7 times, one for each size. The function takes 4 args (as you'd expect) and needs 4 regs (just the args)

:<Q> Is SAD_INC_2x%1P another macro?
:<A> Yes, it's one of three macros, look above, each one does 2 rows worth of SAD for width 4, width 8, and width 16. It picks the right one based on the width and it %reps it based on the height.

 Now, start analyzing the 3 macros above (the sad macros) and trying to figure out how they work. Note mm0 is the accumulator which is why it's zeroed at the start.

:<Q> The order of args is the same as in the C function?
:<A> Yes
:<Q> What does punpckldq do?
:<A> Good question! punpck is a set of instructions that interleave their arguments in some fashion.

To start with, it can be l or h, low or high. So punpckl__ ABCD, EFGH will use AB and EF. And punpbkh__ ABCD, EFGH will use CD and GH.

The next two letters are the source size, and destination size. For example, <tt>punpcklbw</tt> interleaves bytes, to create words. So <tt>punpcklbw ABCD, EFGH</tt> gives you AEBF (if the letters are bytes). So <tt>punpckldq ABCDEFGH, IJKLMNOP</tt> gives us ABCDIJKL. So in other words, it stuffs the two sets of 4 bytes we just loaded into one register

So we can do only one SAD, instead of two. <tt>punpckldq ABCD0000, EFGH0000</tt> results in ABCDEFGH. So it effectively concatenates mm1 and mm2 for us. If we didn't do this, we'd have to do twice as many sads and adds. We do this because the registers are width 8, but our sad is width 4. So we need to stuff sad information side by side to fill the whole reg.

:<Q> Why are we punpckldq'ing the [r0+r1] and not [r0]?
:<A> We're concatenating row 0 and row 1 of each input.
:<Q><tt>lea r0, [r0+2*r1]</tt> Why are we doing this step? Doesn't it move r0 over 2*r1?
:<A> We're incrementing the pointer by 2*stride

 Now you should understand what SAD_INC_2x4P does, the others work similarly except without the punpck magic because they don't need it.

:<Q> Why is the lea out of order in SAD_INC_2x8P? By out of order i mean not next to each other.
:<A> No particular reason.
:<Q> So we rep the SAD for however many times so the 2x%1 is completed?
:<A> Yes, so if it's height 8, we rep it 4 times
:<Q> Why are strides not hardcoded btw?
:<A> SAD can be called on a reference frame thus variable stride
:<Q> I don't really get it...
:A<> It's called on frames, as opposed to some temporary block of memory.

Now, for the kicker: the 16x16 SAD function declared here is 15 times faster than C.

:<Q> What? Why is it so much faster?
:<A> psadbw

 Now let's get a bit to how we measure performance. For any asm instruction, there are three things that matter: latency, inverse throughput, and execution units. The first two are represented with notation like this: "3/1". This means a psadbw takes 3 clocks to finish from when it's started and you can do one of them per cycle. Another example is <tt>mov</tt>. Between two registers it is 1/0.33, takes 1 cycle, and you can do 3 per clock.

Execution unit usage is a bit trickier. Not all execution units can do all instructions. Intel chips have 6 execution units: p0, p1, p2, p3, p4, p5

:<Q> Wait, what is latency?
:<A> time from start to finish, in clocks
:<Q> and inverse throughput and execution units?
:<A> Inverse throughput is how many you can do per clock. Execution units are the things in the chip that do stuff.

Of these 6 execution units, three can do math: p0, p1, p5. <tt>psadbw</tt>, for example, can only use one of these (p1), <tt>pxor</tt> can use all three.

Generally execution units aren't important until you get into serious optimizing but they can often affect the best instruction choices. For example, if an execution unit is sitting around doing nothing for a whole function.

 The instruction tables sheet here http://agner.org/optimize/ has all the information on latency, execution units, and inverse throughput for a wide variety of CPUs.

:<Q> How about branching? I heard branching ruins things.
:<A> Not generally unless it's unpredictable, branch mispredictions are the cause. We can get to a case of that later.

Now, let's just analyze SAD. Suppose we want to analyze the 8x8 SAD. In this function we do: 8 SADs, 8 adds (accumulates), 16 loads, plus the start, end, and calling overhead. 8 SADs: takes 8 cycles (inverse throughput of 1). 8 adds: takes 8 cycles (inverse throughput of 1), and can run at the same time as SADs. 16 loads: takes 16 cycles, and can run at the same time as the above.

So the loads are the bottleneck. This is an important thing to understand: it's possible for one type of operation to bottleneck a function. Loads are a common example. In this case, SAD is *so fast* that it is effectively free, as we're sitting around waiting for loads the whole time. The actual run-time of the function is about 22 clocks. Which is fitting for 16 + start + end + overhead.

 So that's some basic performance analysis for you. How long the function should take in theory, how long each instruction takes in theory, and how you can be bottlenecked.

:<Q> Is there anything that does this automatically for you?
:<A> Analysis? not really. There are Intel performance counters and such on the chip but they're not magic. It might be useful to have some kind of tool to analyze asm functions. In general though, intuition is a powerful tool.

== Example 3: pixel_avg2_w16_sse2 ==

Let's move onto something else. Open common/x86/mc-a.asm, pixel_avg2_w16_sse2 ([http://git.videolan.org/?p=x264.git;a=blob;f=common/x86/mc-a.asm;h=22fb8720c83b72bcd34b41553f4153bb62847462;hb=HEAD#l842 git link]). This function interpolates between two inputs, and outputs to an output. The interpolation is the simplest possible: (A+B+1)>>1

Look at the function signature above: <tt>void pixel_avg2_w4( uint8_t *dst, int dst_stride</tt> etc. This function takes inputs from src1 and src2, averages them together, and writes to dst. src1 and src2 have src_stride and dst has dst_stride.

:<Q> What's the height?
:<A> How many lines to interpolate.

Now this function uses xmm registers (128-bit) so it does 16 bytes at a time. All 128-bit loads must be aligned unless movdqu is used. Since our inputs are unaligned, this is a lot of movdqu.

:<Q> What does movdqu do?
:<A> Loads 128 bits from an unaligned source.
:<Q> What is the difference between movdqa and movdqu?
:<A> movdqa is for aligned data, movdqu is for unaligned data. The output is always aligned, as we control it, whereas the input is an arbitrary pointer into a reference frame and so it could be anything.
:<Q> Why subtract r2 from r4?
:<A> Ah, now here's a fun trick. We need to increment three pointers, right? src1, src2, dst. But src1 and src2 have the same stride, so they're being incremented by the same amount. So we can take src2 and represent it as an offset from src1. Then we only have to increment src1. One lea removed per iteration, bam.
:<Q> And use r6 as the offset + stride?
:<A> Yes

Look through that function and see if there's anything you don't know about and ask questions.

:<Q> How do you keep track of which argument is which?
:<A> You can copy the description from w4 and annotate it.
<pre>; void pixel_avg2_w4( uint8_t *dst (r0), int dst_stride (r1),
; uint8_t *src1 (r2), int src_stride (r3),
; uint8_t *src2 (r4), int height (r5) );</pre>
:There is also a system that can be covered later which helps you keep track of registers or, well, makes it easier to.
:<Q> pavgb, I assume, does some sort of averaging?
:<A> Yes, (A+B+1)>>1 for each pair of input pixels.

Now we come to one of the many jumps that exists in x86, jump if greater than, so if r5d > 0.

:<Q> How does the jump work without a compare instruction?
:<A> The sub sets the same flags as a cmp.
:<Q> So why subtract two?
:<A> It handles two rows at a time.

Here is something to note. The REP_RET you might have been wondering about. In short, if you have a RET after a jump, use REP_RET. Blame AMD.

X264 TODO

2011-03-28T07:31:47Z

Saintdev: /* x86 assembly */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Optimize more for the Phenom.
* Optimize for Sandy Bridge.
* Work more on 10-bit asm.
* If you read this, join #x264dev on irc://irc.freenode.net and ping holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.
* SPARC VIS assembly is only available when high bit-depth is disabled.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol
* Mixed lossless/lossy encoding.

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

X264 Development Newsletters

2011-02-19T07:21:34Z

Saintdev:

This is an archive of the x264 Development Newsletters.

== Volume 1 ==

''9/18/2010''

From now on, with each weekly-ish push of new patches, I'll try to post a newsletter as well to give an update of what's going on and the changes.

===== Fixes: =====

Different versions of 64-bit MinGW gcc were inconsistent with regards to whether or not to add a _ prefix to called functions. This broke compilation because the assembly code wouldn't link in correctly.

Intra refresh got a lot of fixes thanks to bug reports by Chris Brien <chris.brien@tandberg.com>. Because x264 uses column refresh instead of row refresh, it has to be more careful with regards to which pixels it can predict from. This should fix some cases where intra refresh didn't refresh the whole frame correctly. Furthermore, it turns out that the recovery_frame_cnt syntax element, signaling the length of the intra refresh, is limited by the max frame number. Accordingly, x264 will raise the max frame number to ensure that recovery_frame_cnt is in spec.

===== Improvements: =====

I've gone through the file headers and updated them to be more accurate and consistent in terms of dates and authorship. It now also mentions the commercial license, on request from a licensee.

The new --disable-gpl configure option is now added for companies using the commercial license. x264 --version will print both the license information of itself and of the linked libavformat (for decoding). It'll also warn you if the combination you've put together isn't distributable, such as a nonfree ffmpeg + GPL x264, or a GPL ffmpeg + non-GPL x264.

x264 now passes the "full chroma input" flag to swscale. This should fix this quality issue when using point scaling: http://doom10.org/index.php?topic=585.0 .

===== New features: =====

Kieran Kunhya <kieran@kunhya.com> has finally committed his arbitrary SEI patch, which lets calling apps give x264 any SEI they want to include in the stream. This is useful for closed-captioning, frame packing, and all kinds of other SEIs that x264 doesn't explicitly support but which the calling application can create itself if necessary. The advantage of giving these to x264 instead of just inserting them separately is that it allows the resulting stream to remain HRD-compliant, since x264 knows they exist.

===== Upcoming: =====

High bit-depth input and filtering! x264 supports 10-bit encoding, but currently only takes 8-bit input. This change will finally create a stable API for high bit-depth input and allow users to filter in higher precision. It will also add dithering support to x264CLI for downconversion of high-bit-depth material.

HQDN3D (denoiser) and YADIF (deinterlacer) will be making their way into the x264CLI filter framework.

Chroma ME in B-frames will be coming soon, giving an extra bit of compression to those using slower presets.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

After high bit-depth input is complete, a large amount of assembly code will be committed as well, improving high bit-depth encoding performance greatly.

Ubuntu Maverick will include an updated x264, r1653.

== [http://mailman.videolan.org/pipermail/x264-devel/2010-September/007808.html Volume 2] ==
''9/27/2010''

This is the second x264 development newsletter. If you missed the first one, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next.

===== Fixes: =====

libx264 should now give correct DTS and print the correct bitrate if the input PTS do not start at zero. This doesn't affect x264CLI, but does affect calling apps that don't start their PTS at zero.

libx264 ratecontrol should now work correctly in CFR mode with timebase != 1/fps. As documented, x264 was supposed to ignore the timebase if CFR mode is set (for purposes of ratecontrol), but that's not what actually happened; things just broke. This should now be fixed. This is necessary for correct behavior with tune=zerolatency in gstreamer.

A missing emms has been added to dump-yuv, fixing some Heisenbug brokenness when dump-yuv was used.

===== Improvements: =====

Slice-max-size has been made more aggressive. Previously we just made some blind assumptions about the distribution of escape bytes and other such inaccuracies -- now it should work absolutely. It accounts for a few extra bytes that are highly predictable (CABAC's extra byte, CAVLC's skip runs). The primary problem, however, was caused by the terrible design of H.264's CABAC, which outputs a string of zeroes if CABAC is perfectly adapted, thus forcing tons of escape bytes for any perfectly uniform image. Slice-max-size should be fixed in this case.

Chroma mode decision and subpel, mentioned in the last newsletter, is finally in. It's enabled at subme >= 9 and provides a ~0.4-1% compression improvement.

===== New features: =====

High-bit-depth support has been finished in libx264 and x264cli. The assembly code is still yet to be committed, so it's a bit slow, but libx264 now supports high-bit-depth input. Additionally, the filter framework now supports 16-bit, which will be useful for future filters to avoid rounding error and banding. It'll automatically perform dither to go down from 16-bit to 8, 9, or 10-bit, thus making this useful for 8-bit encoding as well.

High 10 Intra profile is supported now, which makes x264 now capable of encoding AVC Intra 50. See the commit message for a sample commandline.

===== Upcoming: =====

High bit depth assembly code is coming soon, which should give a nice (3x+) improvement to high bit depth encoding speed.

HQDN3D (denoiser) and YADIF (deinterlacer) will be making their way into the x264CLI filter framework. This was previously being blocked by the high-bit-depth modifications to the filter framework.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

== [http://mailman.videolan.org/pipermail/x264-devel/2010-October/007858.html Volume 3] ==

''10/10/2010''

This is the third x264 development newsletter. If you missed the first two, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next.

===== Fixes: =====

The sigint handler in x264cli is now volatile, like it should have been. This likely didn't actually affect anything on any real system, but it's more correct.

-DNDEBUG broke the filtering system. This is now fixed.

The bugfix from last week regarding intra refresh predicting from topright blocks didn't work with 8x8dct. This is now fixed.

2-pass ratecontrol now works with CBR HRD. Previously, it ignored filler bits, and thus broke horribly. In general, you don't need 2-pass for CBR (and it probably doesn't help), but it should work better now.

A missing mod4-stack check was added; this should fix ICC builds on Phenom CPUs.

===== Improvements: =====

The build tree has been cleaned a bit. A bunch of old stuff, like the non-working regression test and doxy, have been removed.

Some of the information in doc/ has been updated to be less absurdly out of date, or at least inform the reader that it is absurdly out of date.

DTS compression has been moved out of the libx264 API and into the muxers, because it's a giant hack and it was making things ugly and messy (and starting to interfere with ratecontrol).

Various asm functions have been improved, particularly the qpel MC functions. Should help performance a bit on Core 2 and similar CPUs.

===== Upcoming: =====

High bit depth assembly code is coming soon, which should give a nice (3x+) improvement to high bit depth encoding speed.

HQDN3D (denoiser) and YADIF (deinterlacer) will be making their way into the x264CLI filter framework. This was previously being blocked by the high-bit-depth modifications to the filter framework.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework.

== [http://mailman.videolan.org/pipermail/x264-devel/2010-November/007920.html Volume 4] ==

''11/10/2010''

This is the fourth x264 development newsletter. If you missed the first three, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next. Previous versions can be found in the mailing list archives.

===== Fixes: =====

The Altivec SATD gave wrong results with very small strides (e.g. 8). This made output on PPC slightly different from x86, due to different results in lookahead and chroma ME. This is now fixed.

FPS reporting on Windows 64-bit was very imprecise (the timers were only accurate to 1 second). This was due to _ftime using the wrong struct due to the Windows ABI's complete retardation. This has been fixed by using ftime instead.

ssd_nv12 has been improved to eliminate overflow at high bit depths (this would, in theory, give wrong PSNR).

SATD/SA8D/Hadamard_ac have been improved to eliminate overflow at high bit depths (this would cause wrong analysis results).

Three ratecontrol bugs have been fixed:

#B-frame size prediction has been fixed (it used quantizer instead of linearized qscale). This should improve VBV.
#Overflow compensation, originally designed for 1-pass ABR, has been disabled with CRF on, as it doesn't do anything useful with pure CRF and causes very bizarre results with CRF+VBV in areas of extremely high complexity.
#A bug in the way that clip_qscale caps the amount by which a frame can be higher quality than a previous frame (as a result of attempting to use up extra bits that would otherwise be wasted) has been fixed. This caused CBR to take extremely long times (e.g. seconds) to recover from extreme changes in complexity at low bitrates.

A possible linking problem with lavf has been fixed.

Some MV/ref prefetching code was in the wrong place. Fixing this may result in a tiny performance improvement.

===== Improvements: =====

doc/threads.txt has been updated with modern benchmarks. This information may be useful to anyone looking to compare the impact of various threading strategies on quality and performance, as well as get an idea of how well x264 scales over multiple cores at various encoding speeds.

The presets are now addressable numerically (0=ultrafast...9=placebo). 
The exact mappings may change in the future if new presets are added or old presets removed, but they will always be in linear order of fast to slow.

The ffmpeg -vpre error message (if a user doesn't use a -vpre) has been made more descriptive, to better instruct users about how the ffmpeg -vpre system works.

weightp -1 offset dupes are now disabled in high bit depth mode, as they're a hack to get around crappy rounding in 8-bit encoding, and thus don't help in high bit depth.

PSNR and SSIM measurements are now VFR-aware! This means they will take into account the duration of frames. This may result in PSNR and SSIM results appearing dramatically different from other tools when used on variable framerate video. Don't fret; x264 is correct, they are not. Of course, results will stay the same for constant framerate video. This is the first step to VFR-aware MB-tree.

Quantizer handling has been improved. 
Library change: i_qpplus1 now defaults to X264_QP_AUTO. This doesn't actually change anything, as X264_QP_AUTO is 0, but this may change in the future. As a reminder, all x264_picture_t structs must be initialized using x264_picture_init or x264_picture_alloc! 
CRF values now make sense in high bit depth: --crf 23 means roughly the same quality in 8-bit or 10-bit, instead of being 4 times higher quality in 10-bit than in 8-bit. 
This means that --qp 0 should be used for lossless, not --crf 0. The latter will not result in lossless compression when in high bit depth mode. 
Bit depth has been added to statsfiles, to prevent users from inadvertently using a statsfile with the wrong bit depth.

Scenecut's flash detection has been improved to work more sanely in the case of flashes at the end of the videos.

===== Upcoming: =====

High bit depth assembly code is nearly ready. Speed boost has been measured as about 4.3x and is still improving.

VBV Emergency Mode is finally completed, with just fine-tuning and bugfixing left. This makes x264 able to deal gracefully with extreme input combined with VBV restrictions (e.g. noise, Doremi Labs test boxes). This is critical for some broadcast applications. Unlike most competing encoders, this VBV Emergency Mode does not drop frames or force all blocks to skip or some similarly extreme step: it adds a "denoising" step to reduce the complexity of the video, which scales upwards until it simply removes all content at the most extreme level.

A pad filter is nearly ready for the x264CLI filter framework.

HQDN3D (denoiser) and YADIF (deinterlacer) will be making their way into the x264CLI filter framework.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding mostly finished.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel.

== [http://mailman.videolan.org/pipermail/x264-devel/2010-November/007950.html Volume 5] ==

''11/19/2010''

This is the fifth x264 development newsletter. If you missed the first four, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next. Previous versions can be found in the mailing list archives.

Note that we pushed a bugfix release this time around, so this newsletter includes fixes from those commits as well, i.e. it covers everything since the last newsletter.

===== Fixes: =====

HRD now works correctly when used with intra refresh. Thanks to a certain x264 commercial licensee for the bug report.

QPfile parsing now works as it was supposed to have worked last time: users should be able to omit QP values.

Allocate the correct amount of memory for weightp buffers with weightp + high bit depth (it allocated too much, not too little).

Fix flash detection to work correctly near the end of the keyframe interval.

Fix a crash in dump yuv with some resolutions.

Various fixes have been made to ratecontrol in high-bit-depth mode.

Constrained intra pred is working properly again in all cases.

===== Improvements: =====

x264's SEI header now indicates whether the build used was GPL or proprietary.

x264 is now compatible with FFMS2's most recent API break.

configure now logs test programs that failed, not just the error output.

Merge Oskar's (irock's) 10-bit asm branch: ~4.4x overall speed boost in high bit depth mode.

Chroma weighted prediction: dramatically improved chroma compression and quality in fades.

Custom cropping rectangle support: users can now specify --crop-rect to add values to the H.264 cropping header. This is supposedly useful for 3D television applications (to allow legacy decoders to access only one view of the image).

===== Upcoming: =====

VBV Emergency Mode is finally completed, with just fine-tuning and bugfixing left. This makes x264 able to deal gracefully with extreme input combined with VBV restrictions (e.g. noise, Doremi Labs test boxes). This is important for some broadcast applications.

The pad filter and yadif are nearly ready for the x264CLI filter framework. Both now support high bit depth.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding finished and inter coding begun.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel (these guys move at the speed of a three-toed sloth swimming down a river of bricks).

===== Other news: =====

Since we started a couple of months ago, over 40 companies have contacted us regarding x264 licensing!

At least one major Blu-ray authoring house is switching to x264 for their commercial Blu-rays.

At least one commercial encoding application based on x264 is currently in the works. An announcement will come Soon™.

== [http://mailman.videolan.org/pipermail/x264-devel/2010-November/008003.html Volume 6] ==

''11/25/2010''

This is the sixth x264 development newsletter. If you missed the first five, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next. Previous versions can be found in the mailing list archives.

Note that we pushed a bugfix release this time around, so this newsletter includes fixes from those commits as well, i.e. it covers everything since the last newsletter.

===== Fixes: =====

Fix some crashes in high bit depth on some compilers due to insufficient array alignment.

Fix a bug in chroma weightp which could cause corruption in rare cases.

Fix some weird issues in the resize filter's rounding code.

Fix the build on SPARC Solaris 10 machines.

Fix high bit depth mode on SPARC.

Fix an odd issue in VFR input + forced timebase.

===== Improvements: =====

QPmin default is now 0.

x264 --version now prints a lot more useful information.

x264_encoder_reconfig now copies field order flags.

There's a new API function to return the maximum number of delayed frames with the current parameters (requested by Gstreamer).

Google Code-In is in full swing, with two sets of patches committed: 
SSE versions of some high-bit-depth (i)DCT functions 
x264 now has a totally sweet Python regression test tool. 

--weightp 1 is now a better speed tradeoff. It also doesn't do any reference duplication, so it's suitable when encoding for broken decoders that don't handle reference duplication correctly (while still getting the benefits of fade detection).

===== Upcoming: =====

VBV Emergency Mode is finally completed, with just fine-tuning and bugfixing left. This makes x264 able to deal gracefully with extreme input combined with VBV restrictions (e.g. noise, Doremi Labs test boxes). This is important for some broadcast applications.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding finished and inter coding begun.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel (these guys move at the speed of a three-toed sloth swimming down a river of bricks).

===== Other news: =====

<strike>A commercial encoding application based on x264 is currently in the works. An announcement will come this week.</strike>

This week? Perhaps I should have said "today"!

Pegasys Inc. switches to x264 for their flagship product, TMPGEnc.

English: http://tmpgenc.pegasys-inc.com/en/press/10_1125.html

日本語: http://tmpgenc.pegasys-inc.com/ja/press/10_1126.html

== [http://mailman.videolan.org/pipermail/x264-devel/2010-December/008041.html Volume 7] ==
''12/07/2010''

This is the seventh x264 development newsletter. If you missed the first six, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next. Previous versions can be found in the mailing list archives.

===== Fixes: =====

Fix SPARC and Solaris building (really this time!)

Three timecode input fixes: for nonzero starting pts, for a corner case in auto timebase generation, and fixing a spurious error in case of a frame read error.

Actually fix fittobox resizing code.

Fix incorrect handling of some pixfmts with non-mod2 resolutions in the resize filter.

Fix a file handle leak in libx264 with certain file reading errors in statsfile handling.

Fix intra refresh's quality masking: the ipratio wasn't correctly applied in threaded mode.

Fix an overflow in fdct asm in 10-bit builds.

===== Improvements: =====

Automatically restrict QPs to avoid quantization (under|over)flow, so you don't have to. Will error out if this, combined with insane user constraints, makes encoding impossible. This makes fprofiling work out of the box again.

Use the Avisynth 2.6 API to detect Avisynth initialization failure and print the associated error (only available in Avisynth 2.6 alphas; doesn't affect older versions).

Install an x264_config.h containing information about how x264 was configured. This will allow ffmpeg to link with commercially licensed copies of libx264 without --enable-gpl.

Massive amounts of high bit depth assembly code from Google Code-In (about two dozen functions).

===== Upcoming: =====

--device and automatic --level restriction support is in the works, as part of Google Code-In.

A per-option help system is in the works, as part of Google Code-In.

VBV Emergency Mode is finally completed, with just fine-tuning and bugfixing left. This makes x264 able to deal gracefully with extreme input combined with VBV restrictions (e.g. noise, Doremi Labs test boxes). This is important for some broadcast applications.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding finished and inter coding begun.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel (these guys move at the speed of a three-toed sloth swimming down a river of bricks).

== [http://mailman.videolan.org/pipermail/x264-devel/2010-December/008070.html Volume 8] ==
''12/15/2010''

This is the eighth x264 development newsletter. If you missed the first seven, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next. Previous versions can be found in the mailing list archives.

===== Fixes: =====

Weightp analysis now calculates offsets correctly in high bit depth mode.

High bit depth intra pred x86 asm functions are now fixed (and thus re-enabled).

That nasty gnu ld alignment bug on windows (ld refuses to align .bss to 16 bytes even when asked nicely) has been squashed with -fno-zero-initialized-in-bss.

===== Improvements: =====

.gitignore is now even more useful than before.

Another bit of memory saved in high bit depth weightp.

The regression test script now tests interlaced compression.

8x8dct is now allowed with cavlc+lossless and subme>=6, for a slight compression improvement.

Frame-packing SEI support is now in, for official 3D support in x264.

A metric crapload of new high bit depth asm functions are now in (thanks, Google Code-In!)

x264 now supports Windows threads in addition to pthreads. Thanks to Pegasys Inc. for the original patch.

===== Upcoming: =====

--device and automatic --level restriction support is in the works, as part of Google Code-In.

A per-option help system is in the works, as part of Google Code-In.

VBV Emergency Mode is finally completed, with just fine-tuning and bugfixing left. This makes x264 able to deal gracefully with extreme input combined with VBV restrictions (e.g. noise, Doremi Labs test boxes). This is important for some broadcast applications.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding finished and inter coding begun.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel (these guys move at the speed of a paraplegic three-toed sloth swimming down a frozen river of bricks).

== [http://mailman.videolan.org/pipermail/x264-devel/2011-January/008132.html Volume 9] ==
''01/10/11''

This is the ninth x264 development newsletter. If you missed the
first eight, this is a regular email containing updates on fixes and
improvements in the most recent x264 push, along with updates on
what's coming next. Previous versions can be found in the mailing
list archives.

===== Fixes: =====

The win32 alignment crash issue is actually fixed now. Apparently gcc doesn't work right unless you initialize zero arrays explicitly.

High bit depth now compiles with --disable-asm properly again.

SATD predictors with high bit depth work properly with --no-mbtree on. 
*Bug resulted in bizarre/wrong bitrates in CRF mode with --no-mbtree.

The lavf demuxer no longer memory leaks like crazy on inputs with multiple video streams.

Some overflows in VFR ratecontrol with extreme timebases have been fixed. These affected some actual videos.

SSIM now works properly in 10-bit.

A bug with negative costs that caused assertions on some videos in high bit depth mode has been fixed.

YV12 now works correctly in the resize filter (there was an issue with swapped chroma planes).

The endianness test and as test in configure now work properly in cross-compile mode.

Weightp no longer prints random extra linebreaks to the statsfile.

A few minor improvements have been made to the parameter SEI.

===== Improvements: =====

Some incorrect GCC uninitialized variable warnings have been shut up.

Metric craptons of high bit depth assembly have been added, for significant performance improvements. Thanks to Google Code-In (now over) and our students for their enormous amount of work, particularly Daniel Kang.

Also a few improvements in 8-bit assembly functions as well.

Check an extra offset in weightp analysis, for improved compression on fades with --weightp 1 (and slightly, but much less, improved with weightp 2).

Improve reference ordering in 3D frame-interleaved mode.

VFR-aware ratecontrol: MB-tree and non-MB-tree ratecontrol will now be aware of how long frames last. This results in greatly improved quality in VFR video, particularly dedup'd anime and similar content. It is now even MORE important that calling applications give x264
correct timestamps! Note this also affects the definition of CRF; higher framerate videos will get lower quality at the same CRF, which should improve how well CRF measures quality (as compared to the human brain itself).

===== Upcoming: =====

--device and automatic --level restriction support is in the works, as part of Google Code-In. The patch is done, but needs review.

A per-option help system is in the works, as part of Google Code-In. 
The patch is done, but needs editing of the help entries.

VBV Emergency Mode is finally completed, with just fine-tuning and bugfixing left. This makes x264 able to deal gracefully with extreme input combined with VBV restrictions (e.g. noise, Doremi Labs test boxes). This is important for some broadcast applications. This was
delayed due to Google Code-In.

Adaptive MBAFF development is coming along, with B-frames being finished up currently.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding finished and inter coding begun.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel (these guys move at the speed of a paraplegic three-toed sloth swimming down a frozen river of bricks while chained to an osmium anchor).

== [http://mailman.videolan.org/pipermail/x264-devel/2011-January/008192.html Volume 10] ==
''01/26/2010''

This is the tenth x264 development newsletter. If you missed the first nine, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next. Previous versions can be found in the mailing list archives.

===== Fixes: =====

Some asm fixes to high bit depth: fix cases where SSE2-only instructions were used in MMX functions (caused SIGILL on old CPUs).

Fix the cacheline check in cache32 avg2 code (minor speed improvement on some old CPUs).

Fix CPU detection on Windows systems with >= 64 CPUs. x264 still doesn't use more than one processor group (it really doesn't need to), but at least it'll work now.

Fix reconfiguration of b_tff: make changing field order during encoding work properly. Required for ffmpeg to signal TFF/BFF, since ffmpeg's TFF/BFF is signaled per-frame instead of per-stream.

Fix cases where x264_encoder_encode failures (due to malloc failure) could cause x264_encoder_close to crash.

Write the 3D frame packing SEI values more correctly (thanks to Nero for the bug report).

===== Improvements: =====

Bump all the dates to 2011.

Add some missing values to the non-extended SAR table.

Improve the regression test script to not break with really complex commandlines on OSs with limited filename length, as well as correctly return to the original branch the user is on after testing.

Add an --input-fmt option to serve the same purpose of -f in ffmpeg; useful for cases where ffmpeg can't successfully probe the format of the input file.

Double the base framerate of frame-sequential 3D files; a "60fps" frame-sequential 3D file is really only 30fps.

Add AVX support to x264's assembly abstraction layer as well as one example function: still needs the actual AVX code to be written, but x264 can now take advantage of 3-operand instructions.

VBV Emergency Mode is finally completed. This makes x264 able to deal gracefully with extreme input combined with VBV restrictions (e.g. noise, Doremi Labs test boxes). This is important for some broadcast applications.

===== Upcoming: =====

--device and automatic --level restriction support is in the works, as part of Google Code-In. The patch is done, but needs review.

A per-option help system is in the works, as part of Google Code-In. 
The patch is done, but needs editing of the help entries.

Adaptive MBAFF development is actually coming along for once this time.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding finished and inter coding begun.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel (these guys move at the speed of a paraplegic three-toed sloth swimming down a river of frozen helium while chained to an osmium anchor stuck inside a black hole).

== [http://mailman.videolan.org/pipermail/x264-devel/2011-February/008251.html Volume 11] ==
''02/06/2011''

This is the eleventh x264 development newsletter. If you missed the first nine, this is a regular email containing updates on fixes and improvements in the most recent x264 push, along with updates on what's coming next. Previous versions can be found in the mailing list archives.

===== Fixes: =====

Fix various regressions introduced in VBV emergency.

Fix AVX detection on OSs that don't support AVX.

Fix a crash on Phenom that could occur with the lookahead disabled and a calling application that tweaks the SSE control flags.

Fix a divide-by-zero crash in the MKV/FLV muxers that could occur if encoder initialization failed.

Fix a race condition that resulted in slightly wrong frame durations calculated for VFR input.

Fix an overflow in i16x16 NEON planar prediction, backported from ffmpeg.

In x86inc.asm, error on duplicate functions instead of silently renaming one of them.

===== Improvements: =====

Enable FastShuffle on crippled Nehalem/Penryn CPUs without SSE4.

Output pic_struct information in the libx264 API.

Allow weightp_fake in interlaced mode: it seems to work fine as-is.

In Windows, restore the old console title after encoding. MSDN says this is supposed to happen automatically, but MSDN is wrong.

Make x264cli update its progress every 0.25 seconds instead of every certain number of frames. This should be more visually consistent and fix some problems with extremely fast-updating status bars (e.g. from FFMS indexing of large files) in some situations.

===== Upcoming: =====

--device and automatic --level restriction support is in the works, as part of Google Code-In. The patch is done, but needs review.

A per-option help system is in the works, as part of Google Code-In. The patch is done, but needs editing of the help entries.

Adaptive MBAFF development is actually coming along for once this time. Inter frames are mostly done.

x262 is under development: a best-in-class MPEG-2 encoder built using the x264 framework. Basic structure is done, with intra coding finished and inter coding begun.

Work is planned to integrate x264 with the Sandy Bridge's encoding ASIC for improved encoding performance. Current status is: waiting on Intel (these guys move at the speed of a one-legged paraplegic three-toed sloth swimming down a river of frozen helium while chained to an osmium anchor stuck inside a black hole).

== [http://mailman.videolan.org/pipermail/x264-devel/2011-February/008293.html Volume 12] ==
''02/18/2011''

This is the twelfth x264 development newsletter. If you missed the
first eleven, this is a regular email containing updates on fixes and
improvements in the most recent x264 push, along with updates on
what's coming next. Previous versions can be found in the mailing
list archives.

===== Fixes: =====

Fix some Intel compiler warnings.

Fix a crash in the mp4 muxer in x264cli if x264_encoder_open failed.

Fix a case in which a malloc(0) was done; on some OSs, like Solaris,
this returned a null pointer instead of a pointer to a 0-byte memory
block, thus causing libx264 to think that its malloc failed. Both
these behaviors are POSIX-compatible.

Fix a minor bug in RD caused by a stray semicolon. It probably didn't
affect much.

Fix a memory leak on encoder close if not all frames were flushed
during encoding.

===== Improvements: =====

Save memory: don't init MV cost data that isn't needed.

Use bs_write1 for all 1-bit header bits, not just some.

Even more accurate handling of slice-max-size escape bytes: track the
actual bytes rather than trying to estimate them. Slightly slower,
but should cover even the most insane corner cases (we found a corner
case in which x264 could literally end up outputting one escape byte
for every single macroblock!).

In frame-packing 3D, don't place scenecuts on the right view, ever.
Some players dislike this. Note: x264 won't stop you from using an
odd keyint, which will mess this up.

The first edition of Daniel Kang's massive AVX patch is in --
hopefully this will give a few % of performance on the new Sandy
Bridge chips.

===== Upcoming: =====

--device and automatic --level restriction support is in the works, as
part of Google Code-In. The patch is done, but needs review.

A per-option help system is in the works, as part of Google Code-In.
The patch is done, but needs editing of the help entries.

Adaptive MBAFF development is actually coming along for once this
time. Inter frames are basically done.

x262 is under development: a best-in-class MPEG-2 encoder built using
the x264 framework. P-frames are done and working.

Work is planned to integrate x264 with the Sandy Bridge's encoding
ASIC for improved encoding performance. Current status is: waiting on
Intel (these guys move at the speed of an obese one-legged paraplegic
three-toed sloth swimming down a river of frozen helium while chained
to an osmium anchor stuck inside a black hole).

X264 Development Newsletters

2011-02-07T07:06:41Z

Saintdev:

X264 Development Newsletters

2011-01-31T02:24:52Z

Saintdev:

X264 Development Newsletters

2011-01-30T04:25:36Z

Saintdev:

X264 Development Newsletters

2011-01-29T05:49:02Z

Saintdev:

X264 Development Newsletters

2011-01-28T02:45:09Z

Saintdev: /* Volume 7 */

X264 Development Newsletters

2011-01-28T02:42:48Z

Saintdev:

X264 TODO

2011-01-27T06:50:18Z

Saintdev: /* x86 assembly */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Port functions (where useful) to AVX, talk to Daniel Kang for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work more on 10-bit asm.
* If you read this, join #x264dev on irc://irc.freenode.net and ping holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.
* SPARC VIS assembly is only available when high bit-depth is disabled.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

X264 TODO

2011-01-27T06:46:57Z

Saintdev: /* x86 assembly */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Port functions (where useful) to AVX, talk to Daniel Kang for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work more on 10-bit asm.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.
* SPARC VIS assembly is only available when high bit-depth is disabled.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

X264 TODO

2011-01-27T06:45:52Z

Saintdev: /* x86 assembly */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work more on 10-bit asm.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.
* SPARC VIS assembly is only available when high bit-depth is disabled.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

X264 TODO

2011-01-27T06:45:01Z

Saintdev: /* GPU */

This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.

Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].

If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.

Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.

===Motion Estimation===
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.
* Somehow take into account the effect of motion vector decision on future blocks.
** Hierarchical motion estimation
** Approximations from lookahead MVs
** Iterative ME (as per Snow)
** Trellis motion estimation
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.
** But how do we know which ones we can afford to skip, and when?
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?
* Try sub-8x8 partitions in B-frames. Is it at all useful?
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?
* Fullpel chroma ME?
** For TESA?

===Intra Analysis===
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?

===Mode Decision===
* Can we find more ways to skip more motion searches in multiref?
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?
* See the TODOs for deblock-aware RD in common/deblock.c.
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?

===Psy===
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.
* RD should take into account local variance.
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.
* Lambda should be picked independent of quantizer (i.e. with greater precision).
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.

===Lookahead===
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?

===Quantization===
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?

===Transform===
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].

===Interlacing===
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?
* Constrained intra + adaptive MBAFF. Does anyone care about this?
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.

===Weighted Prediction===
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.

===Ratecontrol===
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.
* Make the frame size and row size predictors better. They currently are kind of crappy.
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.
** Just force all blocks to skip?
** Drop frames?
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.

===GPU===
* Motion estimation?
** Methods
*** Hierarchical?
*** 2D Wave?
*** Something else?
** "Easy": lookahead motion estimation
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.
** "Hard": main motion estimation
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.
*** But potentially more useful...
* Other things?

===x86 assembly===
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.
* Optimize more for the Phenom.
* Optimize for the upcoming Sandy Bridge.
* Work more on 10-bit asm.
* Convince holger to commit his local patches.
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.

===Other assembly===
* NEON assembly is nowhere near complete.
** Chroma MC needs to be rewritten for NV12 support.
* Altivec assembly is very lacking.
* SPARC VIS assembly is only available when high bit-depth is disabled.

===Other CPU optimizations===
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?

===Other features===
* MPEG-2 encoding support
* VP8 encoding support
* '''4:2:2 and 4:4:4 colorspace support'''
** '''RGB?'''
* Support for timecodes
* Merge speedcontrol

===x264CLI===
* Finish audio support. Talk to Kovensky about this one.
* Make the filtering system aware of fullrange vs TV range.
* Make the filtering system aware of BT.601 vs BT.709.
* Add more filters.
** Deinterlacers (YADIF).
** Denoisers (HQDN3D?).
** IVTC, decomb?
* Merge L-SMASH mp4 muxer.
* Add TS muxing support using HRD. Talk to kierank about this one.
* Add --device support.
* Add automatic --level restriction support.

X264 Development Newsletters

2011-01-27T06:37:45Z

Saintdev: /* Other news: */

X264 Development Newsletters

2011-01-27T06:35:45Z

Saintdev:

X264 Development Newsletters

2011-01-27T06:30:30Z

Saintdev: /* Volume 4 */

X264

2011-01-26T07:44:45Z

Saintdev: /* Developers */

{{Lowercase}}
'''x264''' is a free software library for encoding [[H.264/MPEG-4 AVC]] video streams. The code is written by Loren Merritt, Laurent Aimar, Eric Petit, Min Chen, Justin Clay, Måns Rullgård, Radek Czyz, Alex Izvorski, Alex Wright, and Christian Heine from scratch.

x264 has a strong user community based at [http://www.Doom9.org Doom9] where discussions for improved development take place.

== Developers ==

* [http://developers.videolan.org/x264.html x264 home page]
* [[x264 Development Newsletters]]
* [[Development Notes]]
* [[X264 TODO]]
* [[x264 asm intro]]

== Demos ==
[http://lives.rm.org/demos/flyingskunk.mkv flyingskunk.mkv] demo encoded with x264/vorbis/matroska - (96 MB, 640x480, 25 fps, 6 minutes), created using [http://lives.sourceforge.net LiVES]

{{stub}}

[[Category:VideoLAN projects]]

X264 Development Newsletters

2011-01-26T07:41:53Z

Saintdev: /* Volume 5 */

X264 Development Newsletters

2011-01-26T07:40:56Z

Saintdev: /* Volume 4 */

X264 Development Newsletters

2011-01-26T07:40:00Z

Saintdev: /* Volume 2 */

X264 Development Newsletters

2011-01-26T07:39:24Z

Saintdev: /* Volume 3 */

X264 Development Newsletters

2011-01-26T07:36:50Z

Saintdev: /* Volume 2 */

X264 Development Newsletters

2011-01-26T07:27:38Z

Saintdev: /* Improvements: */

X264 Development Newsletters

2011-01-26T07:22:07Z

Saintdev:

X264 Development Newsletters

2011-01-26T07:14:07Z

Saintdev: Created page with "This is an archive of the x264 Development Newsletters. == Volume 1 == ''9/18/2010'' From now on, with each weekly-ish push of new patches, I'll try to post a newsletter as…"