SoC 2008/x264 Improve Fast Inter Refinement and Adaptive Quantization

This project is part of Google Summer of Code 2008. Student: Joey Degges Mentor: Jason Garrett-Glaser

Abstract

The goal of this project is to improve heuristics and decision-making for inter refinement in order to improve efficiency given average encoding settings. This will involve various early termination heuristics along with methods of deciding which partition modes need to be searched while performing minimal actual searching on them. I also plan to experiment with different methods that can be used to improve psycho-visual optimizations for mode decisions and quantization. This will include improving variance adaptive quantization by experimenting with different methods which could be used to weight the variance in order to select a more optimized quantizer.

Goals

Goals:

• Improve inter prediction algorithms through the addition of early termination heuristics

1. Analyze the costs of using the different reference frames and partition modes.
2. Uses these results to create rules for early termination in order to avoid exhaustive searches.

• Explore the usage of texture and shape in inter prediction

1. Implement curvature/first derivative algorithms (edge detection).
2. Compare edge count with the costs associated with the reference frames and partition modes.
3. Explore other metrics of texture/shape and their usefulness in this situation.
4. Implement early termination heuristics that depend on these texture and shape metrics.

• Improve psycho-visual optimizations for mode decisions and quantization

1. Explore methods for SSIM-QNS optimization.
2. Adaptive dead zone / lambda.
3. Make improvements upon the Adaptive Quantization algorithms to achieve higher visual quality.
   1. Investigate the usage of NSSE and noise shaping techniques.
   2. Explore the usage of curvature as a means of weighting the variance in order to achieve a more optimized quantizer.
   3. Determine the most effective metric or combination of metrics.