https://wiki.videolan.org/api.php?action=feedcontributions&feedformat=atom&user=KieranVideoLAN Wiki - User contributions [en]2026-11-04T15:54:46ZUser contributionsMediaWiki 1.43.11https://wiki.videolan.org/index.php?title=VDD19&diff=61957VDD192019-11-10T00:41:54Z<p>Kieran: /* Sunday November 10 */</p>
<hr />
<div>{{VDD}}<br />
[[Image:Vdd2019.png|thumb|500px|]]<br />
= Welcome to Video Dev Days 2019 =<br />
<br />
The [[VideoLAN]] non-profit organization is happy to invite you to the multimedia open-source event of the fall!<br />
<br />
For its eleventh edition, people from the VideoLAN and open source multimedia communities will meet in '''Tokyo''' to discuss and work on the future of the open-source multimedia community.<br />
<br />
This is a very technical conference, focused on low-level multimedia.<br />
<br />
== Who can come? ==<br />
<br />
Literally every one is invited to VDD, we just need registration for legal reason ''(to enter the building)''.<br />
<br />
This is a technical conference, focused on low-level multimedia, like codecs and their implementations like '''x264''' or '''dav1d''', frameworks like '''FFmpeg''' or '''Gstreamer''' or playback libraries like '''libVLC'''.<br />
<br />
Things like Javascript video frameworks or ad-injection frameworks would be considered out of scope.<br />
<br />
The price to enter is totally '''Free'''.<br />
<br />
== Sponsorship for Japan ==<br />
<br />
Since '''Japan''' is a bit further than the usual conferences, often in Europe, going to Japan is more expensive.<br />
<br />
If you want [[VideoLAN]] to sponsor your trip, you should read the [[VDD19/Sponsor_Policy|Sponsorship Policy]].<br />
<br />
== Traveling to Japan ==<br />
<br />
You should consider staying a bit longer to '''Japan''', if you can afford a few extra days. Kyoto is amazing!<br />
<br />
You should take care of your passport and, if applicable, your visa very quickly!<br />
<br />
See the dedicated page: [[VDD19/Traveling_Advice|traveling advice for Japan]].<br />
<br />
You might want to consider to '''offset your carbon footprint'''.<br /><br />
For example: [https://co2.myclimate.org/en/flight_calculators/new MyClimate CO2 Calculator] or [https://www.carbonfootprint.com/carbonoffset.html CarboonFootPrint.com]<br />
<br />
= Schedule =<br />
<br />
== Friday November 8th ==<br />
<br />
Community bonding day !<br/><br />
<br />
=== Pokemon : Codec Wars ! ===<br />
<br />
Discover your team members in a discovery game around Tokyo.<br/><br />
Please meet in the sponsored hotel lobby at 9:30am<br />
<br />
[https://wiki.videolan.org/VDD19/#Accommodation_.2F_Hotel Hotel Villa Fontaine Hakozaki]<br />
<br />
=== Evening drinks ===<br />
<br />
19:?? Evening drinks at ??<br />
<br />
'''Sponsored by Mozilla.'''<br />
<br />
''(NB: food is not provided there)''<br />
<br />
== Saturday November 9th ==<br />
<br />
=== Planning ===<br />
<br />
{| class="wikitable" style="text-align: center;"<br />
|-<br />
! style="width: 10%;" | Time || Opera 1 Auditorium <br />
|-<br />
! '''08:30 - 09:00''' || style="background: silver;" | Registration<br />
|-<br />
| '''09:00 - 09:10''' || '''Welcome words''' by Jean-Baptiste Kempf, VideoLAN<br />
|-<br />
| '''09:10 - 09:20''' || '''Internet Initative Japan''' by Marie Gunji, IIJ<br />
|-<br />
| '''09:20 - 09:50''' || '''Dav1d''' by Ronald S. Bultje, Two Orioles<br />
|-<br />
| '''09:50 - 10:20''' || '''Sash and low latency adaptive streaming''', by Phil Cluff, Mux<br />
|-<br />
| '''10:20 - 10:50''' || '''x265 update''', by Pradeep Ramachandran, MulticoreWare inc.<br />
|-<br />
| '''10:50 - 11:05''' || style="background: silver;" | Break<br />
|-<br />
<br />
<br />
| '''11:05 - 11:35''' || '''rav1e update''' by Thomas Daede, Mozilla<br />
|-<br />
| '''11:35 - 12:05''' || '''VLC 4.0''' by Jean-Baptiste Kempf, VideoLAN<br />
|-<br />
| '''12:05 - 12:20''' || '''Tokyo Video Tech'''<br />
|-<br />
| '''12:30 - 14:00''' || style="background: silver;" | '''Bentō lunch break''' sponsored by J-Stream<br />
|-<br />
| '''13:30 - 14:00''' || style="background: silver;" | '''Group photo''' (Goodies distribution will start before 13:30)<br />
|-<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center;"<br />
|-<br />
! style="width: 10%;" | Time || Opera 1 Auditorium || Opera 2 Auditorium || Meeting Room<br />
! <br />
|-<br />
| '''14:00 - 16:00''' || '''VLC Technical meeting''' || '''AV1 meetup''' || '''Hacking''' open sessions<br />
|-<br />
| '''16:00 - 18:00''' || '''VideoLAN meeting''' || '''Meetups''' || '''Hacking''' open sessions<br />
|-<br />
| '''19:30 - 21:30''' ||colspan=4| '''Community Dinner''' by the Dinner committee<br />
|}<br />
<br />
=== VideoLAN community dinner ===<br />
<br />
You '''really''' need to '''be on time'''<br />
<br />
Warayakiya Ginza<br />
[https://goo.gl/maps/Q8S6DV1u3FdkZUFg6 Ginza, 6 Chome−5−15] <br />
銀座能楽堂ビル 6F<br />
<br />
わらやき屋 銀座店<br />
[https://goo.gl/maps/Q8S6DV1u3FdkZUFg6 銀座６丁目５−15]<br />
銀座能楽堂ビル 6F<br />
<br />
== Sunday November 10 ==<br />
<br />
{| class="wikitable" style="text-align: center;"<br />
|-<br />
!style="width: 10%;" | Time || Description<br />
|-<br />
| '''09:30 - 12:00''' || '''Lightning talks'''<br />
* Intel update from Max<br />
* llvm-mingw by Martin Storsjo<br />
* 360 intra directional predictors by Luc Trudeau<br />
* Vimeo elevator by Rapahel Zummer<br />
* GSoC experiences by Krishnan<br />
* VLC Benchmark by Duncan<br />
* A Japanese Way to Maintain Constant Quality.... byYusuke Goto<br />
* Patents by Vittorio<br />
* TTML in mp4 by JEEB<br />
* BlitzLoop by marcan<br />
* color: daala, dav1d and beyond<br />
* Compiler assisted security<br />
* Rust with C by Luca<br />
* Sub-frame decoding by Kieran<br />
|-<br />
| '''12:30 - 14:00''' || style="background: silver;" | '''Bentō lunch break''' sponsored by Abema TV<br />
|}<br />
<br />
{| class="wikitable" style="text-align: center;"<br />
|-<br />
! style="width: 10%;" | Time || Opera 1 Auditorium || Opera 2 Auditorium || Meeting Room<br />
! <br />
|-<br />
| '''14:00 - 16:00''' || '''Unconferences''' || '''FFmpeg''' meetup || '''Hacking''' open sessions<br />
|-<br />
| '''16:00 - 18:00''' || '''Unconferences''' || '''Meetups''' || '''Hacking''' open sessions<br />
|-<br />
| '''18:30 - 21:30''' || colspan=3 style="background: silver;" | '''Unofficial Dinner''' and Karaoke sessions<br />
|}<br />
<br />
<br />
=== Unofficial Dinner ===<br />
<br />
Amusement Bar J-House<br />
[https://goo.gl/maps/hHFW8suntPcsS6o4A Chiyoda City, Fujimi, 2 Chome−3−14] <br />
CST ビル 5F<br />
<br />
アミューズメントバー ジェイハウス<br />
[https://goo.gl/maps/hHFW8suntPcsS6o4A 東京都千代田区富士見2-3-14] <br />
CSTビル5F<br />
<br />
= Practical information =<br />
<br />
* The [https://www.japan.travel/en/plan/ official '''JNTO''' site] has a wealth of practical information targeting first-time visitors.<br />
* The [[VDD19/Air_travel|air travel]] page provides more info for arrival and departure.<br />
<br />
== Location / main venue ==<br />
The meeting is happening at the [https://goo.gl/maps/jkzwDAz99qDk84V29 IIJ HQ]:<br />
{| class="wikitable"<br />
!In English<br />
!In Japanese<br />
|-<br />
|Internet Initiative Japan Inc.<br/><br />
Iidabashi Grand Bloom<br/><br />
2-10-2 Fujimi, Chiyoda-ku<br/><br />
Tokyo 102-0071<br/><br />
Japan<br />
|株式会社インターネットイニシアティブ<br/><br />
〒102－0071 東京都千代田区富士見2-10-2<br/><br />
飯田橋グラン・ブルーム<br />
|}<br />
<br />
'''Do not lose the issued access pass''', you will need it to re-enter the building during the 2 days.<br />
<br />
The venue is within walking distance of Iidabashi station served by:<br />
* JR East<br />
** Chuo-Sobu (JB16) train line,<br />
* Tokyo Metro<br />
** Tozai (T06),<br />
** Yurakucho (Y13) and<br />
** Namboku (N10) lines,<br />
* Toei<br />
** Odeo (E06) metro line.<br />
From the sponsored attendees' hotel, take the ''Tozai'' line west-bound from ''Kayabachō'' station (T11).<br />
<br />
== Accommodation / Hotel ==<br />
<br />
If you are sponsored by the [[VideoLAN]] association, your hotel is:<br />
{| class="wikitable"<br />
!In English<br />
!In Japanese<br />
|-<br />
|[https://www.hvf.jp/eng/location/hakozaki.php Hotel Villa Fontaine Tokyo Nihombashi Hakozaki]<br/><br />
[https://goo.gl/maps/4RZWFXwiBu76Xcwd6 20-10 Hakozaki-cho Nihombashi]<br/><br />
Chuo-ku Tokyo, 103-0015<br />
|ホテルヴィラフォンテーヌ東京日本橋箱崎<br/><br />
〒103-0015 東京都中央区日本橋箱崎町２０−1０<br />
|-<br />
|colspan="2"|+81-3-3667-3330<br />
|}<br />
Elsewise, please make your own arrangements for accommodation.<br />
<br />
Check-in is 3:00pm. Check-out is 11:00am.<br />
Breakfast available from 7:00 to 9:30am.<br />
<br />
'''Beware''' that there are plenty of ''Villa Fontaine'' hotels in the area. '''Be sure to head to the Hakozaki one.'''<br />
=== Public transit ===<br />
{| class="wikitable"<br />
!Origin<br />
!Recommended route<br />
|-<br />
|Conference venue<br />
|Tokyo Metro ''Tozai'' line east-bound to ''Kayabachō'' station (T11).<br />
|-<br />
|Sengakuji station<br/>Haneda airport via the [https://www.haneda-tokyo-access.com/en/ Keikyu railway]<br />
|Toei ''Asakusa'' metro line north-bound to ''Ningyōchō'' station (A14)<br />
|-<br />
|Ueno station<br/>Narita airport via the [http://www.keisei.co.jp/keisei/tetudou/skyliner/us/index.php Keisei electric railway]<br />
|Tokyo Metro ''Hibiya'' line south-bound to ''Ningyōchō'' station (H13).<br />
|-<br />
|[http://www.tcat-hakozaki.co.jp/en/ T-CAT] terminal<br/>Haneda airport via T-CAT<br/>Narita airport via T-CAT<br />
|Short walk.<br />
|-<br />
|Tokyo station<br/>Yaesu [http://www.keiseibus.co.jp/inbound/tokyoshuttle/en/ Keisei bus] stop<br/>Narita via Keisei bus<br />
|'''1,7 km''' walk from the North exit.<br />
|}<br />
In addition to Kayabachō and Ningyōchō, the Suitengu-mae station (Z10) on the Tokyo metro Hanzōmon line is also located close-by.<br />
<br />
== Transportation ==<br />
<br />
* Get a Japanese electronic [[#Electronic_payments|payment card]] to travel easily on the public transports.<br />
* Check the dedicated page for [[VDD19/Air travel#Airport_transfer|airport transfers]].<br />
* For routing, you can use the following services:<br />
{| class="wikitable"<br />
!Vendor<br />
!Website<br />
!Google Play<br />
!Apple App Store<br />
|-<br />
|Hitachi Hyperdia<br />
|[https://www.hyperdia.com Hyperdia]<br />
|[https://play.google.com/store/apps/details?id=com.hyperdia.android.activity here]<br />
|[https://apps.apple.com/fr/app/hyperdia-by-voice/id379001449 here]<br />
|-<br />
|Japan National Tourism Organization<br />
| -<br />
|[https://play.google.com/store/apps/details?id=jp.go.jnto.jota here]<br />
|[https://apps.apple.com/app/apple-store/id1230367186?mt=8 here]<br />
|}<br />
'''NOTE''': VideoLAN does not refund taxi, car rental and public transport.<br />
<br />
=== Rail pass ===<br />
If you have a voucher for a rail pass, you should save it for before or after the conferencing days. JR passes are of limited within central Tokyo. They are valid for the ''Yamanote'' loop and the ''Chuo-Sobu'' east-west transverse train lines, as well as JR-operated buses, but not on the metro networks.<br />
<br />
== Means of payment ==<br />
<br />
VISA and MasterCard cards are '''not''' universally accepted in Japan.<br />
<br />
=== Cash ===<br />
You will need to withdraw cash from an ''international'' ATM (accepting VISA, MasterCard, AmEx, Discovery and UnionPay). You can find those:<br />
* at arrivals in the airport terminal,<br />
* at '''[https://www.sevenbank.co.jp/intlcard/index2.html 7-Eleven]''' convenience stores,<br />
* at '''[https://www.family.co.jp/for_tourist/en.html Family Mart]''' convenience stores,<br />
* and during office hours in Japanese Post offices.<br />
Do not bother with the local banks ATMs, most of which do not accept overseas cards.<br />
To be on the safe side, you could exchange a few tens of thousands of yens in advance. Compare prices and avoid the extortionate bureaux de change at airports!<br />
<br />
=== Electronic payments ===<br />
<br />
For small day-to-day purchases, you should procure a FeliCa electronic payment card, a.k.a. an ''IC'' card. It can be used to enter and exit public transports (train, metro, bus, etc), to check out at convenience stores and many shops. You can load the card with Japanese bank notes at dedicated machines. There are two options in Tokyo area:<br />
<br />
{| class="wikitable"<br />
!Vendor<br />
!Regular card<br />
!Tourist card<br />
!Tourist point of sale<br />
|-<br />
!JR East<br />
|[https://www.jreast.co.jp/e/pass/suica.html Suica]<br/>¥500 deposit<br />
|[https://www.jreast.co.jp/e/welcomesuica/welcomesuica.html Welcome Suica]<br/>free??<br />
|JR East Travel Service Center<br />
|-<br />
!Tokyo Metro<br />
|[https://www.pasmo.co.jp/visitors/en/normalpasmo/ Pasmo]<br/>¥500 deposit<br />
|[https://www.pasmo.co.jp/visitors/en/about/ Pasmo passport]<br/>¥500<br />
|<br />
*Keisei at Narita airport<br />
*Keikyu at Haneda airport<br />
|}<br />
Tourist cards with their cutesy design are meant to keep as memento and provide third-party discounts. But beware that they expire after 28 days (including any unused credit).<br />
<br />
== Electricity ==<br />
Japan has the lowest household voltage in the world, so '''check the specifications''' on your power adapters.<br />
Outlets are not compatible with European or Commonwealth plugs. North American ''grounded'' plugs (with 3 pins) are also troublesome.<br />
Most attendees will thus need passive adapters. They are easier to find at home, so '''BUY BEFORE DEPARTURE'''.<br />
{| class="wikitable"<br />
!Voltage<br />
|100 V<br />
|-<br />
!Frequency<br />
|50 Hz (60 Hz in Kyoto/Osaka)<br />
|-<br />
!Socket<br />
|Type A with only 2 pins (i.e. ungrounded)<br />
|}<br />
<br />
== Connectivity ==<br />
<br />
The telephone prefix for Japan is '''+81'''.<br />
<br />
=== Mobile data ===<br />
You can buy, or better yet pre-order, a prepaid data SIM card at international arrivals at the airport.<br />
Don't postpone, as it might be difficult to find elsewhere.<br />
<br />
=== Wireless LAN ===<br />
Free Wi-Fi access points are common, but they may need registration, possibly in Japanese only.<br />
There is a map of known Wi-Fi access points in the '''JNTO''' app (see [[#Transportation]]).<br />
<br />
=== Group Chat ===<br />
Participants can share information and discuss using the dedicated [https://t.me/joinchat/GIwXP1WmM84pbxCXStaUew VDD 2019 Telegram group].<br />
<br />
== Emergency ==<br />
<br />
'''119''' is the fire and ambulance emergency number in Japan. '''110''' is the local police number.<br />
<br />
Please carry your travel insurance card or certificate on you at all times.<br />
If you do not have a travel insurance yet, buy a fixed-term one before you depart. EU social security cards are not '''not''' valid in Japan.<br />
Advise your relatives, and if applicable, your government:<br />
* [https://pastel.diplomatie.gouv.fr/fildariane/dyn/public/login.html Registration for French citizens].<br />
<br />
Personal safety risks in Japan primarily stem from natural disasters:<br />
* [https://www.jnto.go.jp/safety-tips/eng/index.html '''JNTO''' safety tips]<br />
** on [https://play.google.com/store/apps/details?id=jp.co.rcsc.safetyTips.android Google Play Store]<br />
** or [https://apps.apple.com/us/app/safety-tips/id858357174 Apple App Store].<br />
* In case of earthquake:<br />
** Seek cover (e.g. under a table) if indoors,<br />
** Get away from structures (especially glass windows) if outdoors.<br />
** After the quake, get away from the shore onto high ground away from tidal waves.<br />
* In case of typhoon, follow instructions from authorities. Typhoon are predicted several days ahead.<br />
<br />
= Contact =<br />
<br />
The VideoLAN Dev Days are organized by the board members of the VideoLAN non-profit organization:<br />
* Jean-Baptiste Kempf,<br />
* Denis Charmet,<br />
* Konstantin Pavlov, and<br />
* Hugo Beauzée-Luyssen.<br />
You can reach us [mailto:board@videolan.org here].<br />
<br />
[[Category:About VideoLAN]]<br />
[[Category:VDD]]</div>Kieranhttps://wiki.videolan.org/index.php?title=VDD15&diff=30403VDD152015-09-16T19:54:22Z<p>Kieran: </p>
<hr />
<div>= Welcome to VideoLAN Dev Days 2015 =<br />
<br />
'''The Open Multimedia Conference that frees the cone in you!'''<br />
<br />
This is a technical un-conference; please bring laptops, power cords, power extensions, RJ45 cables and a '''good mood''' with you.<br />
<br />
= Schedule =<br />
== Thursday, September 17th 2015 ==<br />
<br />
Nothing is prepared for that day, but for people early at the hotel, we will get drinks at the hotel, at around 21:00.<br />
<br />
Find Felix :D<br />
<br />
== Friday, September 18th 2015 ==<br />
<br />
=== Community day ===<br />
If you subscribed for the community day, you should have received instructions by e-mail.<br />
<br />
Meeting point is '''at''' EuroDisney at '''9:45'''. The theme park opens at '''10:00''' and closes at '''21:00'''.<br />
<br />
==== How to get there ====<br />
The best option is to take the Metro/RER. This takes approximately 1 hour and 5 minutes:<br />
* take metro ''Line 13'' from the hotel to '''St Lazare''', then<br />
* take metro ''Line 3'' from '''St Lazare''' to '''Havre-Caumartin''' (1 stop only),<br />
* take the express train ''RER A'' to '''Marne La Vallée-Chessy'''<br />
** check on the platform display before boarding: not all trains go to Marne La Vallée-Chessy!<br />
<br />
'''N.B.:''' You should buy a ticket to Marne La Vallée at the metro station. The ticket is valid for the entire journey.<br />
<br />
=== Evening drinks ===<br />
We will meet at a local bar, for drinks (soft and beers), at around '''20:30'''.<br />
<br />
The name will be announced later, when confirmed, but should not be too far from the hotel.<br />
<br />
== Saturday, September 19th 2015 ==<br />
<br />
=== Morning ===<br />
<br />
{| class="wikitable" style="width: 100%;"<br />
|-<br />
!style="width: 10%;"| Time || Description<br />
|-<br />
| '''8:45 - 9:30''' || Registration and Breakfast<br />
|-<br />
| '''9:30 - 9:45''' || '''Opening remarks''' by Jean-Baptiste Kempf ''(VideoLAN)''<br />
|-<br />
| '''9:45 - 10:20''' || '''Daala''' by Timothy Terriberry ''(Mozilla)''<br />
|-<br />
| '''10:20 - 10:55''' || '''A new AAC free and open-source encoder''' by Rostislav Pehlivanov ''(FFmpeg)''<br />
|-<br />
| '''10:55 - 11:05''' || ''Coffee break''<br />
|-<br />
| '''11:05 - 11:40''' || '''New video compression techniques under consideration for VP10''' by Alex Converse ''(Google)''<br />
|-<br />
| '''11:40 - 12:10''' || '''An update to x265''' by Deepthi Nandakumar ''(MCW)''<br />
|-<br />
| '''12:10 - 12:40''' || '''VLC 3.0''' by Jean-Baptiste Kempf ''(VideoLAN)''<br />
|-<br />
| '''12:40 - 12:45''' || '''Spice''' by Lord Julien Navas ''(Vodkaster)''<br />
|}<br />
<br />
=== Afternoon ===<br />
<br />
{| class="wikitable" style="width: 100%;"<br />
|-<br />
!style="width: 10%;"| Time !! Description<br />
|-<br />
| '''14:00 - 14:40''' || '''The Thor Codec''' by Thomas Davies ''(Cisco)''<br />
|-<br />
| '''14:40 - 15:00''' || '''How to optimize press coverage for FLOSS projects''' by Sebastian Grüner ''(golem.de)''<br />
|-<br />
| '''15:00 - 15:20''' || '''VP9 encoding/decoding performance vs. H.264/HEVC''' by Ronald Bultje ''(FFmpeg)''<br />
|-<br />
| '''15:30 - 18:00''' || ''Break-out sessions''<br />
|}<br />
<br />
==== Break-out sessions, Saturday ====<br />
{| class="wikitable" style="width: 100%; text-align: center"<br />
|-<br />
! style="width: 10%;"| Time<br />
! style="width: 18%;"| Auditorium<br />
! style="width: 18%;"| Room 1<br />
! style="width: 18%;"| Room 2<br />
! style="width: 18%;"| Room 3<br />
! style="width: 18%;"| Room 4<br />
|-<br />
! 15:30 - 16:45<br />
| <br />
| '''VLC technical discussion'''<br />
|<br />
| <br />
|<br />
|-<br />
! 16:45 - 17:00<br />
| colspan="5" | ''Coffee Break''<br />
|-<br />
! 17:00 - 18:00<br />
| <br />
| VLC Mobile Ports discussion<br />
| <br />
| <br />
|<br />
|-<br />
! 18:00 - 19:30<br />
| <br />
| '''VideoLAN annual meeting''' Assemblée Générale Ordinaire, Assemblée Générale Extraordinaire && Réunion<br /><br />
This is for '''VideoLAN non-profit''' members.<br />
| <br />
| <br />
|<br />
|}<br />
<br />
=== Dinner ===<br />
Dinner is at '''20:00''' next to the place, at The Royal Trinité.<br />
<br />
59 Rue de Châteaudun, 75009 Paris, France<br />
<br />
[https://goo.gl/maps/30dCb Royal Trinité]<br />
<br />
== Sunday, September 20th 2015 ==<br />
<br />
{| class="wikitable" style="width: 100%; text-align: center"<br />
|-<br />
! style="width: 10%;"| Time<br />
! style="width: 18%;"| Auditorium<br />
! style="width: 18%;"| Room 1<br />
! style="width: 18%;"| Room 2<br />
! style="width: 18%;"| Room 3<br />
! style="width: 18%;"| Room 4<br />
|-<br />
! 9:30 - 10:00<br />
| colspan="5" | ''Breakfast''<br />
|-<br />
! 10:00 - 10:45<br />
| <br />
| VLC/iOS meetup<br />
|<br />
| <br />
|<br />
|-<br />
! 10:45 - 11:30<br />
| <br />
| VLC/Android meetup<br />
| <br />
| <br />
|<br />
|-<br />
! 11:30 - 11:45<br />
| colspan="5" | ''Coffee break''<br />
|-<br />
! 11:45 - 12:45<br />
| <br />
| <br />
| <br />
| <br />
|<br />
|-<br />
! 12:45 - 14:00<br />
| colspan="5" | ''Lunch break''<br />
|-<br />
! 14:00 - 14:45<br />
| <br />
|<br />
| <br />
| <br />
|<br />
|-<br />
! 14:45 - 15:30<br />
| <br />
|<br />
| <br />
| <br />
|<br />
|-<br />
! 15:30 - 15:45<br />
| '''Closing Remarks'''<br/> by Jean-Baptiste Kempf ''(VideoLAN)''<br />
| —<br />
| —<br />
| —<br />
| —<br />
|-<br />
! 15:45 - 17:00<br />
| <br />
|<br />
| <br />
| <br />
|<br />
|-<br />
! 17:00 - 18:00<br />
| <br />
|<br />
| <br />
| VideoLAN '''roots''' meeting <br />
|<br />
|}<br />
<br />
= Practical information =<br />
<br />
== Emergency situation ==<br />
In case of emergency, call the European emergency number '''112'''.<br />
<br />
== Location / main venue ==<br />
The venue is in Paris (France), 9th district, near St Lazare:<br />
<br />
Criteo<br/><br />
32 rue Blanche<br/><br />
75009 Paris, France<br />
<br />
[https://www.google.fr/maps/place/Criteo/@48.878914,2.3315591,17z/data=!4m6!1m3!3m2!1s0x47e66e04d509ec8f:0xbf35e94464e752f7!2sCriteo!3m1!1s0x47e66e04d509ec8f:0xbf35e94464e752f7 Google Maps]<br />
<br />
== Accomodation / Hotel ==<br />
For sponsored participants, the hotel is: <br />
<br />
Timhotel Berthier Paris 17<br/><br />
4 Boulevard Berthier<br/><br />
75017 Paris, France<br />
<br />
http://www.timhotel-berthier-paris-17.com/<br/><br />
[https://www.google.fi/maps/place/Timhotel+Berthier+Paris+17/@48.894398,2.312611,17z/data=!3m1!4b1!4m2!3m1!1s0x47e66f9878ba126d:0x24a180f7131eebe5 Google Maps]<br />
<br />
== Payment ==<br />
The local currency is the Euro. Merchants are forbidden from accepting other currencies (except exchange).<br />
<br />
Most venues will accept EMV credit and debit cards (i.e. Chip & PIN), at least VISA and MasterCard. Swipe-only cards can be more tricky (it has been illegal for banks to issue them for years in France, so only foreign visitors use them). American Express is widely but not universally accepted. Diners/Discovery, UnionPay, JCB, etc. are mostly only accepted in shops catering to tourists.<br />
<br />
== Transportation ==<br />
<br />
* Public transport: [http://www.ratp.fr/itineraires/en/ratp/recherche-avancee RATP]<br />
* Attendees arrival & departure list: [[VDD15/Attendees]]<br />
<br />
'''NOTE''': VideoLAN does not refund taxi expenses nor car rentals.<br />
<br />
=== Paris-CDG / Roissy airport ===<br />
<br />
* Upon landing, follow the signs for local trains. There is one station for T1/T3 and one for T2 (A-G).<br />
* At the station, look for '''blue/green''' ticket vending machines.<br />
** Do not use the yellow/grey machines (at terminal 2), those are for long distance trains only.<br />
* Buy a ticket for zones 1-5 (to Paris). The cost is 10€ one way. Credit cards are accepted.<br />
** If you only have cash, you have to queue at the staffed ticketing booth ("''Billeterie''" in French)<br />
* Take the regional express(RER) line B (blue). You can use any train toward Paris.<br />
* Alight at ''Gare du Nord''.<br />
* See below from Gare du Nord.<br />
<br />
* 2€ Alternative From/To ''Palais Royal /Louvre'': http://www.easybus.com/fr/paris-cdg (online booking), takes 50min<br />
<br />
=== Orly (ORY) airport ===<br />
* Upon landing, follow the signs for Orlyval.<br />
* At the station, buy the ticket from the vending machines, before boarding.<br />
* Buy a ticket for Paris. The cost is 12€ one way. Credit cards are accepted.<br />
** If you only have cash, you have to queue at the staffed ticketing booth ("''Billeterie''" in French)<br />
* Take the Orlyval to Antony. Then take the regional express(RER) line B (blue) to Paris (Direction Charles de Gaulle Airport).<br />
* Alight at ''Gare du Nord''.<br />
* See below from Gare du Nord.<br />
<br />
=== Beauvais (BVA) airport ===<br />
We feel sorry for you. Take the bus.<br />
<br />
=== Gare du Nord (North railway station) ===<br />
<br />
If you arrive at Gare du Nord, you need to buy a zone 1 / Paris ticket. It is valid on the subway, the tramway, busses and, '''within the Paris city limits''', on the regional express trains (RER).<br />
<br />
==== To the sponsored hotel ====<br />
* Transfer to subway(M) '''line 2''' toward ''Porte Dauphine''.<br />
** You can also take line 4 first and then transfer to line 2 (shorter walking distance)<br />
* Alight at '''Place de Clichy''' (6 stops)<br />
* Transfer to subway(M) '''line 13''' toward ''Asnières''.<br />
* Get off at '''Porte de Clichy''' (4 stops)<br />
<br />
The hotel is just around the corner.<br />
<br />
==== To the conference venue ====<br />
* Transfer to regional express (RER) line E (magenta) toward ''Haussman St Lazare''.<br />
* Get off at the final stop, ''Haussman St Lazare''.<br />
<br />
Walking route:<br />
[https://www.google.fr/maps/dir/32+Rue+Blanche,+75009+Paris,+France/Haussmann+-+Saint-Lazare,+France/@48.8767778,2.3281097,17z/data=!3m1!4b1!4m14!4m13!1m5!1m1!1s0x47e66e49a8a4ba55:0x95f959c0c70dfded!2m2!1d2.3316096!2d48.8788866!1m5!1m1!1s0x47e66e35df16346b:0x5b4b229d8b94e143!2m2!1d2.329037!2d48.875188!3e2 Google Maps]<br />
<br />
=== Gare de l'Est (East railway station) ===<br />
If you arrive at Gare de l'Est, you need to buy a zone 1 / Paris ticket. It is valid on the subway, the tramway, busses and, '''within the Paris city limits''', on the regional express trains (RER).<br />
<br />
==== To the sponsored hotel ====<br />
* Take subway(M) line 4 toward ''Porte de Clignancourt''.<br />
* Get off at ''Barbès-Rochechouart'' (3 stops).<br />
* Transfer to subway(M) line 2 toward ''Porte Dauphine''.<br />
* Alight at '''Place de Clichy''' (5 stops)<br />
* Transfer to subway(M) line 13 toward ''Asnières''...<br />
* Get off at '''Porte de Clichy''' (4 stops)<br />
<br />
The hotel is just around the corner.<br />
<br />
==== To the conference venue ====<br />
* Take subway(M) line 7 toward ''Villejuif''<br />
* Get off at ''Chaussée d'Antin (Lafayette)''<br />
<br />
=== Gare de Lyon (South/East railway station) ===<br />
<br />
=== Other? ===<br />
<br />
== From the hotel to the conference ==<br />
* Take subway(M) line 13 toward ''Chatillon-Montrouge''<br />
* Get off at ''Liège'' and walk<br />
<br />
Feel free to ask for advice on the mailing list if your station is missing.<br />
<br />
== Connectivity ==<br />
<br />
=== Power supply ===<br />
<br />
The AC power sockets are "French" '''IEC type E'''. Connectors from "German" '''IEC type F''' countries should fit safely, as they are usually bi-standards. There should have a hole for the French-style ground pin.<br />
<br />
The electrical system is '''220V / 50Hz'''. If you come from outside Europe, please make sure your devices are compatible beforehand.<br />
<br />
=== Internet ===<br />
Hopefully, there will be Wi-Fi.<br />
<br />
== Weather forecast ==<br />
<br />
:Fri Sep 18 Occasional rain 20°C Low 12°C <br />
:Sat Sep 19 Occasional rain 19°C Low 10°C <br />
:Sun Sep 20 Fog 20°C Low 12°C <br />
:Mon Sep 21 Mostly cloudy 18°C Low 12°C <br />
<br />
In Fahrenheit, this is a high of 64°F-66°F and a low of 50°F.<br />
<br />
[[Category:About VideoLAN]]</div>Kieranhttps://wiki.videolan.org/index.php?title=VDD15/Attendees&diff=30234VDD15/Attendees2015-09-05T10:49:59Z<p>Kieran: </p>
<hr />
<div>= Arrivals =<br />
Please order chronologically by time of arrival.<br />
<br />
{| border="1" class="sortable"<br />
! Name<br />
! IRC<br />
! Time<br />
! Port<br />
! Vessel<br />
! Notes<br />
|-<br />
| Arwa Arif<br />
| arwa<br />
| 2015-09-17 08:20<br />
| CDG airport<br />
| 9W 126<br />
| from BOM<br />
|-<br />
| Tim Miller<br />
| drasticactions<br />
| 2015-09-17 08:35<br />
| CDG airport<br />
| DL 243 (AF 3655)<br />
| from BOS<br />
|-<br />
| Rostislav Pehlivanov<br />
| atomnuker<br />
| 2015-09-17 10:17<br />
| Gare du Nord<br />
| TGV TMST<br />
| from London<br />
|-<br />
| Kieran Kunhya<br />
| kierank<br />
| 2015-09-17 16:47<br />
| Gare du Nord<br />
| Eurostar<br />
| from London<br />
|-<br />
| David Fuhrmann<br />
| d-fu<br />
| 2015-09-17 20:54<br />
| Gare de Est<br />
| TGV 9552<br />
| <br />
|-<br />
| Francois Cartegnie<br />
| <br />
| 2015-09-17 21:16<br />
| Gare de Lyon<br />
| TGV 6198<br />
| <br />
|-<br />
| Konstantin Pavlov<br />
| thresh<br />
| 2015-09-17 22:30<br />
| CDG airport T2C<br />
| SU 2460<br />
| from Moscow/SVO<br />
|-<br />
| Ronald S. Bultje<br />
| BBB<br />
| 2015-09-18 9:20<br />
| CDG airport<br />
| DL 400 (AF 3577)<br />
| from NY/JFK<br />
|-<br />
| Thierry Foucu<br />
| lotharkript<br />
| 2015-09-18 10:45<br />
| CDG airport<br />
| UA 990<br />
| from SFO<br />
|-<br />
| Federico Hlawaczek<br />
| <br />
| 2015-09-18 10:45<br />
| CDG airport<br />
| UA 990<br />
| from SFO<br />
|-<br />
| Rémi Denis-Courmont<br />
| courmisch<br />
| 2015-09-18 21:35<br />
| CDG airport T2D<br />
| AY 877<br />
| from Helsinki<br />
|-<br />
| Janne Grunau<br />
| jannau<br />
| 2015-09-18 22:35<br />
| Gare de l'Est<br />
| TGV 9570<br />
| from Karlsruhe<br />
|-<br />
|}<br />
<br />
= Departures =<br />
Please order chronologically by time of departure.<br />
<br />
{| border="1" class="sortable"<br />
! Name<br />
! IRC<br />
! Time<br />
! Port<br />
! Vessel<br />
! Notes<br />
|-<br />
| Mc Fly Marty<br />
| mcfly<br />
| 1985-10-26 01:22<br />
| Hill Valley, California<br />
| DeLorean OUTATIME<br />
| to Hill Valley<br />
|-<br />
| Tim Miller<br />
| drasticactions<br />
| 2015-09-20 16:45<br />
| CDG airport<br />
| AF 338<br />
| to BOS<br />
|-<br />
| Janne Grunau<br />
| jannau<br />
| 2015-09-20 17:25<br />
| Gare de l'Est<br />
| TGV 9577<br />
| to Karlsruhe<br />
|-<br />
| David Fuhrmann<br />
| d-fu<br />
| 2015-09-20 17:25<br />
| Match found<br />
| -- same train --<br />
|<br />
|-<br />
| Francois Cartegnie<br />
| <br />
| 2015-09-20 17:49<br />
| Gare de Lyon<br />
| TGV 6949<br />
| <br />
|-<br />
| Rémi Denis-Courmont<br />
| courmisch<br />
| 2015-09-20 19:00<br />
| CDG airport T2D<br />
| AY 874<br />
| to Helsinki<br />
|-<br />
| Federico Hlawaczek<br />
| <br />
| 2015-09-21 10:05<br />
| CDG airport<br />
| UA 991<br />
| to SFO<br />
|-<br />
| Rostislav Pehlivanov<br />
| atomnuker<br />
| 2015-09-21 10:13<br />
| Gare du Nord<br />
| TGV TMST<br />
| to London<br />
|-<br />
| Konstantin Pavlov<br />
| thresh<br />
| 2015-09-21 10:40<br />
| CDG airport T2E<br />
| AF 84<br />
| to SFO<br />
|-<br />
| Arwa Arif<br />
| arwa<br />
| 2015-09-21 10:50<br />
| CDG airport<br />
| EY 32<br />
| to HYD<br />
|-<br />
| Ben littler<br />
| <br />
| 2015-09-21 13:30<br />
| CDG airport<br />
| AF 378<br />
| to DTW<br />
|-<br />
| Ronald S. Bultje<br />
| BBB<br />
| 2015-09-21 14:00<br />
| CDG airport<br />
| AF 6<br />
| to NY/JFK<br />
|-<br />
| Thierry Foucu<br />
| lotharkript<br />
| 2015-09-22 10:05<br />
| CDG airport<br />
| UA 991<br />
| to SFO<br />
|-<br />
|}<br />
<br />
= Locals =<br />
People who do not need to travel:</div>Kieranhttps://wiki.videolan.org/index.php?title=VDD15/Attendees&diff=30233VDD15/Attendees2015-09-05T10:49:13Z<p>Kieran: </p>
<hr />
<div>= Arrivals =<br />
Please order chronologically by time of arrival.<br />
<br />
{| border="1" class="sortable"<br />
! Name<br />
! IRC<br />
! Time<br />
! Port<br />
! Vessel<br />
! Notes<br />
|-<br />
| Arwa Arif<br />
| arwa<br />
| 2015-09-17 08:20<br />
| CDG airport<br />
| 9W 126<br />
| from BOM<br />
|-<br />
| Tim Miller<br />
| drasticactions<br />
| 2015-09-17 08:35<br />
| CDG airport<br />
| DL 243 (AF 3655)<br />
| from BOS<br />
|-<br />
| Rostislav Pehlivanov<br />
| atomnuker<br />
| 2015-09-17 10:17<br />
| Gare du Nord<br />
| TGV TMST<br />
| from London<br />
|-<br />
| Kieran Kunhya<br />
| kierank<br />
| 2015-09-17 13:31<br />
| Gare du Nord<br />
| Eurostar<br />
| from London<br />
|-<br />
| David Fuhrmann<br />
| d-fu<br />
| 2015-09-17 20:54<br />
| Gare de Est<br />
| TGV 9552<br />
| <br />
|-<br />
| Francois Cartegnie<br />
| <br />
| 2015-09-17 21:16<br />
| Gare de Lyon<br />
| TGV 6198<br />
| <br />
|-<br />
| Konstantin Pavlov<br />
| thresh<br />
| 2015-09-17 22:30<br />
| CDG airport T2C<br />
| SU 2460<br />
| from Moscow/SVO<br />
|-<br />
| Ronald S. Bultje<br />
| BBB<br />
| 2015-09-18 9:20<br />
| CDG airport<br />
| DL 400 (AF 3577)<br />
| from NY/JFK<br />
|-<br />
| Thierry Foucu<br />
| lotharkript<br />
| 2015-09-18 10:45<br />
| CDG airport<br />
| UA 990<br />
| from SFO<br />
|-<br />
| Federico Hlawaczek<br />
| <br />
| 2015-09-18 10:45<br />
| CDG airport<br />
| UA 990<br />
| from SFO<br />
|-<br />
| Rémi Denis-Courmont<br />
| courmisch<br />
| 2015-09-18 21:35<br />
| CDG airport T2D<br />
| AY 877<br />
| from Helsinki<br />
|-<br />
| Janne Grunau<br />
| jannau<br />
| 2015-09-18 22:35<br />
| Gare de l'Est<br />
| TGV 9570<br />
| from Karlsruhe<br />
|-<br />
|}<br />
<br />
= Departures =<br />
Please order chronologically by time of departure.<br />
<br />
{| border="1" class="sortable"<br />
! Name<br />
! IRC<br />
! Time<br />
! Port<br />
! Vessel<br />
! Notes<br />
|-<br />
| Mc Fly Marty<br />
| mcfly<br />
| 1985-10-26 01:22<br />
| Hill Valley, California<br />
| DeLorean OUTATIME<br />
| to Hill Valley<br />
|-<br />
| Tim Miller<br />
| drasticactions<br />
| 2015-09-20 16:45<br />
| CDG airport<br />
| AF 338<br />
| to BOS<br />
|-<br />
| Janne Grunau<br />
| jannau<br />
| 2015-09-20 17:25<br />
| Gare de l'Est<br />
| TGV 9577<br />
| to Karlsruhe<br />
|-<br />
| David Fuhrmann<br />
| d-fu<br />
| 2015-09-20 17:25<br />
| Match found<br />
| -- same train --<br />
|<br />
|-<br />
| Francois Cartegnie<br />
| <br />
| 2015-09-20 17:49<br />
| Gare de Lyon<br />
| TGV 6949<br />
| <br />
|-<br />
| Rémi Denis-Courmont<br />
| courmisch<br />
| 2015-09-20 19:00<br />
| CDG airport T2D<br />
| AY 874<br />
| to Helsinki<br />
|-<br />
| Federico Hlawaczek<br />
| <br />
| 2015-09-21 10:05<br />
| CDG airport<br />
| UA 991<br />
| to SFO<br />
|-<br />
| Rostislav Pehlivanov<br />
| atomnuker<br />
| 2015-09-21 10:13<br />
| Gare du Nord<br />
| TGV TMST<br />
| to London<br />
|-<br />
| Konstantin Pavlov<br />
| thresh<br />
| 2015-09-21 10:40<br />
| CDG airport T2E<br />
| AF 84<br />
| to SFO<br />
|-<br />
| Arwa Arif<br />
| arwa<br />
| 2015-09-21 10:50<br />
| CDG airport<br />
| EY 32<br />
| to HYD<br />
|-<br />
| Ben littler<br />
| <br />
| 2015-09-21 13:30<br />
| CDG airport<br />
| AF 378<br />
| to DTW<br />
|-<br />
| Ronald S. Bultje<br />
| BBB<br />
| 2015-09-21 14:00<br />
| CDG airport<br />
| AF 6<br />
| to NY/JFK<br />
|-<br />
| Thierry Foucu<br />
| lotharkript<br />
| 2015-09-22 10:05<br />
| CDG airport<br />
| UA 991<br />
| to SFO<br />
|-<br />
|}<br />
<br />
= Locals =<br />
People who do not need to travel:</div>Kieranhttps://wiki.videolan.org/index.php?title=VDD14/Sessions&diff=27137VDD14/Sessions2014-09-20T11:16:33Z<p>Kieran: /* Proposed VDD sessions */</p>
<hr />
<div>__NOTOC__<br />
<noinclude><br />
'''<big>[[VDD14|← Back to the VDD14 page]]</big>'''<br />
</noinclude><br />
<br />
=== Rooms ===<br />
<br />
==== 11th floor ====<br />
* Ocean's eleven, large room with talks, for ~80 people<br />
* Kitchen11, coffee, for ~30 people<br />
<br />
==== 12th floor ====<br />
* PepperClip, small meeting rooms for ~10 people<br />
* FiloFax, small meeting rooms for ~10 people<br />
* Kitchen12, coffee and cool layout for ~20 people<br />
<br />
==== 13rd floor ====<br />
<br />
* Kitchen13, coffee and cool layout for ~20 people<br />
* Galileo, meeting room, for ~20 people<br />
* Docks view, for ~8people<br />
<br />
=== Etherpad to take Notes ===<br />
https://beta.etherpad.org/<br />
<br />
=== Scheduled VDD sessions ===<br />
<br />
==== Saturday ====<br />
{| class="wikitable"<br />
|-<br />
! Time<br />
! Oceans' eleven<br />
! Galileo<br />
! Kitchen11<br />
! Kitchen12<br />
! Kitchen13<br />
|-<br />
! 14:00 - 16:00<br />
| VLC technical discussion<br />
| libav technical discussions <br />
| FFmpeg technical discussion <br />
| <br />
|<br />
|-<br />
! 16:00 - 16:15<br />
| colspan="5" | Coffee Break<br />
|-<br />
! 16:15 - 18:15<br />
| colspan="4" | T.B.D.<br />
|<br />
|-<br />
! 18:30 - 19:00<br />
| Wrapping up<br />
|}<br />
<br />
==== Sunday ====<br />
{| class="wikitable"<br />
|-<br />
! Time<br />
! Room 1<br />
! Room 2<br />
! Room 3<br />
! Room 4<br />
! Room 5<br />
|-<br />
| 14:00 - 16:00<br />
| colspan="4" | T.B.D.<br />
| Hacking<br />
|-<br />
| 16:00 - 16:15<br />
| colspan="5" | Break<br />
|-<br />
| 16:15 - 18:15<br />
| colspan="4" | T.B.D.<br />
| Hacking<br />
|-<br />
| 18:30 - 19:00<br />
| colspan="5" | Wrapping up<br />
|}<br />
<br />
=== Proposed VDD sessions ===<br />
<br />
==== Road towards HWAccel 2 ====<br />
During this year we moved two steps towards version [[https://wiki.libav.org/Blueprint/HWAccel2 2]]:<br />
<br />
* 1.2 -> default context allocators<br />
* 1.3 -> bitstream-oriented hwaccel support<br />
<br />
During this session we will discuss about how to provide an<br />
uniform structure to avoid even more boilerplate code and still<br />
allow full control to the people needing it.<br />
<br />
==== ChromeCast ====<br />
How do we achieve VLC using Chromecast?<br />
<br />
==== Media Library ====<br />
Current state and conception of needed functionality for a cross-platform library to collect meta data, thumbnails and so on for use within media handling applications based on libvlc. Written in C++.<br />
<br />
==== VLC Mobile Remotes ====<br />
There is a major feature missing in VLC's mobile apps, which is currently provided through third party apps. Control VLC remotely running on a desktop. What needs to be done both on client and server-side to provide an advanced and seamless integration between the platforms?<br />
<br />
==== VLC Subpicture Framework improvements for hwaccel rendering ====<br />
As of today for picture type subpictures (DVB Subtitles, VBI pages, ...) a software scale step is forced into the pipeline. To make better use of hardware being able to render scaled overlays this shall be omitted. But it seems some significant changes in the VLC core are required to support this in a common way, which some brainstorming on would be required.<br />
<br />
==== EBU-TT-D integration in VLC and/or ffmpeg ====<br />
<br />
==== Android ====<br />
<br />
==== iOS ====<br />
<br />
==== WinRT ====<br />
<br />
==== x264 meeting ====<br />
<br />
==== Opus in MP4 ====<br />
We should sit down and design this mapping.<br />
<br />
==== Binary Album Art extensions for Ogg ====<br />
We've had a proposal to include binary (not base64-encoded) album art for Ogg in the context of Ogg Opus: <http://www.ietf.org/mail-archive/web/codec/current/msg03061.html>.<br />
This session would determine if there is interest from players in supporting such an extension and working on the design.<br />
<br />
==== libav/FFmpeg discussion ====<br />
<br />
==== Testing ====<br />
Seems like everybody is doing the same very flaky or very manual thing when testing video processing and player software. Can we do better?<br />
<br />
==== Broadcasting things ====<br />
An attempt to justify what broadcasters are doing.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=26286X264 TODO2013-12-20T23:53:31Z<p>Kieran: </p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264. <br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk]. <br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code. <br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC. <br />
<br />
=== Motion Estimation ===<br />
<br />
*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore. <br />
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed. <br />
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
**I have a patch for this in the lookahead, but it didn't help much, since it only added predictors.<br />
*Somehow take into account the effect of motion vector decision on future blocks. <br />
**Hierarchical motion estimation <br />
**Approximations from lookahead MVs <br />
**Iterative ME (as per Snow) <br />
**Trellis motion estimation <br />
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search. <br />
**But how do we know which ones we can afford to skip, and when? <br />
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either. <br />
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs? <br />
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search? <br />
**This seems to be awful from my testing, but maybe there's something we can do?<br />
*Try sub-8x8 partitions in B-frames. Is it at all useful? <br />
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help? <br />
*Fullpel chroma ME? <br />
**For TESA?<br />
<br />
=== Intra Analysis ===<br />
<br />
*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
**With the SSSE3-based fast intra analysis, we no longer do any early terminations for different modes, at least in SAD/SATD analysis. But there might still be improvements to be made.<br />
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
=== Mode Decision ===<br />
<br />
*Can we find more ways to skip more motion searches in multiref?<br />
**A while back, I tried using weaker motion searches on older refs. This helped a bit for speed-vs-compression, but is ironically the opposite of what one wants; older refs will be harder to find good MVs in, and therefore really need better searches.<br />
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
**Can we do something smart by analyzing fenc? It's impossible to tell whether a block is motionless by looking at fdec, but looking at the source pixels is useful. There's still complexity such as lower-QP-than-reference though.<br />
*See the TODOs for deblock-aware RD in common/deblock.c.<br />
**I tried correcting weightp references for deblock RDO, but it didn't help.<br />
**I tried chroma, too, and again, it didn't help measurably.<br />
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
**Doing a merged 4x4/8x8 SATD would help here, but would require new asm.<br />
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle. <br />
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality). <br />
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
*How about saving CABAC state between each trellis call, rather than basing them all on the CABAC state at the start of the macroblock?<br />
*Make subme=11 not do thresholding in qpel RD and bidir RD.<br />
<br />
=== Psy ===<br />
<br />
*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better? <br />
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see. <br />
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs. <br />
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed. <br />
*RD should take into account local variance. <br />
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis. <br />
*Lambda should be picked independent of quantizer (i.e. with greater precision). <br />
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text. <br />
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou). <br />
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"... <br />
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
=== Lookahead ===<br />
<br />
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search. <br />
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
*B-adapt 1 could be made quite a bit better -- it's important because it's used on all the fast speed modes (and even the defaults). "Harbour 4CIF" is a good example of a clip where it does noticeably badly.<br />
<br />
=== Quantization ===<br />
<br />
*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
**This is doubly important now, as CABAC trellis has been made way faster, but CAVLC hasn't. Many of the CABAC trellis improvements can be backported.<br />
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
**How useful is this with an entropy coder that doesn't really bias towards zero-runs, as in CABAC?<br />
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high? <br />
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former. <br />
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.<br />
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
=== Transform ===<br />
<br />
*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help. <br />
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
=== Interlacing ===<br />
<br />
*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower? <br />
*Constrained intra + adaptive MBAFF. Does anyone care about this? <br />
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
=== Weighted Prediction ===<br />
<br />
*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.''' <br />
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one. <br />
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases. <br />
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.<br />
<br />
=== Ratecontrol ===<br />
<br />
*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate. <br />
*Make the frame size and row size predictors better. They currently are kind of crappy. <br />
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
*1-pass ratecontrol often can't adapt fast enough when there are lots of threads (12, 16, 24, etc), especially with smallish VBV buffers. Improve this?<br />
*2-pass VBV is actually a bit more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
**2-pass is still better in the case of many threads, due to the above.<br />
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
=== GPU ===<br />
<br />
*Motion estimation? <br />
**Methods <br />
***Hierarchical? <br />
***2D Wave? <br />
***Something else? <br />
**"Easy": lookahead motion estimation <br />
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once. <br />
**"Hard": main motion estimation <br />
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search. <br />
***But potentially more useful... <br />
*Other things?<br />
*MultiCoreWare is currently looking on a GPU-based lookahead.<br />
<br />
=== x86 assembly ===<br />
<br />
*Find holger, lock him in a closet, and make him commit his local patches. It's been years now!<br />
<br />
=== Other assembly ===<br />
<br />
*There's still a lot of unreviewed NEON assembly. Can we please get this in?<br />
*Altivec assembly is very lacking. <br />
*SPARC VIS assembly is only available when high bit-depth is disabled. But nobody cares about SPARC anyways.<br />
<br />
=== Other CPU optimizations ===<br />
<br />
*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark. <br />
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
=== Other features ===<br />
<br />
*MPEG-2 encoding support<br />
**[https://github.com/kierank/x262/wiki/TODO x262]<br />
*VP8 encoding support<br />
**[https://github.com/DarkShikari/xvp8 xvp8]<br />
*Support for SMPTE timecodes <br />
*Merge speedcontrol <br />
*Mixed lossless/lossy encoding.<br />
*Segment re-encoding<br />
<br />
=== x264CLI ===<br />
<br />
*Finish audio support. Talk to Kovensky about this one.<br />
*Make the filtering system aware of BT.601 vs BT.709.<br />
*Add more filters. <br />
**Deinterlacers (YADIF). <br />
**Denoisers (HQDN3D?). <br />
**IVTC, decomb? <br />
*Merge L-SMASH mp4 muxer. <br />
*Add --device support. <br />
*Add automatic --level restriction support.<br />
<br />
<br />
[[Category:x264]]</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25426SDI API2013-10-01T08:29:13Z<p>Kieran: </p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side the issues can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll response (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 8 or 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame for things like Closed captions<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== SDI extension to V4L2 thoughts ==<br />
Single SDI file descriptor. Open, setup options (bit-depth etc), poll on descriptor with buffers to read (or mmap if the data is large enough?).<br />
<br />
Receive minimum of one line of data starting at EAV to subsequent EAV. Either in v210 for 10-bit or UYVY for 8-bit. Userspace has the responsibility of parsing the datastream (easily doable with a libsdi component).</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25425SDI API2013-10-01T07:10:13Z<p>Kieran: /* SDI extension to V4L2 thoughts */</p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll response (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 8 or 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame for things like Closed captions<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== SDI extension to V4L2 thoughts ==<br />
Single SDI file descriptor. Open, setup options (bit-depth etc), poll on descriptor with buffers to read (or mmap if the data is large enough?).<br />
<br />
Receive minimum of one line of data starting at EAV to subsequent EAV. Either in v210 for 10-bit or UYVY for 8-bit. Userspace has the responsibility of parsing the datastream (easily doable with a libsdi component).</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25289SDI API2013-09-26T19:28:15Z<p>Kieran: /* SDI extension to V4L2 */</p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll response (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 8 or 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame for things like Closed captions<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== SDI extension to V4L2 thoughts ==<br />
Single SDI file descriptor. Open, setup options (bit-depth etc), poll on descriptor with buffers to read (or mmap if the data is large enough?).</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25288SDI API2013-09-26T19:27:44Z<p>Kieran: </p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll response (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 8 or 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame for things like Closed captions<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== SDI extension to V4L2 ==<br />
Single SDI file descriptor. Open, setup options (bit-depth etc), poll on descriptor with buffers to read (or mmap if the data is large enough?).</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25287SDI API2013-09-26T19:16:39Z<p>Kieran: </p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll response (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame for things like Closed captions<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== API thoughts ==<br />
Single file descriptor. Open, setup options (pixel format etc), poll on descriptor with buffers to read (or mmap if the data is large enough?).</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25286SDI API2013-09-26T19:13:38Z<p>Kieran: </p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll response (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame for things like Closed captions<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== API thoughts ==<br />
Single file descriptor. Open, setup options (pixel format etc), poll on descriptor with queued and dqueued mmapped buffers.</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25284SDI API2013-09-26T19:04:08Z<p>Kieran: /* Thoughts about SDI API */</p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll response (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== API thoughts ==<br />
Single file descriptor. Open, setup options (pixel format etc), poll on descriptor with queued and dqueued mmapped buffers.</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25283SDI API2013-09-26T19:01:14Z<p>Kieran: </p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.<br />
<br />
== API thoughts ==<br />
Single file descriptor. Open, setup options (pixel format etc), poll on descriptor with queued and dqueued mmapped buffers.</div>Kieranhttps://wiki.videolan.org/index.php?title=SDI_API&diff=25282SDI API2013-09-26T18:59:26Z<p>Kieran: Created page with "== Thoughts about SDI API == There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issu..."</p>
<hr />
<div>== Thoughts about SDI API ==<br />
<br />
There are a number of manufacturers using V4L2, sometimes alongside ALSA, to provide Linux drivers for their SDI cards. This causes a lot of issues for professional users. There are a number of proprietary APIs that provide a better interface for professional users. On the capture side they can be described as follows:<br />
<br />
* Separate file descriptors for Audio/Video meaning sync must be performed with timestamps (if they exist; sometimes they don't). This is the most serious issue in a professional environment.<br />
* Timestamps are sometimes generated with the system clock and do not reflect the way audio is transported in SDI, nor precise nature of NTSC timestamps.<br />
* Impossible to select number of lines to capture per poll (e.g x lines, one field, one frame)<br />
* Impossible to keep NTSC audio samples per video frame cadence, nor line number where audio starts<br />
* 10-bit capture (though this is easy to solve in V4L2)<br />
* Signals when frames are dropped (so alternative signals can be sent)<br />
* An API for accessing VBI and HBI in 10-bit alongside the frame<br />
<br />
Most of these issues are caused by the separation of Audio/Video/Blanking so the two solutions in that area could be accurate timestamps (using a 27mhz clock) or data returned multiplexed. Another way of solving the issue would be to have an SDI "pixel format" and userspace could parse the datastream itself as VLC does with the Computermodules SD-SDI driver.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=22842X264 TODO2012-07-21T22:59:38Z<p>Kieran: /* Lookahead */</p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264. <br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk]. <br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code. <br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC. <br />
<br />
=== Motion Estimation ===<br />
<br />
*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore. <br />
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed. <br />
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
**I have a patch for this in the lookahead, but it didn't help much, since it only added predictors.<br />
*Somehow take into account the effect of motion vector decision on future blocks. <br />
**Hierarchical motion estimation <br />
**Approximations from lookahead MVs <br />
**Iterative ME (as per Snow) <br />
**Trellis motion estimation <br />
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search. <br />
**But how do we know which ones we can afford to skip, and when? <br />
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either. <br />
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs? <br />
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search? <br />
**This seems to be awful from my testing, but maybe there's something we can do?<br />
*Try sub-8x8 partitions in B-frames. Is it at all useful? <br />
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help? <br />
*Fullpel chroma ME? <br />
**For TESA?<br />
<br />
=== Intra Analysis ===<br />
<br />
*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
**With the SSSE3-based fast intra analysis, we no longer do any early terminations for different modes, at least in SAD/SATD analysis. But there might still be improvements to be made.<br />
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
=== Mode Decision ===<br />
<br />
*Can we find more ways to skip more motion searches in multiref?<br />
**A while back, I tried using weaker motion searches on older refs. This helped a bit for speed-vs-compression, but is ironically the opposite of what one wants; older refs will be harder to find good MVs in, and therefore really need better searches.<br />
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
**Can we do something smart by analyzing fenc? It's impossible to tell whether a block is motionless by looking at fdec, but looking at the source pixels is useful. There's still complexity such as lower-QP-than-reference though.<br />
*See the TODOs for deblock-aware RD in common/deblock.c.<br />
**I tried correcting weightp references for deblock RDO, but it didn't help.<br />
**I tried chroma, too, and again, it didn't help measurably.<br />
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
**Doing a merged 4x4/8x8 SATD would help here, but would require new asm.<br />
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle. <br />
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality). <br />
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
*How about saving CABAC state between each trellis call, rather than basing them all on the CABAC state at the start of the macroblock?<br />
*Make subme=11 not do thresholding in qpel RD and bidir RD.<br />
<br />
=== Psy ===<br />
<br />
*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better? <br />
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see. <br />
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs. <br />
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed. <br />
*RD should take into account local variance. <br />
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis. <br />
*Lambda should be picked independent of quantizer (i.e. with greater precision). <br />
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text. <br />
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou). <br />
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"... <br />
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
=== Lookahead ===<br />
<br />
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search. <br />
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
*B-adapt 1 could be made quite a bit better -- it's important because it's used on all the fast speed modes (and even the defaults). "Harbour 4CIF" is a good example of a clip where it does noticeably badly.<br />
<br />
=== Quantization ===<br />
<br />
*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
**This is doubly important now, as CABAC trellis has been made way faster, but CAVLC hasn't. Many of the CABAC trellis improvements can be backported.<br />
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
**How useful is this with an entropy coder that doesn't really bias towards zero-runs, as in CABAC?<br />
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high? <br />
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former. <br />
**See [https://github.com/saintdev/x264-devel/compare/enquant-base...energy-quant saintdev's github] for one attempt at this.<br />
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
=== Transform ===<br />
<br />
*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help. <br />
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
=== Interlacing ===<br />
<br />
*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower? <br />
*Constrained intra + adaptive MBAFF. Does anyone care about this? <br />
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
=== Weighted Prediction ===<br />
<br />
*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.''' <br />
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one. <br />
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases. <br />
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.<br />
<br />
=== Ratecontrol ===<br />
<br />
*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate. <br />
*Make the frame size and row size predictors better. They currently are kind of crappy. <br />
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
*1-pass ratecontrol often can't adapt fast enough when there are lots of threads (12, 16, 24, etc), especially with smallish VBV buffers. Improve this?<br />
*2-pass VBV is actually a bit more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
**2-pass is still better in the case of many threads, due to the above.<br />
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
=== GPU ===<br />
<br />
*Motion estimation? <br />
**Methods <br />
***Hierarchical? <br />
***2D Wave? <br />
***Something else? <br />
**"Easy": lookahead motion estimation <br />
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once. <br />
**"Hard": main motion estimation <br />
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search. <br />
***But potentially more useful... <br />
*Other things?<br />
*MultiCoreWare is currently looking on a GPU-based lookahead.<br />
<br />
=== x86 assembly ===<br />
<br />
*Find holger, lock him in a closet, and make him commit his local patches. It's been years now!<br />
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
=== Other assembly ===<br />
<br />
*There's still a lot of unreviewed NEON assembly. Can we please get this in?<br />
*Altivec assembly is very lacking. <br />
*SPARC VIS assembly is only available when high bit-depth is disabled. But nobody cares about SPARC anyways.<br />
<br />
=== Other CPU optimizations ===<br />
<br />
*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark. <br />
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
=== Other features ===<br />
<br />
*MPEG-2 encoding support<br />
**[https://github.com/kierank/x262/wiki/TODO x262]<br />
*VP8 encoding support<br />
**[https://github.com/DarkShikari/xvp8 xvp8]<br />
*Support for SMPTE timecodes <br />
*Merge speedcontrol <br />
*Mixed lossless/lossy encoding.<br />
*Segment re-encoding<br />
*Spec compliant AVC-Intra encoding (ask kierank for spec)<br />
**I tried this, but the problem is that the encoders/decoders in reality don't actually abide by their own spec. This is completely insane.<br />
<br />
=== x264CLI ===<br />
<br />
*Finish audio support. Talk to Kovensky about this one.<br />
*Make the filtering system aware of BT.601 vs BT.709.<br />
*Add more filters. <br />
**Deinterlacers (YADIF). <br />
**Denoisers (HQDN3D?). <br />
**IVTC, decomb? <br />
*Merge L-SMASH mp4 muxer. <br />
*Add TS muxing support using HRD. Talk to kierank about this one. <br />
*Add --device support. <br />
*Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=19805X264 TODO2011-10-31T17:33:11Z<p>Kieran: /* Other features */</p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264. <br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk]. <br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code. <br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC. <br />
<br />
=== Motion Estimation ===<br />
<br />
*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore. <br />
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed. <br />
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion. <br />
*Somehow take into account the effect of motion vector decision on future blocks. <br />
**Hierarchical motion estimation <br />
**Approximations from lookahead MVs <br />
**Iterative ME (as per Snow) <br />
**Trellis motion estimation <br />
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search. <br />
**But how do we know which ones we can afford to skip, and when? <br />
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either. <br />
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs? <br />
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search? <br />
*Try sub-8x8 partitions in B-frames. Is it at all useful? <br />
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help? <br />
*Fullpel chroma ME? <br />
**For TESA?<br />
<br />
=== Intra Analysis ===<br />
<br />
*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful. <br />
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
=== Mode Decision ===<br />
<br />
*Can we find more ways to skip more motion searches in multiref? <br />
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here? <br />
*See the TODOs for deblock-aware RD in common/deblock.c. <br />
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial. <br />
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle. <br />
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality). <br />
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation? <br />
*How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
=== Psy ===<br />
<br />
*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better? <br />
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see. <br />
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs. <br />
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed. <br />
*RD should take into account local variance. <br />
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis. <br />
*Lambda should be picked independent of quantizer (i.e. with greater precision). <br />
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text. <br />
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou). <br />
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"... <br />
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
=== Lookahead ===<br />
<br />
*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.''' <br />
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search. <br />
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
=== Quantization ===<br />
<br />
*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c. <br />
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful. <br />
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high? <br />
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former. <br />
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
=== Transform ===<br />
<br />
*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help. <br />
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
=== Interlacing ===<br />
<br />
*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower? <br />
*Constrained intra + adaptive MBAFF. Does anyone care about this? <br />
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
=== Weighted Prediction ===<br />
<br />
*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.''' <br />
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one. <br />
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases. <br />
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.<br />
<br />
=== Ratecontrol ===<br />
<br />
*VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy. <br />
**Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing. <br />
*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate. <br />
*Make the frame size and row size predictors better. They currently are kind of crappy. <br />
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur. <br />
*2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved. <br />
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving). <br />
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
=== GPU ===<br />
<br />
*Motion estimation? <br />
**Methods <br />
***Hierarchical? <br />
***2D Wave? <br />
***Something else? <br />
**"Easy": lookahead motion estimation <br />
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once. <br />
**"Hard": main motion estimation <br />
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search. <br />
***But potentially more useful... <br />
*Other things?<br />
<br />
=== x86 assembly ===<br />
<br />
*Optimize more for the Phenom. <br />
*Yell at holger to commit his local patches. <br />
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
=== Other assembly ===<br />
<br />
*NEON assembly is nowhere near complete. <br />
**Chroma MC needs to be rewritten for NV12 support. <br />
*Altivec assembly is very lacking. <br />
*SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
=== Other CPU optimizations ===<br />
<br />
*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark. <br />
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
=== Other features ===<br />
<br />
*MPEG-2 encoding support <br />
*VP8 encoding support <br />
*Support for SMPTE timecodes <br />
*Merge speedcontrol <br />
*Mixed lossless/lossy encoding.<br />
*Segment re-encoding<br />
*Spec compliant AVC-Intra encoding (ask kierank for spec)<br />
<br />
=== x264CLI ===<br />
<br />
*Finish audio support. Talk to Kovensky about this one. <br />
*Make the filtering system aware of fullrange vs TV range. <br />
*Make the filtering system aware of BT.601 vs BT.709. <br />
*Add more filters. <br />
**Deinterlacers (YADIF). <br />
**Denoisers (HQDN3D?). <br />
**IVTC, decomb? <br />
*Merge L-SMASH mp4 muxer. <br />
*Add TS muxing support using HRD. Talk to kierank about this one. <br />
*Add --device support. <br />
*Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=19689X264 TODO2011-09-23T15:02:58Z<p>Kieran: /* Other features */</p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264. <br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk]. <br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code. <br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC. <br />
<br />
=== Motion Estimation ===<br />
<br />
*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore. <br />
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed. <br />
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion. <br />
*Somehow take into account the effect of motion vector decision on future blocks. <br />
**Hierarchical motion estimation <br />
**Approximations from lookahead MVs <br />
**Iterative ME (as per Snow) <br />
**Trellis motion estimation <br />
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search. <br />
**But how do we know which ones we can afford to skip, and when? <br />
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either. <br />
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs? <br />
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search? <br />
*Try sub-8x8 partitions in B-frames. Is it at all useful? <br />
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help? <br />
*Fullpel chroma ME? <br />
**For TESA?<br />
<br />
=== Intra Analysis ===<br />
<br />
*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful. <br />
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
=== Mode Decision ===<br />
<br />
*Can we find more ways to skip more motion searches in multiref? <br />
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here? <br />
*See the TODOs for deblock-aware RD in common/deblock.c. <br />
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial. <br />
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle. <br />
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality). <br />
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation? <br />
*How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
=== Psy ===<br />
<br />
*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better? <br />
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see. <br />
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs. <br />
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed. <br />
*RD should take into account local variance. <br />
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis. <br />
*Lambda should be picked independent of quantizer (i.e. with greater precision). <br />
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text. <br />
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou). <br />
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"... <br />
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
=== Lookahead ===<br />
<br />
*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.''' <br />
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search. <br />
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
=== Quantization ===<br />
<br />
*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c. <br />
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful. <br />
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high? <br />
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former. <br />
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
=== Transform ===<br />
<br />
*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help. <br />
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
=== Interlacing ===<br />
<br />
*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower? <br />
*Constrained intra + adaptive MBAFF. Does anyone care about this? <br />
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
=== Weighted Prediction ===<br />
<br />
*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.''' <br />
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one. <br />
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases. <br />
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.<br />
<br />
=== Ratecontrol ===<br />
<br />
*VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy. <br />
**Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing. <br />
*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate. <br />
*Make the frame size and row size predictors better. They currently are kind of crappy. <br />
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur. <br />
*2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved. <br />
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving). <br />
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
=== GPU ===<br />
<br />
*Motion estimation? <br />
**Methods <br />
***Hierarchical? <br />
***2D Wave? <br />
***Something else? <br />
**"Easy": lookahead motion estimation <br />
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once. <br />
**"Hard": main motion estimation <br />
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search. <br />
***But potentially more useful... <br />
*Other things?<br />
<br />
=== x86 assembly ===<br />
<br />
*Optimize more for the Phenom. <br />
*Yell at holger to commit his local patches. <br />
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
=== Other assembly ===<br />
<br />
*NEON assembly is nowhere near complete. <br />
**Chroma MC needs to be rewritten for NV12 support. <br />
*Altivec assembly is very lacking. <br />
*SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
=== Other CPU optimizations ===<br />
<br />
*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark. <br />
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
=== Other features ===<br />
<br />
*MPEG-2 encoding support <br />
*VP8 encoding support <br />
*Support for SMPTE timecodes <br />
*Merge speedcontrol <br />
*Mixed lossless/lossy encoding.<br />
*Segment re-encoding<br />
<br />
=== x264CLI ===<br />
<br />
*Finish audio support. Talk to Kovensky about this one. <br />
*Make the filtering system aware of fullrange vs TV range. <br />
*Make the filtering system aware of BT.601 vs BT.709. <br />
*Add more filters. <br />
**Deinterlacers (YADIF). <br />
**Denoisers (HQDN3D?). <br />
**IVTC, decomb? <br />
*Merge L-SMASH mp4 muxer. <br />
*Add TS muxing support using HRD. Talk to kierank about this one. <br />
*Add --device support. <br />
*Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=19542X264 TODO2011-08-22T13:41:36Z<p>Kieran: /* Other features */</p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264. <br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk]. <br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code. <br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC. <br />
<br />
=== Motion Estimation ===<br />
<br />
*Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore. <br />
*(T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed. <br />
*Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion. <br />
*Somehow take into account the effect of motion vector decision on future blocks. <br />
**Hierarchical motion estimation <br />
**Approximations from lookahead MVs <br />
**Iterative ME (as per Snow) <br />
**Trellis motion estimation <br />
*We don't need to check all 11 predictors all the time for 16x16 fullpel motion search. <br />
**But how do we know which ones we can afford to skip, and when? <br />
**Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either. <br />
*libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs? <br />
*With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search? <br />
*Try sub-8x8 partitions in B-frames. Is it at all useful? <br />
*Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help? <br />
*Fullpel chroma ME? <br />
**For TESA?<br />
<br />
=== Intra Analysis ===<br />
<br />
*Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful. <br />
*SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
=== Mode Decision ===<br />
<br />
*Can we find more ways to skip more motion searches in multiref? <br />
*On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here? <br />
*See the TODOs for deblock-aware RD in common/deblock.c. <br />
*Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial. <br />
*Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle. <br />
*Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality). <br />
*With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation? <br />
*How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
=== Psy ===<br />
<br />
*Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better? <br />
*Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see. <br />
*Psy-trellis (and maybe psy-RD?) are too strong at very high QPs. <br />
*Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed. <br />
*RD should take into account local variance. <br />
*Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis. <br />
*Lambda should be picked independent of quantizer (i.e. with greater precision). <br />
*Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text. <br />
**If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou). <br />
*Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"... <br />
*Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
=== Lookahead ===<br />
<br />
*'''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.''' <br />
*Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search. <br />
*Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
=== Quantization ===<br />
<br />
*CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c. <br />
*There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful. <br />
*Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high? <br />
*Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former. <br />
*Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
=== Transform ===<br />
<br />
*Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help. <br />
*Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
=== Interlacing ===<br />
<br />
*Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower? <br />
*Constrained intra + adaptive MBAFF. Does anyone care about this? <br />
*PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
=== Weighted Prediction ===<br />
<br />
*'''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.''' <br />
*Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one. <br />
*Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases. <br />
*Improve weighted prediction analysis to do more searching based on an estimated offset vs scale gradient.<br />
<br />
=== Ratecontrol ===<br />
<br />
*VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy. <br />
**Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing. <br />
*Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate. <br />
*Make the frame size and row size predictors better. They currently are kind of crappy. <br />
*Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur. <br />
*2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved. <br />
*2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving). <br />
*Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
=== GPU ===<br />
<br />
*Motion estimation? <br />
**Methods <br />
***Hierarchical? <br />
***2D Wave? <br />
***Something else? <br />
**"Easy": lookahead motion estimation <br />
***Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once. <br />
**"Hard": main motion estimation <br />
***Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search. <br />
***But potentially more useful... <br />
*Other things?<br />
<br />
=== x86 assembly ===<br />
<br />
*Optimize more for the Phenom. <br />
*Yell at holger to commit his local patches. <br />
*Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
=== Other assembly ===<br />
<br />
*NEON assembly is nowhere near complete. <br />
**Chroma MC needs to be rewritten for NV12 support. <br />
*Altivec assembly is very lacking. <br />
*SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
=== Other CPU optimizations ===<br />
<br />
*x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark. <br />
*Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
=== Other features ===<br />
<br />
*MPEG-2 encoding support <br />
*VP8 encoding support <br />
*'''4:2:2 colorspace support''' <br />
*Support for SMPTE timecodes <br />
*Merge speedcontrol <br />
*Mixed lossless/lossy encoding.<br />
*Segment re-encoding<br />
<br />
=== x264CLI ===<br />
<br />
*Finish audio support. Talk to Kovensky about this one. <br />
*Make the filtering system aware of fullrange vs TV range. <br />
*Make the filtering system aware of BT.601 vs BT.709. <br />
*Add more filters. <br />
**Deinterlacers (YADIF). <br />
**Denoisers (HQDN3D?). <br />
**IVTC, decomb? <br />
*Merge L-SMASH mp4 muxer. <br />
*Add TS muxing support using HRD. Talk to kierank about this one. <br />
*Add --device support. <br />
*Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=SOCIS_x264_2011&diff=19461SOCIS x264 20112011-07-25T16:40:31Z<p>Kieran: </p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are only accepting one student this year as part of SOCIS. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project. There are also experts from ARM Shanghai who are optimizing x264 who may also be able to help.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
'''This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.'''<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.</div>Kieranhttps://wiki.videolan.org/index.php?title=SOCIS_x264_2011&diff=19449SOCIS x264 20112011-07-20T20:40:40Z<p>Kieran: /* Projects */</p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are only accepting one student this year as part of SOCIS. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project. There are also experts from ARM Shanghai who are optimizing x264 who may also be able to help.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.</div>Kieranhttps://wiki.videolan.org/index.php?title=SOCIS_x264_2011&diff=19442SOCIS x264 20112011-07-19T14:14:40Z<p>Kieran: </p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project. There are also experts from ARM Shanghai who are optimizing x264 who may also be able to help.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.</div>Kieranhttps://wiki.videolan.org/index.php?title=SOCIS_x264_2011&diff=19429SOCIS x264 20112011-07-14T00:43:04Z<p>Kieran: </p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.<br />
<br />
x264 is part of the VideoLAN candidature for ESA Summer of Code in Space.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project. There are also experts from ARM Shanghai who are optimizing x264 who may also be able to help.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.</div>Kieranhttps://wiki.videolan.org/index.php?title=SOCIS_x264_2011&diff=19428SOCIS x264 20112011-07-14T00:42:29Z<p>Kieran: </p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.<br />
<br />
x264 is part of the VideoLAN candidature for ESA Summer of Code in Space.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project. There are also experts from ARM Shanghai who are optimizing x264 who may also be able to help.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.<br />
<br />
You should also contact the admin [[User:J-b|jb]].</div>Kieranhttps://wiki.videolan.org/index.php?title=SOCIS_x264_2011&diff=19427SOCIS x264 20112011-07-14T00:40:35Z<p>Kieran: </p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.<br />
<br />
x264 is part of the VideoLAN candidature for ESA Summer of Code in Space.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.<br />
<br />
You should also contact the admin [[User:J-b|jb]].</div>Kieranhttps://wiki.videolan.org/index.php?title=SOCIS_x264_2011&diff=19426SOCIS x264 20112011-07-14T00:39:34Z<p>Kieran: Created page with "==Introduction to x264 and Summer of Code== x264 is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, telev…"</p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in -- we've had tons of successful students, both in Google Summer of Code and Google Code-In, most of whom knew nothing about video compression before applying.<br />
<br />
x264 is part of the VideoLAN candidature for Google Summer of Code 2011.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.<br />
<br />
===4:4:4 and 4:2:2 colorspaces===<br />
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.<br />
<br />
The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.<br />
<br />
This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
===Non-local RD optimization===<br />
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).<br />
<br />
This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.<br />
<br />
===H.264/MVC===<br />
This project would be to implement the multiview coding extension (read: 3D video) of H.264. This would mostly be a high-level project: affecting headers, reference handling, and so forth, as opposed to low level compression steps. This would let x264 be used to author 3D Blu-rays and encode 3D content more efficiently. (kierank can get Blu-ray 3D streams verified)<br />
<br />
This project would involve reading a lot of spec and implementing it in the cleanest way possible. It is probably not inordinately difficult.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===4:4:4 and 4:2:2 Colorspaces===<br />
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.<br />
<br />
A partially done qualification task for reference can be found here: <br />
https://github.com/Andreas-HH/x264-colorspaces<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
===Non-local RD optimization===<br />
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.<br />
<br />
===H.264/MVC===<br />
Read the spec and write an outline of what changes you'll have to make to x264 (specifically, with x264 in mind) in order to implement MVC. In other words, a design document. It doesn't have to be very formal, but it needs to clearly demonstrate that you have put some amount of thought into this.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.<br />
<br />
You should also contact the admin [[User:J-b|jb]].</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=19425X264 TODO2011-07-13T21:46:12Z<p>Kieran: </p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.<br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].<br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.<br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.<br />
<br />
===Motion Estimation===<br />
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.<br />
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.<br />
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
* Somehow take into account the effect of motion vector decision on future blocks.<br />
** Hierarchical motion estimation<br />
** Approximations from lookahead MVs<br />
** Iterative ME (as per Snow)<br />
** Trellis motion estimation<br />
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.<br />
** But how do we know which ones we can afford to skip, and when?<br />
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.<br />
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?<br />
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?<br />
* Try sub-8x8 partitions in B-frames. Is it at all useful?<br />
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?<br />
* Fullpel chroma ME?<br />
** For TESA?<br />
<br />
===Intra Analysis===<br />
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
===Mode Decision===<br />
* Can we find more ways to skip more motion searches in multiref?<br />
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
* See the TODOs for deblock-aware RD in common/deblock.c.<br />
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.<br />
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).<br />
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
===Psy===<br />
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?<br />
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.<br />
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.<br />
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.<br />
* RD should take into account local variance.<br />
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.<br />
* Lambda should be picked independent of quantizer (i.e. with greater precision).<br />
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.<br />
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).<br />
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...<br />
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
===Lookahead===<br />
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''<br />
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.<br />
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
===Quantization===<br />
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?<br />
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.<br />
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
===Transform===<br />
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.<br />
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
===Interlacing===<br />
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?<br />
* Constrained intra + adaptive MBAFF. Does anyone care about this?<br />
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
===Weighted Prediction===<br />
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''<br />
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.<br />
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.<br />
<br />
===Ratecontrol===<br />
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.<br />
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.<br />
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.<br />
* Make the frame size and row size predictors better. They currently are kind of crappy.<br />
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
===GPU===<br />
* Motion estimation?<br />
** Methods<br />
*** Hierarchical?<br />
*** 2D Wave?<br />
*** Something else?<br />
** "Easy": lookahead motion estimation<br />
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.<br />
** "Hard": main motion estimation<br />
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.<br />
*** But potentially more useful...<br />
* Other things?<br />
<br />
===x86 assembly===<br />
* Optimize more for the Phenom.<br />
* Optimize for Sandy Bridge.<br />
* Work more on 10-bit asm.<br />
* If you read this, join #x264dev on irc://irc.freenode.net and ping holger to commit his local patches.<br />
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
===Other assembly===<br />
* NEON assembly is nowhere near complete.<br />
** Chroma MC needs to be rewritten for NV12 support.<br />
* Altivec assembly is very lacking.<br />
* SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
===Other CPU optimizations===<br />
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.<br />
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
===Other features===<br />
* MPEG-2 encoding support<br />
* VP8 encoding support<br />
* '''4:2:2 colorspace support'''<br />
* Support for SMPTE timecodes<br />
* Merge speedcontrol<br />
* Mixed lossless/lossy encoding.<br />
<br />
===x264CLI===<br />
* Finish audio support. Talk to Kovensky about this one.<br />
* Make the filtering system aware of fullrange vs TV range.<br />
* Make the filtering system aware of BT.601 vs BT.709.<br />
* Add more filters.<br />
** Deinterlacers (YADIF).<br />
** Denoisers (HQDN3D?).<br />
** IVTC, decomb?<br />
* Merge L-SMASH mp4 muxer.<br />
* Add TS muxing support using HRD. Talk to kierank about this one.<br />
* Add --device support.<br />
* Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=19424X264 TODO2011-07-13T19:49:55Z<p>Kieran: </p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.<br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].<br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.<br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.<br />
<br />
===Motion Estimation===<br />
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.<br />
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.<br />
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
* Somehow take into account the effect of motion vector decision on future blocks.<br />
** Hierarchical motion estimation<br />
** Approximations from lookahead MVs<br />
** Iterative ME (as per Snow)<br />
** Trellis motion estimation<br />
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.<br />
** But how do we know which ones we can afford to skip, and when?<br />
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.<br />
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?<br />
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?<br />
* Try sub-8x8 partitions in B-frames. Is it at all useful?<br />
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?<br />
* Fullpel chroma ME?<br />
** For TESA?<br />
<br />
===Intra Analysis===<br />
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
===Mode Decision===<br />
* Can we find more ways to skip more motion searches in multiref?<br />
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
* See the TODOs for deblock-aware RD in common/deblock.c.<br />
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.<br />
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).<br />
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
===Psy===<br />
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?<br />
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.<br />
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.<br />
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.<br />
* RD should take into account local variance.<br />
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.<br />
* Lambda should be picked independent of quantizer (i.e. with greater precision).<br />
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.<br />
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).<br />
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...<br />
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
===Lookahead===<br />
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''<br />
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.<br />
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
===Quantization===<br />
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?<br />
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.<br />
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
===Transform===<br />
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.<br />
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
===Interlacing===<br />
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?<br />
* Constrained intra + adaptive MBAFF. Does anyone care about this?<br />
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
===Weighted Prediction===<br />
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''<br />
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.<br />
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.<br />
<br />
===Ratecontrol===<br />
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.<br />
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.<br />
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.<br />
* Make the frame size and row size predictors better. They currently are kind of crappy.<br />
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
===GPU===<br />
* Motion estimation?<br />
** Methods<br />
*** Hierarchical?<br />
*** 2D Wave?<br />
*** Something else?<br />
** "Easy": lookahead motion estimation<br />
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.<br />
** "Hard": main motion estimation<br />
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.<br />
*** But potentially more useful...<br />
* Other things?<br />
<br />
===x86 assembly===<br />
* Optimize more for the Phenom.<br />
* Optimize for Sandy Bridge.<br />
* Work more on 10-bit asm.<br />
* If you read this, join #x264dev on irc://irc.freenode.net and ping holger to commit his local patches.<br />
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
===Other assembly===<br />
* NEON assembly is nowhere near complete.<br />
** Chroma MC needs to be rewritten for NV12 support.<br />
* Altivec assembly is very lacking.<br />
* SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
===Other CPU optimizations===<br />
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.<br />
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
===Other features===<br />
* MPEG-2 encoding support<br />
* VP8 encoding support<br />
* '''4:2:2 colorspace support'''<br />
* Support for timecodes<br />
* Merge speedcontrol<br />
* Mixed lossless/lossy encoding.<br />
<br />
===x264CLI===<br />
* Finish audio support. Talk to Kovensky about this one.<br />
* Make the filtering system aware of fullrange vs TV range.<br />
* Make the filtering system aware of BT.601 vs BT.709.<br />
* Add more filters.<br />
** Deinterlacers (YADIF).<br />
** Denoisers (HQDN3D?).<br />
** IVTC, decomb?<br />
* Merge L-SMASH mp4 muxer.<br />
* Add TS muxing support using HRD. Talk to kierank about this one.<br />
* Add --device support.<br />
* Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=18828X264 TODO2011-03-16T10:39:15Z<p>Kieran: /* Other features */</p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.<br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].<br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.<br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.<br />
<br />
===Motion Estimation===<br />
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.<br />
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.<br />
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
* Somehow take into account the effect of motion vector decision on future blocks.<br />
** Hierarchical motion estimation<br />
** Approximations from lookahead MVs<br />
** Iterative ME (as per Snow)<br />
** Trellis motion estimation<br />
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.<br />
** But how do we know which ones we can afford to skip, and when?<br />
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.<br />
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?<br />
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?<br />
* Try sub-8x8 partitions in B-frames. Is it at all useful?<br />
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?<br />
* Fullpel chroma ME?<br />
** For TESA?<br />
<br />
===Intra Analysis===<br />
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
===Mode Decision===<br />
* Can we find more ways to skip more motion searches in multiref?<br />
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
* See the TODOs for deblock-aware RD in common/deblock.c.<br />
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.<br />
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).<br />
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
===Psy===<br />
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?<br />
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.<br />
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.<br />
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.<br />
* RD should take into account local variance.<br />
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.<br />
* Lambda should be picked independent of quantizer (i.e. with greater precision).<br />
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.<br />
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).<br />
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...<br />
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
===Lookahead===<br />
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''<br />
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.<br />
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
===Quantization===<br />
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?<br />
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.<br />
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
===Transform===<br />
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.<br />
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
===Interlacing===<br />
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''<br />
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.<br />
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?<br />
* Constrained intra + adaptive MBAFF. Does anyone care about this?<br />
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
===Weighted Prediction===<br />
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''<br />
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.<br />
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.<br />
<br />
===Ratecontrol===<br />
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.<br />
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.<br />
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.<br />
* Make the frame size and row size predictors better. They currently are kind of crappy.<br />
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
===GPU===<br />
* Motion estimation?<br />
** Methods<br />
*** Hierarchical?<br />
*** 2D Wave?<br />
*** Something else?<br />
** "Easy": lookahead motion estimation<br />
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.<br />
** "Hard": main motion estimation<br />
*** Difficult synchronization issues, not as heavily parallel in terms of number of macroblocks, but far more partition sizes and refs to search.<br />
*** But potentially more useful...<br />
* Other things?<br />
<br />
===x86 assembly===<br />
* Port functions (where useful) to AVX, talk to Daniel Kang for a patch.<br />
* Optimize more for the Phenom.<br />
* Optimize for the upcoming Sandy Bridge.<br />
* Work more on 10-bit asm.<br />
* If you read this, join #x264dev on irc://irc.freenode.net and ping holger to commit his local patches.<br />
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
===Other assembly===<br />
* NEON assembly is nowhere near complete.<br />
** Chroma MC needs to be rewritten for NV12 support.<br />
* Altivec assembly is very lacking.<br />
* SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
===Other CPU optimizations===<br />
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.<br />
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
===Other features===<br />
* MPEG-2 encoding support<br />
* VP8 encoding support<br />
* '''4:2:2 and 4:4:4 colorspace support'''<br />
** '''RGB?'''<br />
* Support for timecodes<br />
* Merge speedcontrol<br />
* Mixed lossless/lossy encoding.<br />
<br />
===x264CLI===<br />
* Finish audio support. Talk to Kovensky about this one.<br />
* Make the filtering system aware of fullrange vs TV range.<br />
* Make the filtering system aware of BT.601 vs BT.709.<br />
* Add more filters.<br />
** Deinterlacers (YADIF).<br />
** Denoisers (HQDN3D?).<br />
** IVTC, decomb?<br />
* Merge L-SMASH mp4 muxer.<br />
* Add TS muxing support using HRD. Talk to kierank about this one.<br />
* Add --device support.<br />
* Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=SoC_x264_2011&diff=18752SoC x264 20112011-03-03T15:08:49Z<p>Kieran: /* H.264/MVC */</p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in.<br />
<br />
x264 is part of the VideoLAN candidature for Google Summer of Code 2011.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.<br />
<br />
===4:4:4 and 4:2:2 colorspaces===<br />
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.<br />
<br />
The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.<br />
<br />
This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
===Non-local RD optimization===<br />
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).<br />
<br />
This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.<br />
<br />
===H.264/MVC===<br />
This project would be to implement the multiview coding extension (read: 3D video) of H.264. This would mostly be a high-level project: affecting headers, reference handling, and so forth, as opposed to low level compression steps. This would let x264 be used to author 3D Blu-rays and encode 3D content more efficiently. (kierank can get Blu-ray 3D streams verified)<br />
<br />
This project would involve reading a lot of spec and implementing it in the cleanest way possible. It is probably not inordinately difficult.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===4:4:4 and 4:2:2 Colorspaces===<br />
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
===Non-local RD optimization===<br />
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.<br />
<br />
===H.264/MVC===<br />
Read the spec and write an outline of what changes you'll have to make to x264 (specifically, with x264 in mind) in order to implement MVC. In other words, a design document. It doesn't have to be very formal, but it needs to clearly demonstrate that you have put some amount of thought into this.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.<br />
<br />
You should also contact the admin [[User:J-b|jb]].</div>Kieranhttps://wiki.videolan.org/index.php?title=SoC_x264_2011&diff=18751SoC x264 20112011-03-03T15:00:51Z<p>Kieran: </p>
<hr />
<div>==Introduction to x264 and Summer of Code==<br />
[[x264]] is one of the most popular video compression libraries in the world, used worldwide for applications such as web video, television broadcast, and Blu-ray authoring. It outclasses practically all commercial implementations both speed and compression-wise. While not actually part of VLC or ffmpeg, it is a major library used by both, licensed under the GPL. Due to its popularity in the commercial world (for example, Youtube and Facebook rely on it), many companies have offered bounties in the past for features and improvements that they found useful.<br />
<br />
But don't let that all scare you. There's still plenty of projects that a student can effectively get involved in.<br />
<br />
x264 is part of the VideoLAN candidature for Google Summer of Code 2011.<br />
<br />
*Lead mentor (and author of this page): Jason Garrett-Glaser (Dark Shikari)<br />
*Possible other mentors: Ronald Bultje (BBB)?, Kieran Kunhya (kierank)<br />
<br />
An overview of x264's structure and algorithms can be found [http://akuvian.org/src/x264/overview_x264_v8_5.pdf here]. It is somewhat outdated, but still mostly accurate. Do note that understanding this is not necessary for all projects.<br />
<br />
On a lighter note, feel free to check out the [http://www.x264.nl/developers/Dark_Shikari/loren.html x264 developer quotes page].<br />
<br />
==Guide to getting involved==<br />
*Hop on IRC--whenever! We have a friendly community that's happy to help and to talk about pretty much whatever. #x264 is the general discussion/user help channel and #x264dev is the development-only channel (Freenode server). If you've never used IRC, grab Chatzilla. There is no such thing as a stupid question--only stupid people--so don't worry about sounding dumb. Just get involved!<br />
*Ask questions--I cannot stress this enough. We've had students who couldn't get anywhere because they were stuck--but didn't ask questions! There's no shame in asking questions; that's how everyone here got involved.<br />
*'''Stay on IRC'''. Even if you have nothing to say, being around lets you get to know the community, get a feel for what's going on, and even sometimes help out by pointing the semicolon I left at the end of my if statement. This applies during the project period too: we expect all students to '''always be on IRC'''. This doesn't mean you have to be actually active all the time, but whenever you're free, you should be at least pingable on IRC, and whenever you're working on Summer of Code, you should be active on IRC.<br />
**The best option for staying online is to get a remote shell account of some sort and leave an IRSSI client on 24/7 in a screen. This lets you easily come back in the morning and see if you missed anything important. checkers can provide you with a shell if you end up being accepted as a student.<br />
*Even if you don't get a slot as a student, or you don't qualify for Summer of Code, we will happily provide you with the exact same support that we would give a student: it is quite reasonable and not at all uncommon for us to have projects of similar scale to Summer of Code be done by students who are not part of the Summer of Code program.<br />
<br />
==Skills needed==<br />
These are required for all listed projects and probably anything not listed, too.<br />
*Basic C programming.<br />
*Basic understanding of video encoding, or at least willingness to do the appropriate reading up on the topic before the summer begins. Most people who get involved don't know much to start with; we don't expect you to!<br />
*To work on anything related directly to the encoder core (not all projects), you'll need to do some significant background reading on relevant topics.<br />
**A PDF with a chapter that can serve as a primer to video compression can be found [http://www.mediafire.com/download.php?auxd23m2snw here]. It also has some more specific chapters on MPEG-4 Part 10 (H.264).<br />
<br />
==Projects==<br />
We are probably only accepting 3-4 students this year. Thus, you will have to prove that you are absolutely able to do your project over the summer--see the qualification tasks. Do remember that the project ideas listed here are merely suggestions; other ideas are always possible if they fit the time and difficulty constraints of SoC. For example, if you have an idea to improve compression (e.g. implementing a new algorithm) that you think we can benefit from, feel free to bring it up.<br />
<br />
===Optimization (ARM)===<br />
x264 prides itself on being one of the most optimized programs in existence while still being reasonably readable and maintainable. This project is about furthering that goal: make it even faster without sacrificing code quality.<br />
<br />
x264's ARM NEON assembly code is well behind its x86 assembly and is missing many optimizations. This project would involve finishing up the ARM assembly (originally done as part of Summer of Code 2009), giving us a much-needed performance boost on ARM.<br />
<br />
NEON knowledge is highly recommended but not required -- but you will have to learn somehow or another. We have a few NEON experts in the community who can help, but none are available for full-time mentoring, so keep this in mind when picking this project.<br />
<br />
===4:4:4 and 4:2:2 colorspaces===<br />
x264 currently only supports the 4:2:0 colorspace, also known as YV12. However, many profession applications require higher precision in the form of 4:2:2 or 4:4:4 chroma subsampling. Furthermore, such sampling is useful for many artificial video sources, like video game captures and presentations, where sharp chroma edges are blurred by the 4:2:0 sampling. This is a large project that, unlike the previous, probably does not involve modifying any assembly if you don't want to. In cases where things would be easier if the assembly ''was'' modified, the other developers will be willing to do it for you if you aren't an assembly guru.<br />
<br />
The project mostly covers changing every place in the code where YV12 was assumed--and making it variable. It also involves handling the potential new syntax elements and adding a new Hadamard transform for the new chroma DC channels. It also may involve modifying x264_scan8, which could be a rather obnoxious task given how many parts of x264 assume certain properties of it.<br />
<br />
This project will involve a lot of coding and a lot of debugging, but none of it should be particularly complex.<br />
<br />
===GPU motion estimation===<br />
While porting x264 entirely to CUDA or OpenCL is an insane task, there are three possible methods that could be used to offload some work to the GPU:<br />
<br />
*High-complexity motion search designed to get useful predictors to be used by the main motion search.<br />
*Massively parallelized lookahead motion search, designed to do a lot of the work normally done in the lookahead thread. May also improve B-frame decision and other parts of the lookahead.<br />
*Motion search designed to completely replace x264's main motion search: would require a lot of threading trickery to sync it perfectly with the main encoder threads.<br />
<br />
The general algorithm that has been agreed on after a great deal of discussion is the hierarchical search method. If you have a better idea, feel free to propose it, of course. More description of this method is in the Qualification Tasks section.<br />
<br />
This project is not recommended unless you have a very significant amount of experience with CUDA or OpenCL.<br />
<br />
===Non-local RD optimization===<br />
x264's biggest weakness is that it only considers the optimal decisions for the current macroblock; it isn't aware of the effects of its decision on the future. There may be significant benefits to be gained via non-local methods, such as iterative optimization. However, especially in H.264, non-local RD is very difficult to do efficiently. This is a project primarily targeted at someone already familiar with video compression: in particular, it *requires* that you have at least some idea with regards to how you would do it! Your idea must not only improve compression, but also do so in some sane amount of time (a 2x speed cost might be tolerable, 400x most certainly not).<br />
<br />
This is probably not very difficult from a coding standpoint and is really more of an algorithmic problem. Since we haven't done it already, that of course means it's a ''hard'' algorithmic problem.<br />
<br />
===H.264/MVC===<br />
This project would be to implement the multiview coding extension (read: 3D video) of H.264. This would mostly be a high-level project: affecting headers, reference handling, and so forth, as opposed to low level compression steps. This would let x264 be used to author 3D Blu-rays and encode 3D content more efficiently. (kierank can mentor and get Blu-ray 3D streams verified)<br />
<br />
This project would involve reading a lot of spec and implementing it in the cleanest way possible. It is probably not inordinately difficult.<br />
<br />
==Qualification tasks==<br />
Unlike many other projects, such as ffmpeg, x264's policy for qualification tasks is to use tasks that are directly useful to ''you'' for your summer project. That is, the projects directly lead you into the start of your project and create a base for you to work off for the rest of the summer. This is, in our opinion, much better than making you work on something completely unrelated. We're willing to give all the technical help you need, but of course we won't write the code for you. "Passing" a qualification task is at the mentor's discretion. Note these are designed to be '''relatively difficult''' and help lead you into your main project. If you can't do the qualification task for the project, you surely cannot do the project either!<br />
<br />
Again, to reiterate, we will guide you through as much of the codebase as you need to do your work. '''This page is not supposed to give you all the information you need to do these tasks: you are expected to contact us for more information.''' Feel free to ask tons of questions. On #x264dev IRC channel on Freenode, of course.<br />
<br />
None of these tasks are supposed to take more than a few days to a week of work. '''If you successfully complete one, we will almost surely accept you as a student.'''<br />
<br />
===Optimization (ARM)===<br />
If you're interested in the optimization task, the qualification task is to speed up x264 on ARM by at least 2% by writing new NEON functions.<br />
<br />
===4:4:4 and 4:2:2 Colorspaces===<br />
If you're interested in working on this project, your task is to produce an x264-encoded bitstream in 4:4:4 or 4:2:2 format. It does not actually have to be at all watchable (that is, you don't have to implement any of the code to handle motion compensation, deblocking, or anything else involving 4:4:4/4:2:2 chroma data), but the bitstream has to be written correctly (correct syntax elements). The patch you write for this will be the starting point for your main project.<br />
<br />
===GPU Motion Estimation===<br />
Your task for this project will be to write a C version of your final algorithm. It doesn't need to deal with any of the corner cases; all it has to do is run before the main encoding loop, deciding the motion vectors for the frame. It doesn't even have to work with threading. It doesn't have to support sub-16x16 partitions either. Assuming you didn't propose another, the hierarchical search works via the following algorithm:<br />
<br />
* Set N equal to 2^M, where M is an integer. A common M is 4.<br />
* WHILE N is greater than 1:<br />
:* Downscale the image (from the original) by a factor of N.<br />
:* Do an ordinary diamond motion search on the image with block size 16x16. Assume the predicted motion vector to be equal to the median of the top, left, and top right motion vectors (as per H.264 MV prediction)... but use the motion vectors from the previous iteration, not the current for these (this is what allows you to parallelize things with CUDA).<br />
:* For each block after searching, split the motion vectors of that block into 4 separate (but equal) motion vectors. These will be used as the starting point for the searches in the next iteration. Each iteration progressively refines the result at a progressively lesser downscale.<br />
:* N = N/2<br />
* Do a final refine at no downscale at all.<br />
<br />
===Non-local RD optimization===<br />
Implement an extremely minimal prototype of some part of your idea. It can be ugly, hacky, and limited; all that matters is you demonstrate that you can take an idea and turn it into code in x264. Bonus points if the idea actually works.<br />
<br />
===H.264/MVC===<br />
Read the spec and write an outline of what changes you'll have to make to x264 (specifically, with x264 in mind) in order to implement MVC. In other words, a design document. It doesn't have to be very formal, but it needs to clearly demonstrate that you have put some amount of thought into this.<br />
<br />
==Contact info==<br />
If you are interested, drop by #x264dev or #x264 on Freenode and ping Dark Shikari.<br />
<br />
You should also contact the admin [[User:J-b|jb]].</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=18641X264 TODO2011-01-24T08:36:17Z<p>Kieran: </p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.<br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].<br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.<br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.<br />
<br />
===Motion Estimation===<br />
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.<br />
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.<br />
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
* Somehow take into account the effect of motion vector decision on future blocks.<br />
** Hierarchical motion estimation<br />
** Approximations from lookahead MVs<br />
** Iterative ME (as per Snow)<br />
** Trellis motion estimation<br />
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.<br />
** But how do we know which ones we can afford to skip, and when?<br />
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.<br />
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?<br />
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?<br />
* Try sub-8x8 partitions in B-frames. Is it at all useful?<br />
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?<br />
* Fullpel chroma ME?<br />
** For TESA?<br />
<br />
===Intra Analysis===<br />
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
===Mode Decision===<br />
* Can we find more ways to skip more motion searches in multiref?<br />
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
* See the TODOs for deblock-aware RD in common/deblock.c.<br />
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.<br />
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).<br />
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
===Psy===<br />
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?<br />
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.<br />
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.<br />
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.<br />
* RD should take into account local variance.<br />
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.<br />
* Lambda should be picked independent of quantizer (i.e. with greater precision).<br />
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.<br />
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).<br />
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...<br />
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
===Lookahead===<br />
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''<br />
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.<br />
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
===Quantization===<br />
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?<br />
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.<br />
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
===Transform===<br />
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.<br />
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
===Interlacing===<br />
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''<br />
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.<br />
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?<br />
* Constrained intra + adaptive MBAFF. Does anyone care about this?<br />
* PAFF + MBAFF adaptive - PAFF performs better than Adaptive MBAFF on high motion scenes because it can predict from the previous field.<br />
<br />
===Weighted Prediction===<br />
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''<br />
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.<br />
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.<br />
<br />
===Ratecontrol===<br />
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.<br />
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.<br />
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.<br />
* Make the frame size and row size predictors better. They currently are kind of crappy.<br />
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.<br />
** Just force all blocks to skip?<br />
** Drop frames?<br />
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)<br />
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
===GPU===<br />
* Motion estimation?<br />
** Methods<br />
*** Hierarchical?<br />
*** 2D Wave?<br />
*** Something else?<br />
** "Easy": lookahead motion estimation<br />
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.<br />
** "Hard": main motion estimation<br />
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.<br />
*** But potentially more useful...<br />
* Other things?<br />
<br />
===x86 assembly===<br />
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.<br />
* Optimize more for the Phenom.<br />
* Optimize for the upcoming Sandy Bridge.<br />
* Work more on 10-bit asm.<br />
* Convince holger to commit his local patches.<br />
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
===Other assembly===<br />
* NEON assembly is nowhere near complete.<br />
** Chroma MC needs to be rewritten for NV12 support.<br />
* Altivec assembly is very lacking.<br />
* SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
===Other CPU optimizations===<br />
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.<br />
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
===Other features===<br />
* MPEG-2 encoding support<br />
* VP8 encoding support<br />
* '''4:2:2 and 4:4:4 colorspace support'''<br />
** '''RGB?'''<br />
* Support for timecodes<br />
* Merge speedcontrol<br />
<br />
===x264CLI===<br />
* Finish audio support. Talk to Kovensky about this one.<br />
* Make the filtering system aware of fullrange vs TV range.<br />
* Make the filtering system aware of BT.601 vs BT.709.<br />
* Add more filters.<br />
** Deinterlacers (YADIF).<br />
** Denoisers (HQDN3D?).<br />
** IVTC, decomb?<br />
* Merge L-SMASH mp4 muxer.<br />
* Add TS muxing support using HRD. Talk to kierank about this one.<br />
* Add --device support.<br />
* Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=18607X264 TODO2011-01-16T19:35:20Z<p>Kieran: </p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.<br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].<br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.<br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.<br />
<br />
===Motion Estimation===<br />
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.<br />
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.<br />
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
* Somehow take into account the effect of motion vector decision on future blocks.<br />
** Hierarchical motion estimation<br />
** Approximations from lookahead MVs<br />
** Iterative ME (as per Snow)<br />
** Trellis motion estimation<br />
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.<br />
** But how do we know which ones we can afford to skip, and when?<br />
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.<br />
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?<br />
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?<br />
* Try sub-8x8 partitions in B-frames. Is it at all useful?<br />
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?<br />
* Fullpel chroma ME?<br />
** For TESA?<br />
<br />
===Intra Analysis===<br />
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
===Mode Decision===<br />
* Can we find more ways to skip more motion searches in multiref?<br />
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
* See the TODOs for deblock-aware RD in common/deblock.c.<br />
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.<br />
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).<br />
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
===Psy===<br />
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?<br />
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.<br />
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.<br />
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.<br />
* RD should take into account local variance.<br />
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.<br />
* Lambda should be picked independent of quantizer (i.e. with greater precision).<br />
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.<br />
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).<br />
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...<br />
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
===Lookahead===<br />
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''<br />
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.<br />
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
===Quantization===<br />
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?<br />
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.<br />
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
===Transform===<br />
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.<br />
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
===Interlacing===<br />
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''<br />
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.<br />
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?<br />
* Constrained intra + adaptive MBAFF. Does anyone care about this?<br />
* PAFF + MBAFF adaptive.<br />
<br />
===Weighted Prediction===<br />
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''<br />
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.<br />
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.<br />
<br />
===Ratecontrol===<br />
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.<br />
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.<br />
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.<br />
* Make the frame size and row size predictors better. They currently are kind of crappy.<br />
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.<br />
** Just force all blocks to skip?<br />
** Drop frames?<br />
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)<br />
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
===GPU===<br />
* Motion estimation?<br />
** Methods<br />
*** Hierarchical?<br />
*** 2D Wave?<br />
*** Something else?<br />
** "Easy": lookahead motion estimation<br />
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.<br />
** "Hard": main motion estimation<br />
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.<br />
*** But potentially more useful...<br />
* Other things?<br />
<br />
===x86 assembly===<br />
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.<br />
* Optimize more for the Phenom.<br />
* Optimize for the upcoming Sandy Bridge.<br />
* Work more on 10-bit asm.<br />
* Convince holger to commit his local patches.<br />
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
===Other assembly===<br />
* NEON assembly is nowhere near complete.<br />
** Chroma MC needs to be rewritten for NV12 support.<br />
* Altivec assembly is very lacking.<br />
* SPARC VIS assembly is only available when high bit-depth is disabled.<br />
<br />
===Other CPU optimizations===<br />
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.<br />
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
===Other features===<br />
* MPEG-2 encoding support<br />
* VP8 encoding support<br />
* '''4:2:2 and 4:4:4 colorspace support'''<br />
** '''RGB?'''<br />
* Support for timecodes<br />
* Merge speedcontrol<br />
<br />
===x264CLI===<br />
* Finish audio support. Talk to Kovensky about this one.<br />
* Make the filtering system aware of fullrange vs TV range.<br />
* Make the filtering system aware of BT.601 vs BT.709.<br />
* Add more filters.<br />
** Deinterlacers (YADIF).<br />
** Denoisers (HQDN3D?).<br />
** IVTC, decomb?<br />
* Merge L-SMASH mp4 muxer.<br />
* Add TS muxing support using HRD. Talk to kierank about this one.<br />
* Add --device support.<br />
* Add automatic --level restriction support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=17623X264 TODO2010-11-16T01:32:30Z<p>Kieran: </p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.<br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].<br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.<br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.<br />
<br />
===Motion Estimation===<br />
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.<br />
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.<br />
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
* Somehow take into account the effect of motion vector decision on future blocks.<br />
** Hierarchical motion estimation<br />
** Approximations from lookahead MVs<br />
** Iterative ME (as per Snow)<br />
** Trellis motion estimation<br />
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.<br />
** But how do we know which ones we can afford to skip, and when?<br />
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.<br />
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?<br />
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?<br />
* Try sub-8x8 partitions in B-frames. Is it at all useful?<br />
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?<br />
* Fullpel chroma ME?<br />
** For TESA?<br />
<br />
===Intra Analysis===<br />
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
===Mode Decision===<br />
* Can we find more ways to skip more motion searches in multiref?<br />
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
* See the TODOs for deblock-aware RD in common/deblock.c.<br />
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.<br />
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).<br />
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
===Psy===<br />
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?<br />
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.<br />
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.<br />
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.<br />
* RD should take into account local variance.<br />
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.<br />
* Lambda should be picked independent of quantizer (i.e. with greater precision).<br />
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.<br />
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).<br />
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...<br />
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
===Lookahead===<br />
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''<br />
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.<br />
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
===Quantization===<br />
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?<br />
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.<br />
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
===Transform===<br />
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.<br />
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
===Interlacing===<br />
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''<br />
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.<br />
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?<br />
* Constrained intra + adaptive MBAFF. Does anyone care about this?<br />
<br />
===Weighted Prediction===<br />
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''<br />
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.<br />
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.<br />
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.<br />
<br />
===Ratecontrol===<br />
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.<br />
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.<br />
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.<br />
* Make the frame size and row size predictors better. They currently are kind of crappy.<br />
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.<br />
** Just force all blocks to skip?<br />
** Drop frames?<br />
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)<br />
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
===GPU===<br />
* Motion estimation?<br />
** Methods<br />
*** Hierarchical?<br />
*** 2D Wave?<br />
*** Something else?<br />
** "Easy": lookahead motion estimation<br />
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.<br />
** "Hard": main motion estimation<br />
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.<br />
*** But potentially more useful...<br />
* Other things?<br />
<br />
===x86 assembly===<br />
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.<br />
* Optimize more for the Phenom.<br />
* Optimize for the upcoming Sandy Bridge.<br />
* Work on 10-bit asm. Talk to irock about this.<br />
* Convince holger to commit his local patches.<br />
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
===Other assembly===<br />
* NEON assembly is nowhere near complete.<br />
** Chroma MC needs to be rewritten for NV12 support.<br />
* Altivec assembly is very lacking.<br />
<br />
===Other CPU optimizations===<br />
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.<br />
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
===Other features===<br />
* MPEG-2 encoding support<br />
* VP8 encoding support<br />
* '''4:2:2 and 4:4:4 colorspace support'''<br />
** '''RGB?'''<br />
* Support for timecodes<br />
* Merge speedcontrol<br />
<br />
===x264CLI===<br />
* Finish audio support. Talk to Kovensky about this one.<br />
* Add more filters.<br />
** Deinterlacers (YADIF).<br />
** Denoisers (HQDN3D?).<br />
** IVTC, decomb?<br />
* Merge L-SMASH mp4 muxer.<br />
* Add TS muxing support using HRD. Talk to kierank about this one.<br />
* Add --device support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_TODO&diff=17622X264 TODO2010-11-16T01:31:55Z<p>Kieran: </p>
<hr />
<div>This page contains an incomplete list of things available in x264 for you to do. It's organized into sections covering various parts of x264.<br />
<br />
Some useful resources: [http://www.x264.nl/developers/Dark_Shikari/?dir=./src Dark Shikari's pile of junk], [http://akuvian.org/src/x264/ Pengvado's pile of junk].<br />
<br />
If you're interested in doing any of this, drop by #x264dev on Freenode IRC. There are no experience or educational requirements for doing any of this, though you are expected to know how to code.<br />
<br />
Bolded features may have companies willing to sponsor or provide bounties. This is not complete either; just because it's not bolded doesn't mean there aren't resources out there. If your company is interested in offering a bounty, drop by IRC.<br />
<br />
===Motion Estimation===<br />
* Sequential elimination (SEA), used for exhaustive search, might be more generally applicable to algorithms like UMH, by letting us skip a lot of SADs. The downside is we won't be able to use SAD_X4 anymore.<br />
* (T)ESA is currently wrong for motion searches done on weightp duplicates. This effect is miniscule, but it still should be fixed.<br />
* Hierarchical motion estimation might be a useful way to catch very long motion vectors without the cost of UMH or ESA. It might also help regularize motion.<br />
* Somehow take into account the effect of motion vector decision on future blocks.<br />
** Hierarchical motion estimation<br />
** Approximations from lookahead MVs<br />
** Iterative ME (as per Snow)<br />
** Trellis motion estimation<br />
* We don't need to check all 11 predictors all the time for 16x16 fullpel motion search.<br />
** But how do we know which ones we can afford to skip, and when?<br />
** Xvid and libtheora have algorithms for this, but the former's is almost surely 100% useless and the latter doesn't seem impressive either.<br />
* libtheora does fullpel motion estimation on the source pixels instead of decoded pixels. Does this give a better starting point for the subpel search and discourage "weird" MVs?<br />
* With extremely fast encoding settings (subme 0), can we rip off lookahead MVs instead of doing a real search?<br />
* Try sub-8x8 partitions in B-frames. Is it at all useful?<br />
* Try bidir motion estimation for fullpel. That is, considering L1's MV when doing L0 (or vice versa). Xvid does this. How much does it help?<br />
* Fullpel chroma ME?<br />
** For TESA?<br />
<br />
===Intra Analysis===<br />
* Make the early terminations smarter. Currently they're just hacks -- some statistical analysis might be useful.<br />
* SAD (subme 1) i8x8 vs i4x4 decision is a bit bad. Can it be improved without significant speed loss?<br />
<br />
===Mode Decision===<br />
* Can we find more ways to skip more motion searches in multiref?<br />
* On extremely fast encoding settings, fast skip is actually kind of slow. But anything dumber (e.g. SAD) is completely useless. Is there some better balance that can be achieved here?<br />
* See the TODOs for deblock-aware RD in common/deblock.c.<br />
* Is there a faster way than SA8D/SATD to do 8x8dct vs 4x4dct mode decision? At very fast settings, the time this uses is nontrivial.<br />
* Is there a faster way than RD to do 8x8dct vs 4x4dct mode decision that's still better than SATD? RD takes over an order of magnitude more time than SATD, so it might be useful to have something in the middle.<br />
* Is there some value to swapping the mode decision metric from SATD to SA8D if we think that the macroblock will use 8x8dct? [http://akuvian.org/src/x264/x264_dct8_guess.diff This has been tried before], but only helped if our guess was extremely good (better than we could get in reality).<br />
* With trellis 2, can we skip most of CABAC and CAVLC bit cost calculation?<br />
* How about a "brute force" mode decision that takes no shortcuts (no early ref termination in p8x8, no SATD thresholds, etc)?<br />
<br />
===Psy===<br />
* Psy-RD is a hack. It works, but it's a hack. If you apply QNS with Psy-RD as the metric, it goes way overboard and gives terrible results. This means that Psy-RD only works because normal mode decision is limited in the way it can modify the image to better suit the metric. Is there a way to make it better?<br />
* Should RD be linear at all? Perhaps we should weight more heavily against low quality blocks and also try to ignore minuscule distortion that viewers can't see.<br />
* Psy-trellis (and maybe psy-RD?) are too strong at very high QPs.<br />
* Psy-trellis should be merged with Psy-RD. There are patches for this, but they probably won't be committed until psy-RD itself is fixed.<br />
* RD should take into account local variance.<br />
* Lambda should be varied on a per-DCT-block basis instead of a per-macroblock basis.<br />
* Lambda should be picked independent of quantizer (i.e. with greater precision).<br />
* Classic problem: a block is mostly high complexity but has a small area of low complexity. How do we judge whether that area is important? Good example: sharp text on background with film grain; grain gets blurred out because of the text.<br />
** If we think it's important all the time, we ruin the quality of many clips that rely on raising complexity on edges (Touhou).<br />
* Should motion estimation lambda be as high as it is at very high quantizers? There's some value to capturing "true motion"...<br />
* Macroblock tree correlates pretty well with visual perception in that its concept of a "high complexity" matches well with the visual concept. Except for local illumination changes. Talk to Dark Shikari for a patch.<br />
<br />
===Lookahead===<br />
* '''Lookahead should be multithreaded, either by splitting the frame (sliced threads) or running multiple frame analysis calls at once.'''<br />
* Temporal MV predictors in lookahead? There's a patch for these somewhere, but they biased heavily in favor of B-frames, likely by improving the motion search.<br />
* Should lookahead use variable lambda based on quantizer (esp. due to adaptive quant)? If so, should it take into account estimated ratecontrol quantizer, too? If so, how?<br />
<br />
===Quantization===<br />
* CAVLC "trellis" is a hack. It works, but it's a hack. Make it better. See the TODOs in encoder/rdo.c.<br />
* There's room for something between trellis and deadzone in terms of complexity. libvpx has a good example -- it biases towards zero-runs in its "medium speed" quantizer. This can't be SIMD'd easily, but is still vastly faster than trellis. A nonlinear quantizer (be more likely to round up larger coefficients) might also be useful.<br />
* Floyd-Steinberg for quantization? Try pushing quantization error to nearby DCT coefficients. Should this go from high to low or low to high?<br />
* Energy-preserving quantizer -- maintain L1 (or maybe L2? I'm not sure) energy. Should we maintain it in the spatial domain (post-iDCT) or residual domain? Probably the former.<br />
* Decimation is currently just a ripoff of the JVT recommended algorithm. Can we do this more optimally? With RD?<br />
<br />
===Transform===<br />
* Analyze the error characteristics of the fDCT. Is there any way to make it more accurate without much speed loss? Particularly at extremely low quantizers, this might help.<br />
* Before forward transform, run a "blocking filter" that acts as the approximate inverse of the deblock filter. See [http://akuvian.org/src/x264/Shwang_loopfilter_thesis.pdf this paper].<br />
<br />
===Interlacing===<br />
* '''Finish adaptive MBAFF. Talk to horlicks about this one.'''<br />
* Make slice-max-mbs/slice-max-size work with interlacing. Pretty much requires a good portion of adaptive MBAFF.<br />
* Lookahead currently blend-deinterlaces to get the lowres. Is this a good idea? Is there something better that isn't much slower?<br />
* Constrained intra + adaptive MBAFF. Does anyone care about this?<br />
<br />
===Weighted Prediction===<br />
* '''Make weightp work with interlacing. Preferably abuse reference duplication to make it useful for MBAFF.'''<br />
* Make weightp work with chroma. Talk to DylanZA about getting his current patch for this one.<br />
* Finish K-means decision for weightp. Talk to DylanZA about getting his current patch for this one.<br />
* Add explicit weighting for B-frames, too. This helps in nonlinear fades, among other cases.<br />
<br />
===Ratecontrol===<br />
* VBV might be able to utilize the ability to re-encode a row of the frame for improved accuracy.<br />
** Maybe re-encode everything in case of an underflow that row-reencoding can't fix? This might be better than underflowing.<br />
* Current per-frame VBV is a hack. It only adapts per row and is O(N^2), where N is the number of rows. An O(N) solution would be able to react more often and thus be more accurate.<br />
* Make the frame size and row size predictors better. They currently are kind of crappy.<br />
* Ratecontrol code as a whole is a bit of a mess. It could be improved. There's a lot of cruft left over that is probably not needed now, like qblur.<br />
* 2-pass VBV is actually more likely to underflow than 1-pass because it doesn't adapt as aggressively and trusts first pass data a lot. This trust is often misplaced if the first pass was a fast one. This should be improved.<br />
* "Emergency mode" for VBV -- handle cases where abiding by VBV is impossible within normal quantizer bounds.<br />
** Just force all blocks to skip?<br />
** Drop frames?<br />
** Allow QPs higher than 51? (Fake QPs -- just results in "denoising" away DCT coefficients)<br />
* 2-pass macroblock-tree: if we added the ability to do macroblock-tree on real encoded data, we'd get better results (particularly with repeating patterns and multiref, such as an anime character's mouth moving).<br />
* Macroblock-tree: make it more psy-aware. Maybe we should cap how much it lowers the quantizer on extremely static scenes? This might tie into the "just-noticeable error" issue in RD.<br />
<br />
===GPU===<br />
* Motion estimation?<br />
** Methods<br />
*** Hierarchical?<br />
*** 2D Wave?<br />
*** Something else?<br />
** "Easy": lookahead motion estimation<br />
*** Extremely high parallelism, hundreds of frame searches (each with thousands of searches) at once.<br />
** "Hard": main motion estimation<br />
*** Difficult synchronization issues, not as heavily parallel in terms of numb of macroblocks, but far more partition sizes and refs to search.<br />
*** But potentially more useful...<br />
* Other things?<br />
<br />
===x86 assembly===<br />
* Finish AVX support for x86inc.asm. Talk to Dark Shikari for a patch.<br />
* Optimize more for the Phenom.<br />
* Optimize for the upcoming Sandy Bridge.<br />
* Work on 10-bit asm. Talk to irock about this.<br />
* Convince holger to commit his local patches.<br />
* Make a merged SA8D/SATD for the 8x8dct mode decision, since the two share most of their calculation. Hadamard_ac already does this, but slightly differently.<br />
<br />
===Other assembly===<br />
* NEON assembly is nowhere near complete.<br />
** Chroma MC needs to be rewritten for NV12 support.<br />
* Altivec assembly is very lacking.<br />
<br />
===Other CPU optimizations===<br />
* x264 needs more prefetching. How many L1 and L2 cache misses (particularly L1) can we get rid of via smart prefetching in the right places? Warning: this is often hard to benchmark.<br />
* Different CPUs take different relative times for some functions. Is this enough (particularly across architectures) to justify different encoding settings for different CPUs?<br />
<br />
===Other features===<br />
* MPEG-2 encoding support<br />
* VP8 encoding support<br />
* '''4:2:2 and 4:4:4 colorspace support'''<br />
** '''RGB?'''<br />
* Support for timecodes<br />
<br />
===x264CLI===<br />
* Finish audio support. Talk to Kovensky about this one.<br />
* Add more filters.<br />
** Deinterlacers (YADIF).<br />
** Denoisers (HQDN3D?).<br />
** IVTC, decomb?<br />
* Merge L-SMASH mp4 muxer.<br />
* Add TS muxing support using HRD. Talk to kierank about this one.<br />
* Add --device support.</div>Kieranhttps://wiki.videolan.org/index.php?title=X264_GCodeIn_Ideas&diff=17509X264 GCodeIn Ideas2010-11-06T21:40:57Z<p>Kieran: </p>
<hr />
<div>= Google Code In =<br />
<br />
This page is about gathering '''ideas''' for the x264 project for acceptance in the [http://code.google.com/opensource/gci/2010-11/index.html Google Code In 2010-2011] program.<br />
<br />
x264 has been part of Videolan's [http://code.google.com/soc/ Google Summer of Code] in [[SoC_2008|2008]], [[SoC_2009|2009]] and [[SoC_2010|2010]].<br />
<br />
= Ideas for x264 =<br />
<br />
== Warning ==<br />
This is a temporary page for listing ideas for Google Code-in tasks. <br />
<br />
The final tasks will be moved to melange, when needed.<br />
<br />
== Tools ==<br />
'''Category''': Quality Assurance<br><br />
'''Description''': Automated regression test tool<br><br />
'''Details''': x264 has no standard regression test tool, and all the developers rely on their own scripts. An automated regression test tool should be able to:<br />
<br />
* Test any two revisions against each other.<br />
* Test a wide variety of x264 settings combinations. We can help you here by giving you some of the existing scripts.<br />
* Report the status of a revision: working (basic regression test passes), broken (basic regression test fails), crash, etc.<br />
* Report the difference between any two revisions (output differs, output is identical, which tests' output differs, etc).<br />
<br />
'''Outcome''': A regression test script (in Bash, Perl, Python, or similar)<br><br />
'''Difficulty''': medium-hard<br><br />
'''Tools''': x264, JM, ffmpeg, Linux, git<br><br />
'''Time''': 5 days<br><br />
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari)<br><br />
<br />
== CLI Development ==<br />
<br />
'''Category''': Video filtering<br><br />
'''Description''': Port ''any'' video filter to x264<br><br />
'''Details''': x264 recently got a filtering system that allows users to process video before encoding -- resizing, cropping, and so forth. We'd like to add fancier filters by porting them from other projects, like mplayer's libmpcodecs. These could include denoising, sharpening, blurring, and so forth.<br><br />
<br />
You can pick ''any filter you want'' and port it -- just make sure to ask first to make sure that it would make sense in the context of x264.<br><br />
<br />
Do note all filters must reach the code quality standards of x264, so in some cases you may need to do more than just copy-pasting: reformatting code, etc.<br><br />
<br />
'''Outcome''': A new filter in x264.<br><br />
'''Difficulty''': hard<br><br />
'''Tools''': x264<br><br />
'''Time''': 7 days<br><br />
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9)<br><br />
<br />
'''Category''': Add a --device option<br><br />
'''Description''': Add a new option to x264cli so users can specify a particular device<br><br />
'''Details''': Edit x264.c to change the input settings for a particular device in a similar way to --preset and --tune<br><br />
<br />
Most of the settings for specific devices (e.g iPod, Xbox, AppleTV) can be found online. x264 developers can help with more complex devices such as Blu-Ray.<br />
<br />
'''Outcome''': Users can specify a specific device to x264.<br><br />
'''Difficulty''': easy-medium<br><br />
'''Tools''': x264<br><br />
'''Time''': 1-2 days<br><br />
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), James Darnley (J_Darnley), Steven Walters (kemuri-_9), Kieran Kunhya (kierank)<br><br />
<br />
== libx264 Core Development ==<br />
<br />
'''Category''': Assembly Optimization<br><br />
'''Description''': Write ''any'' assembly function: x86 (MMX/SSE), PowerPC (Altivec), or ARM (ARMv6/NEON)<br><br />
'''Details''': x264 has boatloads of C functions with assembly versions that haven't been written yet. Assembly functions are really great for Google Code-In because they can be written without any knowledge of the overall program. Most functions are only a few lines long in C, making an assembly implementation relatively easy. In many cases, some (but not all) versions of a function are already written in assembly, providing good examples for future versions.<br><br />
<br />
If you pick this task, it's highly recommended that you have some experience in assembly coding already. Writing a single function is not too difficult if you already have some experience, but if you've never done it before, it will take significantly more time. Of course, if you're interested in learning, we'd be happy to teach you--Jason has taught many many students before. It usually takes a couple hours to learn the basics.<br><br />
'''Outcome''': A new assembly function in x264.<br><br />
'''Difficulty''': hard<br><br />
'''Tools''': x264, yasm (for x86) or gas (for ARM)<br><br />
'''Time''': 0.5-7 days, depends heavily on function complexity<br><br />
'''Possible Mentors''': Jason Garrett-Glaser (Dark Shikari), Holger Lubitz (holger), Loren Merritt (pengvado)<br><br />
<br />
== Contact ==<br />
For ANY question, contact [[User:Dark Shikari]], preferably on IRC.<br />
<br />
IRC channel: #x264dev on irc://irc.freenode.net</div>Kieran