Documentation:Unicode - Revision history

DoesItReallyMatter: add to :Category:libVLC

2014-11-09T11:07:24Z

← Older revision		Revision as of 11:07, 9 November 2014
Line 36:		Line 36:

	Adapted from a [[vlc-devel]] post by [[User:Courmisch\|Courmisch]] (December 2006).		Adapted from a [[vlc-devel]] post by [[User:Courmisch\|Courmisch]] (December 2006).

			[[Category:libVLC]]

J-b: moved Unicode to Documentation:Unicode

2010-11-23T11:33:41Z

moved Unicode to Documentation:Unicode

← Older revision	Revision as of 11:33, 23 November 2010
(No difference)

Tek: Redone into nicer English.

2009-05-16T13:02:47Z

Redone into nicer English.

Funman: /* Backward compatibility note */

2007-09-29T16:03:28Z

Backward compatibility note

← Older revision		Revision as of 16:03, 29 September 2007
Line 24:		Line 24:

	Now for the good (?) news: in trunk, the only parts that still use the <code>*Locale</code> LibVLC APIs are:		Now for the good (?) news: in trunk, the only parts that still use the <code>*Locale</code> LibVLC APIs are:
	* ncurses (if you are brave, you can try porting to wide ncurses...),
	* GnomeVFS (pretty much dead and not aimed at Win32 anyway),		* GnomeVFS (pretty much dead and not aimed at Win32 anyway),
	* GnuTLS (no support from GnuTLS library),		* GnuTLS (no support from GnuTLS library),

Earthworm: /* Overview */

2007-03-08T23:06:07Z

Overview

← Older revision		Revision as of 23:06, 8 March 2007
Line 2:		Line 2:

	==Overview==		==Overview==
	As a general rules, all character strings passed to LibVLC should be in UTF-8. This includes but is far from limited to the playlist, and all the ''chain'' from the input/access to the outputs. We pretty much needed Unicode anyway; the only options were UTF-8 and wide char. We ~~poke~~ UTF-8 because, as disruptive as it has been, it remains much less disruptive than wide characters. Also wide characters means different things on different platforms (e.g. UCS2LE on old Windows, UTF-16LE on		As a general rules, all character strings passed to LibVLC should be in UTF-8. This includes but is far from limited to the playlist, and all the ''chain'' from the input/access to the outputs. We pretty much needed Unicode anyway; the only options were UTF-8 and wide char. We picked UTF-8 because, as disruptive as it has been, it remains much less disruptive than wide characters. Also wide characters means different things on different platforms (e.g. UCS2LE on old Windows, UTF-16LE on
	modern Windows, UTF-32 on glibc), are a pain to use in POSIX environment (including Mac OS X and Linux which most of the devs are using), and might have needed lots of libc replacements <code>wc*</code> functions on some platforms.		modern Windows, UTF-32 on glibc), are a pain to use in POSIX environment (including Mac OS X and Linux which most of the devs are using), and might have needed lots of libc replacements <code>wc*</code> functions on some platforms.

Courmisch at 18:53, 6 March 2007

2007-03-06T18:53:03Z

New page

LibVLC was switched to UTF-8 internally sometime in 2005, and it's now getting kind of useable again. Still, there's been quite many changes throughout 2006, so let's make a summary.

==Overview==
As a general rules, all character strings passed to LibVLC should be in UTF-8. This includes but is far from limited to the playlist, and all the ''chain'' from the input/access to the outputs. We pretty much needed Unicode anyway; the only options were UTF-8 and wide char. We poke UTF-8 because, as disruptive as it has been, it remains much less disruptive than wide characters. Also wide characters means different things on different platforms (e.g. UCS2LE on old Windows, UTF-16LE on
modern Windows, UTF-32 on glibc), are a pain to use in POSIX environment (including Mac OS X and Linux which most of the devs are using), and might have needed lots of libc replacements <code>wc*</code> functions on some platforms.

==Historical approach==
The original approach has been to rely on <code>ToLocale</code>, <code>FromLocale</code> and <code>LocaleFree</code> (and <code>ToLocaleDup</code>, <code>FromLocaleDup</code> and <code>free</code>) whenever a character string in the OS native representation was needed. It turned out to be a big mistake on Windows, which uses no less than three different representations internally: UTF-16 in the kernel and file system (Unicode/wide), ACP (ANSI Code Page) which is a locale-dependant 8-bit character set (CP1252 in the West), and even "OEM" (cp437 in the US, cp850 in Western Europe). Unfortunately, (To|From)Locale were using ACP, which prevents usage of Unicode code points outside the range of the local charset, while the filesystem and native Win32 supports them. This means VLC cannot ''reach'' some files.

==Current approach==
So we now have a comprehensive (but not fully yet) set of higher-level wrappers that accept UTF-8 character string to represent filenames, or text to be outputted to the console. At the time of writing, trunk includes:

* utf8_open, utf8_fopen, utf8_stat, utf8_lstat, utf8_mkdir, utf8_opendir, utf8_readdir, utf8_scandir for file system operation,

* utf8_fprintf, utf8_vfprintf for console output (or actually output in any file who needs local charset).

All of these functions behave like their non "utf8_"-prefixed equivalent with the exceptions that they expect UTF-8 character strings.

We also have EnsureUTF8 that force a string into UTF-8 (replaces invalid character sequences with '?') and IsUTF8 that checks if the string is valid UTF-8 (returns NULL if not). These are useful when reading UTF-8 from untrusted sources, such as the network, before injecting the data into VLC.

===Backward compatibility note===
ToLocale and FromLocale should NEVER be used for filesystem operations anymore, except on code that is not aimed at Windows. And even then it's better to use wrappers; for instance, Mac OS X has some conversion logic for its directory listing. When using the Win32 API manually, always use the Unicode version (the one ending with 'W' instead of 'A'). If you need to convert to UTF-8 for use in LibVLC, you can use FromWide (Win32 only!). Of course, it's better to resort to the wrappers otherwise you always need a specific Windows implementations.
Please refer to the various pieces of code using Win32 SHFolderGetPath() for reference implementations.

Now for the good (?) news: in trunk, the only parts that still use the <code>*Locale</code> LibVLC APIs are:
* ncurses (if you are brave, you can try porting to wide ncurses...),
* GnomeVFS (pretty much dead and not aimed at Win32 anyway),
* GnuTLS (no support from GnuTLS library),
* skins2,
* WxWidgets (only on non-Win32 and being phased out)

I have however only checked the core and plugins, so there might be problems with browser plugins and bindings. Also, if you intend to write new plugins, please keep this in mind.

==Automatic charset detection==
In addition to this, I remind you that we now have the GetFallbackEncoding() API that returns the ANSI code page used by Windows for the given locale, to be passed to iconv. For instance, if your system is configured to run in German, it returns "CP1252". This is convenient when handling subtitles or various kind of texts that are typically aimed at Windows users.

Adapted from a [[vlc-devel]] post by [[User:Courmisch|Courmisch]] (december 2006).

← Older revision		Revision as of 13:02, 16 May 2009
Line 1:		Line 1:
	LibVLC was switched to UTF-8 internally sometime in 2005, and it's now getting kind of ~~useable~~ again. Still, there's been quite many changes throughout 2006, so let's ~~make a summary~~.		LibVLC was switched to UTF-8 internally sometime in 2005, and it's now getting kind of usable again. Still, there's been quite many changes throughout 2006, so let's summarise.

	==Overview==		==Overview==
	As a general rules, all character strings passed to LibVLC should be in UTF-8. This includes but is far from limited to the playlist, and all the ''chain'' from the input/access to the outputs. We pretty much needed Unicode anyway; the only options were UTF-8 and wide char. We picked UTF-8 because, as disruptive as it has been, it remains much less disruptive than wide characters. Also wide characters means different things on different platforms (e.g. UCS2LE on old Windows, UTF-16LE on		As a general rules, all character strings passed to LibVLC should be in UTF-8. This includes, but is far from limited to the playlist, and all the ''chain'' from the input/access to the outputs. We pretty much needed Unicode anyway; the only options were UTF-8 and wide char. We picked UTF-8 because, as disruptive as it has been, it remains much less disruptive than wide characters. Also wide characters means different things on different platforms (e.g. UCS2LE on old Windows, UTF-16LE on
	modern Windows, UTF-32 on glibc), are a pain to use in POSIX environment (including Mac OS X and Linux which most of the devs are using), and might have needed lots of libc replacements <code>wc*</code> functions on some platforms.		modern Windows, UTF-32 on glibc), are a pain to use in POSIX environment (including Mac OS X and Linux which most of the devs are using), and might have needed lots of libc replacements <code>wc*</code> functions on some platforms.

Line 9:		Line 9:

	==Current approach==		==Current approach==
	So we now have a comprehensive (but not fully yet) set of higher-level wrappers that accept UTF-8 character string to represent filenames, or text to be ~~outputted~~ to the console. At the time of writing, trunk includes:		So we now have a comprehensive (but not fully yet) set of higher-level wrappers that accept UTF-8 character string to represent filenames, or text to be output to the console. At the time of writing, trunk includes:

	* utf8_open, utf8_fopen, utf8_stat, utf8_lstat, utf8_mkdir, utf8_opendir, utf8_readdir, utf8_scandir for file system operation,		* utf8_open, utf8_fopen, utf8_stat, utf8_lstat, utf8_mkdir, utf8_opendir, utf8_readdir, utf8_scandir for file system operation,

	* utf8_fprintf, utf8_vfprintf for console output (or actually output in any file ~~who~~ needs local charset).		* utf8_fprintf, utf8_vfprintf for console output (or actually output in any file which needs local charset).

	All of these functions behave like their non "utf8_"-prefixed equivalent with the ~~exceptions~~ that they expect UTF-8 character strings.		All of these functions behave like their non "utf8_"-prefixed equivalent with the exception that they expect UTF-8 character strings.

	We also have EnsureUTF8 that ~~force~~ a string into UTF-8 (replaces invalid character sequences with '?') and IsUTF8 that checks if the string is valid UTF-8 (returns NULL if not). These are useful when reading UTF-8 from untrusted sources, such as the network, before injecting the data into VLC.		We also have EnsureUTF8 that forces a string into UTF-8 (replaces invalid character sequences with '?') and IsUTF8 that checks if the string is valid UTF-8 (returns NULL if not). These are useful when reading UTF-8 from untrusted sources, such as the network, before injecting the data into VLC.

	===Backward compatibility note===		===Backward compatibility note===
Line 29:		Line 29:
	* WxWidgets (only on non-Win32 and being phased out)		* WxWidgets (only on non-Win32 and being phased out)

	I have ~~however~~ only checked the core and plugins, so there might be problems with browser plugins and bindings. Also, if you intend to write new plugins, please keep this in mind.		However I have only checked the core and plugins, so there might be problems with browser plugins and bindings. Also, if you intend to write new plugins, please keep this in mind.

	==Automatic charset detection==		==Automatic charset detection==
Line 35:		Line 35:


	Adapted from a [[vlc-devel]] post by [[User:Courmisch\|Courmisch]] (~~december~~ 2006).		Adapted from a [[vlc-devel]] post by [[User:Courmisch\|Courmisch]] (December 2006).