Tutorial about character handling in Ada

This is unfinished!!!

Character sets

"A character set" means the groups of characters not representation.

Encodings

"A Encoding" means the representation of "A character set".
A variable in programing languages can hold an encoded string.

Ada types

Writing literals in Ada

ASCII character

   ASCII_Character_1 : constant Character := 'A';
   ASCII_Character_2 : constant Character := ASCII.HT;

Latin-1 character

   Latin_1_Character_1 : constant Character := 'A';
   Latin_1_Character_2 : constant Character := Ada.Characters.Latin_1.UC_A_Grave;

Using GNAT

Save your souce code as Latin-1.

   pragma Wide_Character_Encoding (UPPER);
   Latin_1_Character_3 : constant Character := 'À';

Save your souce code as UTF-8.

   pragma Wide_Character_Encoding (UTF8);
   Latin_1_Character_3 : constant Character := 'À';

UCS-2 (encoding) or UTF-16 character

   UTF_16_Character_1 : constant Wide_Character := 'A';

Using GNAT

Save your souce code as UTF-8.

   pragma Wide_Character_Encoding (UTF8);
   UTF_16_Character_1 : constant Wide_Character := 'Ǡ';

UTF-32 character

   UTF_32_Character_1 : constant Wide_Wide_Character := 'A';

Using GNAT

Save your souce code as UTF-8.

   pragma Wide_Character_Encoding (UTF8);
   UTF_32_Character_1 : constant Wide_Wide_Character := 'Ǡ';

Shift-JIS/EUC-JP character

Ada does not support these encodings in the standard.

Note about JIS code literals in GNAT

Do NOT use pragma Wide_Character_Encoding (SHIFT_JIS); or pragma Wide_Character_Encoding (EUC);.
These options are useless because string literals with these pragmas are encoded to Wide_String contains raw values of JIS code. But raw values of JIS code is not used anywhere in fact. For example, A-version of Windows API requires Shift-JIS, not raw values of JIS code. Mail applications requires ISO-2022-JP (that is another encoding), not raw values of JIS code.
So you have to save source code as UTF-8 and convert strings at run-time. (Or save source code just as Shift-JIS or EUC-JP and write these encoded literals into String directly without pragma.)

Save your source code as UTF-8.

   pragma Wide_Character_Encoding (UTF8);
   UTF_16_Character_1 : constant Wide_Character := 'あ';

And convert it at runtime with libiconv, wcstombs or WideCharToMultiByte API.

(Or save your source code as Shift-JIS.

   Shift_JIS_Character_1 : constant String := "あ";

Please attention, the standard library (or GNAT runtime) can not handle multi-byte encoded String like Shift-JIS/EUC-JP(or UTF-8). You have to write almost string operations yourself.)

Handling real letters of Unicode

Probably, you may be confused in this section.
You shold know one code-point of Unicode is not one real letter for display.
Plural code-points represents one real letter in some cases.
(In this section, "one real letter" means one glyph that you are looking now.)

Composed character

TBD...

Variation selector

TBD...

One real letter

You shold forget considering one Wide_Wide_Character as "one character".
One real letter is composed:

partone code-point having combining class 0(optional) multi code-points having combining class >= 1(optional) one variation selector
example

Perhaps you may think to convert Unicode string by NFC to remove composed characters. But NFC does not help you about this. Because NFC characters are not defined every NFD characters (One of reasons: composable code-points having combining class >= 1 are able to be connected infinity). So some composed characters may be left after NFC.
And, NFC replaces some compatibility characters. It's bad behavior for some languages. You shold not do NFC without much thought.

Iterate each real letter of Wide_Wide_String

TBD...

Iterate each real letter of String containing UTF-8

TBD...

Handling local encoding

On UNIX

In this document, "local encoding" means an encoding depending on user's setting of the operating system.
It's also used in C runtime.
Therefore the method of handling local encoding is same as C language.
At first, set the locale of C runtime to user's setting to use other functions in C runtime.

declare
   LC_ALL : constant Interfaces.C.int := 0;
   function setlocale (
      category : Interfaces.C.int;
      locale : access constant Interfaces.C.char)
      return access constant Interfaces.C.char;
   pragma Import (C, setlocale);
   Empty : aliased Interfaces.C.char_array := (0 => Interfaces.C.nul);
   Previous_Locale : access constant Interfaces.C.char;
begin
   Previous_Locale := setlocale (LC_ALL, Empty (0)'Access);
end;

This is same as C code in below:

    char const *previous_locale = setlocale (LC_ALL, "");

Iterate each multi-byte characters of Interfaces.C.char_array containing local encoding

Import mblen to get length of one multi-byte character.

   function mblen (
      s : access constant C.char;
      n : Interfaces.C.size_t;
      return Interfaces.C.int;
   pragma Import (C, mblen);

Then, iterate.

declare
   Text : Interfaces.C.char_array := ...string containing local encoding...;
   I : Interfaces.C.size_t := Text'First;
begin
   while I <= Text'Last loop
      declare
         Length : Interfaces.C.size_t := Interfaces.C.size_t (
            mblen (Text (I)'Access, Text'Last - I + 1));
         One_Multi_Byte_Character : Interfaces.C.char_array
            renames Text (I .. I + Length - 1);
      begin
         ...
         ... -- use One_Multi_Byte_Character
         ...
         I := I + Length;
      end;
   end loop;
end;

convert between local encoding and string types of Ada

TBD...

On Windows

Iterate each multi-byte characters of Interfaces.C.char_array containing local encoding

Import IsDBCSLeadByte to get length of one multi-byte character.

   function IsDBCSLeadByte (
      TestChar : C.char) -- BYTE
      return Interfaces.C.int; -- BOOL
   pragma Import (C, IsDBCSLeadByte);

Then, iterate.

declare
   Text : Interfaces.C.char_array := ...string containing local encoding...;
   I : Interfaces.C.size_t := Text'First;
begin
   while I <= Text'Last loop
      declare
         Length : Interfaces.C.size_t :=
            Boolean'Pos (IsDBCSLeadByte (Text (I)) /= 0) + 1; -- 1 or 2
         One_Multi_Byte_Character : Interfaces.C.char_array
            renames Text (I .. I + Length - 1);
      begin
         ...
         ... -- use One_Multi_Byte_Character
         ...
         I := I + Length;
      end;
   end loop;
end;

convert between local encoding and string types of Ada

TBD...

Ada standard libraries

Ada.Command_Line/Ada.Environment_Varaibles

Ada.Command_Line.Argument and Ada.Environment_Varaibles.Value are defined as String.
It means that we can not get characters not in Latin-1 (character set) of command line and environment variables.

But, using GNAT

But GNAT runtime does not convert command-line and environment variables from local encoding to Latin-1.
(This behavior seems bug, but useful.)
So we can get raw command-line and environment variables with Ada.Command_Line/Ada.Environment_Varaibles.
And, it's necessary to convert from local encoding to UTF-32 (Wide_Wide_String) or any encoding what you need.

Ada.Text_IO

Name parameter is String that holds Latin-1 (encoding).
But you can use UTF-8 file name by Form parameter.

   Open (
      File,
      Name => Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode (Wide_Wide_File_Name),
      Form => implementation-defined);

Refer http://groups.google.com/group/comp.lang.ada/msg/038c559fd843a19f?hl=en.

Using GNAT

Use "ENCODING=UTF8".

   Open (
      File,
      Name => Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode (Wide_Wide_File_Name),
      Form => "ENCODING=UTF8");

Refer http://gcc.gnu.org/onlinedocs/gnat_rm/FORM-Strings.html.

Ada.Directories

It's same as Text_IO. If function has Form parameter, you can use UTF-8 file name.
But almost functions does not have Form parameter to regret.
It's unclear which are Form parameters added or not in the future. But we have to use functions in C runtime currently. Do not forget the encoding of C runtime is differ from Ada.

Name_Case_Equivalence

Ada.Directories.Name_Case_Equivalence are defined in Ada 2012.
Refer http://ada-auth.org/standards/12rm/html/RM-A-16.html.

But, Actually this function can not help you.
Because it may return Case_Preserving on NTFS (file system of Windows). It's case-insensitive for accessing an existing file but keeping cases of each letter. And, it may return Case_Preserving on HFS+ (file system of Mac OS X). It's case-insensitive for accessing an existing file but keeping cases of each letter. Then, are behaviors of these two file systems same?
No. Case-insensitive rules of these file systems are different. (See below.)
So you have to write function to compare two file names for each file systems.

Ada.Characters/Ada.Strings

Almost subprograms of Ada.Characters/Ada.Strings are defined to work with Latin-1 (encoding).
Do not use these with UTF-8 or local encoding.

Interfaces.C

You who read so far are may awake to bad definition of To_Ada/To_C for char/wchar_t in Interfaces.C.
char of C is local encoding, wchar_t of C is wide version of local encoding. And, wchar_t'Size is able to be 16 (on Windows) or 32 (on UNIX).
Character of Ada is Latin-1 (encoding), Wide_Character of Ada is UCS-2 (encoding).
But in the standard, function To_Ada (Item : wchar_t) return Wide_Character; is defined as Wide_Character'Value (wchar_t'Image (Item)). These functions will raise Constraint_Error instead of encoding, if multi-byte encoding is necessary.
You shold use functions in C runtime to convert between Latin-1/UCS-2/UTF-32 and local encoding instead of To_Ada/To_C for char/wchar_t.

char16_t and char32_t are right.

wchar_t of GNAT

Interfaces.C.wchar_t of GNAT's implementation is more bad.

   type wchar_t is new Wide_Character;
   for wchar_t'Size use Standard'Wchar_T_Size;

On UNIX platforms, it can not hold Unicode characters over 16-bit. Constraint_Error will be raised.

For the real world applications

BSD

BSD is operating system(s) of UNIX.
It's unique about I18N.
wchar_t of BSD is not Unicode.
Functions like wcstombs for wide character set in C runtime of BSD work as character-set indepent.
What is wchar_t in BSD? It's something that holds packed one character that's wide version of local encoding.
You have to use libiconv to convert between Unicode and local encoding.

HFS+

HFS+ is the file system of Mac OSX.
It's unique about normalization/case insensitive of file names.
16#2000# .. 16#2FFF#, 16#F900# .. 16#FAFF#, and 16#2F800# .. 16#2FAFF# are not replaced on the normalization of HFS+.
This variation of normalization is also resolving a kind of problem of normalizations defined in Unicode standard, because compatibility characters are removed from targets of the normalization.
Refer http://developer.apple.com/library/mac/#qa/qa1173/_index.html and http://search.cpan.org/dist/Encode-UTF8Mac/lib/Unicode/Normalize/Mac.pm.
The case insensitive rule of HFS+ is differ from Unicode standard, too.
Refer http://www.opensource.apple.com/source/boot/boot-132/i386/libsaio/hfs_CaseTables.h.

NTFS

NTFS is the file system of Windows.
File names in NTFS are stored as UTF-16.
So a file name having illegal UTF-16 sequence can be existing.
If you want to handle all file names of NTFS, you shold keep file names as Wide_String, not convert to UTF-8/UTF-32.
And, You can use CompareString(W) API for case insensitive of NTFS.

Windows console

Windows console (cmd.exe) can not display UTF-8.
You have to convert to local encoding, if necessary.

Cygwin is an exception. Cygwin console can display UTF-8.

Ada.Wide_Text_IO of GNAT

GNAT's implementation of Wide_Text_IO outputs UTF-8 encoded from UCS-2 in Wide_String that given to Put/Put_Line, irrespective of Windows.
As a result, the output will be garbled to read.
I think it's bug of GNAT.