# Natural and UTF-8 **URL:** \ **Category:** Forum\ **Tags:** Adabas-Natural, Natural, Natural-on-Linux\ **Created:** [May 24, 2006, 4:07pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038 "2006-05-24T16:07:50Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Matthias1](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@Matthias1](https://techcommunity.softwareag.com/u/Matthias1)\ **Post date:** [May 24, 2006, 4:07pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/1 "2006-05-24T16:07:50Z")
I think, from ADABAS point of view, it’s irrelevant if the characters are stored in ISO-8859 or UTF-8 format. But what about Natural? How would Natural display an UTF-8 encoded umlaut or other special characters? As a first test, I write the following simple program: ```auto write 'UTF-8 encoded umlaut u is:' H'C384' / 'UTF-8 encoded ligature sz is:' H'C39F' / 'UTF-8 encoded euro symbol is:' H'E282AC' end ``` This works quite good if I set my terminal-emulation to UTF-8 first! But at the end of the line, some characters from the previous screen are still displayed. The next problem comes with the maps. For example: If I write a map with the EURO-Symbol in UTF-8 on it, it would be written into the map-source with a length of 3 byte, but it would be displayed with a character-width of one. This causes problems in the map-editor. For example it’s impossible to fill a map with a width of 79 with 79 characters. The next thing is: It’s almost impossible to edit such a text-constant. Here’s my test-map: [code] - MAP2: PROTOTYPE — CREATED BY UNIX 6.1.1 — - INPUT USING MAP ‘XXXXXXXX’ FORMAT PS=003 LS=080 ZP=OFF SG=OFF KD=OFF IP=OFF - MAP2: MAP PROFILES \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* 200\*\*\*\*\*\*\*\*\*\*\* - .TTAAAMMOO D I D I N D I D I ?\_)^&:+( \* - 003079 N0NNUCN X 01 SYSDBA NR \* * * * INPUT ( IP=OFF /\* ) 001T ‘----±—1----±—2----±—3----±—4----±—5----±–’- ‘-6----±—7----±—’ / 001T 'Here comes the EUR-symbol: ---
**Author:** ![system](https://global.discourse-cdn.com/techcommunity/original/3X/c/d/cd19967bb605d28a72d77eb83a1cae22f3df2eb8.svg) [@system](https://techcommunity.softwareag.com/u/system)\ **Post date:** [May 24, 2006, 8:54pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/2 "2006-05-24T20:54:15Z")
Hi Matthias, There were a couple of articles on Natural and UNICODE in the last newsletters (and there is one more coming up in June), they are archived at [url][http://developer.softwareag.com/ets/knowhow/Nat\_knowhow.htm[/url]](http://developer.softwareag.com/ets/knowhow/Nat_knowhow.htm%5B/url%5D). Maybe you can get some information from them. Steve ---
**Author:** ![Matthias1](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@Matthias1](https://techcommunity.softwareag.com/u/Matthias1)\ **Post date:** [May 29, 2006, 2:59pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/3 "2006-05-29T14:59:54Z")
Hello Steven Wild! Thanks for the link to the Unicode-documents. Here are some excerpts: > [@Article #1](#): > > Unicode is now being implemented in Natural with Natural Version 4.2 (Mainframe) and 6.2 (Open Systems). At the moment I’m working with Natural 6.1.1 for Solaris. It seems, that I have to wait a little bit … > [@Article #1](#): > > In its basic version, close to 65,000 characters are available; more than enough for standard global business applications. > […] > Natural handles all Unicode data in the UFT16 form, which is the original 2-byte value of the character. > […] > For Unicode fields their functions are changed to base all processing on a character value, i.e. 2 bytes internally. If I understand this correctly, Natural uses UTF-16 and all related statements (like EXAMINE, MOVE SUBSTR) are adapted to handle 2 bytes per character. But UTF-16 does not mean, that every character can be represented by 2 bytes. Characters above U+FFFF are represented by 4 bytes (surrogate pair). See: [UTF-16 - Wikipedia](http://en.wikipedia.org/wiki/UTF-16/UCS-2#Examples) So my question is: Will Natural 6.2 (Open Systems) support UTF-16 with or without surrogates? > [@Article #2](#): > > To store Unicode in a database, the data definition module has to be defined with “W” type fields in Adabas, or as Unicode field in DB2. No further preparation needs to be done. Natural automatically transfers the Unicode data correctly to the database in UTF16 format. In Adabas the data will be converted to the UTF8 format for compression reasons. So in a typical Western Latin-oriented application the amount of data is only slightly larger than the original data stored in a local code page. That’s the reason why UTF-8 became the quasi-standard for Unicode representation. Next question: Does Natural 6.2 only support UTF-16? This could be a problem, because most of XML-Documents I’ve seen until now are using UTF-8. Even my PuTTY-terminal-Emulation can only handle UTF-8. OK, it would be possible to write a converter for the XML-Files (but I will not do this in Natural)… ---
**Author:** ![Peter\_Dowson](https://avatars.discourse-cdn.com/v4/letter/p/eb9ed0/32.png) [@Peter\_Dowson](https://techcommunity.softwareag.com/u/Peter_Dowson)\ **Post date:** [May 31, 2006, 7:06pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/4 "2006-05-31T19:06:15Z")
Just for your information : Natural 6.2 is available on Solaris already, Peter ---
**Author:** ![Matthias1](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@Matthias1](https://techcommunity.softwareag.com/u/Matthias1)\ **Post date:** [June 1, 2006, 12:43pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/5 "2006-06-01T12:43:43Z")
> [@Peter Dowson](#): > > Natural 6.2 is available on Solaris already, Yes, I know. But I’m not the one, who does the installations here. So I have to wait … 😉 ---
**Author:** ![Arthur\_Dent](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Arthur\_Dent](https://techcommunity.softwareag.com/u/Arthur_Dent)\ **Post date:** [June 1, 2006, 5:29pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/6 "2006-06-01T17:29:14Z")
Matthias wrote: > Next question: Does Natural 6.2 only support UTF-16? > \< Natural introduces with Nat 4.2 and Nat 6.2 a new format ‘U’. Natural Unix, Win handles internally this format in UTF-16. But just for ‘U’ variables. All other variables are treated as before. From my point of view it would be the best to read first the documentation, because there are many questions answered. ---
**Author:** ![Matthias1](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@Matthias1](https://techcommunity.softwareag.com/u/Matthias1)\ **Post date:** [June 1, 2006, 6:21pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/7 "2006-06-01T18:21:27Z")
> [@Arthur Dent](#): > > Natural introduces with Nat 4.2 and Nat 6.2 a new format ‘U’. Natural Unix, Win handles internally this format in UTF-16. But just for ‘U’ variables. All other variables are treated as before. > > From my point of view it would be the best to read first the documentation, because there are many questions answered. First of all: I don’t have any documentation of Nat 6.2. I only got the links to the documents that Stephen Wild mentioned above. There I read about the new U-Format in Natural (which is UTF-16) and the corresponding new W-Format in ADABAS (which is UTF-8 ). The conversion between Natural and Adabas is done automatically. From my point of view UTF-8 is the quasi-standard. And my question still remains: Does Natural 6.2 only support UTF-16? In other words: What about Workfiles and XML-docs with UTF-8 encoding? ---
**Author:** ![anon29934956](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@anon29934956](https://techcommunity.softwareag.com/u/anon29934956)\ **Post date:** [June 2, 2006, 3:25pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/8 "2006-06-02T15:25:42Z")
Hello Matthias, Natural internally stores Unicode characters in UTF16 format. This allows quick scanning in a Unicode string as each character starts at a fixed location. The MOVE statement was enhanced to allow conversion from one code page to another. Please refer to ```auto MOVE ENCODED ``` You may specify UTF8 als source and UTF16 as target and vice versa, allowing the conversion. Surrogate characters in UTF16 require 4 bytes of storage and must be handled in pairs of U-characters. The EXAMINE statement has been enhanced to detect surrogates. ```auto EXAMINE [FULL [VALUE [OF]]] {op1 | SUBSTR(op1,op2,op3)} [POSITION-clause] [FOR] [CHARPOSITION op4] [CHARLENGTH op5] [[GIVING] POSITION IN op6] [[GIVING] LENGTH IN op7] ``` With best regards ---
**Author:** ![Matthias1](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@Matthias1](https://techcommunity.softwareag.com/u/Matthias1)\ **Post date:** [June 2, 2006, 5:15pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/9 "2006-06-02T17:15:55Z")
> [@Michael Muenster](#): > > Please refer to > > ```auto > MOVE ENCODED > ``` > > You may specify UTF8 als source and UTF16 as target and vice versa, allowing the conversion. Thanks for answering! So Natural 6.2 supports UTF-8 … [http://techcommunity.softwareag.com/ecosystem/documentation/natural/nat621win/sm/move\_0860.htm#syntax8\_move](http://techcommunity.softwareag.com/ecosystem/documentation/natural/nat621win/sm/move_0860.htm#syntax8_move) [http://techcommunity.softwareag.com/ecosystem/documentation/natural/nat621win/unicode/uni-language.htm](http://techcommunity.softwareag.com/ecosystem/documentation/natural/nat621win/unicode/uni-language.htm) BTW, I have to memorize the following: > [@Logical Condition Criteria](#): > > If the operands are not all of U format, the second operand is converted to the format of the first operand. > […] > Note: > Comparing U and A format can deliver different results, depending on the sequence of the fields. Phew! There a programer can make really bad mistakes. ---
**Author:** ![system](https://global.discourse-cdn.com/techcommunity/original/3X/c/d/cd19967bb605d28a72d77eb83a1cae22f3df2eb8.svg) [@system](https://techcommunity.softwareag.com/u/system)\ **Post date:** [April 2, 2021, 2:40pm UTC](https://techcommunity.softwareag.com/t/natural-and-utf-8/61038/10 "2021-04-02T14:40:40Z")