How to stream.Read UTF16 files?

[paskal]

Here is a procedure:

Code: [Select]

function ReadFile(FileName: string): string;
var
  MyString: AnsiString;
  stream: TFileStream;
  StringLenth: Integer;
begin
    StringLenth := FileSize(FileName);
  stream := TFileStream.Create(UTF8ToSys(FileName), fmShareDenyNone );
  try
    stream.Seek(0, soFromBeginning);
    SetLength(MyString, StringLenth);
    stream.Read(MyString[1], StringLenth);
  finally
    stream.Free();
  end;
  form1.Memo1.lines.Text := ConvertEncoding (MyString,'ansi', 'cp1200');
end;
I try to read a file, saved as UTF16LE (codepage 1200);
I tried different combinations of ConvertEncoding but I got no result.
Memo1 displays only the firsts three chars of the file, the first two ones are shown as question marks (they are the BOM chars), the third one is shown as it should.
What is odd is that when I read a UTF8 file, it is displayed properly, despite ConvertEncoding (MyString,'ansi', 'cp1200');

[marcov]

utf16LE is a 2-bytes per array item type. Use a 2-byte stringtype like widestring or unicodestring.

Note that in that case you need to divide the filesize by two to get the value for setlength.

[theo]

You could try charencstreams.pas
http://wiki.freepascal.org/Theodp

[paskal]

charencstreams.pas seems to work file (at least reading, so far).
Does it guess the encoding for files without BOM, or it finds it some other way? With a very short string it gets a wrong encoding, but with a longer it is fine.

[theo]

Yes, it guesses as far as possible.
If you know the encoding, you can disable this behaviour.

fCES.ForceType:=true;

See also demo/charenc/

[paskal]

I have some problems with longer files (100kB+).
Meanwhile, I found something odd in the examples charenc in unit1.pas there is:



procedure TForm1.btnOpenClick(Sender: TObject);
begin
  if OpenDialog1.Execute then

  begin
    if not cbForceType.Checked then fCES.Reset;
    fCES.LoadFromFile(UTF8ToSys(OpenDialog1.FileName));
    Memo1.text := fCES.UTF8Text;
    cbBOM.Checked := fCES.HasBOM;
    cbForceType.Checked := fCES.ForceType;
    UniEnc.ItemIndex := Ord(fCES.UniStreamType);
    ANSIEnc.Enabled := fCES.UniStreamType = ufANSI;
    if ANSIEnc.Enabled then ANSIEnc.ItemIndex := ANSIEnc.Items.IndexOf(fCES.ANSIEnc) else ANSIEnc.ItemIndex := 0;
  end;
end;   

It works okay.
But if I make a small change:


procedure TForm1.btnOpenClick(Sender: TObject);
var
  mystring: string;
begin
  if OpenDialog1.Execute then

  begin
    if not cbForceType.Checked then fCES.Reset;
    fCES.LoadFromFile(UTF8ToSys(OpenDialog1.FileName));
    mystring:=fCES.UTF8Text;
    Memo1.text := fCES.UTF8Text;
    cbBOM.Checked := fCES.HasBOM;
    cbForceType.Checked := fCES.ForceType;
    UniEnc.ItemIndex := Ord(fCES.UniStreamType);
    ANSIEnc.Enabled := fCES.UniStreamType = ufANSI;
    if ANSIEnc.Enabled then ANSIEnc.ItemIndex := ANSIEnc.Items.IndexOf(fCES.ANSIEnc) else ANSIEnc.ItemIndex := 0;
  end;
end;   


then a wrong result is displayed. Probably each time the function is called, a different result occurs. Should it be this way?

Also, I managed to crash the example.
Here is how it happens:
1. I open a file (UTF16BE)
2. I open another file (UTF8)
3. I open the first file again -> Exception class external....

Opening the first file many time makes no problem.
Then I tried something else:
1. I open a file (UTF16BE)
1. I open a file (UTF16BE again)->  Exception class external....

By the way, tere is a bug in the sample code in http://wiki.freepascal.org/Theodp
fCES.LoadFromFile(OpenDialog1.FileName); shall be fCES.LoadFromFile( UTF8ToSys( OpenDialog1.FileName)); because currently it can open only files with basic Latin names.

[paskal]

Can anyone else reproduce the problem?
I have attached a ZIP containing two files- one of them is in UTF8 and the other one- in UTF16BE (as indicated in the filenames) encoding.
With the example application from UTF8Tools/Demo/charenc/ when the UTF8 file (TestFileUTF8.txt )is open and right after it- the UTF16BE file (TestFileUTF16BE.txt) is open, the application crashes.
I have tried it on two different computers with WinXP, probably with slightly different Lazarus versions.
The last test is done on Lazarus #1.1; Date: 2012-05-29; FPC 2.6.1; Svn 37447.

[KpjComp]

charencstreams calls a procedure called, WideSwapEndian,  This assumes the text is null terminated.

WideSwapEndian ideally needs another parameter for maximum length.

For now though you could just fake it by adding a Null.

eg. In unit1 of the charenc demo project change the btnOpenClick event to->

Code: [Select]

procedure TForm1.btnOpenClick(Sender: TObject);
var
  nullword:word = 0;
begin
  if OpenDialog1.Execute then
  begin
    if not cbForceType.Checked then fCES.Reset;
    fCES.LoadFromFile(OpenDialog1.FileName);
    fCES.Seek(0,soEnd);
    fCES.WriteBuffer(Nullword,sizeof(nullword));
    Memo1.text := fCES.UTF8Text;
    cbBOM.Checked := fCES.HasBOM;
    cbForceType.Checked := fCES.ForceType;
    UniEnc.ItemIndex := Ord(fCES.UniStreamType);
    ANSIEnc.Enabled := fCES.UniStreamType = ufANSI;
    if ANSIEnc.Enabled then ANSIEnc.ItemIndex := ANSIEnc.Items.IndexOf(fCES.ANSIEnc) else ANSIEnc.ItemIndex := 0;
  end;             


[paskal]

Did you try if this solution works for you?

So what you do is add two lines:

Code: [Select]

    fCES.Seek(0,soEnd);
    fCES.WriteBuffer(Nullword,sizeof(nullword));
So, firsts Lazarus did not find NullStr so I replaced it with NullStr.
After running the app, it behaves the same way

[KpjComp]

It's not NullStr,  it's NullWord.
And it's defined at the beginning of the procedure I posted.

Code: [Select]

var
  nullword:word = 0;

IOW: I'm adding two extra 0 bytes to the end of the stream to prevent the WideSwapEndian from overflowing.

Did you try if this solution works for you?

Yes..

[paskal]

It's not NullStr,  it's NullWord.
And it's defined at the beginning of the procedure I posted.

Code: [Select]

var
  nullword:word = 0;


Ooops, I did not see it.

IOW: I'm adding two extra 0 bytes to the end of the stream to prevent the WideSwapEndian from overflowing.

Do you mean that I should either

Code: [Select]

    fCES.WriteBuffer(NullWord,sizeof(NullWord));
    fCES.WriteBuffer(NullWord,sizeof(NullWord));
or

Code: [Select]

var
  nullword:Dword = 0;


Did you try if this solution works for you?

Yes..

Now it works for me, too. Thanks!

Anyway, wont' this Nullwords result in an extra string, which I should remove?

[KpjComp]

Do you mean that I should either

As well as the var, you just need ->

Code: [Select]

    fCES.Seek(0,soEnd);
    fCES.WriteBuffer(Nullword,sizeof(nullword));
You don't need two, as NullWord = 2 bytes.  And we only need two bytes to prevent WideSwapEndian from overflowing.  But it won't harm using Dword = 4 bytes, you never know your UTF16 file might be corrupt and have a none Even size.

For the long term, ideally WideSwapEndian wants fixing, as adding 0 bytes to the end of the memory stream is a bit of a hack.

[KpjComp]

Anyway, wont' this Nullwords result in an extra string, which I should remove?

Possibly, that's why it's a bit of a hack.

For a proper fix try this in charencstreams ->
Change WideSwapEndian to ->

Code: [Select]

procedure WideSwapEndian(PWC: PWideChar;size:integer);
begin
  while size >= sizeof(widechar) do
  begin
    PWC^ := WideChar(SwapEndian(Word(PWC^)));
    inc(PWC);
    dec(size,sizeof(widechar));
  end;
end;

And the two places this is called pass the Size.
eg.

Code: [Select]

function TUniStream.GetUTF8Text: AnsiString;
........
        if fUniStreamType = ufUtf16be then WideSwapEndian(PWC,size);
and

Code: [Select]

procedure TUniStream.SetUTF8Text(AString: AnsiString);   
.......
          if fUniStreamType = ufUtf16be then WideSwapEndian(@WS[1],size);


[paskal]

Okay, it seems to work fine now.
There is something more unclear about this package- is it appropriate for using in GPL and LGPL applications?

[KpjComp]

including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense,

The term sublicense to me would indicate you can choose.