Lnet FTP problem with large files in Win64

[cybersmyth]

I'm not totally sure what the problem is, but I think it's got something to do with Windows 64bit. I built a routine based on the lnet visual FTP example and it works great on my Linux machine (which is 32bit). I run it on my Windows machine and it compiles and runs fine until it suddenly doesn't. 

I tried a lot of stuff and finally figured out it's large files that are giving me the trouble. I don't know exactly how big they have to get before there is an issue, but it definitely has something to do with larger files. Small text and image files copy over fine.

I thought I had screwed something up so I tried the lnet visual example and ran into the same problem! So I'm totally lost, no clue and am only guessing it has to do with 32 vs 64 because I could swear I've gotten this routine to work on a 32bit system before, but not 100% sure. But I am 100% sure it works fine on Linux 32bit.

Here is the problem area:

Code: [Select]

...
var
  s: string;
  i: Integer;
  Buf: array[0..65535] of Byte;
begin
  if FTP.CurrentStatus = fsRetr then begin // getting file, save to file
    i := FTP.GetData(Buf, 65535);
    if i > 0 then begin
      if Length(CreateFilePath) > 0 then begin
        FFile := TFileStream.Create(CreateFilePath, fmCreate or fmOpenWrite);
        CreateFilePath := '';
      end;
      FFile.Write(Buf, i);
    end else begin
      // file download ended
...

The crash happens at FFile.Write(Buf, i); after it writes 70 bytes or so.

Thanks for any help.

[Almindor]

This is really odd, it doesn't seem connected to lNet itself since you get a crash a TFIleStream.Write..

Perhaps try to write a standalone application which just copies a file like this on given platform. Perhaps you actually hit a hard disk problem?

Is it reproducable? If the FTP server/file is accessible send me the URL/name and I could try here too (although I only have win32 XP)

Also what crash is it exactly? Can you try to compile FPC source with -gl and see if you get debug info inside the file stream.

[FabienWang]

Buf: array[0..65535] of Byte; << shouldn't it be 65534 ?

[motaz]

it is better to use SizeOf function:

Code: [Select]

   i := FTP.GetData(Buf, SizeOf(Buf));


[Almindor]

You're right that it's odd usage (yeah.. my fault) but this cannot cause such problems.

Buffer size is 65536 (0..65535) and max 65535 bytes is read in GetData.

FFile.Write gets always a positive number (if i > 0) but <= 65535 (GetData limit) so it should work fine.

There's a bit of a discrepancy in the code but there's no bounds problem as far as I can say.

You need to provide a backtrace otherwise we're fishing in the dark.

[cybersmyth]

Error exemption class 'External: SIGSEGV'

Running -gl just goes to the line where the FFile.Write(Buf, i); (line 348) Without the -gl I get some assembler data.

I did try a couple of FTP servers and lots of different files. It seems like anything bigger than a few K is trouble. While I was testing, I got this weird thing (screenshot attached). Hope that's helpful.

I will try to compile this at work today too and see if it's just something on this PC. All of my office PCs are 64bit Win7 too. So right now I have no way to test Win32 code.

I did try i := FTP.GetData(Buf, SizeOf(Buf)); and it does the same thing.

[FabienWang]

hummm i have a SIGSEGV with visual example project FTP (provided with lnet)

Lazarus 0.9.31 r29829 FPC 2.4.3 i386-win32-win32/win64
latest lnet 0.6.6

It also stops on line FFile.Write(Buf, i); with -gl option

[Almindor]

Thanks for testing.

Sigsegv happens on memory access out of bounds situations, so it might be something with our buffer then o.O.

One thing comes to mind, try changing FFile.Write(buf..) to FFile.Write(buf[0],...) and try again please. You can also do the same with GetData(buf, ..) change to GetData(buf[0]...).

I'll try this at home on win32 and Linux64 (I don't have win64, but perhaps I could get it running in virtualbox).

The only way to get even better backtrace would be for you to recompile FPC from source with -gl so we see exactly where it crashes inside that code. But that's a bit too much to ask at this point.

I'll try things tonight and will report here. You can create a bugreport for this so it doesn't fade away.

Thanks

[FabienWang]

Same here =(

[cybersmyth]

More weirdnesses... I am really getting the impression that it is something wrong with TFIleStream.Write. I am now working at a totally different PC at work, still running Win 7 64bit.

I have tested both binary and text files for download, in both cases larger files cause the error. Small image files download just find and have no errors. Small text files do the same. So this leads me to believe the problem is with all files and not just with binary transfer or anything like that.

There is something particularly odd happening with text files that I think might shed some light on what is happening. I tried to download a really large text file and all of the sudden, just before the crash, the text in the file started filling in the FTP remote file list box! But then the error that gets spit back is "Invalid date format" with a "do you wish to continue" message. I click yes and it then gives me the 'External: SIGSEGV' error. As binary files, it just goes straight to the 'External: SIGSEGV' error. This is making me thing maybe TFIleStream.Write is overwriting the memory instead of going to the hard drive maybe? Something like that.

Another interesting thing, the text files don't always terminate at the same place. I've tried several times on the same file and it always ends at a different spot.

[cybersmyth]

Submitted a bug report:

http://mantis.freepascal.org/view.php?id=18927

My first bug report, hope I did it right.

[Almindor]

Thanks for the bugrep, it's fine.

It's starting to look like a win32 specific TFileStream bug, but it could still be a lNet/example related memory corruption problem too.

I have managed to reproduce it on FPC ftp site downloading FPC pdf files.

It is indeed semi-random, but if I replace TFileStream with FileCreate/FileWrite/FileClose operations on a THandle, it works fine, data is consistent.

So it's either a memory error somewhere in our code, or a proper TFileStream/FPC bug. I'll investigate further tomorrow.

[cybersmyth]

Do you feel the FileCreate/FileWrite/FileClose operations are an effective work-around?

[Almindor]

You've found a nice bug here

It's a bug in the FTP code (both library and console and visual example), specifically the if i > 0 then .. else part (given i = # of bytes received) and FTP handling of GetData.

The thing is, lNet reports 0 bytes received also when a recv fails due to "cannot handle more receives now, wait a bit" (TCP level). This isn't a typical scenario and in fact, it doesn't happen on Linux (at least not for the same FTP example) probably because Linux allocates bigger receive buffers. However FTP should handle this and a new logic should be introduced for these kind of receives that do not mean "end of downloaded file".

A workaround is simple:

In the "else" part where the downloading is "finished" add a condition like this (code from visual example):

Code: [Select]

      FFile.Write(Buf, i);
    end else if FDLDone + i >= FDLSize then begin // done whole file
      // file download ended

This is not a fix however, just a workaround. I'll need to change the FTP library code (and examples too) for this logical path when recv itslef gives -1 (but lNet returns 0 because we just need to wait). There should be 2 "results" out of FTP.GetData. One should mean "data amount read (including 0)" and the other should mean "data download complete". This will require a change which could break old code tho, so it will probably go to 7.x. Until then I'll fix the examples to check for filesize (not exactly correct).

[cybersmyth]

You've found a nice bug here

Yah! I done a good thing.... I think....