FTP Upload Issue

[Zipur]

I created a small FTP client that works fine but if you select a large file, it crashes. The sizes of the files I transfer varies on a daily basis, but I need to do the large ones without any issues.

I transferred a 150 meg file with no issues but it crashed on the 5 GB file. Does anyone have any suggestions?

Thanks

Here is the code of the FTP that I have added to the program.

expandcollapse popup

#include <FTP_Ex.au3>


            
            
            
                Case @Error = 0 ;OK - The string returned is valid
                    $message = "Please select a file you wish to ftp."
                    $var = FileOpenDialog($message, @WindowsDir & "\", "Compression Files (*.rar;*.zip)|All Files (*.*)", 1 + 4 )
                    If @error Then
                        MsgBox(4096,"","No File(s) chosen")
                    Else
                        $var = StringReplace($var, "|", @CRLF)
                        $sFile = StringTrimLeft($var, StringInStr($var, "\", 0, -1))
                        $sDir = StringTrimRight($var, (StringLen($var) - StringInStr($var, "\", 0, -1)))
                        $server = 'fileserver'
                        $username = 'Username'
                        $pass = 'Password'
                        $FTPport = 21
                        
                        $Open = _FTP_Open('MyFTP Control')
                        $Conn = _FTP_Connect($Open, $server, $username, $pass, 1, 21)
                        if @error then
                            msgbox(0,"Error","Connect Error")
                        Exit
                        EndIf
                                
                        $Ftpd = _FTP_DirCreate($Conn, $AScallno)
                        $FTPdir = _FTP_DirSetCurrent ($Conn, $AScallno)
                        $Ftpp = _FTP_ProgressUpload($Conn, $var,$sFile)
                        if $Ftpp Then
                                MsgBox(0, "OK", "File successfully uploaded!")
                                GUICtrlSetData($progress, 0)
                        Else ;[if @error = -6 then button stop pressed]
                                MsgBox(0, "ERROR", "Error during upload (error code: " & @error & ").")
                                GUICtrlSetData($progress, 0)
                        EndIf
                        $Ftpc = _FTP_Close($Open)
                        
                    EndIf
                Case @Error = 1 ;The Cancel button was pushed

                Case @Error = 3 ;The InputBox failed to open

                EndSelect

Edited April 25, 2010 by Zipur

[Zipur]

Nobody has encountered this or has any suggestions? I need to upload the 5 GB files just as much as the smaller ones. What about a dos command line ftp program I can run?

[MrMitchell]

The client side is Windows XP?

Will another FTP client pull back the file without any issues?

Is there a specific or at least general max file size the transfer quits at or is it completely random?

[jchd]

Which AutoIt version are you running? Have you tried the latest release (3.3.6.1)?

Also, what do you mean by "it crashes"? Do you get error?

Edited April 27, 2010 by jchd

[Zipur]

I will try 3.3.6.1. I was using 6.0. If I install filezilla or cuteftp, I can upload with no issues using that software. It just seems like the little util I wrote won't upload files over a certain size. I work in Phone support and some of our customers do not want the third party software installed, but trust the software we provide.

Edited April 27, 2010 by Zipur

[jchd]

OK I see. Don't waste your time testing gigabyte files with 3.3.6.1 as I believe it behaves the same as previous 3.3.6 (but it won't hurt to update anyway!).

I seem to recal there was an issue with large filesize and Inetget of something, but searching proved it was (only) InetFileSize limited to 32-bit return value (ticket #1458).

And do you get error value, or does AutoIt actually crash (Windows exception)?

Edited April 27, 2010 by jchd

[Zipur]

My ftp code is in my original post, but I will post it again. It just "freezes" the ftp progress after it gets to 1%, but the original util still runs fine. You can exit or call other options from the menu. I let it run over night and it did not make it past the 1%

expandcollapse popup

#include <FTP_Ex.au3>


            
            
            
                Case @Error = 0 ;OK - The string returned is valid
                    $message = "Please select a file you wish to ftp."
                    $var = FileOpenDialog($message, @WindowsDir & "\", "Compression Files (*.rar;*.zip)|All Files (*.*)", 1 + 4 )
                    If @error Then
                        MsgBox(4096,"","No File(s) chosen")
                    Else
                        $var = StringReplace($var, "|", @CRLF)
                        $sFile = StringTrimLeft($var, StringInStr($var, "\", 0, -1))
                        $sDir = StringTrimRight($var, (StringLen($var) - StringInStr($var, "\", 0, -1)))
                        $server = 'fileserver'
                        $username = 'Username'
                        $pass = 'Password'
                        $FTPport = 21
                        
                        $Open = _FTP_Open('MyFTP Control')
                        $Conn = _FTP_Connect($Open, $server, $username, $pass, 1, 21)
                        if @error then
                            msgbox(0,"Error","Connect Error")
                        Exit
                        EndIf
                                
                        $Ftpd = _FTP_DirCreate($Conn, $AScallno)
                        $FTPdir = _FTP_DirSetCurrent ($Conn, $AScallno)
                        $Ftpp = _FTP_ProgressUpload($Conn, $var,$sFile)
                        if $Ftpp Then
                                MsgBox(0, "OK", "File successfully uploaded!")
                                GUICtrlSetData($progress, 0)
                        Else ;[if @error = -6 then button stop pressed]
                                MsgBox(0, "ERROR", "Error during upload (error code: " & @error & ").")
                                GUICtrlSetData($progress, 0)
                        EndIf
                        $Ftpc = _FTP_Close($Open)
                        
                    EndIf
                Case @Error = 1 ;The Cancel button was pushed

                Case @Error = 3 ;The InputBox failed to open

                EndSelect

Edited April 27, 2010 by Zipur

[jchd]

I'm looking in the UDF and I believe there is suspuscious code there in case of large files.

I need to rush to something else right now, but will get back to you later in (my evening) and post here.

Be confident we'll get it to work reliably.

[Zipur]

Awesome, thank you so much

[jchd]

Hi again, sorry for interrupt.

Can you try again with this replacement functions for upload and download. Rename them or remove the old versions from the FTPEx.au3 UDF source.

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _FTP_ProgressUpload
; Description ...: Uploads a file in Binary Mode and shows a Progress window or by Calling a User defined Function.
; Syntax.........: _FTP_ProgressUpload($l_FTPSession, $s_LocalFile, $s_RemoteFile[, $FunctionToCall = ""])
; Parameters ....: $l_FTPSession    - as returned by _FTP_Connect().
;   $s_LocalFile    - The local file.
;   $s_RemoteFile   - The remote Location for the file.
;   $FunctionToCall - [Optional] A function which can update a Progressbar and
;   react on UserInput like Click on Abort or Close App.
;   (More info in the end of this comment)
; Return values .: Success: 1
;   Error: 0 and @error:
;   -1 -> Local file couldn't be opened
;   -3 -> Create File failed
;   -4 -> Write to file failed
;   -5 -> Close File failed
;               -6 -> Download aborted by PercentageFunc, Return of Called Function
; Author ........: limette, Prog@ndy
; Modified.......: jchd
; Remarks .......:
; Information about $FunctionToCall:
; Parameter: $Percentage - The Percentage of Progress
; Return Values: Continue Download - 1
;   Abort Download  - 0 Or negative
;   These Return Values are returned by _FTP_UploadProgress, too,
;   so you can react on different Actions like Aborting by User, closing App or TimeOut of whatever
;~ Examples:
;~  Func _UpdateProgress($Percentage)
;~  ProgressSet($percent,$percent &"%")
;~  If _IsPressed("77") Then Return 0 ; Abort on F8
;~  Return 1 ; bei 1 Fortsetzten
;~  Endfunc
;
;~  Func _UpdateProgress($Percentage)
;~  GUICtrlSetData($ProgressBarCtrl,$percent)
;~  Switch GUIGetMsg()
;~  Case $GUI_EVENT_CLOSE
;~  Return -1 ; _FTP_UploadProgress Aborts with -1, so you can exit you app afterwards
;~  Case $Btn_Cancel
;~  Return 0 ; Just Cancel, without special Return value
;~  EndSwitch
;~  Return 1 ; Otherwise contine Upload
;~  Endfunc
; Related .......:
; Link ..........: @@MsdnLink@@ FtpOpenFile
; Example .......:
; ===============================================================================================================================
Func _FTP_ProgressUpload($l_FTPSession, $s_LocalFile, $s_RemoteFile, $FunctionToCall = "")
    If $__ghWinInet_FTP = -1 Then Return SetError(-2, 0, 0)

    Local $fhandle = FileOpen($s_LocalFile, 16)
    If @error Then Return SetError(-1, 0, 0)

    Local $ai_ftpopenfile = DllCall($__ghWinInet_FTP, 'handle', 'FtpOpenFileW', 'handle', $l_FTPSession, 'wstr', $s_RemoteFile, 'dword', $GENERIC_WRITE, 'dword', $FTP_TRANSFER_TYPE_BINARY, 'dword_ptr', 0)
    If @error Or $ai_ftpopenfile[0] = 0 Then Return SetError(-3, _WinAPI_GetLastError(), 0)

    If $FunctionToCall = "" Then ProgressOn("FTP Upload", "Uploading " & $s_LocalFile)

    Local $glen = FileGetSize($s_LocalFile)
    Local Const $ChunkSize = 256 * 1024
    Local $last = Mod($glen, $ChunkSize)

    Local $parts = Ceiling($glen / $ChunkSize)
    Local $buffer = DllStructCreate("byte[" & $ChunkSize & "]")

    Local $ai_InternetCloseHandle, $ai_ftpwrite, $out, $ret
    Local $x = $ChunkSize
    Local $done = 0
    For $i = 1 To $parts
        If $i = $parts And $last > 0 Then
            $x = $last
        EndIf
        DllStructSetData($buffer, 1, FileRead($fhandle, $x))

        $ai_ftpwrite = DllCall($__ghWinInet_FTP, 'bool', 'InternetWriteFile', 'handle', $ai_ftpopenfile[0], 'ptr', DllStructGetPtr($buffer), 'dword', $x, 'dword*', $out)
        If @error Or $ai_ftpwrite[0] = 0 Then
            Local $lasterror = _WinAPI_GetLastError()
            $ai_InternetCloseHandle = DllCall($__ghWinInet_FTP, 'bool', 'InternetCloseHandle', 'handle', $ai_ftpopenfile[0])
            ; No need to test @error.
            FileClose($fhandle)

            Return SetError(-4, $lasterror, 0)
        EndIf
        $done += $x

        If $FunctionToCall = "" Then
            ProgressSet(($done / $glen) * 100)
        Else
            $ret = Call($FunctionToCall, ($done / $glen) * 100)
            If $ret <= 0 Then
                $ai_InternetCloseHandle = DllCall($__ghWinInet_FTP, 'bool', 'InternetCloseHandle', 'handle', $ai_ftpopenfile[0])
                ; No need to test @error.
                DllCall($__ghWinInet_FTP, 'bool', 'FtpDeleteFileW', 'handle', $l_FTPSession, 'wstr', $s_RemoteFile)
                ; No need to test @error.
                FileClose($fhandle)
                Return SetError(-6, 0, $ret)
            EndIf
        EndiF
        Sleep(10)
    Next

    FileClose($fhandle)

    If $FunctionToCall = "" Then ProgressOff()

    $ai_InternetCloseHandle = DllCall($__ghWinInet_FTP, 'bool', 'InternetCloseHandle', 'handle', $ai_ftpopenfile[0])
    ; No need to test @error.
    If @error Or $ai_InternetCloseHandle[0] = 0 Then Return SetError(-5, 0, 0)

    Return 1
EndFunc ;==>_FTP_ProgressUpload


; #FUNCTION# ====================================================================================================================
; Name...........: _FTP_ProgressDownload
; Description ...: Downloads a file in Binary Mode and shows a Progress window or by Calling a User defined Function.
; Syntax.........: _FTP_ProgressDownload($l_FTPSession, $s_LocalFile, $s_RemoteFile[, $FunctionToCall = ""])
; Parameters ....: $l_FTPSession    - as returned by _FTP_Connect().
;   $s_LocalFile    - The local file to create.
;   $s_RemoteFile   - The remote source file.
;   $FunctionToCall - [Optional] A function which can update a Progressbar and
;   react on UserInput like Click on Abort or Close App.
;   (More info in the end of this comment)
; Return values .: Success - 1
;   Error: 0 and @error:
;   -1 -> Local file couldn't be created
;   -3 -> Open RemoteFile failed
;   -4 -> Read from Remotefile failed
;   -5 -> Close RemoteFile failed
;               -6 -> Download aborted by PercentageFunc, Return of Called Function
;   -7 -> Local file write failed
;   -8 -> Unable to get RemoteFile size
; Author ........: limette, Prog@ndy
; Modified.......: jchd
; Remarks .......:
; Information about $FunctionToCall:
; Parameter: $Percentage - The Percentage of Progress
; Return Values: Continue Download - 1
;   Abort Download  - 0 Or negative
;   These Return Values are returned by _FTP_ProgressDownload(), too,
;   so you can react on different Actions like Aborting by User, closing App or TimeOut of whatever
;~ Examples:
;~  Func _UpdateProgress($Percentage)
;~  ProgressSet($percent,$percent &"%")
;~  If _IsPressed("77") Then Return 0 ; Abort on F8
;~  Return 1 ; bei 1 Fortsetzten
;~  Endfunc
;
;~  Func _UpdateProgress($Percentage)
;~  GUICtrlSetData($ProgressBarCtrl,$percent)
;~  Switch GUIGetMsg()
;~  Case $GUI_EVENT_CLOSE
;~  Return -1 ; _FTP_DownloadProgress Aborts with -1, so you can exit you app afterwards
;~  Case $Btn_Cancel
;~  Return 0 ; Just Cancel, without special Return value
;~  EndSwitch
;~  Return 1 ; Otherwise contine Download
;~  Endfunc
; Related .......:
; Link ..........:
; Example .......:
; ===============================================================================================================================
Func _FTP_ProgressDownload($l_FTPSession, $s_LocalFile, $s_RemoteFile, $FunctionToCall = "")
    If $__ghWinInet_FTP = -1 Then Return SetError(-2, 0, 0)

    Local $fhandle = FileOpen($s_LocalFile, 18)
    If $fhandle < 0 Then Return SetError(-1, 0, 0)

    Local $ai_ftpopenfile = DllCall($__ghWinInet_FTP, 'handle', 'FtpOpenFileW', 'handle', $l_FTPSession, 'wstr', $s_RemoteFile, 'dword', $GENERIC_READ, 'dword', $FTP_TRANSFER_TYPE_BINARY, 'dword_ptr', 0)
    If @error Or $ai_ftpopenfile[0] = 0 Then Return SetError(-3, _WinAPI_GetLastError(), 0)

    If $FunctionToCall = "" Then ProgressOn("FTP Download", "Downloading " & $s_LocalFile)

    Local $ai_FTPGetFileSize = DllCall($__ghWinInet_FTP, 'dword', 'FtpGetFileSize', 'handle', $ai_ftpopenfile[0], 'dword*', 0)
    If @error Then Return SetError(-8, _WinAPI_GetLastError(), 0)

    Local $glen = _WinAPI_MakeQWord($ai_FTPGetFileSize[0], $ai_FTPGetFileSize[2]) ;FileGetSize($s_RemoteFile)
    Local Const $ChunkSize = 256 * 1024
    Local $last = Mod($glen, $ChunkSize)

    Local $parts = Ceiling($glen / $ChunkSize)
    Local $buffer = DllStructCreate("byte[" & $ChunkSize & "]")

    Local $ai_InternetCloseHandle, $ai_FTPread, $out, $ret
    Local $x = $ChunkSize
    Local $done = 0
    For $i = 1 To $parts
        If $i = $parts And $last > 0 Then
            $x = $last
        EndIf

        $ai_FTPread = DllCall($__ghWinInet_FTP, 'bool', 'InternetReadFile', 'handle', $ai_ftpopenfile[0], 'ptr', DllStructGetPtr($buffer), 'dword', $x, 'dword*', $out)
        If @error Or $ai_FTPread[0] = 0 Then
            Local $lasterror = _WinAPI_GetLastError()
            $ai_InternetCloseHandle = DllCall($__ghWinInet_FTP, 'bool', 'InternetCloseHandle', 'handle', $ai_ftpopenfile[0])
            ; No need to test @error.
            FileClose($fhandle)
            Return SetError(-4, $lasterror, 0)
        EndIf
        $ret = FileWrite($fhandle, BinaryMid(DllStructGetData($buffer, 1), 1, $ai_FTPread[4]))
        If Not $ret Then
            $ai_InternetCloseHandle = DllCall($__ghWinInet_FTP, 'bool', 'InternetCloseHandle', 'handle', $ai_ftpopenfile[0])
            ; No need to test @error.
            FileClose($fhandle)
            FileDelete($s_LocalFile)
            Return SetError(-7)
        EndIf
        $done += $ai_FTPread[4]

        If $FunctionToCall = "" Then
            ProgressSet(($done / $glen) * 100)
        Else
            $ret = Call($FunctionToCall, ($done / $glen) * 100)
            If $ret <= 0 Then
                $ai_InternetCloseHandle = DllCall($__ghWinInet_FTP, 'bool', 'InternetCloseHandle', 'handle', $ai_ftpopenfile[0])
                ; No need to test @error.
                FileClose($fhandle)
                FileDelete($s_LocalFile)
                Return SetError(-6, 0, $ret)
            EndIf
        EndIf
        Sleep(10)
    Next

    FileClose($fhandle)

    If $FunctionToCall = "" Then ProgressOff()

    $ai_InternetCloseHandle = DllCall($__ghWinInet_FTP, 'bool', 'InternetCloseHandle', 'handle', $ai_ftpopenfile[0])
    If @error Or $ai_InternetCloseHandle[0] = 0 Then
        Return SetError(-5, _WinAPI_GetLastError(), 0)
    EndIf

    Return 1
EndFunc ;==>_FTP_ProgressDownload

I suspect that the UDF version fails in the case of large file because data is transmitted by chunks of about 1/100 of the file total length (to ease percentage progress computation). In the case of a 5Gb file, that amounts to transferring buffers of 50Mb each and this is (probably) what causes havoc. Where exactly is unclear to me as I didn't dig further.

I've changed the code in the attached function to transfer using 16Kb 256Kb chunks units (you can try changing the $ChunkSize constant). While this will cause more calls to the progress function, I believe it will be more reasonable and hopefully more robust.

Tell us how it goes.

Edit: minor changes (see my other post below)

Edit again: added both upload and download revised functions

Edited April 30, 2010 by jchd

[Zipur]

Hi

I replaced the function in the FTPEX.au3. It seems to run on my machine, but I tried it at a store and it is giving me an error of a variable used without being declared?. I put the old FTPex.au3 back and recompiled it. It works in that store now, but now I am having the same issues as I started with. The limit on the ftp upload.

What variable would not be declared? Is it that chunksize?

[jchd]

I double checked and there's no undeclared variable in this code. May be it is in another part of you code, or, more probably, you aren't using the latest AutoIt version (3.3.6.1).

I transfer 20Gb+ files here without any problem. To be honest, there are two (minor) problems with the function I posted above. First is that the transfer chunk size selected 16 Kb is clearly too small and that is slowing the transfer, at least on a fast LAN. I've had good success with a still reasonable 256Kb size. Second is I made a cosmetic mistake in the progress percentage computation (it's the inverse). I'll change the post above rather than posting yet another copy to save space and avoid confusion.

I identified the error occuring in the standard UDF version to be "not enough memory" (error#8) returned by the Windows function when transfering "one hundredth" of a large file. The companion function (download uses similar code and is likely to fail the same way. I'll push the modifications to the developpers.

[Zipur]

Yeah, I traced the declared variable back to the machine. It works fine in other stores, so i will see what is up with that one. Thanks for the help, it is appreciated.

[jchd]

Glad it works. Indeed allocating 100Mb buffer all around for a 10Gb file transfer wasn't the best idea!

I'm making the corresponding modification to the download counterpart and I'll post it here (to have them grouped) until the new versions make it in a beta or release.

Best of luck.

[jchd]

I bump this to let you know I've made available both upload and download revised functions. Look at the post above.

[Zipur]

Thanks again. It works great

[Frescard]

I was having problems too, using _FTP_ProgressUpload (it would always abort after a few MB, whereas _FTP_FilePut would work just fine), but I figured since I'm up to 3.3.6.1 that this fairly old thread would probably not apply anymore.

BUT... once I pasted the code from into my FTPEx.au, all the problems went away!

So — it might be a good idea to update the FTPEx.au package (I just saw that the last update was done on Apr 16, 2010), since without this fix that function is fairly useless (and it took me quite a few tries to actually find this relevant post).

[jchd]

It's fixed in the beta so it should appear in due time.