UDF to support SFTP protocol using PSFTP

[Lupo73 9]

I started to work on this UDF and I'd like to discuss, test and improve it with other interested users.

The idea is to create a UDF similar to FTPEx, but to support SFTP protocol. Given that this protocol is not natively supported by Windows, I found some candidates to manage it:
cURL: http://curl.haxx.se/
PSFTP: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
FZSFTP: http://filezilla-project.org/

I'm trying to use the second one (realied by PuTTY team) and the result is the following code. It is not complete and needs to be widely tested, but main functions work and could be considered as a draft to start the discussion.

A possible alternative could be to use FZSFTP (a modified version of PSFTP realized for FileZilla), that includes some interesting improvements to be used by external programs, but does not offer a related documentation.


Code version: 1.0 beta 9 (not complete)
Previous downloads: 132

SFTPEx.au3

Edited May 11, 2013 by Lupo73

[ProgAndy 69]

Nice work. If you want to avoid using an additional process, I know of two possible DLLs you can use: libcurl and libssh2.

[Lupo73 9]

They could be other good alternatives, but I never worked with DLLs before and I may need few help to start. Do you think use a dll is a better solution?

My idea is to start the discussion about it to decide what solution follow. I think could be interesting for several users to create a stable UDF for SFTP support (and maybe also for other protocols like FTPS using SSL). I don't know what are the limits and rules to create official UDFs, but with the collaboration of some users I'd like to create a good library to extend FTP support to other protocols.

[Lupo73 9]

Code updated: minor fixes and standardized variable names (version 1.0 beta).

[Lupo73 9]

Code updated: more improvements and started to add descriptions as a standard UDF (version 1.0 beta 2).

[Lupo73 9]

Code updated: added _SFTP_FileMove() and _SFTP_ListToArray() functions, improved _SFTP_ListToArrayEx() and fixed some issues (version 1.0 beta 3).

Edited March 2, 2012 by Lupo73

[Lupo73 9]

Code updated: completed _SFTP_ListToArray() and _SFTP_ListToArrayEx(), planned to add _SFTP_FileExists() and _SFTP_FileGetSize(), fixed other issues (version 1.0 beta 4).

[Lupo73 9]

Code updated: added _SFTP_FileExists() and _SFTP_FileGetSize(), partially added _SFTP_ProgressDownload(), fixed other minor issues (version 1.0 beta 5).

ps: something like _SFTP_FileExists() could be easily added also to FTPEx library, what do you think about it?

[Lupo73 9]

Code updated: improved _SFTP_ListToArray() and _SFTP_ListToArrayEx(), added _SFTP_DirDelete(), fixed _SFTP_DirGetContents() and _SFTP_FileGet(), fixed other minor issues (version 1.0 beta 6).

[Lupo73 9]

Code updated: completed most of the functions (tests are welcome!)

Known limits: _SFTP_ListToArrayEx() loads time based on server timezone (not converted to the local timezone), _SFTP_ProgressDownload() doesn't correctly update progress bar downloading folders

I created also 3 functions that could be added to FTPEx:

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _FTP_FileExists
; Description ...: Checks if a file or folder exists on a FTP server.
; Syntax.........: _FTP_FileExists ( $hSession, $sRemoteFile )
; Parameters ....: $hSession - as returned by _FTP_Connect().
;                 $sRemoteFile - The remote file.
; Return values .: Success - 1
;                 Failure - 0, sets @error
;                 |1 - The connection is closed
;                 |2 - Directory not found
;                 |3 - Directory is empty
; Author ........: Lupo73
; Modified.......:
; Remarks .......:
; Related .......: _FTP_Connect
; Link ..........:
; Example .......: No
; ===============================================================================================================================
Func _FTP_FileExists($hSession, $sRemoteFile)
    If $hSession = 0 Then
        Return SetError(1, 0, 0)
    EndIf

    Local $aStringSplit, $aFileList, $sPreviousDir, $iExists = 0, $sRemoteDir = ""
    $aStringSplit = StringSplit($sRemoteFile, "/")
    If $aStringSplit[0] > 1 Then
        $sRemoteDir = StringTrimRight($sRemoteFile, StringLen($aStringSplit[$aStringSplit[0]]) + 1)
        $sRemoteFile = $aStringSplit[$aStringSplit[0]]
        $sPreviousDir = _FTP_DirGetCurrent($hSession)
        _FTP_DirSetCurrent($hSession, $sRemoteDir)
    EndIf
    $aFileList = _FTP_ListToArray($hSession)
    If @error Then
        _FTP_DirSetCurrent($hSession, $sPreviousDir)
        Return SetError(2, 0, 0)
    EndIf
    _FTP_DirSetCurrent($hSession, $sPreviousDir)
    If $aFileList[0] = 0 Then
        Return SetError(3, 0, 0)
    EndIf

    For $A = 1 To $aFileList[0]
        If $sRemoteFile = $aFileList[$A] Then
            $iExists = 1
            ExitLoop
        EndIf
    Next

    Return $iExists
EndFunc   ; ==>_FTP_FileExists

; #FUNCTION# ====================================================================================================================
; Name...........: _FTP_FileGetInfo
; Description ...: Get info of a file or folder from a FTP server.
; Syntax.........: _FTP_FileGetInfo ( $hSession, $sRemoteFile )
; Parameters ....: $hSession - as returned by _FTP_Connect().
;                 $sRemoteFile - The remote file.
; Return values .: Success - returns an Array containing file info.
;                 Failure - 0, sets @error
;                 |1 - The connection is closed
;                 |2 - Directory not found
;                 |3 - Directory is empty
;                 |4 - File not found
; Author ........: Lupo73
; Modified.......:
; Remarks .......: Array[0] Filename
;                 Array[1] Filesize
;                 Array[2] Permissions
;                 Array[3] File Modification datetime
;                 Array[4] File Creation datetime
;                 Array[5] File Access datetime
; Related .......: _FTP_Connect, _FTP_ListToArrayEx
; Link ..........:
; Example .......: No
; ===============================================================================================================================
Func _FTP_FileGetInfo($hSession, $sRemoteFile)
    If $hSession = 0 Then
        Return SetError(1, 0, 0)
    EndIf

    Local $aArray[6], $aStringSplit, $aFileList, $sPreviousDir, $sRemoteDir = ""
    $aStringSplit = StringSplit($sRemoteFile, "/")
    If $aStringSplit[0] > 1 Then
        $sRemoteDir = StringTrimRight($sRemoteFile, StringLen($aStringSplit[$aStringSplit[0]]) + 1)
        $sRemoteFile = $aStringSplit[$aStringSplit[0]]
        $sPreviousDir = _FTP_DirGetCurrent($hSession)
        _FTP_DirSetCurrent($hSession, $sRemoteDir)
    EndIf
    $aFileList = _FTP_ListToArrayEx($hSession)
    If @error Then
        _FTP_DirSetCurrent($hSession, $sPreviousDir)
        Return SetError(2, 0, 0)
    EndIf
    _FTP_DirSetCurrent($hSession, $sPreviousDir)
    If $aFileList[0][0] = 0 Then
        Return SetError(3, 0, 0)
    EndIf

    For $A = 1 To $aFileList[0][0]
        If $sRemoteFile = $aFileList[$A][0] Then
            For $B = 0 To 5
                $aArray[$B] = $aFileList[$A][$B]
            Next
            Return $aArray
        EndIf
    Next

    Return SetError(4, 0, 0)
EndFunc   ; ==>_FTP_FileGetInfo

; #FUNCTION# ====================================================================================================================
; Name...........: __FTP_ListToArrayEx2
; Description ...: Get names, sizes, attributes and times of files/folders into defined remote directory.
; Syntax.........: __FTP_ListToArrayEx2 ( $hSession [, $sRemoteDir = "" [, $ReturnType = 0 [, $iFlags = 0 [, $fTimeFormat = 1]]]] )
; Parameters ....: $hSession - as returned by _FTP_Connect().
;                 $sRemoteDir - Optional, The remote Directory (or use current remote Directory if not defined).
;                 $ReturnType - Optional, 0 = Both Files and Folders (default), 1 = Folders, 2 = Files.
;                 $iFlags - Optional, see _FTP_FindFileFirst().
;                 $fTimeFormat - Optional, type on the date strings: 1 = YYYY/MM/DD[ HH:MM] (default), 0 = MM/DD/YYYY[ HH:MM].
; Return values .: Success - returns an Array containing file info.
;                 Failure - 0, sets @error same of _FTP_ListToArrayEx()
; Author ........: Lupo73
; Modified.......:
; Remarks .......: Array[0][0] = Number of found entries
;                 Array[x][0] Filename
;                 Array[x][1] Filesize
;                 Array[x][2] Permissions
;                 Array[x][3] File Modification datetime
;                 Array[x][4] File Creation datetime
;                 Array[x][5] File Access datetime
; Related .......: _FTP_Connect, _FTP_ListToArrayEx, _FTP_FindFileFirst
; Link ..........:
; Example .......: No
; ===============================================================================================================================
Func __FTP_ListToArrayEx2($hSession, $sRemoteDir = "", $ReturnType = 0, $iFlags = 0, $fTimeFormat = 1)
    If $hSession = 0 Then
        Return SetError(1, 0, 0)
    EndIf

    Local $aFileList, $sPreviousDir, $iError = 0
    $sPreviousDir = _FTP_DirGetCurrent($hSession)
    _FTP_DirSetCurrent($hSession, $sRemoteDir)
    $aFileList = _FTP_ListToArrayEx($hSession, $ReturnType, $iFlags, $fTimeFormat)
    $iError = @error
    _FTP_DirSetCurrent($hSession, $sPreviousDir)
    If $iError Then
        Return SetError($iError, 0, $aFileList)
    EndIf

    Return $aFileList
EndFunc   ; ==>__FTP_ListToArrayEx2

Edited March 13, 2012 by Lupo73

[kazki 0]

Cool UDF. Saves me time from tinkering with psftp

However you may also want to add additional logic on this particular scenario below for first time connection and have it default to either yes or no:

The server's host key is not cached in the registry. You

have no guarantee that the server is the computer you

think it is.

The server's rsa2 key fingerprint is:

ssh-rsa 1024 ***********************************

If you trust this host, enter "y" to add the key to

PuTTY's cache and carry on connecting.

If you want to carry on connecting just once, without

adding the key to the cache, enter "n".

If you do not trust this host, press Return to abandon the

connection.

Store key in cache? (y/n)

Thanks

[kazki 0]

It turns out that the window is still accepting input. It was the stdout that did not get updated. When I inserted some StdinWrite($hSession, @CRLF) on line 160 the window responded with "Too many logon attempts.". This is right after I inserted StdinWrite($hSession, "y"'& @CRLF) on line 140 after the Sleep(10). Login also worked when I did the same thing with the credentials.

[IgneusJotunn 0]

This looks great, but when I try to use it, I get an error. SciTE points at line 1116, in the _SFTP_Open function. Specifically, this line.

Local $hSession = Run($sPath, "", @SW_HIDE, $STDIN_CHILD + $STDOUT_CHILD)

It looks like $STDIN_CHILD and $STDOUT_CHILD aren't declared anywhere else in the script, and aren't passed as input. Can you point me in the right direction as to what those should be doing?

[guinness 1,465]

Check the help file, these can be found in the Constants.au3 file, just include it in your script.

[IgneusJotunn 0]

Thank you Guinness, that worked. And thank you Lupo73 for the UDF in the first place!

[IgneusJotunn 0]

Minor suggested alteration to the code. At the moment, _SFTP_Close() does not log out. It's really easy to do (It only takes one line) but it might be worth making sure that a connection is not left open by logging out before the ProcessClose in _SFTP_Close. Alternitively, you could create an _SFTP_Disconnect function.

The line that does this is

stdinWrite($hSession, "bye")

Not a big contribution I know, but I hope it helps! Great code, very helpful.

[pinkfoyd 1]

Hi Lupo73 for this great UDF !

But i have a very annoying bug with it !

When i use '_SFTP_ListToArray' or '_SFTP_ListToArrayEx', it return me only one file, but with filezilla i can see two files !

Can you check this ? Thanks

For more information, i have put a debug line in your UDF

Line 1032 :

$sLine &= StdoutRead($hConnection)

ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $sLine = ' & $sLine & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console

And here is the answer :

$sLine = Listing directory /Deals/Out

drwxrwx--- 1 no-user no-group 0 Jan 1 1970 ..

-rwxrwx--- 1 no-user no-group 5500 Oct 26 14:07 xxxx.xml

-rwxrwx--- 1 no-user no-group 6159 Oct 26 14:08 xxxxx.xml

psftp>

so the UDF see the two file, but in the array i have only one

I don't find but the problem if after ligne 1032 when you transforme you stringplit to $aarray

I have solve my problem by working the stingsplit by myself, but it will be 'more clean' is you can correct the udf ^^

Edited October 30, 2012 by pinkfoyd

[AngelsFlyHigh 0]

Hi!

Used the SFTPEx.au3 to play with moving files from an SFTP server and found 2 issues.

_SFTP_ListToArray and _SFTP_ListToArrayEx both misses the 2 first files on the FTP. I rewrote them thus:

expandcollapse popup

Func _SFTP_ListToArrayEx($hConnection, $sRemoteDir = "", $ReturnType = 0, $fTimeFormat = 1)
Local $aArray[1][5]
If ProcessExists($hConnection) = 0 Then
$aArray[0][0] = 0
Return SetError(1, 0, $aArray)
EndIf

If $ReturnType < 0 Or $ReturnType > 2 Then
$aArray[0][0] = 0
Return SetError(4, 0, $aArray)
EndIf

Local $sLine, $sTime, $aStringSplit, $aSubStringSplit, $B, $iArrayIndex = 0
If $sRemoteDir <> "" Then
$sRemoteDir = ' "' & $sRemoteDir & '"'
EndIf
StdinWrite($hConnection, 'ls' & $sRemoteDir & @CRLF)
While 1
$sLine &= StdoutRead($hConnection)
If ProcessExists($hConnection) = 0 Then
$aArray[0][0] = 0
Return SetError(1, 0, $aArray)
ElseIf StringInStr($sLine, "Unable to open") Then
$aArray[0][0] = 0
Return SetError(2, 0, $aArray)
ElseIf StringInStr($sLine, "Multiple-level wildcards") Or StringInStr($sLine, ": canonify:") Then
$aArray[0][0] = 0
Return SetError(5, 0, $aArray)
ElseIf StringInStr($sLine, "psftp>") Then
ExitLoop
EndIf
Sleep(10)
WEnd

$aStringSplit = StringSplit(StringStripCR($sLine), @LF)
If IsArray($aStringSplit) = 0 Then
$aArray[0][0] = 0
Return SetError(3, 0, $aArray)
EndIf
; CHANGED BY TCR
; $aArray[0][0] = $aStringSplit[0] - 4
$aArray[0][0] = $aStringSplit[0] - 2
If $aArray[0][0] < 1 Then
$aArray[0][0] = 0
Return $aArray
EndIf
ReDim $aArray[$aArray[0][0] + 1][5]

; For $A = 4 To $aStringSplit[0] - 1
For $A = 2 To $aStringSplit[0] - 1
If ($ReturnType = 1 And StringLeft($aStringSplit[$A], 1) <> "d") Or ($ReturnType = 2 And StringLeft($aStringSplit[$A], 1) <> "-") Then
$iArrayIndex += 1
$aArray[0][0] -= 1
ContinueLoop
EndIf
; $B = $A - 3 - $iArrayIndex
$B = $A - 1 - $iArrayIndex
$aSubStringSplit = _StringExplode(StringStripWS($aStringSplit[$A], 7), " ", 8)
If UBound($aSubStringSplit) < 9 Then
$iArrayIndex += 1
$aArray[0][0] -= 1
ContinueLoop
EndIf
$aArray[$B][0] = $aSubStringSplit[8]
$aArray[$B][1] = 0
If StringLeft($aSubStringSplit[0], 1) <> "d" Then ; Is A File.
$aArray[$B][1] = $aSubStringSplit[4]
EndIf
$aArray[$B][2] = $aSubStringSplit[0]
$sTime = ""
$aSubStringSplit[6] = StringRight("0" & $aSubStringSplit[6], 2)
If StringInStr($aSubStringSplit[7], ":") Then ; Is A Time.
$sTime = " " & $aSubStringSplit[7]
$aSubStringSplit[7] = @YEAR
If _DateDiff('n', $aSubStringSplit[7] & "/" & __MonthToNumber($aSubStringSplit[5]) & "/" & $aSubStringSplit[6] & $sTime, _NowCalc()) < 0 Then
$aSubStringSplit[7] -= 1
EndIf
EndIf
If $fTimeFormat = 1 Then
$aArray[$B][3] = $aSubStringSplit[7] & "/" & __MonthToNumber($aSubStringSplit[5]) & "/" & $aSubStringSplit[6] & $sTime
Else
$aArray[$B][3] = __MonthToNumber($aSubStringSplit[5]) & "/" & $aSubStringSplit[6] & "/" & $aSubStringSplit[7] & $sTime
EndIf
; <<<<<<<<<<<<<<<<<<<<<<<<<<< it may needs to update time based on timezone offset
$aArray[$B][4] = $aSubStringSplit[2] & " " & $aSubStringSplit[3]
Next
ReDim $aArray[$aArray[0][0] + 1][5]

Return $aArray
EndFunc ; ==>_SFTP_ListToArrayEx

And

expandcollapse popup

Func _SFTP_ListToArray($hConnection, $sRemoteDir = "", $ReturnType = 0)
Local $aArray[1]
If ProcessExists($hConnection) = 0 Then
$aArray[0] = 0
Return SetError(1, 0, $aArray)
EndIf

If $ReturnType < 0 Or $ReturnType > 2 Then
$aArray[0] = 0
Return SetError(4, 0, $aArray)
EndIf

Local $sLine, $aStringSplit, $aSubStringSplit, $B, $iArrayIndex = 0
If $sRemoteDir <> "" Then
$sRemoteDir = ' "' & $sRemoteDir & '"'
EndIf
StdinWrite($hConnection, 'ls' & $sRemoteDir & @CRLF)
While 1
$sLine &= StdoutRead($hConnection)
If ProcessExists($hConnection) = 0 Then
$aArray[0] = 0
Return SetError(1, 0, $aArray)
ElseIf StringInStr($sLine, "Unable to open") Then
$aArray[0] = 0
Return SetError(2, 0, $aArray)
ElseIf StringInStr($sLine, "Multiple-level wildcards") Or StringInStr($sLine, ": canonify: ") Then
$aArray[0] = 0
Return SetError(5, 0, $aArray)
ElseIf StringInStr($sLine, "psftp>") Then
ExitLoop
EndIf
Sleep(10)
WEnd
;MsgBox(0, "Inside _SFTP_ListToArray", "Have Just got this file list: " & @CRLF & $sLine)

$aStringSplit = StringSplit(StringStripCR($sLine), @LF)
If IsArray($aStringSplit) = 0 Then
$aArray[0] = 0
Return SetError(3, 0, $aArray)
EndIf
; CHANGED BY TCR:
; Removing first 4 lines... why???
; Lets Try only to remove 2 line and see...
; $aArray[0] = $aStringSplit[0] - 4
$aArray[0] = $aStringSplit[0] - 2
If $aArray[0] < 1 Then
$aArray[0] = 0
Return $aArray
EndIf
ReDim $aArray[$aArray[0] + 1]

; For $A = 4 To $aStringSplit[0] - 1
For $A = 2 To $aStringSplit[0] - 1
If ($ReturnType = 1 And StringLeft($aStringSplit[$A], 1) <> "d") Or ($ReturnType = 2 And StringLeft($aStringSplit[$A], 1) <> "-") Then
$iArrayIndex += 1
$aArray[0] -= 1
ContinueLoop
EndIf
; $B = $A - 3 - $iArrayIndex
$B = $A - 1 - $iArrayIndex
$aSubStringSplit = _StringExplode(StringStripWS($aStringSplit[$A], 7), " ", 8)
If UBound($aSubStringSplit) < 9 Then
$iArrayIndex += 1
$aArray[0] -= 1
ContinueLoop
EndIf
$aArray[$B] = $aSubStringSplit[8]
Next
ReDim $aArray[$aArray[0] + 1]

Return $aArray
EndFunc ; ==>_SFTP_ListToArray

Dont know why you have 3 garbage lines and I only get one. Might be that my test server didnt show hidden files or that it was on a Danish computer. Just guessing.

My suggestion would be to rewrite these functions to take ALL lines and then only process lines starting with d??? or -??? to filter out all garbage lines.

Kind Regards,

Thomas

PS. Great Work!!! DS.

Edited November 29, 2012 by AngelsFlyHigh

[Lupo73 9]

Thanks for your reports, I'll fix the UDF. If you made other improvements in the meanwhile, they are welcome!

Edit: I updated the UDF following your advices.. let me know!

Edited March 5, 2013 by Lupo73

[trainer 0]

Hi Lupo.. maybe you should add the red marked code below to your functions, because if someone (like me) has installed putty and uses a standard session, connecting to any server will fail, because the standard session is being loaded by default and the desired StdOut "psftp>" never appears.

This is the solution to the problem kazki has described in his post.

Cheers

trainer

Func _SFTP_Open($sPath = 'psftp.exe')

Local $hSession = Run($sPath & " -load null", "", @SW_HIDE, $STDIN_CHILD + $STDOUT_CHILD)

[...]

Func _SFTP_Connect($hSession, $sServerName, $sUsername = "", $sPassword = "", $iServerPort = 0)

If ProcessExists($hSession) = 0 Then

Return SetError(1, 0, 0)

EndIf

If $iServerPort = 0 Then

$iServerPort = ""

EndIf

Local $sLine, $sStringSplit

StdinWrite($hSession, 'open ' & $sServerName & ' ' & $iServerPort & @CRLF)

$wait_for_key_saving = 0

$save_key_if_not_saved = True

While 1

;If after 5 seconds no psftp>-prompt appears, putty is probably waiting for the answer whether the (new) ssh-key should be saved in the registry (this appears every time, if the key is not being saved) --> answer the question with 'y' (once).

$wait_for_key_saving += 1

If $wait_for_key_saving >= 500 And $save_key_if_not_saved Then

StdinWrite($hSession, 'y' & @CRLF)

$save_key_if_not_saved = False

EndIf

$sLine = StdoutRead($hSession)

[...]

Edited May 3, 2013 by trainer