How to speed up response time of 'FileExists()'?

supersonic · January 21, 2012

MilesAhead,

could it really be that easy?

I will give it a try - sounds too easy to be true...

Edited January 21, 2012 by supersonic

MilesAhead · January 21, 2012

Strange, but if it works then great.

Might be a good idea to look up the API on MSDN. From what it says, the function doesn't like unc network shares that don't specify a file. So I added a test if the path starts with "" but doesn't contain a dot(not foolproof but better than nothing) then use FileExists() otherwise call the API. I think where it really gains the speed is it likely looks inside some Windows data store and sees a mapped network drive has been disconnected, rather than waiting for the time-out.

Edited January 21, 2012 by MilesAhead

guinness · January 21, 2012

Might be a good idea to look up the API on MSDN.

How do you think I found the API? To be frank I posted it over a week ago and forgot about the remarks, my apologies.

MilesAhead · January 21, 2012

How do you think I found the API? To be frank I posted it over a week ago and forgot about the remarks, my apologies.

No, it was my mistake. I quoted the wrong post.

Too quick on the mouse click.

What I meant was supersonic should look at the doc to see if it's applicable to his situation. The function works great for my folder cache utility. If I had a network drive mapped to a drive letter and it was no longer valid, the popup window would have a big delay while the program validated the folder paths. Now as long as it is a driver letter that was disconnected, the response is orders of magnitude faster. It's only slow when I'm forced to use FileExists(). But I don't use unc paths anyway. I just noted in the readme to use mapped drive letters for quickest response.

Edited January 21, 2012 by MilesAhead

MilesAhead · January 21, 2012

MilesAhead,
could it really be that easy?
I will give it a try - sounds too easy to be true...

Let me know what you find out. So far it's working great for my use.

guinness · January 21, 2012

No, it was my mistake. I quoted the wrong post.

No problems. Edited January 21, 2012 by guinness

supersonic · January 26, 2012

MilesAhead,

apparently the API call does not help in my case.

If I connect via VPN to an other network the local network is - and should be - unreachable.

Then only NSLOOKUP or NBLOOKUP gives me an acceptable timeout time (~ 1 second).

The API call takes about 10-20 seconds to time out - just like 'FileExists()' too...

Greets,

-supersonic.

Edited January 26, 2012 by supersonic

supersonic · January 26, 2012

Here's my latest version - extended with NBLOOKUP [ http://support.microsoft.com/kb/830578/en-us/ ].

I know, this is very much code for doing an apparently easy job - but it is faster than anything I have found so far...

<<< _FileExists.au3 >>>

#include-once


#include "._NBLookupExe.au3"


; Example.
;~ Local $iTimerInit = TimerInit()
;~ Local $iFileExists = _FileExists(@TempDir)
;~ Local $iFileExists = _FileExists(@LogonServer & "NETLOGON*.*")
;~ ConsoleWrite("_FileExists() -> [" & $iFileExists & "][" & @error & "][" & @extended & "][" & Round(TimerDiff($iTimerInit), 0) & "]" & @CRLF)


; [http://www.autoitscript.com/forum/topic/19370-autoit-wrappers/],
; [http://www.autoitscript.com/forum/topic/136479-how-to-speed-up-response-time-of-fileexists/],
; [http://support.microsoft.com/kb/830578/en-us/].
Func _FileExists($sFE_Path)
    Local $iFE_Return = 0
    If StringLen($sFE_Path) > 1 Then
        _StringIsValid($sFE_Path, '/"<>|')
        If Not @error Then
            Local $sFE_Drive = StringLeft($sFE_Path, 2)
            If StringInStr(@HomeDrive, $sFE_Drive, 0, 1, 1, 2) > 0 Then
                $iFE_Return = FileExists($sFE_Path)
                SetExtended(1)
            Else ; Non-error.
                Local $iFE_UNC = 0
                Local $sFE_UNC = ""
                Switch ($sFE_Drive <> "")
                    Case False ; UNC [http://en.wikipedia.org/wiki/Path_(computing)].
                        $sFE_UNC = $sFE_Path
                        ContinueCase
                    Case True ; Network drive/path?
                        If $sFE_UNC = "" Then
                            Local $sFE_DriveMapGet = DriveMapGet($sFE_Drive)
                            If Not @error Then
                                $iFE_UNC = 1
                                $sFE_UNC = $sFE_DriveMapGet
                            Else ; Error.
                                ContinueCase
                            EndIf
                        EndIf
                        If $sFE_UNC <> "" Then
                            Local $sFE_Host = StringMid($sFE_UNC, 3, StringInStr($sFE_UNC, "", 0, 1, 3) - 3)
                            If $sFE_Host <> "" Then
                                ___FileExists_NXLookup(@SystemDir & "nslookup.exe", $sFE_Host, "***", False) ; NSLookup (FQDN).
                                Switch (Not @error)
                                    Case False
                                        _NBLookupExe(Default, True)
                                        If Not @error Then
                                            ___FileExists_NXLookup(@TempDir & "nblookup.exe", $sFE_Host, "The name requested does not exist.", True) ; NBLookup (NetBIOS).
                                            If Not @error Then
                                                ContinueCase
                                            Else ; Error.
                                                SetError(9, 0)
                                            EndIf
                                        Else ; Error.
                                            SetError(8, 0)
                                        EndIf
                                    Case True
                                        Ping($sFE_Host, 1000)
                                        If Not @error Then
                                            $iFE_Return = FileExists($sFE_Path)
                                            If $iFE_UNC = 0 Then
                                                SetExtended(2)
                                            Else ; Non-error.
                                                SetExtended(3)
                                            EndIf
                                        Else ; Error.
                                            SetError(7, 0)
                                        EndIf
                                EndSwitch
                            Else ; Error.
                                SetError(6, 0)
                            EndIf
                        Else ; Error.
                            SetError(5, 0)
                        EndIf
                    Case Else ; Local drive/path.
                        Switch DriveGetType($sFE_Drive)
                            Case "CDROM", "FIXED", "RAMDISK", "REMOVABLE"
                                If DriveStatus($sFE_Drive) = "READY" Then
                                    $iFE_Return = FileExists($sFE_Path)
                                    SetExtended(4)
                                Else ; Error.
                                    SetError(4, 0)
                                EndIf
                            Case "UNKNOWN"
                                $iFE_Return = FileExists($sFE_Path)
                                SetExtended(5)
                            Case Else ; "NETWORK"
                                SetError(3, 0)
                        EndSwitch
                EndSwitch
            EndIf
        Else ; Error.
            SetError(2, 0)
        EndIf
    Else ; Error.
        SetError(1, 0)
    EndIf
    Return $iFE_Return
EndFunc   ;==>_FileExists


; [http://en.wikipedia.org/wiki/Fully_qualified_domain_name],
; [http://en.wikipedia.org/wiki/NetBIOS].
Func ___FileExists_NXLookup($sFE_NXLookup, $sFE_Host, $sFE_StringInStr, $bFE_NetBIOS = False)
    Local $iFE_Return = 0
    If $sFE_Host <> "" Then
        Local $iFE_PID = 0
        Switch $bFE_NetBIOS
            Case True
                $sFE_Host = StringMid($sFE_Host, 1, StringInStr($sFE_Host, ".", 0, 1, 1) - 1) ; FQDN to NetBIOS.
                ContinueCase
            Case Else
                $iFE_PID = Run(@ComSpec & " /C " & $sFE_NXLookup & " " & $sFE_Host, @TempDir, @SW_HIDE, 0x8)
        EndSwitch
        If Not @error Then
            Local $sFE_StdoutRead = ""
            While 1
                $sFE_StdoutRead &= StdoutRead($iFE_PID, False, False)
                If @error Then
                    ExitLoop
                EndIf
            WEnd
            If $sFE_StdoutRead <> "" Then
                If StringInStr($sFE_StdoutRead, $sFE_Host, 0) > 0 Then
                    If Not StringInStr($sFE_StdoutRead, $sFE_StringInStr, 0) Then
                        $iFE_Return = 1
                    Else ; Error.
                        SetError(5, 0)
                    EndIf
                Else ; Error.
                    SetError(4, 0)
                EndIf
            Else ; Error.
                SetError(3, 0)
            EndIf
        Else ; Error.
            SetError(2, 0)
        EndIf
    Else ; Error.
        SetError(1, 0)
    EndIf
    Return $iFE_Return
EndFunc   ;==>___FileExists_NXLookup


; [http://www.autoitscript.com/forum/topic/119453-stringisvalid/].
Func _StringIsValid($sSIV_String, $sSIV_Pattern = '/:*?"<>|')
    $sSIV_Pattern = StringRegExpReplace($sSIV_Pattern, "E", "E", 0)
    Local $iSIV_Result = BitAND(Not StringRegExp($sSIV_String, '[Q' & $sSIV_Pattern & 'E]', 0, 1), 1)
    Return SetError(Not $iSIV_Result, 0, $iSIV_Result)
EndFunc   ;==>_StringIsValid

<<< _NBLookupExe.au3 >>>

[Please see the attachment!]

Please post what you think...

_NBLookupExe.au3

Edited January 26, 2012 by supersonic

rover · January 26, 2012

I think I've come up with one possible solution for the UNC path long file system timeout issue.

Running FileExists in another process, and closing it if the process exit code is not received before the timeout does work, but is slow (400ms> per file).

Try this threaded FileExists function. It's experimental, but seems to hold up well in a test loop of 100000 iterations reading available and unavailable (timeout engaged) shares.

No crashes, memory leaks or growing unclosed handle count.

I removed a modified version of _ThreadWaitForExit() with <10ms sleep. Sleep values were not stable and could go above 10ms.

It has the option to use GetFileAttributes or PathFileExists

A few stackoverflow references:

http://stackoverflow.com/questions/51526...o-quickly-check-if-unc-path-is

Comments on using file attributes vs opening file to check if file exists

http://stackoverflow.com/questions/44039...g-for-file-existance-on-window

Some remarks regarding PathFileExists: (Determines whether a path to a file system object such as a file or directory is valid.).

Community comment from MSDN page:

"it can return failure, if kaspersky antivirus is activated..."

"If the application tries to access the file through this function it can return failure on a valid existing file due to kaspersky antivirus,

retrying after some sleep works..."

A path specified by Universal Naming Convention (UNC) is limited to a file only; that is,

serversharefile is permitted. A UNC path to a server or server share is not permitted; that is,

server or servershare. This function returns FALSE if a mounted remote drive is out of service.

Contrary to the above statement, PathFileExists does seem to work with a UNC network share (servershare) (at least tested on a local LAN share)

Requires Ascend4nt's thread functions.

Download the UDF from Ascend4nt's site:

http://sites.google.com/site/ascend4ntscode/processfunctions

Processes_Threads_nDLLs-v1-06082k11.zip

Tested on XP x86 and Win7 x64

Edit: forgot to put example code inside a function

Edit: removed unused var, spelling

;coded by rover 2k12
;Tested on XP x86 and Win7 x64
#NoTrayIcon
#include 'Processes_Threads_nDLLs[Includes]_ThreadFunctions.au3'

Opt('MustDeclareVars', 1)

Global Const $STILL_ACTIVE = 0x103
Global Const $INVALID_FILE_ATTRIBUTES = 0xFFFFFFFF ;invalid file/path - possibly permissions error, network unreachable, non-existant share

;network shares, shared files, mapped drives
;'AUTOITZ'
;'AUTOITZtest.txt'
;'Z:' ; Z (AUTOIT) (Z:)

_Example('AUTOITZ')

Func _Example($sPath)
    ProcessSetPriority(@AutoItPID, 4)
    Local $iT = TimerInit(), $sTxt = '+Return = '
    Local $iRet = _UNCPathExists($sPath, 1500, 0)
    Local $iErr = @error, $iExt = @extended
    ConsoleWrite('-Time: ' & Round(TimerDiff($iT), 1) & 'ms' & @CRLF)
    ProcessSetPriority(@AutoItPID, 2)
    If $iRet <> 1 Then $sTxt = '!Return = '
    ConsoleWrite($sTxt & $iRet & @CRLF)
    ConsoleWrite('!Exit Code = ' & $iExt & @CRLF) ;;extended =  thread exit code, function error/extended or last error
    ConsoleWrite('!Error = ' & $iErr & @CRLF)
    If $iRet = 0 Then
        ConsoleWrite('!Failed: file or path/drive/share/UNC path not found, access denied, drive not ready' & @CRLF)
    ElseIf $iErr = 0 And $iRet = -1 Then
        ConsoleWrite('-Wait time exceeded: can be one of several errors: file or path/drive/share/UNC path not found, access denied, drive not ready or remote system slow, unavailable or unresponsive' & @CRLF)
    ElseIf $iErr > 0 Then
        ConsoleWrite('!Unknown Error' & @CRLF)
    EndIf
EndFunc   ;==>_Example

Func _UNCPathExists($szPath, $iTimeout = 3000, $iMode = 0)
;$szPath = UNC share/file path
;$iTimeout = timeout in milliseconds - recommended to not use lower than 1500ms
;$iMode = 0 use GetFileAttributesW, $iMode = 1 use PathFileExistsW
;(only single parameter functions can be run directly with CreateThread)
;
;True: Return = 1, Error = 0, @Extended = exit code from thread function - for Mode 0 = File Attribute Constants (16 = directory)
;False: Return = 0, Error = 0, @Extended = exit code from thread function **
;Timeout: Return = -1, Error = 0, @Extended = exit code from thread function - STILL_ACTIVE for existing thread
;Wait time exceeded: can be one of several errors: file or path/drive/share/UNC path not found, access denied,
;drive not ready or remote system slow, unavailable or unresponsive
;
;Error: Return = -1, Error = 1 to 10, @Extended = -1, function error or extended
;**because the function is run in a thread, _WinAPI_GetLastError can't be used to get reason for failure (access denied, etc)
;either run one of the two function in a DllCall or run FileExists() and use _WinAPI_GetLastError()
;
;Author: rover 2k12
;/w ideas from Ascend4nt, Prog@ndy and Wraithdu
;Requires _ThreadFunctions.au3 from Ascend4nt's Process + Threads UDF
;http://sites.google.com/site/ascend4ntscode/processfunctions
;Processes_Threads_nDLLs-v1-06082k11.zip
Local $hMod, $pAddr, $tPath, $hThread, $iResult, $iErr, $iExt, $iIdx = 0, _
   $aFunc[2][2] = [['kernel32.dll', 'GetFileAttributesW'],['shlwapi.dll', 'PathFileExistsW']]
If $iMode <> 0 Then $iIdx = 1
$hMod = DllCall($_COMMON_KERNEL32DLL, 'handle', 'GetModuleHandleW', 'wstr', $aFunc[$iIdx][0])
If @error Or Not IsPtr($hMod[0]) Then Return SetError(2, @error, -1)
$pAddr = DllCall($_COMMON_KERNEL32DLL, 'ptr', 'GetProcAddress', 'handle', $hMod[0], 'str', $aFunc[$iIdx][1])
If @error Or Not IsPtr($pAddr[0]) Then Return SetError(3, @error, -1)
$tPath = DllStructCreate('wchar[' & StringLen($szPath) + 1 & ']')
If @error Then Return SetError(4, @error, -1)
DllStructSetData($tPath, 1, $szPath & ChrW(0)) ; terminating null
If @error Then Return SetError(5, @error, -1)
$hThread = _ThreadCreate($pAddr[0], DllStructGetPtr($tPath))
If @error Or Not IsPtr($hThread) Then Return SetError(6, @error, -1)
If Not IsNumber($iTimeout) Or $iTimeout < 1500 Then $iTimeout = 1500
;_ThreadWaitForExit adds 10ms latency, but script gui will be responsive when waiting on a thread (WaitForSingleObject with a timeout is fast if no gui in script) - see comments in this function.
$iResult = _ThreadWaitForExit($hThread, $iTimeout)
$iErr = @error
$iExt = @extended
If $iErr > 0 And $iErr <> 16 Then
  _ThreadCloseHandle($hThread)
  If $iErr = 1 Then Return SetError(7, 0, -1) ;invalid parameter
  If $iErr = 2 Then Return SetError(8, $iErr, -1) ;return dllcall error
  Return SetError(9, $iExt, -1) ;error 4: returned value from WaitForSingleObject is other than WAIT_OBJECT_0 = 0 or WAIT_TIMEOUT = 0x0102 (WAIT_FAILED,WAIT_ABANDONED,WAIT_IO_COMPLETION)
EndIf
_ThreadGetExitCode($hThread)
If @error Then
  $iErr = @error
  _ThreadCloseHandle($hThread)
  Return SetError(10, $iErr, -1)
EndIf
_ThreadCloseHandle($hThread)
If $iResult And $THREAD_LAST_EXIT_CODE > 0 Then ; thread exited normally with valid exit code indicating file exists
  ;GetFileAttributes returns greater than 0, PathFileExists returns 1 for existing file/directory
  Return SetError(0, $THREAD_LAST_EXIT_CODE, 1);extended set to return from function run in thread
  ;MSDN: GetFileAttributes - File Attribute Constants
  ;http://msdn.microsoft.com/en-us/library/windows/desktop/gg258117%28v=VS.85%29.aspx
ElseIf $iResult And ($THREAD_LAST_EXIT_CODE = $INVALID_FILE_ATTRIBUTES Or $THREAD_LAST_EXIT_CODE = 0) Then ;thread exited normally - exit code indicates file or path/drive/share not found, access denied, drive not ready, etc.
  Return SetError(0, $THREAD_LAST_EXIT_CODE, 0)
ElseIf $iErr = 16 Or $THREAD_LAST_EXIT_CODE = $STILL_ACTIVE Then ;Wait time exceeded: can be one of several errors: file or path/drive/share/UNC path not found, access denied, drive not ready or remote system slow, unavailable or unresponsive'
  Return SetError(0, $THREAD_LAST_EXIT_CODE, -1);either retry, ignore, log error or add file/path to an array for processing in background to see if file accessible with longer wait.
EndIf
Return SetError(1, -1, -1)
EndFunc   ;==>_UNCPathExists

Edited January 27, 2012 by rover

supersonic · January 27, 2012

rover,

thank you for the post. It is a completely different approach - cool!

I will adapt your piece of code to integrate it into my script. I'll give feedback how it works.

Thanks again.

Edited January 27, 2012 by supersonic

supersonic · January 27, 2012

rover,

please, can you explain why 'ProcessSetPriority()' is used?

rover · January 27, 2012

rover,
thank you for the post. It is a completely different approach - cool!
I will adapt your piece of code to integrate it into my script. I'll give feedback how it works.
Thanks again.

It's not a drop in replacement for FileExists(), but with your filtering, hopefully it will be of some use.

rover,
please, can you explain why 'ProcessSetPriority()' is used?

You get a slight performance boost when reading an array of files in a loop.

It does nothing for a single file read.

I left it in the example as a suggestion.

It decreases latency because your process gets higher priority with the cpu.

A bit of a cheat, and not recommended to use a value higher than 3 in practice.

I usually use 3, sometimes 4 doesn't decrease latency.

For instance, when populating a gui with many controls there are ways to speed up the gui creation, one way is to temporarily increase the process priority.

I tested the thread priority function. There was no benefit in this usage and it slightly increased latency.

(only tested that on XP x86)

supersonic · January 31, 2012

rover,

I have now tested your (example) UDF - it works much better than just calling 'FileExists()'.

If I use it for the interaction with "undefined"/unstable (VPN) network connections it sometimes tends to get stuck.

In such cases NBLOOKUP/NSLOOKUP helps me out once again... The timeout behavior is slightly better...

Thanks again for sharing!

Beege · January 31, 2012

supersonic,

I'm curios what kind of timeout values your looking for? I see in your original post you stated you have tried using ping(). My results for AutoIts ping timeout are usually as low as 300ms. I can understand needing lower values than that, but pretty much only when trying to find a network computer that you dont know the address of and are possibly going to be pinging hundreds of ip addresses.

supersonic · January 31, 2012

Beege,

indeed,

I'm talking about a specific "problem"...

If I'm connected to a VPN and pinging a FQDN the request uses "the wrong" IP route -

the ping accesses DNS/WINS server which are currently out of reach. And this takes time (~ 20-30 seconds)...

The IP route is not actually wrong -

but changing the route configuration is not an option.

I utilize NSLOOKUP and NBLOOKUP to resolve a hostname from a share (e. g. "SERVERSHARE") using the current DNS/WINS server(s).

In the above described scenario both tools have rather short and reliable timeout values...

Edited January 31, 2012 by supersonic

Beege · January 31, 2012

Ok I see what you mean (I think ).

Well since you already got it figured out I'll be short with how I handled a similar problem in the past. I believe its kinda what MvGulik was trying to explain. The idea would be to create second script that only does FileExists() check. That script would simply write its result of FileExists() to the console. From your main script you would launch that secondary script, then do a loop checking the stdout for the result. If you don't receive a result within the amount of time you desire, you end the secondary process (or don't because it will eventually end by itself) and move on. That would give you control over any possible hangups that might happen.

Edited January 31, 2012 by Beege

IanN1990 · January 26, 2017

Hi,

The same issue has affected me today and found this thread.

This might be stupid but this seams to be working for me

Func _FileExists($Location)
    $FileHandle = FileOpen($Location)
        If $FileHandle = -1 Then
            Return 0
        Else
            FileClose($FileHandle)
            Return 1
        EndIf
EndFunc

Sign In

How to speed up response time of 'FileExists()'?

Recommended Posts

supersonic

MilesAhead

guinness

MilesAhead

MilesAhead

guinness

supersonic

supersonic

rover

supersonic

supersonic

rover

supersonic

Beege

supersonic

Beege

IanN1990

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta