Jump to content

trying to find a way to improve speed


gcue
 Share

Recommended Posts

hello world

to get "date taken" metadata for most image types, the shell method works and it's very fast. but some image types are not supported by this method nor are video files.  the exiftool method, on the other hand, uses the exiftool works on a much wider range of image types including video files. however it is very slow

processing script

#include <file.au3>

$scan_dir = FileSelectFolder("Please select folder.", "", 2)

$files_array = _FileListToArrayRec($scan_dir, "*", $FLTAR_FILES, 1, $FLTAR_SORT, $FLTAR_FULLPATH)

For $x = 1 To UBound($files_array) - 1
    $file_path_full = $files_array[$x]

;~  $date_formatted = Get_DateTaken_Shell($file_path_full)
    $date_formatted = Get_DateTaken_ExifTool($file_path_full)

    If Not @error Then
        ConsoleWrite($file_path_full & @CRLF & $date_formatted & @CRLF & @CRLF)
    EndIf
Next

Func Get_DateTaken_Shell($file_path_full) ;credit to Melba - thanks!

    Local $sDir_Name = StringRegExpReplace($file_path_full, "(^.*\\)(.*)", "\1")
    Local $sFile_Name = StringRegExpReplace($file_path_full, "^.*\\", "")

    Local $sDOS_Dir = FileGetShortName($sDir_Name, 1)

    Local $oShellApp = ObjCreate("shell.application")
    If IsObj($oShellApp) Then
        Local $oDir = $oShellApp.NameSpace($sDOS_Dir)
        If IsObj($oDir) Then
            Local $oFile = $oDir.Parsename($sFile_Name)
            If IsObj($oFile) Then
                If @OSVersion = "WIN_VISTA" Or @OSVersion = "WIN_7" Then
                    $date = $oDir.GetDetailsOf($oFile, 12) ; Date Taken

                    If $date <> "" Then
                        $date = StringRegExpReplace($date, "(\x{200E}|\x{200F})", "")

                        Return $date
                    Else
                        Return SetError(1)
                    EndIf
                EndIf
            Else
                Return SetError(1)
            EndIf
        Else
            Return SetError(1)
        EndIf
    Else
        Return SetError(1)
    EndIf

EndFunc   ;==>Get_DateTaken_Shell

Func Get_DateTaken_ExifTool($file_path_full) ;credit to mLipok for StringRegExp pattern - thanks!

    $temp_file = _TempFile(@ScriptDir, "~")

    RunWait(@ComSpec & ' /c ' & FileGetShortName(@ScriptDir) & '\exiftool.exe -DateTimeOriginal "' & $file_path_full & '" > "' & $temp_file & '"', "", @SW_HIDE)

    $data = FileRead($temp_file)

    FileDelete($temp_file)

    If $data = "" Then
        Return SetError(1)
    EndIf

    $data = StringStripWS($data, 1 + 2)

    $date_created = StringRegExpReplace($data, "Date/Time Original.*?(\d.+)", "$1")

    If @extended <> 1 Then
        Return SetError(1)
    EndIf

    Return $date_created

EndFunc   ;==>Get_DateTaken_ExifTool

i cant think of a way to make the exiftool method faster so i was thinking of using a combination of both methods... something along the lines of this (still very slow but slightly faster)

#include <file.au3>

$scan_dir = FileSelectFolder("Please select folder.", "", 2)

$files_array = _FileListToArrayRec($scan_dir, "*", $FLTAR_FILES, 1, $FLTAR_SORT, $FLTAR_FULLPATH)

For $x = 1 To UBound($files_array) - 1
    $file_path_full = $files_array[$x]

    $date_formatted = Get_DateTaken_Shell($file_path_full)

    If @error Then
        $date_formatted = Get_DateTaken_ExifTool($file_path_full)

        If Not @error Then
            ConsoleWrite("exiftool method used" & @CRLF)
            ConsoleWrite($file_path_full & @CRLF & $date_formatted & @CRLF & @CRLF)
        EndIf
    Else
        ConsoleWrite("shell method used" & @CRLF)
        ConsoleWrite($file_path_full & @CRLF & $date_formatted & @CRLF & @CRLF)
    EndIf
Next

any better ideas?

thank you in advance!

Edited by gcue
Link to comment
Share on other sites

Yes: in the shellmethod this line:

If @OSVersion = "WIN_VISTA" Or @OSVersion = "WIN_7" Then

checks for OS, try to expand it to newer OS and test if the shellmethod also works succefull with them. For the rest shellmethod throws error take your solution. Then your script runs faster on Win 8x and Win 10x.

Edited by AutoBert
Link to comment
Share on other sites

the shellmethod is very fast already.  i am trying to improve on the exiftool method.

i have also tried using Run instead of RunWait but i would not get an output result when i did that.

i also tried doing the stdoutread but that would take really long because of the while loop.

anyone else have any creative ideas?

thanks

Link to comment
Share on other sites

10 minutes ago, gcue said:

the shellmethod is very fast already.

Yes but only on WIN_7 and WIN_VISTA, when you expand it to newer OS it would'nt fail so your:

 

16 hours ago, gcue said:

 

#include <file.au3>

$scan_dir = FileSelectFolder("Please select folder.", "", 2)

$files_array = _FileListToArrayRec($scan_dir, "*", $FLTAR_FILES, 1, $FLTAR_SORT, $FLTAR_FULLPATH)

For $x = 1 To UBound($files_array) - 1
    $file_path_full = $files_array[$x]

    $date_formatted = Get_DateTaken_Shell($file_path_full)

    If @error Then
        $date_formatted = Get_DateTaken_ExifTool($file_path_full)

        If Not @error Then
            ConsoleWrite("exiftool method used" & @CRLF)
            ConsoleWrite($file_path_full & @CRLF & $date_formatted & @CRLF & @CRLF)
        EndIf
    Else
        ConsoleWrite("shell method used" & @CRLF)
        ConsoleWrite($file_path_full & @CRLF & $date_formatted & @CRLF & @CRLF)
    EndIf
Next

 

would be faster on the newer OS where yet the EXIF-Tool Method is used. But when you are only running on Vista or Win7 there's no need for my suggest changes.

Link to comment
Share on other sites

  • 2 months later...

There is little no value in checking the OS Version every single time the Shell function is called.  I would just check once at the script launch and then determine if shell can be used.  

Also, the temp file might be slowing you down a little (temp file create, write, read, delete).  See if you can employ StdoutRead() instead.


 

Edited by spudw2k
Link to comment
Share on other sites

great suggestions.. id id try just reading the stdoutread but had trouble - varying spacing issues (horizontal and vertical) -  thats why i've been using the tempfile route

ill try again

Link to comment
Share on other sites

Interesting....Instead of doing a while loop does using ProcessWaitClose like below have any impact?
 

;...snippet from Helpfile Example...
    
    Local $iPID = Run(@ComSpec & ' /C DIR "' & $sFilePath & $sFilter & '" /B /A-D /S', $sFilePath, @SW_HIDE, $STDOUT_CHILD)
    ; If you want to search with files that contains unicode characters, then use the /U commandline parameter.

    ; Wait until the process has closed using the PID returned by Run.
    ProcessWaitClose($iPID)

    ; Read the Stdout stream of the PID returned by Run. This can also be done in a while loop. Look at the example for StderrRead.
    Local $sOutput = StdoutRead($iPID)



 

Link to comment
Share on other sites

If you use the _FileGetProperty UDF I have linked in my signature, you can find the data by name and not by the number in case it changes on other OSs.

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Link to comment
Share on other sites

22 hours ago, spudw2k said:

Interesting....Instead of doing a while loop does using ProcessWaitClose like below have any impact?
 

;...snippet from Helpfile Example...
    
    Local $iPID = Run(@ComSpec & ' /C DIR "' & $sFilePath & $sFilter & '" /B /A-D /S', $sFilePath, @SW_HIDE, $STDOUT_CHILD)
    ; If you want to search with files that contains unicode characters, then use the /U commandline parameter.

    ; Wait until the process has closed using the PID returned by Run.
    ProcessWaitClose($iPID)

    ; Read the Stdout stream of the PID returned by Run. This can also be done in a while loop. Look at the example for StderrRead.
    Local $sOutput = StdoutRead($iPID)



 

the processwaitclose makes it take longer - wondering if its really necessary?

Link to comment
Share on other sites

bummer.  thanks for entertaining the idea and trying.  I'm surprised it's slower though.

So it looks like the EXIFtool supports a batch mode.  

Quote

Improving Performance

There is a significant overhead in loading ExifTool, so performance may be greatly improved by taking advantage of ExifTool's batch processing capabilities (the ability to process multiple files or entire directories with a single command) to reduce the number of executed commands when performing complex operations or processing multiple files


That's probably the best way to squeeze performance out of this.  Generate a file list of files that fail with shell method, or simply gen a list and only use the exiftool.  

edit: of course you'll have to parse the entire output of the batch process, but it's still potentially the fastest route.

The try with shell function and if fail use exif function is also going to cause some undesired slowness as well.  Perhaps bypassing the shell method and using the batch mode will be best performing?   

Some things to test I guess.

Edited by spudw2k
Link to comment
Share on other sites

pretty easy to do batch processing..

command:

Quote

exiftool.exe -s -DateTimeOriginal "C:\Users\Smith\Desktop\New Folder1\*.*"

output:

Quote

======== C:/Users/Smith/Desktop/New folder1/20151116_101449_001.mp4
DateTimeOriginal                : 2015:09:16
======== C:/Users/Smith/Desktop/New folder1/20151229_161519.jpg
======== C:/Users/Smith/Desktop/New folder1/20151229_161530.jpg
======== C:/Users/Smith/Desktop/New folder1/angel - DIFF.jpg
DateTimeOriginal                : 2006:12:30 17:06:51
======== C:/Users/Smith/Desktop/New folder1/angel - SAME.jpg
DateTimeOriginal                : 2006:12:30 17:06:51
======== C:/Users/Smith/Desktop/New folder1/angel _SAME.jpg
DateTimeOriginal                : 2006:12:30 17:06:51
======== C:/Users/Smith/Desktop/New folder1/angel2.jpg
DateTimeOriginal                : 2005:01:01 00:04:53
======== C:/Users/Smith/Desktop/New folder1/bfc x64.cfg
======== C:/Users/Smith/Desktop/New folder1/bfc x64.exe
======== C:/Users/Smith/Desktop/New folder1/drinking_bird - Copy.fla
======== C:/Users/Smith/Desktop/New folder1/drinking_bird.fla
======== C:/Users/Smith/Desktop/New folder1/IMG_0066.JPG
DateTimeOriginal                : 2015:10:24 18:06:30
======== C:/Users/Smith/Desktop/New folder1/IMG_3762.jpg
DateTimeOriginal                : 2015:10:29 08:31:30
======== C:/Users/Smith/Desktop/New folder1/IMG_3799.jpg
DateTimeOriginal                : 2016:02:11 10:33:59
======== C:/Users/Smith/Desktop/New folder1/IMG_3817.jpg
DateTimeOriginal                : 2016:02:11 10:38:33
======== C:/Users/Smith/Desktop/New folder1/IMG_3840.jpg
DateTimeOriginal                : 2015:10:29 09:20:33
======== C:/Users/Smith/Desktop/New folder1/IMG_3894 (1).jpg
DateTimeOriginal                : 2015:10:29 09:15:23
======== C:/Users/Smith/Desktop/New folder1/IMG_4330.jpg
DateTimeOriginal                : 2016:02:04 10:51:50
======== C:/Users/Smith/Desktop/New folder1/IMG_4343.jpg
DateTimeOriginal                : 2016:02:04 10:53:42
======== C:/Users/Smith/Desktop/New folder1/IMG_4367.jpg
DateTimeOriginal                : 2016:02:04 10:58:51
======== C:/Users/Smith/Desktop/New folder1/IMG_4368.jpg
DateTimeOriginal                : 2016:02:04 10:59:00
======== C:/Users/Smith/Desktop/New folder1/IMG_4370.jpg
DateTimeOriginal                : 2016:02:04 10:59:11
======== C:/Users/Smith/Desktop/New folder1/IMG_4469.jpg
DateTimeOriginal                : 2016:02:04 11:16:12
======== C:/Users/Smith/Desktop/New folder1/IMG_4499.jpg
DateTimeOriginal                : 2016:02:04 11:20:24
======== C:/Users/Smith/Desktop/New folder1/IMG_4604 (1).jpg
DateTimeOriginal                : 2016:02:09 09:21:16
======== C:/Users/Smith/Desktop/New folder1/John_Martin_-_Sodom_and_Gomorrah - SAME.jpg
======== C:/Users/Smith/Desktop/New folder1/John_Martin_-_Sodom_and_Gomorrah-SAME.jpg
======== C:/Users/Smith/Desktop/New folder1/no.gif
======== C:/Users/Smith/Desktop/New folder1/no.jpg
======== C:/Users/Smith/Desktop/New folder1/no.mp4
======== C:/Users/Smith/Desktop/New folder1/Office Picnic 2006 - 1.PNG
======== C:/Users/Smith/Desktop/New folder1/Office Picnic 2006 - 2.PNG
======== C:/Users/Smith/Desktop/New folder1/Thumbs.db
   33 image files read
 

i need help parsing the information.  i think it might be easiest to get the results into an array for easy referencing (1 column for full file path and the other for datetimeoriginal date)

thank you very much in advance!

Edited by gcue
Link to comment
Share on other sites

for anyone interested.. there's also a way to process a list of files (so doesn't have to be all files contained within a directory)

you need to have the file paths loaded into a text file

read DateTimeOriginal for all files listed in a text file

exiftool -datetimeoriginal -@ file_list.txt

and to write tag instead of read:

exiftool "-datetimeoriginal=2016:04:19 13:29:00" -@ file_list.txt"

i am still having trouble parsing the output into an array - can anyone assist?  

thank you very much in advance!

 

Link to comment
Share on other sites

here is my mickey mouse way to get it.. =)

#include <Array.au3>
#include <File.au3>

$file_types = "*.jpg;*.jpeg;*.mp4;*.png"
$scan_dir = "C:\Users\Smith\Desktop\New folder1"
$temp_file = _TempFile(@TempDir)

$files_array = _FileListToArrayRec($scan_dir, $file_types, $FLTAR_FILES, 1, $FLTAR_SORT, $FLTAR_FULLPATH)

_FileWriteFromArray($temp_file, $files_array, 1)

$exiftool_dir = "C:\Users\Smith\Desktop"
$output_file = $exiftool_dir & "\output.txt"

RunWait(@ComSpec & ' /c ' & $exiftool_dir & '\exiftool.exe -DateTimeOriginal -@ "' & $temp_file & '" > "' & $output_file & '"', @SystemDir, @SW_HIDE)

$file = FileOpen($output_file)
$data = FileRead($file)
FileClose($file)

$array = StringSplit($data, @LF, 1)

Local $final_results[1][1]

For $x = 1 To UBound($array) - 1
    If StringInStr($array[$x], "========") <> 0 Then
        ReDim $final_results[UBound($final_results) + 1][2]

        $final_results[UBound($final_results) - 1][0] = StringReplace($array[$x], "========", "")

        If StringInStr($array[$x + 1], "Date/Time Original") <> 0 Then
            $final_results[UBound($final_results) - 1][1] = StringRegExpReplace($array[$x + 1], "Date/Time Original.*?(\d.+)", "$1")
        Else
            $final_results[UBound($final_results) - 1][1] = "NOT_FOUND"
        EndIf
    EndIf
Next

FileDelete($temp_file)
FileDelete($output_file)

_ArrayDisplay($final_results)

anyone have a more efficient way?

Edited by gcue
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...