Jump to content

Recommended Posts

Posted (edited)

Hello everyone, I am working on a project which requires reading a few values from Excel, the catch is that I need it to be very fast... unfortunatley I found out that read operations using the supplied Excel UDF are very slow, more than 150 ms for each operation on average :(

Here is my testing setup that I made:

#include <Excel.au3>
#include <MsgBoxConstants.au3>

Global $iTotalTime = 0

Test()

Func Test()
    Local $oExcel = _Excel_Open()
    Local $oBook = _Excel_BookAttach("Test.xlsx", "FileName", $oExcel)
    Local $sSheet = "Sheet1"
    If @error Then Return MsgBox($MB_ICONERROR, "Excel Failed", "Failed to attach to Excel")

    Local $iNum
    For $iRow = 1 To 6
        Time()
        Local $iNum = Number(_Excel_RangeRead($oBook, $sSheet, "A" & $iRow))
        If ($iNum = 1) Then
            ConsoleWrite("Row " & $iRow & " is 1 and value of column B is " & _Excel_RangeRead($oBook, $sSheet, "B" & $iRow))
        Else
            ConsoleWrite("Row " & $iRow & " is not 1")
        EndIf
        ConsoleWrite(". Reading took: ")
        Time()
    Next
    ConsoleWrite("The whole operation took " & $iTotalTime & " milliseconds." & @CRLF)
EndFunc

Func Time()
    Local Static $hTimer
    Local Static $bRunning = False
    If $bRunning Then
        Local $iTime = Round(TimerDiff($hTimer), 2)
        $iTotalTime += $iTime
        ConsoleWrite($iTime & @CRLF)
    Else
        $hTimer = TimerInit()
    EndIf
    $bRunning = Not $bRunning
EndFunc

And Test.xlsx in CSV format:

1,-1
-1,1
1,-1
1,1
-1,-1
1,1

Here is the actual xlsx but it should expire in a week: https://we.tl/t-EVkxGp1kc6

And finally output from my script:

Row 1 is 1 and value of column B is -1. Reading took: 276.06
Row 2 is not 1. Reading took: 163.36
Row 3 is 1 and value of column B is -1. Reading took: 302.58
Row 4 is 1 and value of column B is 1. Reading took: 294.65
Row 5 is not 1. Reading took: 152.33
Row 6 is 1 and value of column B is 1. Reading took: 284.92
The whole operation took 1473.9 milliseconds.

 

Taking ~1.5 seconds for reading 6 rows of data is bad for my script, which needs to run as fast as possible :(. It would be nice if I can bring this down to 100 ms somehow, I am not very experienced working with MS office so I thought about asking you folks for help and advice on how I can optimize my script to squeeze out every bit of performance that I can get from this script :D

 

Thanks for the help in advance!

Edited by SDL
excel -> Excel

A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

@SDL
Does this file need to be in XLSX or you can convert it in CSV format as you did above? :)


Click here to see my signature:

Spoiler

Thoughts:

  • I will always thank you for the time you spent for me.
    I'm here to ask, and from your response, I'd like to learn.
    By my knowledge, I can help someone else, and "that someone" could help in turn another, and so on.

/*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------*/

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites
7 minutes ago, FrancescoDiMuro said:

Does this file need to be in XLSX or you can convert it in CSV format as you did above? :)

Good question, the only reason I am using XLSX is because I need Excel to automatically update those values via an addon. Otherwise I would simply have used CSV or anything else which is appropriate :)

I gave the CSV because it is easier to paste it here... and I could not provide the xlsx file permanently without using up my attachment space.


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

As you are only reading a few rows/columns I would use _Excel_RangeRead to read the whole used range and then process the returned array.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2019-08-19 - Version 1.4.13.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2019-07-24 - Version 1.3.6.0) - Download - General Help & Support - Example Scripts - Wiki
Outlook Tools (NEW 2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
PowerPoint (2017-06-06 - Version 0.0.5.0) - Download - General Help & Support
Excel - Example Scripts - Wiki
Word - Wiki
Taks Scheduler (NEW 2019-10-09 - Version 0.9.0.0) - Download - General Help & Support - Wiki

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

@water & @kaisies I see, I will try adapting my script to reduce calls to RangeRead and see how it performs :)

Will report back once I test it!


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

I have modified my original script to read all the data once instead of one by one for every cell... and boy are the speed gains huge! :D

#include <Array.au3>
#include <Excel.au3>
#include <MsgBoxConstants.au3>

Global $iTotalTime = 0

Test()

Func Test()
    Local $oExcel = _Excel_Open()
    Local $oBook = _Excel_BookAttach("Test.xlsx", "FileName", $oExcel)
    Local $sSheet = "Sheet1"
    If @error Then Return MsgBox($MB_ICONERROR, "Excel Failed", "Failed to attach to Excel")

    Local $iNum
    Time()
    Local $aData = _Excel_RangeRead($oBook, $sSheet, "A1:B6")
    Time()
    _ArrayDisplay($aData)
    For $iRow = 0 To 5
        Time()
        Local $iNum = Number($aData[$iRow][0])
        If $iNum = 1 Then
            ConsoleWrite("Row " & $iRow + 1 & " is 1 and value of column B is " & $aData[$iRow][0])
        Else
            ConsoleWrite("Row " & $iRow + 1 & " is not 1")
        EndIf
        ConsoleWrite(". Reading took: ")
        Time()
    Next
    ConsoleWrite("The whole operation took " & $iTotalTime & " milliseconds." & @CRLF)
EndFunc

Func Time()
    Local Static $hTimer
    Local Static $bRunning = False
    If $bRunning Then
        Local $iTime = Round(TimerDiff($hTimer), 2)
        $iTotalTime += $iTime
        ConsoleWrite($iTime & @CRLF)
    Else
        $hTimer = TimerInit()
    EndIf
    $bRunning = Not $bRunning
EndFunc
120.15
Row 1 is 1 and value of column B is 1. Reading took: 0.07
Row 2 is not 1. Reading took: 0.02
Row 3 is 1 and value of column B is 1. Reading took: 0.02
Row 4 is 1 and value of column B is 1. Reading took: 0.02
Row 5 is not 1. Reading took: 0.01
Row 6 is 1 and value of column B is 1. Reading took: 0.02
The whole operation took 120.31 milliseconds.

From ~1500 ms to ~120 ms, that is a huge 12.5x improvement in speed :thumbsup:

And even most of that is the first initial call to Angered, definitely going to try this method of caching in my project... I actually feel a bit dumb not having tried this before asking :lol:


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

You learn something new each day :)


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2019-08-19 - Version 1.4.13.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2019-07-24 - Version 1.3.6.0) - Download - General Help & Support - Example Scripts - Wiki
Outlook Tools (NEW 2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
PowerPoint (2017-06-06 - Version 0.0.5.0) - Download - General Help & Support
Excel - Example Scripts - Wiki
Word - Wiki
Taks Scheduler (NEW 2019-10-09 - Version 0.9.0.0) - Download - General Help & Support - Wiki

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

I discovered another way to hugely cut up the speed... by using the inbuilt AutoIt transpose function! Just set $bForceFunc parameter to true and enjoy the gains :)

#include <Array.au3>
#include <Excel.au3>
#include <MsgBoxConstants.au3>

Global $iTotalTime = 0

Test()

Func Test()
    Local $oExcel = _Excel_Open()
    Local $oBook = _Excel_BookAttach("Test.xlsx", "FileName", $oExcel)
    Local $sSheet = "Sheet1"
    If @error Then Return MsgBox($MB_ICONERROR, "Excel Failed", "Failed to attach to Excel")

    Local $iNum
    Time()
    Local $aData = _Excel_RangeRead($oBook, $sSheet, "A1:B6", 1, True)
    Time()
    _ArrayDisplay($aData)
    For $iRow = 0 To 5
        Time()
        Local $iNum = Number($aData[$iRow][0])
        If $iNum = 1 Then
            ConsoleWrite("Row " & $iRow + 1 & " is 1 and value of column B is " & $aData[$iRow][0])
        Else
            ConsoleWrite("Row " & $iRow + 1 & " is not 1")
        EndIf
        ConsoleWrite(". Reading took: ")
        Time()
    Next
    ConsoleWrite("The whole operation took " & $iTotalTime & " milliseconds." & @CRLF)
EndFunc

Func Time()
    Local Static $hTimer
    Local Static $bRunning = False
    If $bRunning Then
        Local $iTime = Round(TimerDiff($hTimer), 2)
        $iTotalTime += $iTime
        ConsoleWrite($iTime & @CRLF)
    Else
        $hTimer = TimerInit()
    EndIf
    $bRunning = Not $bRunning
EndFunc
24.44
Row 1 is 1 and value of column B is 1. Reading took: 0.04
Row 2 is not 1. Reading took: 0.01
Row 3 is 1 and value of column B is 1. Reading took: 0.01
Row 4 is 1 and value of column B is 1. Reading took: 0.02
Row 5 is not 1. Reading took: 0.01
Row 6 is 1 and value of column B is 1. Reading took: 0.01
The whole operation took 24.54 milliseconds.

From 1500 ms to 120 ms, and finally to 25 ms, that is an astonishing 60x improvement in total :D

The excel function for transposing must really suck... or it accounts for a lot of things that our array transpose function cannot. Either way, I am happy with the outcome.


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By VinMe
      Dear all, i am unable to open a xml file to excel in the "xml table format" Please help me out in where i am missing
      Local $strFileToOpen = _WinAPI_OpenFileDlg('Select xml file', @WorkingDir, 'All Files(*.*)', 1, '', '', BitOR($OFN_PATHMUSTEXIST, $OFN_FILEMUSTEXIST, $OFN_HIDEREADONLY)) Global $xlXmlLoadImportToList = 2 ; Places the contents of the XML data file in an XML table $oExcel = _Excel_Open() $oWorkbook1=$oExcel.Workbooks.OpenXML($strFileToOpen, "", $xlXmlLoadImportToList) If $strFileToOpen <> False Then     Local $oWorkbook1 = _Excel_BookOpen($oExcel, $strFileToOpen) EndIf Error i am getting is:
      ......\81e_Compare_v1.au3" (46) : ==> The requested action with this object has failed.:
      $oWorkbook1=$oExcel.Workbooks.OpenXML($strFileToOpen, "", $xlXmlLoadImportToList)
      $oWorkbook1=$oExcel.Workbooks^ ERROR
      >Exit code: 1    Time: 7.338
    • By VinMe
      Dear all, 
      I am unable to get the right result after applying the filter to the excel. please let me know on the same.
      issue: After applying the filter the output $lastRow11 not giving the right output of complete visible rows. (its breaking at row skips)
       
      ;DATA EXTRACTION FROM LOC EXCEL
      ;=============================================================================
      $oWorkbook = _Excel_BookAttach($sWorkbook)
      Local $sMSN = InputBox("MSN NO", "Enter MSN in XX FORMAT", "")
      ;~ Local $LastRow1 = ($oWorkbook.ACTIVESHEET.Range("A1").SpecialCells($xlCellTypeLastCell).Row)
      $LastRow1 = $oWorkbook.ActiveSheet.UsedRange.Rows.Count
      MsgBox(0, "lastrow1", $LastRow1)
      _Excel_FilterSet($oWorkbook, $oWorkbook.activesheet, "AF1", 32, "*" & $sMSN & "*")
      Local $oLocDS = $oWorkbook.ActiveSheet.Range("S1:S" & $LastRow1).SpecialCells($xlCellTypeVisible)
      Local $LastRow11 = $oLocDS.rows.count    ;error output
      MsgBox(0, "lastrow11", $LastRow11)
      Local $aLocDS1 = _Excel_RangeRead($oWorkbook, Default, $oLocDS)
      Local $oLocNr = $oWorkbook.ActiveSheet.Range("A1:A" & $LastRow1).SpecialCells($xlCellTypeVisible)
      Local $aLocNr1 = _Excel_RangeRead($oWorkbook, Default, $oLocNr)
      _ArrayDisplay($aLocDS1)
      _ArrayDisplay($aLocNr1)
      _ArrayTrim($aLocDS1, 6, 1)
      _ArrayTrim($aLocNr1, 6, 1)
      _ArrayTrim($aLocNr1, 6, 0)
      _ArrayDisplay($aLocDS1)
      _ArrayDisplay($aLocNr1)
    • By VinMe
      I am unable to execute the below script, my requirement is to copy the content from active excel sheet and to display the same.
      Please let me know where i am missing!
      #include <Excel.au3>
      #include <MsgBoxConstants.au3>
      #include <Array.au3>
      #include <StringConstants.au3>
      Local $oExcel = _Excel_Open()
      $LastRow2 = $oExcel.UsedRange.Rows.Count
      $Tissue = _Excel_RangeRead($oExcel, Default, "E1:E" & $LastRow2)
      $TshNr = _Excel_RangeRead($oExcel, Default, "F1:F" & $LastRow2)
      _ArrayDisplay($Tissue)
      _ArrayDisplay($TshNr)
    • By _leo_
      Hey there! 😃
      I am having a problem with the _Excel_RangeFind. I am trying to search for a value in a particular cell range. The script copies the value from the internet. Copying and saving as a variable is working fine, but as soon as it should find the value in excel, nothing happens. ( I am not getting an error)
      #include <Excel.au3> Func Excel() Send("{CTRLDOWN}") Send("{c}") Send("{CTRLUP}") Local $sName = ClipGet() ;Text Local $sShortName = StringTrimRight ( $sName, 1) ;delete one letter Local $bOpenWorkBook = False, $oExcel = _Excel_Open() Local $sFilePath = "C:\Users\Acer\OneDrive\xyz.xlsx" Local $oWorkbook $oWorkbook = _Excel_BookAttach($sFilePath) If @error Then $oWorkbook = _Excel_BookOpen($oExcel, $sFilePath) $bOpenWorkBook = True EndIf sleep(15000) Send("{LWINDown}") Send ("{up}") ;maximize window Send("{LWINup}") sleep(1000) _Excel_RangeFind ($oWorkbook, $sShortName, "A3:A56") EndFunc Is anyone familiar with this problem or am I just missing some basic stuff? 
      Thanks for help!
    • By _leo_
      Hey guys
      I'm new to autoit, so this could be a simple question. I'm trying to read the value from the currently selected cell in Excel. I read on the forum and tired to find videos, but I couldn't quite get to it. 
      This is what I have got so far:
       
      Local $oExcel_1 = _Excel_Open()
      Local $var = "C:\Users\Acer\xy"
      Local $oWorkbook = _Excel_BookOpen($oExcel_1,$var)
       
      Local $_read1 = _Excel_RangeRead($oWorkbook, Default.Application.ActiveCell.Address)
       
      Whatever I try, I either get an error or it only reads "0".
       
      Thank you very much for any helpful thoughts!
×
×
  • Create New...