Jump to content
TheDcoder

Optimizing reading data from Excel

Recommended Posts

Hello everyone, I am working on a project which requires reading a few values from Excel, the catch is that I need it to be very fast... unfortunatley I found out that read operations using the supplied Excel UDF are very slow, more than 150 ms for each operation on average :(

Here is my testing setup that I made:

#include <Excel.au3>
#include <MsgBoxConstants.au3>

Global $iTotalTime = 0

Test()

Func Test()
    Local $oExcel = _Excel_Open()
    Local $oBook = _Excel_BookAttach("Test.xlsx", "FileName", $oExcel)
    Local $sSheet = "Sheet1"
    If @error Then Return MsgBox($MB_ICONERROR, "Excel Failed", "Failed to attach to Excel")

    Local $iNum
    For $iRow = 1 To 6
        Time()
        Local $iNum = Number(_Excel_RangeRead($oBook, $sSheet, "A" & $iRow))
        If ($iNum = 1) Then
            ConsoleWrite("Row " & $iRow & " is 1 and value of column B is " & _Excel_RangeRead($oBook, $sSheet, "B" & $iRow))
        Else
            ConsoleWrite("Row " & $iRow & " is not 1")
        EndIf
        ConsoleWrite(". Reading took: ")
        Time()
    Next
    ConsoleWrite("The whole operation took " & $iTotalTime & " milliseconds." & @CRLF)
EndFunc

Func Time()
    Local Static $hTimer
    Local Static $bRunning = False
    If $bRunning Then
        Local $iTime = Round(TimerDiff($hTimer), 2)
        $iTotalTime += $iTime
        ConsoleWrite($iTime & @CRLF)
    Else
        $hTimer = TimerInit()
    EndIf
    $bRunning = Not $bRunning
EndFunc

And Test.xlsx in CSV format:

1,-1
-1,1
1,-1
1,1
-1,-1
1,1

Here is the actual xlsx but it should expire in a week: https://we.tl/t-EVkxGp1kc6

And finally output from my script:

Row 1 is 1 and value of column B is -1. Reading took: 276.06
Row 2 is not 1. Reading took: 163.36
Row 3 is 1 and value of column B is -1. Reading took: 302.58
Row 4 is 1 and value of column B is 1. Reading took: 294.65
Row 5 is not 1. Reading took: 152.33
Row 6 is 1 and value of column B is 1. Reading took: 284.92
The whole operation took 1473.9 milliseconds.

 

Taking ~1.5 seconds for reading 6 rows of data is bad for my script, which needs to run as fast as possible :(. It would be nice if I can bring this down to 100 ms somehow, I am not very experienced working with MS office so I thought about asking you folks for help and advice on how I can optimize my script to squeeze out every bit of performance that I can get from this script :D

 

Thanks for the help in advance!

Edited by SDL
excel -> Excel

A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

@SDL
Does this file need to be in XLSX or you can convert it in CSV format as you did above? :)


Click here to see my signature:

Spoiler

Thoughts:

  • I will always thank you for the time you spent for me.
    I'm here to ask, and from your response, I'd like to learn.
    By my knowledge, I can help someone else, and "that someone" could help in turn another, and so on.

/*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------*/

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites
7 minutes ago, FrancescoDiMuro said:

Does this file need to be in XLSX or you can convert it in CSV format as you did above? :)

Good question, the only reason I am using XLSX is because I need Excel to automatically update those values via an addon. Otherwise I would simply have used CSV or anything else which is appropriate :)

I gave the CSV because it is easier to paste it here... and I could not provide the xlsx file permanently without using up my attachment space.


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

As you are only reading a few rows/columns I would use _Excel_RangeRead to read the whole used range and then process the returned array.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2019-10-24 - Version 1.4.14.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2019-11-30 - Version 1.4.0.0) - Download - General Help & Support - Example Scripts - Wiki
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
PowerPoint (2017-06-06 - Version 0.0.5.0) - Download - General Help & Support
Excel - Example Scripts - Wiki
Word - Wiki
Task Scheduler (NEW 2019-12-03 - Version 1.5.1.0) - Download - General Help & Support - Wiki

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

@water & @kaisies I see, I will try adapting my script to reduce calls to RangeRead and see how it performs :)

Will report back once I test it!


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

I have modified my original script to read all the data once instead of one by one for every cell... and boy are the speed gains huge! :D

#include <Array.au3>
#include <Excel.au3>
#include <MsgBoxConstants.au3>

Global $iTotalTime = 0

Test()

Func Test()
    Local $oExcel = _Excel_Open()
    Local $oBook = _Excel_BookAttach("Test.xlsx", "FileName", $oExcel)
    Local $sSheet = "Sheet1"
    If @error Then Return MsgBox($MB_ICONERROR, "Excel Failed", "Failed to attach to Excel")

    Local $iNum
    Time()
    Local $aData = _Excel_RangeRead($oBook, $sSheet, "A1:B6")
    Time()
    _ArrayDisplay($aData)
    For $iRow = 0 To 5
        Time()
        Local $iNum = Number($aData[$iRow][0])
        If $iNum = 1 Then
            ConsoleWrite("Row " & $iRow + 1 & " is 1 and value of column B is " & $aData[$iRow][0])
        Else
            ConsoleWrite("Row " & $iRow + 1 & " is not 1")
        EndIf
        ConsoleWrite(". Reading took: ")
        Time()
    Next
    ConsoleWrite("The whole operation took " & $iTotalTime & " milliseconds." & @CRLF)
EndFunc

Func Time()
    Local Static $hTimer
    Local Static $bRunning = False
    If $bRunning Then
        Local $iTime = Round(TimerDiff($hTimer), 2)
        $iTotalTime += $iTime
        ConsoleWrite($iTime & @CRLF)
    Else
        $hTimer = TimerInit()
    EndIf
    $bRunning = Not $bRunning
EndFunc
120.15
Row 1 is 1 and value of column B is 1. Reading took: 0.07
Row 2 is not 1. Reading took: 0.02
Row 3 is 1 and value of column B is 1. Reading took: 0.02
Row 4 is 1 and value of column B is 1. Reading took: 0.02
Row 5 is not 1. Reading took: 0.01
Row 6 is 1 and value of column B is 1. Reading took: 0.02
The whole operation took 120.31 milliseconds.

From ~1500 ms to ~120 ms, that is a huge 12.5x improvement in speed :thumbsup:

And even most of that is the first initial call to Angered, definitely going to try this method of caching in my project... I actually feel a bit dumb not having tried this before asking :lol:


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

You learn something new each day :)


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2019-10-24 - Version 1.4.14.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2019-11-30 - Version 1.4.0.0) - Download - General Help & Support - Example Scripts - Wiki
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
PowerPoint (2017-06-06 - Version 0.0.5.0) - Download - General Help & Support
Excel - Example Scripts - Wiki
Word - Wiki
Task Scheduler (NEW 2019-12-03 - Version 1.5.1.0) - Download - General Help & Support - Wiki

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

I discovered another way to hugely cut up the speed... by using the inbuilt AutoIt transpose function! Just set $bForceFunc parameter to true and enjoy the gains :)

#include <Array.au3>
#include <Excel.au3>
#include <MsgBoxConstants.au3>

Global $iTotalTime = 0

Test()

Func Test()
    Local $oExcel = _Excel_Open()
    Local $oBook = _Excel_BookAttach("Test.xlsx", "FileName", $oExcel)
    Local $sSheet = "Sheet1"
    If @error Then Return MsgBox($MB_ICONERROR, "Excel Failed", "Failed to attach to Excel")

    Local $iNum
    Time()
    Local $aData = _Excel_RangeRead($oBook, $sSheet, "A1:B6", 1, True)
    Time()
    _ArrayDisplay($aData)
    For $iRow = 0 To 5
        Time()
        Local $iNum = Number($aData[$iRow][0])
        If $iNum = 1 Then
            ConsoleWrite("Row " & $iRow + 1 & " is 1 and value of column B is " & $aData[$iRow][0])
        Else
            ConsoleWrite("Row " & $iRow + 1 & " is not 1")
        EndIf
        ConsoleWrite(". Reading took: ")
        Time()
    Next
    ConsoleWrite("The whole operation took " & $iTotalTime & " milliseconds." & @CRLF)
EndFunc

Func Time()
    Local Static $hTimer
    Local Static $bRunning = False
    If $bRunning Then
        Local $iTime = Round(TimerDiff($hTimer), 2)
        $iTotalTime += $iTime
        ConsoleWrite($iTime & @CRLF)
    Else
        $hTimer = TimerInit()
    EndIf
    $bRunning = Not $bRunning
EndFunc
24.44
Row 1 is 1 and value of column B is 1. Reading took: 0.04
Row 2 is not 1. Reading took: 0.01
Row 3 is 1 and value of column B is 1. Reading took: 0.01
Row 4 is 1 and value of column B is 1. Reading took: 0.02
Row 5 is not 1. Reading took: 0.01
Row 6 is 1 and value of column B is 1. Reading took: 0.01
The whole operation took 24.54 milliseconds.

From 1500 ms to 120 ms, and finally to 25 ms, that is an astonishing 60x improvement in total :D

The excel function for transposing must really suck... or it accounts for a lot of things that our array transpose function cannot. Either way, I am happy with the outcome.


A cross-platform implementation of the AutoIt language

My contributions to the AutoIt Community ##AutoIt at freenode, real-time chat

3fHNZJ.gif

Spoiler

If I have hurt or offended you in anyway, Please accept my apologies, I never (regardless of the situation) intend to do that to anybody.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By Rskm
      Hi, I am using excel as input media for my program. The excel file (i tried with .xls, .xlsx and .xlsm format) has inputs which the autoit script reads during the run and performs few calculations. Some times (not always), after the run, when i try to open the excel file manually, the file doesnt open at all in excel. see the screenshot attached. However, if the execute the autoit script, the scripts still reads the existing data from that excel and performs the calcs. I copied the excel file to another computer and there too, it doesnt open.  So, after this, i cannot edit the excel forever (if i need to change any inputs). It is only this particular file that got affected. other excel files works normal.  What could be the problem here.  please help as this is a new challenge for me during my program development. 

    • By Taxyo
      Hi,
       
      I've been trying to automate modification of an excel file and the last thing I am stuck on is deleting all the rows where the value of Column 13 is 0. 
      I believe the error is due to me not fully understanding the syntax so this is where I'm stuck: 
       
      Func Hotkey2() Global $aUsedRange = _Excel_RangeRead($oWorkbook, 1) _ArrayDisplay($aUsedRange) For $iRow = UBound($aUsedRange) - 1 to 3 Step -1 If $aUsedRange[$iRow][13] = 0 Then _Excel_RangeDelete($oWorkbook.Worksheets(1), $aUsedRange[$iRow] & ":" & $aUsedRange[$iRow], default, 1) If @error Then Exit MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeDelete Example 2", "Error deleting rows." & @CRLF & "@error = " & @error & ", @extended = " & @extended) EndIf Next EndFunc  
      While my script properly locates the row which contains value 0 in Column 13, I am not sure how to set it to the corresponding row in the excel workbook?  My above experiment gives me $vRange error and I've been toying around with it to no avail. The only way I get the Script to delete a row is by actually specifying "4:4" or "6:8" etc. 
      Where am I going wrong?
       
      Thanks! 
    • By Most
      #include <Array.au3> #include <Excel.au3> #include <MsgBoxConstants.au3> ; Create application object and open an example workbook Local $oExcel = _Excel_Open() If @error Then Exit MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeRead Example", "Error creating the Excel application object." & @CRLF & "@error = " & @error & ", @extended = " & @extended) Local $oWorkbook = _Excel_BookOpen($oExcel, @ScriptDir & "\trans.xlsx") If @error Then MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeRead Example", "Error opening workbook '" & @ScriptDir & "\trans.xlsx'." & @CRLF & "@error = " & @error & ", @extended = " & @extended) _Excel_Close($oExcel) Exit EndIf ; ***************************************************************************** ; Read data from a single cell on the active sheet of the specified workbook ; ***************************************************************************** Local $sResult = _Excel_RangeRead($oWorkbook, Default, "A1") If @error Then Exit MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeRead Example 1", "Error reading from workbook." & @CRLF & "@error = " & @error & ", @extended = " & @extended) MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeRead Example 1", "Data successfully read." & @CRLF & "Value of cell A1: " & $sResult) Hi, all.
      Ok, here is the deal. I have simple excel file called trans.xlsx. It's located in the directory of script. In general i don't care where to store it. 
      What i do need is to open excel file and copy one by one numbers from cells. I've tried different ways, examples. But i only get error, says: error = 3, extended = 1. I saw different posts from different years. I even tried to use simple example from manual file. But always get error.

      In general my goal get numbers one by one and post it to let's say search filed in my PC one by one. Or to notepad (but one by one, in kind of loop). 
      I've learned how to copy or show in message box some info from other apps. But with excel i'm stuck. 

      I'm able to open needed window based on "title" of excel. But i don't succeed of copying info from cells. 

      Would be appreciate for any help. 
      So, in this code i'm trying at least to read from cell A1. Doesn't matter what Sheet. 

      I use Windows 10, Excel for Office 365. 
      Thank you in advance. 
    • By VinMe
      Dear all, i am unable to open a xml file to excel in the "xml table format" Please help me out in where i am missing
      Local $strFileToOpen = _WinAPI_OpenFileDlg('Select xml file', @WorkingDir, 'All Files(*.*)', 1, '', '', BitOR($OFN_PATHMUSTEXIST, $OFN_FILEMUSTEXIST, $OFN_HIDEREADONLY)) Global $xlXmlLoadImportToList = 2 ; Places the contents of the XML data file in an XML table $oExcel = _Excel_Open() $oWorkbook1=$oExcel.Workbooks.OpenXML($strFileToOpen, "", $xlXmlLoadImportToList) If $strFileToOpen <> False Then     Local $oWorkbook1 = _Excel_BookOpen($oExcel, $strFileToOpen) EndIf Error i am getting is:
      ......\81e_Compare_v1.au3" (46) : ==> The requested action with this object has failed.:
      $oWorkbook1=$oExcel.Workbooks.OpenXML($strFileToOpen, "", $xlXmlLoadImportToList)
      $oWorkbook1=$oExcel.Workbooks^ ERROR
      >Exit code: 1    Time: 7.338
    • By VinMe
      Dear all, 
      I am unable to get the right result after applying the filter to the excel. please let me know on the same.
      issue: After applying the filter the output $lastRow11 not giving the right output of complete visible rows. (its breaking at row skips)
       
      ;DATA EXTRACTION FROM LOC EXCEL
      ;=============================================================================
      $oWorkbook = _Excel_BookAttach($sWorkbook)
      Local $sMSN = InputBox("MSN NO", "Enter MSN in XX FORMAT", "")
      ;~ Local $LastRow1 = ($oWorkbook.ACTIVESHEET.Range("A1").SpecialCells($xlCellTypeLastCell).Row)
      $LastRow1 = $oWorkbook.ActiveSheet.UsedRange.Rows.Count
      MsgBox(0, "lastrow1", $LastRow1)
      _Excel_FilterSet($oWorkbook, $oWorkbook.activesheet, "AF1", 32, "*" & $sMSN & "*")
      Local $oLocDS = $oWorkbook.ActiveSheet.Range("S1:S" & $LastRow1).SpecialCells($xlCellTypeVisible)
      Local $LastRow11 = $oLocDS.rows.count    ;error output
      MsgBox(0, "lastrow11", $LastRow11)
      Local $aLocDS1 = _Excel_RangeRead($oWorkbook, Default, $oLocDS)
      Local $oLocNr = $oWorkbook.ActiveSheet.Range("A1:A" & $LastRow1).SpecialCells($xlCellTypeVisible)
      Local $aLocNr1 = _Excel_RangeRead($oWorkbook, Default, $oLocNr)
      _ArrayDisplay($aLocDS1)
      _ArrayDisplay($aLocNr1)
      _ArrayTrim($aLocDS1, 6, 1)
      _ArrayTrim($aLocNr1, 6, 1)
      _ArrayTrim($aLocNr1, 6, 0)
      _ArrayDisplay($aLocDS1)
      _ArrayDisplay($aLocNr1)
×
×
  • Create New...