Sign in to follow this  
Followers 0
Bowmore

Fast Array Write To Excel

15 posts in this topic

#1 ·  Posted (edited)

I have some scripts that use Excel to present the result of data validation tests and I was becoming a little frustrated at the length of time it was taking to write the data to excel, using _ExcelWriteSheetFromArray() function in the Excel.au3 UDF. So I thought I'd take a look and see if I could improve on it's performance. This was the result. This new function is 200 to 300 times faster than _ExcelWriteSheetFromArray() for large arrays (20000 plus elements) and in the region of 100 times faster for small arrays.

I have also included the option to write only part of the array to the spreadsheet as sometimes I do not need all of the array.

_ExcelSheetWriteFromArray

; #FUNCTION# ====================================================================================================================
; Name...........: _ExcelSheetWriteFromArray
; Description ...: Writes a 2D array, or part of, to a specified location in the active worksheet
; Syntax.........: _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
; Parameters ....: $oExcel - Excel object opened by a preceding call to _ExcelBookOpen() or _ExcelBookNew()
;                $aArray - The array ByRef to write data from (array is not modified)
;                $iExcelStartRow - The spreadsheet row to start writing the array to, default is 1
;                $iExcelStartCol - The spreadsheet column to start writing the array to, default is 1
;                $iArrayStartRow - Start row index of array to start writing data from, default is 0
;                $iArrayStartCol - Start column index of array to start writing data from, default is 0
;                $iArrayEndRow - End row index of array to stop writing data from, default is Ubound($aArray)
;                $iArrayEndCol - End column index of array to stop writing data from, default is Ubound($aArray,2)
; Return values .: Success   - Returns 1
;                Failure - Returns 0 and sets @error on errors:
;                |@error=1   - Excel object does not exist
;                |@error=2   - Not a 2D array
;                |@error=3   - End row greater than start row
;                |@error=4   - End col greater than start col
; Author ........: Bowmore
; Modified.......:
; Remarks .......: Only 2D arrays are currently handeled. Maxim cell content length is 255 characters
; Related .......:
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
Local $iExcelEndRow = 1
Local $iExcelEndCol = 1

If Not IsObj($oExcel) Then Return SetError(1, 0, 0)
If UBound($aArray, 0) <> 2 Then Return SetError(2, 0, 0)
If $iExcelStartRow = Default Then $iExcelStartRow = 1
If $iExcelStartCol = Default Then $iExcelStartCol = 1
If $iArrayStartRow = Default Then $iArrayStartRow = 0
If $iArrayStartCol = Default Then $iArrayStartCol = 0
If $iArrayEndRow = Default Then $iArrayEndRow = UBound($aArray) - 1
If $iArrayEndCol = Default Then $iArrayEndCol = UBound($aArray, 2) - 1
If $iExcelStartRow < 1 Then $iExcelStartRow = 1
If $iExcelStartCol < 1 Then $iExcelStartCol = 1
If $iArrayStartRow < 0 Then $iArrayStartRow = 0
If $iArrayStartCol < 0 Then $iArrayStartCol = 0
If $iArrayEndRow < $iArrayStartRow Then Return SetError(3, 1, 0)
If $iArrayEndCol < $iArrayStartCol Then Return SetError(4, 1, 0)
$iExcelEndRow = $iExcelStartRow + $iArrayEndRow - $iArrayStartRow
$iExcelEndCol = $iExcelStartCol + $iArrayEndCol - $iArrayStartCol

; Check if only part of the array is to written to the speadsheet
If $iArrayStartRow <> 0 Or $iArrayStartCol <> 0 Or $iArrayEndRow <> UBound($aArray) - 1 Or $iArrayEndCol = UBound($aArray, 2) - 1 Then
;Copy specified array range to a temporary array
Local $aTemp[$iArrayEndRow - $iArrayStartRow + 1][$iArrayEndCol - $iArrayStartCol + 1]
Local $iRow = 0
Local $iCol = 0
For $i = $iArrayStartRow To $iArrayEndRow
$iCol = 0
For $j = $iArrayStartCol To $iArrayEndCol
$aTemp[$iRow][$iCol] = $aArray[$i][$j]
$iCol += 1
Next
$iRow += 1
Next
With $oExcel.ActiveSheet
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aTemp)
EndWith
Else
With $oExcel.ActiveSheet
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aArray)
EndWith
EndIf

EndFunc ;==>_ExcelSheetWriteFromArray

Simple Speed Test _ExcelSheetWriteFromArray

#include <excel.au3>

Test_Main()

Func Test_Main()
Local $sXlsFile1 = @ScriptDir & "Test1.xls"
Local $sXlsFile2 = @ScriptDir & "Test2.xls"
Local $oExcel = 0
Local $Tim = 0
Local $aData[5000][10]

For $i = 0 To 4999
$aData[$i][0] = "Y"
$aData[$i][1] = "DATA_TEST_ROW_0" & StringFormat("%03i", $i)
$aData[$i][2] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum ultrices est varius elit consequat sollicitudin mollis libero ultrices. Quisque lobortis lorem quis nunc vestibulum tincidunt. Duis eget urna sapien, nec egestas purus. Fusce massa nunc."
$aData[$i][3] = $i * 10
$aData[$i][4] = $i * 100
$aData[$i][5] = $aData[$i][3] / $aData[$i][4]
$aData[$i][6] = "Pass"
$aData[$i][7] = @YEAR & "-" & @MON & "-" & @MDAY & "T" & @HOUR & ":" & @MIN
$aData[$i][8] = "FAIL"
$aData[$i][9] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum ultrices est varius elit consequat sollicitudin mollis libero ultrices. Quisque lobortis lorem quis nunc vestibulum tincidunt. Duis eget urna sapien, nec egestas purus. Fusce massa nunc."
Next

; Speed Test of new function
$oExcel = _ExcelBookNew(0)
$Tim = TimerInit()
_ExcelSheetWriteFromArray($oExcel, $aData)
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : TimerDiff($Tim) = ' & TimerDiff($Tim) & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console
_ExcelBookSaveAs($oExcel, $sXlsFile1)
_ExcelBookClose($oExcel, 0, 0)

; Speed Test of old function
$oExcel = _ExcelBookNew(0)
$Tim = TimerInit()
_ExcelWriteSheetFromArray($oExcel, $aData)
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : TimerDiff($Tim) = ' & TimerDiff($Tim) & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console
_ExcelBookSaveAs($oExcel, $sXlsFile2)
_ExcelBookClose($oExcel, 0, 0)

EndFunc ;==>Test_Main


; #FUNCTION# ====================================================================================================================
; Name...........: _ExcelSheetWriteFromArray
; Description ...: Writes a 2D array, or part of, to a specified location in the active worksheet
; Syntax.........: _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
; Parameters ....: $oExcel - Excel object opened by a preceding call to _ExcelBookOpen() or _ExcelBookNew()
;                $aArray - The array ByRef to write data from (array is not modified)
;                $iExcelStartRow - The spreadsheet row to start writing the array to, default is 1
;                $iExcelStartCol - The spreadsheet column to start writing the array to, default is 1
;                $iArrayStartRow - Start row index of array to start writing data from, default is 0
;                $iArrayStartCol - Start column index of array to start writing data from, default is 0
;                $iArrayEndRow - End row index of array to stop writing data from, default is Ubound($aArray)
;                $iArrayEndCol - End column index of array to stop writing data from, default is Ubound($aArray,2)
; Return values .: Success   - Returns 1
;                Failure - Returns 0 and sets @error on errors:
;                |@error=1   - Excel object does not exist
;                |@error=2   - Not a 2D array
;                |@error=3   - End row greater than start row
;                |@error=4   - End col greater than start col
; Author ........: Bowmore
; Modified.......:
; Remarks .......: Only 2D arrays are currently handeled. Maxim cell content length is 255 characters
; Related .......:
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
Local $iExcelEndRow = 1
Local $iExcelEndCol = 1

If Not IsObj($oExcel) Then Return SetError(1, 0, 0)
If UBound($aArray, 0) <> 2 Then Return SetError(2, 0, 0)
If $iExcelStartRow = Default Then $iExcelStartRow = 1
If $iExcelStartCol = Default Then $iExcelStartCol = 1
If $iArrayStartRow = Default Then $iArrayStartRow = 0
If $iArrayStartCol = Default Then $iArrayStartCol = 0
If $iArrayEndRow = Default Then $iArrayEndRow = UBound($aArray) - 1
If $iArrayEndCol = Default Then $iArrayEndCol = UBound($aArray, 2) - 1
If $iExcelStartRow < 1 Then $iExcelStartRow = 1
If $iExcelStartCol < 1 Then $iExcelStartCol = 1
If $iArrayStartRow < 0 Then $iArrayStartRow = 0
If $iArrayStartCol < 0 Then $iArrayStartCol = 0
If $iArrayEndRow < $iArrayStartRow Then Return SetError(3, 1, 0)
If $iArrayEndCol < $iArrayStartCol Then Return SetError(4, 1, 0)
$iExcelEndRow = $iExcelStartRow + $iArrayEndRow - $iArrayStartRow
$iExcelEndCol = $iExcelStartCol + $iArrayEndCol - $iArrayStartCol

; Check if only part of the array is to written to the speadsheet
If $iArrayStartRow <> 0 Or $iArrayStartCol <> 0 Or $iArrayEndRow <> UBound($aArray) - 1 Or $iArrayEndCol = UBound($aArray, 2) - 1 Then
;Copy specified array range to a temporary array
Local $aTemp[$iArrayEndRow - $iArrayStartRow + 1][$iArrayEndCol - $iArrayStartCol + 1]
Local $iRow = 0
Local $iCol = 0
For $i = $iArrayStartRow To $iArrayEndRow
$iCol = 0
For $j = $iArrayStartCol To $iArrayEndCol
$aTemp[$iRow][$iCol] = $aArray[$i][$j]
$iCol += 1
Next
$iRow += 1
Next
With $oExcel.ActiveSheet
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aTemp)
EndWith
Else
With $oExcel.ActiveSheet
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aArray)
EndWith
EndIf

EndFunc ;==>_ExcelSheetWriteFromArray
Edited by Bowmore
1 person likes this

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

There is a ticket in BugTracker for a new version of _ExcelReadSheetFromArray() that also eliminates the nested loops that individually access Excel cells in favor of using the Range.Cells method. It shows a similar speed increase.

I would think if you reverted back to the production function name, switched both the $iRowBase ($iArrayStartRow) and $iColBase ($iArrayStartCol) parameters back to defaulting to 1, instead of 0, and maintained the existing error return values (with some of your own additions), that you'd have a backward-compatible function (with 2 new parms) that could replace the one in the production UDF?

Share this post


Link to post
Share on other sites

Thanks for the suggestions. Although I wrote it to meet my own specific needs I would make sense to change the version I posted hear to make it compatible with the existing _ExcelWriteSheetFromArray(). I'll change it in the next couple of days when I have half an hour to make the changes and test it properly.


"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

Hi,

Your Fast Array Write To Excel Script rocks, but only for 65536 rows, if I put one more 65537 I get an error:

: ==> The requested action with this object has failed.:

.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aTemp)

.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aTemp)^ ERROR

can that be fixed?

Share this post


Link to post
Share on other sites

I'm about to rewrite the Excel UDF that comes with AutoIt. It's an early Alpha version but your problem has been solved by GMX's (it is named _Excel_RangeRead).

If you can test, please post the results in the Excel Rewrite thread.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Actually, I need to make changes to the _Excel_RangeWrite function as well. Will post it in the Excel Rewrite thread as soon as I've completed it.

Share this post


Link to post
Share on other sites

I agree with GMK, I have tested _ExcelSheetWriteFromArray with Excel Rewrite.au3 and 65536 issue is still present.

Waiting for changes to the _Excel_RangeWrite function...

Share this post


Link to post
Share on other sites

There appears to be a limit of 65536 cells when using the .transpose method at least up to Excel 2007, might even go beyond that, but I'm not sure.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

I find this post on mrexcel.com forum: http://www.mrexcel.com/forum/excel-questions/555073-limitation-range-size-transpose-function.html#post2742571

(The Transpose function just doesn't work with >65536, just as SpecialCells or AdvancedFilter etc.methods sometimes don't work with ranges above a certain size. You'd have to ask the Microsoft coders why they haven't updated these to allow for the larger sheet size of Excel 2007 and subsequent versions.)

I really don't know is this comment form mrexcel.com forum true or false, but I find one not so pretty but working solution for example in our topic

#include <excel.au3>

Test_Main()

Func Test_Main()
Local $oExcel = 0
Local $Tim = 0
Local $aData[65536][1]
Local $aData1[65536][1]

For $i = 0 To 65535
$aData[$i][0] = "DATA_TEST_ROW_0" & StringFormat("%03i", $i+1)
Next

For $j = 0 To 65535
$aData1[$j][0] = "DATA_TEST_ROW_0" & StringFormat("%03i", $j+65537)
Next

; Speed Test of new function
$oExcel = _ExcelBookNew()
$Tim = TimerInit()
_ExcelSheetWriteFromArray($oExcel, $aData)
_ExcelSheetWriteFromArray($oExcel, $aData1, 65537, 1)
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : TimerDiff($Tim) = ' & TimerDiff($Tim) & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

EndFunc ;==>Test_Main


; #FUNCTION# ====================================================================================================================
; Name...........: _ExcelSheetWriteFromArray
; Description ...: Writes a 2D array, or part of, to a specified location in the active worksheet
; Syntax.........: _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
; Parameters ....: $oExcel - Excel object opened by a preceding call to _ExcelBookOpen() or _ExcelBookNew()
;            $aArray - The array ByRef to write data from (array is not modified)
;            $iExcelStartRow - The spreadsheet row to start writing the array to, default is 1
;            $iExcelStartCol - The spreadsheet column to start writing the array to, default is 1
;            $iArrayStartRow - Start row index of array to start writing data from, default is 0
;            $iArrayStartCol - Start column index of array to start writing data from, default is 0
;            $iArrayEndRow - End row index of array to stop writing data from, default is Ubound($aArray)
;            $iArrayEndCol - End column index of array to stop writing data from, default is Ubound($aArray,2)
; Return values .: Success - Returns 1
;            Failure - Returns 0 and sets @error on errors:
;            |@error=1 - Excel object does not exist
;            |@error=2 - Not a 2D array
;            |@error=3 - End row greater than start row
;            |@error=4 - End col greater than start col
; Author ........: Bowmore
; Modified.......:
; Remarks .......: Only 2D arrays are currently handeled. Maxim cell content length is 255 characters
; Related .......:
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
Local $iExcelEndRow = 1
Local $iExcelEndCol = 1

If Not IsObj($oExcel) Then Return SetError(1, 0, 0)
If UBound($aArray, 0) <> 2 Then Return SetError(2, 0, 0)
If $iExcelStartRow = Default Then $iExcelStartRow = 1
If $iExcelStartCol = Default Then $iExcelStartCol = 1
If $iArrayStartRow = Default Then $iArrayStartRow = 0
If $iArrayStartCol = Default Then $iArrayStartCol = 0
If $iArrayEndRow = Default Then $iArrayEndRow = UBound($aArray) - 1
If $iArrayEndCol = Default Then $iArrayEndCol = UBound($aArray, 2) - 1
If $iExcelStartRow < 1 Then $iExcelStartRow = 1
If $iExcelStartCol < 1 Then $iExcelStartCol = 1
If $iArrayStartRow < 0 Then $iArrayStartRow = 0
If $iArrayStartCol < 0 Then $iArrayStartCol = 0
If $iArrayEndRow < $iArrayStartRow Then Return SetError(3, 1, 0)
If $iArrayEndCol < $iArrayStartCol Then Return SetError(4, 1, 0)
$iExcelEndRow = $iExcelStartRow + $iArrayEndRow - $iArrayStartRow
$iExcelEndCol = $iExcelStartCol + $iArrayEndCol - $iArrayStartCol

; Check if only part of the array is to written to the speadsheet
If $iArrayStartRow <> 0 Or $iArrayStartCol <> 0 Or $iArrayEndRow <> UBound($aArray) - 1 Or $iArrayEndCol = UBound($aArray, 2) - 1 Then
;Copy specified array range to a temporary array
Local $aTemp[$iArrayEndRow - $iArrayStartRow + 1][$iArrayEndCol - $iArrayStartCol + 1]
Local $iRow = 0
Local $iCol = 0
For $i = $iArrayStartRow To $iArrayEndRow
$iCol = 0
For $j = $iArrayStartCol To $iArrayEndCol
$aTemp[$iRow][$iCol] = $aArray[$i][$j]
$iCol += 1
Next
$iRow += 1
Next
With $oExcel.ActiveSheet
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aTemp)
EndWith
Else
With $oExcel.ActiveSheet
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
.Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aArray)
EndWith
EndIf

EndFunc ;==>_ExcelSheetWriteFromArray

Share this post


Link to post
Share on other sites

A better way would be to do something like this:

#include <excel.au3>

Test_Main()


; #FUNCTION# ====================================================================================================================
; Name...........: _ExcelSheetWriteFromArray
; Description ...: Writes a 2D array, or part of, to a specified location in the active worksheet
; Syntax.........: _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
; Parameters ....: $oExcel - Excel object opened by a preceding call to _ExcelBookOpen() or _ExcelBookNew()
;            $aArray - The array ByRef to write data from (array is not modified)
;            $iExcelStartRow - The spreadsheet row to start writing the array to, default is 1
;            $iExcelStartCol - The spreadsheet column to start writing the array to, default is 1
;            $iArrayStartRow - Start row index of array to start writing data from, default is 0
;            $iArrayStartCol - Start column index of array to start writing data from, default is 0
;            $iArrayEndRow - End row index of array to stop writing data from, default is Ubound($aArray)
;            $iArrayEndCol - End column index of array to stop writing data from, default is Ubound($aArray,2)
; Return values .: Success - Returns 1
;            Failure - Returns 0 and sets @error on errors:
;            |@error=1 - Excel object does not exist
;            |@error=2 - Not a 2D array
;            |@error=3 - End row greater than start row
;            |@error=4 - End col greater than start col
; Author ........: Bowmore
; Modified.......:
; Remarks .......: Only 2D arrays are currently handeled. Maxim cell content length is 255 characters
; Related .......:
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _ExcelSheetWriteFromArray($oExcel, ByRef $aArray, $iExcelStartRow = Default, $iExcelStartCol = Default, $iArrayStartRow = Default, $iArrayStartCol = Default, $iArrayEndRow = Default, $iArrayEndCol = Default)
    Local $iExcelEndRow = 1
    Local $iExcelEndCol = 1

    If Not IsObj($oExcel) Then Return SetError(1, 0, 0)
    If UBound($aArray, 0) <> 2 Then Return SetError(2, 0, 0)
    If $iExcelStartRow = Default Then $iExcelStartRow = 1
    If $iExcelStartCol = Default Then $iExcelStartCol = 1
    If $iArrayStartRow = Default Then $iArrayStartRow = 0
    If $iArrayStartCol = Default Then $iArrayStartCol = 0
    If $iArrayEndRow = Default Then $iArrayEndRow = UBound($aArray) - 1
    If $iArrayEndCol = Default Then $iArrayEndCol = UBound($aArray, 2) - 1
    If $iExcelStartRow < 1 Then $iExcelStartRow = 1
    If $iExcelStartCol < 1 Then $iExcelStartCol = 1
    If $iArrayStartRow < 0 Then $iArrayStartRow = 0
    If $iArrayStartCol < 0 Then $iArrayStartCol = 0
    If $iArrayEndRow < $iArrayStartRow Then Return SetError(3, 1, 0)
    If $iArrayEndCol < $iArrayStartCol Then Return SetError(4, 1, 0)
    If $iArrayEndRow > UBound($aArray) - 1 Then $iArrayEndRow = UBound($aArray) - 1 ; <<<<<<<<<<<<<<< added this line, because there's no check to see if the $iArrayEndRow is larger than the end of the array
    $iExcelEndRow = $iExcelStartRow + $iArrayEndRow - $iArrayStartRow
    $iExcelEndCol = $iExcelStartCol + $iArrayEndCol - $iArrayStartCol

    ; Check if only part of the array is to written to the speadsheet
    If $iArrayStartRow <> 0 Or $iArrayStartCol <> 0 Or $iArrayEndRow <> UBound($aArray) - 1 Or $iArrayEndCol = UBound($aArray, 2) - 1 Then
        ;Copy specified array range to a temporary array
        Local $aTemp[$iArrayEndRow - $iArrayStartRow + 1][$iArrayEndCol - $iArrayStartCol + 1]
        Local $iRow = 0
        Local $iCol = 0
        For $i = $iArrayStartRow To $iArrayEndRow
            $iCol = 0
            For $j = $iArrayStartCol To $iArrayEndCol
                $aTemp[$iRow][$iCol] = $aArray[$i][$j]
                $iCol += 1
            Next
            $iRow += 1
        Next
        With $oExcel.ActiveSheet
            .Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
            .Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aTemp)
        EndWith
    Else
        With $oExcel.ActiveSheet
            .Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).Select
            .Range(.Cells($iExcelStartRow, $iExcelStartCol), .Cells($iExcelEndRow, $iExcelEndCol)).value = $oExcel.Application.WorksheetFunction.Transpose($aArray)
        EndWith
    EndIf

EndFunc   ;==>_ExcelSheetWriteFromArray

Func Test_Main()
    Local $oExcel = 0
    Local $Tim = 0
    Local $aData[131072][1]

    For $i = 0 To 131071
        $aData[$i][0] = "DATA_TEST_ROW_0" & StringFormat("%03i", $i + 1)
    Next
    ; Speed Test of new function
    $oExcel = _ExcelBookNew()
    $Tim = TimerInit()
    ; This only writes 65535 elements of the array at a time, if your array is larger than that, then it will split it up
    ; for you.
    For $loop = 0 To UBound($aData) - 1 Step 65536 
        _ExcelSheetWriteFromArray($oExcel, $aData, $loop + 1, 0, $loop, 0, $Loop + 65535)
    Next
    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : TimerDiff($Tim) = ' & TimerDiff($Tim) & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

EndFunc   ;==>Test_Main

This rewrites the test script that tempman posted above and also corrects a bug in the _ExcelSheetWriteFromArray function if you send it an end row of the array that is beyond the UBound dimension of the array.

This will write 65535 rows to the spreadsheet until you've hit the end of the array, and the array can be of any size either larger or smaller than 65535 rows.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites
updated.

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

Thanks GMK!

I hope to release a new alpha version with your changes quite soon.

Edited by water

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

I posted a revised version of _Excel_RangeWrite for you to test.

Would like to hear from Excel 2003 testers as I only can test with Excel 2010.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Hi

Is there a script for Fast Array Write from Excel? I see that it takes a lot of time for this as well - when using big Excel files.

Thank you!

Share this post


Link to post
Share on other sites

Welcome to AutoIt and the forum,

the rewrite of the Excel UDF I'm currently working on has a _Excel_RangeRead function too.

Please check the for the latest download. But be aware that it is still an early Alpha version!

If you search the forum you will find some other fast _ExcelReadSheetToArray functions too.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • ShawnW
      By ShawnW
      I have a script that takes a large excel file, pulls out and reorganizes certain information I need, and spits out a trimmed down csv file which I uses to upload the information on my website. Some of this information contains characters with accents or em dashes. By default it would create a csv file in ANSI which I then uploaded but had to tell my website import system it was windows-1252 in order for it to look correct.
      This was all working fine except now I need to add in a non-breaking space and non-breaking hyphen into parts of my output. At first I tried using ChrW(0xA0) and ChrW(0x2011) as replacements. A quick test in the console looked correct, however opening the csv output in notepad++ showed the space correctly but a ? for the hyphen and the file was still encoded as ANSI. I tried to view it as UTF-8 instead but this just made the space appear as xAO and also other characters appeared that way like my em dashes appeared as x97 and another symbol as xA7 etc.
      If I instead do a convert to UTF-8 from notepad++ then those problems go away except the hyphen still displays as ?. I then noticed on the page I linked for the non-breaking hyphen it lists the UTF-8 hex as 0xE2 0x80 0x91 (e28091). I was unsure how to enter this in autoit but several things i tried all failed to get the hyphen inserted.
      I need a way to get both the space and hyphen added correctly as either ANSI or UTF-8, but if it is UTF-8 then I need a way to convert all of the other data I extracted from the excel file.
      I've included a test excel file with a single line and test script to create a csv demonstrating the problem.
      test.xlsx
      test.au3
    • Ascer
      By Ascer
      1. Description.
      Udf working with MSDN System.Collections.ArrayList. Allow you to make fast operations on huge arrays, speed is even x10 better than basic _ArrayAdd.  Not prefered for small arrays < 600 items. 2. Requirements
      .NET Framework 1.1 - 4.5 (on this version Microsoft destroy old rules) System Windows 3. Possibilities.
      ;=============================================================================================================== ; UDF Name: List.au3 ; ; Date: 2018-02-17, 10:52 ; Description: Simple udf to create System Collections as ArrayList and make multiple actions on them. ; ; Function(s): _ListCreate -> Creates a new list ; _ListCapacity -> Gets a list size in bytes ; _ListCount -> Gets items count in list ; _ListIsFixedSize -> Get bool if list if fixed size ; _ListIsReadOnly -> Get bool if list is read only ; _ListIsSynchronized -> Get bool if list is synchronized ; _ListGetItem -> Get item on index ; _ListSetItem -> Set item on index ; ; _ListAdd -> Add item at end of list ; _ListClear -> Remove all list items ; _ListClone -> Duplicate list in new var ; _ListContains -> Get bool if item is in list ; _ListGetHashCode -> Get hash code for list ; _ListGetRange -> Get list with items between indexs ; _ListIndexOf -> Get index of item ; _ListInsert -> Insert a new item on index ; _ListInsertRange -> Insert list into list on index ; _ListLastIndexOf -> Get index last of item ; _ListRemove -> Remove first found item ; _ListRemoveAt -> Remove item in index ; _ListRemoveRange -> Remove items between indexs ; _ListReverse -> Reverse all items in list ; _ListSetRange -> Set new value for items in range ; _ListSort -> Sort items in list (speed of reading) ; _ListToString -> Get list object name ; _ListTrimToSize -> Remove unused space in list ; ; Author(s): Ascer ;=============================================================================================================== 4. Downloads
      List.au3 5. Examples
      SpeedTest _ArrayAdd vs ListAdd SpeedTest ArraySearch vs ListIndexOf Basic usage - crating guild with members  
    • ronmage
      By ronmage
      So I have a loop that keeps reading data from an array and searching it for the same value. If the value is no there it does work then adds the value to the array to prevent it from doing the same work.
      If _ArraySearch($ID,$filearray[$i]) = -1 Then Work.... _ArrayAdd($ID,$filearray[$i]) EndIf This is in a for loop hence $i
      So what is happening is the code works great for several hours. After a period of time _ArraySearch($ID,$filearray[$i]) will result in -1 even if $ID = $filearray. So it ready as if there is no data in the array. Anyone have this problem? 
       
      Also I am just running in using F5 not compiling it and running it if that makes a difference.
       
    • FMS
      By FMS
      Hello,
      I'm trying to read a binary file to an array but couln't get it to work.
      Also I coul not find any help in the forum around this subject whish was helpfull.
      Is there any way it could be done?
      I tried a lot of ways but maybe somebody know's the right way?
      #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 #include <File.au3> #include <Array.au3> #include <AutoItConstants.au3> Local $in=FileOpen("TEST_labels.idx1-ubyte",16) ; 16+0=Read binary Local $data = FileRead($in) Local $FileArray = BinaryToString($data,4) ;~ $FileArray = StringSplit($BinarydData, @CRLF, 1+2) ;~ Local $FileArray = StringRegExp($BinarydData, "[^\r\n]+", 3) FileClose($in) _ArrayDisplay($FileArray,"$FileArray","",32) MsgBox($MB_SYSTEMMODAL, "", "$FileArray = " & $FileArray )  
      TEST_labels.idx1-ubyte
    • ur
      By ur
      I am reading a CSV file which is tab seperated as below.
      Local $aArray = FileReadToArray($file) And now, I am splitting this main array record wise so that Array contains internally another arrow to represent each row.
      For $i = 0 to (UBound($aArray) - 1) ;MsgBox(0,"",$aArray[$i]) $aArray[$i] = StringSplit(StringStripCR($aArray[$i]), Chr(9),2);Chr(9) for tab ;_ArrayDisplay($aArray[$i]) Next Afther that, _ArrayDIsplay is able to see the individual internal arrays.
      _ArrayDisplay($aArray[1]) But If I try to access the individual element of it as below.It is not showing any result.
      MsgBox(0,"",$aArray[1][1]) Any suggestion, below is the sample csv file.
      New Text Document.csv