Search the Community

Good Morning All, I have an application that takes csv files - sorts, finds, and tries to make it's own individual csv files... Everything runs smooth until I get to the "_myCSV2DCreator" on extremely large CSV files... like 200+ MB with 100,000 + lines. I get an "out of memory" error from AutoIT when I convert the 2D array to a CSV file using this function "_myCSV2DCreator". I get an "out of memory" error when running the _ArrayInsert($aNewArray, "0", $columnheaders, "", ","). I believe it takes whatever array is in memory and doubles it's size as it writes just one row of data. Any ideas on how to get around this? I'm stumped... Func GEN_SeparateOutCSV03() SendAndLog("GEN_SeparateOutCSV03", $tempzipdir & '\' & $LogFileName01, True) SplashTextOn($ProgramTitle, 'Separating CSV files from data...', 400, 60, (@DesktopWidth / 2) - 200, 10, "", "") Sleep ($sleeptime) $array01 = $twoDarray $aUniqueHostname = _ArrayUnique ($array01, 1) ;_ArrayDisplay ($aUniqueHostname, "UniqueHostname ") For $i01 = Ubound($aUniqueHostname) - 1 to 0 Step - 1 For $j01 = Ubound($array01) - 1 to 0 Step - 1 If $array01[$j01][1] == $aUniqueHostname[$i01] and StringRegExp($array01[$j01][5], "MY_VALUE.txt") then ;MsgBox(0, "Computer and MY_VALUE.txt", $aUniqueHostname[$i01] & " : " & $array01[$j01][5]) $FileName01 = $tempzipdir & "\" & $array01[$j01][3] & "_" & $aUniqueHostname[$i01] & "_" & $array01[$j01][2] & ".csv" ; MsgBox(0, "File Name", $FileName01) Local $avResult = _ArrayFindAll($array01, $aUniqueHostname[$i01], 0, 0, 0, 0, 1) ;_ArrayDisplay($avResult, "$avResult") Local $aNewArray = "" Local $aNewArray[UBound($avResult)][UBound($array01, 2)] SplashTextOn($ProgramTitle, 'Loop - Array search for unique hostname', 400, 60, (@DesktopWidth / 2) - 200, 10, "", "") Sleep ($sleeptime) For $i = 0 To UBound($avResult) -1 ; Loop through the returned index numbers. For $j = 0 To UBound($array01, 2) -1 ; Loop through each of the columns. $aNewArray[$i][$j] = $array01 [$avResult[$i]] [$j] ; Populate the new array. Next Next SplashTextOn($ProgramTitle, 'Loop - Column header modification', 400, 60, (@DesktopWidth / 2) - 200, 10, "", "") Sleep ($sleeptime) ;_ArrayDisplay($twoDarray, "Removed 'User-defined Rules:' in column 3") ;_ArrayInsert($aNewArray, "0", $columnheaders, "", ",") _ArrayInsert($aNewArray, "0", $columnheaders, "", ",") ;_ArrayDisplay($aNewArray, "Inserted 0") ;MsgBox (0, "Out of the loop", "Out of the loop - File Write From Array") _myCSV2DCreator($tempzipdir & "\beta_" & $array01[$j01][3] & "_" & $aUniqueHostname[$i01] & "_" & $array01[$j01][2] & ".csv", $aNewArray, True) ;_FileWriteFromArray($FileName01, $aNewArray) ExitLoop EndIf Next Next SplashTextOn($ProgramTitle, 'Please wait a moment...', 400, 60, (@DesktopWidth / 2) - 200, 10, "", "") Sleep ($sleeptime) SplashOff() SendAndLog("GEN_SeparateOutCSV03 - Okay", $tempzipdir & '\' & $LogFileName01, True) MsgBox(262192, $ProgramTitle, "All files stored here:" & @CR & @CR & $tempzipdir) SendAndLog("Final Message Box - Exit Okay", $tempzipdir & '\' & $LogFileName01, True) Exit EndFunc Func _myCSV2DCreator($hFile, $avArray, $bEraseCreate = True) SplashTextOn($ProgramTitle, '2D to CSV file create', 400, 60, (@DesktopWidth / 2) - 200, 10, "", "") Sleep ($sleeptime) If $bEraseCreate Then FileClose(FileOpen($hFile, 2)) Local $sHoldString = "" For $x = 0 To UBound($avArray) - 1 For $i = 0 To UBound($avArray, 2) - 1 $sHoldString &= $avArray[$x][$i] & "," Next $sHoldString &= @CRLF Next Return FileWrite($hFile, StringTrimRight($sHoldString, 3)) EndFunc (out of memory) Is there a better method / solution that I should be using? Thanks Everyone!