A set of question for file

[WolfWorld]

Ok if we read file that lager than 2147483647 characters into a string what will happen if it will be an error what other ways can we do.

and is there anyother ways to read a file faster than fileread

THANKS athiwatc

[/dev/null]

Ok if we read file that lager than 2147483647 characters into a string what will happen if it will be an error what other ways can we do.

and is there anyother ways to read a file faster than fileread

THANKS athiwatc

2147483647 ??? Please explain.....

[PsaltyDS]

The help file says the max string length is 2GB, which is 2^31 - 1 = 2147483647.

I don't know exactly what happens if you try to cross the line...

Well... that was interesting. AutoIt crashed with "Error allocating memory", but it happened much sooner than expected:

$sString = "RYRYRYRYRYRYRYRYRYRYRYRYRYRYRYRY" ; 32bytes = 2^5
For $n = 6 To 33
    $sString &= $sString
    ConsoleWrite($n & ": $sString = " & StringLen($sString) & @LF)
Next

Output:

>Running:(3.2.10.0):C:\Program Files\AutoIt3\autoit3.exe "C:\Temp\Test1.au3"    
6: $sString = 64
7: $sString = 128

; etc...

26: $sString = 67108864
27: $sString = 134217728
->13:35:55 AutoIT3.exe ended.rc:1

That failed at about 128MB.

Edited January 22, 2008 by PsaltyDS

[weaponx]

Remember what happened to the Death Star?

[weaponx]

Well... that was interesting. AutoIt crashed with "Error allocating memory", but it happened much sooner than expected:

$sString = "RYRYRYRYRYRYRYRYRYRYRYRYRYRYRYRY" ; 32bytes = 2^5
For $n = 6 To 33
    $sString &= $sString
    ConsoleWrite($n & ": $sString = " & StringLen($sString) & @LF)
NextoÝ÷ Øë­¦ë¡×ºÙ§)àçÎ4ßmtÓyð.|ÓÝ®+jabë4÷`.¶-ßOvjëhÝÞÅíø4÷dÞ=Ù7¬·V®ß~:çÍ7êÄ­®)à

Edit: Code fixed

Edited January 22, 2008 by weaponx

[PsaltyDS]

Not sure what you did but this code gets to 2048 mb just fine (by fine I mean my system was very slow afterward):

$MegabyteString = ""
For $Y = 1 to 1024
    $MegabyteString &= "X"
Next

;Append megabytes to string until catastrophic failure
$sString = ""
For $n = 1 To 2048
    $sString &= $MegabyteString
    ConsoleWrite($n & " mb: $sString = " & StringLen($sString) & @LF)
Next

Hmm... Your $MegabyteString is only 1KB long, so that only counts up to 2MB, not 2GB. The output ends for me with:

2046 mb: $sString = 2095104
2047 mb: $sString = 2096128
2048 mb: $sString = 2097152
+>14:21:04 AutoIT3.exe ended.rc:0

Did it finish for you if you tried my version?

[/dev/null]

26: $sString = 67108864

27: $sString = 134217728

->13:35:55 AutoIT3.exe ended.rc:1

That failed at about 128MB.

same behavior here.... WinXP SP2, AU3 3.2.10 Edited January 22, 2008 by /dev/null

[weaponx]

Hmm... Your $MegabyteString is only 1KB long, so that only counts up to 2MB, not 2GB. The output ends for me with:

2046 mb: $sString = 2095104
  2047 mb: $sString = 2096128
  2048 mb: $sString = 2097152
  +>14:21:04 AutoIT3.exe ended.rc:0

Did it finish for you if you tried my version?

Sorry. My fault.

This version rendered my computer unusable at 183 mb

;Create a one megabyte string
$MegabyteString = ""
For $X = 1 to 1024
    ;Create a one kilobyte string
    For $Y = 1 to 1024
        $MegabyteString &= "X"
    Next
Next

;Append megabytes to string until catastrophic failure
$sString = ""
For $n = 1 To 256
    $sString &= $MegabyteString
    ConsoleWrite($n & " mb: $sString = " & StringLen($sString) & @LF)
Next

XP Pro Sp2, 1 GB Ram, P4 3.2ghz

Edited January 22, 2008 by weaponx

[PsaltyDS]

Sorry. My fault.

This version rendered my computer unusable at 183 mb

;Create a one megabyte string
$MegabyteString = ""
For $X = 1 to 1024
    ;Create a one kilobyte string
    For $Y = 1 to 1024
        $MegabyteString &= "X"
    Next
Next

;Append megabytes to string until catastrophic failure
$sString = ""
For $n = 1 To 256
    $sString &= $MegabyteString
    ConsoleWrite($n & " mb: $sString = " & StringLen($sString) & @LF)
Next

XP Pro Sp2, 1 GB Ram, P4 3.2ghz

None of my tests have caused problems with the OS, just an error pop up and AutoIt exits. I ran it again on a server with 4 2GHz Xenon CPUs and 5GB RAM on Win2K3 SP1 = 208MB in the string when it crashed. I wonder what the real limitation is?

[AdmiralAlkex]

This is strange, weaponx later version fails at 208 mb

Win XP SP2

Intel Core 2 Duo E6750

2 GB Ram

[weaponx]

Hangs @ 208 mb / 218,103,808 characters on my home machine too.

Intel Q6600 @ 3ghz, 2gb ram @ 835 mhz

XP Pro SP2 32-bit

[The Kandie Man]

Autoit did not hang for me. It went all the way up to 256MB and then exited correctly.

Windows XP Pro x64

2GB RAM

- The Kandie Man ;-)

[Valik]

Come on guys, think about this. Of course you can't hit 2GB. For one thing, the virtual address space of a process is only 2GB. The program itself and all the DLL's it's using are using some of that space. Then there's the reserved space for the stack. The system is only going to give you so much dynamic memory. Then, when you think about it, strings have to grow, so in reality you're only going to get roughly half that size. As the internal string buffer grows, there is a period of time where over double the memory is used - when the new buffer has been allocated and the old elements are copied to the new buffer. At a certain point, the size of the old buffer plus the size of the new buffer is going to exceed what you can have. This is going to be roughly half the maximum potential.

The fact is, the theoretical limit for a string is 2GB. At that point it becomes impossible to address the characters in the string. The real limit is at best less than half of that simply due to the way in which the string buffer grows. Then you have to figure in what your system can even handle, meaning, what the OS will give you. Even if you have 2GB of available resources, you aren't going to get it all, you aren't the only process on the system. The real limit is going to vary by hardware.

The only way you're ever going to hit the string limit in AutoIt is if you are running on a 64-bit OS Under the right circumstances, you should be able to load a string with 2GB of data. But in reality, and especially on 32-bit systems, you're not going to get anywhere near that. Nor do you want to. Anything that reads in 2GB of data at once is flawed, period.

[PsaltyDS]

The only way you're ever going to hit the string limit in AutoIt is if you are running on a 64-bit OS Under the right circumstances, you should be able to load a string with 2GB of data.

Thanks, Valik. Instructional, as always.

Just out of curiosity, since y'all have been knee deep in 64-bit land lately, can the 64-bit versions of XP/Vista/2003 with 64-bit AutoIt or outside of it (i.e. 64-bit C++ compiled) actually generate and work with a 2GB string (given only RAM/PageFile space)? Or would it still require exotic tweaking and trickery?

Edited January 23, 2008 by PsaltyDS

[Valik]

As far as I know it should work.

[PsaltyDS]

As far as I know it should work.

We just got a few 64-bit hardware PCs in, but for the moment all they want to know is that our standard 32-bit XP load will run on them (it did). Once we get some time away from that, I'll try to get a chance to play with 64-bit XP on one of them and see what happens.

(Why does that sound like "...play with one of them fancy new-fangled phones what ain't got no circular dial on it, just buttons!"?)

[The Kandie Man]

Come on guys, think about this. Of course you can't hit 2GB. For one thing, the virtual address space of a process is only 2GB. The program itself and all the DLL's it's using are using some of that space. Then there's the reserved space for the stack. The system is only going to give you so much dynamic memory. Then, when you think about it, strings have to grow, so in reality you're only going to get roughly half that size. As the internal string buffer grows, there is a period of time where over double the memory is used - when the new buffer has been allocated and the old elements are copied to the new buffer. At a certain point, the size of the old buffer plus the size of the new buffer is going to exceed what you can have. This is going to be roughly half the maximum potential.

The fact is, the theoretical limit for a string is 2GB. At that point it becomes impossible to address the characters in the string. The real limit is at best less than half of that simply due to the way in which the string buffer grows. Then you have to figure in what your system can even handle, meaning, what the OS will give you. Even if you have 2GB of available resources, you aren't going to get it all, you aren't the only process on the system. The real limit is going to vary by hardware.

The only way you're ever going to hit the string limit in AutoIt is if you are running on a 64-bit OS Under the right circumstances, you should be able to load a string with 2GB of data. But in reality, and especially on 32-bit systems, you're not going to get anywhere near that. Nor do you want to. Anything that reads in 2GB of data at once is flawed, period.

Yeah, that makes sense. I was watching the process with Process Explorer and watched as the Working Memory usage fluctuated up and down. I realized that this was because when AutoIt concatenates a string, it takes the original string and then allocates memory enough memory for the original string and the string to be concatenated on and then copies the string data byte for byte into the new string. This was horrificly slow at about 430MB because each megabyte increase meant that it would have to allocate memory for 430MB of string data and then copy it byte for byte into the new string buffer.

Also, since this is a unicode version of Autoit, I would also assume that the string array may use two bytes for every character. That means in reality, 120 characters may not be 120 bytes, but rather 240 bytes.

Overall I think i was able to get it up to 512MB before it crashed due to a unable to allocate memory error. I did the whole thing with 32bit AutoIt executables on a 64bit environment. At 480MB the AutoIt.exe process was using 963,023KB of Private Memory with a total working set of 964,724KB. Its peak working set however was 1,587,780KB.

As you can see, the peak working set is extremely high because of the double allocation of memory for the string concatenation.

If I am right, this makes perfect sense. The string's size is double because it is unicode and then the memory allocation is twice the string size. As a result, the program tried to allocate more physical memory than the system has and errors.

512MB * 2 = 1GB ;String size is twice its string length because it is in unicode and two bytes represent on character

(1GB * 2) + ((concatenation string length) * 2) > 2GB; Memory allocation is 2*2*StringLen($sString) + 2*StringLen($MegabyteString)

It is probably something like the above. Let me know if I am off.

- The Kandie Man ;-)

[Valik]

I forgot about the Unicode aspect. You're correct, the memory is going to be double the number of characters.

[Thatsgreat2345]

I'm just really curious when anyone is going to use that big of a string holy crap.

[Richard Robertson]

I'm just curious, why would you attempt to use a string buffer that large in the first place?