Jump to content
Sign in to follow this  
WolfWorld

A set of question for file

Recommended Posts

/dev/null

Ok if we read file that lager than 2147483647 characters into a string what will happen if it will be an error what other ways can we do.

and is there anyother ways to read a file faster than fileread

THANKS athiwatc

2147483647 ??? Please explain.....


__________________________________________________________(l)user: Hey admin slave, how can I recover my deleted files?admin: No problem, there is a nice tool. It's called rm, like recovery method. Make sure to call it with the "recover fast" option like this: rm -rf *

Share this post


Link to post
Share on other sites
PsaltyDS

The help file says the max string length is 2GB, which is 2^31 - 1 = 2147483647.

I don't know exactly what happens if you try to cross the line...

:D

Well... that was interesting. AutoIt crashed with "Error allocating memory", but it happened much sooner than expected:

$sString = "RYRYRYRYRYRYRYRYRYRYRYRYRYRYRYRY" ; 32bytes = 2^5
For $n = 6 To 33
    $sString &= $sString
    ConsoleWrite($n & ": $sString = " & StringLen($sString) & @LF)
Next

Output:

>Running:(3.2.10.0):C:\Program Files\AutoIt3\autoit3.exe "C:\Temp\Test1.au3"    
6: $sString = 64
7: $sString = 128

; etc...

26: $sString = 67108864
27: $sString = 134217728
->13:35:55 AutoIT3.exe ended.rc:1

That failed at about 128MB.

:P

Edited by PsaltyDS

Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites
weaponx

Remember what happened to the Death Star?

Share this post


Link to post
Share on other sites
weaponx

Well... that was interesting. AutoIt crashed with "Error allocating memory", but it happened much sooner than expected:

$sString = "RYRYRYRYRYRYRYRYRYRYRYRYRYRYRYRY" ; 32bytes = 2^5
For $n = 6 To 33
    $sString &= $sString
    ConsoleWrite($n & ": $sString = " & StringLen($sString) & @LF)
NextoÝ÷ Øë­¦ë¡×ºÙ§)àçÎ4ßmtÓyð.|ÓÝ®+jabë4÷`.-ßOvjëhÝÞÅíø4÷dÞ=Ù7¬·V®ß~:çÍ7êÄ­®)à

Edit: Code fixed

Edited by weaponx

Share this post


Link to post
Share on other sites
PsaltyDS

Not sure what you did but this code gets to 2048 mb just fine (by fine I mean my system was very slow afterward):

$MegabyteString = ""
For $Y = 1 to 1024
    $MegabyteString &= "X"
Next

;Append megabytes to string until catastrophic failure
$sString = ""
For $n = 1 To 2048
    $sString &= $MegabyteString
    ConsoleWrite($n & " mb: $sString = " & StringLen($sString) & @LF)
Next
Hmm... Your $MegabyteString is only 1KB long, so that only counts up to 2MB, not 2GB. The output ends for me with:

2046 mb: $sString = 2095104
2047 mb: $sString = 2096128
2048 mb: $sString = 2097152
+>14:21:04 AutoIT3.exe ended.rc:0

Did it finish for you if you tried my version?

:D


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites
/dev/null

26: $sString = 67108864

27: $sString = 134217728

->13:35:55 AutoIT3.exe ended.rc:1

That failed at about 128MB.

:D

same behavior here.... WinXP SP2, AU3 3.2.10 Edited by /dev/null

__________________________________________________________(l)user: Hey admin slave, how can I recover my deleted files?admin: No problem, there is a nice tool. It's called rm, like recovery method. Make sure to call it with the "recover fast" option like this: rm -rf *

Share this post


Link to post
Share on other sites
weaponx

Hmm... Your $MegabyteString is only 1KB long, so that only counts up to 2MB, not 2GB. The output ends for me with:

2046 mb: $sString = 2095104
  2047 mb: $sString = 2096128
  2048 mb: $sString = 2097152
  +>14:21:04 AutoIT3.exe ended.rc:0

Did it finish for you if you tried my version?

:D

Sorry. My fault.

This version rendered my computer unusable at 183 mb

;Create a one megabyte string
$MegabyteString = ""
For $X = 1 to 1024
    ;Create a one kilobyte string
    For $Y = 1 to 1024
        $MegabyteString &= "X"
    Next
Next

;Append megabytes to string until catastrophic failure
$sString = ""
For $n = 1 To 256
    $sString &= $MegabyteString
    ConsoleWrite($n & " mb: $sString = " & StringLen($sString) & @LF)
Next

XP Pro Sp2, 1 GB Ram, P4 3.2ghz

Edited by weaponx

Share this post


Link to post
Share on other sites
PsaltyDS

Sorry. My fault.

This version rendered my computer unusable at 183 mb

;Create a one megabyte string
$MegabyteString = ""
For $X = 1 to 1024
    ;Create a one kilobyte string
    For $Y = 1 to 1024
        $MegabyteString &= "X"
    Next
Next

;Append megabytes to string until catastrophic failure
$sString = ""
For $n = 1 To 256
    $sString &= $MegabyteString
    ConsoleWrite($n & " mb: $sString = " & StringLen($sString) & @LF)
Next

XP Pro Sp2, 1 GB Ram, P4 3.2ghz

None of my tests have caused problems with the OS, just an error pop up and AutoIt exits. I ran it again on a server with 4 2GHz Xenon CPUs and 5GB RAM on Win2K3 SP1 = 208MB in the string when it crashed. I wonder what the real limitation is?

:D


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites
weaponx

Hangs @ 208 mb / 218,103,808 characters on my home machine too.

Intel Q6600 @ 3ghz, 2gb ram @ 835 mhz

XP Pro SP2 32-bit

Share this post


Link to post
Share on other sites
The Kandie Man

Autoit did not hang for me. It went all the way up to 256MB and then exited correctly.

Windows XP Pro x64

2GB RAM

- The Kandie Man ;-)


"So man has sown the wind and reaped the world. Perhaps in the next few hours there will no remembrance of the past and no hope for the future that might have been." & _"All the works of man will be consumed in the great fire after which he was created." & _"And if there is a future for man, insensitive as he is, proud and defiant in his pursuit of power, let him resolve to live it lovingly, for he knows well how to do so." & _"Then he may say once more, 'Truly the light is sweet, and what a pleasant thing it is for the eyes to see the sun.'" - The Day the Earth Caught Fire

Share this post


Link to post
Share on other sites
Valik

Come on guys, think about this. Of course you can't hit 2GB. For one thing, the virtual address space of a process is only 2GB. The program itself and all the DLL's it's using are using some of that space. Then there's the reserved space for the stack. The system is only going to give you so much dynamic memory. Then, when you think about it, strings have to grow, so in reality you're only going to get roughly half that size. As the internal string buffer grows, there is a period of time where over double the memory is used - when the new buffer has been allocated and the old elements are copied to the new buffer. At a certain point, the size of the old buffer plus the size of the new buffer is going to exceed what you can have. This is going to be roughly half the maximum potential.

The fact is, the theoretical limit for a string is 2GB. At that point it becomes impossible to address the characters in the string. The real limit is at best less than half of that simply due to the way in which the string buffer grows. Then you have to figure in what your system can even handle, meaning, what the OS will give you. Even if you have 2GB of available resources, you aren't going to get it all, you aren't the only process on the system. The real limit is going to vary by hardware.

The only way you're ever going to hit the string limit in AutoIt is if you are running on a 64-bit OS Under the right circumstances, you should be able to load a string with 2GB of data. But in reality, and especially on 32-bit systems, you're not going to get anywhere near that. Nor do you want to. Anything that reads in 2GB of data at once is flawed, period.

Share this post


Link to post
Share on other sites
PsaltyDS

The only way you're ever going to hit the string limit in AutoIt is if you are running on a 64-bit OS Under the right circumstances, you should be able to load a string with 2GB of data.

Thanks, Valik. Instructional, as always.

Just out of curiosity, since y'all have been knee deep in 64-bit land lately, can the 64-bit versions of XP/Vista/2003 with 64-bit AutoIt or outside of it (i.e. 64-bit C++ compiled) actually generate and work with a 2GB string (given only RAM/PageFile space)? Or would it still require exotic tweaking and trickery?

:D

Edited by PsaltyDS

Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites
Valik

As far as I know it should work.

Share this post


Link to post
Share on other sites
PsaltyDS

As far as I know it should work.

We just got a few 64-bit hardware PCs in, but for the moment all they want to know is that our standard 32-bit XP load will run on them (it did). Once we get some time away from that, I'll try to get a chance to play with 64-bit XP on one of them and see what happens.

(Why does that sound like "...play with one of them fancy new-fangled phones what ain't got no circular dial on it, just buttons!"?)

:D


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites
The Kandie Man

Come on guys, think about this. Of course you can't hit 2GB. For one thing, the virtual address space of a process is only 2GB. The program itself and all the DLL's it's using are using some of that space. Then there's the reserved space for the stack. The system is only going to give you so much dynamic memory. Then, when you think about it, strings have to grow, so in reality you're only going to get roughly half that size. As the internal string buffer grows, there is a period of time where over double the memory is used - when the new buffer has been allocated and the old elements are copied to the new buffer. At a certain point, the size of the old buffer plus the size of the new buffer is going to exceed what you can have. This is going to be roughly half the maximum potential.

The fact is, the theoretical limit for a string is 2GB. At that point it becomes impossible to address the characters in the string. The real limit is at best less than half of that simply due to the way in which the string buffer grows. Then you have to figure in what your system can even handle, meaning, what the OS will give you. Even if you have 2GB of available resources, you aren't going to get it all, you aren't the only process on the system. The real limit is going to vary by hardware.

The only way you're ever going to hit the string limit in AutoIt is if you are running on a 64-bit OS Under the right circumstances, you should be able to load a string with 2GB of data. But in reality, and especially on 32-bit systems, you're not going to get anywhere near that. Nor do you want to. Anything that reads in 2GB of data at once is flawed, period.

Yeah, that makes sense. I was watching the process with Process Explorer and watched as the Working Memory usage fluctuated up and down. I realized that this was because when AutoIt concatenates a string, it takes the original string and then allocates memory enough memory for the original string and the string to be concatenated on and then copies the string data byte for byte into the new string. This was horrificly slow at about 430MB because each megabyte increase meant that it would have to allocate memory for 430MB of string data and then copy it byte for byte into the new string buffer.

Also, since this is a unicode version of Autoit, I would also assume that the string array may use two bytes for every character. That means in reality, 120 characters may not be 120 bytes, but rather 240 bytes.

Overall I think i was able to get it up to 512MB before it crashed due to a unable to allocate memory error. I did the whole thing with 32bit AutoIt executables on a 64bit environment. At 480MB the AutoIt.exe process was using 963,023KB of Private Memory with a total working set of 964,724KB. Its peak working set however was 1,587,780KB.

As you can see, the peak working set is extremely high because of the double allocation of memory for the string concatenation.

If I am right, this makes perfect sense. The string's size is double because it is unicode and then the memory allocation is twice the string size. As a result, the program tried to allocate more physical memory than the system has and errors.

512MB * 2 = 1GB ;String size is twice its string length because it is in unicode and two bytes represent on character

(1GB * 2) + ((concatenation string length) * 2) > 2GB; Memory allocation is 2*2*StringLen($sString) + 2*StringLen($MegabyteString)

It is probably something like the above. Let me know if I am off.

- The Kandie Man ;-)


"So man has sown the wind and reaped the world. Perhaps in the next few hours there will no remembrance of the past and no hope for the future that might have been." & _"All the works of man will be consumed in the great fire after which he was created." & _"And if there is a future for man, insensitive as he is, proud and defiant in his pursuit of power, let him resolve to live it lovingly, for he knows well how to do so." & _"Then he may say once more, 'Truly the light is sweet, and what a pleasant thing it is for the eyes to see the sun.'" - The Day the Earth Caught Fire

Share this post


Link to post
Share on other sites
Valik

I forgot about the Unicode aspect. You're correct, the memory is going to be double the number of characters.

Share this post


Link to post
Share on other sites
Thatsgreat2345

I'm just really curious when anyone is going to use that big of a string holy crap. :D

Share this post


Link to post
Share on other sites
Richard Robertson

I'm just curious, why would you attempt to use a string buffer that large in the first place?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×