RTFC

HighMem - Maximise your RAM access on x64

10 posts in this topic

#1 ·  Posted (edited)

Please answer me these questions three, ere the other side you see:

  1. Are you running a 64-bit machine with a 64-bit Windows operating system?
  2. Can your AutoIt scripts cope with having directive #AutoIt3Wrapper_UseX64=Y, and thus @AutoItX64=True?
  3. Are you sick and tired of seeing this error message?

OutofMemory.PNG

If you (like me) answered "YES" to all three questions, then the _HighMem library may ease your pain (the name commemorates a useful utility from the days when CPUs were still steam-powered).B) Forget about pathetic boot switches /3GB and /userva; in a full-fledged 64-bit environment, _HighMem can pre-allocate all available physical/virtual RAM you've got (or any smaller size you need), and manage individual allocations therein with four simple functions:

_HighMem_StartUp( $nSize, $sUnit="GB" ) ; parse size of total region to pre-allocate, e.g. (10,"GB")
_HighMem_Allocate( $nSize, $sUnit="B" ) ; returns $pOffset (new allocation's base address)
_HighMem_Release( $pOffset ) ; existing allocations are identified by their offset (base address)
_HighMem_CleanUp() ; close handles, release all pre-allocated memory

Of course, existing AutoIt limitations remain in force (e.g., DllstructCreate() is still limited to 2 GB per call),:( but the maximum of 2-4 GB of virtual memory per Windows process can (under the right circumstances, in the proper environment) be circumvented.;) However, this is the first beta release, so glitches are likely, and performance may vary. In fact, it may not work at all for you (if you're running 32-bit, for example). And since this involves your own hardware, it's unlikely I would be able to reproduce your issues in my own work environment. Nevertheless, if you find obvious bugs or mistakes in the code, please do post. And if it works for you, that's also good to hear. My own motivation for developing it was to supercharge my matrix computing environment (Eigen4AutoIt), so it can handle matrices of any size that fit in machine RAM.

The attached zip contains the library itself (HighMem.au3) and two test examples. HighMem_Test1 performs a dry run stress test of the allocation management system; it does not actually do any memory I/O. By contrast, HighMem_Test2 pre-allocates a 6 GB space, stores 3 x 2GB structs there, performs some basic I/O, and releases the allocations one by one. Obviously, for this to work you'll need at least that much free RAM to begin with (check with Task Manager -> Performance -> Memory if you're unsure). My own test environment has 16 GB of physical RAM, and runs W10Pro/64.

EDIT: minor edits added to improve user experience (many more status messages if $_HighMem_Verbose=True)

 

HighMem.v0.85.7z

Edited by RTFC
2 people like this

Share this post


Link to post
Share on other sites



#2 ·  Posted

@RTFC.  Very interesting.  Is the problem that you describe in your intro inherent to 64-bit systems in general, or is this soley an issue when attempting to use the 64-bit version of AutoIt?  Same question would apply to your scripts use: can I use it if I am running a 64-bit system but the 32-bit version of Autoit, or would I need to have the 64-bit version of Autoit in order to get any use out of this script?

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

@MattHiggs: Hi, thanks. The problem I'm describing (running out of memory when trying to allocate multiple large chunks of data) occurs solely when running AutoIt scripts as 32-bit (even though I'm on x64 machine + x64 OS). But it would of course also occur on a 32-bit machine+OS. I haven't really tested this library in partial 32-bit environments, but as far as I can tell, you would need all three aspects (machine/OS/AutoIt) to be running in x64-mode for this to work. I presume that if you were to use the directive "#AutoIt3Wrapper_UseX64=Y" in a 32-bit environment (machine and/or OS), it would fail, and @AutoitX64 would still return False (0). If the OS is not x64, then how is AutoIt going to receive 64-bit memory pointers from it (which are needed to address the entire virtual memory space, in my case 128 TB (with 16 GB physical RAM))? Ditto for machine. Since my environment only required me to get the AutoIt part up to x64 standard (and my own Eigen4AutoIt.dll to be compiled for x64), that's what I did.

It's because I deal with large datasets in large computations all the time, and so far my E4A library couldn't handle these in full, so I either had to write complicated segmented computational codes:x or develop a computation in E4A on a small dummy dataset, iron out all the kinks, and then translate that into a separate (C++) Eigen code (which is quite painful to debug, hence using E4A first, which is much more user-friendly). Using HighMem, that's all history now. (Well, except for the initial debugging, that is.;))

Another advantage of using this memory mapping (that I hadn't mentioned yet) is that it can be shared between processes, so conceivably parallelised processing becomes possible (even networked, if #RequireAdmin can be added to the script). So for me it was the next step. Not sure it can be of any use to others (such as yourself), but it's a safe bet to assume that in x86-mode (i.e., 32-bit), there's no point in using it as your mappable address space would be restricted to 2-4 GB, which is the standard Windows-allotted virtual RAM per process anyway. Does that answer your question?:)

Edited by RTFC
clarification
1 person likes this

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

9 minutes ago, RTFC said:

in my case 128 TB

:blink:  Damn!  Dude.  Do you have the entire internet stored on this machine or what?  I cannot even comprehend having that amount of storage, much less a scenario where it would be necessary.  And I thought my 6 TB  file server was hot shit.  Now I feel emasculated..:>

As for the rest of your explanation, if I am understanding your explanation correctly, this udf essentially removes 32 bit operating system max memory allocation and allows operations which require a large amount of memory (more than what a 32-bit system is capable of supporting normally) to be performed on 32 bit systems.  That correct?

Edited by MattHiggs

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

This is virtual memory, not physical memory.:lol: And I don't use all of it, I just need more than 4 GB in individual computations. Used memory pages get swapped in and out of physical RAM as needed. Explanation here.

RE. your second question, not quite. This library is purely for x64 environments. It's a RAM allocation manager that pre-allocates a named space of desired size, in which you can then allocate smaller chunks that can be larger than 4GB each (identified by their base address) for your personal use. If you're mapping AutoIt structs to these (see second provided test script), these will still be restricted to 2GB (-1), but my E4A x64 dll is native-x64, so larger-sized matrices are also supported there. And when you subsequently release some of these chunks they can be re-assigned for new allocations, just like a file manager would for a harddrive. I cannot use AutoIt structs for matrices >2GB; that's why HighMem provides an alternative for larger chunks (but only in x64).

 

 

Edited by RTFC
typos
1 person likes this

Share this post


Link to post
Share on other sites

#6 ·  Posted

2 minutes ago, RTFC said:

This is virtual memory, not physical memory

As in the "pagefile.sys" for Windows operating systems?  As for the rest, I believe I am starting to understand.  It specifically for 64 bit systems, but, especially for memory intesive processes, there was not a way to limit the size of the chuncks of memory that a process would utilize while running (at least until this UDF), and would normally be determined by the operating system, which might not make the best decision as to the size of the these chunks which could lead to errors included in first post (Os might think 1GB of memory available is plenty of memory to allocate to a process requesting access to RAM, without realizing that the chunk of memory that the process will be using is 1 1/2 GB of memory).  So a more effective memory management tool.  That closer?

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Yap. Although it's not so much the OS that's the bottleneck for me as it is the maximum size of AutoIt structs. I used to stuff my matrices in structs, which is fine in 32-bit up to 2 GB. But now I need larger containers, so this library provides me with an easy way to manage these (on x64), without either creating or mapping (AutoIt) structs. Of course, you could also stick a large file in there (e.g. to compute a hash, as trancexx has shown here), or use it as a large graphics buffer for animations, for example.

Edited by RTFC
1 person likes this

Share this post


Link to post
Share on other sites

#8 ·  Posted

4 minutes ago, RTFC said:

Yap. Although it's not so much the OS that's the bottleneck for me as it is the maximum size of AutoIt structs. I used to stuff my matrices in structs, which is fine in 32-bit up to 2 GB. But now I need larger containers, so this library provides me with an easy way to manage these (on x64), without either creating or mapping (AutoIt) structs. Of course, you could also stick a large file in there (e.g. to compute a hash, as trancexxhas shownhere), or use it as a large graphics buffer for animations.

I see.  Ok.  That answers my questions.  Very cool udf.  Thank you sir

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

Since I did a less than stellar job of explaining yesterday:baby:, I'll try to correct myself here.:> Although AutoIt_x64 will not generate an @error when attempting to allocate a struct >2GB, only that maximum amount will be allocated, as far as I can tell (at least, I am unable to read/write data from/to addresses higher than that). Here's an example (for users on an x64 machine + x64 OS):

;#AutoIt3Wrapper_UseX64=N   ; always FAILS, even on x64 machine + x64 OS
#AutoIt3Wrapper_UseX64=Y    ; works on x64 machine + x64 OS

Global $GB=1024^3
Global $GBtoAllocate=6
Global $structs[$GBtoAllocate+1]
Global $lastbyte=$GB * 1
Global $testvalue=123       ; use 0-255 only (byte)

ConsoleWrite("Attempting to allocate " & $GBtoAllocate & " 1 GB chunks of memory..." & @CRLF)
For $cc=1 To $GBtoAllocate
    $structs[$cc]=DllStructCreate("byte[" & 1 * $GB & "]")  ; should work on x64
    if @error Then
        ConsoleWrite("Creating struct " & $cc & " has failed; aborting." & @CRLF)
        Exit(-1)
    EndIf
    ConsoleWrite("Setting last value: " & ((DllStructSetData($structs[$cc],1,$testvalue,$lastbyte)=$testvalue)?("works"):("FAILS!")) & @CRLF)
Next

ConsoleWrite(@CRLF & "Attempting to allocate a SINGLE chunk of " & $GBtoAllocate & " GB now..." & @CRLF)
$cc=0
Global $lastbyte=$GB*$GBtoAllocate
$structs[$cc]=DllStructCreate("byte[" & $lastbyte & "]")    ; partial failure (silent)
if @error Then
    ConsoleWrite("Creating struct " & $cc & " has failed; aborting." & @CRLF)
    Exit(-1)
EndIf
ConsoleWrite("Setting last value: " & ((DllStructSetData($structs[$cc],1,$testvalue,$lastbyte)=$testvalue)?("works"):("FAILS!")) & @CRLF)

So if you just need to allocate lots of small structs that together exceed 4 GB, you don't need HighMem (on x64 machine + OS that is, as long as you set the UseX64 directive; on x86 machine+OS you're up the proverbial creek). But if any single memory region you need exceeds 2 GB, HighMem may be quite useful on x64. Another advantage is control; you decide in advance how much total space you need, and within that space, HighMem will spatially optimise allocations of various sizes automatically, and keep track of additional free space when memory regions are released again. The drawback is that you'll still have to figure out yourself how to access that space. One way would be to map it with consecutive 1 GB structs. The WInAPI provides other ways. My E4A wrapper library now uses DllCalls for all matrix memory I/O.

And regarding that TB-sized virtual memory, of course I don't have a pagefile that size. Virtual memory is a joint address space shared by, and accessible to all processes on your machine. Only a tiny fraction thereof will be in use, and only a proportion of that will get swapped to disk if/when physical RAM is unable to cope. It's like the Dewey libary book topic classification; just because it spans thousands of sub-classes does not mean that every library using the system will have all existing books on each possible subtopic on its shelves.:D

I hope the above goes some way towards clearing the fog left by my earlier ramblings.:think:

Edited by RTFC

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • iXX
      By iXX
      Hi!
      Looking for working code to  get full path of process  - both 32 & 64 bit.
      I tryed this bellow, but it works only for 32-bit processes, even if compiled for x64...
      Thanx for suggestions!
       
      Func _ProcessGetPath($vProcess) ;get the program path done by MrCreatoR Local $iPID = ProcessExists($vProcess) If NOT $iPID Then Return SetError(1, 0, -1) Local $aProc = DllCall('kernel32.dll', 'hwnd', 'OpenProcess', 'int', BitOR(0x0400, 0x0010), 'int', 0, 'int', $iPID) If NOT IsArray($aProc) OR NOT $aProc[0] Then Return SetError(2, 0, -1) Local $vStruct = DllStructCreate('int[1024]') Local $hPsapi_Dll = DllOpen('Psapi.dll') If $hPsapi_Dll = -1 Then $hPsapi_Dll = DllOpen(@SystemDir & '\Psapi.dll') If $hPsapi_Dll = -1 Then $hPsapi_Dll = DllOpen(@WindowsDir & '\Psapi.dll') If $hPsapi_Dll = -1 Then Return SetError(3, 0, '') DllCall($hPsapi_Dll, 'int', 'EnumProcessModules', _ 'hwnd', $aProc[0], _ 'ptr', DllStructGetPtr($vStruct), _ 'int', DllStructGetSize($vStruct), _ 'int_ptr', 0) Local $aRet = DllCall($hPsapi_Dll, 'int', 'GetModuleFileNameEx', _ 'hwnd', $aProc[0], _ 'int', DllStructGetData($vStruct, 1), _ 'str', '', _ 'int', 2048) DllClose($hPsapi_Dll) If NOT IsArray($aRet) OR StringLen($aRet[3]) = 0 Then Return SetError(4, 0, '') Return $aRet[3] EndFunc  
    • Meobdg
      By Meobdg
      Hi, I have a problem, my functions do not work with each other. Separately, each works very well. Features that do not work together it's loot() and  pos1(). It looks that after spreadsheet function loot() script stops working even though everything is in the While 1-Wend loop.  Can someone help me please?
       
         
    • Dequality
      By Dequality
      Since my last topic were closed because bot scripting aren't allowed to be discussed anymore on here.
      Could anyone possibly give me a good example to learn memory read/write? , i can't figure out anything else which would be a good level of difficulty to practice than "tetris bot" but since it aint legal, i wont be asking for that let me know ur ideas and i would highly appreciate if examples could be posted (My last project was a imgsrch/pxlsrch) so thought i woud move on to memory read/write, if this somehow came out wrong lmk.
       
      AND NO I'M NOT ASKING FOR A FULL CODE I WANNA CODE/SCRIPT IT MY SELF, Just show me some simple examples of Memory read/write if u can/will TYVM.
       
      Dequality.
    • Dequality
      By Dequality
      I honestly didn't try anything yet.
      But i was thinking about my next step would be to try creating a tetris bot for educational purpose only. Last time i created a automation script / bot for ClubPenguin, which was simply made with PixelSearch If NOT(@error) Then , bla bla u know the drill aight, so i thought i would give my self a little harder challenge instead of keep making Pixelbots, i would love to try making a Memory bot or w/e u call , i read something about memory read/record was used for Tetris bots, since i haven't used Memory, i would love to know if anyone could give me a good guide or smth to follow :-) Either a complete tetris bot guide or a guide that explains memory totally in depth ish. :-)
       
       
      Dequality. #ANY-HELP-APPRECIATET
    • giangnguyen
      By giangnguyen
      I am looking for a way to set up either VIRTUAL_PROTECT or PAGE_GUARD for memory protection. I currently don't know how to do this, I have made the encryption for my EXE Protector, the RunPE module, and basically everything that I need. I also have made an advanced obfuscation tool, which I might release here on the forums in the future, to make sure the code is impossible to be understood. However, people can dump the original EXE from memory when I am injecting it. So how would I implement VIRTUAL_PROTECT, PAGE_GUARD or other methods of protecting memory?