Jump to content
Sign in to follow this  
SmOke_N

Ini+Ex Functions - exceed 32kb limit

Recommended Posts

Could you please update the script to 3.3.10.2?

Now _IniReadSectionEx works only for the section in line 1.

When the section is in line 2 it doesn`t work.

Also it doesn`t work for section 2, 3, ...

I think the bug is at this lines

; data between sections or till end of file
    Local $s_datapatt = "(?is)(?:^|\v)(?!;|#)\h*\[\h*\Q"
    $s_datapatt &= $s_section
    $s_datapatt &= "\E\h*\]\h*\v+(.*?)(?:\z|\v\h*\[)"
    Local $a_data = StringRegExp($s_fread, $s_datapatt, 1)
    If @error Then Return SetError(-3, 0, 0)

Share this post


Link to post
Share on other sites

Tweaky,

And what exactly have you tried to do to get the script working with v3.3.10.2 in the past 11 days? :huh:

M23


Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Works just fine for me with no bugs with an INI file with multiple sections, and where the first section name isn't on line 1.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

Couldn't it be done using the Windows APIs? So many string based calculation for such a simple thing.

I wrote an Ini class using the PrivateProfile APIs, didn't tested that for the 32kb limitation. May I ask for an example for the situation which current AutoIt Ini functions would fail?

Share this post


Link to post
Share on other sites

I have tried to understand the RegExp Pattern. But it`s to difficutly for me :

Please test it with the file in the attachment.

As @error I get -3

#Region    ;************ Includes ************
#include <ini_ex.au3>
#include <Array.au3>
#EndRegion    ;************ Includes ************

$welche_ini = @ScriptDir & "\" & "ini_ex.ini"
$array_ini = _IniReadSectionex($welche_ini, "Section 1")
MsgBox(0,0,@error)
_ArrayDisplay($array_ini, 1)
$array_ini = _IniReadSectionex($welche_ini, "Section 2")
MsgBox(0,0,@error)
_ArrayDisplay($array_ini, 2)

ini_ex.rar

Edited by Tweaky

Share this post


Link to post
Share on other sites

AutoIt currently uses GetPrivateProfileString.


UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites

AutoIt is, but this UDF isn't.

The PrivateProfile APIs have no size limit by themselves as MSDN says, that's why I guessed maybe the limitation is coming from how AutoIt is implementing them.

Would be nice if anybody could show me some code where current AutoIt Ini functions will fail because of the size limit.

Edited by D4RKON3

Share this post


Link to post
Share on other sites

Just remember this script was written 7 years ago, things have changed since then. The 32K limitation of the function IniReadSection is listed in the help file, and does seem to be still valid.

Edited by BrewManNH

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

If there isn't any limitation, why using this kind of UDFs instead of the built-in functions?

If the limitation is still there, why even it's limited as the used APIs don't have any limitation?

And if it's because of how AutoIt is using the APIs, why not writing an UDF based on the PrivateProfile APIs instead of regexp?

That's the whole point of all of my posts in this thread.

Share this post


Link to post
Share on other sites

>INI File Processing Functions

 

example

#include <IniEx.au3>
#include <Array.au3>

Local $sData = @CRLF, $aData, $iData, $iTimerDiff, $hIniFile
FileDelete(@DesktopDir & "\Test.ini")
For $i = 1 To 10
    $sData &= "[SectionName_" & $i & "]" & @CRLF
    For $y = 1 to 100000
        $sData &= "KeyName" & $y & " = Value" & $y & @CRLF
    Next
Next
FileWrite(@DesktopDir & "\Test.ini", $sData)    ;25 MB - 1.000.000 Line, Key-Value
$sData = ""

$iTimerDiff = TimerInit()   ;;_IniOpenFileEx()
$hIniFile = _IniOpenFileEx(@DesktopDir & "\Test.ini")
If @Error Then
    ConsoleWrite("Error - " & @Error & @LF)
    Exit
EndIf
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("_IniOpenFileEx TimerDiff - " & $iTimerDiff & @LF)


$iTimerDiff = TimerInit()   ;;_IniGetSectionNumberEx()
$iData = _IniGetSectionNumberEx($hIniFile)
If @Error Then
    ConsoleWrite("Error - " & @Error & @LF)
    Exit
EndIf
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniGetSectionNumberEx TimerDiff - " & $iTimerDiff & @LF)
ConsoleWrite("SectionNumber - " & $iData & @LF & @LF)


$iTimerDiff = TimerInit()   ;;_IniRenameSectionEx()
_IniRenameSectionEx($hIniFile, "SectionName_10", "SectionName_111")
If @Error Then
    ConsoleWrite("Error - " & @Error & @LF)
    Exit
EndIf
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniRenameSectionEx TimerDiff - " & $iTimerDiff & @LF & @LF)


$iTimerDiff = TimerInit()   ;;_IniReadSectionNamesEx()
$aData = _IniReadSectionNamesEx($hIniFile)
If @Error Then
    ConsoleWrite("Error - " & @Error & @LF)
    Exit
EndIf
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniReadSectionNamesEx TimerDiff - " & $iTimerDiff & @LF & @LF)
_ArrayDisplay($aData)


$iTimerDiff = TimerInit()   ;;_IniReadSectionEx()
$sData = _IniReadSectionEx($hIniFile, "SectionName_8")
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniReadSectionEx TimerDiff - " & $iTimerDiff & @LF & @LF)
;;ConsoleWrite($sData & @LF & @LF)
;;MsgBox(0, "SectionName_8", $sData)

$iTimerDiff = TimerInit()   ;;_IniReadSectionEx()
$aData = _IniReadSectionEx($hIniFile, "SectionName_8", $INI_2DARRAYFIELD)   ;$INI_NOOCCURRENCE + $INI_MERGE + $INI_2DARRAYFIELD
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniReadSectionEx TimerDiff - " & $iTimerDiff & @LF & @LF)
_ArrayDisplay($aData, "IniReadSectionEx")


$iTimerDiff = TimerInit()   ;;_IniReadEx()
$sData = _IniReadEx($hIniFile, "SectionName_8", "KeyName99980")
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniReadEx TimerDiff - " & $iTimerDiff& @LF)
ConsoleWrite($sData & @LF & @LF)


$iTimerDiff = TimerInit()   ;;_IniReadEx()
$aData = _IniReadEx($hIniFile, "SectionName_8", "KeyName99988", "Default Return", $INI_ARRAYDATA)   ;$INI_NOOCCURRENCE + $INI_MERGE + $INI_ARRAYDATA
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniReadEx TimerDiff - " & $iTimerDiff & @LF)
ConsoleWrite($aData[1] & @LF & @LF)
_ArrayDisplay($aData, "IniReadEx")


$iTimerDiff = TimerInit()   ;;_IniDeleteEx()
_IniDeleteEx($hIniFile, "SectionName_8", "KeyName99988")    ;$INI_IGNOREDUPLICATE + $INI_NOOCCURRENCE + $INI_REMOVE
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniDeleteEx TimerDiff - " & $iTimerDiff & @LF & @LF)


$iTimerDiff = TimerInit()   ;;_IniWriteSectionEx()
_IniWriteSectionEx($hIniFile, "SectionName_8", "KeyNameX =  ValueX")    ; $INI_NOOCCURRENCE + $INI_IGNOREDUPLICATE + $INI_MERGE + $INI_REMOVE + $INI_OVERWRITEALL +  $INI_APPENDDATA
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniWriteSectionEx TimerDiff - " & $iTimerDiff & @LF & @LF)
ConsoleWrite(_IniReadEx($hIniFile, "SectionName_8", "KeyNameX") & @LF)


$iTimerDiff = TimerInit()
For $i = 1 To 5
    _IniWriteSectionEx($hIniFile, "SectionName_8", "KeyNameX" & $i & " =  ValueX" & $i)
Next
For $i = 6 To 10
    _IniWriteEx($hIniFile, "SectionName_8", "KeyNameX" & $i, "ValueX" & $i)
Next
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniWrite*Ex TimerDiff - " & $iTimerDiff & @LF)


$iTimerDiff = TimerInit()   ;;_IniReadEx()
For $i = 1 To 10
    ConsoleWrite(_IniReadEx($hIniFile, "SectionName_8", "KeyNameX" & $i) & @LF)
Next
$iTimerDiff = TimerDiff($iTimerDiff)
ConsoleWrite("IniReadEx TimerDiff - " & $iTimerDiff & @LF)

_IniCloseFileEx($hIniFile)

Exit


;~ >Running:(3.3.11.3):C:\Program Files (x86)\AutoIt3\Beta\autoit3.exe "C:\Users\DXRW4E\Desktop\New AutoIt v3 Script.au3"    
;~ --> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
;~ _IniOpenFileEx TimerDiff - 2656.48548057389
;~ IniGetSectionNumberEx TimerDiff - 0.0343594296492653
;~ SectionNumber - 10
;~  
;~ IniRenameSectionEx TimerDiff - 0.204971769976651
;~  
;~ IniReadSectionNamesEx TimerDiff - 0.0584505240010489
;~  
;~ IniReadSectionEx TimerDiff - 6.80316707055452
;~  
;~ IniReadSectionEx TimerDiff - 1831.04639075913
;~  
;~ IniReadEx TimerDiff - 24.1380917325683
;~ Value99980
;~  
;~ IniReadEx TimerDiff - 111.67841469541
;~ Value99988
;~  
;~ IniDeleteEx TimerDiff - 43.9488700091941
;~  
;~ IniWriteSectionEx TimerDiff - 28.2560891228314
;~  
;~ ValueX
;~ IniWrite*Ex TimerDiff - 261.353224415258
;~ ValueX1
;~ ValueX2
;~ ValueX3
;~ ValueX4
;~ ValueX5
;~ ValueX6
;~ ValueX7
;~ ValueX8
;~ ValueX9
;~ ValueX10
;~ IniReadEx TimerDiff - 261.622965684803
;~ +>00:07:21 AutoIt3.exe ended.rc:0
;~ >Exit code: 0    Time: 5.914
Ciao. Edited by DXRW4E

Share this post


Link to post
Share on other sites

If there isn't any limitation, why using this kind of UDFs instead of the built-in functions?

There is a limitation.

If the limitation is still there, why even it's limited as the used APIs don't have any limitation?

And if it's because of how AutoIt is using the APIs, why not writing an UDF based on the PrivateProfile APIs instead of regexp?

That's the whole point of all of my posts in this thread.

AutoIt uses a Windows provided API for reading/writing ini files. These functions were created back in the Windows 3.1 days and haven't been updated because they're only kept in Windows to provide backwards compatibility. Since Windows 95 Microsoft has recommended the registry and recently XML files instead of ini files.

If you need more than 64K in an ini file, or more than 32K per section, you should probably use something else, or as has been done here roll your own functions.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

Here is a quick test with IniRead. The test writes the section header and then appends keys and values with each loop using FileWrite (for faster write speed using a handle). Tested on Windows 7.

$count = 1048576
$content = '1234567890abcdef'
$start = 0

; write the section header
$hIniFile = FileOpen('test.ini', 2)
If $hIniFile = -1 Then Exit 1
FileWrite($hIniFile, '[section]' & @CRLF)
FileClose($hIniFile)

Do
    Sleep(100)
    ; write key value pairs
    $hIniFile = FileOpen('test.ini', 1)
    If $hIniFile = -1 Then ExitLoop
    For $1 = $start To $count
        FileWrite($hIniFile, $1 & '=' & $content & @CRLF)
    Next
    $last = $count + 1
    ; use a last key to read from
    FileWrite($hIniFile, $last & '=' & $content & '&' & $count & @CRLF)
    FileClose($hIniFile)
    ;
    ConsoleWrite($count & ' ' & Round(FileGetSize('test.ini')/1048576, 2) & @CRLF)
    $start = $last + 1
    $count = $start + 1048576
Until IniRead('test.ini', 'section', $last, 'failed') == 'failed'

MsgBox(0, '', $count)

Output

1048576 24.94
2097154 50.94
3145732 76.94
4194310 102.94
5242888 128.94
6291466 154.94
7340044 180.94
8388622 206.94
9437200 232.94
10485778 259.4
11534356 286.4
12582934 313.4
13631512 340.4
14680090 367.4
15728668 394.4
16777246 421.4
17825824 448.4
18874402 475.4
19922980 502.4
20971558 529.4
22020136 556.4
23068714 583.4
24117292 610.41
25165870 637.41
26214448 664.41
27263026 691.41
28311604 718.41
29360182 745.41
30408760 772.41
31457338 799.41
32505916 826.41
33554494 853.41
34603072 880.41
35651650 907.41
36700228 934.41
37748806 961.41
38797384 988.41
39845962 1015.41
40894540 1042.41
41943118 1069.41
42991696 1096.41
44040274 1123.41
45088852 1150.41
46137430 1177.41
47186008 1204.41
48234586 1231.41
49283164 1258.41
50331742 1285.41
51380320 1312.41
52428898 1339.41
53477476 1366.41
54526054 1393.41
55574632 1420.41
56623210 1447.41
57671788 1474.41
58720366 1501.41
59768944 1528.41
60817522 1555.41
61866100 1582.41
62914678 1609.41
63963256 1636.41
65011834 1663.41
66060412 1690.41
67108990 1717.41
68157568 1744.41
69206146 1771.41
70254724 1798.41
71303302 1825.41
72351880 1852.41
73400458 1879.41
74449036 1906.41
75497614 1933.41
76546192 1960.41
77594770 1987.41
78643348 2014.41
79691926 2041.41
80740504 2068.41

It stopped at 2 GB. The 32 KB limit was removed from AutoIt3 when 9x support was removed. I could presume similar limits with the other Ini functions though testing would confirm.

Share this post


Link to post
Share on other sites

IniRead isn't the problem, IniReadSection is. If you take your 2GB ini file and try to read a single section with more than 32767 characters in it, it will only read the first 32767 characters. You can make it read any size ini file you'd like one key/value at a time, reading in bulk is limited.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×
×
  • Create New...