Jump to content
Sign in to follow this  
AutID

BinaryToString() Error

Recommended Posts

AutID

Hello,

I have a database of 36mb in a .txt file and it is binary. I am trying to read the file and then write it in another file returned in ANSI.

However I am having an error saying this.

It's an AutoIt msgbox saying "Error allocating memory".

the code is this:

FileWrite($file, BinaryToString($read))
Any ideas?

Cheers

EDIT:

Hmmmm, it could be the File functions giving this error. ConsoleWrite gives a good return...sometimes...

Filewrite gives a true return but it doesn't write the read from the previous file... weird, it doesn't work everytime though...

Edited by AutID

Share this post


Link to post
Share on other sites
czardas

There's some information >here. There are probably a few questions like this on the forum, so do a search.

Edited by czardas

Share this post


Link to post
Share on other sites
AutID

I already looked at these posts. And other similar. I have a simple consowrite without the binary return on scite running and it is still running 25 minutes now...

Share this post


Link to post
Share on other sites
czardas

Well perhaps writing so much data to Scite console is not a good idea. I just did a quick test on multiple writes to a binary file and it worked fine. First try to achieve what you want with a small file and if it works then step it up to the big test. I split the binary using the following function after removing "0x" prefix. Then I wrote 512 bytes at a time to a file using a For loop.

;

Func _StringEqualSplit($sString, $iNumChars) ; Returns a 0 based array
    If Not IsString($sString) Or $sString = "" Then Return SetError(1, 0, 0)
    If Not IsInt($iNumChars) Or $iNumChars < 1 Then Return SetError(2, 0, 0)
    Return StringRegExp($sString, "(?s).{1," & $iNumChars & "}", 3)
EndFunc ; _StringEqualSplit

;

I haven't tried the method on a large file. I hope this information helps.

Edited by czardas

Share this post


Link to post
Share on other sites
jchd

AutId,

Unless you machine is really very short in memory I don't believe it could be that instruction causing the error.

Post your code, the error is most probably somewhere else.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
trancexx

That would be weird but sometimes it's possible. Like jchd says the error could be caused by something else. For example this simple code can cause issues depending how much RAM your system has:

DllStructCreate("byte[1234567865465]")

Try lowering the number to get rough idea of how much is max for your script.


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites
czardas

I was doubting myself about this after I ran further tests this morning. Apologies for misleading anyone.

Share this post


Link to post
Share on other sites
AutID

8GB RAM

#include <File.au3>

$file = _FileCreate(@ScriptDir & "\database.txt")
$read = FileRead(@ScriptDir & "\database 2012.txt")
$write = FileWrite($file, BinaryToString($read))
code is very simple. I changed the .sql file of the database to .txt but problem is still the same.

As I looked around people having similar problems, File functions are propably causing this error. If instead of using FileWrite I do a consolewrite then it works fine but it take about 2 hours to consolewrite the entire file. When I change the .sql to .txt the txt file became 261 MB...

Share this post


Link to post
Share on other sites
FireFox

You open the file in text read mode, not in binary mode.


 

OS : Win XP SP2 (32 bits) / Win 7 SP1 (64 bits) / Win 8 (64 bits) | Autoit version: latest stable / beta.
Hardware : Intel(R) Core(TM) i5-2400 CPU @ 3.10Ghz / 8 GiB RAM DDR3.

My UDFs : Skype UDF | TrayIconEx UDF | GUI Panel UDF | Excel XML UDF | Is_Pressed_UDF

My Projects : YouTube Multi-downloader | FTP Easy-UP | Lock'n | WinKill | AVICapture | Skype TM | Tap Maker | ShellNew | Scriptner | Const Replacer | FT_Pocket | Chrome theme maker

My Examples : Capture toolIP Camera | Crosshair | Draw Captured Region | Picture Screensaver | Jscreenfix | Drivetemp | Picture viewer

My Snippets : Basic TCP | Systray_GetIconIndex | Intercept End task | Winpcap various | Advanced HotKeySet | Transparent Edit control

 

Share this post


Link to post
Share on other sites
FireFox

$read = FileRead(@ScriptDir & "\database 2012.txt")
$write = FileWrite($file, $read)

;OR
$hFile = FileOpen(@ScriptDir & "\database 2012.txt", $FO_BINARY)
$read = FileRead($hFile)
FileClose($hFile)
$write = FileWrite($file, BinaryToString($read))


 

OS : Win XP SP2 (32 bits) / Win 7 SP1 (64 bits) / Win 8 (64 bits) | Autoit version: latest stable / beta.
Hardware : Intel(R) Core(TM) i5-2400 CPU @ 3.10Ghz / 8 GiB RAM DDR3.

My UDFs : Skype UDF | TrayIconEx UDF | GUI Panel UDF | Excel XML UDF | Is_Pressed_UDF

My Projects : YouTube Multi-downloader | FTP Easy-UP | Lock'n | WinKill | AVICapture | Skype TM | Tap Maker | ShellNew | Scriptner | Const Replacer | FT_Pocket | Chrome theme maker

My Examples : Capture toolIP Camera | Crosshair | Draw Captured Region | Picture Screensaver | Jscreenfix | Drivetemp | Picture viewer

My Snippets : Basic TCP | Systray_GetIconIndex | Intercept End task | Winpcap various | Advanced HotKeySet | Transparent Edit control

 

Share this post


Link to post
Share on other sites
AutID

You open the file in text read mode, not in binary mode.

I also tried a fileopen to force in a binary mode* but the same error message comes directly...

Edited by AutID

Share this post


Link to post
Share on other sites
FireFox

You will have to process the file by chunks.


 

OS : Win XP SP2 (32 bits) / Win 7 SP1 (64 bits) / Win 8 (64 bits) | Autoit version: latest stable / beta.
Hardware : Intel(R) Core(TM) i5-2400 CPU @ 3.10Ghz / 8 GiB RAM DDR3.

My UDFs : Skype UDF | TrayIconEx UDF | GUI Panel UDF | Excel XML UDF | Is_Pressed_UDF

My Projects : YouTube Multi-downloader | FTP Easy-UP | Lock'n | WinKill | AVICapture | Skype TM | Tap Maker | ShellNew | Scriptner | Const Replacer | FT_Pocket | Chrome theme maker

My Examples : Capture toolIP Camera | Crosshair | Draw Captured Region | Picture Screensaver | Jscreenfix | Drivetemp | Picture viewer

My Snippets : Basic TCP | Systray_GetIconIndex | Intercept End task | Winpcap various | Advanced HotKeySet | Transparent Edit control

 

Share this post


Link to post
Share on other sites
jchd

I don't get it. Either your file is a genuine .sql file, that is a text file in which case you don't have to process it as binary, or it is a true database and I wonder what processing you intend to apply to it.

Nevertheless, FileRead-ing a 36Mb binary shouldn't burst in you face.  Which AutoIt version are you using and do you run for x86 or x64?

BTW, this seems to work flawlessly over a 36Mb binary file (an old SQLite database, actually) and produces a ... 36 Mb file (unsurprisingly):

$hread = FileOpen(@ScriptDir & "\test.db3", 16)
$read = FileRead($hread)
FileClose($hread)
$hfile = FileOpen(@ScriptDir & "\binarbug.bin", 2)
$write = FileWrite($hfile, BinaryToString($read))
$read = 0
FileClose($hfile)

The whole process is nothing more than a sophisticated NOP operation, or rather an overly complexified file copy, BTW.

May I ask what the actual need is?

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
AutID

I don't get it. Either your file is a genuine .sql file, that is a text file in which case you don't have to process it as binary, or it is a true database and I wonder what processing you intend to apply to it.

 

Nevertheless, FileRead-ing a 36Mb binary shouldn't burst in you face.  Which AutoIt version are you using and do you run for x86 or x64?

 

BTW, this seems to work flawlessly over a 36Mb binary file (an old SQLite database, actually) and produces a ... 36 Mb file (unsurprisingly):

$hread = FileOpen(@ScriptDir & "\test.db3", 16)
$read = FileRead($hread)
FileClose($hread)
$hfile = FileOpen(@ScriptDir & "\binarbug.bin", 2)
$write = FileWrite($hfile, BinaryToString($read))
$read = 0
FileClose($hfile)
The whole process is nothing more than a sophisticated NOP operation, or rather an overly complexified file copy, BTW.

 

May I ask what the actual need is?

the .sql file was 36MB. As soon as I opened and saved it in Notepad it turned to 261MB. As I am understanding the filewrite times out or the data written in the FileWrite function is greater than it is supposed to be.

It is a true database.

ConsoleWrite managed to turn it to string (BinaryToString) and write it after some hours. I thought I could then copy it from the consolewrite and paste it in a file manually but copy doesn't seem to work on the consolewrite because the amount of the lines... 800k+ lines of consolewrite...

I will try split the database in parts and try again.

Share this post


Link to post
Share on other sites
BrewManNH

Why are you trying to read a DB file using FileRead, you're not going to get much usable information out of doing that.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites
jchd

AutId,

Instead of talking about what you think it does wrong, did you try my code with your own 36 Mb file (fairly similar to the one I used)?

Did you think twice about what it actually does (hint: nothing but copy the file verbatim)?

Did you bother to even explain in plain english what you really want to achieve?

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×