Write binary data to a file?

[GaryC]

I want to store data in a file to allow the program to recover if it crashes. I want this to happen fast so that it doesn't interrupt the user. I have been able to write integers to a binary file one at a time but haven't found a way to write an array at one time-- actually I think I read that you can't. I looked at the binary type but can't see a way to concatenate data to it to hold more than one number. I tried a DllStruct but it wouldn't write. The only other way I can think of is to "manyually" convert the bytes to a string. Otherwise I'll have to write the stuff in ASCII numbers as in a print-out, which seems like a lot of wasted conversion just so that it can read it back in and convert it back to binary. My data is primarily a list of up to about 3000 2-byte integers indicating selected items.

Any ideas?

Thanks.

Gary

[Zedna]

Look at this my example

Instead of "byte" you can use "int","float",... structure ;-)

[GaryC]

Look at this my example

Instead of "byte" you can use "int","float",... structure ;-)

Thanks for your prompt response.

In some places the docs seem to indicate that you can assign a whole array at a time, in others they seem to say you can assign only a number or a string (or maybe a binary type). Can I assign a whole array at once, or do I have to set each element of the struct?

This program tries to assign an array to a struct array of ushorts and then read it back. The data returned by DllStructGetData is a number rather than an array, and the contents of the struct is all 0.

expandcollapse popup

#include <array.au3>

Const $gciCheckinsSize = 3000
Const $gcsLogFile = @ScriptDir & "\tststruct.log"
Global $gaCheckins[$gciCheckinsSize]

Global $gsLog = "" ; holds output messages.

Main()

Func Main()
    MakeData()
    ;_ArrayDisplay($gaCheckins, "test data")

    Local $tagBuf = "USHORT[" & UBound($gaCheckins) & "]"
    MsgOut(0, "Info", "Creating struct " & $tagBuf)
    Local $stBuf = DllStructCreate($tagBuf)
    If @error Then
        MsgOut(0, "Info", "After DllCreateStruct($tagBuf) @error was " & @error)
    EndIf
    DllStructSetData($stBuf, 1, $gaCheckins)
    If @error Then
        MsgOut(0, "Info", "After DllStructSetData($stBuf) @error was " & @error)
    EndIf
    Local $sMsg = "First elements of $stBuf: "
    For $i = 1 To 5
        $sMsg &= DllStructGetData($stBuf, 1, $i) & " "
        If @error Then
            $sMsg &= @CRLF & "After GetData for element " & $i & " @error = " & @error & @CRLF
        EndIf
    Next ; $i
    $sMsg &= @CRLF
    MsgOut(0, "Info", $sMsg)
    Local $gaCheckins2 = DllStructGetData($stBuf, 1)
    If @error Then
        MsgOut(0, "Info", "After DllStructGetData($stBuf) @error was " & @error)
    EndIf
    If Not IsArray($gaCheckins2) Then
        MsgOut(0, "Error", "DllStructGetData for $gaCheckins2 did not return array, type is " & VarGetType($gaCheckins2) & ", value = " & $gaCheckins2)
    EndIf
    ;_ArrayDisplay($gaCheckins2, "$gaCheckins2")
    CheckArrays($gaCheckins, $gaCheckins2)
    If @error Then
        MsgOut(0, "Error", "CheckArrays returned @error = " & @error)
    EndIf

    ; Write log.
    FileDelete($gcsLogFile)
    FileWrite($gcsLogFile, $gsLog)
    MsgBox(0, "Done", "Log written to " & $gcsLogFile)
EndFunc   ;==>Main

Func MakeData()
    Local $iIdx = 1
    For $i = 0 To UBound($gaCheckins) - 1
        $gaCheckins[$i] = ($iIdx + 1) * 256 + $iIdx
        If $iIdx >= 253 Then
            $iIdx = 1
        Else
            $iIdx += 2
        EndIf
    Next ; $i
EndFunc   ;==>MakeData

; Compares 2 arrays.  If they match returns 0, if different sizes sets @error=2, if contents differ sets @error=1 and @extended=first differing element.
; If either is not an array sets @error = 3.
Func CheckArrays(Const ByRef $a1, Const ByRef $a2)
    If Not IsArray($a1) Or Not IsArray($a2) Then Return SetError(3, 0, 0)
    If UBound($a1) <> UBound($a2) Then Return SetError(2, 0, 0)
    Local $fDiff = False
    For $i = 0 To UBound($a1) - 1
        If $a1[$i] <> $a2[$i] Then
            $fDiff = True
            ExitLoop
        EndIf
    Next ; $i
    If $fDiff Then
        Return SetError(1, $i)
    EndIf
    Return 0
EndFunc   ;==>CheckArrays

Func MsgOut($iFlag, $sTitle, $sMsg)
    $gsLog &= $sMsg & @CRLF
EndFunc   ;==>MsgOut

tststruct.log contents:

Creating struct USHORT[3000]

First elements of $stBuf: 0 0 0 0 0

DllStructGetData for $gaCheckins2 did not return array, type is Int32, value = 0

CheckArrays returned @error = 3

Thanks.

Gary

[kylomas]

Gary,

Try the following for writing bin values (I'm sure that they are string representations of bin values).

I'm sure that this can be made to run faster if it satisfies your requirements.

;
; The following write 300 8 byte bin values to a file
;

#include <array.au3>
#include <file.au3>

global $bNums[300]

for $i = 0 to 299
    $bNums[$i] = binary(random(1,10000,1))
Next

_arraydisplay($bnums)

local $bFl = fileopen("c:\bin_test",17)
if $bFl = -1 then msgbox(0,'','Unable to open file')

; each element followed by a crlf
;_FileWriteFromArray($bFl,$bNums)

; comma delimited string
FileWrite($bFl,_ArrayToString($bNums,','))


fileclose($bFl)

kylomas

Edited January 18, 2011 by kylomas

[kylomas]

Hello Structure Experts (especially you, Zedna),

I have been trying to get my arms around dll structures. I'm not a "C" coder, however, my language of choice is 386 assembler and in many ways these structures are like "control blocks".

I cannot figure out why Gary's code does not work. I've tried changing the structure element type to int64 thinking that ushort may not be big enough. I tries changing the way the array is generated by limiting array element values to random integers between 1 and 256. None of this worked.

I compared Zedna's example (in the thread) to what Gary is doing and the only difference I see is that Gary's array contains elements that are more than 1 byte long.

Lastly,

IS the winapi for file reading/writing being used because non string data is being written?

Thanks,

kylomas

P.S. Gary please excuse me if it seems like I am hijacking this thread but it seemed like the place to ask!

[Zedna]

Hello Structure Experts (especially you, Zedna),

:-)

IS the winapi for file reading/writing being used because non string data is being written?

My example was written especially as example for

_WinAPI_SetFilePointer()

because in that time Autoit's native file functions didn't support it.

Latest version can do SetFilePointer already.

But with this API and structures you can also easily read/write all data types directly to files.

[kylomas]

Structure Experts,

I tried to simplify the code as follows. I think I am missing the point entirely cause it looks like this should work but does'nt.

;
;

#include <array.au3>

local $struct,$a10[3000],$i,$s1_var

for $i = 1 to 3000 - 1
    $a10[$i] = hex(random(1,32767,1))
Next

;_arraydisplay($a10)

$struct = dllstructcreate("int64 s1_var[50000]")
if @error then msgbox(0,'','struct create error = ' & @error)

dllstructsetdata($struct,$s1_var,$a10)                          ; <----------    this returns 2, element out of range
if @error then msgbox(0,'','struct set data error = ' & @error) 

for $i = 1 to 100
    consolewrite(dllstructgetdata($struct,$s1_var,$i) & ' ')
next

consolewrite(@crlf)

Thanks,

kylomas

[Zedna]

You must do DllStructSetData for each element in FOR/NEXT loop.

[kylomas]

Yes,

I finally figured it out. This works. Gary this may or may not be applicable to your needs. Chalk it up to "too stuborn to let it go without understanding it".

;~ ;
;~ ;

#include <array.au3>

local $struct,$i

$struct = dllstructcreate("int[4000]")
if @error then msgbox(0,'','struct create error = ' & @error)

for $i = 1 to 1000
    dllstructsetdata($struct,1,binary(random(1,32767,1)),$i)
    if @error then msgbox(0,'','struct set data error = ' & @error)
Next

for $i = 1 to 1000
    consolewrite(binary(dllstructgetdata($struct,1,$i)) & ' ')
next

consolewrite(@crlf)

kylomas

[GaryC]

;
...
local $struct,$a10[3000],$i,$s1_var
...
$struct = dllstructcreate("int64 s1_var[50000]")
...
dllstructsetdata($struct,$s1_var,$a10)                          ; <----------    this returns 2, element out of range
...

It gets an error because $s1_var is not set. You would need either

$s1_var = "s1_var"

dllstructsetdata($struct,$s1_var,$a10)

or

dllstructsetdata($struct,"s1_var",$a10)

What I am trying to do is save some data to implement a recovery file. I want to save it as fast as possible so as to be as transparent as possible. If I need to recover I basically want to restore the array as it was before the crash, so I don't want to have to convert to binary or hex. I am using ushort because I know my numbers will fit in two bytes and don't want to waste disk space if I don't have to. That's why I am using the WinAPI functions. The documentation says if you set a struct element[...] to an array it takes as much of the array as there is room for, but it also talks about only being able to store strings or binary types, otherwise it has to be numbers. I guess what it means is that you can set any kind of array you want as long as it's a string or binary type. I have it working using the for loop to load the struct and I guess that's what I'll have to use. I was just hoping not to have to copy the data in a loop. Oh well, I guess most of the time there won't be much data.

About hijacking the thread: that was exactly what I wanted to know, so it's fine with me :-).

Thanks.

Gary

[kylomas]

Gary,

The one that I finally got working did NOT use the s1_var variable...my confusion is that the doc seems to indicate that you can load an array in dllstructsetdata but I could not make it work til I loaded it in a loop.

Anyway, good luck,

kylomas

[Sundance]

Hi folks,

i got one more question. I try to write a byte value in an array. How to get this going?

$structByteStructure = DllStructCreate("byte[1024]")

For $iCounter = 1 To 1024

$iResult = DllStructSetData($structByteStructure , 1, random(1,15,1), $iCounter)

ConsoleWrite(DllStructGetData($sStructure, 1,$iCounter) & @CRLF)

Next

for example won't work. I know that the random value is a int but how to create a byte value between 0 and F ?

Has someone a hint for me?

[trancexx]

What won't work?

[Sundance]

Hi trancexx,

the variable $iResult gets 0 and the dllstructure stays at 0 also.

I want to write 1024 randomly choosen bytes into the structure. Do you think i'am missing something?

Mmmhh. Can be. Sometimes i won't see the forest because of to many trees

PS: Now i'am trying it with an int array.

Edited January 29, 2011 by Sundance

[Jos]

Why are you using a different variable in the ConsoleWrite statement?

[Jos]

try this:

$structByteStructure = DllStructCreate("byte[1024]")

For $iCounter = 1 To 1024
    $iResult = DllStructSetData($structByteStructure, 1, Random(1, 15, 1), $iCounter)
Next
ConsoleWrite(DllStructGetData($structByteStructure, 1) & @CRLF)

[Sundance]

Ups. Sorry. It's a copy paste error.

I did a function with 'byref' the structure. In the function the structure is called $sStructure and in the main its called $structByteStructure . Sorry. But it won't run anyway. I'll think about it...

try this:

$structByteStructure = DllStructCreate("byte[1024]")

For $iCounter = 1 To 1024
    $iResult = DllStructSetData($structByteStructure, 1, Random(1, 15, 1), $iCounter)
Next
ConsoleWrite(DllStructGetData($structByteStructure, 1) & @CRLF)

[trancexx]

Maybe something is broken with my machine but that works just fine here.

[Jos]

Maybe something is broken with my machine but that works just fine here.

Your machine is as fine and mine when it comes to this script so guess the OP has an issue somewhere....

[trancexx]

Your machine is as fine and mine when it comes to this script so guess the OP has an issue somewhere....

As else maybe.

Btw, you never know, I've been watching some porn yesterday.