DIRECT ACCESS FILE UDF (fixed-length records)

[joseLB]

This UDf ( _FileDirectAccess.au3 ) implements direct access file method. It allows to create files with fixed-legth records  with many fields, so you can write or read these records giving their record number.

version 2 in sept 10 2017. I dis some importante changes.

Records can have up to 180 fields. Each field can be:

• type 1= int32 (occupies 4 bytes in record)
• type 2= int64 (occupies 8 bytes in record)
• type 3= string (size defined by user)
• type 4= real - double (occupies 8 bytes)  (not working yet, but for now you can use string with m chars -> ex: m=10, 16 or 20), where the most sgnificant digits are preserved (left digits)

If there is any interest on them, I can put the formal UDF comments describing better the functions.

How they work:

open/create a DA file:   _fileDA_Open (ByRef $file, $filename, $mode, $flush=10 )

• Param1= variable previously created by user, ex: $file=0. It will be converted in an array, where all fle information is saved, including record fields.
•  Param2= open mode = read (1) , read+write file (2), read+write+create (3)    mode=1 => file must exists and can only be read,  mode=2 => file + path will be created if does not exists. Record pointer will be after last record (so a write will be at file´s end),  mode 3 => a previous file will be erased. File + path will be created.
• Param3= flush at each "n" writes. If 0, there are no flushes. default = 10 = a flush at each 10 writes

define record fields and size:   _FileDA_defineRecord(ByRef $file, $fields)

• Param1= array describing file, created by _fileDA_Open
• Param2= array [n][2]: n= # fields, [n][0]=field type, [n][1]= field lenght (used just if string, otherwise ignored)

writes a record:   _fileDA_Write (ByRef $file, ByRef $record)

• Param1= array describing file, created by _fileDA_Open
• Param2= array[n], n= number of record´s fields. [0]= field1, [1]= field2... each field must have same definition´s datatypes
• ** strings smaller than field´s size => will be filled with " ", bigger will be right trimmed (left n chars will remain)

reads a record:     _fileDA_Read (ByRef $file, ByRef $record)

• Param1= array describing file, created by _fileDA_Open
• Param2= array with  # columns >= # fields. Each array´s column receives a record field

Positions file pointer at next record to be read or write:   _fileDA_Pos (ByRef $file, $pos, $origin=0)

• Param1= array describing file, created by _fileDA_Open
• Param2= pointer position.
• Param3= type of positioning: 0= absolute form start (default), 1= relative to the current position

Closes file:    _fileDA_Close (ByRef $file, $free=1)

• Param1= array describing file, created by _fileDA_Open
• Param2= type of close:  1 (defaut), clears $file array, so another file can be created on this variable. 0= another open is necessary, but defineRecord is not (ex: read)

Besides attached UDF, there is na example that covers it´s usage.

Spoiler

expandcollapse popup

#include <_FileDirectAccess.au3>

Local $file1=0                  ;must be declared. Wil be modified to an array by fileopen holding file definitions
Local $file1_fields[4][2]=[ [3,10], [1,0], [2,0], [4,0] ]       ;field definitions=> field1= string 10 chr, field2= int32, field3= int64, field4= double real (not working yet)
Local $recwrite[4]= ["ABCDEFGHIJKLMN", 1023, 1024, 1024.5 ]     ;record to write: 4 fields respecting field definitions (string can be bigger or smaller). Must be declared
Local $recRead[4]                                               ;array where record will be read. Must be declared. 

    _ArrayDisplay($file1_fields, "fields definitions")          ;show array with field definitions
    _ArrayDisplay($recwrite, "record to write")                 ;show record to write

_fileDA_Open($file1 , "C:\GoogleDrive\bateria\VII\Central_PC\V6\Test1.txt" , 3) ;open a new file for read and write. Create file/path. If opened for read (1), file must exists
_FileDA_defineRecord( $file1, $file1_fields )                           ;defines each field type/size and record size = sum of fields sizes

_fileDA_Write( $file1, $recwrite)                                       ;writes 1st record
$recwrite[0]= "second record"
_fileDA_Write( $file1, $recwrite)                                       ;writes 2nd record
$recwrite[0]= "3th"
_fileDA_Write( $file1, $recwrite)                                       ;writes 3rd record (string smaller than 10 => will b filled with " "

_fileDA_Pos($file1, 1,0)                                                ;**** rewind pointer to 1st record to start read
_fileDA_Read( $file1, $recRead )                                        ;reads 1st record
    _ArrayDisplay( $recRead, "record 1 readed")
_fileDA_Read( $file1, $recRead )                                        ;reads 2nd record
    _ArrayDisplay( $recRead, "record 2 readed")
_fileDA_Read( $file1, $recRead )                                        ;reads 3rd record
    _ArrayDisplay( $recRead, "record 3 readed")

_fileDA_Pos($file1, 2 ,0)                                               ;**** pointer set to re-read 2nd record
_fileDA_Read( $file1, $recRead )                                        ;reads 2nd record
    _ArrayDisplay( $recRead, "rec. 2 re-readed")

_fileDA_Close( $file1)

;+++++  speed tests +++++++
_fileDA_Open($file1 , "C:\GoogleDrive\bateria\VII\Central_PC\V6\Test2.txt" , 3) ;open a new file for read and write
_FileDA_defineRecord( $file1, $file1_fields )                           ;defines each field type/size and record size = sum of fields sizes
$tempo= TimerInit()
$recwrite[0]= "10K"
For $K=1 to 10000                                   ;will record 10.000 records
    $recwrite[0]= $K/1.001                              ;***string[n] can be a floating point (this example 10 lef most significant digits are saved, as string has 10 chars)
    $recwrite[1]= $K
    $recwrite[2]= $K+1
    $recwrite[3]= $K/7
    _fileDA_Write( $file1, $recwrite)
Next
MsgBox(4096,"10K rec. writes", "time  to write (ms)= " & int(TimerDiff( $tempo))) ; time spent recording

$tempo= TimerInit()
_fileDA_Pos($file1, 1 ,0)                               ;rewind to record 1 to read 10K records and measure their time
For $K= 1 To 10000                                      ; time to read 10.000 records
    _fileDA_Read( $file1, $recRead )
Next
MsgBox(4096,"10K rec. reads", "time to read (ms)= " & int(TimerDiff( $tempo)))

For $K=1 To 7
    $R= Random(1,10000,1)
    _fileDA_Pos($file1, $R ,0)      ;randomic record
    _fileDA_Read( $file1, $recRead )
    _ArrayDisplay( $recRead, "rec." & $R & " OK?")

Next

MsgBox(4096, "skipped writes", "up to now 10K regs where writted sequentialy" &@cr& "now just reg 15K and 14K would be writted" &@cr& "and some reads will be done")
_fileDA_Pos($file1, 15000 ,0)                       ;write a new record "skipping" 5K records (rec. 15K)
 $recwrite[1]= 15000
_fileDA_Write( $file1, $recwrite)
_fileDA_Pos($file1, 14000 ,0)                       ;write a new record (rec. 14K)
 $recwrite[1]= 14000
_fileDA_Write( $file1, $recwrite)
_fileDA_Pos($file1, 15000 ,0)
_fileDA_Read( $file1, $recRead )                    ;reads rec. 15K to see if OK
_ArrayDisplay( $recRead, "rec 15K" )

_fileDA_Pos($file1, 13000 ,0)                       ;reads non-writed record 13K
_fileDA_Read( $file1, $recRead )
_ArrayDisplay( $recRead, "rec 13K not writed" )

_fileDA_Pos($file1, 14000 ,0)                       ;reads record 14K
_fileDA_Read( $file1, $recRead )
_ArrayDisplay( $recRead, "rec 14K" )

_fileDA_Close( $file1)

 

version 2 :

 

_FileDirectAccess.au3

Edited September 11, 2017 by joseLB

[joseLB]

About real/float/double numbers that are not working:

** meanwile you  can use a string field type with the desired size to preserve the precision you want: ex: 1024.123456 will need 11 chars

The real (float) seems to be writen  OK.  So, the problem is to read them.

In a record, where the real/double number writen was  1024.5   using     FileWrite( $file[1] ,  Number($record[$K], 3 )  )  ,    later  reading it with   $X= FileRead( $file[1], 8) , and observing $X in MsgBox(...) =  0x0000000000029040  that seems to be a real number, as:

• 1024.5 = 0x0...29040
• 1025.5 = 0x0...69040 
• 1026.5 = 0x0...A9040 

I suppose I tried all conversion routines avaiable... any clue? I googled for some float point converters, but could reach a resonable answer.

Thanks in advance.

Edited September 10, 2017 by joseLB