Need Expert Advise about files and DB

[topten]

I have multi-processes and several scripts , where I need to open files for reading/writing encrypt info, save it in files, read it, update and when the amount of process becomes too large the files become irresponsive (I am talking about 50-70 processes in minute)

I need an expert advise, which way should I organize my data: Should I continue working with files or should I do something with data bases or structure the info in arrays? Which way is the best to accomplish that?

[jchd]

This is way too vague to guess anything.

[topten]

Sure, I should have given more info

The main problem I am facing

$FileOP4 = FileOpen ($pathfile & "idcounter.dat",0)
   $idcounter = FileRead ($FileOP4)
   FileClose ($FileOP4)

I am getting the value of $idcounter for while untill I have more scripts reading and writing into this file. And at some point I even can't read this file, getting zero value. Which is the best way to deal with this?

Edited July 3, 2015 by topten

[jchd]

Then your problem is contention created by multiple asynchronous concurrent read/write accesses. Then yes, a database is precisely suited to handle this smartly.

If all your processes run on a single machine, then SQLite can greatly help. If you're accessing data remotely over a LAN, then a client-server engine will be necessary (MS SQL, MySQL, Postgress, ...) but they need administration and maintainance, contrary to the simple SQLite engine.

[TheSaint]

Do you need everything to be read so quickly?

If not, then some sleeps would help.

If so, then using a database would be better.