Jump to content
Jewtus

Storing files in a better way

Recommended Posts

Jewtus

I am dealing with an issue that I'm trying to think of a clever workaround for. The network drives that I have for deployment are painfully slow and when I put EXE files on them, it takes forever to load. That being said, I use the same network directory for handling "attachments" in various parts of my tool. I crypt the files and store the files extension as well as the crypted password in a database and put the crypted files into a subdirectory.

I'm not going to include my code here because really what I'm trying to figure out is if there is a way to deconstruct any file (mainly pdfs, msg files, and word docs) and store it as a string in my database or something of that nature. Sometimes the files are hundreds of MB and I don't know if I can even store something that big in a DB (I know you can with BULK statements, but that needs a path relative to the server which I cannot do). I want to get rid of the network directory dependency to improve performance.

Share this post


Link to post
Share on other sites
jchd

All SQL engines I know off support BLOBS (binary data) columns. If your DB is local you could just encrypt the whole DB and store everthing you want in BLOBs. For instance you can use SQLite for that. The system.data.sqlite.dll is 100% compatible with the legacy dll but adds crypt support so you don't even have to launch a third-party program to decrypt your precious data. This works at DB page level, so that the whole DB file is equivalent to random binary to anyone who doesn't have the passphrase. Works like a charm but don't forget the key!


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
Jewtus

Ya, but to do blob storage, don't you need to use BULK commands with a filepath that is relative for the server?

 

Because the people who would be attaching files are likely doing it from their local machines, the server wouldn't have permissions to their desktop for example (or the documents dir). I wanted to use BULK, but unfortunately, I cannot push files to the server to do a BULK statement and the server cannot path to end user machines.

 

If I were to use SQLite for blob storage, I would still need a place to store the SQLite DB that all end users would be able to access (thats why I'm looking to put it directly on the SQL Server)

 

I was even looking into fileread to do an insert statement with the read string, but after 30 minutes of reading a file (only about 80 MB) I gave up on that method. The network drives are slow but not THAT slow.

Share this post


Link to post
Share on other sites
jchd

If the DB won't be local to user, then forget abour bare SQLite. Yet other engines can be used like PostGres, MySQL, ...

  • Like 1

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
Jewtus

MySQL did it! Thanks for the suggestion!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • dangr82
      By dangr82
       
      I have created this function for a database, but I can not make it work. I always have two error messages: "not an error" ... and the file created, in the script directory, does not contain anything.
      Global $sDBName = "Hen.db" Func DatabaseTable() Local $sConnDB _SQLite_Startup() If @error Then MsgBox($MB_SYSTEMMODAL, "SQLite Error", "SQLite.dll Can't be Loaded!") Exit -1 EndIf $sConnDB = _SQLite_Open($sDBName) If @error Then MsgBox($MB_SYSTEMMODAL, "SQLite Error", "Can't Load Database!") Exit -1 EndIf If Not _SQLite_Exec($sDBName, 'CREATE TABLE Animal ("Name", "Age");') = $SQLITE_OK Then MsgBox($MB_SYSTEMMODAL, "SQLite Error", _SQLite_ErrMsg()) If Not _SQLite_Exec($sDBName, 'INSERT INTO Animale VALUES ("Charlie","5");') = $SQLITE_OK Then MsgBox($MB_SYSTEMMODAL, "SQLite Error", _SQLite_ErrMsg()) _SQLite_Close($sConnDB) _SQLite_Shutdown() EndFunc  
    • argumentum
      By argumentum
      ..let's say I store in a mysql column "[1,2,11,12,123]".
      How can I make s query to get all those rows that is 1, or is 12, etc ?
      I'm using  mysqlnd 5.0.11-dev - 20120503 
      Any advise on what technique I should use is welcomed
      Thanks
      PS: ..I'm adding a comma to each end, as in  [,1,2,11,12,123,] and will search with like %,12,% .
      Not pretty but I'm in a hurry and can't come up with a better solution =/
    • sc4ry
      By sc4ry
      Dear all,
      I already have a formula which will select doublicates, but sometimes there might be entries with additional information starting after "-", which I now want to trim away.
      Current formula is:
      Select DISTINCT a.title, a.oid, a.size, a.disk from DB a, DB b where (a.title like b.title and a.oid <> b.oid)" & $ActiveDisks_Filter & " AND (a.comment <> 'Delete' AND b.comment <> 'Delete') Order by a.title ASC, a.size DESC; testwise I want to add a simple solution like left, but afterwards I get an error but I do not know why.
      Select DISTINCT a.title, a.oid, a.size, a.disk from DB a, DB b where (left(a.title,15) like b.title and a.oid <> b.oid) Order by a.title ASC, a.size DESC; Error: near "(": syntax error: Finally I would like to insert intstr to search for "-" but currently I am not able to extend my current code =/
      Thanks a lot for your help.
      EDIT: hey, I managed it, left was not working also it was somehow shown that the tool knowed the comment. I did it now with substr+instr
      (substr(a.title, 1, instr(a.title, ' - '))  
    • AndyS19
      By AndyS19
      I have code that does a WMI SQL query to find all defined printers, and I want to parse the returned object in several places.  However, after parsing it the first time, all other times fail to find any printer objects.
      Here is my test code:
      test() Func test() Local $oPrinters, $oPrinter, $err, $cnt, $oP, $query $query = "SELECT * FROM Win32_Printer" $oPrinters = doQuery($query) $err = @error LogMsg("+++: $err = " & $err & ", isObj($oPrinters) = " & IsObj($oPrinters)) If ($err == 0) Then LogMsg("FIRST LOOP") ; <=== FIRST LOOP $cnt = 0 $oP = $oPrinters LogMsg("+++: isObj($oP) = " & IsObj($oP)) For $oPrinter In $oP $cnt += 1 LogMsg("+++: isObj($oPrinter): " & IsObj($oPrinter) & ", $oPrinter.Name ==>" & $oPrinter.Name & "<==") Next LogMsg("+++: Found " & $cnt & " printers") LogMsg("SECOND LOOP") ; <== SECOND LOOP $cnt = 0 $oP = $oPrinters LogMsg("+++: isObj($oP) = " & IsObj($oP)) For $oPrinter In $oP $cnt += 1 LogMsg("+++: isObj($oPrinter): " & IsObj($oPrinter) & ", $oPrinter.Name ==>" & $oPrinter.Name & "<==") Next LogMsg("+++: Found " & $cnt & " printers") EndIf EndFunc ;==>test Func doQuery($sQuery, $lnum = @ScriptLineNumber) #forceref $lnum LogMsg("+++:" & $lnum & ": doQuery(" & '"' & $sQuery & '"' & ") entered") Local $oWMIService, $oResults, $errstr Local $wbemFlags = BitOR(0x20, 0x10) ; $wbemFlagReturnImmediately and wbemFlagForwardOnly $oWMIService = ObjGet("winmgmts:\\" & "localhost" & "\root\CIMV2") If (IsObj($oWMIService)) Then $oResults = $oWMIService.ExecQuery($sQuery, "WQL", $wbemFlags) If (IsObj($oResults)) Then LogMsg("+++: doQuery() returns @error = 0, Good: returning the object") Return (SetError(0, 0, $oResults)) ;;; Good: return the object Else $errstr = "" _ & "WMI Query failed." & @CRLF _ & "This is the query:" & @CRLF _ & " " & $sQuery LogMsg("+++: ====>" & $errstr & "<===") LogMsg("+++: doQuery() returns @error = 1") Return (SetError(1, 0, $errstr)) ; Error: Query faled EndIf Else $errstr = "" _ & "WMI Output" & @CRLF _ & "No WMI Objects Found for class: " & @CRLF _ & "Win32_PrinterDriver" & @CRLF _ & "using this query:" & @CRLF _ & " " & $sQuery LogMsg("+++: ====>" & $errstr & "<===") MsgBox(0, "ERROR", $errstr) ; Error: Cannot get $oWMIService object Exit (1) EndIf EndFunc ;==>doQuery Func LogMsg($msg, $lnum = @ScriptLineNumber) ConsoleWrite("+++:" & $lnum & ": " & $msg & @CRLF) EndFunc ;==>LogMsg Parsing the returned $oPrinters object shows 5 printers:
      +++:15: FIRST LOOP +++:18: +++: isObj($oP) = 1 +++:22: +++: isObj($oPrinter): 1, $oPrinter.Name ==>Microsoft XPS Document Writer<== +++:22: +++: isObj($oPrinter): 1, $oPrinter.Name ==>Microsoft Office Document Image Writer<== +++:22: +++: isObj($oPrinter): 1, $oPrinter.Name ==>Fax<== +++:22: +++: isObj($oPrinter): 1, $oPrinter.Name ==>Canon MG7100 series Printer WS<== +++:22: +++: isObj($oPrinter): 1, $oPrinter.Name ==>Canon MG6100 series Printer WS<== +++:24: +++: Found 5 printers Parsing it again, shows no printers:
      +++:26: SECOND LOOP +++:29: +++: isObj($oP) = 1 +++:35: +++: Found 0 printers  
    • GhostLine
      By GhostLine
      Hello guys !
      I'm stuck in something stupid (I guess), but since I've no clue on how to solve it, here I come
      I'm trying to collect the result of the query "select @@hostname" on MSSQL Server, but it doesn't work ... and I'm sure I'm connected to the database (I've successfully seen the database treeview). 

      #Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseX64=n #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include<_sql.au3> #include<array.au3> Local $aData,$iRows,$iColumns ;Server ID and credentials for tCards Global $ServerAddressT = "10.200.88.1" Global $ServerUserNameT = "user" Global $ServerPasswordT = "password" Global $DatabaseNameT = "DataBASE" ;Connect to DB _SQL_RegisterErrorHandler();register the error handler to prevent hard crash on COM error $OADODB = _SQL_Startup() If $OADODB = $SQL_ERROR Then MsgBox(0 + 16 + 262144, "Error", _SQL_GetErrMsg()) If _sql_Connect(-1, $ServerAddressT, $DatabaseNameT, $ServerUserNameT, $ServerPasswordT) = $SQL_ERROR Then     MsgBox(0 + 16 + 262144, "Error 1", _SQL_GetErrMsg())     _SQL_Close()     Exit Else     If _SQL_Execute(-1,"select @@SERVERNAME") = $SQL_ERROR then         Msgbox(0 + 16 +262144,"Error",_SQL_GetErrMsg())     Else         _ArrayDisplay(_SQL_GetTable(-1,"select @@SERVERNAME",$aData,$iRows,$iColumns)) ;~         $toto = _SQL_Execute(-1,"select @@SERVERNAME") ;~         _ArrayDisplay($toto)     EndIf EndIf
×