Sign in to follow this  
Followers 0
Jewtus

Updating sqlite with values from MSSQL db

9 posts in this topic

I'm trying to figure out a couple things and I'm not sure what the best way to go is.

First I'm trying to create a sqlite DB with a csv file, but I also need to grab data from a MS SQL server associated with the csv file and associate them. Is there any examples someone can point me to so I can understand what I'm getting into here?

I'd like to create a separate table with info associated with the csv and import them both to a SQLite DB.

Share this post


Link to post
Share on other sites



You can use the command-line tool (sqlite3.exe) with the .import command to import .csv data into an sqlite table. This may not build the most suitable schema for your application but at least you should have the data there ready for manipulation, or transfer to another table created by you later having the right definition.

Now in your applicative code, you can use ADO support (search ADO in the example forum) to open ths MSSQL source and query it to grab data, which you can then insert/merge or otherwise process as needed.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Those are actually what I've been researching but I'm trying to find some examples that I can work from. My expertise is really more in SQL than general scripting. I have been able to figure out the connection string stuff with ADO but I'm not sure how to actually execute the query or save the executed data into an array so I can process it into the sqlite DB (I'm also not sure if it makes sense to save the data or just display it, so I'm weighing my options).

Share this post


Link to post
Share on other sites

I'll try to work out somehow soon a (very generic) skeleton to help you figure out how to get started. Of course, lacking more specific guidance, it won't follow your actual needs accurately.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Much appreciated. Once I understand how it actually functions, I will likely ask more questions and be more specific. I'm not really even sure if its worth loading to a DB (I need to run through about 50k records and use the results, but I don't need to save the results... just query, fill in info, and delete the query data).

Share this post


Link to post
Share on other sites

For the record, SQLite can handle disk-based or memory-based DBs. Of course a memory DB is way faster to populate, query, update... but it's temporary unless backed up to disk. I'll point you to an easy way to copy an SQLite DB while it's being used to/from disk or memory. 50k rows is a very small DB. This little thing is *very* flexible.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

That would be great. I didn't think I needed a DB (I have to find out if anyone actually cares if I keep logs or a DB to track the system). If I can process it via memory (even if it is just partial data that I log), that would be perfect.

I just need to get a functional design setup and I can flush out the details after the fact. 

Share this post


Link to post
Share on other sites

Exploring ever increasing logs is a pain. A simple DB is certainly more efficient, but up to you.

You have a number of CSV to/from Array examples (use the search) floating around. I posted one with extended format which preserves basic (and even not so basic) datatypes but it uses a somehow precise format.

I run out of time right now. I'll be back to you late in (my) evening.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

No rush. I'm just exploring my options at this point. I have a feeling I will need to log the input data (into a DB simply for trace-ability) but the data that I'm pulling using ADO doesn't need to be replicated to a different database.

Thanks again! 

PS I've been reading up on ADO on w3school (because they have examples that you can try realtime) but they syntax seems significant enough for me to be completely lost. I'm also running into weird issues with my import statement (works when I execute it via command line but when I use autoit, it doesn't import the data correctly). I'll post another thread if I end up going with the cmd line sqlite (I'll probably need to pack it in with the EXE so I don't have to configure environment variables on deployment.. I was hoping to use Autoit's SQLite connection, but that seems a bit out of reach until I get more versed in Autoit scripting)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • rynow
      By rynow
      romaSQL
      This autoIt UDF is built on the concept of Laravel Query & doctrine.
      RomaSQL provides a new, comfortable and easy to use way for SQL-queries in autoIt.
      Most of the common SQL-queries are supported already and more are coming soon.
      All of your support is much appreciated.
      Connections
      For the connection the object ADODB is used. Therefore the connection string is based on ODBC.
      You can also use OLEDB connection strings or other database connections.
      In order for this to work your add-ons have to be installed in the function: __4ern_SQL_Connection.
      I’d be very glad if you shared your modifications with me.
      Currently supported connections
      -       MySQL (odbc)
      -       Microsoft SQL Server (odbc)
      -       SQLite (odbc)
      -       Microsoft Access (odbc)
      Command reference
      $SQL_connect; establishing connection $SQL_returnType; return a Array or Dictionary ('oDict') Object (Default = Array) $SQL_setDefaultTable; Default Tablename $SQL_setDefaultKey; Default Colmn Key (Default = id) $SQL_debug; if True, show SQL Statment in Console $SQL_get $SQL_update $SQL_delete $SQL_insertInto $SQL_take $SQL_limit $SQL_table $SQL_select $SQL_distinct $SQL_where $SQL_orWhere $SQL_whereBetween $SQL_whereNotBetween $SQL_whereIn $SQL_whereNotIn $SQL_whereNull $SQL_whereNotNull $SQL_having $SQL_orHaving $SQL_havingBetween $SQL_havingNotBetween $SQL_havingIn $SQL_havingNotIn $SQL_havingNull $SQL_havingNotNull $SQL_groupBy $SQL_orderBy  
      Examples
      establishing connection
      ;-----/ ; SQLite Connection ;-----/ $SQL_setDatabase('sqlite') $SQL_connect('C:\project.db') ;-----/ ; Access Connection ; Database, User, Password ;-----/ $SQL_setDatabase('access') $SQL_connect('C:\project.mdb') ;or as Admin $SQL_connect('C:\project.mdb', '4ern', 'root') ;-----/ ; SQLServer Connection ; Database, User, Password, Server, Driver ;-----/ $SQL_setDatabase('sqlserver') $SQL_connect('myDB', '4ern', 'root', 'localhost') ;or with Driver $SQL_connect('myDB', '4ern', 'root', 'localhost', 'SQL Server') ;-----/ ; MySQL Connection ; Database, User, Password, Server, Driver ;-----/ $SQL_setDatabase('mysql') $SQL_connect('myDB', '4ern', 'root', 'localhost') ;or with Driver $SQL_connect('myDB', '4ern', 'root', 'localhost', 'MySQL ODBC 5.2 UNICODE Driver')  
      simple SQL query
      $SQL_table('albums') $aRet = $SQL_get() if IsArray($aRet ) then _ArrayDisplay($aRet ) else ConsoleWrite('Keine Ergebnisse' & @LF) endif  
      Select
      $SQL_table('albums') $SQL_select('id', 'Name', 'Artist', 'Song') ;or pass to an Array Local $aSelect = ['id', 'Name', 'Artist', 'Song'] $SQL_select($aSelect) $aRet = $SQL_get() if IsArray($aRet ) then _ArrayDisplay($aRet ) else ConsoleWrite('Keine Ergebnisse' & @LF) endif where
      $SQL_table('albums') $SQL_select('id', 'Name', 'Artist', 'Song', 'Votes') $SQL_where('Artist', 'adele') $SQL_where('Votes', '>=' ,'9') $SQL_orWhere('Artist', '=' ,'Rag'n'Bone Man') ;or pass to an 2dArray Local $aSelect = [['Artist','adele'],['Votes', '>=' ,'9']] $SQL_where($aSelect) $aRet = $SQL_get() if IsArray($aRet ) then _ArrayDisplay($aRet ) else ConsoleWrite('Keine Ergebnisse' & @LF) endif  
      If you need more examples, then tell me exactly what you need.
      I hope you like my UDF and find some use for it.
      ---
      ->DONWLOAD romaSQL
       
       
    • BisherSH
      By BisherSH
      Good day ,
      I have the code below , and i would like to embed SQLite3.dll to the compiled file 
      Is it possible ? 
      Thanks in advance
      #include <SQLite.au3> #include <SQLite.dll.au3> _SQLite_Startup() If @error Then     MsgBox($MB_SYSTEMMODAL, "SQLite Error", "SQLite3.dll Can't be Loaded!")     Exit -1 EndIf $DB = _SQLite_Open("C:\Temp\Test.db") $Action = "TestAction" $Time = @HOUR&":"&@MIN&":"&@SEC $Date = @YEAR&"-"&@MON&"-"&@MDAY $User = @UserName $Computer = @ComputerName $DC = @LogonServer If @error Then     MsgBox($MB_SYSTEMMODAL, "SQLite Error", "Couldnt open Database")     Exit -1 EndIf _SQLite_Exec($DB,"INSERT INTO QLogs (Action,Date,Time,User,Computer,DC) " & _                               "VALUES ("& _SQLite_FastEscape($Action) & "," & _                                           _SQLite_FastEscape($Date) & "," & _                                           _SQLite_FastEscape($Time) & "," & _                                           _SQLite_FastEscape($User) & "," & _                                           _SQLite_FastEscape($Computer) & "," & _                                           _SQLite_FastEscape($DC) & ");") If @error Then     MsgBox($MB_SYSTEMMODAL, "SQLite Error", "Couldnt insert!")     Exit -1 EndIf _SQLite_Shutdown()
    • supersonic
      By supersonic
      Hello -
      For a SQLite insert statement I'm passing a 2D array through two For/Next loops -  code snippet:
      Local $sInsertInto2 = "" For $i = 0 To $iCount - 1 $sInsertInto2 &= " (" For $j = 0 To UBound($aTmp, 2) - 2 $sInsertInto2 &= "'" & StringReplace($aTmp[$i][$j], "'", "''") & "', " Next $sInsertInto2 &= "'" & StringReplace($aTmp[$i][$j], "'", "''") & "')" & ((($iCount - 1) = $i) ? (";") : ("," & @CRLF)) Next This works great but due to StringReplace() it is very slow. StringReplace() is used to "escape" single quotes (') and it is only required if a single quote actually is present...
      Somewhere on the forum there is a faster example by UEZ but I can't find it...
      Anyone any idea?
    • FrancescoDiMuro
      By FrancescoDiMuro
      Hi guys! How are you? Hope you're fine
      I'm trying to insert a text that contains quotes in a SQLite database, and I don't know how to do.
      I tried with:
      Local $sString = "1P6AV2104'0HA04'0AA0" StringReplace($sString, "'", "''") but I didn't managed to, obtaining this error:
      --> Error:    unrecognized token: "0HA04" How can I manage to solve this problem? Thanks  
    • FrancescoDiMuro
      By FrancescoDiMuro
      Hi guys! How are you? Hope you're fine
      I'm trying to use SQLite for managing some data, and, I would like to display my "retrieved" data, but I'm trying to do a select from 2 table, and seems to not work properly...
      What I'd like to do is retrieve data from 2 tables and display in a listview...
      I tried with:
      Local $aRisultato, $iRighe, $iColonne, $iRVal $iRval = _SQLite_GetTable2d($hDatabase, "SELECT * FROM Magazzino_Rockwell, Magazzino_Siemens;", $aRisultato, $iRighe, $iColonne) If $iRVal = $SQLITE_OK Then ;_SQLite_Display2DResult($aRisultato) _ArrayDisplay($aRisultato) EndIf ... but it displays a single record 41 times, and it does this thing for every record in the database...
      What do you need to help me?
      Thanks  

      EDIT:
      Managed with a double If and double query like this:
       
      $iRval = _SQLite_GetTable2d($hDatabase, "SELECT * FROM Magazzino_Rockwell;", $aRisultato, $iRighe, $iColonne) If $iRVal = $SQLITE_OK Then Local $aRisultatoRockwell = $aRisultato $iRval = _SQLite_GetTable2d($hDatabase, "SELECT * FROM Magazzino_Siemens;", $aRisultato, $iRighe, $iColonne) If $iRVal = $SQLITE_OK Then _ArrayConcatenate($aRisultato, $aRisultatoRockwell) Local $i, $sRiga, $s_LV_Item For $i = 1 To Ubound($aRisultato) - 1 $sRiga = $aRisultato[$i][0] & "|" & _ $aRisultato[$i][1] & "|" & _ $aRisultato[$i][2] & "|" & _ $aRisultato[$i][3] & "|" & _ $aRisultato[$i][4] & "|" & _ $aRisultato[$i][5] & "|" & _ $aRisultato[$i][6] & "|" & _ $aRisultato[$i][7] & "|" & _ $aRisultato[$i][8] & "|" & _ $aRisultato[$i][9] & "|" & _ $aRisultato[$i][10] & "|" & _ $aRisultato[$i][11] $s_LV_Item = GUICtrlCreateListViewItem($sRiga, $lv_Lista) Next And so, the listview is created
      If anyone has another more efficient way, I'm here Thanks guys