Sign in to follow this  
Followers 0
Jewtus

Updating sqlite with values from MSSQL db

9 posts in this topic

I'm trying to figure out a couple things and I'm not sure what the best way to go is.

First I'm trying to create a sqlite DB with a csv file, but I also need to grab data from a MS SQL server associated with the csv file and associate them. Is there any examples someone can point me to so I can understand what I'm getting into here?

I'd like to create a separate table with info associated with the csv and import them both to a SQLite DB.

Share this post


Link to post
Share on other sites



You can use the command-line tool (sqlite3.exe) with the .import command to import .csv data into an sqlite table. This may not build the most suitable schema for your application but at least you should have the data there ready for manipulation, or transfer to another table created by you later having the right definition.

Now in your applicative code, you can use ADO support (search ADO in the example forum) to open ths MSSQL source and query it to grab data, which you can then insert/merge or otherwise process as needed.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Those are actually what I've been researching but I'm trying to find some examples that I can work from. My expertise is really more in SQL than general scripting. I have been able to figure out the connection string stuff with ADO but I'm not sure how to actually execute the query or save the executed data into an array so I can process it into the sqlite DB (I'm also not sure if it makes sense to save the data or just display it, so I'm weighing my options).

Share this post


Link to post
Share on other sites

I'll try to work out somehow soon a (very generic) skeleton to help you figure out how to get started. Of course, lacking more specific guidance, it won't follow your actual needs accurately.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Much appreciated. Once I understand how it actually functions, I will likely ask more questions and be more specific. I'm not really even sure if its worth loading to a DB (I need to run through about 50k records and use the results, but I don't need to save the results... just query, fill in info, and delete the query data).

Share this post


Link to post
Share on other sites

For the record, SQLite can handle disk-based or memory-based DBs. Of course a memory DB is way faster to populate, query, update... but it's temporary unless backed up to disk. I'll point you to an easy way to copy an SQLite DB while it's being used to/from disk or memory. 50k rows is a very small DB. This little thing is *very* flexible.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

That would be great. I didn't think I needed a DB (I have to find out if anyone actually cares if I keep logs or a DB to track the system). If I can process it via memory (even if it is just partial data that I log), that would be perfect.

I just need to get a functional design setup and I can flush out the details after the fact. 

Share this post


Link to post
Share on other sites

Exploring ever increasing logs is a pain. A simple DB is certainly more efficient, but up to you.

You have a number of CSV to/from Array examples (use the search) floating around. I posted one with extended format which preserves basic (and even not so basic) datatypes but it uses a somehow precise format.

I run out of time right now. I'll be back to you late in (my) evening.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

No rush. I'm just exploring my options at this point. I have a feeling I will need to log the input data (into a DB simply for trace-ability) but the data that I'm pulling using ADO doesn't need to be replicated to a different database.

Thanks again! 

PS I've been reading up on ADO on w3school (because they have examples that you can try realtime) but they syntax seems significant enough for me to be completely lost. I'm also running into weird issues with my import statement (works when I execute it via command line but when I use autoit, it doesn't import the data correctly). I'll post another thread if I end up going with the cmd line sqlite (I'll probably need to pack it in with the EXE so I don't have to configure environment variables on deployment.. I was hoping to use Autoit's SQLite connection, but that seems a bit out of reach until I get more versed in Autoit scripting)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • Burgs
      By Burgs
      Greetings,
        I have SQLite setup within my AutoIT program...I'm trying to accomplish what should be a relatively simple task.  I want to be able to return an array of 'table' names for an established database...I believe this might be possible using the '_SQLite_SQLiteExe' command...since it seems to be able to access SQLite schemas...?  The ".tables" command is one of them...if I am not mistaken that command returns a list of all table names in the active database.
        I am attempting the following:
      #include <SQLite.au3> #include <SQLite.dll.au3> Global $hDb, $sIn, $sOut ... $sIn = ".tables" & @CRLF _SQLite_SQLiteExe($hDb, $sIn, $sOut) if @error == 0 Then ;Show Table (using SQLite3.dll) Else if @error == 2 Then ConsoleWrite("ERROR: Sqlite3.exe file not found" & @CRLF) Else ConsoleWrite("ERROR: @error=" & @error & " when calling _SQLite_SQLiteExe" & @CRLF) EndIf ;@error is "2"...OR NOT... EndIf ;@error is "0"...OR NOT... ...   The error being thrown is "ERROR: Sqlite3.exe file not found" ...
        Am I required to have the Sqlite3.exe installed in my directory (i.e. @ScriptsDir)...???  I do not have it in there at present because I did not believe it was necessary with the 'include' calls to "SQLite.au3" and "SQLite.dll.au3"...any advice appreciated.  Thanks in advance. 
      Regards
       
       
       
       
       
       
       
    • FrancescoDiMuro
      By FrancescoDiMuro
      Good evening guys
      I am working on a little project, in which I have to retrieve 8000+ rows of data from a table, from a SQLite database, to populate a combobox 
      This is what I tried 'til now, but it still takes about 12 seconds to populate the combobox.
      _SQLite_Exec($objDatabase, "BEGIN TRANSACTION;") If _SQLite_GetTable($objDatabase, "SELECT DISTINCT Comune FROM LISTA_COMUNI;", $arrRisultatoQuery, $intRighe, $intColonne) = $SQLITE_OK Then For $i = 2 To UBound($arrRisultatoQuery) - 1 If $i < UBound($arrRisultatoQuery) - 1 Then GUICtrlSetData($cbo_ComuneNascita, $arrRisultatoQuery[$i] & "|") Else GUICtrlSetData($cbo_ComuneNascita, $arrRisultatoQuery[$i]) EndIf Next _SQLite_Exec($objDatabase, "COMMIT;") Are there any other solution to retrieve 8000+ records from a SQLite database?
      Thank you very much  
    • FrancescoDiMuro
      By FrancescoDiMuro
      Good morning community!
      I am working on a script which read from a text file ( .txt ) and should import all the content in a SQLite3 DB, in order to execute some queries that should be difficult to execute on a text file.
      So, I was looking for something very very fast, because the file could be very large ( I don't know exaclty how much can became big, but I know a lot of rows, it's a log file ... )
      I found the "Import method", but I don't know If I can implement it in a query ( @jchd, it's your turn!  )
      Do you know some methods that I can implement in my script to have a very very fast import of thousands and thousands rows in a SQLite3 DB?
      Thanks a lot
      Francesco
    • FrancescoDiMuro
      By FrancescoDiMuro
      Good morning guys
      I was trying to use again SQLite in AutoIt, and I've again an issue: I can't startup SQLite...
      Can you please show me the way to set the enviornment for a SQLIte usage in AutoIt? Step by step, I'll follow your suggestion(s).

      Thanks for everyone will reply!

      Francesco
    • FrancescoDiMuro
      By FrancescoDiMuro
      Good evening everyone
      I'm working on this little project for a week, and, what I'm trying to do could be useful for many users as well...
      I'm trying to do a "Report Generator", which reads the data that have to report from a text file (.txt) formatted with this pattern;
      Data1;Data2;Data3;Data4;Data5;;

      YES, there are 2 semi-colon at the end of the line.

      In detail, Data1 is a date/time stamp with this format: YYYY/MM/DD HH:MM:SS ;

      When the script starts, the user is prompted to choose 2 dates which I'll call as:
      Report_Date_Start; Report_Date_Start. So, the report, should cover all dates between Report_Date_Start AND Report_Date_End.

      And, already at this point, I don't know how to do the query... How can I say to the script:
      SELECT * FROM (.txt) WHERE Data1 BETWEEN Report_Date_Start AND Report_Date_End; ?

      I thought that I could do a _DateDiff, but if the difference between the two dates is months and not days, how can I do the trick?
      Should I make a Switch...Case with the _DateDiff() and see then calculate all the dates between Report_Date_Start AND Report_Date_End... But then, how can I compare the dates in the file with all the dates between Report_Date_Start AND Report_Date_End? I'm going crazy, I know...

      I've already made a "Export Tool", which exports the content of the .txt file in a .db, managed with SQLite... I mean, there I could easily do a query like I did above the thread, but, this "export", for 1080 rows, takes 28 seconds to be done. And, 1080 rows are daily rows that are added every day in the .txt file, so, in a week, the file could be easily 7000+ rows, which means that the "export" would take 3 minutes to be done... And we can go over and over...
      I'll post just for be "complete" what I've done about the export, so, maybe, someone could say how to improve it in terms of efficency...
       
      Local $aContenutoFileAuditReport = "" _FileReadToArray($sFileAudit_Report, $aContenutoFileAuditReport) If(IsArray($aContenutoFileAuditReport) And Not @error) Then Local $aContenutoFileAuditReport_Splitted = "" Local $sQuery = "" Local $hInizioConteggio = TimerInit() For $i = 1 To UBound($aContenutoFileAuditReport) - 1 $aContenutoFileAuditReport_Splitted = StringSplit($aContenutoFileAuditReport[$i], ";") $sQuery = "INSERT INTO FileDB_Report(DATESTAMP, TIMESTAMP, USER_ID, OBJECT_ID, DESCRIPTION, COMMENT) " & _ "VALUES(" & _ _SQLite_FastEscape($aContenutoFileAuditReport_Splitted[0]) & "," & _ _SQLite_FastEscape($aContenutoFileAuditReport_Splitted[1]) & "," & _ _SQLite_FastEscape($aContenutoFileAuditReport_Splitted[3]) & "," & _ _SQLite_FastEscape($aContenutoFileAuditReport_Splitted[4]) & "," & _ _SQLite_FastEscape($aContenutoFileAuditReport_Splitted[5]) & "," & _ _SQLite_FastEscape($aContenutoFileAuditReport_Splitted[6]) & ");" If(_SQLite_Exec($hFileDB_Report, $sQuery) <> $SQLITE_OK) Then ConsoleWrite("Errore durante l'esecuzione della query #" & $i & @CRLF) Else ConsoleWrite("Query eseguita correttamente #" & $i & @CRLF) EndIf Next ConsoleWrite("Esportazione completata in: " & Round(TimerDiff($hInizioConteggio)/1000, 0) & " secondi") Else MsgBox($MB_ICONERROR, "Errore!", "Errore durante la lettura del file nell'array." & @CRLF & "Errore: " & @error) EndIf I know that I can't do queries from a .txt file...
      [19:18] 
      I've been writing this post from 18:40 maybe...
      By the way, if @jchd or someone else could tell me if I can import a formatted .txt file in SQLite and then, do queries on the DB, I'd be very happy for that...
      About the report in PDF, I'm talking with @taietel in order to know how to create a PDF.
      I hope someone will help me
      Sorry for the "long" list of questions...
      Thank you for everything you've done for me
      I have to say that this is the community of programming language that I've loved most!
      By the way, I'll be back tomorrow in the morning ( ~ 9:15 a.m. Italian time ), so, excuse me if I can't answer before that time.
      Hope you guys have a wonderful day/night.

      Thanks again

      Francesco