Jump to content
FrancescoDiMuro

SQLite Query for populating combobox

Recommended Posts

FrancescoDiMuro

Good evening guys :)

I am working on a little project, in which I have to retrieve 8000+ rows of data from a table, from a SQLite database, to populate a combobox :)

This is what I tried 'til now, but it still takes about 12 seconds to populate the combobox.

_SQLite_Exec($objDatabase, "BEGIN TRANSACTION;")
If _SQLite_GetTable($objDatabase, "SELECT DISTINCT Comune FROM LISTA_COMUNI;", $arrRisultatoQuery, $intRighe, $intColonne) = $SQLITE_OK Then
    For $i = 2 To UBound($arrRisultatoQuery) - 1
        If $i < UBound($arrRisultatoQuery) - 1 Then
            GUICtrlSetData($cbo_ComuneNascita, $arrRisultatoQuery[$i] & "|")
        Else
            GUICtrlSetData($cbo_ComuneNascita, $arrRisultatoQuery[$i])
        EndIf
    Next
_SQLite_Exec($objDatabase, "COMMIT;")

Are there any other solution to retrieve 8000+ records from a SQLite database?

Thank you very much :) 


Click here to see my signature:

Spoiler

Thoughts:

  • I will always thank you for the time you spent for me.
    I'm here to ask, and from your response, I'd like to learn.
    By my knowledge, I can help someone else, and "that someone" could help in turn another, and so on.

/*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------*/

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites
Danyfirex

Hello. Use _GUICtrlComboBox_AddString is faster than GUICtrlSetData

 

Saludos

Share this post


Link to post
Share on other sites
jchd

@FrancescoDiMuro,

Remove BEGIN and COMMIT. You don't need to enclose a single select (or any other single SQL statement BTW) in a transaction. SQLite will do that for you under the hood.

Now why is selecting distinct values so slow? Do you have duplicates in your table? If yes you should remove extra copies. If you don't have duplicate rows but only several rows for the same "Comune" column, then the design is flawed: you shouldn't duplicate data like that. Use a separate table and foreign keys for that.

To make you understand what I mean, imagine you have a table of all streets in a country. A bad design would be:
Id, Region, City, Street
where Region and City will be repeated for every street.

A better design will use a Regions table, a Cities Table and a Streets table.

Regions:
RegionId, RegionName (Unique)

Cities:
CityId, CityName (Unique in a given region)

Streets:
Id, RegionId, CityId, StreetName
where RegionId is a foreign key to the Regions table and CityId a foreign key to the Cities table.

Of course this is just a simplified example to fix ideas. This process of avoiding data duplication is called DB normalization.


Besides these considerations, you can also use the power of SQL to return you a single string with all the data and delimiters in place, for instance:

_SQLite_QuerySingleRow($objDatabase, "select group_concat(Comune, '|') FROM (select distinct comune from LISTA_COMUNI) group by null;", $Row)

On my table of 75377 zip codes for 32 countries, this request takes 300ms on my very slow PC. This way you can feed $Row to the combobox all in once. Yet, I personally suspect that a combobox with 8000+ entries is definitely unpractical, but that is up to you.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
FrancescoDiMuro

Good morning @jchd:)

The table I have does not contain more than a couple of duplicated rows...

What I'd like to have, is retrieve all the data from "Comune", and display them in a combobox, in order to let the user select what he wants to.

Do you have any pratical suggestion on how to display so many rows, and let select just one of them, in order to manage the data selected?

For SQLite: I can try with what you did suggest to me... But then, I have to split data, am I right? :)

Thank you a lot :) 


Click here to see my signature:

Spoiler

Thoughts:

  • I will always thank you for the time you spent for me.
    I'm here to ask, and from your response, I'd like to learn.
    By my knowledge, I can help someone else, and "that someone" could help in turn another, and so on.

/*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------*/

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites
jchd

I find it irritating when I'm offered a choice in some overpopulated, endless and boring listbox of combobox.

Restricting the choices to the few which loosely match user input seems preferable. I've made available an SQLite extension DLL named "Unifuzz" which offers a number of Unicode string and collations functions, among which a typos() function returning the Damerau-Levenshtein distance between two strings. I often use it this way, loosely adapted to what I understand of your context:

select distinct comune from comuni where typos(comune, $input) < 3 order by comune;

This returns all distinct comune that are 4 character change away from the string in variable $input. For instance, searching cities in Belgium,
select distinct ville from codespostaux where paysiso = 'BE' and typos(ville, 'böan') < 3 order by ville;
returns:

Ville
BAAL
BOHAN
BOOM
BOST
BRA
BRAS
BRAY
DOHAN
MEAN
MOEN
ON


You can see that allowing as little as two typos already grabs a fair but manageable number of plausible choices. This function works by first folding case and accents on the whole BMP unicode range. Sorry I don't have a list of italian comuni to examplify.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
jchd

_ArrayToString is a possibility but using the power of fast SQL function will always reveal faster than UDF code.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • Eminence
      By Eminence
      Hello,
      Using SQLite, I was trying to select a specific column with having duplicate entries removed by using the DISTINCT function. 
      Local $sQuery = "SELECT DISTINCT supervisor, COUNT(DISTINCT employee_name) FROM data_db What the above snippet does is just list only one supervisor but with the total count of unique employee names in the whole database when it should be listing out all supervisor entries from the database and list the total count of employees per each supervisors. If I remove the COUNT function, it does list out all unique entries of supervisor names from the list. Attached is a screenshot of an example database as well. 
      Any help will be much appreciated. Thank you!

    • Seminko
      By Seminko
      Hey,
      I have a script that uses SQLite. It worked without a hiccup on my W7 system.
      However, last week I bought a new machine, installed W10 and autoit, and now I'm getting an error during _SQLite_Startup.
      Can anyone advise on how to find what's wrong? Tried checking $__g_hPrintCallback_SQLite but that returns nothing.
       
      Tried downloading the latest version of the dll from the link in the function page and I also replaced the default SQLite.dll.au3 that came with AutoIt with the one that came with the latest SQLite version zip.
       
      Relevant parts of my script:
      #include <SQLite.au3> #include <SQLite.dll.au3> _SQLite_Startup(@ScriptDir & "\Config\sqlite3_x64.dll", False, 1) If @error Then MsgBox(16, "SQLite Error", "SQLite3.dll Can't be Loaded! - " & $__g_hPrintCallback_SQLite & @CRLF & @CRLF & "Exiting application / Zavolej Honzovi") Exit -1 EndIf  
      Thanks
      S.
    • FrancescoDiMuro
      By FrancescoDiMuro
      Good evening everyone
      I am building a management for the company I work with, and I just imported a real amount of rows ( about 29000 ), in my SQLite DB.
      The thing I am not understanding, is the time that the script takes to build this amount of rows in the ListView.
      I didn't measure it, but I think it took 2 minutes or so to create each ListView item...
      It is normal that it takes so much time?
      What can I do to improve the creation of the items?

      Here's the code I am using to query and to create ListView items...
      ; Articles ListView: Global $lvwArticles = GUICtrlCreateListView("ID|Fornitore|Codice|Descrizione|EU|Prezzo|Sconto Applicato|Note", 14, 87, 1507, 660, BitOR($GUI_SS_DEFAULT_LISTVIEW,$LVS_SORTASCENDING,$LVS_SORTDESCENDING), BitOR($WS_EX_CLIENTEDGE,$LVS_EX_GRIDLINES,$LVS_EX_FULLROWSELECT)) ; Query $strQuery = "SELECT * FROM ARTICOLI;" ; Query Execution _SQLite_GetTable2d($objDatabase, $strQuery, $arrResult, $intRows, $intColumns) If @error Then ; Error Handling Else ; Cleaning the ListView _GUICtrlListView_DeleteAllItems($lvwArticles) If @error Then ; Error Handling Else ; No records in the Table If UBound($arrResult) < 2 Then ; Error Handling Else _GUICtrlListView_BeginUpdate($lvwArticles) For $intCounter = 1 To UBound($arrResult) - 1 $strListViewItem = $arrResult[$intCounter][0] & "|" & _ $arrResult[$intCounter][1] & "|" & _ $arrResult[$intCounter][2] & "|" & _ $arrResult[$intCounter][3] & "|" & _ $arrResult[$intCounter][4] & "|" & _ $arrResult[$intCounter][5] & "|" & _ $arrResult[$intCounter][6] & "|" & _ $arrResult[$intCounter][7] $objListViewItem = GUICtrlCreateListViewItem($strListViewItem, $lvwArticles) Next _GUICtrlListView_EndUpdate($lvwArticles) EndIf EndIf EndIf Thanks in advance


      Best Regards.
    • odaylton
      By odaylton
      Hello everyone ,
      Text translated from Portuguese by google - please apologize for any errors

      Once again I need a light from the gurus of this wonderful website
      I am fanatic by combobox for its usability but I am not able to uncertain images within lists ...
      I know uncertain icons, and cursors but I am not able to insert gif or jpg images.
      1) How to solve this ...
      2) will the path is to convert gif into bmp and then uncertain ... how to do that?
      #include <GUIConstantsEx.au3> #include <GuiImageList.au3> #include <GuiListView.au3> #include <WinAPI.au3> #include <WindowsConstants.au3> ;dados de mouse #include <WinAPIRes.au3> ;COMBO #include <GuiComboBoxEx.au3> ;Fonts #include <FontConstants.au3> ;obter imagem de arquivo #include <GDIPlus.au3> #include <WinAPIGdi.au3> Example() Func Example() Local $hGUI, $hImage, $hCombo, $hFont Local $idListview, $hImage Local $sWow64 = "" Local $Pasta = "C:\WINDOWS\Cursors\3dgarro.cur" Local $PastaROMs = @MyDocumentsDir & "\Stella\ROMs\" ;com barra no final $hGUI = GUICreate("ImageList", 500, 310) GUISetFont(14, 400, 0, "Arial") ;<<<<<<<<<<this does not work for _GUICtrlComboBoxEx_Create $hCombo = _GUICtrlComboBoxEx_Create($hGUI, "", 2, 2, 494, 200) ;GUICtrlSetFont($hCombo, 14, 400, 0, "Arial");<<<<<<<<<<It does not work ; ;Create a handle to a font object $hFont = _WinAPI_CreateFont(30, 0, 0, 0, 400, False, False, False, $DEFAULT_CHARSET, $OUT_DEFAULT_PRECIS, $CLIP_DEFAULT_PRECIS, $DEFAULT_QUALITY, 0, 'Arial') ;Pass the handle to _WinAPI_SetFont _WinAPI_SetFont($hCombo, $hFont) $idListview = GUICtrlCreateListView("", 2, 50, 494, 250, BitOR($LVS_SHOWSELALWAYS, $LVS_NOSORTHEADER, $LVS_REPORT)) _GUICtrlListView_SetExtendedListViewStyle($idListview, BitOR($LVS_EX_FULLROWSELECT, $LVS_EX_GRIDLINES, $LVS_EX_DOUBLEBUFFER)) GUISetState(@SW_SHOW) ; Initialize GDI+ library _GDIPlus_Startup() ; Load images $hImage = _GUIImageList_Create(120, 70, 6, 2) ;32,32,5,5) ; $DadosCombo = "Escolha|" & _GUIImageList_Add($hImage, _GUICtrlComboBoxEx_CreateSolidBitMap($hCombo, 0x0000FF, 120, 70)) ;Local $aCursors[] = [32650, 32512, 32515, 32649, 32651, 32513, 32648, 32646, 32643, 32645, 32642, 32644, 32516, 32514] For $i = 1 To 10 $sFile = FileOpenDialog("Please select an image", $PastaROMs, "Image (*.jpg;*.png;*.bmp;*.gif;*.tif)", BitOR($FD_PATHMUSTEXIST, $FD_FILEMUSTEXIST)) If @error Then Exit MsgBox(BitOR($MB_TOPMOST, $MB_ICONERROR), "Error", "No image file has been selected", 30) $a = _GDIPlus_BitmapCreateFromFile ($sFile);<<<<< ok If @error Or Not $hImage Then MsgBox(BitOR($MB_TOPMOST, $MB_ICONERROR), "Error", "This file isn't supported by GDIPlus!") Else $b=_GUIImageList_Add($hImage, $a);<<<<<< Return -1 $DadosCombo &= "|" & $i & "_img_princ.gif""|" & $b EndIf Next _GUICtrlListView_SetImageList($idListview, $hImage, 1) _GUICtrlComboBoxEx_SetImageList($hCombo, $hImage) ; Add columns _GUICtrlListView_AddColumn($idListview, "Icons - Mouses", 350) ; Add items $a = StringSplit($DadosCombo, "|") For $i = 1 To $a[0] Step 2 _GUICtrlListView_AddItem($idListview, $a[$i], $a[$i + 1]) _GUICtrlComboBoxEx_AddString($hCombo, $a[$i], $a[$i + 1], $a[$i + 1]) Next ;AutoIt_Debugger_Command:Disable_Debug ; Loop until the user exits. Do Until GUIGetMsg() = $GUI_EVENT_CLOSE GUIDelete() ;AutoIt_Debugger_Command:Enable_Debug EndFunc ;==>Example tks
    • aiter
      By aiter
      I  possibly misunderstood the value of combo boxes, but I thought you could type a value, press enter and the combo box would store that value.
      The reason for wanting this is to store a history of typed commands into the combo box (accepted by pressing enter).
       
      After hours of trying to trap enter on a combobox I realized that the enter event was not registering.
      I had code like this (which used to work on a input box), but it does not for a combobox (because it does not accept enter)
      unc WM_COMMAND($hWnd, $Msg, $wParam, $lParam) Local $nNotifyCode = BitShift($wParam, 16) Local $nID = BitAND($wParam, 0x0000FFFF) If $hWnd = $CmdForm Then If $nID = $ComboInput Then ; useed to work on a normal input box If _IsPressed('0D') Then ; enter was pressed, respond DoCmd() Anyone able to help?
×