Jump to content
FrancescoDiMuro

SQLite Query for populating combobox

Recommended Posts

Good evening guys :)

I am working on a little project, in which I have to retrieve 8000+ rows of data from a table, from a SQLite database, to populate a combobox :)

This is what I tried 'til now, but it still takes about 12 seconds to populate the combobox.

_SQLite_Exec($objDatabase, "BEGIN TRANSACTION;")
If _SQLite_GetTable($objDatabase, "SELECT DISTINCT Comune FROM LISTA_COMUNI;", $arrRisultatoQuery, $intRighe, $intColonne) = $SQLITE_OK Then
    For $i = 2 To UBound($arrRisultatoQuery) - 1
        If $i < UBound($arrRisultatoQuery) - 1 Then
            GUICtrlSetData($cbo_ComuneNascita, $arrRisultatoQuery[$i] & "|")
        Else
            GUICtrlSetData($cbo_ComuneNascita, $arrRisultatoQuery[$i])
        EndIf
    Next
_SQLite_Exec($objDatabase, "COMMIT;")

Are there any other solution to retrieve 8000+ records from a SQLite database?

Thank you very much :) 


Click here to see my signature:

Spoiler

Thoughts:

  • I will always thank you for the time you spent for me.
    I'm here to ask, and from your response, I'd like to learn.
    By my knowledge, I can help someone else, and "that someone" could help in turn another, and so on.

/*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------*/

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites

Share this post


Link to post
Share on other sites

@FrancescoDiMuro,

Remove BEGIN and COMMIT. You don't need to enclose a single select (or any other single SQL statement BTW) in a transaction. SQLite will do that for you under the hood.

Now why is selecting distinct values so slow? Do you have duplicates in your table? If yes you should remove extra copies. If you don't have duplicate rows but only several rows for the same "Comune" column, then the design is flawed: you shouldn't duplicate data like that. Use a separate table and foreign keys for that.

To make you understand what I mean, imagine you have a table of all streets in a country. A bad design would be:
Id, Region, City, Street
where Region and City will be repeated for every street.

A better design will use a Regions table, a Cities Table and a Streets table.

Regions:
RegionId, RegionName (Unique)

Cities:
CityId, CityName (Unique in a given region)

Streets:
Id, RegionId, CityId, StreetName
where RegionId is a foreign key to the Regions table and CityId a foreign key to the Cities table.

Of course this is just a simplified example to fix ideas. This process of avoiding data duplication is called DB normalization.


Besides these considerations, you can also use the power of SQL to return you a single string with all the data and delimiters in place, for instance:

_SQLite_QuerySingleRow($objDatabase, "select group_concat(Comune, '|') FROM (select distinct comune from LISTA_COMUNI) group by null;", $Row)

On my table of 75377 zip codes for 32 countries, this request takes 300ms on my very slow PC. This way you can feed $Row to the combobox all in once. Yet, I personally suspect that a combobox with 8000+ entries is definitely unpractical, but that is up to you.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Good morning @jchd:)

The table I have does not contain more than a couple of duplicated rows...

What I'd like to have, is retrieve all the data from "Comune", and display them in a combobox, in order to let the user select what he wants to.

Do you have any pratical suggestion on how to display so many rows, and let select just one of them, in order to manage the data selected?

For SQLite: I can try with what you did suggest to me... But then, I have to split data, am I right? :)

Thank you a lot :) 


Click here to see my signature:

Spoiler

Thoughts:

  • I will always thank you for the time you spent for me.
    I'm here to ask, and from your response, I'd like to learn.
    By my knowledge, I can help someone else, and "that someone" could help in turn another, and so on.

/*--------------------------------------------------------------------------------------------------------------------------------------------------------------------------*/

ALWAYS GOOD TO READ:

 

Share this post


Link to post
Share on other sites

I find it irritating when I'm offered a choice in some overpopulated, endless and boring listbox of combobox.

Restricting the choices to the few which loosely match user input seems preferable. I've made available an SQLite extension DLL named "Unifuzz" which offers a number of Unicode string and collations functions, among which a typos() function returning the Damerau-Levenshtein distance between two strings. I often use it this way, loosely adapted to what I understand of your context:

select distinct comune from comuni where typos(comune, $input) < 3 order by comune;

This returns all distinct comune that are 4 character change away from the string in variable $input. For instance, searching cities in Belgium,
select distinct ville from codespostaux where paysiso = 'BE' and typos(ville, 'böan') < 3 order by ville;
returns:

Ville
BAAL
BOHAN
BOOM
BOST
BRA
BRAS
BRAY
DOHAN
MEAN
MOEN
ON


You can see that allowing as little as two typos already grabs a fair but manageable number of plausible choices. This function works by first folding case and accents on the whole BMP unicode range. Sorry I don't have a list of italian comuni to examplify.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

_ArrayToString is a possibility but using the power of fast SQL function will always reveal faster than UDF code.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By DannyJ
      Hello Forum, 
      I have one special Combo-box [with BitOR($CBS_DROPDOWNLIST, $GUI_SS_DEFAULT_COMBO, $CBS_SIMPLE) ] and if I press one button I want to change the Combo-box's background color, and after I press Button B I want to change it to the basic settings and it not works.
      I wanna do this with one combo-box.
      I have already tried several methods and I tried  GUICtrlComboSetColors that I have found on this forum.
      (But this methods works perfectly with Cobo boxes that don't have BitOR($CBS_DROPDOWNLIST, $GUI_SS_DEFAULT_COMBO, $CBS_SIMPLE). 
      Here is the example code
      #include <ButtonConstants.au3> #include <ComboConstants.au3> #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> #Region ### START Koda GUI section ### Form= ;$CBS_DROPDOWNLIST ;$GUI_SS_DEFAULT_COMBO $Form1 = GUICreate("Form1", 504, 249, 252, 227) $Combo1 = GUICtrlCreateCombo("", 136, 24, 193, 25,BitOR($CBS_DROPDOWNLIST, $GUI_SS_DEFAULT_COMBO, $CBS_SIMPLE)) ; I want to change this special combo's background color if I press button 1 $Button1 = GUICtrlCreateButton("Button1", 112, 96, 75, 25) $Button2 = GUICtrlCreateButton("Button2", 264, 96, 75, 25) GUISetState(@SW_SHOW) #EndRegion ### END Koda GUI section ### GUICtrlSetData($Combo1," " & "|" & "apple" & "|" & "banana" & "|" & "cherry" ," ") While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit Case $Button1 GUICtrlSetBkColor($Combo1,0x0078D7) Case $Button2 ;Makes the original bc color GUICtrlSetBkColor($Combo1,0xFFFFFF) EndSwitch WEnd  
      Thanks you in advance your help
    • By Skysnake
      Hi
      I am trying to set Accelerator keys from an array.
      I select the KEY and CONTROL from a SQLite table, the Array looks like that generated for the Helpfile, but I can't get the CONTROLS to resolve...
      I though about Assign & Eval, but not sure if that's a step in the right direction. IsDeclared shows that the $var exists in Local Scope -1.  
       
      Local $Main = GUICreate("Custom MsgBox", 225, 80) GUICtrlCreateLabel("Please select a button.", 10, 10) Local $idButton_Yes = GUICtrlCreateButton("Yes", 10, 50, 65, 25) Local $idButton_No = GUICtrlCreateButton("No", 80, 50, 65, 25) Local $idButton_Exit = GUICtrlCreateButton("Exit", 150, 50, 65, 25) Local $query, $aResult, $iRows, $iColumns $query = "" ;reset $query = "Select hotkey_key, hotkey_ctrl from mytable where mykeys = 'hotkey' ; " ; ; Query $iRval = _SQLite_GetTable2d($sqliteDb, $query, $aResult, $iRows, $iColumns) If $iRval = $SQLITE_OK Then Local $sizeofHotkeys = UBound($aResult) - 1 ConsoleWrite("$sizeofHotkeys " & $sizeofHotkeys & @CRLF) If $sizeofHotkeys > 0 Then Local $main__aAccelKeys[$sizeofHotkeys][2] For $i = 0 To $sizeofHotkeys - 1 $j = $i + 1 ; replace friendly text with code -- ! alt + Shift ^ Ctrl # Windows $aResult[$j][0] = StringReplace($aResult[$j][0], "Alt", "!") $aResult[$j][0] = StringReplace($aResult[$j][0], "Shift", "+") $aResult[$j][0] = StringReplace($aResult[$j][0], "Ctrl", "^") $main__aAccelKeys[$i][0] = $aResult[$j][0] ;--- $main__aAccelKeys[$i][1] = $aResult[$j][1] ;--- Next ;~ Row|Col 0|Col 1 ;~ Row 0|F2|$idButton_Yes ;~ Row 1|F3|$idButton_No _DebugArrayDisplay($main__aAccelKeys) Local $rv = GUISetAccelerators($main__aAccelKeys, $Main) GUISetState(@SW_SHOW) ; Display the GUI.  
      Please note that this is a modified Helpfile example.
      The Helpfile specifies (a) WinHandle and (b) last Gui created.
      --> the example uses a control not a WinHandle and (b) what happens with ChildGuis? Also, the HelpFile specifies lower case, yet the examples show "{F1}" upper case?
      Also, is there a way to check the result of the GuiSetAccelerator function? 
      Note, if I add these to lines after the FOR loop, then the F1 works, and the DebugArrayDisplays shows control 4... not it's name... So I am in the right place, but my $vars names do not convert to their control numbers in the GUI
      Next $main__aAccelKeys[$sizeofHotkeys - 1][0] = "{F1}" ; -- -- use the extra row for the F1 $main__aAccelKeys[$sizeofHotkeys - 1][1] = $ChmHLP ;--- Skysnake
    • By Skeletor
      Hi All,
      Here's a really simple question. 
      I ran the code from the helpfile under: _SQLite_Open

      Issue is I end up with an error message: SQLite3.dll Can't be Loaded!
      I placed the *.dll in the include folder, but still nothing. 

      Where must this file be placed.
       
    • By Reher
      Hi
      so i ran into a Problem i didn't expect to get stuck on.
      It sounds simple (and probably is) but i can't seem to get it to work
      So i created a Combobox and filled it with some things from an array which worked fine.
      $combobox = GUICtrlCreateCombo($list[0], 160, 200, 265, 25, BitOR($CBS_DROPDOWN,$CBS_AUTOHSCROLL)) GuiCtrlSetData($list, "|" & _ArrayToString($list,Default,1),$list[2]) GUICtrlSetFont(-1, 9, 400, 0, "Arial") Now i want to change the style of the combobox from $CBS_DROPDOWN to $CBS_DROPDOWNLIST via a checkbox.
      $Checkbox1 = GUICtrlCreateCheckbox("Checkbox1", 160, 96, 105, 25) Case $Checkbox1 If GUICtrlRead($Checkbox1) = $GUI_CHECKED Then Call ("Read_only") Else Call ("Edit") Endif Func Read_only() GUICtrlSetStyle ($Combobox1,something to change it to $CBS_DROPDOWNLIST) EndFunc Func Edit() GUICtrlSetStyle ($Combobox1,something to change it to $CBS_DROPDOW) EndFunc I tried using $CBS_DROPDOWN and $CBS_DROPDOWNLIST but it simply does nothing.
      I also tried $ES_READONLY which kinda worked, it changed the combobox so i couldn't change the input via dropdown anymore but i could still type in it via keyboard (which i dont want).
      It's essential that its Editable but once the checkbox is checked the combobox becomes readonly so you can't type in it anymore but you can still choose between the items in the list. (It does not have to save your current input if you write something in it and then check the Checkbox it would be best if it would simpy jump back to the first item in the list)
      Im sure this is a simple thing but after long google search im tired of my stupidity can someone give me a push in the right direction?
    • By Eminence
      Hello,
      Using SQLite, I was trying to select a specific column with having duplicate entries removed by using the DISTINCT function. 
      Local $sQuery = "SELECT DISTINCT supervisor, COUNT(DISTINCT employee_name) FROM data_db What the above snippet does is just list only one supervisor but with the total count of unique employee names in the whole database when it should be listing out all supervisor entries from the database and list the total count of employees per each supervisors. If I remove the COUNT function, it does list out all unique entries of supervisor names from the list. Attached is a screenshot of an example database as well. 
      Any help will be much appreciated. Thank you!

×
×
  • Create New...