The reason for the listbox resizing is that by default, the height of a listbox is locked to a multiple of the height of a single item. You need to add the $LBS_NOINTEGRALHEIGHT style to make it behave otherwise.
As for your resizing, try this:
GUICtrlSetResizing($chatrcv,$GUI_DOCKBORDERS)
GUICtrlSetResizing($nicklist,$GUI_DOCKWIDTH+$GUI_DOCKTOP+$GUI_DOCKBOTTOM)
GUICtrlSetResizing($chinputbox,$GUI_DOCKBOTTOM+$GUI_DOCKHEIGHT+$GUI_DOCKLEFT+$GUI_DOCKRIGHT)
GUICtrlSetResizing($chsendbtn,$GUI_DOCKSIZE+$GUI_DOCKRIGHT+$GUI_DOCKBOTTOM)
While testing this, I ran into something strange: The controls seem to resize in discrete steps, a few pixels at a time, instead of continuously. Maybe a bug, maybe not. Anyone observed this before?