If it's just the 5 of you most of the time, the 23" is going to do you just fine (I own one and can easily cook for 6 adults on it). But, if you like to entertain and need to cook for a lot of folks, then consider the 32".
As others have noted, the other direction is to go with 2 KKs of different sizes - say a 23" and a 19", so you could do different cooks at the same time - example ribs on the 23" and burgers/dogs on the 19" (or your large BGE, assuming you plan to keep it.)
Bottomline - you can't go wrong in upgrading to the KK over the BGE, as you'll quickly learn after you start cooking on it. I've cooked on both and the KK is the clear winner in versatility and quality.
Talk to Dennis. He's a pleasure to talk to and will help you figure out what's going to be best option for you - no sales pressure at all!!
Keep us posted on what you decide.