The servers with "1" are placeholders and can be misleading since they aren't really "players" The "doorman" server is what you are talking about and I don't quite understand why it says there are "10" players no matter when you look, and then you go in there and there are either none or like 1 or 2.
If you go on a server and no one is there, just wait a little. People join quickly when they actually find players, like you were doing. Games fill up rather quickly.
Good Luck.
