Is it just servers being hosted by your friends that are giving you the issue? Are they hosting it on their own computer?
The reason I ask is I host a server on mine and while people can connect to it fine from the server browser, people cannot join me via the steam friend list, or if I invite them because for some reason, instead of using my public IP, it uses my local IP of 10.0.0.2. Seems to be a steam issue, as it does it on other games that use steamworks as well, such as Ark.
If this is the case, the only workaround I have found is to use the server browser and join that way. But then again, I haven't really looked into the issue of why steam uses my local IP vs my public IP.