I have the exact same issue on Xbox Series X, wired connection and open NAT type. I was looking forward to the patch 1.07, but unfortunately nothing has changed.
Same issue when returning to lobby from race (ranked mode in my case). If one player is unreachable / unavailable, the game gets stuck in "waiting for player"-process. There is no timeout, as written before. You have to quit the game and restart it. No chance of going back to main menu.