The gateway cannot associate between the key exchange traffic and the encrypted packets.
The VPN server expects that all of the traffic will come from the same source so he could associate the keys for each packet.