I also wouldn't be surprised. Results in transfer learning from synthetic to real world vision tasks suggest to me that if you could train this system with (for example) GTA-V as the environment, it may work reasonably well in the real world.