The Nazis had plans to invade -- Operation Tannenbaum -- it just wasn't a priority. You're fighting the Soviet Juggernaut in the East, and the British Commonwealth + the US in the West. Why piss away tons of troops to get a small mountain country with no strategic resources that's effectively bottled up?

But knock out the UK or the Soviets, lower the pressure, and there is little doubt the Nazis would have made a go of it eventually.