gcc has the ability to target different architectures (look up the -march and -mtune flags for example). Linux distributions are typically set to be compatible with a pretty wide range of devices, so they often don't take advantage of recent instructions.
Compiling a big program can be a bit of a pain, though, so it is probably only worthwhile if you have a program that you use very frequently. Also compilers aren't magic, the bottleneck in the program you want to run could be various things: CPU stuff, memory bandwidth, weird memory access patterns, disk access, network access, etc. The compiler mostly just helps with the first one.
Also, note that some libraries, like Intel's MKL, are able to check what processor you are using and just dispatch the appropriate code (your mileage may vary, they sometimes don't keep up with changes in AMD processors, causing great annoyance).
Compiling a big program can be a bit of a pain, though, so it is probably only worthwhile if you have a program that you use very frequently. Also compilers aren't magic, the bottleneck in the program you want to run could be various things: CPU stuff, memory bandwidth, weird memory access patterns, disk access, network access, etc. The compiler mostly just helps with the first one.
Also, note that some libraries, like Intel's MKL, are able to check what processor you are using and just dispatch the appropriate code (your mileage may vary, they sometimes don't keep up with changes in AMD processors, causing great annoyance).