IDL is a fantastic stand-alone programming language (you do not need ENVI). I particularity like it for very fast matrix processing on large arrays. @Aaron makes IDL sound much less flexible then it really is. The majority of IDL development came out of the Physics and Astronomy communities. There is robust support for mathematical and statistical programming. If bundled with ENVI, you have all the library calls (functions) available in ENVI including support for spatial vector objects. There are also a large number of functions and models developed by the user community. One advantage to learning IDL is that it will make you marketable in "analytic" remote sensing shops.
Also, do not forget that ERDAS has a scripting language (EML) that is quite good and easy to learn. EML is the backbone of the graphic modeler and gmd's are just packaged EML scripts that sit under the graphic modeler interface. The advantage of using EML directly is that you can use for/while loops and have access to more ERDAS functionality in a scripting language.
MATLAB is also very good for matrix processing and there are open source versions (e.g., Octave) that have exactly the same syntax with similar benchmarks. This is a highly flexible language with considerable power. It is one of the preferred language for applied mathematics and engineering.
The Python alternatives NumPy and SciPy are flexible but not as optimized as IDL and MATLAB. As such, you need to deal with addressing space and speed when working with large arrays. One huge advantage of Python are the additional libraries for performing a variety of analytical tasks. There are packages for remote sensing, nonparametric statistics, bindings to spatial classes (e.g. GDAL, LibLAS) to name just some of the added functionality available through packages.
This brings us to R. I am primarily a spatial statistician so, this is my everyday language. The number of available packages is staggering which, in turn, provides access to cutting edge cross-discipline statistical methodologies. However, I have to say that it is very cumbersome when dealing with large data problems. The spatial classes are getting much better and because of the raster package providing the ability to hold large data out of memory I am now able to implement some fairly complex statistical models utilizing large raster arrays. But still, R is slow when dealing with large memory problems. The BigMatrix package allows for writing and processing huge arrays from disk but the coding overhead is not insignificant There are also bindings to GDAL and GIS software (e.g., GRASS, SAGA) that allow spatial object processing to occur outside of R in a GIS specific software, which is how I interact with GIS software these days. This allows me to leverage functionality across multiple software without leaving R.
So, now that the software cheerleading is out of the way, my recommendation is "yes to all of the above options". Programming is a skill that, once learned, is easily applied to other languages. There are striking similarities between C++, R, IDL and Python. Aside from some coding idiocentricities, what one must learn are the available functions to implement a given model/task. Once this is done it is just a matter of syntax that implements common coding structures.
Sometimes there are things that just work better in a different software or language. I occasionally write code in FORTRAN or C++ because it is just the best choice for a given task. It is a matter of adaptability. You may want to start with Python because, as a scripting language, it can be applied to numerous tasks it also provides availability of packages for specialized analysis, has a number of free online resources and is somewhat easy to learn.