Both Michael Markieta and gm70560 are correct. If you are running running large geoprocessing tasks, I would definitely do it via a stand-alone python script, preferably launched from the command line and not an IDE. For this sort of task, the overhead of importing ArcPy is well worth it.
However, a small task, especially one which requires user input, is usually faster and more convenient in ArcMap - unless you have to open ArcMap specifically for the purpose in which case you have the overhead of of starting ArcMap itself plus some downtime to connect to all your databases referenced in the MXD (which if they are over a slow network connection can be tedious).
Likewise, batch processing even small tasks is, in my opinion, better done in a stand-alone Python script than via a toolbox in ArcMap because you take that import hit only once and don't have the ArcMap overhead.
Having done a lot of automated geoprocessing via Python scripts, I would default to running them as stand alone processes because I like the additional speed and flexibility. Most tasks which run faster in the toolbox are usually small enough that I would have knocked them up in ModelBuilder without bothering to write any code.