Pandas requires compilation. If you don't have an appropriate compiler and build dependencies installed, you'll need to install a precompiled version.
Pandas depends on numpy. The Numpy application binary interface (ABI) is not forward compatible which means you can't use a version of Pandas that was compiled against a later version of numpy even though the application programming interface (API) is compatible.
Current versions of Pandas were compiled against numpy 1.8.1, so you can use them in ArcGIS 10.4 which requires numpy 1.9.2. This means you can use the methods described in Gene's answer.
However, ArcGIS 10.2 and 10.3 require numpy 1.7.1. Although the current versions of pandas are compatible with the numpy 1.7+ API, as they were compiled against numpy 1.8.1, they will not work with the numpy 1.7.1 installed by ArcGIS. If you install a newer version that was compiled against numpy 1.8.1, importing pandas will fail with the exception:
ValueError: numpy.dtype has the wrong size, try recompiling.
If you want to install Pandas to use with ArcGIS 10.2 and 10.3, you need to use the older version 0.13 that was compiled against numpy 1.7.1. This version can't be installed with pip. Instead, download the appropriate installer executable and run it manually.
And please note that upgrading numpy is dangerous and you are likely to break your ArcGIS installation if you try (unless you set up a virtualenv).