Technically you could link one python install to libraries from other installs, but it's quite an ugly thing to do, and prone to errors. And, more than that, you don't have to.
Libraries are located at /lib/site-packages, each python install will have this folder, and in it all its external libraries. You can download any library for any install. What the OSGEO4W, ArcGIS, Anaconda and such do, is install for you a python version and a bunch of libraries that are part of its package. It's just a helping hand, but you could also download a standalone python and keep adding libraries manually. Or add them to existing installs.
For installing Arcpy, check these instructions: https://pythongisandstuff.wordpress.com/2015/09/14/enable-python-installations-that-were-not-installed-by-arcgis-to-access-arcpy-functionality/
Take note that ArcGIS Desktop is a 32-bit software, and will not work with 64-bit python. Also, ArcGIS 10.4 requires python 2.7, no other release will suffice. Your Anaconda python must observe both these restrictions if you wish for the arcpy library to work with it.
Similarly you can do so with all the different libraries that make up the OSGEO suite. They'll also need to observe correct python specifics. As an example, here you can get the GDAL/OGR python bindings: http://www.gisinternals.com/release.php
You'll notice that, aside from matching the python release and version, it must also match the release and version of your GDAL install, and even the version of Visual C++ used to compile your python install.
It is a bit troublesome, but it's doable. The standard repository libraries like numpy can be more easily downloaded with pip.