I have pyproj installed in my conda environment on my local machine (MacBook Pro). Everything works properly. I am able to use pyproj within the pyspark installation on my local machine. However, I mainly work on a Linux machine I SSH to and I have shipped my conda environment from that Linux machine to an HDFS cluster so that I can use pyproj (and pyspark) on a large dataset.
When I try to run the same rdd.map() code on that HDFS cluster in which some pyproj stuff is inside the .map(), I get the following error:
ExtractionError: (ExtractionError("Can't extract file(s) to egg cache The following error occurred while trying to extract file(s) to the Python egg cache: [Errno 13] Permission denied: '/home/.cache' The Python egg cache directory is currently set to: /home/.cache/Python-Eggs Perhaps your account does not have write access to this directory? You can change the cache directory by setting the PYTHON_EGG_CACHE environment variable to point to an accessible directory. ",).
The '/home/.cache/Python-Eggs' is a directory on the Linux machine I SSH to, not in HDFS, nor my local machine (MacBook Pro).
How can I debug this?