1

I have pyproj installed in my conda environment on my local machine (MacBook Pro). Everything works properly. I am able to use pyproj within the pyspark installation on my local machine. However, I mainly work on a Linux machine I SSH to and I have shipped my conda environment from that Linux machine to an HDFS cluster so that I can use pyproj (and pyspark) on a large dataset.

When I try to run the same rdd.map() code on that HDFS cluster in which some pyproj stuff is inside the .map(), I get the following error:

ExtractionError: (ExtractionError("Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg
cache:

  [Errno 13] Permission denied: '/home/.cache'

The Python egg cache directory is currently set to:

  /home/.cache/Python-Eggs

Perhaps your account does not have write access to this directory?  You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
",).

The '/home/.cache/Python-Eggs' is a directory on the Linux machine I SSH to, not in HDFS, nor my local machine (MacBook Pro).

How can I debug this?

3
  • Does your login account have a directory within /home? If so try setting "export PYTHON_EGG_CACHE=/home/<<username>>/.cache/Python-Eggs. You'll need to create this directory if it doesn't exist. Aug 18 '17 at 7:12
  • I will try when I get home. Thanks for the suggestion.
    – mrphysics
    Aug 18 '17 at 22:30
  • Tried the suggestion, but same problem.
    – mrphysics
    Aug 22 '17 at 18:16
1

Try the following configuration:

pyspark/spark-submit 
--conf spark.executorEnv.PYTHON_EGG_CACHE="./.python-eggs/" 
--conf spark.executorEnv.PYTHON_EGG_DIR="./.python-eggs/" 
--conf spark.driverEnv.PYTHON_EGG_CACHE="./.python-eggs/" 
--conf spark.driverEnv.PYTHON_EGG_DIR="./.python-eggs/"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.