I am trying to use a user-defined function in pyspark in order to check whether the points in CANmep_GDF lie within the polygons in gda_GDF.

from geopandas import GeoDataFrame 

def check_exist(geom):   
    gda_GDF = GeoDataFrame.from_file("/dbfs/FileStore/tables/gda_000a11a_e.shp")  
    exist = gda_GDF.contains(geom)   
    return exist

check_exist_udf = udf(check_exist,BooleanType())

CANmep_GDF_exist = CANmep_GDF.withColumn('EXIST',check_exist_udf('geometry'))

When I run the code above, I get the following error:

AttributeError: 'GeoDataFrame' object has no attribute 'withColumn'

I think geopandas does not support .withColumn(). What would be an alternate way to write the same?

  • 1
    Geopandas doesn't have a function withColumn() for the GeoDataFrame object - it looks like you might be mixing modules. You can find methods that work on GeoDataFrames in the geopandas reference: geopandas.org/reference.html – jyingling Apr 5 '19 at 18:31
  • .withColumn is a method of pyspark.DataFrame class. It's different from pandas.DataFrame and geopandas.GeoDataFrame classes. – Kadir Şahbaz Apr 6 '19 at 10:49
  • what do expect withColumn to do? – Paul H Apr 6 '19 at 13:47

To add a new column into a (geo)pandas.(Geo)DataFrame, you should use the .assign method. The column names are passed as keyword arguments, and the values can be scalars, sequences, or callable functions and methods that accept the dataframe in its current state as the first argument.

So your code becomes:

shpfile = "/dbfs/FileStore/tables/gda_000a11a_e.shp"
gda_GDF = (
        .assign(exists=lambda gdf: gdf.contains(other_geom)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.