I have original data as a 700000's rows excel, with geometry data in hexwkb format. I want to plot it and make some spatial operations on it like intersect, distance and buffer, on a jupyter notebook. Half of the geometries are null. Part of the remaining ones have apparently invalid geometries.
I had succeed in plotting the first 100 lines, apparently with fine geometries, but I'm struggling to plot all the data.
This code worked for lines with no geometry problems:
Data example
id (int64) | geom (object)
1 | '0105000020E610000001000...
import pandas as pd
import geopandas as gpd
from shapely import wkb
from shapely.geometry import MultiLineString
df=pd.read_excel(r'trips.xlsx')
trips= df.dropna(subset=['geom'])
# create column with converted geometry
def convert_to_hexwkb(geomString):
geometry = wkb.loads(geomString, hex=True)
return geometry
trips['geometry']=trips['geom'].map(convert_to_hexwkb)
trips.drop('geom', axis=1, inplace=True)
# convert to GeoDataFrame
gtrips=gpd.GeoDataFrame(trips)
print(gtrips['geometry'][0])
>> MULTILINESTRING ((-9.1474504470825 38.744022369385, -9.147529602050801 38.744121551514, -9.14...
# show trips map
gtrips.plot()
>> got a nice map here
But when I run this to the entire data I get the following error given from the shapely -> geos library, during conversion of string to hexwkb:
WKBReadingError: Could not create geometry because of errors while reading input.
IllegalArgumentException: point array must contain 0 or >1 elements
I couldn't find how to apply Geopandas.GeoSeries.is_empty
function to find empty geometries, because it was not a geodataframe at that point.
Same for is_valid
function from shapely library, since the field was not geometry type.
I've tried to delete exceptional rows with this:
for i, item in enumerate(trips['geom']):
try:
trips.at[i,'geometry'] = wkb.loads(item, hex=True)
#wkb.loads(item,hex=True)
except:
trips.drop([i], inplace=True)
And got the errors:
IllegalArgumentException: point array must contain 0 or >1 elements
WKBReadingError: Could not create geometry because of errors while reading input.
another exception occurred: KeyError: '[791] not found in axis' - This related with how the for cycle was constructed probably.
If I run this for debug:
def convert_to_hexwkb(geomString):
try:
geometry = wkb.loads(geomString, hex=True)
return geometry
except:
return np.nan
trips['geometry']=trips['geom'].map(convert_to_hexwkb)
I get these errors, multiplied for several lines:
ParseException: Unexpected EOF parsing WKB
ParseException: Premature end of HEX string
Any clue of what should I do to clean geometries of my dataset?
trips = df.dropna(subset=['geom']).reset_index(drop=True)
and thenfor
loop? – Kadir Şahbaz May 11 '20 at 21:29