2

I am trying to build GeoDataFrame row by row, from an empty one. The equivalent using only pandas would be something like this :

df = pandas.DataFrame(columns=['a','b','c','d'])    
df.loc['y'] = pandas.Series({'a':1, 'b':5, 'c':2, 'd':3})

(from [this answer])


My current method

So far, I build a Python list of dict with a specific structure, and then use it to create a GeoFataFrame, here is a complete example :

import geopandas as gpd
from shapely.geometry import  Point

my_dict = {
  '007': {
    'name': 'A',
    'lat': 48.843664, 
    'lon': 2.302672,
    'type': 'small'
  },
  '008': {
    'name': 'B',
    'lat': 50.575813,
    'lon': 7.258148,
    'type': 'medium'
  },
  '010': {
    'name': 'C',
    'lat': 47.058420, 
    'lon': 15.437464,
    'type': 'big'
  }
}

tmp_list = []
for item_key, item_value in my_dict.items() :
  tmp_list.append({
    'geometry' : Point(item_value['lon'], item_value['lat']),
    'id': item_key,
    'name': item_value ['name'],
    'type': item_value ['type']
   })
my_gdf = gpd.GeoDataFrame(tmp_list)
print(my_gdf.head())

Here is the result :

                    geometry   id name    type
0   POINT (2.30267 48.84366)  007    A   small
1   POINT (7.25815 50.57581)  008    B  medium
2  POINT (15.43746 47.05842)  010    C     big

What I am looking for

I would like to create an empty GeoDataFrame (my_gdf = gpd.GeoDataFrame()), and then fill it directly in the for loop, without using the temporary list after the loop (my_gdf = gpd.GeoDataFrame(tmp_list))

I think that a row by row building would, in this case, have better performance. It would also allow me to use the id key from my_dict as the GeoDataFrame index, so that the result would be :

                     geometry  name    type
007   POINT (2.30267 48.84366)    A   small
008   POINT (7.25815 50.57581)    B  medium
010  POINT (15.43746 47.05842)    C     big
  • 1
    In order for us to do tests and help you as best we can, can you illustrate a sample of your input data and the desired result? – Vincent Bré Dec 16 '19 at 11:15
  • Thank you for editing your post. Concerning the index, I think you don't have control over that because it's the very structure of the geodataframe. Why do you want this? – Vincent Bré Dec 16 '19 at 13:08
  • Well, working with dataframes (non-geographical), it is possible using df.loc['y'] =, so I thought there was a way to do the same with geodataframes. Concerning the index, it is about not having duplicated fields (I don't really need the iterative default index), and being able to quickly access the data with the same ids from another data source. – Arkeen Dec 16 '19 at 13:12
1

I don't think that a row by row building would have better performance. I've tested.

Result:

import geopandas as gpd
import pandas as pd
from shapely.geometry import  Point

d = {'007': {'name': 'A', 'lat': 48.843664, 'lon': 2.302672, 'type': 'small' },
     '008': {'name': 'B', 'lat': 50.575813, 'lon': 7.258148, 'type': 'medium'},
     '010': {'name': 'C', 'lat': 47.058420, 'lon': 15.437464,'type': 'big'}}

## IN THE ABOVE CASE. Time: ~1 ms (milisecond)
tmp_list = []
for item_key, item_value in d.items() :
    tmp_list.append({
      'geometry' : Point(item_value['lon'], item_value['lat']),
      'id': item_key,
      'name': item_value ['name'],
      'type': item_value ['type']
     })
gdf = gpd.GeoDataFrame(tmp_list)
##


## SOLUTION 1. Time: ~2.3 ms, @gene's answer.
df = pd.DataFrame.from_dict(d, orient='index')
df["geometry"] = df.apply (lambda row: Point(row.lon,row.lat), axis=1)
gdf = gpd.GeoDataFrame(df, geometry=df.geometry)
##


## SOLUTION 2. Time: ~2.5 ms
gdf = gpd.GeoDataFrame()    
gdf["id"]   = [k for k in d.keys()]
gdf["name"] = [d[k]["name"] for k in d.keys()]
gdf["type"] = [d[k]["type"] for k in d.keys()]
gdf["geometry"]  = [Point(d[k]["lon"], d[k]["lat"]) for k in d.keys()]    
gdf.set_index('id', inplace=True)
##


## SOLUTION 3. Time: ~9.5 ms
gdf = gpd.GeoDataFrame(columns=["name", "type", "geometry"])
for k, v in d.items():
    gdf.loc[k] = (v["name"], v["type"], Point(v["lon"], v["lat"]))
##

print(gdf)

# OUTPUT for last solution
#     name    type                   geometry
# 007    A   small   POINT (2.30267 48.84366)
# 008    B  medium   POINT (7.25815 50.57581)
# 010    C     big  POINT (15.43746 47.05842)
1

You don't need to build the GeoDataFrame row by row here, look at pandas.DataFrame.from_dict¶

import pandas as pd
df = pd.DataFrame.from_dict(my_dict,orient='index')
print(df)
     name     lat        lon    type
007    A  48.843664   2.302672   small
008    B  50.575813   7.258148  medium
010    C  47.058420  15.437464     big
from shapely.geometry import Point
df["geometry"] = df.apply (lambda row: Point(row.lon,row.lat), axis=1)

Convert to a GeoDataFrame

gdf = gpd.GeoDataFrame(df, geometry=df.geometry)
print(gdf)
     name    lat        lon      type            geometry
007    A  48.843664   2.302672   small  POINT (2.302672 48.843664)
008    B  50.575813   7.258148  medium  POINT (7.258148 50.575813)
010    C  47.058420  15.437464     big  POINT (15.437464 47.05842)

Or directly:

gdf = gpd.GeoDataFrame(df, geometry=df.apply(lambda row: Point(row.lon,row.lat), axis=1)

In other words, do not form a new DataFrame for each row. Instead, collect all the data in a list of dicts, and then call df = pd.DataFrame(data) once at the end, outside the loop.
Each call to df.append requires allocating space for a new DataFrame with one extra row, copying all the data from the original DataFrame into the new DataFrame, and then copying data into the new row. All that allocation and copying makes calling df.append in a loop very inefficient. The time cost of copying grows quadratically with the number of rows. Not only is the call-DataFrame-once code easier to write, it's performance will be much better -- the time cost of copying grows linearly with the number of rows. (from How to append rows in a pandas dataframe in a for loop?)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.