2

I wrote some code that basically loops over all features in a shapefile with Fiona in Python, doing some things, calculating some things, etc.

I found that it would exit the for loop prematurely, depending on the data set I fed it. I found that it would work fine if I removed some code that (to me) has nothing to do with the loop we are in. This is my first attempt at GIS with Python, so I might be doing some newbie mistage with data structures I guess?

Below is a minimal example using Natural Earth data which you can grab at http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_countries.zip

import fiona
from shapely.geometry import shape, Point

def find_containing_area(point, collection):
    for record in collection:
        geometry = shape(record['geometry'])
        if point.within(geometry):
            return record
    return None

with fiona.open('/tmp/ne_110m_admin_0_countries.shp', 'r') as source:
    for admin_record in source:
        properties = admin_record['properties']
        print(properties['name'])

        # in my original code, i created a point related to the admin_record
        point = Point(0,0)

        # if the following line exists, the loop will stop after the first
        # admin_record in this example. In my original code, it stopped
        # after just few records, not the first one though.
        containing_area = find_containing_area(point, source)

If I run it, it prints "Afghanistan" instead of the whole list of ~177 names. If I comment out the last line, it prints them all as intended.

My only guess is that someone an iterator is passed along with the collection? The only hint in the documentation that I found is "Fiona’s Collection is like a Python file, but is iterable for records rather than lines." which does help stupid-me not very much. http://toblerity.org/fiona/manual.html#data-model

edit: I have tried to copy the collection and pass the copy to the function instead, got the same bevahiour:

newsource = source[:]
containing_area = find_containing_area(point, newsource)
5
  • Could you give a little more detail on how/why you are creating the point? It seems like you are creating a point based on a country boundary, and then seeing what country boundary that point falls within. If that's not the case, you may be able to reorganize the loops.
    – mr.adam
    Aug 13, 2015 at 14:22
  • In this example I create a point anywhere, it does not matter where it is and if find_containing_area() returns a valid record/feature or not. The problem really is that the (main) loop is not iterating through all records as it should. Aug 13, 2015 at 14:29
  • I guess my question is why do you need the main loop? I just wanted to double-check that the point is created directly from the shapefile. If it's not, just give a description of what the main loop is for.
    – mr.adam
    Aug 13, 2015 at 14:35
  • As I said, the point does not matter here. In my actual code it is generated based on the admin_record's geometry but it has no relation to my problem. In my actual code the main loop is used to do calculations (involving the point and other things) per admin_record. Aug 13, 2015 at 14:40
  • ok cool. sometimes people do some extra looping that causes problems and isn't actually necessary.
    – mr.adam
    Aug 13, 2015 at 16:32

2 Answers 2

3

According to the documentation:

Seeking the beginning of the file is not supported. You must reopen the collection to get back to the beginning.

Open the file again for the second loop. I changed your last line to the following and got the loop to run all the way through.

with fiona.open('ne_110m_admin_0_countries.shp', 'r') as source2:
        containing_area = find_containing_area(point, source2)
5
  • Thanks, that is a workaround but a very wasteful solution, especially since in actual usage that would mean reading the file tens or hundreds of times. I would like read it in once and then work in memory. Aug 13, 2015 at 14:48
  • 2
    I see. What if you read all the records into a list, and then loop through the list?
    – HishivS
    Aug 13, 2015 at 14:58
  • I feel a bit stupid now, that seems like the best idea. I thought fiona.open "kinda" did that. Aug 13, 2015 at 15:01
  • 3
    fiona.open returns an iterator so it will only return one element at a time in memory. The documentation has some examples on storing iterator results in memory, e.g. lists
    – Kersten
    Aug 13, 2015 at 15:32
  • Thank you! I would accept that as answer if you post it as such :) How could I have known that it is an iterator? The docs say it returns a Collection object and that is a " file-like interface to features". Aug 14, 2015 at 8:53
0

Reposting Kersten's comment as an answer so I can accept it as solution:

fiona.open returns an iterator so it will only return one element at a time in memory. The documentation has some examples on storing iterator results in memory, e.g. lists

My solution was to create a list from the source:

newsource = list(source)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.