Large Open Street Map OSM file to Shapefile. Is there any hope for 32-bit FME?

Question

I have a large OSM XML file (around 8 GB) that I was hoping to process in FME. I have a workbench that I previously used to process smaller files (500 MB) into feature type-specific Shapefiles (e.g. waterway_line, waterway_polygon etc.).

In this workbench I simply added the OSM reader, left everything as default, and FME helpfully grouped the features into useful categories - sending each to the appropriate Shapefile.

Unfortunately my 32-bit OS cannot manage the larger file. Because the entire XML file has to be read at the start of the transformation the memory just builds and builds until the process limit is reached before FME has even read all the incoming features.

One of the things FME does by default is build polygons from lines (raw OSM data doesn't recognise polygons), so if I break the area up into smaller files I could end up with polygons that are artificially split (e.g. landuse areas) and end up being labelled twice and having spurious borders through the middle.

One option is to load my OSM file into PostGIS, and FME can read data from PostGIS (I'm assuming this will help overcome the memory limit as the database can be read row-by-row) but then FMW won't know to group features into those useful categories and which Shapefiles they should end up in.

Essentially, it looks like I simply can't use FME to process a big OSM file to Shapefiles. What I really want to know is:

1. Is there any way I can convince FME to do what I want?
2. Are there other tools that can create similar convenient Shapefile groupings of feature types?
3. EDIT How do Cloudmade and GeoFabrik create their huge Shapefiles? Are they using a different tool I don't know about?

Any input much appreciated.

(note: I don't have access to a 64-bit FME machine, and the pre-processed Shapefiles from Cloudmade / GeoFabrik are insufficient for my requirements)

Answer 1

Updated: OSM Reader for FME 2013 (Beta)

=========================== BUILD 13082  20120417 ===========================
===========================================================================
OSM reader: Updated to support reading very large datasets, for example
~764 million features on a European OSM dataset (PR#37345)

ftp://ftp.safe.com/fme/beta/whatsnew.txt

FME 2013 Beta

http://www.safe.com/support/support-resources/fme-downloads/beta/

(Only use in a development/testing environment)

Answer 2

If you load the data into PostGIS, is there a column created for this category you mention? If so, you could use that attribute to specify the output shapefile name by setting the dynamic writer properties to use that attribute as the feature type name.

Another possibility is using something like imposm.parser and Python to parse the XML file and generate FME features in a PythonCreator or PythonCaller. Presumably since it is multithreaded it does not read the entire file in at once but element by element. Update: Yes, looking at the parser's source it actually uses cElementTree's iterparse function so it does not read the entire file in at once. They also claim to have parsed the entire planet.osm (250GB uncompressed) so hopefully it will work with your data as well.

Answer 3

I hit a lot of walls with FME too (including the RAM one despite having 36GB to play with) but typically they can be worked around. I'm not sure if there's anything specific about the OSM XML reader that requires grouping so I'll assume not.

The first thing I'd suggest is read my reply here - Debugging FMW memory usage near Group-based Transformers - which basically says: don't use groupers. Except you want to aggregate and group.

To do it within FME my suggestion would be as follows:

• a) Create a workspace that has the reader and then writes the data directory to a database of your choice. You may want to use a FeatureTypeFilter in there to split into seperate tables. You can add attribute keepers/removers too, but whatever you do, don't add a Grouping transformer. Because there are no groupers this won't require any RAM as everything will just pass straight through.

• b) Once the data is in your database you can then process it with a second workspace. Ideally you can do this on a table-by-table basis and so grouping should be less of an issue. You can also do it with "Start_feature" and "Max_features_to_read" to also limit your RAM use. If you can figure out what the grouping items are, you'll be able to write some SQL load only those features using SQLCreator.

Also - at least on my machine FME never really "runs out" of RAM. What happens is when it gets close to using all the RAM, FME simply starts dumping stuff to the hard-drive in a temporary location (system variable: FME_TEMP). I'd guess a x86 build would do this. It's slower of course but.

---

Convertor http://help.openstreetmap.org/questions/1963/conversion-from-osm-to-shapefiles - these folks seem to think there was no convertor 18 months ago.

There's also an information thread/post on their mailing lists here: http://lists.openstreetmap.org/pipermail/dev/2008-March/009315.html

Obviously its technically possible, so maybe email the Cloudmade / GeoFabrik folks and ask how they're doing it.