Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I have a large OSM XML file (around 8 GB) that I was hoping to process in FME. I have a workbench that I previously used to process smaller files (500 MB) into feature type-specific Shapefiles (e.g. waterway_line, waterway_polygon etc.).

In this workbench I simply added the OSM reader, left everything as default, and FME helpfully grouped the features into useful categories - sending each to the appropriate Shapefile.

Unfortunately my 32-bit OS cannot manage the larger file. Because the entire XML file has to be read at the start of the transformation the memory just builds and builds until the process limit is reached before FME has even read all the incoming features.

One of the things FME does by default is build polygons from lines (raw OSM data doesn't recognise polygons), so if I break the area up into smaller files I could end up with polygons that are artificially split (e.g. landuse areas) and end up being labelled twice and having spurious borders through the middle.

One option is to load my OSM file into PostGIS, and FME can read data from PostGIS (I'm assuming this will help overcome the memory limit as the database can be read row-by-row) but then FMW won't know to group features into those useful categories and which Shapefiles they should end up in.

Essentially, it looks like I simply can't use FME to process a big OSM file to Shapefiles. What I really want to know is:

  1. Is there any way I can convince FME to do what I want?
  2. Are there other tools that can create similar convenient Shapefile groupings of feature types?
  3. EDIT How do Cloudmade and GeoFabrik create their huge Shapefiles? Are they using a different tool I don't know about?

Any input much appreciated.

(note: I don't have access to a 64-bit FME machine, and the pre-processed Shapefiles from Cloudmade / GeoFabrik are insufficient for my requirements)

share|improve this question
    
What Version of FME are you using (including Build Number)? –  Mapperz Apr 30 '12 at 14:47
    
@Mapperz FME Professional 2011, Build 6508 –  tomfumb Apr 30 '12 at 16:28
add comment

3 Answers 3

up vote 4 down vote accepted

Updated: OSM Reader for FME 2013 (Beta)

=========================== BUILD 13082  20120417 ===========================
===========================================================================
OSM reader: Updated to support reading very large datasets, for example
~764 million features on a European OSM dataset (PR#37345)

ftp://ftp.safe.com/fme/beta/whatsnew.txt

FME 2013 Beta

http://www.safe.com/support/support-resources/fme-downloads/beta/

(Only use in a development/testing environment)

share|improve this answer
1  
I downloaded the beta and so far it is doing much better - FME has already read-in 6,000,000 features and memory usage is hovering around 130 MB. The product I'm testing this for is scheduled to go live around October 2012 so hopefully I will be able to get a stable copy of FME 2013 by then. –  tomfumb Apr 30 '12 at 20:23
    
I'd guess based on this and your comment to my answer that the 2012 reader acted as a grouper and the 2013 isn't (130MB is the normal RAM use I see for FME). Note: FME 2013 won't be released until Jan next year so I'm not sure about "stable copy". –  GIS-Jonathan May 1 '12 at 7:24
    
There are few caveats with Beta Software - but Safe Software do run nightly benchmarks/stress-tests on each build. So for Beta software is it quiet stable though not recommended for production work. But the OSM reader has been vastly improved. Please provide feedback to safe they are very good at taking users experiences and wanting to improve on it. –  Mapperz May 1 '12 at 13:28
    
@Mapperz I'm generally not a shirt-and-tie guy so I'll assume you mean caveats :) FME 2013 dealt with all 40,000,000 features in my dataset and memory peaked around 600 MB. I watched my C drive space slowly decrease but in the end I think it only used about 1 GB. –  tomfumb May 1 '12 at 15:37
    
@GIS-Jonathan a co-worker told me the same thing, but also suggested Safe could be persuaded to include the changes in a future 2012 Service Pack if I made a good enough argument. I'm working on it... –  tomfumb May 1 '12 at 15:38
add comment

If you load the data into PostGIS, is there a column created for this category you mention? If so, you could use that attribute to specify the output shapefile name by setting the dynamic writer properties to use that attribute as the feature type name.

Another possibility is using something like imposm.parser and Python to parse the XML file and generate FME features in a PythonCreator or PythonCaller. Presumably since it is multithreaded it does not read the entire file in at once but element by element. Update: Yes, looking at the parser's source it actually uses cElementTree's iterparse function so it does not read the entire file in at once. They also claim to have parsed the entire planet.osm (250GB uncompressed) so hopefully it will work with your data as well.

share|improve this answer
    
I have to check later as I don't have access to the db right now, but I assume not - I think the db schema very closely minics the XML format which doesn't provide any grouping. When classifying features in the way I described FME gives the option of default, very broad, or very narrow categories. I think this means that somewhere buried in FME there are 3 definitions of FME-specific OSM categories and a set of rules which group features into them. If I could find these rules I might be able to use them against the db schema, but I'm working on a number of assumptions here. –  tomfumb Apr 29 '12 at 19:22
    
Are these the files you're talking about? <FME InstallDir>\xml\osm\AreaFeatureLists\osm_specific_areas.txt and <FME InstallDir>\xml\osm\AreaFeatureLists\osm_broad_areas.txt. –  blah238 Apr 29 '12 at 19:41
1  
Or perhaps these? {FME Directory”\xml\osm\schemaMap\osm_broad_schema.fmi and osm_specific_schema.fmi. –  blah238 Apr 29 '12 at 19:45
    
Yes! the osm_broad_schema.fmi file is perfect for explaining how FME maps the XML to feature types. Unfortunately this just made me realise how 'bluntly' this is actually done, e.g. if the field 'highway' is not "" then it's a highway. Unfortunately the OSM data seems a bit too messy for this approach. –  tomfumb Oct 11 '12 at 18:07
add comment

I hit a lot of walls with FME too (including the RAM one despite having 36GB to play with) but typically they can be worked around. I'm not sure if there's anything specific about the OSM XML reader that requires grouping so I'll assume not.

The first thing I'd suggest is read my reply here - Debugging FMW memory usage near Group-based Transformers - which basically says: don't use groupers. Except you want to aggregate and group.

To do it within FME my suggestion would be as follows:

  • a) Create a workspace that has the reader and then writes the data directory to a database of your choice. You may want to use a FeatureTypeFilter in there to split into seperate tables. You can add attribute keepers/removers too, but whatever you do, don't add a Grouping transformer. Because there are no groupers this won't require any RAM as everything will just pass straight through.

  • b) Once the data is in your database you can then process it with a second workspace. Ideally you can do this on a table-by-table basis and so grouping should be less of an issue. You can also do it with "Start_feature" and "Max_features_to_read" to also limit your RAM use. If you can figure out what the grouping items are, you'll be able to write some SQL load only those features using SQLCreator.

Also - at least on my machine FME never really "runs out" of RAM. What happens is when it gets close to using all the RAM, FME simply starts dumping stuff to the hard-drive in a temporary location (system variable: FME_TEMP). I'd guess a x86 build would do this. It's slower of course but.


Convertor http://help.openstreetmap.org/questions/1963/conversion-from-osm-to-shapefiles - these folks seem to think there was no convertor 18 months ago.

There's also an information thread/post on their mailing lists here: http://lists.openstreetmap.org/pipermail/dev/2008-March/009315.html

Obviously its technically possible, so maybe email the Cloudmade / GeoFabrik folks and ask how they're doing it.

share|improve this answer
    
I tried reading OSM and writing straight out into another format using FME and turning off all the options that would use grouping under the covers (e.g. build areas from lines). It still ran out of memory - it all happens during the XML parsing stage. I have plenty of temp space on my machine and the disk is not being filled, so FME is definitely giving up when a memory limit is reached and is not off-loading memory requirements to disk. If I was able to get the raw data into a database I think I would still have problems as a single category (highway) makes up so much of the data –  tomfumb Apr 30 '12 at 20:30
    
Weird, I've not seen that behaviour myself though I've still to try loading OSM. At least the 2013 beta seems to fix it. –  GIS-Jonathan May 1 '12 at 7:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.