Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

SimpleGeo has recently made their "Places" data set freely available which is GREAT, because now we can use it for our own purposes, even offline. So I went about downloading the 2GB dataset, which contains zipped country files in GeoJSON format.

Here is where my problem begins... when I try loading a country file (*.geojson) directly into QGIS or even converting it to shapefile using ogr2ogr, I end up with just 1 point feature.

Clearly there is more data available in these files that I am not able to "see" or retrieve, so what am I doing wrong? Can someone provide me any suggestions on how to turn these files into useable GIS features?

Updated Question: I really want to convert the geojson file into a shapefile (or similar standard GIS format) for use in a desktop GIS software.

Snippet from SimpleGeo blog on Aug 1, 2011:

"SimpleGeo’s CC0 Places data set is now available for download at no cost. If you’d like to get your hands on 21M+ POIs that cover 63 countries, we’re ready to hand that over to you in one file. The file is about 2GB in .ZIP format, and remember, with the CC0 license, this data becomes yours – free and clear – to do whatever you want. We hope you do awesome things with it! You can download it now by clicking here."

Update: January 12th, 2012: After SimpleGeo was acquired by Urban Airship in Oct 2011, Urban Aiship said “wind down the availability of the current versions of [SimpleGeo's] Places, Context, and Storage over the next few months.”...The target date for pulling the plug officially will be March 31, 2012. Link

share|improve this question
3  
+1. Listened to Matt Galligan of SimpleGeo sometime back saying locational data is a matter of fact and should be freely available. It is context which should be sold. Good to see 'they' meant it. Also eager to leverage this data. –  Erick Aug 19 '11 at 8:48

3 Answers 3

up vote 9 down vote accepted

Try wrapping the features in the following:

{"type":"FeatureCollection","features":[ 

****ALL THE DATA HERE -- COMMA SEPARATED****

]}

and separating each feature with a comma.

For example, here's the first two features from the IE dataset:

{"geometry": {"type": "Point", "coordinates": [-6.422587, 53.293363]}, "type": "Feature", "id": "SG_41u80gC4971D4Gc0Fv1p8q_53.293363_-6.422587@1308163237", "properties": {"website": "http://www.buongiorno.com", "city": "Dublin", "name": "Buongiorno", "tags": ["mobile", "vas", "community", "social-networking", "connected-devices", "android", "tablets", "smartphones"], "country": "IE", "classifiers": [{"category": "Professional", "type": "Services", "subcategory": "Computer Services"}], "href": "http://api.simplegeo.com/1.0/features/SG_41u80gC4971D4Gc0Fv1p8q_53.293363_-6.422587@1308163237.json", "address": "Lake Drive City West Digital Park", "owner": "simplegeo", "postcode": "3050"}}
{"geometry": {"type": "Point", "coordinates": [-6.250848, 53.339347]}, "type": "Feature", "id": "SG_1dnWbWIg8hX3VyfUKIDRz9_53.339347_-6.250848@1308163237", "properties": {"website": "http://www.simchronise.com", "city": "Dublin", "name": "SIMchronise", "tags": ["mobile-solutions", "mobile-data", "data-synchronisation", "mobile-security", "backup", "restore", "mobile-contacts", "web-2.0-addressbook", "phonebackup"], "country": "IE", "classifiers": [{"category": "Professional", "type": "Services", "subcategory": "Computer Services"}], "href": "http://api.simplegeo.com/1.0/features/SG_1dnWbWIg8hX3VyfUKIDRz9_53.339347_-6.250848@1308163237.json", "address": "78 Merrion Square", "owner": "simplegeo", "postcode": "Dublin2"}}

... and here they are as a GeoJSON FeatureCollection:

{"type":"FeatureCollection","features":[ 
{"geometry": {"type": "Point", "coordinates": [-6.422587, 53.293363]}, "type": "Feature", "id": "SG_41u80gC4971D4Gc0Fv1p8q_53.293363_-6.422587@1308163237", "properties": {"website": "http://www.buongiorno.com", "city": "Dublin", "name": "Buongiorno", "tags": ["mobile", "vas", "community", "social-networking", "connected-devices", "android", "tablets", "smartphones"], "country": "IE", "classifiers": [{"category": "Professional", "type": "Services", "subcategory": "Computer Services"}], "href": "http://api.simplegeo.com/1.0/features/SG_41u80gC4971D4Gc0Fv1p8q_53.293363_-6.422587@1308163237.json", "address": "Lake Drive City West Digital Park", "owner": "simplegeo", "postcode": "3050"}},
{"geometry": {"type": "Point", "coordinates": [-6.250848, 53.339347]}, "type": "Feature", "id": "SG_1dnWbWIg8hX3VyfUKIDRz9_53.339347_-6.250848@1308163237", "properties": {"website": "http://www.simchronise.com", "city": "Dublin", "name": "SIMchronise", "tags": ["mobile-solutions", "mobile-data", "data-synchronisation", "mobile-security", "backup", "restore", "mobile-contacts", "web-2.0-addressbook", "phonebackup"], "country": "IE", "classifiers": [{"category": "Professional", "type": "Services", "subcategory": "Computer Services"}], "href": "http://api.simplegeo.com/1.0/features/SG_1dnWbWIg8hX3VyfUKIDRz9_53.339347_-6.250848@1308163237.json", "address": "78 Merrion Square", "owner": "simplegeo", "postcode": "Dublin2"}}
]}
share|improve this answer
    
After your posting I realized that I really didn't make my question clear so I edited it. I would really like to convert the data to shapefile or other standard GIS format for use in a desktop GIS. Sorry for the poorly worded question but thanks for the answer @fod. –  RyanDalton Aug 19 '11 at 14:24
1  
fod's answer will get you half-way there. You will need to pre-process all of these dumps to wrap the features in a FeatureCollection as fod suggested, and separate with commas. Easy way to do it is to replace all newlines with a , + newline (i.e. \n with ,\n), and then enclose in FeatureCollection, either through a script, or manually with a text editor. After you've fixed all the files to be proper GeoJSON, then you can use ogr2ogr v1.8.0 or later to convert GeoJSON to shapefile. –  Sasa Ivetic Aug 19 '11 at 19:43
    
Excellent! I didn't realize this is what it would take to make them "readable". Using the method described above, I was able to make the file directly readable via QGIS. Thanks a bunch! I used the Mexico dataset as a good sample, because it was only 9k file size. Now to tackle "search & replace" for the US file @ 7.4 GB. –  RyanDalton Aug 19 '11 at 22:32
    
Apologies - I didn't have time to follow up sooner but FYI you can open GeoJSON files directly in QGIS - no need to convert to shapefile. Then, if you really want a shapefile, you can export the layer in that format (Layer->Save As...). –  fod Aug 20 '11 at 18:41
    
The first thing I tried was opening up the files in QGIS directly. The problem was that all of the data was "wrapped up" in one record, and therefore one point. I needed to follow the instructions you left above in order for QGIS to recognize each individual feature in the dataset. –  RyanDalton Aug 21 '11 at 18:49

You could try FME to do this. It reads GeoJSON and writes Shape and - more importantly - has a bunch of "transformers" for handling JSON (JSONExploder, JSONFlattener, JSONExtractor). So I think it would help to get the content out into a usable format.

It also has a free trial available!

NB: I do work for Safe Software, makers of FME.

share|improve this answer

Thank you very much to @fod & @Sasa Ivetic for putting me on the right track! I had to put this project down for a while but finally got back to it today and wanted to pass along my results to help out anyone else who may have struggled with this.

Using the information they provided in @fod's answer, I wrote this Python script to convert the dump files in "proper" geojson format. This py script will convert the data into format readable by ogr2ogr, then you can convert it to whatever format you would like.

infile = sys.argv[1]
outfile = sys.argv[2]


## Prepare the input/output for read/write of files
fin = open(infile, 'r')
fout = open(outfile, 'w')


##Prepend the new file with approriate header info
fout.write('{"type":"FeatureCollection","features":[' + '\n')


## Read each line from the input and write to the output
for line in fin:
    ##write each line in the table, and replace carriage returns with "," + carriage return
    fout.write(line.replace("\n",",\n"))

##Append the new file with approriate footer info
fout.write(']}')

##Close the input and output files to release file locking
fin.close()
fout.close()
share|improve this answer
    
Note: you may run into trouble converting the converted geojson files via ogr2ogr, because of OS memory limitations. Take a look at this related question: gis.stackexchange.com/questions/16307/… –  RyanDalton Oct 29 '11 at 16:52
    
Great ! Microsoft has memory limitations –  twisig Aug 28 '13 at 13:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.