70

Recently I've been spending a lot of time converting perfectly good field names like "Percent of citizens age 25 and over with a bachelor's degree or higher" into things like "edbchogtr" to meet the DBF's 10 character field name limit.

In another thread (“Oddities” in the Shapefile technical specification), geospatialpython commented that "Despite the shapefile format's flaws, oddities, and limitations it persists stubbornly in and around the field of GIS. Every other attempt to replace it has been too bloated for simple vector storage or too proprietary."

This activity coupled with Mr. Lawhead's comment has me wondering:

  • have any explicit attempts ever been made to replace the shapefile as GIS's ubiquitous data storage and interchange format?
  • Are there any contenders?
  • If there have been competing formats, why have they failed?
  • Has Esri refused to support them, or is the story simply one of technological inertia?
  • If there haven't been attempts... why not?

It seems like we could do a little better for ourselves, both as GIS developers and users.

Improve this question
13
  • 2
    @Mapperz Other than the recently released Geodatabase API, I don't see any tools for writing a geodatabase that are free. I don't think this could count as a replacement except in the ESRI portion of the world.
    – canisrufus
    Commented Feb 20, 2012 at 21:23
  • 2
    You can write and read geodatabases (via API) using GDAL gdal.org/ogr/drv_filegdb.html using resources.arcgis.com/content/geodatabases/10.0/file-gdb-api
    – Mapperz ♦
    Commented Feb 20, 2012 at 22:12
  • 1
    Would like to see Python API to read/write File Geodatabase (at least Simple Features) without ArcGIS license - that would be Open.
    – PolyGeo ♦
    Commented Feb 21, 2012 at 4:16
  • 2
    @PolyGeo you and everyone else :)
    – Ragi Yaser Burhum
    Commented Feb 21, 2012 at 18:21
  • 3
    @celenius From gdal.org/ogr/drv_shapefile.html "Geometry: The Shapefile format explicitly uses 32bit offsets and so cannot go over 8GB (it actually uses 32bit offsets to 16bit words). Hence, it is is not recommended to use a file size over 4GB. Attributes: The dbf format does not have any offsets in it, so it can be arbitrarily large." So you can have dbfs that are pretty big, but you have to be careful with your shp going over 4GB. Then you are playing with fire.
    – Ragi Yaser Burhum
    Commented Feb 21, 2012 at 19:08

7 Answers 7

Reset to default
52

This is a topic that always comes up. I may not have the right answer, but I can give you my personal opinion.

The reason that they are supported, can be attributed to several characteristics about them, so let me mention a few.

  • First, there is a spec. I mean, I am in my early thirties and this thing existed since I was a teenager. So it is safe to say that this spec has been around for some time. Of course, there are several other formats that are also published, but the difference about this one is that...

  • It is relatively simple! It is built on top of the DBF Format, which at the time already existed and was widely supported in several platforms/OSs. There were already parsers that could read half of this format (the DBF part), so it made supporting the extra addition easier. You have a geometry? Sure just serialize it and write it. You are done. Contrast this with a coverage! Try to explain to somebody in simple terms what a topology clean does. It is not trivial to write a topologically clean coverage.

  • Most importantly, I think the #1 reason for shapefiles to still be popular is that they are supported in both Open Source and Proprietary systems alike. What GIS do you know that doesn't support shapefiles?!? Unheard of.

As a replacement, we hear of File GeoDatabases and Spatialite. Both formats, are vastly superior in terms of functionality, flexibility, speed, etc. when compared to Shapefiles. In their own way, they have certain things that make them better than each other in different areas, but a comparison of spatialite and FileGDB is certainly out of the scope of this question.

Do I think that either of this formats will replace Shapefiles? Not in their current incarnations.

Why?

Not because of a technological argument (I did say they were superior in that aspect after all), but because of something else: licensing.

So what are their problems?

FileGDB:

FileGDB provides interoperability through the new FileGDB API. Nevertheless, this API is provided in binary format by ESRI. This is not a specification. Having worked in the GeoDatabase team in the past, I can tell you, contrary to all the tin-foil-hat-wearing conspiracy theorists, this is not malicious at all. It is because the internals of the GeoDatabase change on every release. Publishing a full spec would entail basically giving all the details of how everything is supposed to be maintained and then carefully documenting the changes to the format with every yearly release. It doesn't make sense. So the FileGDB API, even though it is not a spec, it abstracts out all those little changes. And now it can be used cross-platform! Mind you, this is a huge step forward! Considering the conservative nature of ESRI, this is definitely a reaction in the right direction.

And yet, binary-only support doesn't make anybody in the Open Source world too happy. How do you then take advantage of porting some code to say to some other flavor of Linux if ESRI doesn't support it. You can't. This is what makes Open Source powerful, and now, you cannot take advantage of this. If ESRI decides to stop supporting Debian, that's it. You are done. And there is nothing you can do to change it.

Spatialite:

Spatialite is awesome because it gets all the free functionality from SQLite. SQLite is used everywhere. It is on your Android Phone, on your iPhone/iPad, on Firefox, on Google Chrome, on several commercial embedded devices - can go on forever. To truly make it into a Geoformat (and not just do dumb bounding box operations), it needs to leverage the same geometry library that PostGIS uses: GEOS. Sadly, GEOS is based on another even more awesome geometry library known as JTS. All the algorithms in JTS are extremely powerful, so what is the problem?

Well, JTS is licensed as Open Source LGPL, and LGPL is a viral license. JTS is LGPL, means GEOS is LGPL, means spatialite linked statically with GEOS is LGPL. This sucks. Why? Without explaining open source licenses too much, I can tell you that, for example, I cannot use spatialite on, say, an iPhone app because that would make my entire app automatically open source (iOS only allows static linking). Any type of GPL license (reasonably) scares the crap out of ESRI, and so they will not touch it with a 10 foot pole. Hence, ArcGIS, the most popular GIS system in the world does not (and will probably never) support spatialite natively. This automatically kills it as a viable format.

And thus we go back to crappy shapefiles that are supported everywhere.

Update:

Apparently my answer was controversial enough that someone decided it was OK to freely edit and change the entire meaning of my answer to put their point of view. Please don't do that. If you disagree with me, that is completely fine, just post your opinion in a different answer and let the community decide. I rolled backed the edits to my answer to show the original meaning. I am adding this update in case you read the edited answer that claimed that sqlite was a viable format.

Improve this answer
21
  • The problem with SQLite/Spatialite is that it's not a format, it's a relational database engine with spatial library on top of it. While it does what it does very well, it forces the data to be stored in the relational manner, which isn't always the most suitable way. Also, the complexity of SQLite file format (sqlite.org/fileformat2.html) makes it difficult to access the data without the SQLite engine and is thus not suitable to be an open & easily accessible file format for data exchange. It wasn't really designed for that.
    – Igor Brejc
    Commented Feb 21, 2012 at 6:31
  • 8
    Actually, LGPL isn't a viral license - it was specifically designed to avoid this. Additionally, Spatialite is licensed under the MPL tri license (source), meaning among other things you can choose the Mozilla Public License as the best fit license and operate under its (very weak copyleft) terms. My reading at least is that ESRI have no reason not to support Spatialite because of the license - whether they will (given it competes in almost the same space as FileGDB) is another story...
    – om_henners
    Commented Feb 21, 2012 at 13:49
  • 4
    @Ragi, you mix using a library and porting it. Of course porting will have to be LGPL, since this is in essence a derivative work. But if you link it dynamically, it is not considered a derivative work, it's "work that uses the library" and you get to keep your license (en.wikipedia.org/wiki/GNU_Lesser_General_Public_License). So saying "LGPL is viral" without additional explanation is not accurate.
    – Igor Brejc
    Commented Feb 21, 2012 at 20:34
  • 2
    But again, this is a moot point, since Spatialite is licensed under a tree-licensed schema (groups.google.com/forum/?fromgroups#!topic/spatialite-users/…), so you get to choose the license that suits you most - MPL allows static linking.
    – Igor Brejc
    Commented Feb 21, 2012 at 20:34
  • 2
    @bugmenot123 Fine, then correct it if you wish, but dont accuse me of spreading FUD about OS because it is insulting. I have been writing OS code for over a decade (would not be surprised that you have used some of mine actually) and that was not an angry rant. It was true - and it still is. Dynamic linking in iOS of LGPL (well, to be precise, frameworks, were allowed in iOS 8). This has never been a technical issue, but a legal one. Distribution in the Appstore requires code signing - and sadly for all OS lovers like me - LGPL is a fuzzy license for this. No precedent in court.
    – Ragi Yaser Burhum
    Commented Jun 1, 2017 at 15:27
18

The SHP+SHX part itself isn't so bad. The real problem lies in the DBF part. That could do with a new format, which supported unicode and all sorts of modern field types. The problem is getting it well supported by all the software out there.

Improve this answer
7
  • 6
    +1 Improving on the DBF part isn't at all difficult, either: it really does come down to persuading software developers to agree on something.
    – whuber
    Commented Feb 20, 2012 at 21:57
  • 1
    Has there been an attempt?
    – canisrufus
    Commented Feb 21, 2012 at 1:28
  • 6
    I've often pined for a Shapefile amendment that simply substituted a UTF-8 CSV file for the DBF. It would be simple to support and require minimum changes to existing software packages.
    – scw
    Commented Feb 21, 2012 at 19:03
  • 1
    @canis Fox Software made a minor (proprietary) attempt in the late '80s. After MS purchased them (c. 1990), that was that. The community created a DBF 3 standard and that pretty much froze all development. MS released Access; FoxPro died out; the world moved on.
    – whuber
    Commented Feb 22, 2012 at 17:38
  • 1
    On the contrary, @Uffe, CSV files can be randomly accessed: you only need an index, just like DBF files do for efficient searches. The biggest problem I see is that seemingly minor changes that happen naturally to CSV files, like quoting strings or CR/LF conversions, will screw up all the byte offsets. The fixed length record structure of a DBF file, although less efficient in storage, does not have that problem.
    – whuber
    Commented Feb 22, 2012 at 17:47
11

GeoPackage is a promising successor. It's similar to Spatialite but from OGC and it's been adopted by many software, inlcuding ArcGIS and OGR.

See the official homepage http://www.geopackage.org/ and e.g. this presentation: http://www.slideshare.net/JeffYutzler/geopackage-swg-overview

Improve this answer
7

At least spatialite has the intention, see eg this presentation http://www.sourcepole.ch/assets/2010/9/10/foss4g2010_spatialite.pdf

On the other hand, I do believe that the main reason that it failed is that shp is well supported by many applications and only has minor deficiencies.

Others share this opinion as well:

This isn’t because the SpatiaLite project hasn’t given tools to us to implement, it has been the community could care less about it. SHP works for them and there isn’t any reason to change.

http://www.spatiallyadjusted.com/2010/09/16/spatialite-is-not-the-shapefile-of-the-future/

More thoughts on Esri file geodatabase, spatialite and autodesk sdf here: http://www.spatialdbadvisor.com/blog/121/the-shapefile-manifesto

Improve this answer
4
  • As great as I think spatiaLite is, it's ~3 megabytes of overhead in functions, reference systems, etc. that keep it from being a good all-around exchange format.
    – Scro
    Commented Feb 20, 2012 at 21:51
  • Actually, the license for spatialite is less than ideal - it has nothing to do with the tools.
    – Ragi Yaser Burhum
    Commented Feb 21, 2012 at 1:53
  • @Scro, 3 megabyte is too big? It certainly isn't too big for the desktop. You must be considering mobile devices. Also, is there another Spatial API, with equivalent functionality, in a smaller size than Spatialite?
    – klewis
    Commented Feb 21, 2012 at 14:14
  • @klewis -- it's not too big per se, it's just very inefficient when you consider there are a lot of small (think < 200kb) datasets out there. That's a lot of overhead, especially in light of the fact that, once received, you would typically leave each dataset in it's 3mb file, or roll it into an existing database. Just to be clear, I <3 spatiaLite -- but we're talking about data transmission, where some sort of flat file/xml/wkb would be much more efficient.
    – Scro
    Commented Feb 21, 2012 at 14:58
6

Esri's been promoting File Geodatabases for several years now as a replacement for shapefiles.

More recently they've provided an API that hides any oddities.

Improve this answer
3
  • I haven't worked with geodatabases a great deal. Wikipedia says they're a "closed" standard, e.g. the geodatabase spec hasn't been published. It seems hard to get very wide adoption without publishing the internals of the format. While I'm too young to know the history, it's my guess that shapefiles are in part so popular because of the public portion of the specification. The API does seem like a good step.
    – canisrufus
    Commented Feb 20, 2012 at 21:10
  • 2
    @canis you are correct. At the time nobody would have adopted shapefiles except that ESRI specifically promoted them as an open GIS data exchange format. Even with the limited software tools available at the time, with ESRI's release of a clear .shp/.shx specification (and a commitment to stick to it), it became a matter of just a few hours' work to write code to read and write shapefiles: no reverse-engineering necessary.
    – whuber
    Commented Feb 22, 2012 at 17:41
  • As long as the API is a black box binary blob, FGDB won't see the same adoption as SHP. Even if Esri convinces all their customers to switch to FGDB from SHP, the API isn't really compatible with open source.
    – dericke
    Commented Jul 1, 2015 at 19:28
3

An XML dialect, like GML, is definitely not optimized to operate huge datasets, but, can be used as an exchange format between software or between platforms.

I don't believe there is any problem with the licensing (see Ragi Yaser Burhum's post about the viral characteristics of Spatialite) and it is fairly easy to adapt existing parsers if required.

Improve this answer
3
  • 1
    I think it hasn't been mentioned for just the reason you're bringing up, that it's not optimized for large datasets. XML is bloated. The formats mentioned here are binary, where GML stores points as strings. The size can be over an order of magnitude different.
    – canisrufus
    Commented Feb 21, 2012 at 18:05
  • 3
    Canisrufus is right. There are several problems with GML. The Infoset can be be navigated using XPath, but anybody that has tried to implement spatial indexing on top of XML will tell you how irrational this is and how badly it maps to traditional relational databases. Without going into many details, if something as basic as indexing and querying become not trivial, the format is bloated, and it basically requires you to have the entire dataset in memory to do anything with it, then this is not a good option.
    – Ragi Yaser Burhum
    Commented Feb 21, 2012 at 18:56
  • 4
    xml is bloated when stored as plain text. There are freely (both free of charge and free to modify and redistribute) available binary xml libraries that can serve as drop in replacements for xml readers, giving people the freedom to utilize both the human readability of xml and the performance and storage efficiency of binary. The only reason I can think of for it never being taken up in a large way is as johandvw observes above: no one cares, .shp has been "good enough" as is.
    – matt wilkie
    Commented Feb 21, 2012 at 20:46
1

Just to come at this from a different perspective, I'm not sure the use of "Percent of citizens age 25 and over with a bachelor's degree or higher" is a perfectly good field name. While mixing spaces and apostrophes can be handled, if you are writing code or queries it is more likely to introduce bugs.

In my opinion the future of spatial data distribution should focus on the web and web services, and the WFS specification (which uses GML) is open and established. GeoJSON is smaller, and can be easier to work with in JavaScript. However with compression the sizes are comparable.

I'd also like to throw in a vote for ESRI's Personal Geodatabases. It may be an oft maligned Microsoft format, but it supports ODBC, SQL queries, views, and allows non-developers to create easy data entry forms, and include at least some level of data integrity checks (data types, lengths, unique values).

Improve this answer
2
  • That's a valid point. What's good about them is that given knowledge of the English language, one can figure out what the fields mean.
    – canisrufus
    Commented Feb 23, 2012 at 14:17
  • That's really the role of the datasets metadata though. The shapefile can use an XML file with the same name to store this.
    – geographika
    Commented Feb 23, 2012 at 18:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.