6

Esri Shapefiles are commonly considered the standard format for GIS vector data.

I noticed on Wikipedia and the OpenStreetMap wiki it is noted (emphasis mine):

The shapefile format is a popular geospatial vector data format for geographic information system (GIS) software. It is developed and regulated by Esri as a (mostly) open specification for data interoperability among Esri and other GIS software products.

The citation for this is the ESRI shapefile technical description, which doesn't make mention of the term "open specification".

The Wikipedia definition for an open specification is:

An open specification is a specification created and controlled, in an open and fair process, by an association or a standardization body intending to achieve interoperability and interchangeability.

It also cautions against confusing open standards and open specifications, which perhaps I'm doing. Is the shapefile a standard or a specification? Does the Esri shapefile meet the above definition, and if not, what does it mean to say the shapefile is an open specification? Moreover, what does it mean that it is "mostly" open? What are the limitations to this specification's openness?

  • 3
    You'd have to ask the person who made that edit what they mean by "mostly". The shapefile specification is open (published and unchanged for 17 years). – Vince Apr 17 '15 at 12:36
  • So the wikis are incorrect? I have clarified the question a little with a definition for "open specification". If a specification is published and has longevity, does that make it open? I'd be happy to accept your statement as an answer, particularly if you can back it up with a source! :) – Ivan Apr 17 '15 at 12:46
  • Wikis are often incorrect, so much so that those with the correct information don't bother to change them. Case in point: That definition of "open specification" is gravely flawed. – Vince Apr 17 '15 at 13:28
  • I don't deny that wikis are often incorrect, in fact that is why I am approaching stack exchange. If you have a better source would you mind directing me to it? – Ivan Apr 17 '15 at 13:35
  • 2
    The shapefile specification describes the structure of .shp and .shx parts, and says that the .dbf part is of dBase format. It does not tell anything about the other parts which are listed in the Wikipedia article (.sbn, sbx, .fbn etc.). It does not even define the dBase version. Most software seem to label the .dbf file to use dBase III format despite Wikipedia claims it is IV. Read a story about reverse engineering the non-open sbn part from geospatialpython.com/2011/10/… – user30184 Apr 17 '15 at 13:39
5

It seems that a Shapefile is indeed an open specification. As Vince suggested, the "mostly" comment should be verified with who-ever written it.

Here are some sources:

This document also provides all the technical information necessary for writing a computer program to create shapefiles without the use of ESRI ® software for organizations that want to write their own data translators.

Such a feature makes ESRI Shape file format to be a format that is

open specification for data interoperability

As mentioned above, and also in the Library of Congress / Digital Preservation of the US Federal Government.

It might be that the "mostly" comment comes from the rest of the "open specification" definition, that is:

a specification created and controlled, in an open and fair process, by an association or a standardization body

Thus, one can ask whether ESRI is an association or a standardization body? and does the regulation process of the format (specification) is fair and open? Browsing the Open Geospatial Consortium website; I couldn't find a standard/specification relating to ESRI Shapefile. That is, that at least this standrdiaztion body didn't find ESRI shapefile in compliance with its own openness standards, or at least that this issue isn't or wasn't on its agenda.

  • Thanks for your answer. It seems odd to me though that since the Esri Shapefile is such a central specification within GIS, it would not have something more concrete available with regards to its licensing and deliberation process. Thanks for your research and pointing me in the right direction. I will let the question stand for the weekend and accept your answer if nothing more comes up! – Ivan Apr 17 '15 at 13:39
  • "Licensing and deliberation process"? Back in the day, companies floated ideas and implementations for public consumption all the time. Shapefile was designed long before it was released, so the only deliberation was around what files would be included (which, of course, was non-germane to the actual specification). – Vince Apr 17 '15 at 14:04
2

I always thought of the shapefile as an open format, however I cannot be 100% sure for its "openness". I also think that:

a) the (rather justified) term "mostly open" can be translated as "publicly accessible" i.e. without any obvious restrictions to the accessibility of the information itself. Anybody can download a copy without any cost.

b) moreover, a standard is usually something formal and well defined, sometimes abstract, that can be adopted by an organisation (e.g. ISOs are all standards, like the ISO 19115 regarding geospatial metadata)

c) a specification may be (amongst other things) a technical description or an implementation of a standard (e.g. ISO 19115 has an XML/XSD documentation and description)

In this sense, a shapefile is both a type of standard coupled with its specification, as the published document's title implies ("technical description").

If there was a standard related to the shapefile, then it should refer to how the information itself is organised. It should not be tightly coupled with its specification and implementation. A shapefile standard could have other means of implementation.

If someone says that "a shapefile is a widespread format and has become the GIS industry's standard" is a different thing. It does not mean that every single GIS software must support the shapefile format (although that is the trend).

I'd also like to point to the terms included in the PDF of the shapefile's technical description:

Copyright © 1997, 1998 Environmental Systems Research Institute, Inc. All rights reserved.

...

The information contained in this document is the exclusive property of Environmental Systems Research Institute, Inc. This work is protected under United States copyright law and other international copyright treaties and conventions. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or by any information storage or retrieval system, except as expressly permitted in writing by Environmental Systems Research Institute, Inc. All requests should be sent to Attention: Contracts Manager, Environmental Systems Research Institute, Inc., 380 New York Street, Redlands, CA 92373-8100 USA.

The information contained in this document is subject to change without notice.

If the shapefile description was as open as it could be, then the copyright terms could probably be less strict, especially the ones about reproducing the technical description.

Also there should be a place where changes in the specification should be available to anyone.

Finally, the intention of the published document (as stated) is to provide means of data translators.

To conclude, I think that the widespread adoption of the shapefile will soon make it a clearly open standard. It's up to ESRI to express that in written in a future document or a license agreement.

EDIT:::: This text from the Library of Congress is a relevant and useful read (http://www.digitalpreservation.gov/formats/fdd/fdd000280.shtml)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.