I was wondering about the difference in spatial data storage methodology used by Coverages, Shapefiles and Geodatabases in ArcGIS. Coverage was the initial format, followed by Shape Files and now Geodatabases. So what has improved in these newer formats of Shapefiles and Geodatabases?
It would be great if someone could please explain it with examples.
This is such a great question. Coverages, Shapefile and Geodatabases are fundamentally different geospatial data stores from an implementation standpoint as well as from a philosophical one. I'll try to summarize without going too deep into it.
1. Coverages:
Coverages are interesting geospatial data structures. They concentrate on storing topology. So you will see that the emphasis is in storing the geometry elements first, that is the nodes, edges that make up all the geometries. You will then see a separate set of tables that relate those geometries to the attributes (and hence they "become" features).
A "clean" coverage guarantees certain rules, for example, that there are nodes at every node intersection, you will not have two (or more) nodes on top of each other (or even within a fuzzy tolerance distance), that there are not two edges on top of each other, etc. They also have a sense of direction (from->to) and can distingish between what is to its left and right side.
Coverages work really well for edits that require awareness of topological relationships (imagine editing a parcel boundary). In addition, coverages compress very well since they remove geometric redundancy by design. In fact, you will see that nowadays, modern formats like TopoJSON started using the same techniques that we learned from coverages several decades back.
Coverages can be a bit more complicated to work with when you are dealing with 3D data (for example modeling a bridge that has an upper side and a lower side right below) because the algorithms that we used to use to deal with them were inherently meant for 2D planar graph math.
So why did we move away from it? That would take a longer answer, but perhaps we should explain a bit more what made ESRI Shapefiles popular first.
2. ESRI Shapefiles:
Along came the Shapefile. Probably the most important characteristic that it had was that it was/is an Open Specification that was (comparatively) simple to implement. The attributes leveraged DBF files, so there were already many libraries that implemented a big part of the spec. There was no concept of "clean", which meant that each individual geometry only had to worry about representing itself without taking in consideration the geometries around them or that they intersected. This meant that we did not have to do any complicated math to make sure that a shapefile was correct (unlike the coverage counterpart).
Have multiple geometries that cross each other? Sure why not. Two points on top of each other? Be my guest.
Sometimes, the (arguably) "best" format is not the one that wins, but the one that gets adopted. If a format is easy to implement, it has better chances to be adopted than a complicated one. That was the Shapefile.
All of a sudden you had several libraries (open source and proprietary) and software vendors that supported it. So all was great.
The obvious question is then - why Geodatabases?
3. Geodatabases:
I believe Geodatabases are one of the most misunderstood geospatial data stores. People usually think of them as just "a geospatial format". A couple of years ago, somebody asked "What are ESRI Geodatabases?". Instead of repeating what my answer was then, I welcome you to read that first. I'll wait :)
Now that you read that answer and know what a Geodatabase is, I can expand a bit more on that answer. At the time, there was a lot of research optimizing SQL and writing query optimizers that leveraged indexes, column stores, etc (there still is). By building the Geodatabase on top of a SQL datastore, we can leverage all that research for free. We only need to concentrate on the geospatial concepts, and as the SQL data stores get better, the Geodatabase gets better, too, for free. Not a bad proposition huh?
Nowadays, there are several specifications for geospatial data that come out. The jury is still out there on what is going to replace these technologies (if anything). Nevertheless, if you are interested in this topic, I recommend reading the answer to a questions asked here in GIS.SE some years back: "Are there any attempts to replace the shape file"
I hope this helps!
Most of the information can be found in Esri Help and some search, so I've just compiled some good reads.
How coverages are stored? Since it is a proprietary format, you won't find any technical specifications on how the algorithms are implemented (unless @Vince will shed some light).
Shapefiles came later on and were implemented as a standard which provided a certain level of interoperability. ESRI Shapefile Technical Description contains full description.
Geodatabases were introduced later on. First personal geodatabases came (MS Access), then file geodatabases (binary encrypted format) and enterprise (or ArcSDE) geodatabases which took advantage of the ArcSDE and DBMS technology. You can read more about geodatabases here: Types of geodatabases and The architecture of a geodatabase.
A good read on GIS.SE: Whether to use File Geodatabase (*.gdb), Personal Geodatabase (*.mdb) or shapefiles?
Concerning performance, there were many benchmarks performed and file geodatabases show to be the fastest in terms of reading/writing information. Personal geodatabases and shapefiles are by far slower and probably the only reason to use them is to support older systems which were built with some MS Access business logic or shapefile reading/conversion in mind.
ArcSDE based geodatabase often perform nearly as well as file geodatabases when the DBMS is tuned, but it all depends on the type of data stored, networks, and hardware. For more information on benchmarks, refer to the Esri system design strategies resources (and here).
The primary difference between these formats is the way that features relate to geometries. Back in the heyday of coverages, the coding language was FORTRAN, which meant fixed buffer sizes in COMMON blocks. The most restrictive of these was 500 vertices per line primitive ("arc"). This restriction introduced the concept of "pseudo-nodes" (places where arcs join with only one other arc), and complicated many other data access operations.
The coverage model used a "polygon arc list" (PAL) to describe polygons, which required a polygon shading algorithm to read one file to obtain the list of arcs, then read the arcs themselves to obtain the vertex count, then allocate enough RAM to store all the vertices, then go back to read the arcs again, this time copying the vertices in forward or reverse order to assemble a complete polygon. Only after two visits to the ARC file could the polygon be adequately described, and then many of the same arcs would need to be accessed (in an opposite direction) to shade the polygon neighbors.
By comparison, the shapefile and various geodatabase formats store the complete geometry as a single object (with various implementation details on how the object is physically implemented). This has drawbacks when trying to edit shared boundaries, but that operation is significantly less frequent than polygon shading.
The "whole shape" storage model is the key difference between coverage format and the new ones, and this difference is so fundamental that it's hard to see any real difference between the shapefile and the various geodatabase formats. In fact, the shapefile format was used to access FGDB geometries through the FGDB API, even though FGDB doesn't use that exact format, just because it was simpler than introducing a new vertex layout.
One more difference between the formats is that a geodatabase can model relationships between feature classes. As Ragi noted,
Coverages work really well for edits that require awareness of topological relationships (imagine editing a parcel boundary).
But this awareness exists only within a single coverage - if you want to model the relationships between 2 or more coverages, it's your responsibility to write the code which checks for any "illegal" topological relationships.
For example, if parcel polygons cannot have gaps, and parcel boundaries should align exactly with roads, with coverages or shapefiles it's up to you to verify that this is the case, and manually fix any errors where these rules are not upheld.
A geodatabase can optionally support a Topology object, which allows you to define the allowable topological rules for your data. Importantly, these rules can occur both within and between feature classes in your geodatabase.
The topology edit tools within ArcMap help you to find topological violations, and in some cases to fix them automatically.
Prior to the introduction of geodatabase topology ("the good old days"), it was common to write long and complicated AML scripts to detect topological violations, then edit the coverages manually in ArcEdit (not so much fun).
