8

I am wondering what storage method will result in the fastest reading of the map vectors for rendering. SHP? PostGres? SQLite? (They do not change often and I do not need spatial functions for these vectors).

4
  • 1
    How many features are you talking?
    – Roger D.
    Jul 8, 2011 at 18:30
  • 2
    Your question seems to imply that the bottleneck is the process of reading data. My experience is that actual load is in the rendering process itself, at least with MapServer. That is, I do not see that using shapefiles or an spatial database will make a great difference.
    – dariapra
    Jul 8, 2011 at 18:41
  • @Roger - I'm not sure. Its a Map of the US using the highest resolution (states, counties, roads, rivers). I got the data from naturalearthdata.com
    – Nate
    Jul 8, 2011 at 20:26
  • @Nate, that being the case, why note read in OSM, or Google for your map? Then overlay on top of them? To render the entire US live each-and-every time is a large process. Then you need to consider network performance, disk-io performance, CPU. There are many things that can and will impact performance.
    – D.E.Wright
    Jul 11, 2011 at 3:32

4 Answers 4

12

Speed Tests

There are some very speed tests of shapefiles versus database (PostGIS) for MapServer in this presentation (from 2007).

In summary:

  • For a dataset of 3 million features running requests for 30 features one after another PostGIS was faster than shapefile (although this may have since changed by a fix to reading the shapefile index)
  • For a dataset of 10,000 features shapefile was slightly faster.
  • For concurrent requests shapfile was faster

And the times in detail, which can also help to decide if the storage format is an important factor.

                       PostGIS   Shapefile 
Start mapserv process  15ms      15ms
Load mapfile           3ms       3ms
Connect to DB          14ms      n/a
Query                  20ms      n/a
Fetch                  7ms       n/a
Draw                   11ms      28ms
Write Image            8ms       8ms
Network Delay          3ms       3ms

Always use FastCGI in MapServer if using a database, as the database connections can be reused, otherwise a new connection must be created on every request.

Implementations for Shapefile Readers

The speed of reading a shapefile (and data from a database) depends on the specific coding implementation.

The source code for MapServer opening a shapefile can be seen here. Following the comments you can see how important it is to have an index. Normally you can only read a file in one direction to get a record, but with an index you can read in two directions.

345   /*    Read the .shx file to get the offsets to each record in             */
346   /*    the .shp file.   

Another is example of opening a shapefile can be seen in the Python source for PyShp. Again you can see how an index is used to find specific shapes directly.

Other Factors to Condsider

The limitations of the DBF format (limits on field size, no null support, limits on text storage), should also be taken into consideration when deciding on whether or not to use a database.

A database also offers means of securing data, easier joining and creation of views, logging and many other features you won't get with a standalone file.

10

Contrary to what dariapra says, my experience in developing Maperitive tells me that the greatest bottleneck is in actual loading of the data before rendering. It all very much depends on how large the overall stored dataset is and how large is the dataset you are trying to render in one go. If you can load it all up into memory, then shapefiles are probably better than using a database engine.

3
  • 4
    +1 for pointing out that the form of the bottleneck matters and providing information from actual experience. Good stuff.
    – whuber
    Jul 8, 2011 at 19:34
  • Where is the bottleneck seems an interesting and open question that depends on a number of variables, perhaps the kind of workload is one important. I remember that once I indexed some shapefiles with shapetree for getting a faster rendering of some vector layers with MapServer, and I did not get a significative performance gain.
    – dariapra
    Jul 8, 2011 at 21:17
  • 2
    @dariapra True, it very much depends on the use case. If he's able to load all of the data in one go, then the spatial index is not really necessary, but loading of data from a shapefile should be much quicker than executing SQL queries. On the other hand, if there's a huge amount of data that needs to be filtered, I would put my bet on a database and not shapefiles.
    – Igor Brejc
    Jul 9, 2011 at 3:32
4

Which program will you use for rendering? This may influence the results. Anyway, having a shapefile with a spatial index (eg http://mapserver.org/utilities/shptree.html ) which is used is often the fastest technique. Apart from that: it depends on you application, but caching your rendered results is often much more useful for improving performance.

1
  • Thank you. I am using MapGuide at the moment (still kind of exploring our options), and I have been looking at Mapserver.
    – Nate
    Jul 8, 2011 at 20:19
2

Shapefile will be the quickest and probably your best bet. There is overhead for any SQL database, then there is managing the return of large result sets (conversion from database datatypes to native datatypes will also slow things).

Try using an open source package from maptools.org to do your reads. The ArcGIS tools, though purpose built, do have a bit of overhead to get started and are expensive.

Hope this helps

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.