Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

How do you manage your geospatial data? I have terabytes of data spread out over hundreds of datasets, and have an ad-hoc solution using symbolic links within projects which link back to an domain-name based archive directory for each dataset. This works mostly, but has its own issues.

I'm also keen to hear if anyone manages their geospatial data in a revision control system; I currently use one for my code and small datasets, but not for full datasets.

share|improve this question
1  
It would be useful to know what kind of files you use, what applications require access to the files, etc, etc. –  JasonBirch Aug 4 '10 at 8:00
    
I'm interested in this problem generally, so any answers are great. –  scw Aug 4 '10 at 17:56
    
I realized this question should probably be community wiki so we can get a single solid answer; hindsight is an exact science. –  scw Aug 12 '10 at 20:45

7 Answers 7

up vote 29 down vote accepted
+50

I think the stock/obvious answer would be to use a spatial database (PostGIS, Oracle, SDE, MSSQL Spatial, etc) in conjunction with a metadata server such as esri's GeoPortal or the open source GeoNetwork application, and overall I think this is generally the best solution. However, you'll likely always have a need for project-based snapshots / branches / tags. Some of the more advanced databases have ways of managing these, but they're generally not all that easy to user/manage.

For things you store outside of a database (large images, project-based files) I think the key is to have a consistent naming convention and again a metadata registry (even something low-tech like a spreadsheet) that allows you to track them and ensure that they are properly managed. For instance, in the case of project-based files this can mean deleting them when records management policy dictates, or rolling them into the central repository on project completion.

I have seen some interesting solutions though...

Back when the BC Ministry of Environment was running things off of Arc/Info coverages, they had a really cool rsync-based two way synchronization process in place. The coverages that were under central control were pushed out to regions nightly, and regional data was pushed back in. This block-level differential transfer worked really well, even over 56k links. There were similar processes for replicating the Oracle-based attribute databases, but I don't think they they typically did too well over dial-up :)

My current place of work uses a similar hybrid solution. Each dataset has its authoritative copy (some in Oracle, others in MapInfo, others in personal geodatabases) and these are cross-ETL'd nightly using FME. There is some pretty major overhead here when it comes to maintenance though; the effort to create any new dataset and ensure organisational visibility is considerably higher than it should be. We're in the process of a review intended to find some way of consolidating to avoid this overhead.

share|improve this answer
8  
If you're using PostGIS, its worth mentioning the History Tables feature new in 1.5 –  fmark Aug 4 '10 at 8:22
    
@fmark + 1 to you for mentioning it :P. –  George Aug 4 '10 at 12:51
    
If the data sets are related, also worth considering Postgresql inheritance to help retain consistency, improve performance, and allow hierarchical summaries. –  Adrian Aug 15 '11 at 13:06

Metadata is by far the most important issue here. If metadata answers whom, when, why, where it's an acceptable metadata record.

Having work experience in large companies with just a few GIS users (around 30) we had major issues to control data, specially versions and permissions. One side of this can be solved with extensive documenting of data (metadata) and the other problems are most likely solved with a central repository, in which PostGIS shines.

GeoNetwork is a good start to handle metadata issues. Solving the central repository is more complicated, because it might take a specialized person to design/maintain the database.

The complicated issue is who will be in charge of QA/QC these datasets and their metadata. Although computer driven processes work great they cannot be as rigorous as a good data manager/data keeper, which was made in this company I worked. Now there is someone exclusively there to review/commit metadata and organize geospatial data that is not centralized in a DBMS.

share|improve this answer

We have used a file system organized hierarchically by: - geographic extent (country or continent) - data provider, licensor - domain/dataset - date/version

After that we have a policy to separate the source data (in the same format that was on whatever CD/DVD that we got from the provider) from any derived datasets that we produced within our company.

The file system makes it really easy to ingest any data from the customer and also allows for some flexibility in terms of the physical storage - we keep our archives on larger, slower disks and we have special file servers (transparently linked into the hierarchy) for the more frequently used datasets.

To facilitate management within projects, we use symbolic links. We keep our vectors in a database (Oracle) and we make it a rule to have at least one database instance per customer (and several users/schemas for the projects). We haven't been keeping many rasters in a database, though, as they tend to take too much space even outside one. Also, we like to keep our database instances as lightweight as possible.

And yes, we have someone in charge of 'policing' the whole thing so it doesn't get too messy.

The biggest issue we have with this setup currently is the lack of a nice user interface which would help us have a better overview over the whole thing, and we've been planning to include a metadata storage on top of all that. We're still considering our options here.

We're using version control for our code and we've used it for documents, but it turns out that version control isn't really made for large datasets, especially if they're mostly binary files, so i wouldn't recommend that, except if you're dealing with GML or something similarly text-like (problems include huge overheads on the server-side disk usage as well as clients crashing when checking out huge repositories).

share|improve this answer

As @JasonBirch said, version control is a huge issue.

Also we've found that an appropriate workflow is hugely important. For example when we're collecting field data we tend to use staging databases where the field data can be QA'd before being merged into the master dataset. Depending on how much data needs to be QA'd this will always create some overhead though.

Also, if you haven't seen it I recommend taking a look at the Geo-communication and information design ebook by Lars Brodersen, at least for some of what he has to say on data modelling.

share|improve this answer

As the previous post mentions, spatial DB's and a metadata server are the usual setup. I think one key thing to remember is that 'one size does not fit all'. You'll end up with data that fits best in Oracle, file servers, SQL server, whatever. I've tried shoe-horning all data needs into one solution and it usually fails.

Expect to use different solutions that fit the data and plan for them. This is where the Geo-portal (metadata server) really comes in.

share|improve this answer

I have to agree with 'George' above that metadata should play a big role in managing geospatial data. Really with any digital data, metadata is key -- think of a photographer who tries to manage his digital photo files w/o proper metadata. Life gets so much easier if you tag things religiously, and have good software that can utilize the data. Now the original question about 'manage geospatial data' is pretty broad -- this could be data formats to store in, naming conventions, hierarchy of datasets and features, editing roles and privileges, etc. etc. etc.

share|improve this answer

Postgres all the way as others have said, however if you want to keep it portable and easy to move, then you could always look at using SQLite + the Spatialite extension.

Not as easy to use as Postgres in terms of management tools, but QGis CAN talk directly to a spatialite enabled GIS Database without any problems.

I actually use SQLite + Spatialite for backup, I have a windows service that run's in the background (Custom written) which monitors my PGSql instance, and mirrors my GIS Data into various SQLite DB's that reside on external USB drives.

One more tip with PG too, use schemas

Many folks I know just drop everything in "public" and be done with it, but if you organize your database correctly it makes the world of difference.

For instance, my "Ordnance_Survey" database has schemas for VectormapDistrict VectormapLocal Topo50 LookupGrids CodePointWithPolygons CodePointOpen

where I keep all the associated data.

Meanwhile the metadata tables, like geometry columns etc, all just live in Public, the Postgis extension is also only enabled on the public schema, but is accessible from all the other schemas in use.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.