13

My company has collected about 30 TBs of GIS data over the past 8 years, and I always find myself asking the following questions:

  1. What type of data do we have for a given geographic area?
  2. What are the details about that data (e.g., resolution in meters per pixel)?
  3. Where does the data exist on the hard drive so I can actually use it?
  4. Have we processed the data already, or is it in an unaltered form from the source?

Up to and including now, I have attempted to address these questions by devising an appropriate folder and file taxonomy/hierarchy. Does anyone have any ideas/suggestions on some understandable, perhaps even standard ways of organizing GIS data using files and folders?

I am also open to learning more about how using a database might benefit my company; we are software developers, not GIS experts, so I suspect we're quite a bit behind the curve on how best to approach the problem of storing/organizing GIS data for ease of use. I did see the question Best practices for managing geospatial data but was only able to draw marginal use out of the answers because I am so unfamiliar with geodatabases.

UPDATE: This last week I spent a fair bit of time reading about GIS databases, and got started familiarizing myself with PostGIS. Long-term, I think we will end up moving towards employment of a database plus metadata server as recommended by JasonBirch in Best practices for managing geospatial data.

closed as too broad by PolyGeo Jul 18 '18 at 22:05

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

2

If you are actually trying to edit data or develop a map, you will need to keep the data you are actively working on separate from the data you started with. When I start a project, I create a SourceData folder, with subdirectories named by the type of data (DEM, Orthophoto, Hydrology, etc.) This will hold all of the layers that I am merely using for reference. Any data I am working on will be copied into a different folder called Working. The Working folder holds data, MXDs and anything else that I modify or create in subdirectories that usually correlate to a phase of the project (MXDs, RoadEdits, Delivery, etc)

In addition to the actual GIS data, you should create a Communications or Specifications folder to hold any documents from your client/internal client/professor. This can serve as metadata when you come back to the project at a later date, as well as creating a centralized location where anyone else can see what is supposed to be happening.

  • 1
    Good points; our company makes maps that our software uses, and we have already developed a folder scheme for separating "raw" data from "working" data from "finalized" data. One of the problems is tracking down what set of raw data was used as the original basis for a final map; seems like your suggestion for a "Specifications" folder would address that. For each map we create, we would be sure to note what raw data source was used in the creation of the map (something we currently do not do). Thanks for the tips! – Sipp Nov 20 '10 at 17:35
1

It seems to me that you need a set of metadata to store this information, and a retrieval system that uses the metadata to let you extract data on the basis of the information.

I think you'd want a solution that supported an OGC Catalog Service, for maximum interoperability. I've seen colleagues use Deegree - though of course there are other solutions you should check out.

Here's an example of how we tied Deegree into our software (the live demo is down for maintenance right now - wouldn't you know! - but should be back up next week)

As for file naming, if you have a catalog service and delivery mechanism, then there's less of an issue about what the files are named and where they are. Otherwise I think it depends on how you look for the data. Do you first start by narrowing down the geographic area, or the type of data? That will determine whether the hierarchy starts by splitting data into tiles, then types of data per tile; or by splitting it into types of data, each of which has a set of tiles.

Of course with a spatial database you don't have the same issues about dividing the data into tiles, so that is often a preferential method - providing the end use application supports using that type of data.

  • Thanks for the suggestions Mark. It seems you are suggesting that there are a few components at play here: the metadata itself (e.g., an XML file), a retrieval system (Deegree?) that knows how to find data based on certain metadata requriements from the user, and a storage backend component (e.g., PostGIS?) that stores both data and metadata. Is that accurate? – Sipp Nov 20 '10 at 17:29
1

I would choose SpatiaLite which is an one-file database where you can insert all your shapefiles, rasters and tables. Then as a relational SQL database, you have the power of SQL queries in your disposal to do all necessary actions (join,select,merge,union,split etc) between attributes and files.

SpatiaLite is also accessible from programming languages such as Python for a greater degree of automation. The sky is the limit.

SpatiaLite Documentation and tutorials

0

I find it useful to create Word documents titled "Map name or theme - Metadata comments.doc". Document major edits and workflows in chronological order (YYYY-MM-DD) for each map and/or data set theme. If you need to figure out the history of a data set: i) Include the date modified / date created of related files which are useful as historical references or potential source files. Include a brief summary of the content of each file (layer names, # of records) while paying attention to general similarities or differences (i.e. what is new in each version of a map or data set). Keep the "- Metadata comments" file in the same working folder as the most recent version of the map or data set. Place older versions of the map or data in an Archive sub-folder. The three step process works well for software development, database development and file management: 1) Develop (& document); 2) Test (& document); 3) Publish (including metadata). 1) Working folder; 2) Archive sub-folder; 3) Published version.

Not the answer you're looking for? Browse other questions tagged or ask your own question.