8

At my work have inherited several shapefiles that originate from MapInfo that I am bringing into a new project in QGIS. I have the opportunity to change the column names, add and subtract columns and because there is not too much data in them yet I can start afresh and adjust the field lengths too.

I notice that some of the field lengths are a lot bigger than they need to be and I remember from past database creation 20 years ago, or so, that it is best to keep the field lengths to be no longer than they need to be to save on 'space', to improve efficiency.

Is this still desirable or does the field length not matter anymore?

Improve this question
3
  • It depends on the format you are using.
    – bugmenot123
    Sep 28, 2015 at 13:46
  • 1
    The field length should probably be kept to "no longer than it needs to be" by definition you don't need anything longer. Of course it depend what you are capturing to determine what length you "need".
    – DMusketeer
    Sep 28, 2015 at 13:50
  • 3
    IMO the most important best practice is to stop using shapefiles if possible.
    – alphabetasoup
    Sep 29, 2015 at 19:06

1 Answer 1

Reset to default
12

The answer depends on data format. dBase-III+ files, which are used in shapefiles for attributes, are fixed-width, so defining a FIPS column to be 254 width text uses 254 bytes. Worse yet, dBase has a maximum record width of 4000 bytes, so the 249 wasted on a five character field aren't available for other fields (of which there is a maximum of 100 or 255, depending on who's implementing the standard). Limits also apply to the total size of the dBase file (2Gb), which could be approached by a 536k records at maximum width, when 5.36m records would be available at 400 byte width.

There is another reason to limit field width -- data quality. If a designator can only legally hold two characters, but you define it with ten, then you increase the possibility of having a fat-fingered invalid value accepted by the data file.

On the other hand, if you provide only the width necessary, and get international data in UTF-8 format, you may find yourself short on space when a character can use 2-6 bytes.

So, for database string fields (which includes file geodatabase), that are generally terminated, and therefore don't waste row space, flexibility is an option, but for fixed-width formats the old rules still apply.

Improve this answer
4
  • Thanks for the answers. I am not sure I completely understand the answer from Vince as I don't know that much about different database structures, but I do get the gist of it. I think my main consideration then will be with data integrity and not setting up anything longer than it needs to be - which seems obvious now - thanks
    – Martin Hügi
    Sep 28, 2015 at 17:18
  • The principal tools for thanking those who answer your question are to upvote and to mark the question answered. If you're unclear on something, ask. The idea here is to build good answers.
    – Vince
    Sep 30, 2015 at 5:12
  • 1
    @Vince answered this well, there's just one more little reason I would add: Making the intent clear. When a field named "state" has only 2 characters it's obvious that the field is supposed to hold the standard abbreviation for a state. However, if you make that field 50 or 200 characters, it might be interpreted as holding the full name of the state. This ties back into overall data quality.
    – RustProof Labs
    Sep 30, 2015 at 13:08
  • Follow up - After 18 months this all makes a lot more sense now - great answer
    – Martin Hügi
    Mar 21, 2017 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.