6

I am trying to wrap my head around storing attribute values in the most efficient way for a large shapefile (>1 000 000 records). I have found excellent descriptions of data representations in 8-bit, 16-bit as interger vs. float, etc (e.g. https://www3.ntu.edu.sg/home/ehchua/programming/java/datarepresentation.html). But I can't find the equivalencies between QGIS and standard names for those formats. Furthermore, QGIS uses different terms depending on the function to further complicate things. Does QGIS only support two integer formats : 32-bit and 64-bit? Are real numbers assigned to Float32 or Float64 based on the length and precision?

Formats that I have found as accessed from Properties and Source fields with the corresponding options from add field are:

  • type/type name, selection from add field

  • double/real, Decimal number (real) with less than 10 digits (32-bit?)

  • double/double, Decimal number (real) with over 11 digits (64-bit?)

  • int/integer, Whole number (integer) (8-bit? 16-bit?)

  • qlonglong/integer64, Whole number (integer64)

enter image description here

enter image description here


Raster format has the following options with fairly straight-forward translations between names and data types:

enter image description here

  • 3
    don't use a shapefile for a large dataset! use a GeoPackage and don't worry about the size limits, plus you can add indexes. – Ian Turton Apr 4 at 16:15
  • 1
    I'm not worried about size limits in this case -- I just want the smallest file possible. With rasters, I have reduced file size from 30MB to 8MB by saving the data type (which was just values from 1 to 10) as 8bit integer vs. 32bit float. Also, GeoPackage has not yet caught on in most circles: my clients will only accept shapefiles. I hope to convince them one by one but it's not a fast turnaround. – user3386170 Apr 4 at 17:17
  • 1
    @IanTurton And regardless of the type of file, choices still need to be made about the data type, which is the point of my question. – user3386170 Apr 4 at 17:30
  • 1
    @user3386170 - In vector formats, real values are seen as double. Float32 or Float64 values are mostly seen in raster formats. – Joseph Apr 5 at 14:23
  • 1
    @user3386170 - Thanks but I would recommended you post your edition as an answer. If someone has more evidence, they could always post their own answer or comment it on yours :) – Joseph Apr 5 at 15:17
1

Thanks to the comments, I have come to understand how data is stored in vector formats (e.g. shapefiles). Most importantly, data values in vector and raster format are stored in completely different ways. Record attribute data is stored in the .dbf file associated with a shapefile uses the standard dbase file storage as explained here:

All fields values occupy the number of bytes specified in the field length property of their field descriptor.

Each record in each attribute (i.e. each field value) takes up the number of bits/bytes as set by the length of the attribute regardless of the number of characters/numbers in a given entry. The size is the same regardless of whether the entry has digits or letters. The only difference between a decimal and an integer in storage terms is that the "." counts as a character. Unlike raster format, there is not a sudden jump from the amount of space taken up by a number with 4 digits (16-bit) vs. 2 digits, less than 256 (8-bit) nor a difference between float and integer..

Based on the link that Joseph provided, I've made up the following table with the formats as QGIS calls them with the names OGR uses.

QVariant::LongLong -- OFTInteger64 -- ogrWidth 1-21
QVariant::Int      -- OFTInteger   -- ogrWidth 1-10
QVariant::String   -- OFTString    -- ogrWidth 1-255
QVariant::Bool     -- OFSTBoolean  -- ogrWidth = 1
QVariant::Double   -- OFTReal      -- ?
QVariant::Date     -- OFTDate      -- ?
QVariant::Time     -- OFTTime      -- ?
QVariant::DateTime -- OFTDateTime  -- ?

Each of the OGR field types is explained here, but mostly intuitive (real is Double Precision floating point and Integer is Simple 32bit integer).

The essentiel element to consider in minimizing the storage size that attribute data uses each to choose the minimum length for each attribute, as QGIS prompts us. So, don't overthink it!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.