4

I am using PostgreSQL with PostGIS to store a large amount of polygons. I want to minimize the space my polygons are taking up in my database and since all the coordinate values of the polygons I am working with only need to be 32 bit integers, I could definitely save space if the EWKB format that PostGIS uses for storage could be shortened down to use 32-bit integers instead of the 64-bit doubles for each coordinate value. Here is where I found the code for the format: https://libgeos.org/specifications/wkb/#extended-wkb (GEOS is apparently a core dependency of PostGIS)

Is it possible for me to go in and edit the few lines of code that currently define doubles to define 32-bit integers instead? If it's possible, how would I be able to access the details of the datatype in order to change it? Also, when I asked ChatGPT about this, it said I would have to create a custom format and convert things from the standard EWKB to my EWKB format. Is that the only solution or is there a way I can just customize my database so that it only stores coordinate values in 32 bit integers?

I have found the GEOS library linked above which details the EWKB format, and I also know that there's an ALTER TYPE command in PostgreSQL, but I'm not sure how to go about altering such a specific part of the geometry datatype / the geometry storage format (EWKB)...

Any insight on this?

EDIT: For clarification, all my data are already all integers, so I am not losing any data by wanting to get rid of the extra 32 bits.

5
  • 2
    So you're proposing a new standard, and replacement for PostGIS?
    – Vince
    Commented Jun 4 at 17:15
  • 4
    Have you considered TinyWKB postgis.net/docs/ST_AsTWKB.html? Or postgis.net/docs/ST_QuantizeCoordinates.html?
    – user30184
    Commented Jun 4 at 19:41
  • 7
    PostGIS doesn't use EWKB for storage - see the internal format manifest - and you will not be able to change the format without a fundamental reformatting of the core framework. Look into better compression via ST_QuantizeCoordinates
    – geozelot
    Commented Jun 4 at 19:42
  • 3
    Here's a good blog post on the effect of using ST_QuantizeCoordinates: danbaston.com/posts/2018/02/15/….
    – dr_jts
    Commented Jun 5 at 18:42
  • Thanks everyone for your help! I tried ST_QuantizeCoordinates after you all recommended it but it didn't decrease as much as I thought. Then I realized this is because my values are already integers so maybe ST_QuantizeCoordinates does not help in my case. It is not that I want to round but that I want to take away the extra bits that are not needed even though they are already 0. Are there any other suggestions for a way to decrease the extra space taken up by the 64 bit doubles that I don't need? Or is Postgres's compression already shrinking the 0's as much as possible?
    – L K
    Commented Jun 11 at 13:08

1 Answer 1

7

By default, PostgreSQL will TOAST your large geometry values and compress them with pglz. If you're looking to save space, you can swap that out for a bit faster lz4 algorithm that also tends to offer higher compression ratio:

ALTER TABLE your_table 
  ALTER column_name SET STORAGE extended, 
  ALTER column_name SET COMPRESSION lz4;

Zstd isn't available yet but it's being discussed here and here.

You could also go the other way and SET STORAGE external, disabling the compression entirely to increase query performance at the cost of additional space required to accommodate raw values.

You can also play some column tetris. Or look into file systems with built-in compression (ZFS, BTRFS) - coincidentally, that thread asks about the exact same thing on the exact same setup, but seems to address only lossless compression.

Since you're already considering lossy compression, reducing the precision of all your coordinates to decrease storage consumption, you could trim the amount of vertices in your polygons with ST_Simplify() and dial down the precision of all coordinates with ST_SnapToGrid(). They both lead to simpler, more compact definitions of the same shapes, which are lighter on their own and compress better.

ST_QuantizeCoordinates() suggested by @geozelot and @dr_jts will do almost the exact thing you wanted to try, wiping bits beyond your target precision. Your idea to cut off some bits would effectively work the same: this function flips them to 0's and lets lz4/pglz compress that away. It won't collapse the vertices that become identical after dropping precision so it's still worth processing further with ST_Simplify() and ST_SnapToGrid().

ST_SnapToGrid() can kind of achieve the same thing as ST_QuantizeCoordinates() but the former thinks in measurement units, the latter in actual bits, so it's a bit tricky.

2
  • thanks for this information. As I commented above just now, it seems because my values are all already integers and the extra bits are already 0, ST_QuantizeCoordinates and ST_SnapToGrid don't seem to help much. Does this mean the compressor is already compressing the extra 0 bits that I don't need? Is there any more that I can do? Thanks also for pointing out ST_Simplify, I think I will be able to use that. lz4 for some reason didn't have an effect when I used it like you mentioned above.
    – L K
    Commented Jun 11 at 13:18
  • 2
    You can test and compare how much space your data occupies if you disable compression. Create one table with the geom column set up with storage parameter extended, then another one with external, copy the same subset of rows from your real table to each of these and see select pg_size_pretty(pg_total_relation_size('the_name_of_the_table'));
    – Zegarek
    Commented Jun 11 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.