0

I'm trying to work with a MIF file containing 5 really big multipolygons, but every operation I try takes hours, or just crash because of memory issues. Mainly using ogr2ogr, to transform MIF file to other format, like SHP or PostGIS. Unsuccessful, so far.

The file weights 700MB. I could load it into PostGIS using ogr2ogr. I just left the command running all night and it finished silently. The size of the table in disk is about 160MB. So, I suspect something went wrong (looks too small)

UPDATE

This is the header of MIF file

Version 300
Charset "Neutral"
Delimiter ","
CoordSys Earth Projection 8, 28, "m", -3, 0, 0.9996, 500000, 0
Columns 2
  Float Float
  String Char(32)
Data

After that, 5 regions with:

  • 124222 polygons
  • 462270 polygons
  • 798815 polygons
  • 1124915 polygons
  • 1706436 polygons

About the PostGIS table, I'm trying to check it. It contains 5 rows, like the original MIF file, but every single operation I try to execute over this table using the geometry column also takes hours. I tried to add a new integer field, and after 4 hours is still running. With ps, the process says "ALTER TABLE waiting".

This is the schema of the table

    Column    |       Type       |                         Modifiers
--------------+------------------+----------------------------------------------
 ogc_fid      | integer          | not null default nextval('my_table_ogc_fid_seq'::regclass)
 wkb_geometry | geometry         |
 float        | double precision |
 string       | character(32)    |
Indexes:
    "my_table_pk" PRIMARY KEY, btree (ogc_fid)
    "my_table_geom_idx" gist (wkb_geometry)
Check constraints:
    "enforce_dims_wkb_geometry" CHECK (st_ndims(wkb_geometry) = 2)
    "enforce_srid_wkb_geometry" CHECK (st_srid(wkb_geometry) = 23030)

So, how could I handle this big MIF file? I can't do anything with it, neither with the result PostGIS table.

5
  • Just to clarify, did you load the file into PostGIS, with the resultant table being 160MB? You suspect that something went wrong, but have you actually tried to load the new layer from PostGIS into QGIS, for example, and see if it comes up? Also, it wouldn't hurt to provide some more detail about the features in the layer, source of the data, etc. Mar 12, 2014 at 23:00
  • I suggest this only because it is entirely possible for a layer that is extremely large in one format, like MIF, to be much smaller in the PostGIS database simply because of the way that the data is encoded in one format vs the other. Mar 12, 2014 at 23:02
  • Yes, I think something went wrong because the table looks small. But ok. Please see update. And thanks for your response! Mar 13, 2014 at 10:18
  • You realize that with only 5 extremely big objects, a spatial index is not going to help you much. All operations need to process almost all data. Isn't it possible to split these 5 regions into smaller objects? Mar 13, 2014 at 16:17
  • Yep, that's what I'm trying to do. Seems to work modifying some PostgreSQL configuration parameters. I'll answer the question to show them. Thanks! Mar 13, 2014 at 19:27

1 Answer 1

1

Finally, looks like the file can be loaded to PostGIS in less than 1 hour just modifying these parameters in PostgreSQL configuration:

shared_buffers=2GB
temp_buffers=50MB
work_mem=1GB

Of course, you need enough RAM. My box has 8GB.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.