0

I'm trying to work with a MIF file containing 5 really big multipolygons, but every operation I try takes hours, or just crash because of memory issues. Mainly using ogr2ogr, to transform MIF file to other format, like SHP or PostGIS. Unsuccessful, so far.

The file weights 700MB. I could load it into PostGIS using ogr2ogr. I just left the command running all night and it finished silently. The size of the table in disk is about 160MB. So, I suspect something went wrong (looks too small)

UPDATE

This is the header of MIF file

Version 300
Charset "Neutral"
Delimiter ","
CoordSys Earth Projection 8, 28, "m", -3, 0, 0.9996, 500000, 0
Columns 2
  Float Float
  String Char(32)
Data

After that, 5 regions with:

  • 124222 polygons
  • 462270 polygons
  • 798815 polygons
  • 1124915 polygons
  • 1706436 polygons

About the PostGIS table, I'm trying to check it. It contains 5 rows, like the original MIF file, but every single operation I try to execute over this table using the geometry column also takes hours. I tried to add a new integer field, and after 4 hours is still running. With ps, the process says "ALTER TABLE waiting".

This is the schema of the table

    Column    |       Type       |                         Modifiers
--------------+------------------+----------------------------------------------
 ogc_fid      | integer          | not null default nextval('my_table_ogc_fid_seq'::regclass)
 wkb_geometry | geometry         |
 float        | double precision |
 string       | character(32)    |
Indexes:
    "my_table_pk" PRIMARY KEY, btree (ogc_fid)
    "my_table_geom_idx" gist (wkb_geometry)
Check constraints:
    "enforce_dims_wkb_geometry" CHECK (st_ndims(wkb_geometry) = 2)
    "enforce_srid_wkb_geometry" CHECK (st_srid(wkb_geometry) = 23030)

So, how could I handle this big MIF file? I can't do anything with it, neither with the result PostGIS table.

5
  • Just to clarify, did you load the file into PostGIS, with the resultant table being 160MB? You suspect that something went wrong, but have you actually tried to load the new layer from PostGIS into QGIS, for example, and see if it comes up? Also, it wouldn't hurt to provide some more detail about the features in the layer, source of the data, etc. – Get Spatial Mar 12 '14 at 23:00
  • I suggest this only because it is entirely possible for a layer that is extremely large in one format, like MIF, to be much smaller in the PostGIS database simply because of the way that the data is encoded in one format vs the other. – Get Spatial Mar 12 '14 at 23:02
  • Yes, I think something went wrong because the table looks small. But ok. Please see update. And thanks for your response! – Jorge Arévalo Mar 13 '14 at 10:18
  • You realize that with only 5 extremely big objects, a spatial index is not going to help you much. All operations need to process almost all data. Isn't it possible to split these 5 regions into smaller objects? – Uffe Kousgaard Mar 13 '14 at 16:17
  • Yep, that's what I'm trying to do. Seems to work modifying some PostgreSQL configuration parameters. I'll answer the question to show them. Thanks! – Jorge Arévalo Mar 13 '14 at 19:27
1

Finally, looks like the file can be loaded to PostGIS in less than 1 hour just modifying these parameters in PostgreSQL configuration:

shared_buffers=2GB
temp_buffers=50MB
work_mem=1GB

Of course, you need enough RAM. My box has 8GB.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.