4

I have shapefile with lines, some of them are perfectly overlaps - duplicate, although each one have unique id etc.

  • When I view the file in QGIS I see they overlap.
  • When I convert the file with ogr2osm to osm XML & import to JOSM, and running validation I also get warning of duplicate ways (the lines converted to ways...).

How can I "programmatically" detect and remove such situations? I don't mind doing it in the shapefile or later in the OSM XML.

Note 1 - The attributes (tags) are unique - the actual spatial geometry is duplicate.

Note 2 - I want to use something like ogrinfo, ogr2ogr etc GDAL libraries either as commandline tools or as import library for python/java code. If other code tools exits it's perfect also.

  • 1
    If geometries are identical but tags differ, do you care which features would be deleted? Are only some technical tags like id or timestamp different, or do you also have different tags for highway or waterway etc? – user30184 Jan 4 '17 at 15:18
  • @user30184 Hi, first if I have 2 duplicates so I want to remove one of them (I don't care which). The identical tags are only technical, because if let's say it's same spatial geometry shape and one is highway and other is waterway so I can say it's "OK" and it's 2 use cases for same spatial area... – michael Jan 4 '17 at 15:22
  • 1
    can you add a simple example to the question? – Ian Turton Jan 4 '17 at 17:50
  • @iant It's simple situation when you have 2 rows in the shapefile representing identical spatial geometry with different id key. If you ogr2osm it so in the xml you see that both 'ways' consist of same 'nodes'. The question is how to access this kind of details via software tool and not QGIS GUI. Always java code scanning the xml with lot's of if statements is possible, so I'm looking for already exist libraries for that. – michael Jan 4 '17 at 19:42
4

It might be possible to do with ogrinfo and SQLite/SpatiaLite dialect http://www.gdal.org/ogr_sql_sqlite.html directly with shapefiles but it may be hard to handle the fids/rowids with that route. But if you convert your data into SpatiaLite you can use SQL for finding and deleting the duplicates. With spatialite command line tools https://www.gaia-gis.it/fossil/spatialite-tools/index you can use SQL in scripts.

Convert data from shapefile into SpatiaLite with ogr2ogr

ogr2ogr -f SQLite -dsco spatialite=yes my_data.sqlite my_data.shp

A query for finding the duplicates

SELECT a.ogc_fid AS a_fid, b.ogc_fid AS b_fid 
FROM equal_lines AS a, equal_lines AS b
WHERE ST_Equals(a.geometry,b.geometry)
AND a.ogc_fid!=b.ogc_fid
AND a.ogc_fid<b.ogc_fid;

A result will be like

a_fid   b_fid
1       3
5       6
5       7
6       7

Thus 1=3, 5=6, 5=7, and 6=7 (means that 5=6=7). Delete rows which have their ogc_fid listed in the right hand column and you have done.

DELETE FROM equal_lines WHERE ogc_fid IN
(SELECT b.ogc_fid  
FROM equal_lines AS a, equal_lines AS b
WHERE ST_Equals(a.geometry,b.geometry)
AND a.ogc_fid!=b.ogc_fid
AND a.ogc_fid<b.ogc_fid);

If you really need shapefiles you can export the processed table also with SQL by using the ExportSHP function https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html.

2

You can do this using GeoTools in Java using a method like:

public Collection<SimpleFeature> getDuplicates() {
    Set<SimpleFeature> ret = new HashSet<>();
    WKTWriter2 writer = new WKTWriter2();
    String geomName = features.getSchema().getGeometryDescriptor().getLocalName();
    try (FeatureIterator<SimpleFeature> itr = features.features()) {
      while (itr.hasNext()) {
        SimpleFeature feature = itr.next();
        Filter f = ECQL.toFilter("EQUALS(" + geomName + "," + writer.write((Geometry) feature.getDefaultGeometry())
            + ") AND NOT IN('" + feature.getID() + "')");

        try (FeatureIterator<SimpleFeature> itr2 = index.subCollection(f).features()) {
          while (itr2.hasNext()) {
            SimpleFeature next = itr2.next();
            ret.add(next);
            // System.out.println("dups " + feature + "\n " + next);
          }
        }
      }
    } catch (CQLException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
    return ret;
  }

The full code is here but the bulk of the work is in the filter which looks for matching geometries but different FIDs (or everything is a duplicate). For speed I've used a spatially indexed collection to help with the spatial queries (if you know that you have a database underlying your collection then feel free to skip this).

  • Thanks, 2 short questions: 1. Generally speaking - does via GeoTools I can have access to a shapefile and modify it? By modify I mean reflect also the change in the corresponding dbf file etc. 2. Does the ECQL acts like PostGis for GeoTools? Can I have access to PostGis in GeoTools? (or just by querying some DB via java code etc.)? Thanks, – michael Jan 6 '17 at 15:21
  • 1
    GeoTools has both Shapefile and PostGIS datastores, filters will work on both of them, the only difference is that the filter will be converted to SQL and passed to postgis but used in memory for the nonspatial parts of the shapefile. – Ian Turton Jan 6 '17 at 15:28
  • I'm struggling to find in the geotools tutorials example of how I modify a shapefile thus reflecting the changes in the dbf etc. Same with example how I use postgis from the java code (without remote DB). Do you familiar with such links? thanks, – michael Jan 6 '17 at 15:36
  • 1
    see gis.stackexchange.com/questions/215660/… or ask a new question – Ian Turton Jan 6 '17 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.