3

I have 2 shapefiles of point type. Both layers have lat/long associated with each point. In one shapefile, I have about 23 hundred thousand points and in another about 20k points. Because the first one has 23 hundred thousand points or features, it is very difficult to process the data or compute any algorithms on it.

Hence I want to somehow compare both shapefiles such that i get a layer with only those points which are in common in both the shapefiles. The shapefiles just have one common attribute that is geom (lat long) between them.

I am using WGS 4326 CRS .

  • have you tried Select-by-Location with a small search tolerance to select points in common? – klewis Oct 17 '16 at 19:02
3

In general you can use the function Join Attributes by Location for such an operation.

If your computer is not capable of doing a join with that amount of data, I would recommend to split the layer with the 23 lakh points into several layers, run the join function with the 20k points layer separately for each of the new layers and rejoin the result layers afterwards into one single layer.

  • Do you mean to say..that i need to split the layer with 23 lakh points into several layers . BUT that layer has only one unique column i.e ID.. So that would mean splitting the layer into 23 lakh differenet layers as it splits according to the unique attribute .ie Id – jeff devasia Oct 18 '16 at 4:52
  • If these are the only two files you want to merge, it would probably also be possible to manual split the big layer with QGIS into e.g. five or six layers. Besides that, does your computer/QGIS crash, when you try to run the join-function on the original files, or does it only take a longer time to process? – tallistroan Oct 18 '16 at 7:38
  • it takes as long as 3 hrs to process .. and yes computer hangs too sometimes – jeff devasia Oct 19 '16 at 19:51
1

Great answer by @tallistroan! Another method is to use the Field Calculator to geospatially return a set of attributes if a point from one layer lies in the exact same position as in another layer.

You need to have the refFunctions plugin installed and enabled by going to the menubar:

Plugins > Manage and Install Plugins...

Then use the Field Calculator on your 20k points layer, add a field (e.g. string) and use an expression like:

 geomequals( 'layer_name', 'field_name')

where:

  • layer_name = the name of your 20k points layer.
  • field_name = the name of any field from your 23 lakh points layer.

It does not matter what field you use because if there is a value, it means that that point exists in the exact same place for both layers. If the value is empty or NULL then it means only the point from the 20k layer exists at that point (assuming the field you chose for the expression has no blank values).

Now from the Attribute Table of your 20k points layer, you can sort the new field into ascending order and select all features which has a value in the new field. Then right-click the layer and select Save As... and make sure you check the Save only selected features option to save it as a new layer (or use the Save selected features tool from the Processing Toolbox).


Edit:

Another possible solution is to create a single string field which will contain your geom values. So if you have lat and lon fields, you can create a new field (in both your layers) with the expression:

concat( "Lat", "Long")

I would suggest using virtual fields to speed things up (make sure your two shapefiles are saved in a project for the time being). Then right-click your 20k points layer and go to Properties > Joins. Then select the other layer and select the new string fields you created. The attributes of the other layer will be shown in your 20k points layer. Use one of the fields from the other layer and if there are any NULL values, it means it doesn't exist in the 20k layer. Then apply the final steps as described in my previous approach.

  • I tried your method . But as soon as i enter the expression it says – jeff devasia Oct 19 '16 at 19:16
  • It says "too many features to compare " and hence does not process – jeff devasia Oct 19 '16 at 19:18
  • @jeffdevasia - Strange, the limit is said to be 100k points so not sure why the 20k points layer should have a problem. – Joseph Oct 20 '16 at 8:59
  • Any other solution? – jeff devasia Oct 23 '16 at 14:24
  • @jeffdevasia - Yup, edited my post to include another method using Joins. – Joseph Oct 24 '16 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.