2

I want to remove duplicated/overlapping lines from a multiline layer using the sf package in R.

The layer I have has a lot of attributes, and in many cases one line is recognized as 5 overlapping lines. I want to remove the repetitive lines so that every line is unique. It is fine to lose attributes, but I need a way to reconnect those coordinates into lines. The layer is big enough that I can't query based on conditions to remove each repetitive line.

A similar question-- https://github.com/r-spatial/sf/issues/669 --was asked regarding spatial points, and the solution requires the distinct() function from tidyverse. However, when you do that with lines, you lose the attributes necessary to reconnect coordinates as lines in the first place.

Here would be example code, where the goal would be to remove the duplicate line3.

library(sf)
library(dplyr)

# Example lines (note these are taken from the sf vignette)
line1 <- rbind(c(0,3),c(0,4),c(1,5),c(2,5))
line2 <- rbind(c(0.2,3), c(0.2,4), c(1,4.8), c(2,4.8))
line3 <- rbind(c(0.2,3), c(0.2,4), c(1,4.8), c(2,4.8)) # Repetitive to line2
line4 <- rbind(c(0,4.4), c(0.6,5))

# Merge together as sf object
lines <- st_multilinestring(list(line1,line2,line3,line4))

# Recommended method to remove identical spatial objects using sf
lines_distinct <- data.frame(st_coordinates(lines)) %>% distinct(X,Y)
head(lines_distinct)

The problem here is that the object lines_distinct just has the coordinates, I have no way to differentiate line1 from line4.

I would also accept any answer using the sp package, but I'm trying to transition more to sf.

2
  • the distinct function is from dplyr, not tidyverse - do not use library(tidyverse) when you can more minimally use library(dplyr).
    – Spacedman
    Sep 4 '18 at 20:07
  • 1
    in distinct set .keep_all = TRUE and then it should keep the attributes
    – see24
    Oct 30 '20 at 15:13
2

How about using the unique() function:

library(sf)

# Example lines (note these are taken from the sf vignette)
line1 <- rbind(c(0,3),c(0,4),c(1,5),c(2,5))
line2 <- rbind(c(0.2,3), c(0.2,4), c(1,4.8), c(2,4.8))
line3 <- rbind(c(0.2,3), c(0.2,4), c(1,4.8), c(2,4.8)) # Repetitive to line2
line4 <- rbind(c(0,4.4), c(0.6,5))

# Merge together as sf object
lines <- st_multilinestring(list(line1,line2,line3,line4))

# Only unique lines
lines2 <- st_multilinestring(unique(lines))
2
  • library(tidyverse) is not needed here, unique is a base R function. In general, library(tidyverse) is a bad idea since it pulls in so many packages and functions, and doing it when you really only want one or two functions from, say, dplyr is not best practice.
    – Spacedman
    Sep 4 '18 at 20:05
  • Thank you! That was careless of me to forget about unique() and I've updated the question to library(dplyr) Bad habit
    – B. R.
    Sep 4 '18 at 21:57
1

To address the original question, just change the following line to include a group_by_all. As noted in the help file, "groups are not modified" and "grouping variables are always included."

lines_distinct <- data.frame(st_coordinates(lines)) %>% group_by_all() %>% distinct(X,Y)

Anyway, I spent the better part of the day trying to figure out how to apply this to large feature classes. The first issue I ran into was that the distinct function would remove duplicate geospatial objects from a feature class, but it also removed all of other fields from the data (i.e. the attribute table's data). I tried to use group_by_all, but dplyr doesn't like it when you try to group the shape field. When trying to use unique, I got other errors, so I went back to testing out distinct.

Although using .keep_all = TRUE allows users to retain what appears to be all of the attribute data, it actually removes unique attribute table records for any duplicate geospatial objects. While this might be okay in some applications, if you're tracking the changing progress of a single geospatial object across multiple time periods, then you do not want to use that argument.

To treat such a record as a unique record and to work around the limitations of distinct(), I created the following function:

Remove_Dupe_Geos <- function(data) {
  Non_Geo_Fields <- names(data)[!names(data) == "Shape"]  
  Temp_Uniques <- data %>%
    group_by(across(Non_Geo_Fields)) %>%
    distinct() %>%
    ungroup()
}

Calling the function basically creates a character vector with all of the feature class's field names with the exception of "Shape" so that the group_by and across will function correctly. Now when you call distinct, all of the original data from the attribute table is retained and considered when removing duplicates.

To test this out and allow others to review my work, I added a few more lines in the middle to record the name of the feature class that was passed through, which I use for saving an RDS object.

Remove_Dupe_Geos <- function(data) {

  Non_Geo_Fields <- names(data)[!names(data) == "Shape" ]
  
  Feature_Class_Name <- deparse(substitute(data))
    
  Temp_Dupes <- data %>%
    group_by(across(Non_Geo_Fields)) %>%
    filter(n() > 1) %>%
    ungroup() %>%
    saveRDS(file = paste0("Temp_Dupes (", Feature_Class_Name,").RDS"))
        
  Temp_Uniques <- All_Veg_Lines_NAD_1983 %>%
    group_by(across(Non_Geo_Fields)) %>%
    distinct() %>%
    ungroup()

}

Please note that this process can take a while with very large feature classes; however, I can't find another solution that removes true duplicates from a feature class. If someone else has a more efficient way that does not involve dropping the geometries, then please let me know. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.