6

I'm working in QGIS, and my current method to change CSVs to shapefiles is to import them to QGIS, right click and save as shapefile. For large CSVs (over 100 million rows) it can take quite some time to save. Is there a quicker, more efficient way to do this? Perhaps using GeoPandas?

10

1 Answer 1

0

I think that your process is not efficient because of the limitation with shp (see this). QGIS can create those shp with millions of features, but may be there are other ways.

In case you are familiar with R, you can try to do so programatically. It's no big deal. In fact, you can make R script to do so through QGIS. Here I propose 2 methods to do so:

  1. Reading all .csv within a folder and export all them to shp in other folder (this csv could be a manual or macro splited from your original .csv file).
  2. CSV --> GPKG (allows much more features per layer: see here)

So here is method 1:

You can transform csv's into shp's fast, regarding that this csv's are splits of your original data.

library(landtools)
library(mapview)
library(stringr)
library(sf)
#-------------------------------------------------------------------------------
# METHOD 1 --> CSV'S TO SHP
#-------------------------------------------------------------------------------
# List cvs files
csvs <- list.files("./data/", '.csv', full.names = TRUE)

# Read all csv files
dfs <- lapply(csvs, function(x) read.csv(x, sep = ';'))

# See where your coords are stored (next AtriCoords parameter)
names(dfs[[1]])

# TRANSFORM CSV (data.frame) INTO SPATIAL
layers <- lapply(dfs, function(x) dftopoints(x,
                                             "EPSG:25829",
                                             "EPSG:25829",
                                             AtriCoords = c("x", "y")))

# Export the layers, setting the name as the original name
outnames <- paste0(str_replace(basename(csvs), '.csv', ''), '.shp')
outfiles <- paste0('./output/', outnames)
for(i in 1:length(layers)){st_write(layers[[i]], outfiles[[i]], append = FALSE)}

# Check the results
# mapview(layers[[1]], zcol = 'Altura..m.')

enter image description here

And here method 2:

Notice that, as I don't have one of those .csv of yours, I just create a huge data.frame. In your case, x will be your .csv. Also, you can get rid of the system.time() part because that was only to test how much it takes to create a .gpkg with a > 100 million rows.

# Create a huge (data.frame) to imitate a huge csv 
x <- dfs[[1]]
while(dim(x)[[1]] < 100000000){x <- rbind(x, x)}
str(x)

# TRANSFORM CSV (data.frame) INTO SPATIAL
system.time({
        lay <- dftopoints(x,
                          "EPSG:25829",
                          "EPSG:25829",
                          AtriCoords = c("x", "y"))
})

# Export file
system.time({
        fname <- paste0("./output/", 'try01.pgkg')
        st_write(lay, fname, append = FALSE)     
})

NOTES:

  • The code need the library landtools just for one function(dftopoints). So, if you can't install the library or you don't want to, you can get the function here. The function needs a data.frame, an epsg of input and output (allows transformations) and the names of the attributes of lon/lat coords.

Not the answer you're looking for? Browse other questions tagged or ask your own question.