I want to remove specific fields that I don't need from multiple shapefiles stored directly in memory (the raw file data stored in a variable).

Unfortunately I haven't found any way to do this without having to write those files on disk, loading them, removing the unwanted fields, then reloading the files to memory and deleting the copy on the disk. And due to the shapefiles sizes, it takes a lot of time that could be shortened.

I only know PyQGIS so far, but I'm not bounded to it.

I actually do the following (using disk):

to_keep = ['Layers', 'I want', 'To keep']
vl = QgsVectorLayer(path, "tmp", "ogr")
attrib = vlIn.attributeList()

for elem in to_keep:
     i = vl.dataProvider().fieldNameIndex(elem)
     del attrib[i]
QgsVectorFileWriter.writeAsVectorFormat(vl, path, "utf-8", crs, "ESRI Shapefile")

Note : Shapefiles are downloaded from internet and are zipped.

  • First of all I doubt you can use downloaded data without saving it to your harddrive. You may create temporary layers, but I think even these are kept on the harddrive somewhere in the appdata for QGis. Second: Why do you not want to save the data? – Erik Sep 14 '18 at 7:51
  • You can load any file only using memory (maybe QGis don't), which is faster, the only limit is your RAM Size, and I have enough to handle the shapefiles I need. (For example you can't load a 22GO csv file with only 16GO RAM, but you can read it line by line, which should be easily feasable) As stated I'm not bounded to QGis, I don't care if I can't do it with it, and it's for speeding up the process time – Chris Prolls Sep 14 '18 at 8:06
  • @People who want to edit, please stop add QGis tags. I clearly stated in the post and the comments that I'm not bounded to it, the code provided is only an example. Thanks. – Chris Prolls Sep 14 '18 at 8:55
  • Could you please add a tag to narrow down the programming language (e.g. python, R, etc)? – Aaron Sep 14 '18 at 12:35
  • @Aaron Just did it, I don't look for a specific language, but if I have to, I prefer it to be Python. – Chris Prolls Sep 14 '18 at 12:50

None of the python libraries that handles dbf can both use only memory and remove fields, the better is do it by yourself.

Here is the recipe which I took inspiration from, which handles StringIo buffers, and helps a lot with the file structure : http://code.activestate.com/recipes/362715-dbf-reader-and-writer/

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.