How to create multiple QGIS layers from multiple text files?

Question

I have a few hundred txt files (each representing a line of a river bed profile) that I need to load as (point-) layers in QGIS. Partial example:

4504764.8331;5374251.3024;397.8871;2
4504764.7761;5374250.8056;397.8555;2
4504764.7191;5374250.3089;397.8295;2
4504764.6621;5374249.8121;397.8149;2
4504764.6051;5374249.3154;397.7977;2
4504764.5481;5374248.8186;397.7725;2

It works manually (layer -> add delimited text layer, then renaming the columns after import) but given the volume I can't do it all by hand.

I found a similar two year old question here: How to import multiple textfiles in one step , but "create a plain text file called .vrt file for each of csv files" doesn't sound so peachy. The files are also missing the header line, but I know the format: "X; Y; Z; classification".

Is there any way to do this without having to code up a text generator for those vrt files and inserting header lines into all csvs?

Don't have time to flesh out a full answer, but I have a blog post which goes through automatically adding a header to the start of each file, and automatically creating the .vrt for each file at (this is in Linux though, so will require work for other OS's): gisforthought.com/… — HeikkiVesanto, Commented May 8, 2015 at 14:19
Instead of a separate .vrt file for each .csv it should be possible since GDAL 1.10.0 to have only one .vrt with an union layer gdal.org/drv_vrt.html. I am not sure if that is easier to create or not - with .csv it would mean an union layer made of a bunch of other vrt files. — user30184, Commented May 8, 2015 at 14:25

PolyGeo · Accepted Answer · 2016-07-04 04:47:35Z

8

Assuming all your text files are into the same directory, you can run this code snippet in the QGIS Python console to get your files loaded as individual layers in QGIS:

import os.path, glob
layers=[]
for file in glob.glob('/tmp/xy/*.txt'): # Change this base path
  uri = "file:///" + file + "?delimiter=%s&xField=%s&yField=%s&useHeader=no&crs=epsg:3116" % (";", "field_1","field_2")
  vlayer = QgsVectorLayer(uri, os.path.basename(file), "delimitedtext")
  vlayer.addAttributeAlias(0,'X')
  vlayer.addAttributeAlias(1,'Y')
  vlayer.addAttributeAlias(2,'Z')
  vlayer.addAttributeAlias(3,'classification')
  layers.append(vlayer)

QgsMapLayerRegistry.instance().addMapLayers(layers)

As you see, it works for GNU/Linux paths, I don't know how the uri (specifically the file path) should look like for Windows. Also, adjust the crs for your own case.

I also wanted to tell you how to change field names automatically, but it seems that delimited-text based layers cannot be edited. I guess the most you can do is to assign aliases for your columns, as the code does.

edited Jul 4, 2016 at 4:47

PolyGeo♦

65.4k29 gold badges114 silver badges345 bronze badges

answered May 24, 2015 at 18:02

Germán Carrillo

37k5 gold badges125 silver badges180 bronze badges

I solved my initial problem with some extra data I managed to get. But your solution looks like it would be the better way. However, I couldn't get it to work ("Errors in file / File cannot be opened or delimiter parameters are not valid" for every file). It's probably due to the path? I changed the path stuff but with little success: for file in glob.glob('C:\\temp\\linetest\\*.txt'): # Change this base path uri = file + "?delimiter=%s&xField=%s&yField=%s&useHeader=no&crs=epsg:31468" % (";", "field_1","field_2") print(uri).
– MapEngine
Commented May 28, 2015 at 14:52
(comment too long) This is the print result: C:\temp\linetest\test_line_5.txt?delimiter=;&xField=field_1&yField=field_2&useHeader=no&crs=epsg:31468. Pasting "C:\temp\linetest\test_line_5.txt" into the explorer bar opens the file however.
– MapEngine
Commented May 28, 2015 at 14:54
Yes, I gues the problem is related to Windows paths and the "file:///" prefix QGIS expects. In a couple of days I'll test paths on Windows and let you know the proper way of doing it.
– Germán Carrillo
Commented May 28, 2015 at 15:47
1

That's it - we just need to specify "file:///" with all three slashes. The rest of the script works like it is in Windows, no need for the escaped backslashes in the base path either. Thanks!
– MapEngine
Commented May 29, 2015 at 8:10
That's great! I've edited the post to add the third slash in the script.
– Germán Carrillo
Commented May 29, 2015 at 13:34

Add a comment |

Jakob · Accepted Answer · 2015-05-09 11:52:59Z

3

Combine all text files in a windows directory with the copy command:

copy *.txt river.txt

On Linux use the cat (concatenate) command:

cat * > river.txt

Then in QGIS for the merged text file use 'Add Delimited Text Layer' button.

edited May 9, 2015 at 11:52

answered May 9, 2015 at 10:19

Jakob

7,5011 gold badge24 silver badges42 bronze badges

1

This answer is correct, if you want a combined result. However my question was edited to ask for a single layer from multiple text files, which wasn't what I asked about initially.
– MapEngine
Commented May 11, 2015 at 0:08

Add a comment |

Community · Accepted Answer · 2017-04-13 12:33:56Z

The answer from @GermanCarrillo worked for me and saved much time.

With a text file structured like so: Time[s],Easting[m],Northing[m],Height[m],Roll[deg],Pitch[deg],Yaw[deg] 170258.002391,332731.5085,5794908.7794,46.1642,0.409501631306,-5.248541020385,0.684043081908

The code that worked for me in windows is:

import os.path, glob
layers=[]
for file in glob.glob('C:/Projects/2829/locations/20160524/*.txt'): # Change this base path
  uri = "file:///" + file + "?delimiter=%s&xField=%s&yField=%s&useHeader=no&crs=epsg:28355" % (",", "field_2","field_3")
  vlayer = QgsVectorLayer(uri, os.path.basename(file), "delimitedtext")
  vlayer.addAttributeAlias(1,'X')
  vlayer.addAttributeAlias(2,'Y')
  layers.append(vlayer)

QgsMapLayerRegistry.instance().addMapLayers(layers)

brodegon · Accepted Answer · 2017-07-10 08:15:00Z

1

I would like to add something to Germán Carrillo answer. For windows and QGIS 2.18.9 his code works perfectly for me. I just added a line of code to identify the path on windows:

pathName = r'put/your/path/here/spearated/by/slashes/'

you can copy+paste your path from the windows folder and change backslashes \ to slashes / before copying it to pathName.

Then, you just change the for line like follows

for file in glob.glob(pathName+'*.txt'):

Hope this helps

answered Jul 10, 2017 at 8:15

brodegon

3612 silver badges8 bronze badges

Add a comment |

Leo · Accepted Answer · 2019-01-30 04:15:19Z

from @nitrofruit @GermánCarrillo

Modified for QGIS 3:

import os.path, glob
from qgis.core import QgsProject, QgsVectorLayer
layers=[]
for file in glob.glob('C:/filepath/*.txt'): # Change this base path and change backslashes to forward slashes
  uri = "file:///" + file + "?delimiter=%s&xField=%s&yField=%s&useHeader=no&crs=epsg:28355" % (",","field_2","field_3") #(Delimeter, XField, YField) #1st column in file is column 1
  vlayer = QgsVectorLayer(uri, os.path.basename(file), "delimitedtext")
  vlayer.setFieldAlias(1,'X') #these order of columns go: 0, 1, 2 etc
  vlayer.setFieldAlias(2,'Y')
  layers.append(vlayer)

QgsProject.instance().addMapLayers(layers)

ZKB · Accepted Answer · 2019-09-20 08:08:35Z

I adapted Germán Carrillo's code to import xyz files (I use it to import river cross sections).

As the fields in my cross profile tables are separated by multiple spaces, what results in adding several empty columns if I use " " as decimal separator, I added a piece of code to replace the spaces by a comma (consecutive spaces are replaced by one coma). For the rest I used the same procedure as described in Germán Carrillo's post to import the cross profiles.

import os.path, glob
layers=[]
profile = 0
directory = 'X:/My/Working/Directory/' # Change this base path
for file in glob.glob(directory+'*.xyz'): 
  f = open(file, 'r')
  lines = f.readlines()
  f.close()
  new_file = directory+'PT_'+str(profile+1)+'.xyz'
  fw = open(new_file, 'w')
  for l in lines:
     fw.write(re.sub("\s+", ",", l.strip())+'\n')
  fw.close()
  uri = "file:///" + new_file + "?delimiter=%s&xField=%s&yField=%s&useHeader=no&crs=epsg:3943" % (",", "field_2","field_3")
  vlayer = QgsVectorLayer(uri, os.path.basename(file), "delimitedtext")
  vlayer.setFieldAlias(0,'id')
  vlayer.setFieldAlias(1,'X')
  vlayer.setFieldAlias(2,'Y')
  vlayer.setFieldAlias(3,'Y')
  layers.append(vlayer)
  profile +=1

QgsProject.instance().addMapLayers(layers)

As you can see in QGIS 3:

addAttributeAlias has been replaced by setFieldAlias
QgsMapLayerRegistry has been replaced by QgsProject

Below the original structure of my cross profiles:

1  1950459.685  2218176.491    37.574     0
2  1950468.259  2218173.250    37.466     0
3  1950471.806  2218171.910    37.466     0
4  1950474.013  2218171.076    35.000     0
5  1950477.754  2218169.662    35.000     0
6  1950480.095  2218168.777    36.465     0
7  1950483.372  2218167.539    36.465     0
8  1950495.173  2218163.079    37.214     0

And the resulting table after applying the strip function on each line.

1,1950459.685,2218176.491,37.574,0
2,1950468.259,2218173.250,37.466,0
3,1950471.806,2218171.910,37.466,0
4,1950474.013,2218171.076,35.000,0
5,1950477.754,2218169.662,35.000,0
6,1950480.095,2218168.777,36.465,0
7,1950483.372,2218167.539,36.465,0
8,1950495.173,2218163.079,37.214,0

There are probably some improvements that can be done, but at least it works for me.

Stack Exchange Network

How to create multiple QGIS layers from multiple text files?

6 Answers 6

Your Answer

Not the answer you're looking for? Browse other questions tagged
qgis
pyqgis
layers
batch
text
or ask your own question.

Linked

Hot Network Questions

How to create multiple QGIS layers from multiple text files?

6 Answers 6

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged qgispyqgislayersbatchtext or ask your own question.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
qgis
pyqgis
layers
batch
text
or ask your own question.