Loading multiple CSV files in QGIS and declaring encoding

Question

By the following code from the other post, I was able to load multiple CSV files into QGIS.

import glob, os

# Define path to  directory of your csv files
path_to_csv = "C:/Users/You/Desktop/csv folder/"  

# Set current directory to path of csv files
os.chdir(path_to_csv) 

# Find each .csv file and load them as vector layers
for fname in glob.glob("*.csv"):  
    uri = "file:///" + path_to_csv + fname + "?delimiter=%s&crs=epsg:4326&xField=%s&yField=%s" % (",", "x", "y")
    name = fname.replace('.csv', '')
    lyr = QgsVectorLayer(uri, name, 'delimitedtext')
    QgsProject.instance().addMapLayer(lyr)

However, not all my file headers are in English: instead of x,y I have: "經度", "緯度" so I tried to declare the encoding like common python files. As a result, QGIS load up the files but failed to get the x,y and the name directory.

# -*- coding: UTF-8 -*-
import glob, os

# Define path to  directory of your csv files
path_to_csv = "C:/Users/You/Desktop/csv folder/"  

# Set current directory to path of csv files
os.chdir(path_to_csv) 

# Find each .csv file and load them as vector layers
for fname in glob.glob("*.csv"):  
    uri = "file:///" + path_to_csv + fname + "?delimiter=%s&crs=epsg:4326&xField=%s&yField=%s" % (",", "x", "y") #THE "x" AND "Y" is not in ENGLISH
    name = fname.replace('.csv', '')
    lyr = QgsVectorLayer(uri, name, 'delimitedtext')
    QgsProject.instance().addMapLayer(lyr)

Where did I go wrong?

The one I load with ordinary ctrl+shift+T :

The one that failed with python script:

Answer 1

To achieve what you want, you need to also encode the field names for longitude/latitude and it will works (tested using your sample file name at https://gist.github.com/ThomasG77/1aa1bf66d847be043a246df8bf74793e You just need to change path_to_csv to your own path)

import glob
import os
from urllib.parse import quote


# Define path to  directory of your csv files
path_to_csv = "your_path"
x_field = '經度' # longitude
y_field = '緯度' # latitude

# Set current directory to path of csv files
os.chdir(path_to_csv)

for fname in glob.glob("*.csv"):
    uri = "file:///" + path_to_csv + quote(fname) + "?delimiter=%s&crs=epsg:4326&xField=%s&yField=%s" % (",", quote(x_field), quote(y_field))
    # Uncomment below line and comment above line
    # if when you open the layer table you see garbage
    # characters as it will deal with UTF-8 encoding
    # using additional `encoding=UTF-8` option
    # uri = "file:///" + path_to_csv + quote(fname) + "?delimiter=%s&crs=epsg:4326&encoding=UTF-8&xField=%s&yField=%s" % (",", quote(x_field), quote(y_field))
    name = fname.replace('.csv', '')
    lyr = QgsVectorLayer(uri, name, 'delimitedtext')
    QgsProject.instance().addMapLayer(lyr)

Answer 2

Try this:

# -*- coding: UTF-8 -*-
import glob, os
import urllib.parse

# Define path to  directory of your csv files
path_to_csv = "/home/rafd/Downloads/test_data/"
#print ('encoding in use:'+ sys.getfilesystemencoding())
# Set current directory to path of csv files
os.chdir(path_to_csv) 

# Find each .csv file and load them as vector layers
for fname in os.listdir():
    if fname.endswith(".csv"):
        print('This file is named: '+urllib.parse.quote_plus(fname))
        uri =  'file://'+path_to_csv +urllib.parse.quote_plus(fname) +'?type=csv&maxFields=10000&detectTypes=yes&xField=x&yField=y&crs=EPSG:27700&spatialIndex=no&subsetIndex=no&watchFile=no' 
        name = fname.replace('.csv', '')
        lyr = QgsVectorLayer(uri, name, 'delimitedtext')
        #print('importing: '+uri)
        QgsProject.instance().addMapLayer(lyr)

On a Linux box it works also with CSVs whose filename contain chinese characters ('經度.csv'); try it out on your OS.

For further import options, the official reference is:

https://qgis.org/pyqgis/master/core/QgsVectorLayer.html