QVariant error when converting DataFrame into attribute table using PyQGIS

Question

I have a df which looks like that:

ID	Happy
0	Very
1	Little

I'm trying to convert it into an attribute table:

headers = [col for col in df.columns]
fieldlist = QgsFields()
fieldlist.append(QgsField(headers[0],QVariant.Int))

for name in headers[1:]:
   fieldlist.append(QgsField(name, QVariant.String))

for i in df.index.to_list():
   featur = QgsFeature()
   newrow = df[headers].iloc[i].tolist()
   featur.setAttributes(newrow)
   sink.addFeature(featur,QgsFeatureSink.FastInsert)

Although I get this error:

Feature could not be written to Output_layer_9a58cfa4_ee1e_4b97_a6e8_dfc0410a280a: Could not store attribute "ID": Could not convert value "" to target type

Why do I get this even though I state that the first column (QgsField) has numbers and the others have Strings?

Answer 1

I believe it is because you are passing Pandas datatypes to the attributes of the QgsFeatures.

import pandas as pd

## make a toy dataframe
data = [[0, 'Very'], [1, 'Little']]
df = pd.DataFrame(data, columns = ['ID', 'Happy'])

## check dtypes
df.dtypes

## returns
df.dtypes
ID        int64
Happy    object
dtype: object

Here is a function that will convert int and float Pandas dtypes to native Python types. It may have to be be extended if you have more fields with different types.

def convert_dtype(data):
    """ function to convert pandas data types to native python type """
       
    def conversion(element):
        ## try and except are used because strings (in this context) behave differently  
        ## to ints/floats and do not have a dtype attribute
        try:
            if element.dtype.name == 'int64':
                return int(element)
            elif element.dtype.name == 'float64':
                return float(element)
            else:
                return element
        except:
            return element
       
    return [conversion(x) for x in data]

    
headers=[col for col in df.columns]
fieldlist=QgsFields()
fieldlist.append(QgsField(headers[0],QVariant.Int))
for name in headers[1:]:
   fieldlist.append(QgsField(name,QVariant.String))
for i in df.index.to_list():
   featur=QgsFeature()
   newrow=df[headers].iloc[i].tolist()

   ######
   ## convert the pandas dtypes to python types
   converted = convert_dtype(newrow)
   ######

   ## use the converted types to pass to the feature
   featur.setAttributes(converted)
   sink.addFeature(featur)

Answer 2

A standalone sample correction. The issue is the fact you try to fill QGIS content using for integer a <class 'numpy.int64'> (due to pandas type) whereas it should be a native Python int.

The important part is newrow = [c.item() if hasattr(c, 'item') else c for c in newrow] that cast numpy objects to their Python equivalent and then avoid your code to fail.

import pandas as pd
 
# initialize list of lists
data = [[0, 'Very'], [1, 'Little']]
 
# Create the pandas DataFrame
df = pd.DataFrame(data, columns = ['ID', 'Happy'])


headers=[col for col in df.columns]
fieldlist=QgsFields()
fieldlist.append(QgsField(headers[0],QVariant.Int))
for name in headers[1:]:
   fieldlist.append(QgsField(name,QVariant.String))

sink = QgsVectorLayer("NoGeometry?crs=EPSG:4326&field=ID:integer(10,0)&field=Happy:string(10,0)", "temp", "memory")

for i in df.index.to_list():
   featur=QgsFeature()
   newrow=df[headers].iloc[i].tolist()
   print(type(newrow[0]))
   newrow = [c.item() if hasattr(c, 'item') else c for c in newrow]
   featur.setAttributes(newrow)
   sink.dataProvider().addFeature(featur)

QgsProject.instance().addMapLayer(sink)