You are using the old cursor which are much slower than the data access cursor. Create a view of the csv and use cursors on both shape and view to list all values then compare using sets:

    import arcpy
    
    fc = r'C:\GIS\ArcMap_default_folder\Default.gdb\ak_riks'
    fcfield = 'polenr'
    poles_in_fc = [f[0] for f in arcpy.da.SearchCursor(fc, fcfield)]
    
    csv = r'C:\GIS\data\testdata\plist.csv'
    csvfield = 'polenr'
    arcpy.MakeTableView_management(in_table=csv, out_view='csvview')
    poles_in_csv = [f[0] for f in arcpy.da.SearchCursor('csvview',csvfield)]
    
    print 'Poles in csv but not in fc: {}'.format(list(set(poles_in_csv)-set(poles_in_fc))) #https://www.geeksforgeeks.org/python-set-difference/

[![enter image description here][1]][1]


  [1]: https://i.stack.imgur.com/38Ka0.jpg