Encode CSV to utf-8 before publishing to Enterprise Portal

Question

I have a two step process here that's not working. 1) A script successfully writes some requested *.json to a *.csv file. 2) The file is then supposed to be added then published to Enterprise Portal, but is failing at this point with a KeyError. My earlier post shows that once the *.csv is set to utf-8 it publishes without the KeyError. But, I'm lacking a way to do this pythonically.

1. (run as a stand-alone script)

import json,requests
import os,sys
import csv

path_to_file = os.path.join(r'C:\Users\jpm\Downloads', 'c19_Vaccine_Current.csv')
#idph url goes here
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'

##get json data from idph
response = requests.get(idph_data,verify=True)
#read the json response and keep the VaccineAdministration part
data = response.json()['VaccineAdministration']

#write to file
with open(path_to_file, 'w', newline='', encoding='utf-8') as csvfile:
    f = csv.writer(csvfile) 
    #write the headers of the csv file
    f.writerow(['County','AdminCount','AdminCountChange', 'RollAvg', 'AllocDoses', 'FullyVaccinated',                    'FullyVaccinatedChange', 'ReportDate', 'Pop', 'PctVaccinated', 'LHDInventory', 'CommInventory',
                'TotalInventory', 'InventoryDate'])
    for elem in data:
        #get the values for all the keys (i.e. CountyName, AdministeredCount, etc...)
        f.writerow([elem['CountyName'], elem['AdministeredCount'], elem['AdministeredCountChange'], 
                    elem['AdministeredCountRollAvg'], elem['AllocatedDoses'], 
                    elem['PersonsFullyVaccinated'], elem['PersonsFullyVaccinatedChange'], 
                    elem['Report_Date'], elem['Population'], elem['PctVaccinatedPopulation'], 
                    elem['LHDReportedInventory'], elem['CommunityReportedInventory'], 
                    elem['TotalReportedInventory'], elem['InventoryReportDate']])

2. (run from a Pro Notebook)

from IPython.display import display
from arcgis.gis import GIS
import os
gis = GIS('Home') #uses current Notebook as workspace

csv_file = r'C:\Users\jpm\Downloads\c19_Vaccine_Current.csv' #path to CSV
csv_item = gis.content.add({}, csv_file) #add CSV to Enterprise Portal
display(csv_item) #display it here 

#location parameter needs to be set to "none" when publishing to Enterprise Portal
params={"type":"csv","locationType":"none"} 
csv_item.publish(publish_parameters=params) #publish to Enterprise Portal


#results in this error!
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [7]:
Line 3:     csv_item.publish(publish_parameters=params) #publish to Enterprise Portal

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish:
Line 10353: elif not buildInitialCache and ret[0]['type'].lower() == 'image service':

KeyError: 'type'
---------------------------------------------------------------------------

If I open the *.csv after the 1st step with this

with open(r'C:\Users\jpm\Downloads\c19_Vaccine_Current.csv') as f:
    print(f)

it prints the encoding as cp1252-- which I've read is because open() uses my machine's environment to choose the appropriate encoding.

I've set the encoding to utf-8 in the original with open() as csvfile. So, how do I bridge the gap? How do I make sure the *.csv is encoded as utf-8 in step 1 before I try publishing it to Enterprise Portal in step 2?

Answer 1

@mikewatt was on to something. The error lies somewhere in the publish() parameters using the python API. I was never able to figure it out.

#BUG-000136260: Unable to overwrite a hosted table in ArcGIS Enterprise using ArcGIS API for Python.

Workaround: Manually overwrite the file inside of Portal or overwrite through ArcGIS Pro.

Related cases: 02769409, 02753889