0
# Create a dataframe from sorted JSON dictionary containing desired data
df = pd.DataFrame(data, columns=['lat', 'lon', 'id'])

# coerce dtype 'O' object values to np array native float64 type
df['id'] = pd.to_numeric(df['id'], errors='coerce')
df['lat'] = pd.to_numeric(df['lat'], errors='coerce')
df['lon'] = pd.to_numeric(df['lon'], errors='coerce')

# remove  all rows containing NaNs (no data2 information)
df = df[~np.isnan(osmdataframe).any(axis=1)]

# Create a sorted np array
sorted_osmNParray = np.array(osmdataframe.to_records())

# reconvert to np array so it can be passed to NumpyArrayToTable method
osmNParray = np.array(sorted_osmNParray, np.dtype([('index', '<i8'), ('lat', '<f8'),  ('lon', '<f8'), ('id', '<i8')]))

# Convert NP Array to table
arcpy.da.NumPyArrayToTable(osmNParray, output)

My goal for this script is to extract some information from an Overpass API query and ultimately join it with the table of another feature class I have created.

So far I have completed the following steps relevant to my problem:

Extract the desired data from the JSON dictionary

  • Put data into a pandas dataframe

  • convert said dataframe to a sorted numpy array, then back to nparray

  • export the data in my numpy array to an ArcGIS table via NumPyArrayToTable()

A look at the contents (view from dataframe without nans)

             lat        lon          id
0      48.132752  11.566575  1576559984
1      48.138378  11.576535  4785090864
...          ...        ...         ...
23703  48.166622  11.587858  1601228494
23704  48.166629  11.587299  1601228506
23705  48.166637  11.587443  1601228508
23706  48.166638  11.587735  1601228509
23707  48.166641  11.587711  1601228510
23708  48.166649  11.587303  1601228511
23709  48.166994  11.587971  1601228515
...          ...        ...         ...

A look at the data types of the numpy array

print(sorted_osmNParray_way.dtype)
...
(numpy.record, [(u'index', '<i8'), (u'lat', '<f8'), (u'lon', '<f8'), (u'id', '<i8')])

Finally the contents of the sorted numpy array that is passed to the method

[20532 rows x 3 columns]
(numpy.record, [(u'index', '<i8'), (u'lat', '<f8'), (u'lon', '<f8'), (u'id', '<i8')])
array([(    0, 48.1327519, 11.566575 , 1576559984),
       (    1, 48.1383783, 11.5765354, 4785090864),
       ( 3199, 48.1196275, 11.5380289, 1079422305), ...,
       (23726, 48.1658095, 11.5900809, 3955898986),
       (23727, 48.1626348, 11.5999409, 1324415281),
       (23728, 48.1627603, 11.6000468, 1324415506)],
      dtype=(numpy.record, [(u'index', '<i8'), (u'lat', '<f8'), (u'lon', '<f8'), (u'id', '<i8')]))

This unfortunately returns the following error:

NumPyArrayToTable() RuntimeError: The value type is incompatible with the field type. [id]

This is rather interesting as I have used this method with dozens of other datasets without any problem ! Could someone please tell me what exactly isn't properly working out with the use of 'id' here?

1
osmNParray = np.array(sorted_osmNParray, np.dtype([('index', '<i8'), ('lat', '<f8'), ('lon', '<f8'), ('id', '<f8')]))

Changing the field type of 'id' to float fixed the issue!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.