3

I have a database application that I connect to using SQLAlchemy. In the database, I store coordinates in the fields x_coord and y_coord, with a third field spatial_reference_code that stores the EPSG code of the coordinate system that x_coord and y_coord are in for the record.

I've written a generic function that takes a SQLAlchemy query object with any preexisting filters on the data table and exports it to a feature class. The function converts the SQLAlchemy object to a pandas data frame, then uses the pandas data frame (after using data_frame.as_matrix()) as the data parameter to arcpy.da.NumPyArrayToFeatureClass. When running this code, it fails in NumPyArrayToFeatureClass with the error "TypeError: narray.fields require"

Below is a minimum reproducible example, adapted from Esri's sample code, that generates the problem:

import os
import arcpy
import numpy
import pandas

out_fc = os.path.join(os.getcwd(), "test.shp")

# Create a numpy array with an id field, and coordinate fields

array = numpy.array([(1, -116.809, 34.271),
                     (2, -114.241, 33.684)],
                    numpy.dtype([('idfield',numpy.int32),('x_coord', '<f8'), ('y_coord', '<f8')]))

# converting to pandas data frame to simulate conditions in my actual code - this causes the failure
df = pandas.DataFrame(array)

# Define a spatial reference for the output feature class
sr = arcpy.SpatialReference(4269)

# Export the numpy array to a feature class using the X and Y fields to
#  represent the output point feature
arcpy.da.NumPyArrayToFeatureClass(df.as_matrix(), out_fc, ['x_coord', 'y_coord'], sr)

My code does not start with a numpy array and convert it to a data frame before converting it back, but the example does that to simulate the conditions of my code (where the data is retrieved from SQLAlchemy into a pandas data frame). If I run that sample code and pass "array" directly to NumPyArrayToFeatureClass, it works - but when I convert it to a pandas data frame and then back using df.as_matrix() - it fails.

For machine specs, this code is running in the Python interpreter provided by ArcGIS Pro 1.3 (Python 3.4.4 - in a conda environment I created from the main ArcGIS Pro environment). Pandas is version 0.17.1, numpy is version 1.9.3, and SQLAlchemy is version 1.1.2. Running on Windows 10 (10.0.14393).

I see one other reference to this issue over on StackOverflow, but that post does not resolve my problem.

Has anyone worked with this arcpy.NumPyArrayToFeatureClass and can help me debug this issue?

  • You have tags for Desktop and Pro. Which architecture and ArcPy are you using? Also, as it stands, I don't think the code that you appear to have copy/pasted from your actual code, will qualify as a code snippet. – PolyGeo Dec 2 '16 at 3:41
  • I mentioned ArcGIS Pro 1.3 in the post but tagged Desktop 10.4 as well since we're targeting that platform too and it should provide an equivalent environment - sorry if that creates confusion. I'll see if I can further reduce the snippets. I'd included more to make it runnable, but I'll leave out some extra checks, etc. – nicksan Dec 2 '16 at 3:47
  • OK - reduced code snippets - thanks for the feedback! – nicksan Dec 2 '16 at 3:54
  • That code still would not qualify as a code snippet that I would volunteer my time looking into because there are far too many unknowns that I would have to guess or waste time asking about. Others may or may not be keen to help. I think you need to focus this question on the Pro or the Desktop flavour of ArcPy. Once you have one fixed you can then see if you need to ask more about the other. For example arcpy.mapping is only available for Desktop and arcpy.mp is only available for Pro. Similar issues arise between the Extent and Camera objects. – PolyGeo Dec 2 '16 at 4:02
  • OK - thanks for the feedback - I made a minimum reproducible example and rewrote the post around it. – nicksan Dec 2 '16 at 4:33
3

The reason it doesn't work for you is because the df.as_matrix() returns an array with dtypes you had missing.

df.as_matrix()
array([[   1.   , -116.809,   34.271],
       [   2.   , -114.241,   33.684]])

Compare with array:

array([(1, -116.809, 34.271), (2, -114.241, 33.684)], 
      dtype=[('idfield', '<i4'), ('x_coord', '<f8'), ('y_coord', '<f8')])

You can use df.to_records(index=False) to create an array of numpy.records. The dtypes will be preserved:

df.to_records(index=False)
rec.array([(1, -116.809, 34.271), (2, -114.241, 33.684)], 
      dtype=[('idfield', '<i4'), ('x_coord', '<f8'), ('y_coord', '<f8')])

The complete code:

import sys
import os
import arcpy
import numpy
import pandas

print(sys.version,'\n',pandas.__version__,'\n',numpy.version.version)

out_fc = os.path.join(os.getcwd(), "test_out.shp")

# Create a numpy array with an id field, and coordinate fields

array = numpy.array([(1, -116.809, 34.271),(2, -114.241, 33.684)],
                    numpy.dtype([('idfield',numpy.int),
                                 ('x_coord', '<f8'), 
                                 ('y_coord', '<f8')
                                ]))

# converting to pandas data frame to simulate conditions in my actual code - this causes the failure
df = pandas.DataFrame(array)

# Define a spatial reference for the output feature class
sr = arcpy.SpatialReference(4269)

records = df.to_records(index=False)
# Export the numpy array to a feature class using the X and Y fields to
#  represent the output point feature
arcpy.da.NumPyArrayToFeatureClass(records, out_fc, ['x_coord', 'y_coord'], sr)

print([f.name for f in arcpy.ListFields(out_fc)])
for row in arcpy.da.SearchCursor(out_fc,"*"):
    print(row)

Out:

3.4.5 |Continuum Analytics, Inc.| (default, Jul  5 2016, 14:53:07) [MSC v.1600 64 bit (AMD64)] 
 0.17.1 
 1.9.3
['FID', 'Shape', 'idfield']
(0, (-116.809, 34.271), 1)
(1, (-114.241, 33.684), 2)
  • 3
    Thanks - this completed the picture for me - as a note to anyone else coming from a pandas data frame, I had to recast types not allowed in ArcGIS before using arcpy.da.NumPyArrayToFeatureClass - check the data types Esri allows here: pro.arcgis.com/en/pro-app/arcpy/get-started/… – nicksan Dec 2 '16 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.