Prevent arcpy.Append from crashing if data is too long for field

Question

We're appending a bunch of information from somebody else's database into ours. Much of it is text information, and the origin database doesn't appear to have much limit on the length of those strings. Ours was set up largely with default limits of 50 on the text fields.

So naturally it will sometimes fail to import, throwing an error like:

ExecuteError: ERROR 001156: Failed on input OID 69838, could not write value 'James Island - Subdivision not indicated, no marker' to output field SUBDIVISION
Failed to execute (Append).

SUBDIVISION can take 50 characters, and ArcPy just tried to push 51 characters, and so it failed and crashed.

Short of increasing the length of all our database's text fields (which seems like it's going to work great until I encounter the next super-long string that exceeds the newly expanded limit), is there an elegant way to handle such cases and still get the data to append? In other words, is there a way to tell the Append function I'd like it to just truncate strings that don't fit (ideally while throwing a message letting me know that happened)?

Answer 1

When using the append tool through the GUI, if you use the "NO_TEST" option for schema type, you can use the Field map control. Expand the field map tree for the text field you're having issues with, and you'll see a node for every input dataset. If you right click on each node, and click format, you'll get options to specify the start and end position of what you're going to read out of that input field. If your target dataset will only accept 50 characters, set the end position to 50 characters, and you should be fine.

You're using arcpy though. You could throw the append into a model, and export the model to a python script, but the field mapping argument will wind up a big long string. If your workflow is static, you could manually edit the string, and that should work. If the data you're appending is a little more dynamic, you may have to programmatically construct your field mapping. See the arcgis help topics for FieldMapping and FieldMap. FieldMap has a method call setEndTextPosition which would be the equivalent to what I described in the GUI.

I'm not sure what version of arcgis you're using, but my copy of arcgis is a little old at 10.0 thanks to my companies IT, so the help topics I linked are for 10.0. Make sure to use the help topic for your version.

Answer 2

I am not too familiar with arcpy.append but I guess you could iterate through your dataset before using append and just truncate any entry that is too long.

You would have to use an UpdateCursor to iterate through your dataset, and then you would check the length of each entry using len(). If the length exceeds 50 characters, then you would chop it.

The code would look like this:

yourFC = "pathToYourFeatureClass"
fields = ('toBeChecked')

with arcpy.da.UpdateCursor(yourFC, fields) as cursor:
    for row in cursor:
        if len(row[0]) > 50:
            longEntry = row[0]
            row[0] = longEntry[:50]
            cursor.updateRow(row)

(I am unable to test this right now and writing from memory; you might have to do small adjustments; I can test it tomorrow)

This does not directly change anything in your append code, but it would surely keep it from failing. Yes, it might take a little bit to run depending on how large your dataset is, but in the end it will take less time than getting annoyed at a crashed tool :-)

Should you opt for this, remember that you should call your cursor from the Data Access module, such as arcpy.da.SearchCursor, as it is much faster than the original cursor!

EDIT:
I just had a chance to test the code snippet I posted last night and it works fine.

To illustrate it, a table like this:

Would turn into this:

Answer 3

This relates to @BritishSteel's answer and their code.

Using a cursor and that verbose code seems like a bit of overkill for this.

If you merely want to truncate a field to 50 characters for all records/features, just use the CalculateField tool (or arcpy.CalculateField_Management in a script).

This ends up being a one liner script! Something like...

arcpy.CalculateField_management(featureClass, "FIELD_NAME", "'!FIELD_NAME![:50]'", "PYTHON_9.3")

But @MWrenn's answer about using the Append tool's built in format option to read only the first few characters of the field is even better. You don't have to modify your source data all, that way.