5

My script takes separate 3D feature classes, calculates a field, appends to a target feature class, and then deletes the originals. This is a huge data processing task of over 4700 mass point feature classes appending to a single one. I know it was a big mistake to put so many feature classes into SDE, but given they're in there now, I've got to deal with them there (yes, the sheer fc count is bringing SDE performance to its knees--hence my urgency).

Early on it worked great, in that the merged result showed valid z-values. I was able to build a terrain and it looked great. I'd been running this in the background for weeks, and have just noticed z-values of 0 in the output, where the originals were not 0! Oh no!!! I'm certainly glad I still have the file-based originals!

Pseudocode:

An in_memory feature class is created from a z-aware template

For each fc in the feature dataset:

It is appended to the empty in_memory fc

The TILE_ID field is calculated using a portion of the name of the source fc

It is then appended to the target SDE fc

The in_memory feature class is emptied

Each processed fc is then deleted

When I initially posted, I was curious if it was my use of in_memory that caused this trouble. I had not initially been using it, and only switched to it because performance was far too slow otherwise, and WAY faster in_memory. To find where trouble started, I ran the Add XY Coordinates tool, which proved the trouble started with in_memory use. At that time I had also switched from CalculateField_management to da.UpdateCursor, so that's likely the bigger reason for the performance enhancement. I will now change my code to avoid in_memory.

Improve this question
7
  • I've never had a problem retaining z values using in_memory (although this was in model builder, not in straight python). One thing that comes to mind, if you haven't already thought of it, is your z-aware template field the same type as the z field that it is being populated with? Seeing as how this was working for you before, I doubt this is the problem, but it's worth a shot. Also, can you try running the process on a sample set with proper results?
    – eric
    Commented Mar 8, 2013 at 18:35
  • Thanks for the reply, @eric. I am suspicious of the in_memory use only because Esri is very non-specific in what they describe being supported in-memory (i.e. no extended geodatabase parameters...). The z-aware template field is simply the Shape field, because these are 3D features. The rows in the target fc all have Shape as Point ZM, as they do in the source FCs. So I'm not sure how the template could be different. I will try on Monday to test on a small sample set and see what happens. By then I'll have the results of the Add XY Coordinates tool to see where the 0 z-values began. Thanks
    – Justin
    Commented Mar 8, 2013 at 21:27
  • It would not come as a total surprise to me if processing in_memory does not provide access to z values because I know it does not calculate Shape_Area while in memory - see here - you may or may not be encountering (something like) NIM063795
    – PolyGeo ♦
    Commented Mar 9, 2013 at 6:31
  • 1
    in_memory usage was not the culprit. It was instead my neglect to include "ENABLED" for the has_z parameter of CreateFeatureclass_management(). Simply depending on the z-enabled characteristic of the template fc is insufficient, as I've learned.
    – Justin
    Commented Mar 21, 2013 at 19:02
  • 2
    @Justin, can you post this as an answer for other users who may encounter this same problem?
    – scw
    Commented Jul 5, 2013 at 8:09

1 Answer 1

Reset to default
2

It would not come as a total surprise to me if processing in_memory does not provide access to z values because I know it does not calculate Shape_Area while in memory - see In-Memory datasets and Shape_Area field at GeoNet - you may or may not be encountering (something like) NIM063795:

The Dissolve tool creates empty Shape_Length and Shape_Area fields in an in-memory output layer during background geoprocessing.

The status of that since 11 Jun 2020 has been “In product plan”.

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.