Return to Question

My script takes separate 3D feature classes, calculates a field, appends to a target feature class, and then deletes the originals. This is a huge data processing task of over 4700 mass point feature classes appending to a single one. I know it was a big mistake to put so many feature classes into SDE, but given they're in there now, I've got to deal with them there (yes, the sheer fc count is bringing SDE performance to its knees--hence my urgency).

Early on it worked great, in that the merged result showed valid z-values. I was able to build a terrain and it looked great. I'd been running this in the background for weeks, and have just noticed z-values of 0 in the output, where the originals were not 0! Oh no!!! I'm certainly glad I still have the file-based originals!

Pseudocode:

An in_memory feature class is created from a z-aware template

For each fc in the feature dataset:

It is appended to the empty in_memory fc

The TILE_ID field is calculated using a portion of the name of the source fc

It is then appended to the target SDE fc

The in_memory feature class is emptied

Each processed fc is then deleted

EDIT: When I initially posted, I was curious if it was my use of in_memory that caused this trouble. I had not initially been using it, and only switched to it because performance was far too slow otherwise, and WAY faster in_memory. To find where trouble started, I ran the Add XY Coordinates tool, which proved the trouble started with in_memory use. At that time I had also switched from CalculateField_management to da.UpdateCursor, so that's likely the bigger reason for the performance enhancement. I will now change my code to avoid in_memory.

Thank you to eric and PolyGeo.

My script takes separate 3D feature classes, calculates a field, appends to a target feature class, and then deletes the originals. This is a huge data processing task of over 4700 mass point feature classes appending to a single one. I know it was a big mistake to put so many feature classes into SDE, but given they're in there now, I've got to deal with them there (yes, the sheer fc count is bringing SDE performance to its knees--hence my urgency).

Early on it worked great, in that the merged result showed valid z-values. I was able to build a terrain and it looked great. I'd been running this in the background for weeks, and have just noticed z-values of 0 in the output, where the originals were not 0! Oh no!!! I'm certainly glad I still have the file-based originals!

Pseudocode:

An in_memory feature class is created from a z-aware template

For each fc in the feature dataset:

It is appended to the empty in_memory fc

The TILE_ID field is calculated using a portion of the name of the source fc

It is then appended to the target SDE fc

The in_memory feature class is emptied

Each processed fc is then deleted

When I initially posted, I was curious if it was my use of in_memory that caused this trouble. I had not initially been using it, and only switched to it because performance was far too slow otherwise, and WAY faster in_memory. To find where trouble started, I ran the Add XY Coordinates tool, which proved the trouble started with in_memory use. At that time I had also switched from CalculateField_management to da.UpdateCursor, so that's likely the bigger reason for the performance enhancement. I will now change my code to avoid in_memory.

My script takes separate 3D feature classes, calculates a field, appends to a target feature class, and then deletes the originals. This is a huge data processing task of over 4700 mass point feature classes appending to a single one. I know it was a big mistake to put so many feature classes into SDE, but given they're in there now, I've got to deal with them there (yes, the sheer fc count is bringing SDE performance to its knees--hence my urgency).

Early on it worked great, in that the merged result showed valid z-values. I was able to build a terrain and it looked great. I'd been running this in the background for weeks, and have just today noticed z-values of 0 in the output, where the originals were not 0! Oh no!!! I'm certainly glad I still have the file-based originals!

Pseudocode: An in_memory feature class is created from a z-aware template

For each fc in the feature dataset: It is appended to the empty in_memory fc The TILE_ID field is calculated using a portion of the name of the source fc It is then appended to the target SDE fc The in_memory feature class is emptied

Each processed fc is then deleted

I am curious if it was my use of in_memory that's caused this trouble, because I was not using it initially. I only switched to it because performance was far too slow, but WAY faster in_memory.

  To find where trouble started, I'm presently running Add XY Coordinates tool. It looks like it'll take a few hours to finish :(

Will someone please let me know if in_memory is the cause of my trouble, or what else it might be? I can post the code if it helps.

My script takes separate 3D feature classes, calculates a field, appends to a target feature class, and then deletes the originals. This is a huge data processing task of over 4700 mass point feature classes appending to a single one. I know it was a big mistake to put so many feature classes into SDE, but given they're in there now, I've got to deal with them there (yes, the sheer fc count is bringing SDE performance to its knees--hence my urgency).

Early on it worked great, in that the merged result showed valid z-values. I was able to build a terrain and it looked great. I'd been running this in the background for weeks, and have just noticed z-values of 0 in the output, where the originals were not 0! Oh no!!! I'm certainly glad I still have the file-based originals!

Pseudocode:

An in_memory feature class is created from a z-aware template

For each fc in the feature dataset:

It is appended to the empty in_memory fc

The TILE_ID field is calculated using a portion of the name of the source fc

It is then appended to the target SDE fc

The in_memory feature class is emptied

Each processed fc is then deleted

EDIT: When I initially posted, I was curious if it was my use of in_memory that caused this trouble. I had not initially been using it, and only switched to it because performance was far too slow otherwise, and WAY faster in_memory. To find where trouble started, I ran the Add XY Coordinates tool, which proved the trouble started with in_memory use. At that time I had also switched from CalculateField_management to da.UpdateCursor, so that's likely the bigger reason for the performance enhancement. I will now change my code to avoid in_memory.

Thank you to eric and PolyGeo.