.NET console application running super slow all of the sudden

Question

I have a standalone .NET console application that (through ArcObjects) processes DWG files, extract coordinates, and pushes them to SQL Server. It's smoking fast and can process a DWG in a matter of seconds. It's been running fine in production for quite a while now (over a year). All of the sudden last Wednesday it started running dead slow (40x longer). My server had a major power failure on the Monday night before this all started going wrong. When the server came back up, iSCSI drives were missing, all kinds of stuff was jacked up and it look 4-5 guys in infrastructure hours to get it all back up. I just found out about my process running slow today.

My question is this: Does anyone have any idea what I could look for as the culprit in something like this. The process runs, just really, really sloooooow. Like its CPU starved. If it was permissions, it would just fail. I don't expect to get an answer from my infrastructure team. Any insight is greatly appreciated.

EDIT: We just realized that ArcGIS Server hasn't written to any log files since this hard crash. The plot thickens...

Another edit: The drive this executable resides on is connected through iSCSI. I don't know a whole lot about iSCSI, but we are wondering if this is a communications issue between the server blade and iSCSI (which I think communicates via a fiber card). I have moved the .NET application over to one of the physical drives on the blade and we'll see what happens...

Answer 1

So, we can rule out memory leaks in the code, as it worked fine for a year. It sounds def hardware related.

I am assuming your console application is isolated from ArcGIS Server? - Although assuming that it is up and running, I would imagine the inability to write log files would be permissions related, try re-running the post-installs and see if they fail anywhere.

Where is SQL Server in relation to the Console application/DWG files?

• Assuming they are across multiple servers, if you ping from one to another, what sort of delay are you getting? (Rule out network latency)
• Whats the CPU load like on the server where the process is running?
• Can you ping the database with Telnet ok? (from server machine)

May also be worth installing tools like WireShark, NetMon, NetStat, to find out which IP is causing the delay.

Answer 2

Have you run perfmon on the server during a session of this Console app running? I would create performance counters for

1. CPU Usage
2. Disk Read/Writes
3. Memory Usage

This will help narrow down the area that the application is being resource starved in.

Having an iSCSI disk means that your drive is on a SAN, which is a shared resource. If other services with storage on the SAN are gobbling up all available IO resources on the SAN your app may have performance issues.

Depending on the nature of the power failure, if the SAN failed at the same time, the SAN could be re-building the array internally, and that could give the appearance of poor performance.