7

Am I correct that if my model has a model-only tool (Calculate Value), and then I run the model as a batched process, it will not run the Calculate Value expression?

I wrote a model to upload, define, project (and clip) XY data from excel files. I had thought to use iterate tables, but it wasn't working. (I assume due to having to choose which worksheet within xlxs). So I thought to do a batch, but this seems to leave that annoying $ that arc appends to table names (and is an invalid character in most tools...)

My guess is that the Calculate value that removes my $ is not being run in the batch. Ideas? Confirmations?

2 Answers 2

6

The Model Only Tool called Calculate Value can be run in Batch.

To prove this to myself I did the following:

  1. Create a test feature class, add one feature and an empty field to it - I used Short Integer
  2. Write first tiny model with Iterator called For and Copy Features and run it to create five copies of the test feature class called test1, test2, ..., test5
  3. Write second tiny model that uses Calculate Value tool to do perhaps 12 + 17 and then feed that output_value to a Calculate Field tool that uses it to populate the field added in 1.
  4. Run the second model in Batch with the five feature classes created in 2.

Result is the number 29 in the field for each of the five feature classes so the Calculate Value tool was run in Batch.


The "proof" above was poorly designed because only proves that the Calculate Value tool run in Batch gives the right result when Calculate Value is not intended to provide a different answer for each row in the Batch Grid.

Consequently, I've now run another set of tests using this model.

enter image description here

The model takes a feature class (which must have an integer as the 5th character in its name e.g. test1, test2) as input, uses Calculate Value to grab the 5th character and return it as an integer so that Calculate Field can add that to the value of a field called Count (always has value of 1) to get a value written into another field called NewCount. Details of dialog settings to accomplish this are below.

enter image description here enter image description here enter image description here

The tool produces the expected result when run using Open or as a Batch with one row.

However, when the tool is run in batch using two input feature classes of test1 and test2 you would expect the NewCount fields to be updated to 2 (=1+1) and 3 (=1+2) respectively. Instead, what happens is that if test1 is in the first row of the batch grid then 2 gets written to both and if test2 is first then 3 gets written to both.

From the following it is obvious that while every other tool gets run once per row of the batch grid, the Calculate Value tool is only getting run once in total.

Executing: BatchGridBug C:\Support\MBarretts20\test.gdb\test1
Start Time: Fri Oct 26 15:17:26 2012
Executing (Parse Path): ParsePath C:\Support\MBarretts20\test.gdb\test1 NAME
Start Time: Fri Oct 26 15:17:26 2012
Executing (Parse Path): ParsePath C:\Support\MBarretts20\test.gdb\test2 NAME
Succeeded at Fri Oct 26 15:17:26 2012 (Elapsed Time: 0.00 seconds)
Executing (Calculate Value): CalculateValue UpdateValue("test1") "def UpdateValue(Value):\n  Value = Value\[4:\]\n  return int(Value)" Variant
Start Time: Fri Oct 26 15:17:26 2012
Value = 1
Succeeded at Fri Oct 26 15:17:26 2012 (Elapsed Time: 0.00 seconds)
Executing (Calculate Field): CalculateField C:\Support\MBarretts20\test.gdb\test1 NewCount "!Count! + 1" PYTHON #
Start Time: Fri Oct 26 15:17:26 2012
Executing (Calculate Field): CalculateField C:\Support\MBarretts20\test.gdb\test2 NewCount "!Count! + 1" PYTHON #
Succeeded at Fri Oct 26 15:17:27 2012 (Elapsed Time: 1.00 seconds)
Succeeded at Fri Oct 26 15:17:27 2012 (Elapsed Time: 1.00 seconds)

Feedback from Redlands via local support was "We have tested the case in house and observed that ... Issue is reproducible in house ... this is a known limit with Calculate Value tool ... NIM053898: "Calculate Value tool stops after first expression variable in batch mode (or when defined as a list of values)" Status of bug: Rejected as designed Reasons for rejection: Known limit. Use Iterate multivalue in 10." I'm disappointed that this is being treated as a Known Limit so have asked whether that Status can at least be documented under Calculate Value and Using Batch Grids.

13
  • Interesting- Thank you for the test run. Must be another issue then. My calculate value is to strip the $ from an xy event layer, but I get an "invalid character' error, that the $ hasn't been removed when run as a batch process. Commented Oct 17, 2012 at 17:27
  • If you can describe your model in more detail, perhaps I can run a test closer to your precise situation to see if I can replicate your result.
    – PolyGeo
    Commented Oct 17, 2012 at 21:17
  • #PolyGeo thanks. 1. Upload xy table 2. Parse Name to get (Name) (removing.dbf) 3. Calculate Value to remove $ from Name 4. Define 5. Copy Features, creating %table_name%.shp 6. Project, creating %table_name%_pj.shp 7. *a different tool is a copy of the above, but adds the clip tool at the end to upload, project and clip data to a study area, using the original xy table name as the file name. The $ that arc adds to the $ complicates things, b/c most tools count it as invalid character, hence Calc Value tool- when this is run as batch, the $ isn't removed, and I get inv. char error. Commented Oct 22, 2012 at 14:43
  • What tools are you using to do steps 1, 2 & 4? I'm keen not to guess.
    – PolyGeo
    Commented Oct 22, 2012 at 21:46
  • 1
    Feedback from Redlands via local support was "We have tested the case in house and observed that ... Issue is reproducible in house ... this is a known limit with Calculate Value tool ... NIM053898: "Calculate Value tool stops after first expression variable in batch mode (or when defined as a list of values)" Status of bug: Rejected as designed Reasons for rejection: Known limit. Use Iterate multivalue in 10." I'm disappointed that this is being treated as a Known Limit so have asked whether that Status can at least be documented under Calculate Value and Using Batch Grids.
    – PolyGeo
    Commented Nov 9, 2012 at 2:52
2

This sounds like the issue described in this thread: Force batch to run processes sequentially

Basically, when you run a model in batch mode, it runs the first step in the model once for each row in the batch dialog, goes to the next step and runs it for each row, etc.

So it is in a way running the processes in parallel instead of in sequence, which can have all sorts of stupid side effects and I have seen no explanation from ESRI as to why it behaves this way.

I've had to wrap many of my models in Python scripts to work around this annoying behavior.

Update: Here's an even simpler test (doesn't use Calculate Value).

  1. Open up ArcMap and using the Catalog window, create a table with a Long Integer column and populate it with 1 row whose value for the column is 1. Count = 1
  2. Create a model in ArcMap that looks like this: Model
    • Set up the Get Field Value tool: Get Field Value
    • Set up the Calculate Field tool: enter image description here
  3. Start editing in ArcMap (so you can see the changes without having to Reload Cache)
  4. Run the model normally (not batch mode)
    • Notice the COUNT value increments by 1 each time you run it.
  5. Run the model through batch mode with say 10 entries.
    • Notice the COUNT value only increments by 1 no matter how many entries you have.

What's happening is during the first iteration (batch grid row), it runs the Get Field Value Tool. Then it runs it again. And again. For each row in the batch grid it gets the SAME field value over and over.

When it's done with that it goes onto the Calculate Field tool and runs that for the first batch grid row, then runs it again for the next row, etc., with that same field value, so it never increments.

So in the text output you see something like this:

Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Get Field Value): GetFieldValue batch_count COUNT Long 0
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #
Executing (Calculate Field): CalculateField batch_count COUNT "2 + 1" VB #

Silliness.

8
  • That does indeed sound like it may be the same issue. Running in parallel would be great if it is multi-processing and always worked!
    – PolyGeo
    Commented Oct 26, 2012 at 7:19
  • 1
    Yeah it's odd; the Calculate Value tool, if present, only seems to get run once. Even if not present, however, the processes aren't run in sequence before moving onto the next row in the batch grid.
    – blah238
    Commented Oct 26, 2012 at 7:40
  • Added an example without using Calculate Value to illustrate the "parallel processing" behavior.
    – blah238
    Commented Oct 26, 2012 at 8:13
  • This is in 10.1, BTW, and I don't remember seeing anything about it in the SP1 notes.
    – blah238
    Commented Oct 26, 2012 at 8:15
  • 10.1 for me too, and I've not uncovered a bug number for it yet, working with local support to try to, so I don't hold out much hope for SP1. Hopefully it can get addressed by 10.2 (if that's on way) at latest or preferably SP2.
    – PolyGeo
    Commented Oct 26, 2012 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.