Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I have an FME workspace that takes a set of street centerlines and converts it into address points. I have been working my way across the state of Missouri with this translator one county at a time, since each county can take up to an hour to process. I select a county at the start of the translator as a choice parameter.

My problem is that I keep having to double check to make sure of what counties I have processed so I do not process a county twice. When I have a file of several million address points, I do not want to be removing duplicates.

What I would like to have FME do is take the county I have selected, and check against a shapefile of counties to see if the county has been processed yet. I have a flag attribute, "COMPLETE" that is set to 0 if the county has not been processed and 1 if it has.

The county shapefile has nothing to do with any other readers or writers in the translator.

Edit: To clarify, the counties are not being processed at all by the translator, only the centerlines.

Since there are over 2 million centerlines, I do not want to read the centerlines at all (much less maintain an attribute on the centerlines) if the county has already been processed.

Edit: Here is the solution. I moved up the st_county10 reader (the one in the image) to the top of the reader list. This executes before anything else. By using the feature holder, I avoid writing over the st_county10 shapefile if the translation is terminated.

One last piece, since I had a python shutdown script, I had to add this line:

import pyfme
if FME_Status == True:

To the front of the shutdown script to stop execution of the shutdown script too.

If COMPLETE=0, translation proceeds, otherwise it goes to the Terminator

share|improve this question
    
After Chad Coopers post - you can use the Tester to move the 0's into the WorspaceRunner fmepedia.safe.com/articles/Samples_and_Demos/WorkspaceRunner and control the number of processes you have running at once. So larger processes can be done over night. –  Mapperz Nov 15 '11 at 16:24
    
The centerlines are all in one statewide shapefile. Basically for each pass, I select all the centerlines in the county being processed, then process just those centerlines. I will not be processing all of the counties; it has not been determined which ones I will process. –  blord-castillo Nov 15 '11 at 17:24
    
That looks a good solution to me. Another way would be the FeatureReader transformer to query the dataset, rather than reading it all. That would be the better solution for a database format, but for Shape it won't make any difference I think because it can't be queried as well. –  Mark Ireland Nov 15 '11 at 19:29
add comment

1 Answer

up vote 1 down vote accepted

If I understand the problem correctly, I think you want to incorporate your county shapefile as a Reader and then execute a Tester transformer against that, then route your job accordingly. Use your published parameter of state name in the tester as one of the test clauses, along with the processed yes/no flag. Here's a quickie I threw together:

enter image description here

This looks for features with a Status = 'Other' in this cities shapefile, then routes according to pass/fail. Now, I think FME is sensitive to the order you put your Readers into the workspace, so if you put this reader in after you already have some others in, the others will get executed first. Try it, you might have to make a new workspace, put this logic in first, then put your other readers in after that to get the read order correct. If you want to kill the job if the query fails, there may be a way to do that with pyfme, or you could route the FAILED feature (when status = Completed already) to a Terminator transformer.

share|improve this answer
    
The counties are not the features being processed. The road centerlines are the features being processed. But I will look at the terminator transformer, that might be exactly what I need. –  blord-castillo Nov 15 '11 at 17:20
    
@blord-castillo - Yes, I know. What I'm suggesting is an additional Reader that reads the county shapefile, runs it through a Tester, then, based on those results (centerlines already completed or not) if you still need to process the road centerlines for that county, run that part of the workflow next. –  Chad Cooper Nov 15 '11 at 17:26
    
Terminator was exactly what I wanted. I simply had the county shapefile read first and put together a simple testing series before executing the other 45 steps of the process :) –  blord-castillo Nov 15 '11 at 17:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.