7

I am a newbie in FME and is trying to accomplish a similar implementation like this: Merging edited data back into ArcSDE master geodatabase?

But we are still in the proof of concept stage.So we are just starting of with 2 non-spatial databases (master and updated) being populated by FME using a single shapefile. Initially both database are equal. Also, we added a FeatureID column using the Counter transformer in FME.

We made a couple of updates to the updated database so that we can try sync up the updates to the master database. We are using this sample as reference: http://fmepedia.safe.com/articles/Samples_and_Demos/Creating-a-Unique-identifier-using-CRCCalculator.

During the population of the master database. We added a "CRC" column which we set as a varchar(100). Before writing to the master database, there is a CRCCalculator added so that we can populate the CRC column.

Syncing process: - we have 2 readers which is set for the update db and master db. - the update db will pass through a CRCCalculator first and will then be passed to a FeatureMerger as a supplier. - the master db will pass through a AttributeRenamer for the CRC column and will then be passed to the same FeatureMerger as the requestor.

CRCCalculator parameters: - Attributes = all of the columns - Calculate CRC = Geometry and Attributes - CRC output attribute = CRC

I have added an inspector in the "Complete" output port of the FeatureMerger and noticed that all of the CRC(from update db) and CRC_Master(from master db) are not equal, even though there are no changes to those rows. So FME is thinking that all of the data have changed.

What am i doing wrong in the syncing process?

Here is a screenshot of how the syncing process looks in FME:

enter image description here

UPDATE - So i tried to test this again and added another CRCCalculator for the Master database. So now i can see 3 CRC columns in the FeatureMerger results. I noticed that the CRC value that was created in the translation matched and the CRC value that was saved in the Master database did not.

  • Now im wondering if the CRCCalculator changes something in every translation.

thanks.

Improve this question
5
  • 2
    provide information on your parameters in the crc calculator. in your screen shot I can see that 1444165 objects are being passed to the featuremerger but none from the masterdb. have you used an inspector on the masterdb?
    – Brad Nesom
    Apr 30, 2014 at 17:31
  • (Safe) FME have youtube video on Syncing MS SQL and Oracle youtube.com/watch?v=9qNC3eeOJzI
    – Mapperz ♦
    Apr 30, 2014 at 17:52
  • hi, i tried to removed the CRC column from the Master DB and tried to and another CRCCalculator to check if the is when i was saving the CRC value in the Master DB but they are still not the same. That is why the count was refreshed. It should be the same count. This is my CRC parameters: Attributes=all of the columns, Calculate CRC = Geometry and Attributes, CRC output attribute = CRC.
    – Juno
    Apr 30, 2014 at 17:59
  • @Juno please touch the edit button on your original question and add more information to that. not as comments thanks
    – Brad Nesom
    Apr 30, 2014 at 18:16
  • You say you are calculating the CRC against all attributes. There must be an attribute that is different in all cases. I would be suspicious of floating point attributes since precision discrepancies may come into play. I would suggest limiting the CRC'd attributes to just the ones you know are significant, and if this includes floating point attributes, to round/truncate them to a specific number of decimal places. Note: this also applies to geometries!
    – blah238
    Apr 30, 2014 at 18:38

2 Answers 2

Reset to default
1

I am going to try to answer your question, well, what I think is your question. What you have as a question is really a methodology for doing something, rather than stating what you're doing.

Here's what I think your situation is: You have a MASTER and an UPDATED dataset. You want to find out where there have been changes between MASTER and UPDATED and insert/update the MASTER dataset with the changes in the UPDATED dataset.

Here's how I'd implement that with FME:

  1. Get both datasets in (you have this, but note that there is something wrong with the MASTER on in your screenshot, since no features are being read
  2. Run them into a CHANGE DETECTOR transformer, with either all attributes selected, or just the ones that change
  3. Get the features from the Added and Deleted side and decide what to do with them! (Here you can use a featuremerger, or some SQL writers with appropriate settings.
Improve this answer
2
  • Hi alex, we opted to go with the CRCCalculator instead of the ChangeDetector because it is faster. The issue i am having is that the CRC value generated from the 2 databases are not the same even though they contain the same data.
    – Juno
    May 1, 2014 at 20:33
  • Hi @Juno well, if they are different in the CRCCalculator, then they're probably different! Why not try it with the Change Detector and get it working there, then optimise later if required?
    – Alex Leith
    May 1, 2014 at 22:50
0

Be sure the attrs used to calculate crc only include the attrs of change. Don't include FME format attrs like fme_dataset as the crc would change every time.

Improve this answer
2
  • Welcome to GIS SE! As a tip, your answer is hard to read due to your use of a contraction (attrs) that is not written out in full even once. For readability I recommend that such contractions are not used.
    – PolyGeo ♦
    May 1, 2014 at 8:19
  • hi, i'll check this out. i generally just selected all of the attributes from the feature when calculating for the CRC. Also, is it correct that i stored the CRC value as a varchar datatype?
    – Juno
    May 1, 2014 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.