5

We are developing a system to disseminate information about roadkills. The source data comes from three different databases. We have set up a FME project which pulls geojson from the source databases, joins them according too a set of business rules and then exports them to a front-end database.

We are currently exporting our data to a PostGIS database. FME does this very nicely. Now my colleague suggests we use Elasticsearch since this will make the presentation of our data in leaflet less complicated. Elasticsearch evidently indexes geojson on the fly.

We are using FME version 15.0. How can I write the data to Elasticsearch from FME? Which writer should I use?

  • 1
    Postgres as of 9.1 has supported JSON as a native datatype and as of 9.3 there is JSONB, which allows for proper indexing of JSON as well. Could you explain how you want to present data in Leaflet that makes Elastic Search preferable to Postgres? – John Powell Apr 20 '15 at 22:22
  • 2
    You can't get data out of Postgres without an API. That's why Elasticsearch would be preference: a built in web-accessible API. – Alex Leith Apr 20 '15 at 23:02
  • 2
    @alexgleith. Getting data out of Postgres in JSON(B) format could be done with less than 5 lines of something like nodejs, which may be less work than moving everything to ElasticSearch -- an API sure, but hardly heavy weight. ElasticSearch is great, but designed to solve different issues than easily getting JSON to Leaflet, I would argue. This is not clear from the question. – John Powell Apr 21 '15 at 13:35
  • See tweet twitter.com/FMEEvangelist/status/590518826225246208 (PR61198) look out at the end of September 2015 when 2016 betas become nightly downloads.safe.com/fme/beta/whatsnew_sp.txt – Mapperz Apr 21 '15 at 15:48
  • 1
    What's funny here is that we did a webinar once that used roadkill data from the city of Vancouver as an example - and got complaints that it was in bad taste! Even though it's real data that depicts a real issue. So whatever you do with your data - make sure you do it tastefully ;-) – Mark Ireland Apr 22 '15 at 15:50
3

GDAL/OGR supports Elasticsearch, which means that FME could support it reasonably easily.

I reckon you might like to hit up the FME people on Twitter and put in a request. Try Mark Ireland @FMEEvangelist or Dale Lutz @DaleAtSafe and Don Murray @DonAtSafe.

But anyway, you could use a FME script to prepare your data, then write it to a temporary file, and then use Python to write, either directly (tricky but solid), or using OGR (easy, less flexible), to Elasticsearch. I'd implement that, and then contact the FME guys and tell them to hurry up and implement an Elasticsearch writer!

3

Give this recent talk by Paul Ramsey a view:

http://blog.cleverelephant.ca/2015/03/magical-postgis.html

He suggests that the PostgreSQL Full Text search make ElasticSearch unnecessary...

Something to consider anyway...

3

Yes, I think we could implement this in FME pretty easily. I've filed an enhancement request with our developers and we'll see what they say. If you contact our support team (http://safe.com/support) and let them know you are interested then they can add your contact details to the request. The reference number is PR#61198

Also, do please add it to the Trello board that Mapperz mentions, as the more users who upvote it on there, the more likely it is that we will implement it.

In the meantime, I took a quick look and it says ElasticSearch is accessible through an API in JSON format. So why not write the data as JSON and use the API to upload it?

I can think of a few ways:

  • Convert your data to JSON with a transformer like the JSONTemplater and then upload it with a HTTPCaller transformer.

  • Write the data with a JSON Writer, then put the HTTPCaller in a second workspace.

  • Write the data with a JSON Writer, then use a Python shutdown script to upload the data through a call to their API

  • Message sent to the support team. Also signed up on Trello, but I do not have the sufficient access to make suggestions there yet. – ragnvald Apr 21 '15 at 18:41
  • 1
    I've never thought about using FME to upload to an API using JSON! Very interesting... – Alex Leith Apr 21 '15 at 22:49
  • @ragnvald: If you add a comment to the "Other Items" list in Trello I think someone in the dev team with permission will upgrade it to a full suggestion. – Mark Ireland Apr 22 '15 at 15:41
  • Support for this planned for FME2016.1 – Mark Ireland Jan 25 '16 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.