2

Is it possible to run ogr2ogr or Python/Fiona in a serverless environment where each feature gets processed in parallel and where you don't have to specify the properties of a virtual machine upfront?

I'm looking for a tool that can convert a variety of geospatial data types to something cloud GIS solutions such as AWS Athena, Google Biqguery and EarthEngine (in the future, current upload types are very limited) can work with.

An example I can think of is leveraging the power of Apache Beam but I have not seen any I/O connector capable of reading geospatial data types and create parallel tasks.

Improve this question
3
  • Can you provide an example of the input and output? I'm not completely clear reading your question: Is the input a spatial database and you want to perform operations on each feature stored in there in parallel or do you want to perform tasks on a number of vector files, i.e. batch processing of files with serverless architecture - such as converting user uploaded content?
    – Kersten
    Sep 18, 2018 at 7:27
  • The latter: My use case is primarily ingesting a multitude of geospatial data types into these cloud services, performing some simple tasks during the process that cleans up the geometries, fixes issues (crossing 0180 meridian) etc.
    – RutgerH
    Sep 18, 2018 at 10:58
  • So you are looking at an ETL pipeline that could be constructed, i.e. on AWS, of the type: S3 bucket -> start process on new file -> process -> push to other service or bucket. Right? That's certainly doable with the likes of geolambda. The only issue I see is how do you determine your clean-up steps? If you have those formalized as a script the rest can be constructed relatively easy with queue processing - be it with AWS, Hadoop, Kubernetes or Beam.
    – Kersten
    Sep 18, 2018 at 12:05

1 Answer 1

Reset to default
1

This is an addon for Apache Beam that can load and process GIS data on Google Dataflow: https://github.com/GoogleCloudPlatform/dataflow-geobeam.

It includes I/O connectors for GeoTiff, Shapefiles, and Geodatabases.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.