Python, OGR and Mapnik: Possibility of passing existing ogr.Open() datasource into Mapnik without using mapnik.Ogr()?

Question

I'm trying to modify create_utfgrids.py to use an OGR VRT file, rather than shapefiles, and while the OGR stuff in this code responds to the VRT, the Mapnik stuff is not. It's failing with..

ERROR 1: Failed to open datasource `PG:host=127.0.0.1 user=postgres dbname=gis password=passw0rd tables=parcels_20140829_pmerc'.

I believe the VRT is fine, because it not only works with ogrinfo directly, but as stated above, the OGR-related code in the Python file also works.

While I'm pretty confident the VRT isn't the issue, for those who might ask, the VRT I tried looks like this. I know there's some redundant stuff in there, but that's because I thought maybe Mapnik was looking for something specific, so I was adding any node I could think of which might be relevant:

<OGRVRTDataSource>
  <OGRVRTLayer name="parcels_20140829_pmerc">
    <SrcDataSource>PG:host=127.0.0.1 user=postgres dbname=gis password=passw0rd tables=parcels_20140829_pmerc</SrcDataSource>
    <SrcLayer>parcels_20140829_pmerc</SrcLayer>
    <LayerSRS>EPSG:3857</LayerSRS>
    <SrcSQL>SELECT ST_AsBinary(wkb_geometry) as geomm, tms, owner_name, the_wkt FROM parcels_20140829_pmerc</SrcSQL>
    <GeometryField encoding="WKB" field="geomm"></GeometryField>
    <Field name="tms" src="tms" type="String"/>
    <Field name="owner_name" src="owner_name" type="String"/>
    <Field name="the_wkt" src="the_wkt" type="String"/>
  </OGRVRTLayer>
</OGRVRTDataSource>

I eventually got past the problem by passing in an OGR PostgreSQL connection string, parsing over that, and then pushing the values into a mapnik.PostGIS() constructor. Sure, this solves my immediate problem, but it's not quite as generic/universal as I would like.

So I'm wondering, since I can't get Mapnik to like my VRT, if it's possible to reference the existing ogr.Open() datasource object from Mapnik and avoid having to use something like mapnik.PostGIS() or mapnik.Ogr() in the first place.

To get this idea, this is kind of how I imagine it..

import mapnik
import ogr

# create_utfgrids.py uses shapefiles, hence "shppath".
# But for this exercise it represents a path to a VRT.
ds = ogr.Open(shppath)

# A bunch of OGR stuff here.

# This is where I want the magic to happen, 
# pushing the OGR dataset into Mapnik..
ds = mapnik.Ogr(ds)

# And then Mapnik stuff follows.

Answer 1

Depending on your version of mapnik, yes it's possible. Mapnik 2.1 introduced the Python Plugin as a data source.

You would need to subclass mapnik.PythonDatasource and implement a features method that based on an incoming mapnik.Query object will return mapnik.PythonDatasource.wkb_features objects.

For example:

import mapnik
import ogr

class VRTDataSource(mapnik.PythonDatasource):

    def __init__(self, shppath, layer_idx=0):
        self.ds = ogr.Open(shppath)
        self.layer_idx = layer_idx
        super(VRTDatasource, self).__init__()

    def features(self, query):
        bounding_box = query.bbox

        layer = self.ds[self.layer_idx]

        layer.SetSpatialFilterRect(
            bounding_box.minx,
            bounding_box.miny,
            bounding_box.maxx,
            bounding_box.maxy
        )

        return mapnik.PythonDatasource.wkb_features(
            keys = layer[0].keys(),  # Assuming that all features have the
                                     # same table definition
            features = [
                ( feature.GetGeometryRef().ExportToWkb(), dict(feature) ), 
                for feature in layer # OGR layers are feature iterators
            ]
        )


# Then when building your mapnik map object

ds = mapnik.Python(factory='VRTDataSource', shppath=path, layer_idx=0)
layer = mapnik.Layer('python')
layer.datasource = ds

The only point I'd worry about would be the Projection - I'm not sure if mapnik will pass the CRS as part of the mapnik.Query object and expect a projected WKB in return, or whether setting the projection of the mapnik layer means mapnik will project on the fly. My guess would be the former, but I'd want to test first.

UPDATE Expand the example code so that you don't need to specify anything beyond the data source and the layer name (assuming that the layer schema is consistent).

Answer 2

Stepping back, if you have data in PostGIS and you want to render it with Mapnik you really should be using Mapnik's native PostGIS support - aka mapnik.PostGIS datasource in python (https://github.com/mapnik/mapnik/wiki/PostGIS). That will be faster and more efficient than going through OGR. That said, if you have a good reason for connecting to PostGIS through an OGR VRT file then that should work just fine via the Mapnik mapnik.Ogr() datasource. I would recommend getting that working (please file an issue at https://github.com/mapnik/mapnik-support with details on how to replicate) instead of using the python plugin because the python plugin is going to be the slowest of all options.