Geoserver Dynamic Data Interpolation

Question

I'm hoping to get some feedback on styling strategies in Geoserver WMS when dealing with dynamic (e.g. not stored with the layer) data and layers that have a large number of features. We've tried a number of different techniques, all with varying levels of performance (mostly poor) but can't help feeling like we're missing something (like we're going about this wrong from the out).

The Actual Question: Is there a better way than a huge, brute force SLD to interpolate dynamically generated data against a GeoServer feature? We're basically looking for a dynamic color map that interpolates against values not stored with the layer.

The Details: We're trying to create thematic maps that colorize each feature according to some external/dynamic value we've calculated using input from the front end. The resultant of which ends up being an interpolation of each feature in the layer matched with a color we've assigned via those calculations. These calculations change every time they're run, so there is no way to store this data with the layer.

Because the data is dynamic and not stored with the layer, the only means to translate this to Geoserver (we can see) is to interpolate by a known property of the layer...which means every feature in the layer needs to be represented in the Recode or Interpolate function. In situations where the layer has thousands of features, this interpolation can result in gigantic SLD files that cause performance issues when rendering on the client (and frankly, feels like a "brute force" method to get what we're after). Additionally, because of the nature of these calculations, there isn't any order which allows for using a "between" or similar operators.

For example, a typical dyanmic SLD looks something like:

<StyledLayerDescriptor xmlns:ogc="http://www.opengis.net/ogc" xmlns:xlink="http://www.w3.org/1999/xlink">
  <NamedLayer>
    <Name>blockgroups</Name>
    <UserStyle>
      <Name>7ba84d10-0310-43ce-840e-018e284d8785</Name>
      <Abstract>Dynamic Style</Abstract>
      <FeatureTypeStyle>
        <Rule>
          <PolygonSymbolizer>
            <Fill>
              <CssParameter name="fill">
                <ogc:Function name="Recode">
                  <ogc:PropertyName>CODE</ogc:PropertyName>
                  <ogc:Literal>060375028014</ogc:Literal>
                  <ogc:Literal>#ca7df0</ogc:Literal>
                  <ogc:Literal>060372270202</ogc:Literal>
                  <ogc:Literal>#ca7df0</ogc:Literal>
                  <ogc:Literal>060650427063</ogc:Literal>
                  <ogc:Literal>#8936c7</ogc:Literal>
                  ..... Hundreds or Thousands more of these Combos.

And the output looks something like:

What We've Tried to work around this:

1. SLD_BODY parameter with a dynamically generated SLD that uses a Recode or Interpolate function to apply colors by property to each feature. - This results in a 2-5Mb SLD file included with every request (and thus must be X-Form-Encoded/POST-ed to Geoserver rather than a simple inline GET) and renders very slowly (especially when panning zooming). Tiles themselves take upwards of 2-4 seconds to render.
2. SLD parameter with the same dynamically generated SLD from #1 but offloaded to the filesystem and included as a URL in the WMS request. - The size of the SLD is the same but because it's offloaded we can use a GET request and rendering is marginally faster.
3. Using WFS to pull GeoJSON and style on the client. - This requires a 15Mb+ GeoJSON file to be pulled from WFS, consumed and styled on the client. It does seem to work better on small areas (say for a single county) than #1/#2 but once you get into larger geometries the map becomes sluggish and/or strops responding entirely due to both the timing of the initial request and the load of adding such a large set of GeoJSON to the client.
4. Using the Vector Tiles plugin to pull GeoJSON and style on the client - This one seemed promising but due to the lack of viewparams support, it just doesn't work for our use case. Additionally, even prototyping this to work with Google Maps was extremely difficult and didn't always work (partial tiles, service lockups, etc).
5. Reformatting our SLD to make it as small as possible - We were able to get the size of the SLD down by using the in function (rather than using Recode or Interpolate) but even reducing the size to <1.5Mb (for a medium sized area) doesn't seem to reduce the rendering time by any noticeable amount.

E.g:

<Rule>
    <Filter>
        <PropertyIsEqualTo>
          <ogc:Function name="in">
            <ogc:PropertyName>CODE</ogc:PropertyName>
            <ogc:Literal>060855053044</ogc:Literal>
            <ogc:Literal>060374818001</ogc:Literal>
            <ogc:Literal>060570001052</ogc:Literal>
            ...Create a rule over and over for each color grouping...ETC

So, our question is less about how to tune our Geoservers performance (it runs swimmingly for just about everything else we're asking it to do) and more about:

Is there a better way to interpolate dynamically generated data against a GeoServer feature? Is there an extension, service (WPS?) or approach generally used to do what we're after that doesn't require brute force?

---

The layer we're rendering in this particular case is from a ShapeFile store which really limits our options. We're going to see about changing that and taking a swipe at @JGH's proposed solution.

---

After spending a day or two working through the proposed solutions, we ended up taking JGH's suggestion and moved the layer(s) from ShapeFile stores into the database, created a cache table for the dynamic data, then used a SQL View to join the new database layer to the cache table. IMHO, Ian's solution is pretty darn clever and we're going to try and prototype that as well at some point...just between our low level of expertise with WPS and not being a Java shop, it's just a bit more than we're really able to manage at the moment. If/when we do use that solution, I'll post some details here on our experience.

In terms of performance here's a quick breakdown of what we were seeing for posterity. This is against a single non-production GeoServer instance over the wire (not local) so the actual timings are less relevant than the comparative between them.

• Huge SLD Saved to Geoserver and Rendered Via style parameter: 9-17 sec/per tile
• Huge SLD Embedded in Request and Rendered Via sld_body parameter: 9-17 sec/per tile
• Huge SLD Saved to File System and Rendered Via sld parameter: 10-19 sec/per tile
• Joined Cached Table (e.g. JGH's Solution): 2-6 sec/per tile
• Joined Cached Table Rendered Via GWC/WMTS (not primed): 6-7 sec/per tile (then milliseconds after prime)

So, we're seeing (in our environment) that JGH's solution well outperformed our other attempts to resolve this...plus for the more frequently occurring datasets allows us to leverage GWC/WMTS.

Answer 1

You could store the computed categories (colors) on the server. You would basically have a table with sessionID (or requestID, or userID etc) / featureID / colorID. You then publish in geoserver a parameterized view between the layer and this table, the parameter beeing the sessionID (or requestID, or userID etc). The SLD then only has one entry per color (colorID1 -> red etc).

The table should be indexed on sessionID and on featureID, and you would need to do some housekeeping to remove old entries (ex, when a session expires)

Answer 2

A possible option is to use create a join using my experimental table joining process. Then you could save as a new temporary layer and style it directly.