A bug in ESRI Line Geometry WKB that adds needless, trailing 0's behind otherwise valid WKB?

Question

[Update 7.13.2012]

My coworker noticed the the ESRI Line WKB is malformed contains unspecified trailing data, and since he's not active on GIS.SE, I posted his remarks as an answer. I'll wait a few days for critiques before accepting it in case someone has a more definitive reaction.

In the meantime, I can recreate the exception between ESRI Line Geometry WKB and SqlServer.Types by making a brand new feature class, giving it a single line geometry, and bingo---it's repeatable. Every time. But I Googled this all day yesterday and didn't find any confirmation of this. I must be a bad Googler!

---

[Original Post]

When I try converting an esriPolylineGeometry to IWkb, then go through the motions of exporting to wkbBytes and consuming it in SqlServer.Types.SqlGeometry.STGeomFromWKB() the conversion always fails with the error "well-known binary (WKB) input is not valid". However, if I use the same approach to get wkbBytes, it works fine when I send it through SharpMap converters. I've done everything I can imagine to rule-out poor geometries, and I'm thinking surely someone has run into this before.

On the ESRI-side, I've tried using ITopologyOperator to perform .Simplify() and .Buffer(0), and neither makes a difference. I've also tried adding .MakeValid() behind STGeomFromWKB() to no use. And there is also no difference from either IWkb.ExportToWkb() or the alternative IGeometryFactory.CreateWkbVariantFromGeometry().

So my question is: Is the WKB really invalid? If so, then why is it converting properly in SharpMap?

The WKB coming out of ESRI looks like this:

010200000009000000BA854FAC7A273541B2325A112E643341B6FB21797E273541F080DDD356643341C87DB90187273541DA6E0FE4776433410692AC108B27354140AF88AF81643341CD1A8D1FA127354163B43190B9643341486A0F7CDF2735415B63352B7065334146AFE4D2E12735411BC18BCB776533413DA83565EC273541C03198AA9E653341861DA529F02735414B65EE25C46533410000000000

Below, the first code block shows the failing approach when I attempt to consume using SqlGeometry's static method STGeomFromWKB(). The second block shows the SharpMap approach, which works when consuming the same wkbBytes data.

This approach using SqlServer.Types fails..

// At this point, feature.Shape is esriPolylineGeometry
IWkb wkb = feature.Shape as IWkb;
int byteCount = wkb.WkbSize;
byte[] wkbBytes = new byte[byteCount];
wkb.ExportToWkb(ref byteCount, out wkbBytes[0]);

// FYI: This alternate approach to get the byte[] is no different..
//IGeometry geom = feature.Shape;
//IGeometryFactory3 geomFactory = new GeometryEnvironment() as IGeometryFactory3;
//byte[] wkbBytes = geomFactory3.CreateWkbVariantFromGeometry(wkb) as byte[];

int srid = feature.Shape.SpatialReference.FactoryCode;

// It fails here noting...
// "24115: The well-known binary (WKB) input is not valid."
SqlGeometry sqlGeometry = SqlGeometry.STGeomFromWKB(new SqlBytes(wkbBytes), srid);

string wkt = sqlGeometry.ToString();
MessageBox.Show(wkt);

But this approach using SharpMap converters works.. ?!?!

IWkb geomWKB = feature.Shape as IWkb;
int wkbSize = wkb.WkbSize;
byte[] geomWkbBytes = new byte[wkbSize];
wkb.ExportToWkb(ref wkbSize, out geomWkbBytes[0]);

SharpMap.Geometries.IGeometry sharpGeom = SharpMap.Converters.WellKnownBinary.GeometryFromWKB.Parse(geomWkbBytes);
string sharpWKT = SharpMap.Converters.WellKnownText.GeometryToWKT.Write(sharpGeom);

Both approaches accept Polygon geometries, but I'm hoping someone knows how to get the SqlServer.Types approach to work on Line geometries, as we have other projects already using that library. I won't hesitate to use SharpMap converters if that is the answer, but it bothers me that an unknown condition is causing SqlServer.Types to fail in the context of Line geometries. :/

Answer 1

No offense, but, actually, the conclusions of your co-worker about how something he thinks should work doesn't make it the "official way" it should work. For that, we refer to the OGC specifications.

You can download the OpenGIS Simple Features Specification for SQL 1.1 from the OGC website (a bit older) or even better, the newer OpenGIS Implementation Specification for Geographic information - Simple feature access - Common architecture 1.2.1. Either one would work.

If you refer to pages 66-70 you will find the specification for the older document, or pages 65- 72 for the newer specification.

To sum it off, there is no part that prohibits the wkb from having trailing data. In fact, the specification includes counters for every single element so you know exactly how much of the binary blob you should be parsing. It goes down to even including a byte for byte ordering so you can switch between endianness of the binary representation of every inner geometry element as you parse it!

My point is you always know exactly how much of it you should be reading and how you should be interpreting it.

Microsoft has chosen, in their parser, to continue consuming the binary content even though nothing in the OGC specification says it should. Then they chose to consider that condition "invalid". Are they wrong? Well, that is how they interpreted it even though the binary blob itself does satisfy the spec.

Saying the WKB provided by ESRI is "invalid", for this particular case, is incorrect.

By the way, every single other WKB parser out there (GEOS', PostGIS', GDAL's, ESRI's, SharpMap's, Autodesk's, etc) accepts this condition as OK.

Here is your example using PostGIS:

mydb=# select st_astext('010200000009000000BA854FAC7A273541B2325A112E643341B6FB21797E273541F080DDD356643341C87DB90187273541DA6E0FE4776433410692AC108B27354140AF88AF81643341CD1A8D1FA127354163B43190B9643341486A0F7CDF2735415B63352B7065334146AFE4D2E12735411BC18BCB776533413DA83565EC273541C03198AA9E653341861DA529F02735414B65EE25C46533410000000000');
                                                                                                                                                          st_astext                                                                                                                                                           
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 LINESTRING(1386362.67308842 1270830.06778256,1386366.47317479 1270870.82759863,1386375.00673662 1270903.89086049,1386379.06513322 1270913.68567939,1386401.12324684 1270969.56325843,1386463.48461022 1271152.16878339,1386465.82380195 1271159.79510123,1386476.39534999 1271198.6663848,1386480.16267571 1271236.14816888)
(1 row)

mydb=#

Now I am going to add any random binary data at the end of your geometry:

mydb=# select st_astext('010200000009000000BA854FAC7A273541B2325A112E643341B6FB21797E273541F080DDD356643341C87DB90187273541DA6E0FE4776433410692AC108B27354140AF88AF81643341CD1A8D1FA127354163B43190B9643341486A0F7CDF2735415B63352B7065334146AFE4D2E12735411BC18BCB776533413DA83565EC273541C03198AA9E653341861DA529F02735414B65EE25C46533411010101010101010101010101010101010101010101fff01010101fff');
                                                                                                                                                          st_astext                                                                                                                                                           
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 LINESTRING(1386362.67308842 1270830.06778256,1386366.47317479 1270870.82759863,1386375.00673662 1270903.89086049,1386379.06513322 1270913.68567939,1386401.12324684 1270969.56325843,1386463.48461022 1271152.16878339,1386465.82380195 1271159.79510123,1386476.39534999 1271198.6663848,1386480.16267571 1271236.14816888)
(1 row)

mydb=

Still works.

The bug is actually in the Microsoft parser.

Answer 2

Basically my co-worker answered my main questions, so I'm punching them in here for the sake of answering the thread. Hopefully a few people will up-vote it, giving it some added credibility.

Q1: Is the ESRI Line Geometry as WKB really invalid?

[Yes.] The wkb says there is 9 points in the line, and then the parser should read 9 points. MS (rightfully) detects more data (being the 10 0's) and tells its messed up.

The 10 0's are 5 bytes in the original array. Remove the last 10 0's, and its good, like..

010200000009000000BA854FAC7A273541B2325A112E643341B6FB21797E273541F080DDD356643341C87DB90187273541DA6E0FE4776433410692AC108B27354140AF88AF81643341CD1A8D1FA127354163B43190B9643341486A0F7CDF2735415B63352B7065334146AFE4D2E12735411BC18BCB776533413DA83565EC273541C03198AA9E653341861DA529F02735414B65EE25C4653341

Q2: If the WKB is really invalid, why is the SharpMap library able to convert it?

I am 100% sure why sharplib does not fail: the format dictates that there is 9 points... if you are not picky, you just stop reading and everything SEEMS dandy. And... why would you actually validate that there are no more bytes. So im pretty sure that SharpLib does not handle it specifically.. they just stop reading after the 9 points, and everything seems to work... Thanks to MS for not just leaving Bits & Bytes hanging around.

As for WHY [the ESRI WKB is malformed] i got a few guesses: They allocate x amount of bytes at a time, and are not [precise about closing the byte array].... OR!! they , when counting the number of bytes, they take the body length and adds the header length too (1 byte encodinginfo, 1 byte geom type, 8 bytes of number of points in the line), and somehow when counting the body, they include the length of the header.. causing double header size... Anyway, thats only speculations and im not sure what the info is good for as we cant do anything about it anyway :-)