GIS vector data for example GML or shapefile contains polygonal data. Lets assume, we have polygon A which has self-intersection which means that the edges in this polygon intersect with each other. Lets say if we want to intersect A with polygon B, then the steps of computing the overlaid polygon differs whether A is a polygon with self-intersections or not. My question is do the polygons in one map layer have self-intersections or not? I searched for this question on internet and I found couple of instances where people complain about error caused while importing data due to the presence of self-intersecting polygons. Also I got the impression that self intersecting polygons are not possible in certain file formats.
Here are relevant excerpts from pages 8 and 9 of the Shapefile Technical Description (emphasis mine):
So, as far as the shapefile specification is concerned, the answer is no, polygons may not intersect themselves.
This is a conceptual rule, not a limitation of the file format. It is quite possible to construct shapefiles containing self-intersecting polygons. However, since self-intersecting geometry is invalid according to the specification, applications may differ in how they handle such cases: some may attempt to clean up the features, while others may [justifiably, if less helpfully] consider the shapefile malformed. Most problematic may be cases where the software assumes a shapefile conforms to the specification (ie, contains no self-intersections) but does not validate it before performing geometric operations, which may yield wrong results.
It's unclear exactly what you want to do, but if you're generating shapefiles, I'd advise against outputting polygons with self-intersections, as it may cause difficulty for your data users. Likewise, if you're writing some sort of algorithm to process shapefiles, you do not need to support self-intersecting polygons to comply with the specification - but if possible, I'd advise you check the input files are not malformed before proceeding.
I am less familiar with GML so I can't say whether self-intersecting polygons are allowed in GML.