ArcGIS servers are configured with a maximum records count that they're allowed to return following a REST query. This maximum is usually 1000 records, which means that GDAL/OGR clients should implement an iterator to go through all the data in a layer.
My attempt to solve this problem was to use returnIdsOnly=true option to ArcGIS, which results in the list of object ids in the current layer. Next, I read these layers using a query on object id range, and this works fine as far as ArcGIS is concerned.
However, GDAL/OGR does not seem to have an easy way of concatenating results from different ArcGIS queries belonging to the same layer. Every time a data source is opened with OGRSFDriverRegistrar::Open function, a separate OGRDataSource is created with its own set of information.
I wonder if anyone could guide me on how to merge results of OGRSFDriverRegistrar::Open calls into the same OGRLayer object? I've tried using layer union operation, but it seems to be failing. Any hints?