0

I am using GeoTools with OGR plugin to read .gdb data, and I found that the performance is rather horrible:

Test code:

    public static void main(String[] args) throws MalformedURLException {
        String gdb = "/data/x.gdb";

        Map<String, Object> params = new HashMap<>();
        //geotools pg
        params.clear();
        params = new HashMap<>();
        params.put("dbtype", "postgis");
        //other connection string omitted

        //ogr gdb
//        params.clear();
//        params.put("DriverName", "OpenFileGDB");
//        params.put("DatasourceName", new File(gdb).getAbsolutePath());

        try {
            long start = System.nanoTime();
            DataStore dataStore = DataStoreFinder.getDataStore(params);
            SimpleFeatureSource sfs = dataStore.getFeatureSource("Water");
            System.out.println("total count " + sfs.getCount(Query.ALL));
            FilterFactory2 ff = CommonFactoryFinder.getFilterFactory2();
            ReferencedEnvelope bbox = new ReferencedEnvelope(x1,x2,y1,y2, sfs.getSchema().getCoordinateReferenceSystem());
            Filter bboxFilter = ff.bbox(ff.property(sfs.getSchema().getGeometryDescriptor().getLocalName()), bbox);
            Query query = new Query();
            query.setFilter(ff.and(Arrays.asList(bboxFilter)));

            System.out.println("filter count " + sfs.getCount(query));

            long end = System.nanoTime();
            long used = end - start;
            System.out.println("used:" + TimeUnit.NANOSECONDS.toMillis(used) + " ms");
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

The test result for OpenFileGDB driver:

total count 821328
filter count 1803
used:11710 ms

Then I exported that data to PG, and got the following result:

total count 821328
filter count 1805
used:1259 ms

PostGIS is much, much better than OpenFileGDB.

Did I miss anything?

8
  • 1
    what is your question? A real database will always outperform a file pretending to be a database, especially when there is no native support for it in GeoTools.
    – Ian Turton
    Sep 27, 2020 at 10:38
  • Geotools use gt-ogr module which depends on jni. So I’d expected OpenFileGDB should perform well. And 11 seconds on a 800K records layer surprised me.
    – giser
    Sep 27, 2020 at 11:18
  • 1
    @IanTurton I've usually found that flat files outperform a relational database due to ACID overhead. File geodatabase in particular outperformed both Oracle 10gR2 and PG 9.4 on a 512GB server-class machine by a order of magnitude. I wonder whether the FGDB in this case had a spatial index, and whether the Esri FGDB API would give better results. I once used a DB2 database tuned by IBMers that returned 1.2 million rows in ~3 seconds when Oracle took ~5 minutes, but that was ~5 years before the FGDB API was published.
    – Vince
    Sep 27, 2020 at 13:43
  • 3
    There are a number of undefined variables here: Presence of spatial index, Local vs networked drive, cache state of the database vice OS filesystem cache, number of test iterations,... I wonder how the Esri DLL would perform under the same conditions.. Does the OpenFGDB API even use the spatial index?
    – Vince
    Sep 27, 2020 at 13:54
  • @Vince no need to defend that desperately FGDB format. To your question yes, OpenFileGDB can use a spatial index. On the other hand OpenFileGDB driver's purpose is certainly not filtering performance. It is just to help making data open where one company was not keen to do so.
    – Miro
    Sep 27, 2020 at 15:04

2 Answers 2

2

It is hard to give an objective answer. Generally speaking OpenFileGDB driver's main purpose is to help GIS users make data accessible without need of proprietary software - read only. In other words fix accessibility problem. OpenFileGDB driver was created by reverse-engineering, and so might not / does not have some functionalities as good as native proprietary driver (which is dependent on third party libraries).

So I believe comparing it with PostGIS at something which was not it's goal is not the best idea. If you are interested in more detail, you can find more informations on official gdal website:

https://gdal.org/drivers/vector/openfilegdb.html

0

Finally, I found the answer.

Why it cost too long is that the GDAL(3.1.2) used for test does not support spatial index at all.

Since the docs said:

Since GDAL 3.2, the driver can use the native .spx spatial indices for spatial filtering.

Why I did not notice the version was that I thought the 3.2.x would be the current stable version since the docs mentioned this. So when I downloaded the binary from GISInternal I did not pay much attention to the version number espacially when the PATCH version is 2. Stupid mistake. :(

Then I tried the 3.2.0dev version, the same test cost 80ms instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.