2

I have a hard time figuring out how to write features to a geopackage file with geotools without committing after adding each feature (which makes the writing process unbearably slow).

What I would like to achieve in my example below is to have a batch size of 50 (BATCH_INSERT_SIZE) so that only after 50 features have been written the geopackage file is actually updated.

What happens instead is that after each feature is added it is immediately written to the geopackage.

If it matters: I use GeoTools 24.2 + OpenJDK 11 + Linux

import java.io.IOException;
import java.nio.file.Files;
import java.util.HashMap;

import org.geotools.data.DataStore;
import org.geotools.data.DataStoreFinder;
import org.geotools.data.DataUtilities;
import org.geotools.data.simple.SimpleFeatureStore;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.feature.simple.SimpleFeatureTypeBuilder;
import org.geotools.geometry.jts.JTSFactoryFinder;
import org.geotools.geopkg.GeoPkgDataStoreFactory;
import org.geotools.jdbc.JDBCDataStoreFactory;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.GeometryFactory;
import org.locationtech.jts.geom.LineString;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;

public class GpkgBulkWritingTest {

    public static void main(String[] args) throws IOException, InterruptedException {
        HashMap<String, Object> map = new HashMap<>();
        map.put(GeoPkgDataStoreFactory.DBTYPE.key, GeoPkgDataStoreFactory.DBTYPE.sample);
        String fileName = Files.createTempFile("test_", ".gpkg").toString();
        map.put(GeoPkgDataStoreFactory.DATABASE.key, fileName);
        map.put(JDBCDataStoreFactory.BATCH_INSERT_SIZE.key, 50);

        DataStore dataStore = DataStoreFinder.getDataStore(map);

        SimpleFeatureTypeBuilder builder = new SimpleFeatureTypeBuilder();
        String layerName = "layerName";
        builder.setName(layerName);
        builder.setSRS("EPSG:4326");
        builder.add("geom", LineString.class);
        builder.add("id", Integer.class);
        SimpleFeatureType featureType = builder.buildFeatureType();
        dataStore.createSchema(featureType);

        GeometryFactory gf = JTSFactoryFinder.getGeometryFactory();
        Coordinate[] coordinates = new Coordinate[] { new Coordinate(16, 48), new Coordinate(14, 48) };
        LineString lineString = gf.createLineString(coordinates);

        SimpleFeatureStore featureStore = (SimpleFeatureStore) dataStore.getFeatureSource(layerName);
        System.out.println("writing to " + fileName);
        for (int i = 0; i < 100; i++) {
            System.out.println("writing feature #" + i);
            Object[] attributes = new Object[] { lineString, i };
            SimpleFeature feature = SimpleFeatureBuilder.build(featureType, attributes, null);
            featureStore.addFeatures(DataUtilities.collection(feature));
            Thread.sleep(1000);
        }

        dataStore.dispose();
    }

}

1 Answer 1

1

Your best plan for more speed is to take control of the transactions by creating one yourself and by batching up all the features into a single (or at least larger) list.

    DefaultTransaction transaction = new DefaultTransaction("faster");
    dataStore.createSchema(featureType);

    featureStore = (SimpleFeatureStore) dataStore.getFeatureSource(layerName);
    featureStore.setTransaction(transaction);
    SimpleFeatureCollection collection = new ListFeatureCollection(featureType, features);
    start = System.currentTimeMillis();
    try {
        featureStore.addFeatures(collection);
        transaction.commit();
    } catch (Exception problem) {
        problem.printStackTrace();
        transaction.rollback();
    } finally {
        transaction.close();
    }
    end = System.currentTimeMillis();
    System.out.println("Runtime: " + (end - start) / 1000.0);
}

This code gives me a time of 0.2 seconds compared to your code at 8.3 seconds (onto a local SSD).

4
  • Thank you very much for the proposed solution! Could you also help me to understand when the BATCH_INSERT_SIZE would jump into action?
    – evod
    Nov 22, 2022 at 20:57
  • Somewhere deeper down, you could trace it with a debugger to be sure,
    – Ian Turton
    Nov 23, 2022 at 8:33
  • It is used by JDBCInsertFeatureWriter. Interestingly doing a bulk insert (i.e. adding a large ListFeatureCollection) and setting the BATCH_INSERT_SIZE to 1 is still much faster than inserting one feature after another. Anyways, I'm happy now, thanks again :)
    – evod
    Nov 23, 2022 at 9:12
  • I think that by default you create a transaction for each insert which would defeat the batch size
    – Ian Turton
    Nov 23, 2022 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.