2

I try to use FME Search Envelope to clip my data smaller. Unfortunately it interrupt all after few seconds.

I am using GeoPackage, is it fatal problem to use FME:s search envelope?

2019-03-14 14:43:17|   0.3|  0.0|INFORM|FME Configuration: Start freeing memory when process usage exceeds 95.48 GB of virtual memory
2019-03-14 14:43:17|   0.3|  0.0|INFORM|FME Configuration: Stop freeing memory when process usage is below 71.61 GB of virtual memory
2019-03-14 14:43:17|   0.3|  0.1|INFORM|Creating writer for format: NULL (Nothing)
2019-03-14 14:43:17|   0.3|  0.0|INFORM|Trying to find a DYNAMIC plugin for writer named `NULL'
2019-03-14 14:43:17|   0.3|  0.0|INFORM|Loaded module 'NULL' from file 'C:\Program Files\FME\plugins/NULL.dll'
2019-03-14 14:43:17|   0.3|  0.0|INFORM|FME API version of module 'NULL' matches current internal version (3.8 20180604)
2019-03-14 14:43:17|   0.4|  0.0|INFORM|Creating reader for format: 
2019-03-14 14:43:17|   0.4|  0.0|INFORM|MULTI_READER(MULTI_READER): Will fail with first member reader failure
2019-03-14 14:43:17|   0.4|  0.0|INFORM|MULTI_READER(MULTI_READER): Adding GEOPACKAGE Reader with keyword GEOPACKAGE_1
2019-03-14 14:43:17|   0.4|  0.0|INFORM|Using Multi Reader with keyword `MULTI_READER' to read multiple datasets
2019-03-14 14:43:17|   0.5|  0.1|INFORM|Creating reader for format: OGC GeoPackage
2019-03-14 14:43:17|   0.5|  0.1|INFORM|Trying to find a DYNAMIC plugin for reader named `GEOPACKAGE'
2019-03-14 14:43:17|   0.5|  0.0|INFORM|FME Configuration: Source coordinate system for reader GEOPACKAGE_1[GEOPACKAGE] set to `EPSG:3067'
2019-03-14 14:43:17|   0.9|  0.3|INFORM|Coordinate System `EPSG:3067' parameters: CS_NAME=`ETRF89.TM-35/Fin' DESC_NM=`ETRF89 / ETRS-TM35FIN' DT_NAME=`ETRS89/01' GROUP=`EUROPE' MAP_SCL=`1' MAX_LAT=`70.13' MAX_LNG=`31.59' MIN_LAT=`59.3' MIN_LNG=`19.09' PARM1=`27' PROJ=`TM' QUAD=`1' SCL_RED=`0.9996' SOURCE=`EPSG, V6.11.2, 3067 [National Land Survey of Finland; http:/]' UNIT=`Meter' X_OFF=`500000' ZERO_X=`0.001' ZERO_Y=`0.001' 
2019-03-14 14:43:17|   0.9|  0.0|INFORM|Loaded module 'GEOPACKAGE' from file 'C:\Program Files\FME\plugins/gdal/GDAL.dll'
2019-03-14 14:43:17|   0.9|  0.0|INFORM|FME API version of module 'GEOPACKAGE' matches current internal version (3.8 20180604)
2019-03-14 14:43:17|   0.9|  0.0|INFORM|FME Configuration: Using FME Reprojection Engine
2019-03-14 14:43:17|   0.9|  0.0|INFORM|Reprojected search envelope from (550000, 6750000) - (560000, 6760000) in coordinate system `EPSG:3067' to (550000, 6750000) - (560000, 6760000) in coordinate system `EPSG:3067'
2019-03-14 14:43:17|   0.9|  0.0|INFORM|GEOPACKAGE reader: Directive '_ENFORCE_FOREIGN_KEY_INTEGRITY' has value 'YES'
2019-03-14 14:43:17|   0.9|  0.0|INFORM|GEOPACKAGE reader: Opening file 'C:\cygwin64\data\mtk\mtk_muokattu.gpkg'
2019-03-14 14:43:17|   0.9|  0.0|INFORM|FME API version of module 'NULL' matches current internal version (3.8 20180604)
2019-03-14 14:43:17|   0.9|  0.0|INFORM|FME API version of module 'NULL' matches current internal version (3.8 20180604)
2019-03-14 14:43:17|   0.9|  0.0|INFORM|Loaded module 'LogCount_func' from file 'C:\Program Files\FME\plugins/LogCount_func.dll'
2019-03-14 14:43:17|   0.9|  0.0|INFORM|FME API version of module 'LogCount_func' matches current internal version (3.8 20180604)
2019-03-14 14:43:17|   0.9|  0.0|INFORM|Loaded module 'InspectorFactory' from file 'C:\Program Files\FME\plugins/InspectorFactory.dll'
2019-03-14 14:43:17|   0.9|  0.0|INFORM|FME API version of module 'InspectorFactory' matches current internal version (3.8 20180604)
Error running translation.
  • 1
    What version of FME do you use? Do you use a classic reader or a FeatureReader? Does this happen with other CRS's as well? (Do you think this is a GeoPackage problem or a CRS problem.) – nielsgerrits Mar 14 at 13:44
  • The log file has only INFORM lines which are just informative. Log does not tell anything about the error. Can you clip your data with ogr2ogr gdal.org/ogr2ogr.html? – user30184 Mar 14 at 14:16
  • 2
    can you put FME in DEBUG mode? FME Options>Translations> click Debug on – Mapperz Mar 14 at 15:15
  • @nielsgerrits - it's FME2018.1 - the "internal version (3.8 20180604)" tells me that (actually one of our developers told me!) – Mark Ireland Mar 14 at 20:27
  • Thanks for answers and comment! Hopefully FME should repair this bug because Search Envelope is very important tool when testing FME functions using very large dataset. I will try FeatureReader with Spatial Filter Bounding Boxes if I wont need to read whole dataset to test this, thanks! – Mickeler Mar 15 at 15:30
2

So, yes, this does look like a fault in FME somewhere. Although I can't reproduce your scenario (reading in Workbench) I do get a crash when I try "View Source Data" to send the data to the FME Data Inspector. The crash only happens when I have a search envelope set, so it sounds like part of the same issue.

Anyway, the obvious workaround is to read all the data and clip it with a Clipper transformer. Use a Creator transformer to create the clip boundary, using the same coordinates as in the search envelope. Route it into the Clipper with the clip boundary as the Clipper and the GeoPackage data as the Clippee.

The less obvious workaround - and which I haven't tried yet - would be to put the envelope as parameters in a WHERE clause. Each table on the FME GeoPackage reader has a WHERE clause parameter. If GeoPackage supports it, you could put something like "ST_MinX > [Minimum X] AND ST_MinY > [Minimum Y] etc". Of course, it depends if the fault is in the underlying library or FME's command to that library. You might still get the same error if the underlying library is at fault.

Meanwhile I've filed an issue with our developers (FMEENGINE-59447) and if they have any other suggestions to get it working, I'll add them here.

1

Created a test geopackage in QGIS with these points in EPSG:3067:

wkt_geom
Point (558723 6758680)
Point (562057 6762107)
Point (552149 6752199)
Point (547750 6747893)
Point (560000 6760000)
Point (550000 6750000)

Using 2018.1.1.2 (20190121 - Build 18586 - WIN64):

  • With a Classic Reader using a Search Envelope I can reproduce the error.
  • Using a FeatureReader with Spatial Filter Bounding Boxes Intersect I get the needed 4 features without a problem.

@Mickeler could you test if this works for you? Cheers.

points on map

workspace

Creator Parameters

Creator Box Parameters

FeatureReader Parameters

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.