-1

The current situation:

Carto is installed on a on a cloud VM. Everything is working fine except that the programmatical import using the Import API gives inconsistent results. (I am using the NodeJS wrapper for Carto in my GUI, see: NodeJS Carto)

I have performed my tests using the same CSV file (with UTF-8 encoding FYI). First time the import finished and everything was perfect. However, right after this, when I wanted to re-import this file, the import API threw an unknown error 99999.

The error message associated with this code was:

Sorry, something went wrong and we're not sure what. Try uploading your file again, or contact us and we'll try to help you quickly.

I went ahead and checked the logs in the rails console and for my second import attempt - that had the error - I found that there was a line for IO error. That is now something, but right now I am feeling clueless.

Below I am going to attach two log extracts for your better understanding of this problem. One for the perfect import and another one for the failed import.

Does anybody have a clue as to what might cause this issue and more importantly how a resolution of this problem could be approached?

#<CartoDB::Log @values={
:id=>"1fa505f4-e981-4539-b64b-5c91d758b847",
 :type=>"import",
 :user_id=>"48cb4b67-12a6-46e6-a581-d63a77dce9fc",
 :created_at=>2019-05-02 10:45:25 +0000,
 :updated_at=>2019-05-02 10:45:29 +0000,
 :entries=>"2019-05-02 10:45:28 UTC: Running on server test-carto-install with PID: 31783
2019-05-02 10:45:28 UTC: new_importer()
2019-05-02 10:45:28 UTC: Fetching datasource public_url metadata for item id public/uploads/07760e1a9295c0475b4e/my_csv.csv
2019-05-02 10:45:28 UTC: File will be downloaded from public/uploads/07760e1a9295c0475b4e/my_csv.csv
2019-05-02 10:45:28 UTC: Before importer run
2019-05-02 10:45:28 UTC: Starting import for public/uploads/07760e1a9295c0475b4e/my_csv.csv
2019-05-02 10:45:28 UTC: Unpacking public/uploads/07760e1a9295c0475b4e/my_csv.csv
2019-05-02 10:45:28 UTC: Filename: /tmp/imports/20190502-31783-1vusfu6/my_csv.csv Size (bytes): 238263
2019-05-02 10:45:28 UTC: Importing data from /tmp/imports/20190502-31783-1vusfu6/my_csv.csv
2019-05-02 10:45:28 UTC: File-based import load
2019-05-02 10:45:28 UTC: Detected encoding UTF-8
2019-05-02 10:45:28 UTC: Using database connection with .....
2019-05-02 10:45:28 UTC: ogr2ogr output:          
2019-05-02 10:45:28 UTC: ogr2ogr exit code:       0
2019-05-02 10:45:28 UTC: Georeferencing...
2019-05-02 10:45:28 UTC: Disabling autovacuum for \"cdb_importer\".\"importer_665a6a426cc711e994516a1e6aacd3aa\"
2019-05-02 10:45:28 UTC: Creating the_geom from the_geom column
2019-05-02 10:45:28 UTC: replace empty strings by nulls?
2019-05-02 10:45:28 UTC: string column found,
 replacing
2019-05-02 10:45:28 UTC: geometrifying
2019-05-02 10:45:28 UTC: casting the_geom to geometry
2019-05-02 10:45:28 UTC: Converting to 2D point
2019-05-02 10:45:28 UTC: geometrified
2019-05-02 10:45:28 UTC: Enabling autovacuum for \"cdb_importer\".\"importer_665a6a426cc711e994516a1e6aacd3aa\"
2019-05-02 10:45:28 UTC: Georeferenced
2019-05-02 10:45:28 UTC: Finished importing data from /tmp/imports/20190502-31783-1vusfu6/my_csv.csv
2019-05-02 10:45:28 UTC: Proceeding to register
2019-05-02 10:45:28 UTC: Before renaming from importer_665a6a426cc711e994516a1e6aacd3aa to gence
2019-05-02 10:45:28 UTC: Before moving schema 'my_csv_18' from cdb_importer to public
2019-05-02 10:45:28 UTC: Before persisting metadata my_csv_18' data_import_id: 8a91808f-70ce-4eb1-b4a4-c5f0e554950e
2019-05-02 10:45:29 UTC: Table 'gence_18' registered
2019-05-02 10:45:29 UTC: After importer run
2019-05-02 10:45:29 UTC: After dispatch
2019-05-02 10:45:29 UTC: Import finished
"}

And the log of the problematic import

#<CartoDB::Log @values={:id=>"45ac77a9-5826-4fb4-abf1-a42724c679d9",
 :type=>"import",
 :user_id=>"48cb4b67-12a6-46e6-a581-d63a77dce9fc",
 :created_at=>2019-05-02 10:40:23 +0000,
 :updated_at=>2019-05-02 10:40:28 +0000,
 :entries=>"2019-05-02 10:40:27 UTC: Running on server test-carto-install with PID: 29138
2019-05-02 10:40:27 UTC: new_importer()
2019-05-02 10:40:27 UTC: Fetching datasource public_url metadata for item id public/uploads/695ed5cbfd66b4edb035/my_csv.csv
2019-05-02 10:40:27 UTC: File will be downloaded from public/uploads/695ed5cbfd66b4edb035/my_csv.csv
2019-05-02 10:40:27 UTC: Before importer run
2019-05-02 10:40:27 UTC: Starting import for public/uploads/695ed5cbfd66b4edb035/my_csv.csv
2019-05-02 10:40:27 UTC: Unpacking public/uploads/695ed5cbfd66b4edb035/my_csv.csv
2019-05-02 10:40:27 UTC: Filename: /tmp/imports/20190502-29138-197g5si/my_csv.csv Size (bytes): 238263
2019-05-02 10:40:27 UTC: Importing data from /tmp/imports/20190502-29138-197g5si/my_csv.csv
2019-05-02 10:40:27 UTC: File-based import load
2019-05-02 10:40:27 UTC: Detected encoding UTF-8
2019-05-02 10:40:27 UTC: Using database connection with ...
2019-05-02 10:40:27 UTC: ogr2ogr output:          
2019-05-02 10:40:27 UTC: ogr2ogr exit code:       0
2019-05-02 10:40:27 UTC: Georeferencing...
2019-05-02 10:40:27 UTC: Disabling autovacuum for \"cdb_importer\".\"importer_b31ef1146cc611e99bf86a1e6aacd3aa\"
2019-05-02 10:40:27 UTC: Creating the_geom from the_geom column
2019-05-02 10:40:27 UTC: replace empty strings by nulls?
2019-05-02 10:40:27 UTC: string column found,
 replacing
2019-05-02 10:40:27 UTC: geometrifying
2019-05-02 10:40:27 UTC: casting the_geom to geometry
2019-05-02 10:40:28 UTC: Converting to 2D point
2019-05-02 10:40:28 UTC: geometrified
2019-05-02 10:40:28 UTC: Enabling autovacuum for \"cdb_importer\".\"importer_b31ef1146cc611e99bf86a1e6aacd3aa\"
2019-05-02 10:40:28 UTC: Georeferenced
2019-05-02 10:40:28 UTC: Finished importing data from /tmp/imports/20190502-29138-197g5si/my_csv.csv
2019-05-02 10:40:28 UTC: Proceeding to register
2019-05-02 10:40:28 UTC: Before renaming from importer_b31ef1146cc611e99bf86a1e6aacd3aa to my_csv
2019-05-02 10:40:28 UTC: Before moving schema 'my_csv_17' from cdb_importer to public
2019-05-02 10:40:28 UTC: Before persisting metadata 'my_csv_17' data_import_id: 62cbd604-d93f-45ff-a796-f1713c920688
2019-05-02 10:40:28 UTC: Exception: Input/output error @ io_write - <STDERR>
2019-05-02 10:40:28 UTC: ERROR!

In the stack trace of the IO error I found this:

"/var/lib/gems/2.2.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:435:in `write'\"

and it made me even more confused as I can't believe that what is working ok at one time does not work the next minute.

0

After applying this small change everything is working perfectly.

In the Carto configuration (app_config.yml) there is a section where you can configure Redis settings. What I did was, I incremented the write_timeout attribute by an arbitrary value and amazingly enough that did the trick after a restart.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.