Import point data from amazon s3 into QGIS

Question

I'm working with spatial/temporal data, stored in s3 (AWS). The data is in the format:

ID,    Date,                 Lat,    Lon
23424  2017-02-03T23:15:15Z  23.434  -5.2343
23424  2017-02-03T23:20:26Z  23.4356 -5.2637
etc...

What I want to do is import this data into QGIS. It is very large, several gb.

My current method is to use an Athena query to download the relevant data, as a csv. I then import the csv into a PostgreSQL database on my local machine, change the lat/long columns into a point geometry column, and import into QGIS as a POSTGIS layer.

This is time consuming, and also uses up space on my local machine.

Is there a way of connecting to the data stored externally on s3 instead?

I am pretty new to python, and I haven't used the python console much.

Answer 1

This question/answer explains how to use a .vrt file to create a link to the URL that corresponds to the S3 file/bucket. This assumes that you are trying to read the entire S3 file, not a subset of it (though it's probably possible to do that, but it's not explained in the referenced question).