I would recommend using PostgreSQL/PostGIS, since it is natively supported by QGIS, has some handy built-in functions to join with other spatial data (like census tracts), and the use of a database should limit memory issues with large datasets. My recommended set of steps is below.
To execute SQL commands you can either use PGAdmin or QGIS, the former will give you more informative errors on queries, the latter will let you load the results of queries as layers on a map. To access the latter go to
Database > DB Manager > DB Manager and click on the second button.
- Get started setting up PostGIS, also quick installers
- Create a table in your database using the
CREATE TABLEcommand (example) in either the QGIS SQL Window or in PGAdmin's SQL Window
- Import your csv with either a COPY sql command or PGAdmin's built in import function by right-clicking on your table in PGAdmin and clicking on
Import...(this latter can be delicate so I'd recommend COPY for larger datasets).
Add a geometry column to your table by executing the following SQL in either PGAdmin or the QGIS SQL window.
ALTER TABLE some_table ADD COLUMN geom geometry(Point,4326);
Create the point geometries using something like
UPDATE yourtable SET geom = ST_SetSRID(ST_MakePoint( x, y), 4326);
Display a subset of data by using a
SELECTstatement with something like
Or join to census data with something like
SELECT c.gid, c.geom
FROM census c
INNER JOIN yourdata ON ST_Within(yourdata.geom, c.geom)
I recently worked with a dataset of 1.4 million points imported from a CSV. I made sure to delete any irrelevant fields in the csv. It worked fine for me, even though some processes took a bit of time to execute. (QGIS 2.12, 64 bit Windows 7, 8 Gb RAM)
An R candidate fwiw, in pseudocode:
library(rgdal) ## for spatial export library(readr) ## for fast file read x <- read_csv("file.csv") names(x) ## some as yet unknown columns coordinates(x) <- c("x", "y") ## your coordinate names may be different writeOGR(x, ".", layer = "filepoints", driver = "MapInfo File")
This will give you a filepoints.tab in the working directory you can read with QGIS. Or choose "ESRI Shapefile" to create a filepoints.shp, or whatever format it is you need. QGIS uses GDAL much like rgdal does, so there's a lot of overlap. There are analogous Python mechanisms.
Since you don't have any metadata, you could have a set after assigning coordinates(x) to set proj4string(x) <- CRS("+proj=something +etc") but we can only guess for your data.