You can read a CSV one line at a time, deal with that line, then either output it or skip to the next one.
Tutorials on reading CSV files in python using the
csv module from the standard library are everywhere. First duckduckgo hit is:
Here's a full working example. This will not read the whole file in, but it still might take a while to process a huge CSV. Its done in the blink of an eye on 100,000 records, a couple of seconds on 2 million records.
Given lines of CSV like this:
Then the following python:
return float(p)>25 and float(p)<30 and \
float(p)>17 and float(p)<19
with open('pts.csv', 'r') as read_obj:
csv_reader = csv.reader(read_obj)
header = next(csv_reader)
for row in csv_reader:
will print out the rows where x is between 25 and 30, and y is between 17 and 19:
$ python filter.py
If you want to write this to another file, either redirect the output or use standard python file writing (well-documented elsewhere).