How to select features where column matches a regex and save them in a new shapefile?

Question

I want to select features from a shapefile where the value of column matches a regex then save only the selected features in a new shapefile.(I used regex101.com to write the regex and check if it's captured in the test string. So the regex is good). Here's what I did:

def filterByRegex(shapefile, output):

# Read the shapefile
shpfile = gpd.read_file(shapefile)

p = re.compile(
    ur'(^[\d]\shour\sparking\s[\d]am.[\d]pm\smon.fri\sexcept\sby\szone\s.*|^[\d]\shour\svisitor\slimit\s[\d]am.[\d]pm\smon.sat\sexcept\sby\szone\sk\spermit.*|^[\d]\shour\svisitor\sparking\s[\d]am.[\d]pm\smon.fri\sexcept\sby\szone.*|^visitor\sparking\stime\slimits\s[\d][\d]\smin\s.\s\[chart\]\s.\sexcept\sby\szone.*|^stickered\s-\svisitor\sparking\stime\slimits\s.\s\[chart\]\s.\sexcept\sby\szone\s[A-Z].*|^Sign.\s[A-Z][\d][\d][\d][\d].\sReserved\sfor\scarpools\swith\spermit\s[\d]am-[\d][\d]am\smon-fri.*|^Sign.\s[A-Z][\d][\d][\d][\d].\s[\d]\shour\sparking\s[\d]am-[\d]pm\smon-sat\sexcept\sby\szone\s.*|^Sign.\s[A-Z]444[\d].\sparking.\srectangle.\sgreen.white.\s.*)', re.MULTILINE | re.IGNORECASE | re.VERBOSE)

test_str = shpfile['Comments']
print test_str
res = re.findall(p, str(test_str))
print res #empty geodataframe?

# Save the selected features to a new shapefile
res.to_file(output)

filterByRegex('./Outputs/joinedInput.shp', './Outputs/output.shp')

When I run the script I get an empty geodataframe :

Name: Comments, dtype: object
[]
Process finished with exit code 0

I tried using shpfile.Comments.str.extract() and shpfile.Comments.str.contains(), but I get the same result.

Any suggestions?

Answer 1

I don't have your data to test the regex, but here is a simple piece of code that works with a sample data I have:

import geopandas as gpd
import re

def filterByRegex(shapefile, output):
    shpfile = gpd.read_file(shapefile)
    idx = shpfile.EDITBY.str.contains('^ERob', flags=re.IGNORECASE, regex=True, na=False)
    print shpfile[idx]
    shpfile[idx].to_file(output)

filterByRegex(r'C:\GIS\Temp\Roads.shp', r'C:\GIS\Temp\RoadsFiltered.shp')

The shpfile.EDITBY refers to the column name, in your case it would be Comments. The idx returns a Boolean index such as this:

0     False
1     False
2     True

Then we apply this index on the data frame to filter the rows.

Consider reviewing the pandas tutorials to learn more about working data frames, here are some really good ones: Lessons for New pandas Users.