The problem with buffering is that the buffers will overlap, and you need to set an exact value for the buffer distance which may be difficult to get right.

Instead, measure the distance from each point to the closest polygon.  Then decide on a threshold for inclusion in a community, or interactively choose for outliers.

Possible methods are available by [Calculating the minimum distance between points and polygons in QGIS][1].


  [1]: https://gis.stackexchange.com/questions/28038/calculating-the-minimum-distance-between-points-and-polygons-in-qgis