Hold Ctrl down while clicking and it will add or remove from the selection.