Note: You don't need the FeatureHolders that you have in the above; they accomplish nothing, although they probably aren't hurting either in this case.
While you will eventually be able to get your solution to work, it will be a royal pain. Address data is about as inconsistent as it gets as you're discovering. i.e. Is "st" for "street" or "Saint"? How about hyphens, some datasets will use them, some won't. Etc etc.
Instead I'd propose a different solution - pass all the addresses through a geocoder, and then use the coordinates that come back as the unique ID which you can match by.
FME includes several GeoCoding transformers, but if you don't want to go that route you can do that separately in another package, other questions may help:
Bulk Geocode 20 million records
and Geocoding USA addresses that cannot be sent over internet? for instance
This way you're giving the address problem to other people who have spent years and millions of $ trying to fix it rather than doing it yourself.
This solution won't be perfect either, but it should hopefully help as a follow-up round from your above solution.