Answering my own question, in case it might be useful for someone else:
I will keep the poi positions in an excel spreadsheet with number, lat, lon, description, long description, priority for now.
In the future I might write a short program that parses a the google maps urls out of a structured textfile and populates the csv. This way I'll be able to dump the information into a textfile quickly.
Install QGIS
Add an OpenStreetMap layer
Add an delimited text layer using the csv file
Add new gpx layer for route planning
Zooming and panning are on the slow side but manageable.
I haven't found out how to automatically create gpx waypoints from the delimited text layer, and unfortunately it is impossible to edit the data with the pois.
I might consider going directly from csv to gpx to have only one layer to deal with.
Overall a solution that is a bit on the complicated and limited side, but at least the data is in an open format and QGIS is certainly extensible.
When I go and visit the points, I will probably just dump all the data as POI waypoints to my gps and write down the order on a pice of paper due to the limited via point capabilities of the device.
MS Streets and Trips is a cheap commercial software, strangely without any even remotely similar open source equivalent. I haven't tested it, but reading this on https://en.wikipedia.org/wiki/Microsoft_Streets_&_Trips makes it clear that proprietary solutions are no good for long term data availablity:
Although version 2013 uses the same file name extension (.est) as
previous versions, it does not recognize files created in previous
versions, including the last major version, 2010. This forces users to
re-input all locations as if using the product for the first time.