Ok, thanks for your concern on the self answering everyone. I would now like to answer my own question, so anyone with a similar problem might get put on the right track.
Attaching a debugger (*) gives the following error (caught as a first chance exception):
"The application called an interface that was marshalled for a different thread."
So, I'm using callback functions in ctypes to call
arcpy.SetProgressor from the DLL, and something (ArcMap?) complains that these get called by more than one thread (even though it wasn't happening concurrently - I was using critical sections to ensure that). Possibly it would complain if called by any thread other than ArcSOCP - the one that loaded my dll in the first place - i.e. the master thread within the dll.
I think the dll code terminates at that point, but arcmap doesn't respond until a while later when it suddenly shows the geoprocessing task as cancelled. Maybe due to some timeout, or maybe it cancelled straight away but took its time cancelling (anyone who has tried to cancel gp operations mid-flow will know what I mean!).
SetProgressor is only called from the DLL's master thread fixes the problem.
(*) attaching the debugger for some reason, didn't work the way I'd normally do it, from 'attach process' in visual studio. For some reason it worked best to (1) load the project in visual studio; (2) run from arcmap; (3) go to task manager - not visual studio - to find the process (arcmap or arcsocp depending on fg/bg execution); (4) right click -> debug; (5) open the 'debug->threads' pane in visual studio to select the correct thread, of many.