I had similar problems when trying to use georeferencer in QGIS 3.26.3. Added .TIF to georeferencer, placed some reference points (residual less than 0,5 m), then tried to click "Start Georeferencing". Referencing failed. Tried to go to Georeferencer settings, changed the output file name and tried again with no luck. Then I went back to settings, clicked "set target resolution" and did not change anything else. Tried to play "Start Georeferencing" and it did the job like a charm. Went back to settings and again clicked "set target resolution" and tried, then again the referencing job went through. Then Tried this with a new TIF file, placed some reference points from map, clicked "play" and again referencing failed. Went to settings, clicked the box of target resolution and tried again, the referencing job went through again like it should. So in my opinion, there seems to be some sort of bug, which prevents the referencer from working properly without first changing some settings for the referencer. It does not matter, at least in my case, if i renamed the output file, but changing some other settings "there and back" seemed to resolve my problem.
ps. Windows 10 21H2