I'm georeferencing a Landsat 8 oli no QGIS trip. I include the control points and the system calculates the residuals for each point. At the end it indicates the polynominal mean error of the transformation.
How does the system do this calculation?
It is the root mean square error, adjusted for the degrees of freedom of the transform:
Source: QgsGeoreferencerMainWindow::calculateMeanError
on github.