While it is technically possible for ArcGIS 10.5 to connect to ancient geodatabase releases as far back as 10.1 (according to the compatibility matrix), there is a caveat in the support policy for Enterprise geodatabases, which states:
Specific versions supported for enterprise geodatabases may vary
depending on the database you use to store the geodatabase. The
database version must be supported for both the client and geodatabase
version. For more information on supported database versions and
client connections made to enterprise geodatabases, see the topics
appropriate to your database management system
The Microsoft SQL Server database requirements for ArcGIS 10.5 page does not list SQL Server 2008 among the supported databases, and has the following Note (emphasis mine):
ArcGIS 10.4 and newer clients connecting to SQL Server must use a SQL
Server 2012 or newer client. Connections to an older release
geodatabase in SQL Server 2008 or 2008 R2 are not supported and may
not work with newer SQL Server clients.
I think you'll need to upgrade your database before Desktop 10.5 will be able to connect. Given that 10.4 didn't support 10.0, you'll probably want to upgrade your geodatabase to at least 10.3.1, possibly 10.4.1, as soon as is feasible.