As a general rule, Esri only supports three generations of a database product with each ArcGIS release, but this is strongly influenced by the database vendor's support for products (if the vendor won't support the driver/database combination, Esri can't either).
In your particular case, you can go to the ArcGIS Server System Requirements page, which links to "Supported databases" which links to the Microsoft SQL Server page, where it states that Esri supports (and has certified) SQL Server 2016, 2014, and 2012 (SP3) with ArcGIS 10.5.x.
There is also a "Note" which states:
ArcGIS 10.4 and newer clients connecting to SQL Server must use a SQL
Server 2012 or newer client. Connections to an older release
geodatabase in SQL Server 2008 or 2008 R2 are not supported and may
not work with newer SQL Server clients.
Esri tries to telegraph their support intentions with updates to the Deprecation Plan documents, which you can find by googling "ArcGIS Deprecation plan 10.n" (in this case, 10.4), which includes:
[Updated June 27, 2015] Support for Microsoft SQL Server 2008/2008 R2
will end with the last release of the ArcGIS 10.3 series of releases
(ArcGIS 10.3.x). ArcGIS Pro 1.x does not support SQL Server 2008 and
2008 R2
So taken together, a plan to end 2008 support, with a lack of explicit 2008 support, and a note indicating that 2008 is not supported, all add up, unfortunately, to inability to connect to SQL Server 2008/2008R2 from ArcGIS 10.4.x or 10.5.x clients.
Since SQL Server 2008 support ended with ArcGIS 10.3.1, and SQL Server 2016 support started at ArcGIS 10.4.0, it is not possible to have an ArcGIS Server configuration which will support both SQL Server 2008 and 2016.
