ArcGIS 10.0 can connect directly to a database. Although, the connection properties that you provide to the SdeWorksapceFactory are slightly different. You could just try to connect to the database as a geodatabase. You should get an error if it isn't a geodatabase. If you have a workspace reference already, you can test the workspace to see if it is an ISqlWorkspace (database) or an IVersionedWorkspace (geodatabase).
if (myWorkspace is IVersionedWorkspace)
else if (myWorkspace is ISqlWorkspace)
//something else like shapefile, personal or file gdb...
At 10.1, the connection properties are the same, so it is a little simpler. All you have to do then is connect to the database/geodatabase and query the workspace properties.
You could check for the sde schema (and posibly the dbo schema on SQLServer), and then check for the SDE.Version (or dbo.SDE_Version) table. That will tell if there is a geodatabase. Obviously this is fairly rudimentary, and only tells you if you have a geodatabase. Nothing about your ability to connect to it. For example, the geodatabase may not be accepting connections, or it may be corrupt.