Is there a way to register with geodatabase a table stored in an Oracle 11g database, without using ArcMap or Python script? I'd like to do it with a command line like when registering with ArcSDE (sdelayer -o register ...)
No, it is not possible. Only ArcObjects can register tables with a geodatabase (vice registering with ArcSDE, which can be done with command line tools). ArcPy is the command line for ArcObjects.
The register with geodatabase/ArcSDE dichotomy will dissappear post-10.2 when the now-deprecated ArcSDE command line tools will be eliminated.