Configure a firewall rule either on the machine itself, or on your network firewall to block access to ports 6080 (http) and 6443 (https) for anything other than the local host and the host that has the web server (which may or may not be the same host).
NB: You may not actually need either Web Adaptor, nor an IIS reverse proxy if your firewall rule does all that you need. It depends on if you use the PC's firewall or the network firewall, and if you are OK with other users on the same network accessing the manager or only users on the manager host itself. If you would clarify this, then it may be possible to provide a simpler answer.
If you do require control within your web server and do not want to use Web Adaptor, then...
Use reverse proxy settings on your web server to configure a different URL on which the manager can be accessed. I'm not sure of how to do this in IIS, but in Apache it's fairly straightforward, and the Esri website even has instructions for this at:
See if you can find equivalent instructions for IIS. Googling for "iis reverse proxy" seems to show relevant results (but I'm more of an Apache person, so I don't know how relevant they really are.)
Then on the web server you can configure rules as to who (or what) can access this new reverse proxy URL.