VS Express (the free, stripped down version of Visual Studio) has nearly full functionality for programming ArcObjects add-ins. You should be able to do everything you need with the free version. Just about the only thing you can't do with VS Express that you might need to is use plug-ins (also known as extensions) such as IronPython.
Download Visual Basic 2010 Express (or C# Express if you want to use C#) and install the ArcObjects SDK for .NET from your ArcGIS CD or .iso. Once installed you can create pre-wired ArcGIS projects as templates.
If you would like to integrate your ArcObjects call from within your Python script, please see this guide.
EDIT: It appears you can also call ArcObjects from Python without installing VS Express, according to the above link. So the answer, OP, to your question, is YES, it is possible to use ArcObjects without VB (or C#).
Also note that although VS Express is free, Microsoft wants you to register it within 30 days of download to continue using it.