I'm not that familiar with QGIS, but I wonder how it compares with ArcGIS in terms of extensibility. Unfortunately there seems to be at least some tradeoffs between extensibility and performance. The best way I've found to get a feel for ArcGIS extensibility is to take a look at Esri's COM component categories found in the registry.
Each category represents a place where users can register dlls containing classes that implement an Esri interface. There are a lot of categories. These categories also contain dog food - Esri uses them not only to discover 3rd party customizations, but also out of the box functionality. While this provides a very fine-grained level of customization, it also means that all these fine grains need to be discovered and loaded at run time. I'm not sure what the relocation cost is, but it must be significant.
C:\Program Files (x86)\ArcGIS\Desktop10.0\Bin\Categories.exe
When you create a dll in Visual Studio there is a place where you can specify the base address for the dll to load into. Since there are so many dlls of different sizes being loaded knowing this ahead of time for an ArcObjects customization would be very difficult. Still, I wonder if a config file could be created instructing where the dll should be loaded into memory. If so, once a user has arcmap running with the dlls loaded that he will typically use he could run a routine that would write the dll base addresses to a config file. That way when arcmap starts it could avoid relocation by loading into those addresses. Then again maybe with 64 bit this won't matter.
At 10.0 Esri introduced Add-ins. The categories of add-ins is much smaller, and discovery doesn't rely on the windows registry. Instead, the add-in dlls are zipped up and placed in a known folder. I'm not sure how this compares performance-wise with dlls discovered via the windows registry. I think the main goal was to allow installation by non-admins.
I'm assuming the question is referring to the Desktop product. The new ArcGIS Runtime product is much lighter weight. I've heard it described as a replacement for MapObjects. It will be interesting to see how it evolves. If Esri does introduce extensibility for WPF Runtime, I hope they don't use the same mechanism for discovery used by Visual Studio when it populates the list of assemblies. That first time clicking "Add Reference..." has gotten painfully slow.