Application domains, unloading assemblies and Visual Studio add-ins

It could be the title of a nice story, but it is not... a nice story :-(

So, to be able to unload dynamically loaded assemblies, we need to load them in an AppDomain. Then it is possible to unload the AppDomain using AppDomain.Unload.
So far, so good. It works well when I test it in a WinForm application.
The problem is that I need to do the same in a Visual Studio add-in. There arise the problems! I won't go into the details here, but I suspect that it's due to the fact that a VS add-in is a COM object.

If someone already had to create a VS add-in that can load and unload assemblies, please help!