What is the difference to send around ISessionFactory and use ISessionFactory.GetCurrentSession() with the usage of NHibernate.Context.ManagedWebSessionContext in the configuration? Instead of create your own session holder? Of course you need to bind and unbind the session to the context as you do in ActionFilter, but the rest is only inventing the wheel again :).
I encountered a hidden problem with this approach and delayed execution of queries (via IQueryable and EF4). Seemed like a good route to take, until we started disposing the connection in ActionExecuted, and discovered that the view result isn't executed as part of the action (of course!), but instead happens after the action is complete. The result was exploding views where the connection was disposed when it attempted to execute the query. I had to move to Dispose() call into OnResultExecuted, which solved that problem.