boost::ptr_vector::erase() will delete the element, but fortunately there
is a transfer() method that can be used here.