boost::ptr_set was actually quite nice here, pity about the obnoxious
warnings...

Change-Id: I46973635fd26e4f1db96f2806c211b83436bef5e