possible that the callback-less flavors of the ref or proxy could have
been added during GC, so we don't want to replace them.