Invalidate properly the cache whenever all_models or app_configs change.
This fixes some isolation issues in the test suite.