This allows us to ignore empty dirs or other unrelated dirs.
Thanks Sebastian Pawluś for the report and the initial patch.