In non-UTF8 locales open() will have the wrong encoding for the test
data. We need to open the file via tokenize, to get a detected
encoding instead.