And make it re-entrant for thread safety. This method should not modify
the internal state of the token array object. The one with ScCompiler
is not re-entrant. Still some way to go.

Change-Id: I06de3637341727aef0963dddfb98527f415bf7fa