Instead of a bare NAN.

There are still many places in this function where the code bluntly returns a
bare NAN. That is always the wrong thing to do. However, it is not certain
that the #DIV/0! is the right error in all cases. One would need to check in
each case how to get there, and what the reference C++ implementation and
other spreadsheet products do in such a case.

Change-Id: I5454609082a92144b1afc3c1745586bfe8ef87f8