tdf#88352 correct triangulator numerical problem

The basegfx Triangulator is used in slideshow with canvas to triangulate
the mask geometres. This uses tools::isPointInTriangle which uses
basegfx::tools::arePointsOnSameSideOfLine. This uses the cross product
to solve and for tests against zero the fTools::equalZero call. The
triangulator then uses the more precise rtl::math::approxEqual to test
if one of the points of the triangle is equal to the test point.
In rare cases this can lead to a position where a point is seen as inside
the triangle wrongly because it is not detected as equal to one of the
triangle points. To solve, use increased accuracy.

Change-Id: I73e12b711f14d4c48e829d5db1cadefa0917c19b
Reviewed-on: https://gerrit.libreoffice.org/19925Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Thorsten Behrens <Thorsten.Behrens@CIB.de>

basegfx/source/polygon/b2dpolygontools.cxx

@@ -2186,7 +2186,9 @@ namespace basegfx
             const B2DVector aVectorToA(rEnd - rCandidateA);
             const double fCrossA(aLineVector.cross(aVectorToA));
 
-            if(fTools::equalZero(fCrossA))
+            // tdf#88352 increase numerical correctness and use rtl::math::approxEqual
+            // instead of fTools::equalZero which compares with a fixed small value
+            if(rtl::math::approxEqual(fCrossA, 0.0))
             {
                 // one point on the line
                 return bWithLine;
@@ -2195,7 +2197,8 @@ namespace basegfx
             const B2DVector aVectorToB(rEnd - rCandidateB);
             const double fCrossB(aLineVector.cross(aVectorToB));
 
-            if(fTools::equalZero(fCrossB))
+            // increase numerical correctness
+            if(rtl::math::approxEqual(fCrossB, 0.0))
             {
                 // one point on the line
                 return bWithLine;