THE RANT / THE SCHPLOG
Schmorp's POD Blog a.k.a. THE RANT
a.k.a. the blog that cannot decide on a name

This document was first published 2016-04-23 18:51:49, and last modified 2016-04-24 13:35:18.

Math::BigFloat maintainer fail

What would be your reaction if a maths function in your programming language, that worked unchanged for 15 years, suddenly returned wrong results, or failed to work for valid inputs?

Well, it's a bug, of course.

And what would be your reaction if you study the CHANGES file, and find out it was a deliberate change? And the rationale is completely bogus?

Well, my reaction in such cases is to consider the language to be so unreliable as to be potentially useless - you can use it for toys, but not for anything important.

Well, didn't happen (although perl5porters came close number of times). The following is what really happened.

CBOR::XS mysteriously fails

I got a report that CBOR::XS fails one of it's test cases, the bignum-related one. And indeed, after upgrading Math::BigFloat to the current versions, it perfectly reproduces on my system.

The test in question decodes a bignum with mantissa 3 and binary exponent -1, i.e. 3 * 2 ** -1, or simply 1.5. And fails to do so.

The relevant code is this (made a bit nicer for yours truly):

Math::BigFloat->new ($mantissa)
  ->blsft ($exponent, 2)

This converts the $mantissa to a MAth::BigFloat object, and then calls the blsft method on it with $exponent and the base, 2.

Now, the blsft method isn't exactly well-documented - it is documented, but there is no explanation of what it's valid inputs are:

$x->blsft($y, $n);      # left shift by $y places in base $n

The best you can get is the comment in the sources:

  # shift left by $y (multiply by power of $n)

Or in other words, blsft does exactly what we need:

$mantissa->blsft ($exponent, 2)

And this gave correct results for 15 years (long before CBOR::XS even existed). Until, apparently, release v1.999718 of Math::BigFloat. The 0.000001 version change between that and the previous version has had quite the impact: blsft has been deliberately broken according to the CHANGES file:

 * For Math::BigFloat, modify bitwise operations brsft() and brsft(), and add
band(), bior(), bxor(), and bnot(). These now handle floating point numbers
the same way as core Perl does, i.e., truncate non-integers to integers
before applying the bitwise operator. This change will also make Perl's
behaviour more consistent whether 'use bignum' is in effect or not.

So, a mathematical function that multiplied a number by a power of another number is being restricted to positive powers, and apparently even truncate floating point numbers to integers.

A rationale has been given (do it the same way as the Perl core), but of course it is completely, utterly and hopelessly bogus:

1. Perl core has no blsft method on numbers, so there is no "as core Perl does".
2. Perl core has a left shift operation, but it is a binary operator, while blsft is ternary - blsft is and always has been a superset of Perl's left shift. Perl core never had an operation or operator that works like blsft.
3. use bignum has no effect on explicit blsft method calls. It has an effect on the overloaded << operator, but that isn't blsft.
4. Perl core's left shift operator also gives interesting results for certain shift values (1 << 70 == 64 on my perl) - blsft does not (and should not) emulate that, either.
5. Last and least - nothing in the sources nor the documentation says that these are bitwise functions, whatever that means for floating point numbers. You can't even guess from the name, as all the (computing) methods start with a b in their name, such as bdiv, which simply divides two numbers. Calling them "bitwise" is at best an attempt to redefine the existing API.

The smoking gun

Now, I was told there will be a bug report for this, but of course the outcome is very open - the maintainer might instantly realise that breaking blsft was a bad idea and fix it, or might stand by his/her/their bogus reasoning and keep it broken (see update below: he chose to keep it broken).

The real issue is not whether the maintainer will fix it or not, however, the real issue is the reasoning behind changing a maths function that worked in a certain way for more than a decade.

What happened is change a function that did this:

$mantissa * $base ** $exponent

Into this (per documentation):

(int $mantissa) * $base ** ($exponent < 0 ? NaN : $exponent)

Based on bogus reasoning over a related, but completely different operator.

No, the real issue is taking a mathematical function from a package supposed to do high-precision floating point arithmetic, that has been in that package for 15 years and that is mathematically well defined and add arbitrary limitations based on a different operator - three limitations, pick any two and keep the other out.

This is on the level of only supporting integer exponents in **, or only positive numbers for the addition operator. Or limiting Math::BigInt numbers to 32 bit, for compatibility to the Perl core.

It doesn't matter whether this is fixed or not - the very fact that the maintainer made this change to such an established maths package simply disqualifies him/her/them from maintaining that package.

You don't deliberately change mathematical functions used by a large body of software to give wrong results or fail when they worked before. You. Just. Don't.

The maintainer replies

As an update to the original story, there is now an official reply for the bug report, and unsurprisingly, it's supposedly not a bug, because:

Shifting floats make no sense, because the internal representation is
not just a sequence of bits that can be shifted left and right. If you
search the net for bit-shifting floats you will find explanations for
this.

Of course, shifting float numbers makes a lot of sense, because it is simply a multiplication by a power of two. The concept of "bits" is irrelevant because blsft formerly was not a bit-based function, it worked on ternary and decimal numbers just as well, because it had a base argument, so clearly it was never even meant to work on bits.

Or in other words, blsft is (or rather, was) a general base shifting function which had nothing to do with bits or bytes - it simply shifted digits in arbitrary bases.

Just reading this "explanation" is so painful, I have to stop here and now and try to purge this episode from my memory :)

More seriously, that means that with the current maintainer in place, Math::BigFloat has become dangerously unusable: you can't rely on a float library where basic mathematical functions suddenly stop working and give wrong results.