Number sign issue when porting JavaScript code to Lua

I’ve been trying to port some JavaScript code to Lua. The code encodes and decodes data to and from a binary stream.

I am having an issue when trying to port some JavaScript code that can contain signed integers. The JavaScript code looks as such:

function encodeVarint(val) {
    const bigval = BigInt(val);
    console.log('r1: ' + (bigval >> 63n).toString());
    console.log('r2: ' + (bigval << 1n).toString());
}

encodeVarint(-332222);

This code is based on code found in this library: https://github.com/hathora/bin-serde-ts/blob/develop/index.ts

I’ve added simplified versions of these functions here for testing: https://playcode.io/883024/

My Lua code is as such (with some logging to compare with PlayCode):

function encodeVarint(val)
    print("v", val)

    local r1 = bit.rshift(val, 63)
    local r2 = bit.lshift(val, 1)

    print("r1", r1)
    print("r2", r2)

    -- a work-around to get similar sign as in JavaScript
    local sign = val > 0 and 1 or -1

    local n = bit.bxor(r1, r2) * sign
    print("n", n)
    return tonumber(n)
end

function decodeVarint(val)
    local r1 = bit.rshift(val, 1)
    local r2 = -(bit.band(val, 1))
    return bit.bxor(r1, r2)
end

Based on code found here: https://github.com/wolf81/bit-serde-lua/blob/main/bit-serde/bit-serde.lua

The simplified example can be executed here: https://replit.com/@WolfgangSchreur/MysteriousEmbellishedSpools#main.lua

Now the problem that I’m seeing is that in the JavaScript code I get the following values for r1, r2:

r1: -1
r2: -664444

Whereas in the Lua code I get the following:

r1:  1
r2:  -664444

So I am guessing this makes the bit.bxor() method not do the thing it’s supposed to. Since the sign is lost in the Lua version. Now as can be seen I manually add the sign, however this causes issues when the data byte length is too much (I guess an extra bit is added for the sign)

Any idea how to make the Lua code behave the same as the JavaScript code?

Answer

JavaScript’s right shift operator keeps the sign bit:

The right shift operator (>>) shifts the first operand the specified number of bits to the right. Excess bits shifted off to the right are discarded. Copies of the leftmost bit are shifted in from the left. Since the new leftmost bit has the same value as the previous leftmost bit, the sign bit (the leftmost bit) does not change. Hence the name “sign-propagating”.

MDN

Lua’s bit library has two functions for right shift:

1. bit.rshift fills with 0 (positivie sign bit) and
2. bit.arshift keeps the sign bit.

Logical shifts treat the first argument as an unsigned number and shift in 0-bits. Arithmetic right-shift treats the most-significant bit as a sign bit and replicates it.

Lua Bit API Functions

bit.arshift has the same behavior for right shifts up to 31 bit.