Arithmetic right shift does not preserve sign on overshift

[Daniel Schemmel]

All current solvers (STP, Z3 and MetaSMT) return 0 when arithmetically overshifting to the right.

While this makes sense for a logic overshift or an arithmetic overshift to the left, the arithmetic right shift operation should preserve the sign. This means that the proper result would be 0 or -1, depending on the sign (native code does so).

This is also how e.g., z3 implements the ashr instruction.

[Daniel Schemmel]

The following program shows that STP thinks 0x80 ashr 100 == 0xFF, i.e., overshifting a negative value also gives minus one here.

ret is false
COUNTEREXAMPLE BEGIN: 
ASSERT( b = 0xFF );
ASSERT( a = 0x80 );
COUNTEREXAMPLE END:

#include <stp/c_interface.h>
#include <iostream>

int main(int argc, char **argv)
{
  VC vc = vc_createValidityChecker();

  Expr a = vc_varExpr(vc, "a", vc_bvType(vc, 8));
  Expr b = vc_varExpr(vc, "b", vc_bvType(vc, 8));
  Expr hundred = vc_bvConstExprFromInt(vc, 8, 100);
  Expr minusone = vc_bvConstExprFromInt(vc, 8, 128);
  Expr zero = vc_bvConstExprFromInt(vc, 8, 0);

  vc_assertFormula(vc, vc_sbvLtExpr(vc, a, zero));

  int ret = vc_query(vc, vc_notExpr(vc, vc_eqExpr(vc, b, vc_bvSignedRightShiftExprExpr(vc, 8, a, hundred))));
  if (ret)
  {
    std::cout << "ret is true\n";
  }
  else
  {
    std::cout << "ret is false\n"
              << std::flush;
    vc_printCounterExample(vc);
  }

  vc_Destroy(vc);

  return 0;
}
``