xiph / rav1e

The fastest and safest AV1 encoder.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unsafe bounds checking is off-by-one in `sgrproj_box_ab_internal`

CodesInChaos opened this issue · comments

The function sgrproj_box_ab_internal looks like this:

  pub(crate) fn sgrproj_box_ab_internal<const BD: usize>(
    r: usize, af: &mut [u32], bf: &mut [u32], iimg: &[u32], iimg_sq: &[u32],
    iimg_stride: usize, start_x: usize, y: usize, stripe_w: usize, s: u32,
  ) {
    let d: usize = r * 2 + 1;
    let n: usize = d * d;
    let one_over_n = if r == 1 { 455 } else { 164 };

    assert!(iimg.len() > (y + d) * iimg_stride + stripe_w + d);
    assert!(iimg_sq.len() > (y + d) * iimg_stride + stripe_w + d);
    assert!(af.len() > stripe_w);
    assert!(bf.len() > stripe_w);

    for x in start_x..stripe_w + 2 {
      // SAFETY: We perform the bounds checks above, once for the whole loop
      unsafe {
        let sum = get_integral_square(iimg, iimg_stride, x, y, d);
        let ssq = get_integral_square(iimg_sq, iimg_stride, x, y, d);
        let (reta, retb) =
          sgrproj_sum_finish::<BD>(ssq, sum, n as u32, one_over_n, s);
        *af.get_unchecked_mut(x) = reta;
        *bf.get_unchecked_mut(x) = retb;
      }
    }
  }

The safety comment claims that the bounds are checked above. The safety condition required by af.get_unchecked_mut(x) is x < af.len(). The maximum value x can have is stripe_w + 2 (exclusive) or stripe_w + 1 (inclusive).

However the assertion only checked that stripe_w < bf.len(), not stripe_w + 1 < bf.len(), so this function doesn't actually ensure its safety preconditions, and is thus unsound.

The same issue applies to all 4 of these assertions. Luckily the stricter assertions do not fail in the unit test suite, so it looks like a theoretical soundness issue, not a practical problem.


An additional complication are integer overflows. If the calculation inside the assertion overflows, the assertion can pass, while the index can still be out of bounds for a smaller x. This currently affects only the first two assertions, but would also affect a "fixed" assertion like assert!(af.len() > stripe_w + 1).