Under / over-sized buffer parsing

[demberto]

Hey devs and contributors, nice project! I was looking for an alternative to Python's construct in Rust for a long time and it seems to me that I finally found one. I had a small question:

If a struct definition adds up to say 100 bytes, but while parsing I get a buffer which is having less / more size, I don't want to have problems.

Now this is perfectly normal in my case. In case the buffer is, say, only 80 bytes, I want remaining struct fields to be set to Option::None (I would definitely wrap their data type in an Option wherever needed).

Also, if I get a buffer of, say, 120 bytes, I want deku to ignore the rest of the buffer while parsing but not forget about it during serialisation (since that would lead to loss of data my parser doesn't understand).

Is this possible or if not, is there some easy way I can do it with where deku is currently?

[wcampbell]

Thanks!

As for a solution to your problem, or at least how I read it. I have the following solution:

src/main.rs

use deku::{
    bitvec::{BitVec, Msb0},
    prelude::*,
};

#[derive(Debug, DekuRead, DekuWrite)]
struct A {
    /// Attempt to read in 10 values of Items
    #[deku(count = "10")]
    pub bit_opts: Vec<Item>,
}

#[derive(Debug, DekuRead, DekuWrite)]
struct Item {
    #[deku(
        // With deku "v0.16.0"
        cond = "!deku::rest.is_empty()",

        // With deku "v0.16.0"
        writer = "Item::none_writer(deku::output, *inner)"
    )]
    inner: Option<u8>,
}

impl Item {
    // With deku "v0.16.0"
    /// Instead of not writing anything for `None` values (non-read), this outputs an
    /// empty u8 to fill to `count`
    fn none_writer(output: &mut BitVec<u8, Msb0>, inner: Option<u8>) -> Result<(), DekuError> {
        match inner {
            Some(s) => {
                s.write(output, ())?;
            }
            None => {
                0_u8.write(output, ())?;
            }
        }

        Ok(())
    }
}

fn main() {
    let bytes = [0x01, 0x02, 0x03];
    let a = A::from_bytes((&bytes, 0)).unwrap();
    dbg!(&a);

    let out = a.1.to_bytes().unwrap();
    assert_eq!(
        out,
        [0x01, 0x02, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00]
    );
}

[demberto]

Nah, what I really meant is that, if:

• buffer size > struct size, serialise the rest of the (buffer) data as-is.
• buffer size < struct size, deserialise only the fields present in buffer, rest of the struct fields should be None. Serialise back only uptil the first None field (hence keeping resulting buffer's length same).

Notice how I am keeping a distinction here between None and other falsy values. None is like a boundary marker. I hope I am clear.

[wcampbell]

Sorry if this also isn't it! But I think it is? If not, lmk what bytes you expect before and after, as that's what is easiest to me 😉

Could different ways to go about this.

use deku::prelude::*;

#[derive(Debug, DekuRead, DekuWrite)]
struct A {
    #[deku(cond = "!deku::rest.is_empty()")]
    b: Option<u8>,

    #[deku(cond = "!deku::rest.is_empty()")]
    c: Option<u8>,

    #[deku(cond = "!deku::rest.is_empty()")]
    d: Option<u8>,

    #[deku(cond = "!deku::rest.is_empty()")]
    e: Option<u8>,
}

#[derive(Debug, DekuRead, DekuWrite)]
struct StoreLeftover {
    #[deku(cond = "!deku::rest.is_empty()")]
    b: Option<u8>,

    #[deku(cond = "!deku::rest.is_empty()")]
    c: Option<u8>,

    #[deku(cond = "!deku::rest.is_empty()")]
    d: Option<u8>,

    #[deku(cond = "!deku::rest.is_empty()")]
    e: Option<u8>,

    #[deku(count = "(deku::rest).len() / 8")]
    the_rest: Vec<u8>,
}

fn main() {
    // Let from_bytes return the rest
    let bytes = [0x01, 0x02, 0x03, 0x04, 0x05, 0x06];
    let ((leftover, _), a) = A::from_bytes((&bytes, 0)).unwrap();
    assert_eq!(leftover, [0x05, 0x06]);

    let mut out = a.to_bytes().unwrap();
    out.append(&mut leftover.to_vec());
    assert_eq!(out, [0x01, 0x02, 0x03, 0x04, 0x05, 0x06]);

    // smaller
    let bytes = [0x01, 0x02];
    let ((leftover, _), a) = A::from_bytes((&bytes, 0)).unwrap();
    assert_eq!(leftover, []);

    let mut out = a.to_bytes().unwrap();
    out.append(&mut leftover.to_vec());
    assert_eq!(out, [0x01, 0x02]);

    // use StoreLeftover
    let bytes = [0x01, 0x02, 0x03, 0x04, 0x05, 0x06];
    let ((leftover, _), a) = StoreLeftover::from_bytes((&bytes, 0)).unwrap();
    assert_eq!(leftover, []);

    let mut out = a.to_bytes().unwrap();
    out.append(&mut leftover.to_vec());
    assert_eq!(out, [0x01, 0x02, 0x03, 0x04, 0x05, 0x06]);
}

[demberto]

Cool! Is there any way to get rid of using the #[deku(cond = "!deku::rest.is_empty()")] for every field?

[wcampbell]

We could make this an attribute in the future... But this works:

use deku::prelude::*;

#[derive(Debug, DekuRead, DekuWrite)]
struct NotEmpty<T: for<'a> DekuRead<'a> + DekuWrite> {
    #[deku(cond = "!deku::rest.is_empty()")]
    inner: Option<T>,
}

#[derive(Debug, DekuRead, DekuWrite)]
struct A {
    b: NotEmpty<u8>,
    c: NotEmpty<u8>,
    d: NotEmpty<u8>,
    e: NotEmpty<u8>,
}

fn main() {
    // Let from_bytes return the rest
    let bytes = [0x01, 0x02, 0x03, 0x04, 0x05, 0x06];
    let ((leftover, _), a) = A::from_bytes((&bytes, 0)).unwrap();
    assert_eq!(leftover, [0x05, 0x06]);

    let mut out = a.to_bytes().unwrap();
    out.append(&mut leftover.to_vec());
    assert_eq!(out, [0x01, 0x02, 0x03, 0x04, 0x05, 0x06]);

    // smaller
    let bytes = [0x01, 0x02];
    let ((leftover, _), a) = A::from_bytes((&bytes, 0)).unwrap();
    assert_eq!(leftover, []);

    let mut out = a.to_bytes().unwrap();
    out.append(&mut leftover.to_vec());
    assert_eq!(out, [0x01, 0x02]);
}