Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make header size functions public #21

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mullr
Copy link

@mullr mullr commented May 23, 2024

I'm using this library in a streaming context, where I don't necessarily have the whole buffer available when reading data. By exposing these functions, I can interpret the framing protocol directly, allowing me to stream in a single message at a time.

For reference, my usage looks something like this:

/// Read a single, complete DLT message from `stream`, and parse it.
pub async fn read_dlt_message<S>(
    stream: &mut S,
) -> Result<dlt_core::parse::ParsedMessage, anyhow::Error>
where
    S: AsyncRead + Unpin,
{
    // dlt_core only works on a byte buffer. So here we interpret the
    // framing part of the protocol just enough to determine the
    // message size, then read it into a buffer, and pass that down to
    // dlt_core.

    // Read the first byte, from which we can calculate the header size
    let header_type_byte = stream.read_u8().await?;
    let headers_len = dlt_core::dlt::calculate_all_headers_length(header_type_byte) as usize;

    // Read the whole header
    let mut header_buf = vec![0u8; headers_len];
    header_buf[0] = header_type_byte;
    stream.read_exact(&mut header_buf[1..]).await?;
    let (_, header) = dlt_core::parse::dlt_standard_header(&header_buf)?;

    // Read the payload. Put it in a single buffer together with the header
    let total_message_size = headers_len + header.payload_length as usize;
    let mut msg_buf = vec![0u8; total_message_size];
    (&mut msg_buf[0..headers_len]).copy_from_slice(&header_buf);
    stream.read_exact(&mut msg_buf[headers_len..]).await?;

    let (remaining_data, dlt_msg) = dlt_core::parse::dlt_message(&msg_buf, None, false)?;
    if !remaining_data.is_empty() {
        return Err(anyhow!(
            "Remaining data after loading DLT message: {remaining_data:?}"
        ));
    }

    Ok(dlt_msg)
}

@marcmo
Copy link
Member

marcmo commented Feb 3, 2025

Hello @mullr , your usecase is actually very familiar to us ;) I thought about it and I think it makes sense to also offer a stream based API for the dlt parser.
your implementation already looks quite good, maybe we can further improve on the memory allocation performance and also the error handling. we have a lot of test data, so I suggest starting with your version we implement this and create a PR, then you can also take a look at it.
alternatively you are welcome to create a PR. The issue for this is here: #34
that way, we also do not have to expose the private API

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants