DataPipes

Data Pipes

FIXME: Pipes, Buffers, or ByteStreams? Producer/consumer, or sender/receiver?

Once data has been consumed, it is no longer the concern of the producer (i.e. no backwards-seeking).

To avoid unnecessary data copying, I would like to propose a mechanism that allows data producers and data consumers to cooperate as what were called Communicating Sequential Processes in my CompSci degree!

Each "end" of the buffer can be mapped into a TaskSlot's memory, taking twice the virtual size of the allocated buffer size. This allows both producer and consumer to write "packets" of data up to the size of the buffer in one go to contiguous (virtual) memory. (Device drivers may also write to buffers, but if they use DMA to transfer the data, they will have to deal with the problems of wrap-around, perhaps through a DMA service.)

The buffered data will be indexed internally using 64-bit values (avoiding any chance of reaching overflow, unless there's 1GiB/s transfer for over 500 years).

A consumer task may block waiting for (at least) a certain number of bytes to be made available by the producer. A producer task may block waiting for a certain amount of space to be made available by the consumer.

The tasks may also be unblocked by the end of transmission, timeout, or some other reasons.

Multiple producers could be possible, by a simple lock at their end. Multiple consumers by delaying consumption until all consumers reach the same point in the stream. So, the equivalent of "tee" would transfer data at the rate of the slowest destination, the next program or the filesystem.

ByteStream_CreateStream max_packet_size TO handle, allocated_packet_size
  // Allocate at least max_packet_size of contiguous physical memory (the actual size returned by the SWI)

ByteStream_CreateBlock  block_size TO handle
  // ByteStream_CreateBlock creates a stream that will be filled by block_size bytes once, and the data not consumed by the reader. (e.g. an executable file)

ByteStream_ClaimInsert handle TO write_address, max_write_space
  // Called by the task whose job it is to put data into the buffer.

ByteStream_EnsureSpace handle, amount TO write_address, max_write_space
  // Blocks the task until sufficient space is available to write to

ByteStream_Inserted handle, amount TO write_address, max_write_space
  // Informs the receiver that some data is available

ByteStream_ClaimRemove handle TO read_address, max_read_amount
  // Called by the task who wants to receive the data

ByteStream_WaitFor handle, amount, timeout TO read_address, max_read_amount, remainder
  // max_read_amount may be less than the requested amount, if remainder = 0 or the sender has closed the stream
  // remainder will show the milliseconds remaining before the timeout would be completed
  // read address is the virtual address of the max_read_amount bytes of data that can be read

ByteStream_Consumed handle, amount TO read_address, max_read_amount
  // Called after processing the data, making space for more to be inserted into the stream.

ByteStream_Create

IN
  r0 = max packet size

OUT
  r0 = handle
  r1 = buffer size (>= 2*max packet size)

ByteStream_ClaimInsert

Associates the current task with the producer end of the stream.

IN
  r0 = handle
  r1 = Pointer to requested virtual address or -1
OUT
  r0 = Unchanged
  r1 = Pointer to writable memory

ByteStream_ClaimRemove

Associates the current task with the consumer end of the stream.

IN
  r0 handle
  r1 = Pointer to requested virtual address or -1
OUT
  r0 = Unchanged
  r1 = Pointer to readable memory

ByteStream_ReleaseInsert

Wakes the consumer, if blocked, no more insertions are allowed.

IN
  r0 handle

ByteStream_ReleaseRemove
IN
  r0 handle

Wakes the producer, if blocked, no more insertions are allowed.

ByteStream_WaitUntil

Block the current thread until a enough data has been received.

IN
  r0 handle
  r1 max centiseconds to wait
  r2, r3 input value to wait for (64-bit index)

OUT
  r1 remaining centiseconds, 0 for timeout
  r2 (virtual) address of new data
  r3 amount of data available (may be less than requested, on timeout, or if producer has released stream)

ByteStream_WaitFor

Block the current thread until enough space is available.

IN
  r0 handle
  r1 max centiseconds to wait
  r2, r3 input value to wait for (64-bit index)

OUT
  r1 remaining centiseconds, 0 for timeout
  r2 (virtual) address where data may be written
  r3 amount of space available (may be less than requested, on timeout, or if consumer has released stream)

``