Protocol

Protocol layers, to be wrapped around a stored::Debugger or stored::Synchronizer instance.

Every embedded device is different, so the required protocol layers are too. What is common, is the Application layer, but as the Transport and Physical layer are often different, the layers in between are often different too. To provide a common Embedded Debugger interface, the client (e.g., GUI, CLI, python scripts), we standardize on ZeroMQ REQ/REP over TCP.

Not every device supports ZeroMQ, or even TCP. For this, several bridges are required. Different configurations may be possible:

In case of a Linux/Windows application: embed ZeroMQ server into the application, such that the application binds to a REP socket. A client can connect to the application directly.
Terminal application with only stdin/stdout: use escape sequences in the stdin/stdout stream. python/libstored.wrapper.stdio is provided to inject/extract these messages from those streams and prove a ZeroMQ interface.
Application over CAN: like a python/libstored.wrapper.stdio, a CAN extractor to ZeroMQ bridge is required.

Then, the client can be connected to the ZeroMQ interface.

Test it using the terminal example, started using the python/libstored.wrapper.stdio. Then connect one of the clients to it.

libstored suggests to use the protocol layers below, where applicable. Standard layer implementations can be used to construct the following stacks (top-down):

If you have to implement you own protocol layer, start with stored::ProtocolLayer. Especially, override stored::ProtocolLayer::encode() for messages passed down the stack towards the hardware, and stored::ProtocolLayer::decode() for messages from the hardware up.

The inheritance of the layers is shown below.

stored::AsciiEscapeLayer

class AsciiEscapeLayer : public stored::ProtocolLayer 

Escape ASCII control characters.

This is required to encapsulate messages within stored::TerminalLayer, for example.

Public Types

typedef ProtocolLayer base

Public Functions

explicit AsciiEscapeLayer(bool all = false, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Constructor.

Parameters:

up – the layer above, which receives our decoded frames
down – the layer below, which receives our encoded frames
all – when true, convert all control characters, instead of only those that conflict with other protocols

virtual ~AsciiEscapeLayer() override = default

Destructor.

Ties to the layer above and below are nicely removed.

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

Public Static Attributes

static char const Esc = '\x7f'

static char const EscMask = '\x1f'

stored::BufferLayer

class BufferLayer : public stored::ProtocolLayer 

Buffer partial encoding frames.

By default, layers pass encoded data immediately to lower layers. However, one might collect as much data as possible to reduce overhead of the actual transport. This layer buffers partial messages until the maximum buffer capacity is reached, or the last flag is encountered.

Public Types

typedef ProtocolLayer base

Public Functions

explicit BufferLayer(size_t size = 0, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Constructor for a buffer with given size.

If size is 0, the buffer it not bounded.

virtual ~BufferLayer() override = default: Destructor.

virtual void encode(void const *buffer, size_t len, bool last = true) override: Collects all partial buffers, and passes the full encoded data on when last is set.

virtual void reset() override: Reset the stack (top-down), and drop all messages.

stored::CallbackLayer

template<typename Up, typename Down, typename Connected> class CallbackLayer : public stored::ProtocolLayer 

Callback class that invokes a callback for every messages through the stack.

Use as follows:

auto cb = stored::make_callback(
              [&](void*, size_t){ ... },
              [&](void const&, size_t, bool){ ... });

The first argument (a lambda in the example above), gets the parameters as passed to decode(). The second argument get the parameters as passed to encode().

Public Types

typedef ProtocolLayer base

Public Functions

inline CallbackLayer(CallbackLayer &&l) noexcept

CallbackLayer(CallbackLayer const&) = delete

virtual ~CallbackLayer() override = default

inline virtual void connected() override: (Re)connected notification (bottom-up).

inline virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

inline virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

void operator=(CallbackLayer&&) = delete

void operator=(CallbackLayer const&) = delete

stored::CompressLayer

class CompressLayer : public stored::ProtocolLayer 

Compress/decompress streams.

The compress layer uses heatshrink for compression. It is a general-purpose algorithm, which is not the best compression, and not also not the fastest, but has a limited memory usage and allows streams, which makes is appropriate for embedded systems.

Compression works best on longer streams, but this layer works per message. So, although it may be stacked in any protocol stack, the compression ratio may be limited. It is nicely used in stored::Stream, where it compresses a full stream (not separate messages), which are sent in chunks to the other side.

When heatshrink is not available, this layer is just a pass-through.

Public Types

enum [anonymous]

Values:

enumerator Window: Window size. See heatshrink documentation.

enumerator Lookahead: Lookahead. See heatshrink documentation.

enumerator DecodeInputBuffer: Include buffer size in bytes. See heatshrink documentation.

enumerator FlagEncoding: Flag for m_state to indicate an active encoder.

enumerator FlagDecoding: Flag for m_state to indicate an active decoder.

typedef ProtocolLayer base

Public Functions

explicit CompressLayer(ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr): Ctor.

virtual ~CompressLayer() override: Dtor.

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

bool idle() const

Check if the encoder and decoder are both in idle state.

Returns:: true if there is no data stuck in any internal buffer.

virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

inline virtual void setPurgeableResponse(bool purgeable = true) final

Flags the current response as purgeable.

This may influence how a response is handled. Especially, in case of retransmits of lost packets, one may decide to either reexecute the command, or to save the first response and resend it when the command was retransmitted. In that sense, a precious response (default) means that every layer should handle the data with case, as it cannot be recovered once it is lost. When the response is flagged purgeeble, the response may be thrown away after the first try to transmit it to the client.

By default, all responses are precious.

stored::Crc16Layer

class Crc16Layer : public stored::ProtocolLayer 

A layer that adds a CRC-16 to messages.

Like stored::Crc8Layer, but using a 0xbaad as polynomial.

Public Types

enum [anonymous]

Values:

enumerator polynomial

enumerator init

typedef ProtocolLayer base

Public Functions

explicit Crc16Layer(ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr): Ctor.

virtual ~Crc16Layer() override = default: Dtor.

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

virtual void reset() override: Reset the stack (top-down), and drop all messages.

stored::Crc8Layer

class Crc8Layer : public stored::ProtocolLayer 

A layer that adds a CRC-8 to messages.

If the CRC does not match during decoding, it is silently dropped. You probably want stored::DebugArqLayer or stored::ArqLayer somewhere higher in the stack.

An 8-bit CRC is used with polynomial 0xA6. This polynomial seems to be a good choice according to Cyclic Redundancy Code (CRC) Polynomial Selection For Embedded Networks (Koopman et al., 2004).

8-bit is quite short, so it works only reliable on short messages. For proper two bit error detection, the message can be 256 bytes. For three bits, messages should only be up to 30 bytes. Use an appropriate stored::SegmentationLayer somewhere higher in the stack to accomplish this. Consider using stored::Crc16Layer instead.

Public Types

enum [anonymous]

Values:

enumerator polynomial

enumerator init

typedef ProtocolLayer base

Public Functions

explicit Crc8Layer(ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr): Ctor.

virtual ~Crc8Layer() override = default: Dtor.

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

virtual void reset() override: Reset the stack (top-down), and drop all messages.

stored::DebugArqLayer

class DebugArqLayer : public stored::ProtocolLayer 

A layer that performs Automatic Repeat Request operations on messages for stored::Debugger.

Only apply this layer on stored::Debugger, as it assumes a REQ/REP mechanism. For a general purpose ARQ, use stored::ArqLayer.

This layer allows messages that are lost, to be retransmitted on both the request and response side. The implementation assumes that lost message is possible, but rare. It optimizes on the normal case that message arrive. Retransmits may be relatively expensive.

Messages must be either lost or arrive correctly. Make sure to do checksumming in the layer below. Moreover, you might want the stored::SegmentationLayer on top of this layer to make sure that packets have a deterministic (small) size.

Every message is prefixed with a sequence number in the range 0-0x7ffffff. Sequence numbers are normally incremented after every message. It can wrap around, but if it does, 0 should be skipped. So, the next sequence number after 0x7ffffff is 1.

This sequence number is encoded like VLQ (Variable-length quantity, see https://en.wikipedia.org/wiki/Variable-length_quantity), with the exception that the most significant bit of the first byte is a reset flag. So, the prefix is one to four bytes.

A request (to be received by the target) transmits every chunk with increasing sequence number. When the target has received all messages of the request (probably determined by a stored::SegmentationLayer on top), the request is executed and the response is sent. When everything is OK, the next request can be sent, continuing with the sequence number. There should be no gap in these numbers.

The client can decide to reset the sequence numbers. To do this, send a message with only the new sequence number that the client will use from now on, but with the reset flag set. There is no payload. The target will respond with a message containing 0x80 (and no further payload). This can be used to recover the connection if the client lost track of the sequence numbers (e.g., it restarted). After this reset operation, the next request shall use the sequence number used to reset + 1. The response will start with sequence number 1.

Individual messages are not ACKed, like TCP does. If the client does not receive a response to its request, either the request or the response has been lost. In any case, it has to resend its full request, using the same sequence numbers as used with the first attempt. The target will (re)send its response. There is no timeout specified. Use a timeout value that fits the infrastructure of your device. There is no limit in how often a retransmit can occur.

The application has limited buffering. So, neither the request nor the full response may be buffered for (partial) retransmission. Therefore, it may be the case that when the response was lost, the request is reexecuted. It is up to the buffer size as specified in DebugArqLayer’s constructor and stored::Debugger to determine when it is safe or required to reexected upon every retransmit. For example, writes are not reexecuted, as a write may have unexpected side-effects, while it is safe to reexecute a read of a normal variable. When the directory is requested, the response is often too long to buffer, and the response is constant, so it is not buffered either and just reexecuted. Note that if the buffer is too small, reading from a stream (s command) will do a destructive read, but this data may be lost if the response is lost. Configure the stream size and DebugArqLayer’s buffer appropriate if that is unacceptable for you.

Because of this limited buffering, the response may reset the sequence numbers more often. Upon retransmission of the same data, the same sequence numbers are used, just like the retransmission of the request. However, if the data may have been changed, as the response was not buffered and the request was reexecuted, the reset flag is set of the first response message, while it has a new sequence number. The client should accept this new sequence number and discard all previously collected response messages.

Within one request or response, the same sequence number should be used twice; even if the request or response is very long. Worst-case, when there is only one payload byte per message, this limits the request and response to 128 MB. As the payload is allowed to be of any size, this should not be a real limitation in practice.

This protocol is verified by the Promela model in tests/DebugArqLayer.pml.

Public Types

typedef ProtocolLayer base

Public Functions

explicit DebugArqLayer(size_t maxEncodeBuffer = 0, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr): Ctor.

virtual ~DebugArqLayer() override = default: Dtor.

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

virtual void reset() override: Reset the stack (top-down), and drop all messages.

virtual void setPurgeableResponse(bool purgeable = true) override

Flags the current response as purgeable.

This may influence how a response is handled. Especially, in case of retransmits of lost packets, one may decide to either reexecute the command, or to save the first response and resend it when the command was retransmitted. In that sense, a precious response (default) means that every layer should handle the data with case, as it cannot be recovered once it is lost. When the response is flagged purgeeble, the response may be thrown away after the first try to transmit it to the client.

By default, all responses are precious.

Public Static Attributes

static uint8_t const ResetFlag = 0x80

stored::DebugZmqLayer

class DebugZmqLayer : public stored::ZmqLayer

Constructs a protocol stack on top of a REQ/REP ZeroMQ socket, specifically for the stored::Debugger.

Public Types

enum [anonymous]

Values:

enumerator DefaultPort

typedef ZmqLayer base

Public Functions

explicit DebugZmqLayer(void *context = nullptr, int port = DefaultPort, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Constructor.

The given port used for a REQ/REP socket over TCP. This is the listening side, where a client like the libstored.gui can connect to.

See also

stored::Debugger

virtual ~DebugZmqLayer() override = default: Dtor.

virtual int recv(long timeout_us = 0) override

Try to receive all available data from the ZeroMQ REP socket, and decode() it.

Parameters:: timeout_us – if zero, this function does not block. -1 blocks indefinitely.

stored::DoublePipeLayer

class DoublePipeLayer : public stored::PolledFileLayer 

Server end of a pair of named pipes.

This is like NamedPipeLayer, but it uses two unidirectional pipes instead of one bidirectional.

Subclassed by stored::XsimLayer

Public Types

typedef PolledFileLayer base

Public Functions

explicit DoublePipeLayer(char const *name_r, char const *name_w, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

virtual ~DoublePipeLayer() override

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual fd_type fd() const override: The file descriptor you may poll before calling recv().

bool isConnected() const: Checks if both pipes are connected.

virtual bool isOpen() const override: Checks if the file descriptor is open.

virtual int recv(long timeout_us = 0) override

Try to receive and decode data.

Returns:: 0 on success, otherwise an errno

virtual void reopen()

virtual void reset() override: Reset the stack (top-down), and drop all messages.

stored::FifoLoopback

template<size_t Capacity, size_t Messages = impl::defaultMessages(Capacity)> class FifoLoopback

Bidirectional loopback for two protocol stacks with thread-safe FIFOs.

The loopback has an a and b side, which are symmetrical. Both sides can be used to connect to a stored::Synchronizer, for example.

Public Types

using FifoLoopback1_type = FifoLoopback1<Capacity, Messages>

Public Functions

inline FifoLoopback()

inline FifoLoopback(ProtocolLayer &a, ProtocolLayer &b)

inline ~FifoLoopback()

inline ProtocolLayer &a()

a endpoint.

Use this layer to register in a stored::Synchronizer, for example, or to wrap another stack.

inline FifoLoopback1_type &a2b()

The a to b FIFO.

Use this FIFO to call recv() on at the b side of the loopback.

inline ProtocolLayer &b()

b endpoint.

Use this layer to register in a stored::Synchronizer, for example, or to wrap another stack.

inline FifoLoopback1_type &b2a()

The b to a FIFO.

Use this FIFO to call recv() on at the a side of the loopback.

stored::FifoLoopback1

template<size_t Capacity, size_t Messages = impl::defaultMessages(Capacity)> class FifoLoopback1 : public stored::PolledLayer 

A ProtocolLayer that buffers downstream messages.

To get the messages from the fifo, call recv(). If there are any, the are passed upstream. Blocking on a recv() is not supported; always use 0 as timeout (no waiting).

This fifo is thread-safe by default. Only encode() messages from one context, and only recv() (and therefore decode()) from another context. Do not mix or have multiple encoding/decoding contexts.

Public Types

typedef PolledLayer base

typedef MessageFifo<Capacity, Messages, true> Fifo_type

using OverflowCallback = bool()

Public Functions

inline explicit FifoLoopback1(ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

virtual ~FifoLoopback1() override = default

inline size_t available() const noexcept

inline constexpr bool bounded() const noexcept

inline bool empty() const noexcept

inline virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

When the FIFO is full and new messages are dropped, the overflow() is invoked. As long as it returns true, the FIFO push is retried.

See also

setOverflowHandler()

inline bool full() const noexcept

inline virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

inline virtual bool overflow()

Invoke overflow handler.

If no callback is set, lastError() is set to ENOMEM, and false is returned. This flag is only reset by reset().

See also

setOverflowHandler()

Returns:: true if the overflow situation might be resolved, false when no other fifo push is to be attempted and the data is to be dropped.

inline virtual int recv(long timeout_us = 0) override

Pass at most one message in the FIFO to decode().

timeout_us is here for compatibility with the ProtocolLayer interface, but must be 0. Actual blocking is not supported.

The return value is either 0 on success or EAGAIN in case the FIFO is empty. This value is not saved in lastError(), as that field is only used by encode() and is not thread-safe.

inline virtual void recvAll(): Pass all available messages in the FIFO to decode().

inline virtual void reset() override: Reset the stack (top-down), and drop all messages.

template<typename F = std::nullptr_t, SFINAE_IS_FUNCTION(F, OverflowCallback, int) = 0> inline void setOverflowHandler(F &&cb = nullptr): Set the handler to be called by overflow().

inline constexpr size_t size() const noexcept

inline size_t space() const noexcept

stored::FileLayer

class FileLayer : public stored::PolledFileLayer 

A layer that reads from and writes to file descriptors.

For POSIX, this applies to everything that is a file descriptor. read() and write() is done non-blocking, by using a stored::Poller.

For Windows, this can only be used for files. See stored::NamedPipeLayer. All files reads and writes use overlapped IO, in combination with a stored::Poller. The file handle must be opened that way. The implementation always has an overlapped read pending, and checks the corresponding event for completion. Every decode() triggers an overlapped write. It uses a completion routine, so the thread must be put in an alertable state once in a while (which the stored::Poller does when blocking).

Subclassed by stored::NamedPipeLayer, stored::SerialLayer

Public Types

enum [anonymous]

Values:

enumerator DefaultBufferSize

typedef PolledFileLayer base

Public Functions

explicit FileLayer(char const *name_r, char const *name_w = nullptr, size_t bufferSize = DefaultBufferSize, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Ctor for two files to be opened.

If name_w is nullptr, name_r is used to write to as well. Files are created when required. If the file exists, writing will append data to the file (it is not truncated).

Do not use this ctor when inheriting this class.

It sets lastError() appropriately.

explicit FileLayer(int fd_r, int fd_w = -1, size_t bufferSize = DefaultBufferSize, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Ctor for two already opened file descriptors.

If fd_w is -1, fd_r is used to write to as well.

Do not use this ctor when inheriting this class.

It sets lastError() appropriately.

virtual ~FileLayer() override

Dtor.

It calls close_(), not close() as the latter is virtual.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

Sets lastError() appropriately.

virtual fd_type fd() const override: Returns the file descriptor to be used by a stored::Poller in order to call recv().

virtual bool isOpen() const override: Check if the file descriptors are open.

virtual int recv(long timeout_us = 0) override

Try to receive data from the file descriptor and forward it for decoding.

Parameters:: timeout_us – if zero, this function does not block. -1 blocks indefinitely.
Returns:: 0 on success, otherwise an errno

stored::IdleCheckLayer

class IdleCheckLayer : public stored::ProtocolLayer 

A layer that tracks if it sees communication through the stack.

This may be used to check of long inactivity on stalled or disconnected communication channels.

Public Types

typedef ProtocolLayer base

Public Functions

inline explicit IdleCheckLayer(ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

virtual ~IdleCheckLayer() override = default

inline virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

inline virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

inline bool idle() const: Checks if both up and down the stack was idle since the last call to setIdle().

inline bool idleDown() const: Checks if downstream was idle since the last call to setIdle().

inline bool idleUp() const: Checks if upstream was idle since the last call to setIdle().

inline void setIdle(): Resets idle flags.

stored::Loopback

class Loopback

Loopback between two protocol stacks.

Public Functions

Loopback(ProtocolLayer &a, ProtocolLayer &b): Constructs a bidirectional loopback of stacks a and b.

~Loopback() = default

void reserve(size_t capacity)

Reserve heap memory to assemble partial messages.

The capacity is allocated twice; one for both directions.

stored::NamedPipeLayer

class NamedPipeLayer : public stored::FileLayer 

Server end of a named pipe.

On Windows, the client end is easier; it is just a file-like create/open/write/close API.

Public Types

enum [anonymous]

Values:

enumerator BufferSize

enumerator Inbound

enumerator Outbound

enumerator Duplex

typedef FileLayer base

Public Functions

NamedPipeLayer(char const *name, DWORD openMode = Duplex, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Ctor for the server part of a named pipe.

The given name is prefixed with \\.\pipe.

virtual ~NamedPipeLayer() override: Dtor.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

Sets lastError() appropriately.

HANDLE handle() const: Returns the pipe handle.

bool isConnected() const: Checks if the pipe is connected.

String::type const &name() const

Returns the full name of the pipe.

This name can be used to open it elsewhere as a normal file.

virtual int recv(long timeout_us = 0) final

Try to receive data from the file descriptor and forward it for decoding.

Parameters:: timeout_us – if zero, this function does not block. -1 blocks indefinitely.
Returns:: 0 on success, otherwise an errno

virtual void reopen(): Resets the connection to accept a new incoming one.

stored::PrintLayer

class PrintLayer : public stored::ProtocolLayer 

Prints all messages to a FILE.

Messages are printed on a line. Decoded message start with <, encoded messages with >, partial encoded messages with *.

Printing can be suspended and resumed by calling enable() or disable(). The default state is enabled.

Mainly for debugging purposes.

Public Types

typedef ProtocolLayer base

Public Functions

explicit PrintLayer(FILE *f = stdout, char const *name = nullptr, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Constructor to print all decoding/encoding messages to the given FILE.

If f is nullptr, printing is suppressed. The name is used as prefix of the printed messages.

See also

setFile()

virtual ~PrintLayer() override = default

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

void disable()

Disable printing all messages.

This is equivalent to calling enable(false).

void enable(bool enable = true): Enable printing all messages.

bool enabled() const: Returns if printing is currently enabled.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

FILE *file() const: Return the FILE that is written to.

void setFile(FILE *f)

Set the FILE to write to.

Parameters:: f – the FILE, set to nullptr to disable output

stored::SegmentationLayer

class SegmentationLayer : public stored::ProtocolLayer 

A layer that performs segmentation of the messages.

Messages to be encoded are split with a maximum chunk size (MTU). At the end of each chunk, either ContinueMarker or the EndMarker is inserted, depending on whether this was the last chunk. Incoming messages are reassembled until the EndMarker is encountered.

This layer assumes a lossless channel; all messages are received in order. If that is not the case for your transport, wrap this layer in the stored::DebugArqLayer or stored::ArqLayer.

Public Types

typedef ProtocolLayer base

Public Functions

explicit SegmentationLayer(size_t mtu = 0, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr): Ctor.

virtual ~SegmentationLayer() override = default: Dtor.

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

size_t lowerMtu() const: Returns the MTU used to split messages into.

virtual size_t mtu() const final

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

virtual void reset() override: Reset the stack (top-down), and drop all messages.

Public Static Attributes

static char const ContinueMarker = 'C'

static char const EndMarker = 'E'

stored::SerialLayer

class SerialLayer : public stored::FileLayer 

A serial port layer.

This is just a FileLayer, but initializes the serial port communication parameters during construction.

Public Types

enum [anonymous]

Values:

enumerator BufferSize

typedef FileLayer base

Public Functions

explicit SerialLayer(char const *name, unsigned long baud, bool rtscts = false, bool xonxoff = false, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

virtual ~SerialLayer() override = default

int resetAutoBaud()

stored::StdioLayer

class StdioLayer : public stored::PolledFileLayer 

A stdin/stdout layer.

Although a Console HANDLE can be read/written as a normal file in Windows, it does not support polling or overlapped IO. Moreover, if stdin/stdout are redirected, the Console becomes a Named Pipe HANDLE. This class handles both.

For POSIX, the StdioLayer is just a FileLayer with preset stdin/stdout file descriptors.

Public Types

enum [anonymous]

Values:

enumerator DefaultBufferSize

typedef PolledFileLayer base

Public Functions

explicit StdioLayer(size_t bufferSize = DefaultBufferSize, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr): Ctor.

virtual ~StdioLayer() override

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual fd_type fd() const override: The file descriptor you may poll before calling recv().

virtual bool isOpen() const override: Checks if the file descriptor is open.

bool isPipeIn() const

bool isPipeOut() const

virtual int recv(long timeout_us = 0) override

Try to receive and decode data.

Returns:: 0 on success, otherwise an errno

stored::SyncZmqLayer

class SyncZmqLayer : public stored::ZmqLayer

Constructs a protocol stack on top of a PAIR ZeroMQ socket, specifically for the stored::Synchronizer.

Public Types

typedef ZmqLayer base

Public Functions

SyncZmqLayer(void *context, char const *endpoint, bool listen, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Constructor.

The given endpoint is used for a DEALER socket. If listen is true, it binds to the endpoint, otherwise it connects to it.

See also

stored::Synchronizer

virtual ~SyncZmqLayer() override = default: Dtor.

stored::TerminalLayer

class TerminalLayer : public stored::ProtocolLayer 

Extracts and injects Embedded Debugger messages in a stream of data, such as a terminal.

The frame’s boundaries are marked with APC and ST C1 control characters.

Subclassed by CaseInverter

Public Types

enum [anonymous]

Values:

enumerator MaxBuffer

typedef ProtocolLayer base

using NonDebugDecodeCallback = void(void *buf, size_t len)

Public Functions

template<typename F, SFINAE_IS_FUNCTION(F, NonDebugDecodeCallback, int) = 0> inline explicit TerminalLayer(F &&cb, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

explicit TerminalLayer(ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

virtual ~TerminalLayer() override

Destructor.

Ties to the layer above and below are nicely removed.

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

virtual void nonDebugEncode(void const *buffer, size_t len)

virtual void reset() override: Reset the stack (top-down), and drop all messages.

Public Static Attributes

static char const Esc = '\x1b'

static char const EscEnd = '\\'

static char const EscStart = '_'

stored::XsimLayer

class XsimLayer : public stored::DoublePipeLayer 

XSIM interaction.

It is based on a DoublePipeLayer, having two named pipes. In VHDL, normal file read/write can be used to pass data. This class also adds keep alive messages, such that a read from the pipe never blocks (which lets XSIM hang), but gets dummy data.

This keep alive uses a third pipe. XSIM is supposed to forward all data it receives to this third pipe. This way, the C++ side knows how many bytes are in flight, and if XSIM would block on the next one. Based on this counter, keep alive bytes may be injected.

Public Types

typedef DoublePipeLayer base

Public Functions

explicit XsimLayer(char const *pipe_prefix, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Ctor.

The pipe_prefix species the name of the pipe in Windows, which will be prepended with \\.\pipe\ . In other POSIX-like systems, the name is used as the filename of the (to be created) FIFO.

virtual ~XsimLayer() override

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

void keepAlive()

virtual int recv(long timeout_us = 0) override

Try to receive and decode data.

Returns:: 0 on success, otherwise an errno

virtual void reopen() override

NamedPipeLayer &req()

virtual void reset() override: Reset the stack (top-down), and drop all messages.

Public Static Attributes

static char const KeepAlive = '\x16'

Abstract classes

stored::ArqLayer

class ArqLayer : public stored::ProtocolLayer 

A general purpose layer that performs Automatic Repeat Request operations on messages.

This layer does not assume a specific message pattern. For stored::Debugger, use stored::DebugArqLayer.

Every message sent has to be acknowledged. There is no window; after sending a message, an ack must be received before continuing. The queue of messages is by default unlimited, but can be set via the constructor. If the limit is hit, the event callback is invoked.

This layer prepends the message with a sequence number byte. The MSb indicates if it is an ack, the 6 LSb are the sequence number. Sequence 0 is special; it resets the connection. It should not be used during normal operation, so the next sequence number after 63 is 1. Messages that do not have a payload (so, no decode() has to be invoked upon receive), should set bit 6. This also applies to the reset message. Bit 6 is implied for an ack.

Retransmits are triggered every time a message is queued for encoding, or when flush() is called. There is no timeout specified.

One may decide to use a stored::SegmentationLayer higher in the protocol stack to reduce the amount of data to retransmit when a message is lost (only one segment is retransmitted, not the full message), but this may add the overhead of the sequence number and round-trip time per segment. If the stored::SegmentationLayer is used below the ArqLayer, normal-case behavior (no packet loss) is most efficient, but the penalty of a retransmit may be higher. It is up to the infrastructure and application requirements what is best.

The layer has no notion of time, or time out for retransmits and acks. The application must call flush() (for the whole stack), or keepAlive() at a regular interval. Every invocation of either function will do a retransmit of the head of the encode queue. If called to often, retransmits may be done before the other party had a chance to respond. If called not often enough, retransmits may take long and communication may be slowed down. Either way, it is functionally correct. Determine for your application what is wise to do.

A reset connection or peer can be recovered from by sending the reset message. The other peer will also reset the communication. To prevent recursive resets, the ack and the reset response must be in the same message; this way the peer knows that the reset was processed properly. Successive resets are not required. A typical flow between peer A and B will be:

* A -> B: reset (0x40)
* B -> A: ack (0x80), reset (0x40)
* A -> B: ack (0x80)
*

Queued messages are retransmitted after the reset, although they may be duplicated when an ack is lost during the reset. Messages are never completely lost.

Public Types

enum [anonymous]

Values:

enumerator RetransmitCallbackThreshold: Number of successive retransmits before the event is emitted.

typedef ProtocolLayer base

enum Event

Values:

enumerator EventNone: No event.

enumerator EventReconnect

An unexpected reset message has been received.

The reset message remains unanswered, until reset() is called. The callback function should probably reinitialize the whole stack.

enumerator EventEncodeBufferOverflow

The maximum buffer capactiy has passed.

The callback may reset the stack to prevent excessive memory usage. Memory allocation will just continue. If no callback function is set (the default), abort() is called when this event happens.

enumerator EventRetransmit

RetransmitCallbackThreshold has been reached on the current message.

This is an indicator that the connection has been lost.

using EventCallback = void(ArqLayer&, Event): Callback type for setEventCallback(F&&).

using EventCallbackArg = void(ArqLayer&, Event, void*): Callback type for setEventCallback(EventCallbackArg*,void*).

Public Functions

explicit ArqLayer(size_t maxEncodeBuffer = 0, ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Ctor.

If maxEncodeBuffer is non-zero, it defines the upper limit of the combined length of all queued messages for encoding. If the limit is hit, the EventEncodeBufferOverflow event is passed to the callback.

virtual ~ArqLayer() override: Dtor.

virtual void connected() override: (Re)connected notification (bottom-up).

virtual void decode(void *buffer, size_t len) override

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

bool didTransmit() const

Checks if a full message has been transmitted.

This function can be used to determine if this layer transmitted anything. For example, when a response is decoded, this function can be used to check if anything has sent back, or the message was dropped. Or, when flush() is called, this flag can be used to check if anything was actually flushed.

To use the function, first call resetDidTransmit(), then execute the code you want to check, and then check didTransmit().

See also

resetDidTransmit()

virtual void encode(void const *buffer, size_t len, bool last = true) override

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

virtual bool flush() override

Flushes all buffered message out of the stack (top-down), if possible.

Any buffered, held back, queued messages are tried to be sent immediately. A flush is always safe; it never destroys data in the stack, it only tries to force it out.

Returns:: true if successful and the stack is empty, or false if message are still blocked

void keepAlive()

Send a keep-alive packet to check the connection.

It actually retransmits the message that is currently processed (waiting for an ack), or sends a dummy message in case the encode queue is empty. Either way, retransmits() and the EventRetransmit can be used afterwards to determine the quality of the link.

virtual size_t mtu() const override

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

virtual void reset() override: Reset the stack (top-down), and drop all messages.

void resetDidTransmit()

Reset the flag for didTransmit().

See also

didTransmit()

size_t retransmits() const

Returns the number of consecutive retransmits of the same message.

Use this function to determine whether the connection is still alive. It is application-defined what the threshold is of too many retransmits.

inline void setEventCallback(EventCallbackArg *cb = nullptr, void *arg = nullptr): Set event callback.

template<typename F> inline setEventCallback(F &&cb): Set event callback.

void shrink_to_fit(): Free all unused memory.

bool waitingForAck() const: Checks if this layer is waiting for an ack.

Public Static Attributes

static uint8_t const AckFlag = 0x80U: Ack flag.

static uint8_t const NopFlag = 0x40U: Flag to indicate that the payload should be ignored.

static uint8_t const SeqMask = 0x3fU: Mask for sequence number.

stored::PolledFileLayer

class PolledFileLayer : public stored::PolledLayer 

A generalized layer that reads from and writes to a file descriptor.

Subclassed by stored::DoublePipeLayer, stored::FileLayer, stored::StdioLayer, stored::ZmqLayer

Public Types

typedef PolledLayer base

typedef int fd_type

Public Functions

virtual ~PolledFileLayer() override: Dtor.

virtual fd_type fd() const = 0: The file descriptor you may poll before calling recv().

stored::PolledLayer

class PolledLayer : public stored::ProtocolLayer 

A generalized layer that needs a call to recv() to get decodable data from somewhere else.

This includes files, sockets, etc. recv() reads data, and passes the data upstream.

Subclassed by stored::FifoLoopback1< Capacity, impl::defaultMessages(Capacity) >, stored::FifoLoopback1< Capacity, Messages >, stored::PolledFileLayer

Public Types

typedef ProtocolLayer base

Public Functions

virtual ~PolledLayer() override

Dtor.

Make sure to close() the related file descriptor prior to destruction.

inline virtual bool isOpen() const: Checks if the file descriptor is open.

inline int lastError() const

Returns the last error of an invoked method of this class.

This is required after construction or encode(), for example, where no error return value is possible.

virtual int recv(long timeout_us = 0) = 0

Try to receive and decode data.

Returns:: 0 on success, otherwise an errno

stored::ProtocolLayer

class ProtocolLayer

Protocol layer base class.

A layer is usually part of the protocol stack. Bytes are decoded and forwarded to the layer above this one, and the layer above sends bytes for encoding down. Moreover, decode() is the inverse of encode(). It is wise to stick to this concept, even though the interface of this class allows more irregular structures, such that decoding and encoding take a different path through the protocol layers.

The implementation of this class does nothing except forwarding bytes. Override encode() and decode() in a subclass.

Subclassed by stored::Stream< false >, LossyChannel, PrintfPhysical, stored::ArqLayer, stored::AsciiEscapeLayer, stored::BufferLayer, stored::CallbackLayer< Up, Down, Connected >, stored::CompressLayer, stored::Crc16Layer, stored::Crc8Layer, stored::DebugArqLayer, stored::Debugger, stored::FrameMerger, stored::IdleCheckLayer, stored::PolledLayer, stored::PrintLayer, stored::SegmentationLayer, stored::Stream< Compress >, stored::Stream< true >, stored::SyncConnection, stored::TerminalLayer, stored::XsimLayer::DecodeCallback, stored::impl::Loopback1

Public Functions

inline explicit ProtocolLayer(ProtocolLayer *up = nullptr, ProtocolLayer *down = nullptr)

Constructor.

Parameters:

up – the layer above, which receives our decoded frames
down – the layer below, which receives our encoded frames

virtual ~ProtocolLayer()

Destructor.

Ties to the layer above and below are nicely removed.

inline ProtocolLayer &bottom(): Return the lowest layer of the stack.

inline ProtocolLayer const &bottom() const: Return the lowest layer of the stack.

inline virtual void connected(): (Re)connected notification (bottom-up).

inline virtual void decode(void *buffer, size_t len)

Decode a frame and forward the decoded frame to the upper layer.

The given buffer may be decoded in-place.

inline ProtocolLayer *down() const

Returns the layer below this one.

Returns:: the layer, or nullptr if there is none.

inline void encode(): Encodes the last part of the current frame.

inline virtual void encode(void const *buffer, size_t len, bool last = true)

Encode a (partial) frame and forward it to the lower layer.

The given buffer will not be modified. A new buffer is allocated when required.

inline virtual bool flush()

Flushes all buffered message out of the stack (top-down), if possible.

Any buffered, held back, queued messages are tried to be sent immediately. A flush is always safe; it never destroys data in the stack, it only tries to force it out.

Returns:: true if successful and the stack is empty, or false if message are still blocked

inline virtual size_t mtu() const

Returns the maximum amount of data to be put in one message that is encoded.

If there is a MTU applicable to the physical transport (like a CAN bus), override this method to reflect that value. Layers on top will decrease the MTU when there protocol adds headers, for example.

Returns:: the number of bytes, or 0 for infinity

inline virtual void reset(): Reset the stack (top-down), and drop all messages.

inline void setDown(ProtocolLayer *down = nullptr)

Change the layer that receives our encoded frames.

Parameters:: down – the layer, which can be nullptr

inline virtual void setPurgeableResponse(bool purgeable = true)

Flags the current response as purgeable.

This may influence how a response is handled. Especially, in case of retransmits of lost packets, one may decide to either reexecute the command, or to save the first response and resend it when the command was retransmitted. In that sense, a precious response (default) means that every layer should handle the data with case, as it cannot be recovered once it is lost. When the response is flagged purgeeble, the response may be thrown away after the first try to transmit it to the client.

By default, all responses are precious.

inline void setUp(ProtocolLayer *up = nullptr)

Change the layer that receives our decoded frames.

Parameters:: up – the layer, which can be nullptr

inline ProtocolLayer &stack(ProtocolLayer &down)

Sets the up/down layers of this layer and the given layer, such that this layer is stacked on (or wrapped by) the given one.

If the given layer was not the top of the stack, this layer injects itself between the given layer and its stacked one.

Returns:: the new top layer of the stack.

inline ProtocolLayer &top(): Return the highest layer of the stack.

inline ProtocolLayer const &top() const: Return the highest layer of the stack.

inline ProtocolLayer *up() const

Returns the layer above this one.

Returns:: the layer, or nullptr if there is none.

inline ProtocolLayer &wrap(ProtocolLayer &up)

Sets the up/down layers of this layer and the given layer, such that this layer wraps the given one.

If the given layer was not the bottom of the stack, this layer injects itself in between the given layer and its wrapper.

Returns:: the new bottom layer of the stack