Streams¶
Streams are used to send data in ERDOS applications.
ERDOS streams are similar to ROS topics, but have a few additional desirable properties. Streams facilitate one-to-many communication, so only 1 operator sends messages on a stream. ERDOS broadcasts messages sent on a stream to all connected operators. In addition, streams are typed when using the Rust API.
Streams expose 3 classes of interfaces:
Read-interfaces expose methods to receive and process data. They allow pulling data by calling
read()
andtry_read()
. Structures that implement read interfaces include:
ReadStream
: used by operators to read data and register callbacks.
ExtractStream
: used by the driver to read data.
Write-interfaces expose the send method to send data on a stream. Structures that implement write interfaces include:
WriteStream
: used by operators to send data.
IngestStream
: used by the driver to send data.
Abstract interfaces used to connect operators and construct a dataflow graph. Structures that implement the abstract :py:class:.Stream interface include:
OperatorStream
: representing a stream on which an operator sends messages.IngestStream
: used to send messages to operators from the driver.LoopStream
: used to create loops in the dataflow graph.
Some applications may want to introduce loops in their dataflow graphs which
is possible using the LoopStream
.
Sending Messages¶
Operators use Write Streams to send data.
- class erdos.WriteStream(_py_write_stream)¶
A
WriteStream
allows an operator to send messages and watermarks to other operators that connect to the correspondingReadStream
.Note
This class is created automatically when ERDOS initializes an operator, and should never be initialized manually.
Receiving Messages¶
Operators receive data by reading messages from Read Streams. Operators also receive data by implementing callbacks that are automatically invoked upon the receipt of a message.
- class erdos.ReadStream(_py_read_stream)¶
A
ReadStream
allows an operator to read and do work on data sent by other operators on a correspondingWriteStream
.An operator that takes control of its execution using the
run
method can retrieve the messages on aReadStream
using theReadStream.read()
orReadStream.try_read()
methods.Note
This class is created automatically during
run
, and should never be initialized manually. No callbacks are invoked if an operator takes control of the execution inrun
.
Abstract Streams¶
These streams represent edges in the dataflow graph, which ERDOS materializes using its communication protocols, and the :py:class:.ReadStream and :py:class:.WriteStream interfaces.
- class erdos.Stream(internal_stream)¶
Base class representing a stream to operators can be connected. from which is subclassed by streams that are used to connect operators in the driver.
Note
This class should never be initialized manually.
- property id: str¶
The id of the stream.
- Return type
str
- property name: str¶
The name of the stream. The stream ID if none was given.
- Return type
str
- class erdos.OperatorStream(operator_stream)¶
Bases:
erdos.streams.Stream
Returned when connecting an operator to the dataflow graph.
Note
This class is created automatically by the connect functions, and should never be initialized manually.
Ingesting and Extracting Data¶
Some applications have trouble placing all of the data processing logic inside operators. For these applications, ERDOS provides special stream interfaces to ingest and extract data.
A comprehensive example is available here.
- class erdos.IngestStream(name=None)¶
Bases:
erdos.streams.Stream
An
IngestStream
enables drivers to inject data into a running ERDOS application.The driver can initialize a new
IngestStream
and connect it to an operator through theconnect
family of functions. Similar to aWriteStream
, anIngestStream
provides aIngestStream.send()
to enable the driver to send data to the operator to which it was connected.- send(msg)¶
Sends a message on the stream.
- Parameters
msg (
Message
) – the message to send. This may be aWatermarkMessage
or aMessage
.
- class erdos.ExtractStream(stream)¶
An
ExtractStream
enables drivers to read data from a running ERDOS applications.The driver can initialize a new
ExtractStream
by passing the instance ofOperatorStream
returned by theconnect
family of functions. Similar to aReadStream
, anExtractStream
providesread()
andtry_read()
for reading data published on the correspondingOperatorStream
.- Parameters
stream (
OperatorStream
) – The stream from which to read messages.
Loops¶
Certain applications require feedback in the dataflow. To support this use case, ERDOS provides the LoopStream interface to support loops in the dataflow.
A comprehensive example is available here.
- class erdos.LoopStream¶
Bases:
erdos.streams.Stream
Stream placeholder used to construct loops in the dataflow graph.
Note
Must call connect_loop with a valid
OperatorStream
to complete the loop.