Extend Rogue With C++

Python is fast enough for most stream work, because the heavy lifting already happens in Rogue’s C++ core. But when a per-frame transform sits in the data path at high rate, the Python callback becomes the bottleneck — and the fix is to write that stage in C++ and expose it to Python.

This tutorial builds a working C++ stream module, loads it into Python, and wraps it in a Device. Rather than inventing new code, it walks the module Rogue’s own continuous integration compiles as a downstream build test, so the source is guaranteed to work.

Note

No hardware required. The module is a pure stream transform, driven by a Python source and sink.

Prerequisites

Finish Build Your First Rogue Application, and read Sending Frames and Receiving Frames — the C++ class implements the same Master and Slave roles in native code.

You also need a C++ toolchain (g++), cmake, and a Rogue installation built with Python support.

The Module

The source lives at tests/downstream/src/BitInverter.cpp in the Rogue repository. It inverts every payload byte (byte ^ 0xFF) and forwards the result downstream, subclassing both Master and Slave so it can sit in the middle of a pipeline.

Four patterns in that file are worth understanding, because every custom module repeats them:

  • A static create() factory returns a std::shared_ptr. That is the ownership model Rogue uses throughout its graph — never construct these objects on the stack.

  • acceptFrame() is the single ingress hook. It takes the frame lock, walks the payload with a FrameIterator (which transparently spans the frame’s underlying buffers), mutates bytes in place, releases the lock, and calls sendFrame() inherited from Master.

  • The Python bindings are guarded by #ifndef NO_PYTHON, so the same source compiles into a Python-free C++ build.

  • BOOST_PYTHON_MODULE registers the bindings. Rogue manages the GIL in its own bindings, so no explicit thread initialisation is needed.

The full annotated source is in Custom Module Source.

Building It

find_package(Rogue) supplies the include paths and libraries. Configure against the Rogue installation you will import from — see Troubleshooting for why that matters:

cd tests/downstream
cmake -B build \
    -DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
    -DCMAKE_SHARED_LINKER_FLAGS="-L$CONDA_PREFIX/lib"
cmake --build build

The linker flag is needed because RogueConfig.cmake lists the core library by bare name (rogue-core) without a search path, so the linker fails with cannot find -lrogue-core unless you supply one. If your Rogue lives somewhere other than a conda environment, substitute that prefix in both places.

This produces BitInverter.so. Put its directory on PYTHONPATH, and make sure the Rogue library is findable at run time:

export PYTHONPATH=$PWD/build:$PYTHONPATH
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

Using It From Python

The compiled class behaves exactly like a built-in stream module — it connects with >> and needs no wrapper to function. Drive it with a Python source and sink:

class ByteSource(rogue.interfaces.stream.Master):
    """Sends one frame carrying the supplied bytes."""

    def sendBytes(self, data):
        frame = self._reqFrame(len(data), True)
        with frame.lock():
            frame.write(bytes(data))
        self._sendFrame(frame)
class ByteCapture(rogue.interfaces.stream.Slave):
    """Keeps the payload of every frame it receives."""

    def __init__(self):
        super().__init__()
        self.frames = []

    def _acceptFrame(self, frame):
        with frame.lock():
            buffer = bytearray(frame.getPayload())
            frame.read(buffer, 0)
            self.frames.append(bytes(buffer))

Then put the C++ module between them:

def invertThroughModule(payload):
    """Push bytes through the compiled module and return what comes out."""
    import BitInverter

    source = ByteSource()
    capture = ByteCapture()

    source >> BitInverter.BitInverter() >> capture
    source.sendBytes(payload)
    time.sleep(0.5)

    return capture.frames[0] if capture.frames else None
input  : 000fff
output : fff000
correct: True

0x00 became 0xFF, 0x0F became 0xF0, 0xFF became 0x00. A Python Master fed a C++ transform which fed a Python Slave — mixing languages freely in one pipeline is the point of the binding layer.

Wrapping It In A Device

The C++ object is a stream endpoint, not a Device, so it has no place in a tree by itself. A thin wrapper gives it a name, a description, and somewhere to expose status:

class BitInverterDevice(pr.Device):
    """Wraps the C++ module so it belongs to a tree.

    The C++ object is a stream endpoint, not a Device, so it has no place in a
    tree by itself. A thin Device wrapper gives it a name, a description, and
    somewhere to hang status Variables -- and exposes the raw object so stream
    connections can still be made.
    """

    def __init__(self, inverter, **kwargs):
        super().__init__(description='C++ bit-inverting stream module', **kwargs)
        self._inverter = inverter

        self.add(pr.LocalVariable(
            name        = 'Description',
            description = 'What the underlying C++ module does',
            mode        = 'RO',
            value       = 'Inverts every payload byte (b ^ 0xFF)'))

    def __rshift__(self, other):
        """Let the wrapper sit in a ``a >> wrapper >> b`` chain."""
        self._inverter >> other
        return other

    def __lshift__(self, other):
        self._inverter << other
        return other

The __rshift__/__lshift__ overrides let the wrapper stand in for the raw object inside a a >> wrapper >> b chain, so tree membership does not cost you the connection syntax. Wrapping Custom Module With PyRogue Device covers richer wrappers that surface C++ counters as PyRogue variables.

Troubleshooting

Two failures are worth calling out, because both produce errors that do not obviously point at their cause.

“extension class wrapper … has not been created yet”

RuntimeError: extension class wrapper for base class
rogue::interfaces::stream::Master has not been created yet

The module was compiled against one Rogue installation and imported alongside a different one. Boost.Python registers a class hierarchy per library instance, so two copies of librogue-core mean the base class your module inherits from is not the one Python already knows about.

This is easy to hit when a source-tree build and a packaged install coexist. Find out how many you have:

find / -name 'librogue-core.so*' 2>/dev/null

Then make sure CMAKE_PREFIX_PATH at build time, LD_LIBRARY_PATH at run time, and the rogue package Python actually imports all refer to the same installation.

“No rule to make target …/lib/librogue-core.so”

make: *** No rule to make target '<path>/lib/librogue-core.so.v6.15',
needed by 'BitInverter.so'.  Stop.

RogueConfig.cmake records an absolute library path assuming an installed layout, with the library under lib/. A plain in-place source build leaves it in the build directory instead, so that path does not exist. Point CMAKE_PREFIX_PATH at a proper installation — a conda environment, or the result of make install — rather than at a raw build tree.

When To Reach For C++

Situation

Recommendation

Prototyping any stream stage

Start in Python; it is far quicker to iterate

Python stage measurably limiting throughput

Port that one stage to C++

Per-frame work on a high-rate path

C++ from the outset

Orchestration, configuration, tree logic

Keep it in Python regardless of rate

Measure before porting. A Python Slave handles modest frame rates without difficulty, and a C++ module you did not need is a maintenance cost with no return.

Where To Go Next