Extend Rogue With C++
Python is fast enough for most stream work, because the heavy lifting already happens in Rogue’s C++ core. But when a per-frame transform sits in the data path at high rate, the Python callback becomes the bottleneck — and the fix is to write that stage in C++ and expose it to Python.
This tutorial builds a working C++ stream module, loads it into Python, and wraps
it in a Device. Rather than inventing new code, it walks the module Rogue’s
own continuous integration compiles as a downstream build test, so the source is
guaranteed to work.
Note
No hardware required. The module is a pure stream transform, driven by a Python source and sink.
Prerequisites
Finish Build Your First Rogue Application, and read
Sending Frames and Receiving Frames — the C++
class implements the same Master and Slave roles in native code.
You also need a C++ toolchain (g++), cmake, and a Rogue installation
built with Python support.
The Module
The source lives at tests/downstream/src/BitInverter.cpp in the Rogue
repository. It inverts every payload byte (byte ^ 0xFF) and forwards the
result downstream, subclassing both Master and Slave so it can sit in the
middle of a pipeline.
Four patterns in that file are worth understanding, because every custom module repeats them:
A static
create()factory returns astd::shared_ptr. That is the ownership model Rogue uses throughout its graph — never construct these objects on the stack.acceptFrame()is the single ingress hook. It takes the frame lock, walks the payload with aFrameIterator(which transparently spans the frame’s underlying buffers), mutates bytes in place, releases the lock, and callssendFrame()inherited fromMaster.The Python bindings are guarded by
#ifndef NO_PYTHON, so the same source compiles into a Python-free C++ build.BOOST_PYTHON_MODULEregisters the bindings. Rogue manages the GIL in its own bindings, so no explicit thread initialisation is needed.
The full annotated source is in Custom Module Source.
Building It
find_package(Rogue) supplies the include paths and libraries. Configure
against the Rogue installation you will import from — see
Troubleshooting for why that matters:
cd tests/downstream
cmake -B build \
-DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
-DCMAKE_SHARED_LINKER_FLAGS="-L$CONDA_PREFIX/lib"
cmake --build build
The linker flag is needed because RogueConfig.cmake lists the core library
by bare name (rogue-core) without a search path, so the linker fails with
cannot find -lrogue-core unless you supply one. If your Rogue lives
somewhere other than a conda environment, substitute that prefix in both places.
This produces BitInverter.so. Put its directory on PYTHONPATH, and make
sure the Rogue library is findable at run time:
export PYTHONPATH=$PWD/build:$PYTHONPATH
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
Using It From Python
The compiled class behaves exactly like a built-in stream module — it connects
with >> and needs no wrapper to function. Drive it with a Python source and
sink:
class ByteSource(rogue.interfaces.stream.Master):
"""Sends one frame carrying the supplied bytes."""
def sendBytes(self, data):
frame = self._reqFrame(len(data), True)
with frame.lock():
frame.write(bytes(data))
self._sendFrame(frame)
class ByteCapture(rogue.interfaces.stream.Slave):
"""Keeps the payload of every frame it receives."""
def __init__(self):
super().__init__()
self.frames = []
def _acceptFrame(self, frame):
with frame.lock():
buffer = bytearray(frame.getPayload())
frame.read(buffer, 0)
self.frames.append(bytes(buffer))
Then put the C++ module between them:
def invertThroughModule(payload):
"""Push bytes through the compiled module and return what comes out."""
import BitInverter
source = ByteSource()
capture = ByteCapture()
source >> BitInverter.BitInverter() >> capture
source.sendBytes(payload)
time.sleep(0.5)
return capture.frames[0] if capture.frames else None
input : 000fff
output : fff000
correct: True
0x00 became 0xFF, 0x0F became 0xF0, 0xFF became 0x00.
A Python Master fed a C++ transform which fed a Python Slave — mixing
languages freely in one pipeline is the point of the binding layer.
Wrapping It In A Device
The C++ object is a stream endpoint, not a Device, so it has no place in a
tree by itself. A thin wrapper gives it a name, a description, and somewhere to
expose status:
class BitInverterDevice(pr.Device):
"""Wraps the C++ module so it belongs to a tree.
The C++ object is a stream endpoint, not a Device, so it has no place in a
tree by itself. A thin Device wrapper gives it a name, a description, and
somewhere to hang status Variables -- and exposes the raw object so stream
connections can still be made.
"""
def __init__(self, inverter, **kwargs):
super().__init__(description='C++ bit-inverting stream module', **kwargs)
self._inverter = inverter
self.add(pr.LocalVariable(
name = 'Description',
description = 'What the underlying C++ module does',
mode = 'RO',
value = 'Inverts every payload byte (b ^ 0xFF)'))
def __rshift__(self, other):
"""Let the wrapper sit in a ``a >> wrapper >> b`` chain."""
self._inverter >> other
return other
def __lshift__(self, other):
self._inverter << other
return other
The __rshift__/__lshift__ overrides let the wrapper stand in for the raw
object inside a a >> wrapper >> b chain, so tree membership does not cost you
the connection syntax. Wrapping Custom Module With PyRogue Device covers richer wrappers that
surface C++ counters as PyRogue variables.
Troubleshooting
Two failures are worth calling out, because both produce errors that do not obviously point at their cause.
“extension class wrapper … has not been created yet”
RuntimeError: extension class wrapper for base class
rogue::interfaces::stream::Master has not been created yet
The module was compiled against one Rogue installation and imported alongside a
different one. Boost.Python registers a class hierarchy per library instance, so
two copies of librogue-core mean the base class your module inherits from is
not the one Python already knows about.
This is easy to hit when a source-tree build and a packaged install coexist. Find out how many you have:
find / -name 'librogue-core.so*' 2>/dev/null
Then make sure CMAKE_PREFIX_PATH at build time, LD_LIBRARY_PATH at run
time, and the rogue package Python actually imports all refer to the same
installation.
“No rule to make target …/lib/librogue-core.so”
make: *** No rule to make target '<path>/lib/librogue-core.so.v6.15',
needed by 'BitInverter.so'. Stop.
RogueConfig.cmake records an absolute library path assuming an installed
layout, with the library under lib/. A plain in-place source build leaves it
in the build directory instead, so that path does not exist. Point
CMAKE_PREFIX_PATH at a proper installation — a conda environment, or the
result of make install — rather than at a raw build tree.
When To Reach For C++
Situation |
Recommendation |
|---|---|
Prototyping any stream stage |
Start in Python; it is far quicker to iterate |
Python stage measurably limiting throughput |
Port that one stage to C++ |
Per-frame work on a high-rate path |
C++ from the outset |
Orchestration, configuration, tree logic |
Keep it in Python regardless of rate |
Measure before porting. A Python Slave handles modest frame rates without
difficulty, and a C++ module you did not need is a maintenance cost with no
return.
Where To Go Next
Creating Custom Modules — the reference treatment of this workflow.
Custom Module Source — the annotated C++ source.
Building The Custom Module — build configuration in detail.
The RogueConfig.cmake Package — the variables
find_package(Rogue)exports.Testing The Custom Module — testing custom modules.
C++ API — the C++ API reference.