Capture Data To Disk And Replay It

Build Your First Rogue Application built a stream that validated itself and threw the data away. Real acquisition systems keep it. This tutorial writes a stream to disk, reads it back, and measures what compression buys.

Note

No hardware required. The PRBS generator produces test frames in software, so the whole capture and replay path is exercised without a detector or a DMA card.

Warning

PrbsTx runs as fast as the CPU allows. During development of this tutorial, an unthrottled one-second capture wrote 281 MB. Every example here includes a RateDrop stage. Keep it, or expect to fill a disk.

Prerequisites

Finish Build Your First Rogue Application. You should recognise PrbsTx and the >> stream operator.

Writing A Capture

Three stages: a generator, a rate limiter, and a writer.

class CaptureRoot(pr.Root):
    """PRBS generator -> rate limiter -> file writer.

    The ``RateDrop`` stage matters. An unthrottled ``PrbsTx`` saturates the
    CPU and writes hundreds of megabytes per second, which is useful for
    benchmarking and hostile to a tutorial. ``RateDrop(True, period)`` forwards
    at most one frame per ``period`` seconds.
    """

    def __init__(self, period=0.01, **kwargs):
        super().__init__(description='Throttled capture to disk', **kwargs)

        self._prbs = pr.utilities.prbs.PrbsTx(name='PrbsTx')
        self.add(self._prbs)

        # True selects time-based dropping; period is in seconds.
        self._rate = rogue.interfaces.stream.RateDrop(True, period)

        self._writer = pr.utilities.fileio.StreamWriter(name='Writer')
        self.add(self._writer)

        self._prbs >> self._rate >> self._writer.getChannel(0)

    def captureTo(self, path, seconds=1.0):
        """Record for a fixed wall-clock duration, then close the file.

        The file must be closed before it is read: the writer buffers, so a
        reader opening it early sees a truncated final record.
        """
        self.Writer.DataFile.set(path)
        self.Writer.Open()

        self.PrbsTx.txEnable.set(True)
        time.sleep(seconds)
        self.PrbsTx.txEnable.set(False)

        self.Writer.Close()
        return os.path.getsize(path)

RateDrop(True, period) selects time-based limiting — at most one frame per period seconds. Passing False instead switches to count-based dropping, forwarding one frame in every N. Time-based is usually what you want for a monitor or a sampled capture; count-based suits decimating a known frame rate.

StreamWriter is a Device, so it joins the tree with self.add() and exposes its controls as variables. Note that getChannel(0) is what you connect a stream to — a single writer can capture several channels into one file, tagging each record with its channel number.

Opening and closing the file is done through variables and commands rather than Python methods:

    def captureTo(self, path, seconds=1.0):
        """Record for a fixed wall-clock duration, then close the file.

        The file must be closed before it is read: the writer buffers, so a
        reader opening it early sees a truncated final record.
        """
        self.Writer.DataFile.set(path)
        self.Writer.Open()

        self.PrbsTx.txEnable.set(True)
        time.sleep(seconds)
        self.PrbsTx.txEnable.set(False)

        self.Writer.Close()
        return os.path.getsize(path)

Important

The API is DataFile.set(path) followed by the Open() command — not open(path). Likewise Close(), capitalised. Lowercase open raises AttributeError with a hint pointing at Open.

Closing matters. The writer buffers, so a reader that opens the file before Close() sees a truncated final record.

Open() always creates a new file. If the DataFile path already exists the command raises an error and leaves the existing file untouched, so rerunning a capture script with the same path fails on the second run. Either point DataFile at a fresh path each run, call AutoName() to generate a timestamped filename, or remove the previous output first.

Replaying A Capture

FileReader iterates records rather than raw bytes, handing back one frame’s payload at a time with its channel intact:

def countRecords(path, limit=None):
    """Replay a capture file and count the records inside it.

    ``FileReader`` iterates records rather than raw bytes, so each iteration
    hands back one frame's worth of payload with its channel intact.
    """
    reader = pr.utilities.fileio.FileReader(path)
    count = 0
    for _ in reader.records():
        count += 1
        if limit is not None and count >= limit:
            break
    return count
plain capture bytes      : 103200
records read back        : 100

One hundred records for a one-second capture at a 0.01 s period — exactly what the rate limiter should produce. That arithmetic is worth checking in your own captures; if the record count does not match the expected rate, the limiter is not doing what you think.

Adding Compression

StreamZip compresses each frame in flight, and StreamUnZip reverses it. Insert the compressor before the writer:

class CompressedCaptureRoot(pr.Root):
    """The same capture path with compression inserted before the writer.

    ``StreamZip`` compresses each frame as it passes. PRBS data is
    pseudo-random and therefore close to incompressible, so expect little or
    no saving here -- real detector payloads with headers, padding, or idle
    regions compress far better.
    """

    def __init__(self, period=0.01, **kwargs):
        super().__init__(description='Compressed capture to disk', **kwargs)

        self._prbs = pr.utilities.prbs.PrbsTx(name='PrbsTx')
        self.add(self._prbs)

        self._rate = rogue.interfaces.stream.RateDrop(True, period)
        self._zip = rogue.utilities.StreamZip()

        self._writer = pr.utilities.fileio.StreamWriter(name='Writer')
        self.add(self._writer)

        self._prbs >> self._rate >> self._zip >> self._writer.getChannel(0)

    def captureTo(self, path, seconds=1.0):
        self.Writer.DataFile.set(path)
        self.Writer.Open()
        self.PrbsTx.txEnable.set(True)
        time.sleep(seconds)
        self.PrbsTx.txEnable.set(False)
        self.Writer.Close()
        return os.path.getsize(path)
plain capture bytes      : 103200
compressed capture bytes : 96819

About 6% — an honest result that is worth understanding rather than hiding. PRBS data is pseudo-random, and random data is close to incompressible. This is the worst case for a compressor, and it is exactly why you should measure with your own payloads instead of trusting a documented ratio. Real detector frames with headers, fixed padding, zero-suppressed regions, or idle patterns often compress by large factors.

Compression is lossless regardless of ratio. Proving that does not need a file at all — put the two stages back to back and compare bytes:

def roundTripCheck(payload=b'ABCD' * 256):
    """Prove StreamZip/StreamUnZip is lossless, without involving a file.

    Note there is no wait here. A Rogue stream connection is a direct call
    chain, not a queue: ``_sendFrame`` runs each downstream ``_acceptFrame``
    on the calling thread, so by the time it returns the payload has already
    been compressed, decompressed, and captured. Stages that *do* hand frames
    to a thread -- ``Fifo``, ``TcpClient``, the batcher -- are the ones that
    need you to wait for a completion signal.
    """

    class Source(rogue.interfaces.stream.Master):
        def sendBytes(self, data):
            frame = self._reqFrame(len(data), True)
            with frame.lock():
                frame.write(data)
            self._sendFrame(frame)

    class Capture(rogue.interfaces.stream.Slave):
        def __init__(self):
            super().__init__()
            self.data = None

        def _acceptFrame(self, frame):
            with frame.lock():
                buffer = bytearray(frame.getPayload())
                frame.read(buffer, 0)
                self.data = bytes(buffer)

    source = Source()
    capture = Capture()
    source >> rogue.utilities.StreamZip() >> rogue.utilities.StreamUnZip() >> capture

    source.sendBytes(payload)
    return capture.data == payload
zip -> unzip is lossless : True

Note that reading a file written by StreamZip requires decompressing it: either insert StreamUnZip in the replay path, or keep captures uncompressed if you want them directly readable by other tools.

Running It

python3 file_capture.py
plain capture bytes      : 103200
records read back        : 100
compressed capture bytes : 96819
zip -> unzip is lossless : True

Sizes vary slightly between runs, since the capture is bounded by wall-clock time rather than frame count.

Choosing A Capture Strategy

Goal

Approach

Capture everything at full rate

No RateDrop; watch disk space and use MaxFileSize

Sampled monitoring

RateDrop(True, period) — one frame per interval

Decimate a known rate

RateDrop(False, N) — one frame in N

Long unattended runs

Set MaxFileSize and enable AutoName to roll files

Rerunning the same script

Use AutoName or a new DataFile path; Open refuses an existing file

Compressible payloads

Add StreamZip; measure your own ratio first

Where To Go Next