Capture Data To Disk And Replay It
Build Your First Rogue Application built a stream that validated itself and threw the data away. Real acquisition systems keep it. This tutorial writes a stream to disk, reads it back, and measures what compression buys.
Note
No hardware required. The PRBS generator produces test frames in software, so the whole capture and replay path is exercised without a detector or a DMA card.
Warning
PrbsTx runs as fast as the CPU allows. During development of this
tutorial, an unthrottled one-second capture wrote 281 MB. Every example
here includes a RateDrop stage. Keep it, or expect to fill a disk.
Prerequisites
Finish Build Your First Rogue Application. You should recognise PrbsTx and the
>> stream operator.
Writing A Capture
Three stages: a generator, a rate limiter, and a writer.
class CaptureRoot(pr.Root):
"""PRBS generator -> rate limiter -> file writer.
The ``RateDrop`` stage matters. An unthrottled ``PrbsTx`` saturates the
CPU and writes hundreds of megabytes per second, which is useful for
benchmarking and hostile to a tutorial. ``RateDrop(True, period)`` forwards
at most one frame per ``period`` seconds.
"""
def __init__(self, period=0.01, **kwargs):
super().__init__(description='Throttled capture to disk', **kwargs)
self._prbs = pr.utilities.prbs.PrbsTx(name='PrbsTx')
self.add(self._prbs)
# True selects time-based dropping; period is in seconds.
self._rate = rogue.interfaces.stream.RateDrop(True, period)
self._writer = pr.utilities.fileio.StreamWriter(name='Writer')
self.add(self._writer)
self._prbs >> self._rate >> self._writer.getChannel(0)
def captureTo(self, path, seconds=1.0):
"""Record for a fixed wall-clock duration, then close the file.
The file must be closed before it is read: the writer buffers, so a
reader opening it early sees a truncated final record.
"""
self.Writer.DataFile.set(path)
self.Writer.Open()
self.PrbsTx.txEnable.set(True)
time.sleep(seconds)
self.PrbsTx.txEnable.set(False)
self.Writer.Close()
return os.path.getsize(path)
RateDrop(True, period) selects time-based limiting — at most one frame per
period seconds. Passing False instead switches to count-based dropping,
forwarding one frame in every N. Time-based is usually what you want for a
monitor or a sampled capture; count-based suits decimating a known frame rate.
StreamWriter is a Device, so it joins the tree with self.add() and
exposes its controls as variables. Note that getChannel(0) is what you
connect a stream to — a single writer can capture several channels into one file,
tagging each record with its channel number.
Opening and closing the file is done through variables and commands rather than Python methods:
def captureTo(self, path, seconds=1.0):
"""Record for a fixed wall-clock duration, then close the file.
The file must be closed before it is read: the writer buffers, so a
reader opening it early sees a truncated final record.
"""
self.Writer.DataFile.set(path)
self.Writer.Open()
self.PrbsTx.txEnable.set(True)
time.sleep(seconds)
self.PrbsTx.txEnable.set(False)
self.Writer.Close()
return os.path.getsize(path)
Important
The API is DataFile.set(path) followed by the Open() command — not
open(path). Likewise Close(), capitalised. Lowercase open raises
AttributeError with a hint pointing at Open.
Closing matters. The writer buffers, so a reader that opens the file before
Close() sees a truncated final record.
Open() always creates a new file. If the DataFile path already exists
the command raises an error and leaves the existing file untouched, so rerunning
a capture script with the same path fails on the second run. Either point
DataFile at a fresh path each run, call AutoName() to generate a
timestamped filename, or remove the previous output first.
Replaying A Capture
FileReader iterates records rather than raw bytes, handing back one frame’s
payload at a time with its channel intact:
def countRecords(path, limit=None):
"""Replay a capture file and count the records inside it.
``FileReader`` iterates records rather than raw bytes, so each iteration
hands back one frame's worth of payload with its channel intact.
"""
reader = pr.utilities.fileio.FileReader(path)
count = 0
for _ in reader.records():
count += 1
if limit is not None and count >= limit:
break
return count
plain capture bytes : 103200
records read back : 100
One hundred records for a one-second capture at a 0.01 s period — exactly what the rate limiter should produce. That arithmetic is worth checking in your own captures; if the record count does not match the expected rate, the limiter is not doing what you think.
Adding Compression
StreamZip compresses each frame in flight, and StreamUnZip reverses it.
Insert the compressor before the writer:
class CompressedCaptureRoot(pr.Root):
"""The same capture path with compression inserted before the writer.
``StreamZip`` compresses each frame as it passes. PRBS data is
pseudo-random and therefore close to incompressible, so expect little or
no saving here -- real detector payloads with headers, padding, or idle
regions compress far better.
"""
def __init__(self, period=0.01, **kwargs):
super().__init__(description='Compressed capture to disk', **kwargs)
self._prbs = pr.utilities.prbs.PrbsTx(name='PrbsTx')
self.add(self._prbs)
self._rate = rogue.interfaces.stream.RateDrop(True, period)
self._zip = rogue.utilities.StreamZip()
self._writer = pr.utilities.fileio.StreamWriter(name='Writer')
self.add(self._writer)
self._prbs >> self._rate >> self._zip >> self._writer.getChannel(0)
def captureTo(self, path, seconds=1.0):
self.Writer.DataFile.set(path)
self.Writer.Open()
self.PrbsTx.txEnable.set(True)
time.sleep(seconds)
self.PrbsTx.txEnable.set(False)
self.Writer.Close()
return os.path.getsize(path)
plain capture bytes : 103200
compressed capture bytes : 96819
About 6% — an honest result that is worth understanding rather than hiding. PRBS data is pseudo-random, and random data is close to incompressible. This is the worst case for a compressor, and it is exactly why you should measure with your own payloads instead of trusting a documented ratio. Real detector frames with headers, fixed padding, zero-suppressed regions, or idle patterns often compress by large factors.
Compression is lossless regardless of ratio. Proving that does not need a file at all — put the two stages back to back and compare bytes:
def roundTripCheck(payload=b'ABCD' * 256):
"""Prove StreamZip/StreamUnZip is lossless, without involving a file.
Note there is no wait here. A Rogue stream connection is a direct call
chain, not a queue: ``_sendFrame`` runs each downstream ``_acceptFrame``
on the calling thread, so by the time it returns the payload has already
been compressed, decompressed, and captured. Stages that *do* hand frames
to a thread -- ``Fifo``, ``TcpClient``, the batcher -- are the ones that
need you to wait for a completion signal.
"""
class Source(rogue.interfaces.stream.Master):
def sendBytes(self, data):
frame = self._reqFrame(len(data), True)
with frame.lock():
frame.write(data)
self._sendFrame(frame)
class Capture(rogue.interfaces.stream.Slave):
def __init__(self):
super().__init__()
self.data = None
def _acceptFrame(self, frame):
with frame.lock():
buffer = bytearray(frame.getPayload())
frame.read(buffer, 0)
self.data = bytes(buffer)
source = Source()
capture = Capture()
source >> rogue.utilities.StreamZip() >> rogue.utilities.StreamUnZip() >> capture
source.sendBytes(payload)
return capture.data == payload
zip -> unzip is lossless : True
Note that reading a file written by StreamZip requires decompressing it:
either insert StreamUnZip in the replay path, or keep captures uncompressed
if you want them directly readable by other tools.
Running It
python3 file_capture.py
plain capture bytes : 103200
records read back : 100
compressed capture bytes : 96819
zip -> unzip is lossless : True
Sizes vary slightly between runs, since the capture is bounded by wall-clock time rather than frame count.
Choosing A Capture Strategy
Goal |
Approach |
|---|---|
Capture everything at full rate |
No |
Sampled monitoring |
|
Decimate a known rate |
|
Long unattended runs |
Set |
Rerunning the same script |
Use |
Compressible payloads |
Add |
Where To Go Next
File I/O Utilities — file format, multi-channel captures, and file rolling.
Reading Frames From A File — replay options beyond
countRecords.Compression Utilities — compression modules in detail.
Rate Limiting With RateDrop — rate limiting options.
Extend Rogue With C++ — continue by extending Rogue in C++.