Testing

Tests usually only need to be ran by those actively developing the suite, especially after resolving merge-conflicts, adding in large features, refactoring, etc.

Running on S3DF and ssh'd into the psana pool (to use psana library) is required to run the entire suite of the tests, but a subset can still be run on a local machine (utilitiy-function tests, etc). This non-psana subset runs automatically in github actions

In short, running the following will do some useful check the code: (do so after commiting your current changes, since linting/format\ting will modify files):

# cd to project root
source setup_developers.sh # only need to run once when initially setting up suite
./lint_and_format.sh # note: will likely code style, but should not modify functionality
pytest .

Details

To setup for running the tests, run the following cmd from the repo root dir:

source setup_developers.sh

This installs the 'pillow' python dependency (used for diffing .png files), the ruff formatter/linter, and downloads the /test_data submodule. (note: only needs to be run once per-repo, unless you manually de-init'd the submodule and want it setup again) (note: this is unlike setup.sh, which needs to be run per-each terminal session (or added to ~/.bashrc))

The code can be checked with linting by running:

./lint_and_format.sh

This will lint and attempt to auto-apply any fixes, and then format the code. The script may output linting-errors that need to be manually fixed. (note: this will modify your local files, so reccomended to commit changes first -> run script -> commit changes from running script) (note: running 'ruff --line-length 120' will lint the code without auto-fixing)

You can run all the tests with the following:

pytest -vs .

(note: this works from either the project root-dir and also the /tests dir) _(note: the -vs flags format the pytest output in a nice way and let you see the output of any prints added to test files.)

You can run an individual test-file by specifying the path:

pytest -vs tests/test_SuiteScripts.py

And run an individual test-case within the test-file by using '::':

pytest -vs tests/test_SuiteScripts.py::test_Noise

These tests verify that the code in /calibrationSuite and /suite_scripts is currently working as expected.

test_SuiteScripts.py runs the files in /suite_script and diffs their real output against expected output in /tests/test_data

/tests/test_data stores data from this repo: https://github.com/slaclab/beamtime-calibration-suite-test-data The repo is added as a git submodule, and will get setup by the setup_developers.sh script. Since the test data is large, LFS is used and needs to be installed. (LFS is installed already on S3DF)

If you wish to run the tests without them diffing the output files against the golden data (and want to just test that the code runs to completion without crash/error), you can just remove the entire /tests/test_data dir manually and the tests will not attempt to diff. If you wish to redownload the /test_data dir, run source setup_developers.sh again.

Some more info on testing, such as adding now data to the test_data submodule or adding a new suite_script test, can be found in test_SuiteScripts.py.