TODO:

Repo setup

• TODO:
  • Persist volumes for databases
  • Docs for injection and container
  • Docs!
  • Set up all loggers
  • Move environments, flows, tasks, and results under model directory
  • Add tests for templated repo
  • Fix docker cred handling in workflows
  • Set up with conda
  • Could test the templated flows directly with the services by using the lume-services pytest fixtures
• Done:
  • Versioneer
  • Action for testing package install & running pytests
  • Tests for injection and container 7/5
  • Environment variables with extensible prefix 7/5
  • License should be passed or something generic. SLAC license shouldn't be in LUME 4/12
  • Add versioneer config. Versioneer install needs to happen on init. 4/12
  • Transition from pytest-mysql to remove pip dependency 4/15
  • Basics of injection and container 4/15
  • Action for building docs
  • Automatically generate diagram from .sql schema? 5/6s
  • Move any services in init files to a service.py
  • Fix prefect context bug during pytest
  • Fix image issue... Build a designated image in deployment using a lume-services base image
  • Remove tests in pip installation

CLI

• TODO:
  • Integrate template with LUME-services CLI tooling. Once the template is tracked as a git submodule, a command can be implemented for creating the template directly using the lume-services entrypoint.
  • Implement CLI tools for registering model deployments. At present, you need to call the in-code API as demonstrated in
  • Add CLI tool for listing available models and deployments

Template

• TODO:
  • Use git submodule to track template with version
  • Use relative path for cookiecutter generation
• Done:
  • Action for testing package install & running pytests
  • Upload docker image to repo artifacts 5/3
  • Docker build action for templated repos 4/12
  • Instructions for templating tool and configuration
  • Templated README
  • Handling of input/output variables at outset
  • Raise not implemented for to-be-replaced template functions

Databases

• TODO:
  • Test database failure cases
  • Dashboard image field in mongodb for impact
  • Docs
• Done:
  • Initial tests for mysql model db
  • Abstraction of database schema and auth config from query execution
  • Add flow_to_deployments into db
  • Change model_versions to deployments in schema
  • Make mongodb multiprocessing safe
  • Add uniqueness to results
  • Fix test connection with mysql in-package plugin. "Connection refused" 4/13
  • Rename base classes for DBService

Backlog

• TODO:
  • Synchronous snapshot service (-> SLAC-services)
  • HPC interface
  • Slurm interface (Can we just mount the binary?)
  • Output service (this probably belongs in LUME-EPICS)
• Done:
  • Finish util testing

Scheduler

• TODO:

  • Kubernetes backend tests
  • Docs
  • Authentication for private repositories

• Done:

  • Refactor scheduler 7/13
  • Create prefect flow image 6/26
  • Remove redundant flow storage during build in pytest 7/6
  • Pin docker-py version 7/15
  • Constrain KubernetesBackend image pull policy to existing options
  • Drop prefect subfolder 6/24
  • Create docker backend 6/27
  • Tesing infrastructure for prefect w/ docker compose 6/27
  • Result tasks
  • Add scheduler to context
  • Drop all but apollo from config 8/3
  • Improve service wait_until_responsive checks and move into docker 8/9
  • Test flow of flows composition
  • Add requirements table
  • Environment solving for containerized jobs
  • Interace to model
  • How do we handle submission of environment variables to the scheduler? For example, how do we communicate the aliasing of services in a docker-compose app?

Misc

• TODO:
  • Use environment variable fixture in tests instead of modifying env
• Done:
  • Rename file.systems to file.filesystems and all files names service.py
  • Move fixtures from conftest to designated files under tests/fixtures
  • Change LUME-model SurrogateModel to BaseModel for generalizability

Files

• TODO:
  • Do we need file locks for the file handlers?
• Done:
  • Implement local file handler 5/2
  • Implement mounted filesystem handler 5/2

Models

• TODO:
  • Create model interface with injected services
  • Add utility for loading flow objects