Problem: Build time takes 3+ hours with the main offenders being the int tests
- Build CMR ~ 45 min
- Migrate DB ~ 14 min
- Unit Tests ~ 28 min
- Int Test 1 ~ 81 min
- Int Test 2 ~ 93 min
- Int Test 3 ~ 46 min
Option D :: Create new 'on-demand' build (DONE) - JYNA
- How it works:
- Create new bamboo build that runs the feature branch with unit tests, but zero int tests to allow for fast deployment only to SIT
- No auto-triggers
- PROS
- Easy to implement
- CONS
- Have two separate build plans for one branch
- Increase bamboo agent usage
Option E :: Increase parallel test groups (DONE) - JYNA
- How it works:
- We have 3 int test groups, we can separate them out even further and have them run in parallel
- Split Group 2 (Search) in half. So 4 groups total.
- PROS
- Easy to implement
- CONS
- Increase bamboo agent usage
Option P :: Parallelize Unit Tests (DONE) - Thomas
- How it works:
- Options:
- Roll our own
- https://github.com/jonpither/lein-pjotest
- https://www.gnu.org/software/parallel/
- https://github.com/Raynes/lein-pdo
- using lein modules command and run those commands in parallel
- Eftest instead of kaocha (will this be worth it? no, since it will require replacement of kaocha which could be very involved)
- Options:
TODO: Add the below as a step in PR template
Option L :: Turn int tests to unit tests (COMMIT TO DO - OVER TIME)
- How it works:
- If test doesn't need to be int test, move it to be a unit tests
- PROS:
- CONS:
Option X :: Initialize database once per test namespace (COMMIT TO DO - OVER TIME)
- How it works:
- Instead of tests reseting the db state along with queue, elastic, etc, we can setup an initial state per test and use that once
- This could work well for search tests
- We can combine tests to be in the same file too, if they are similar enough
- You see this: (use-fixtures :each (ingest/reset-fixture {"provguid1" "PROV1"}))
then try to make it :once
- PROS:
- Reduce time from reseting the db, elastic, queue, etc for every single test (which is happening now)
- CONS:
- Will take some time to understand the tests and reduce
Option A :: De-dup, Cleanup Unnecessary Int tests (COMMIT TO DO - OVER TIME)
- How it works:
- Consolidate, de-dup and clean up unnecessary int tests
- PROS
- No change to workflow necessary
- CONS
- Will take time to go through all tests and think about consolidation efforts
- I suspect this will not give us that big of a time save
Option Y :: Unit Tests in every merge & Int Tests daily
- Question:
- Is there a code coverage threshold that would make us more comfortable to rely on unit tests?
- Would we still do the int tests if we separated search and ingest into their own projects?
- Should int test should be blocking step at all in merge requests??
- Idea: Keep int tests as daily runs, and then unit tests on commit changes
- Is there a code coverage threshold that would make us more comfortable to rely on unit tests?
Option Z :: Only run changed project int tests
- How it works:
- Having the ability to only run int tests that are affected by code changes
- writing some tool/script that separate all the project tests and any project changed will run
- Would have to think about which apps would be affected by what change (if you change access-control, then you should run ingest tests too, etc.)
- Having the ability to only run int tests that are affected by code changes
Option Q :: Parallel TestContainer
- How it works:
- spinning up multiple docker containers to parallelize tests (multiple redis containers, etc.) BUT dev-system setup is not going to make it easy to transition into this and will require big refactor
- This is for the far future if we rewrite CMR to keep this in mind
Option K :: Dev (only unit tests) + Master (int tests) + GitFlow
Option B :: Separate int tests to 'core' and 'full' + GitFlow
- How it works:
- Have feature branches (for devs to work on), development branch (for devs to branch from), and master branch (the final deployable branch) (GitFlow)
- Create a set of "core" integration tests that go through the most important test pathways on feature branches
- Once the feature branch is merged into development branch, that will trigger the "full" int tests
- Release branches are created from development branch and merged into master
- Have feature branches (for devs to work on), development branch (for devs to branch from), and master branch (the final deployable branch) (GitFlow)
- PROS
- Easily separate and reduce build time on feature branches
- Protect master branch from issues with 'bad code' or failing full int tests
- CONS
- More complex branching and release structure
- Change to our workflow needed
- Will not know if you failed an edge case until you merged to development branch
- What is 'core' vs 'full' test is subjective
Option C :: Separate int tests to 'core' and 'full' + Current Trunk Based Workflow
- How it works:
- PROS
- No change to our workflow needed
- Easily separate and reduce build time on feature branches
- CONS
- Master branch is vulnerable 'bad code' because new code can fail full int tests that were not caught in feature branch
Option F :: Allow forked CMR repos to deploy to SIT
- How it works:
- Allow forked or separate CMR repos can deploy to SIT
- PROS
- each developer can control the bamboo process for their own builds and deploy to SIT for testing
- CONS
- The devs would have to turn it back on and make sure the full build works before merging to master which opens up to human error vulnerabilities
Option G :: Moving some tests to GitActions
- How it works:
- Running tests (unit, int or both) on Git side
- PROS
- Less strain on bamboo agents for running our tests
- CONS
- Time to understand, setup, implement
Overview
Content Tools