...

• Build CMR ~ 45 min
• Migrate DB ~ 14 min
• Unit Tests ~ 28 min
• Int Test 1 ~ 81 min
• Int Test 2 ~ 93 min
• Int Test 3 ~ 46 min

Option

...

D :: Create new 'on-demand' build (COMMIT TO DO) - JYNA

• How it works:
  • Create new bamboo build that runs the feature branch with unit tests, but zero int tests to allow for fast deployment only to SIT
  • No auto-triggers
• PROS
  • Easy to implement
• CONS
  • Have two separate build plans for one branch
  • Increase bamboo agent usage

Option E :: Increase parallel test groups (COMMIT TO DO) - JYNA

• How it works:
  • We have 3 int test groups, we can separate them out even further and have them run in parallel
  • Split Group 2 (Search) in half. So 4 groups total.
• PROS
  • Easy to implement
• CONS
  • Increase bamboo agent usage

Option P :: Parallelize Unit Tests (COMMIT TO DO) - Jyna write ticket (Thomas assign)

• How it works:
  • Options:
    • Roll our own
    • https://github.com/jonpither/lein-pjotest
    • https://www.gnu.org/software/parallel/
    • https://github.com/Raynes/lein-pdo
    • using lein modules command and run those commands in parallel
    • Eftest instead of kaocha (will this be worth it? no, since it will require replacement of kaocha which could be very involved)

TODO: Add the below as a step in PR template

Option L :: Turn int tests to unit tests (COMMIT TO DO - OVER TIME)

• How it works:
  • If test doesn't need to be int test, move it to be a unit tests
• PROS:
• CONS:

Option X :: Initialize database once per test namespace (COMMIT TO DO - OVER TIME)

• How it works:
  • Instead of tests reseting the db state along with queue, elastic, etc, we can setup an initial state per test and use that once
  • This could work well for search tests
  • We can combine tests to be in the same file too, if they are similar enough
  • You see this: (use-fixtures :each (ingest/reset-fixture {"provguid1" "PROV1"}))
    then try to make it :once
• PROS:
  • Reduce time from reseting the db, elastic, queue, etc for every single test (which is happening now)
• CONS:
  • Will take some time to understand the tests and reduce

Option A :: De-dup, Cleanup Unnecessary Int tests (COMMIT TO DO - OVER TIME)

• How it works:
  • Consolidate, de-dup and clean up unnecessary int tests
• PROS
  • No change to workflow necessary
• CONS
  • Will take time to go through all tests and think about consolidation efforts
  • I suspect this will not give us that big of a time save

Option Y :: Unit Tests in every merge & Int Tests daily

• Question: 
  • Consolidate, de-dup and clean up unnecessary int tests
  Options:
  • Switch to new test tool like eftest that allows for parallel test runs (kaocha does not allow for parallel test runs)
    • parallel runs do not work for int tests because tests require specific database state that is shared and reset every new test. Running in parallel will mess up the db states. BUT maybe we can use this for unit tests?
  • Parallel testcontainers? (Ask Tyler)
  • Setup each namespace to share one instance of db, elastic, etc to reduce # of times we need to reset these resources
  • Move some int tests to unit tests
  • Introducing multi-threading in applicable int tests (example: cmr.system-int-test.misc.health-test) – we may not have the computational power for this in docker
  Should int test should be blocking step at all in merge requests?? Idea: Keep int tests as daily runs, and then unit tests on commit changes
  • Is there a code coverage threshold that would make us more comfortable to rely on unit tests?
    • Would we still do the int tests if we separated search and ingest into their own projects?
• PROS
  • No change to workflow necessary
CONS
• Will take time to go through all tests and think about consolidation efforts
I suspect this will not give us that big of a time save
• • Should int test should be blocking step at all in merge requests?? 
    • Idea: Keep int tests as daily runs, and then unit tests on commit changes

Option Z :: Only run changed project int tests

• How it works:
  • Having the ability to only run int tests that are affected by code changes
    • writing some tool/script that separate all the project tests and any project changed will run
    • Would have to think about which apps would be affected by what change (if you change access-control, then you should run ingest tests too, etc.)

Option Q :: Parallel TestContainer

• How it works:
  • spinning up multiple docker containers to parallelize tests (multiple redis containers, etc.) BUT dev-system setup is not going to make it easy to transition into this and will require big refactor
  • This is for the far future if we rewrite CMR to keep this in mind

Option K :: Dev (only unit tests) + Master (int tests) + GitFlow

Option B :: Separate int tests to 'core' and 'full' + GitFlow

• How it works:
  • Have feature branches (for devs to work on), development branch (for devs to branch from), and master branch (the final deployable branch) (GitFlow)
    
    • https://www.toptal.com/software/trunk-based-development-git-flow
  
  • Create a set of "core" integration tests that go through the most important test pathways on feature branches
  • Once the feature branch is merged into development branch, that will trigger the "full" int tests
  • Release branches are created from development branch and merged into master
• PROS
  • Easily separate and reduce build time on feature branches
  • Protect master branch from issues with 'bad code' or failing full int tests
• CONS
  • More complex branching and release structure
  • Change to our workflow needed
  • Will not know if you failed an edge case until you merged to development branch
  • What is 'core' vs 'full' test is subjective

Option C :: Separate int tests to 'core' and 'full' + Current Trunk Based Workflow

• How it works:
  • Trunk based workflow is CMR's current workflow (one master and feature branches)
    • 
  • Have 'core' int tests run on feature branches
  • When feature branch merges into master then 'full' int tests run
• PROS
  • No change to our workflow needed
  • Easily separate and reduce build time on feature branches
• CONS
  • Master branch is vulnerable 'bad code' because new code can fail full int tests that were not caught in feature branch

Option D :: Create new 'on-demand' build

• How it works:
  • Create new bamboo build that runs the feature branch with 'core' int tests or zero int tests to allow for fast deployment
• PROS
  • Easy to implement
• CONS
  • Have two separate build plans for one branch
  • Increase bamboo agent usage

Option E :: Increase parallel test groups

...

...

• Easy to implement

...

• Increase bamboo agent usage

Option F :: Allow forked CMR repos to deploy to SIT

• How it works:
  • Allow forked or separate CMR repos can deploy to SIT
• PROS
  • each developer can control the bamboo process for their own builds and deploy to SIT for testing
• CONS
  • The devs would have to turn it back on and make sure the full build works before merging to master which opens up to human error vulnerabilities

Option G :: Moving some tests to GitActions

• How it works:
  • Running tests (unit, int or both) on Git side
• PROS
  • Less strain on bamboo agents for running our tests
• CONS
  • Time to understand, setup, implement