Problem: Build time takes 3+ hours with the main offenders being the int tests

• Build CMR ~ 45 min
• Migrate DB ~ 14 min
• Unit Tests ~ 28 min
• Int Test 1 ~ 81 min
• Int Test 2 ~ 93 min
• Int Test 3 ~ 46 min

Option D :: Create new 'on-demand' build (COMMIT TO DO) - JYNA

• How it works:
  • Create new bamboo build that runs the feature branch with unit tests, but zero int tests to allow for fast deployment only to SIT
  • No auto-triggers
• PROS
  • Easy to implement
• CONS
  • Have two separate build plans for one branch
  • Increase bamboo agent usage

Option E :: Increase parallel test groups (COMMIT TO DO) - JYNA

• How it works:
  • We have 3 int test groups, we can separate them out even further and have them run in parallel
  • Split Group 2 (Search) in half. So 4 groups total.
• PROS
  • Easy to implement
• CONS
  • Increase bamboo agent usage

Option P :: Parallelize Unit Tests (COMMIT TO DO) - Jyna write ticket (Thomas assign)

• How it works:
  • Options:
    • Roll our own
    • https://github.com/jonpither/lein-pjotest
    • https://www.gnu.org/software/parallel/
    • https://github.com/Raynes/lein-pdo
    • using lein modules command and run those commands in parallel
    • Eftest instead of kaocha (will this be worth it? no, since it will require replacement of kaocha which could be very involved)

TODO: Add the below as a step in PR template

Option L :: Turn int tests to unit tests (COMMIT TO DO - OVER TIME)

• How it works:
  • If test doesn't need to be int test, move it to be a unit tests
• PROS:
• CONS:

Option X :: Initialize database once per test namespace (COMMIT TO DO - OVER TIME)

• How it works:
  • Instead of tests reseting the db state along with queue, elastic, etc, we can setup an initial state per test and use that once
  • This could work well for search tests
  • We can combine tests to be in the same file too, if they are similar enough
  • You see this: (use-fixtures :each (ingest/reset-fixture {"provguid1" "PROV1"}))
    then try to make it :once
• PROS:
  • Reduce time from reseting the db, elastic, queue, etc for every single test (which is happening now)
• CONS:
  • Will take some time to understand the tests and reduce

Option A :: De-dup, Cleanup Unnecessary Int tests (COMMIT TO DO - OVER TIME)

• How it works:
  • Consolidate, de-dup and clean up unnecessary int tests
• PROS
  • No change to workflow necessary
• CONS
  • Will take time to go through all tests and think about consolidation efforts
  • I suspect this will not give us that big of a time save

Option Y :: Unit Tests in every merge & Int Tests daily

• Question: 
  • Is there a code coverage threshold that would make us more comfortable to rely on unit tests?
    • Would we still do the int tests if we separated search and ingest into their own projects?
  • Should int test should be blocking step at all in merge requests?? 
    • Idea: Keep int tests as daily runs, and then unit tests on commit changes

Option Z :: Only run changed project int tests

• How it works:
  • Having the ability to only run int tests that are affected by code changes
    • writing some tool/script that separate all the project tests and any project changed will run
    • Would have to think about which apps would be affected by what change (if you change access-control, then you should run ingest tests too, etc.)

Option Q :: Parallel TestContainer

• How it works:
  • spinning up multiple docker containers to parallelize tests (multiple redis containers, etc.) BUT dev-system setup is not going to make it easy to transition into this and will require big refactor
  • This is for the far future if we rewrite CMR to keep this in mind

Option K :: Dev (only unit tests) + Master (int tests) + GitFlow

Option B :: Separate int tests to 'core' and 'full' + GitFlow

• How it works:
  • Have feature branches (for devs to work on), development branch (for devs to branch from), and master branch (the final deployable branch) (GitFlow)
    
    • https://www.toptal.com/software/trunk-based-development-git-flow
  
  • Create a set of "core" integration tests that go through the most important test pathways on feature branches
  • Once the feature branch is merged into development branch, that will trigger the "full" int tests
  • Release branches are created from development branch and merged into master
• PROS
  • Easily separate and reduce build time on feature branches
  • Protect master branch from issues with 'bad code' or failing full int tests
• CONS
  • More complex branching and release structure
  • Change to our workflow needed
  • Will not know if you failed an edge case until you merged to development branch
  • What is 'core' vs 'full' test is subjective

Option C :: Separate int tests to 'core' and 'full' + Current Trunk Based Workflow

• How it works:
  • Trunk based workflow is CMR's current workflow (one master and feature branches)
    • 
  • Have 'core' int tests run on feature branches
  • When feature branch merges into master then 'full' int tests run
• PROS
  • No change to our workflow needed
  • Easily separate and reduce build time on feature branches
• CONS
  • Master branch is vulnerable 'bad code' because new code can fail full int tests that were not caught in feature branch

Option F :: Allow forked CMR repos to deploy to SIT

• How it works:
  • Allow forked or separate CMR repos can deploy to SIT
• PROS
  • each developer can control the bamboo process for their own builds and deploy to SIT for testing
• CONS
  • The devs would have to turn it back on and make sure the full build works before merging to master which opens up to human error vulnerabilities

Option G :: Moving some tests to GitActions

• How it works:
  • Running tests (unit, int or both) on Git side
• PROS
  • Less strain on bamboo agents for running our tests
• CONS
  • Time to understand, setup, implement