You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 13 Next »

Problem: Build time takes 3+ hours with the main offenders being the int tests

  • Build CMR ~ 45 min
  • Migrate DB ~ 14 min
  • Unit Tests ~ 28 min
  • Int Test 1 ~ 81 min
  • Int Test 2 ~ 93 min
  • Int Test 3 ~ 46 min

Option A :: Clean up Int tests

  • How it works:
    • Consolidate, de-dup and clean up unnecessary int tests
    • Options:
      • Switch to new test tool like eftest that allows for parallel test runs (kaocha does not allow for parallel test runs)
        • parallel runs do not work for int tests because tests require specific database state that is shared and reset every new test. Running in parallel will mess up the db states. BUT maybe we can use this for unit tests?
      • Parallel testcontainers? (Ask Tyler)
      • Setup each namespace to share one instance of db, elastic, etc to reduce # of times we need to reset these resources
      • Move some int tests to unit tests
      • Introducing multi-threading in applicable int tests (example: cmr.system-int-test.misc.health-test) – we may not have the computational power for this in docker
      • Should int test should be blocking step at all in merge requests?? 
        • Idea: Keep int tests as daily runs, and then unit tests on commit changes
      • Is there a code coverage threshold that would make us more comfortable to rely on unit tests?
        • Would we still do the int tests if we separated search and ingest into their own projects?
  • PROS
    • No change to workflow necessary
  • CONS
    • Will take time to go through all tests and think about consolidation efforts
    • I suspect this will not give us that big of a time save

Option B :: Separate int tests to 'core' and 'full' + GitFlow

  • How it works:
    • Create a set of "core" integration tests that go through the most important test pathways on feature branches
    • Once the feature branch is merged into development branch, that will trigger the "full" int tests
    • Release branches are created from development branch and merged into master
  • PROS
    • Easily separate and reduce build time on feature branches
    • Protect master branch from issues with 'bad code' or failing full int tests
  • CONS
    • More complex branching and release structure
    • Change to our workflow needed
    • Will not know if you failed an edge case until you merged to development branch

Option C :: Separate int tests to 'core' and 'full' + Current Trunk Based Workflow

  • How it works:
    • Trunk based workflow is CMR's current workflow (one master and feature branches)
    • Have 'core' int tests run on feature branches
    • When feature branch merges into master then 'full' int tests run
  • PROS
    • No change to our workflow needed
    • Easily separate and reduce build time on feature branches
  • CONS
    • Master branch is vulnerable 'bad code' because new code can fail full int tests that were not caught in feature branch

Option D :: Create new 'on-demand' build

  • How it works:
    • Create new bamboo build that runs the feature branch with 'core' int tests or zero int tests to allow for fast deployment
  • PROS
    • Easy to implement
  • CONS
    • Have two separate build plans for one branch
    • Increase bamboo agent usage

Option E :: Increase parallel test groups

  • How it works:
    • We have 3 int test groups, we can separate them out even further and have them run in parallel
  • PROS
    • Easy to implement
  • CONS
    • Increase bamboo agent usage

Option F :: Allow forked CMR repos to deploy to SIT

  • How it works:
    • Allow forked or separate CMR repos can deploy to SIT
  • PROS
    • each developer can control the bamboo process for their own builds and deploy to SIT for testing
  • CONS
    • The devs would have to turn it back on and make sure the full build works before merging to master which opens up to human error vulnerabilities


  • No labels