EDSC Contingency Plan

Contingency-Related Actions:

The following are a list of various actions that can be taken in order to respond to various scenarios.  After this section, we will describe scenarios and which of these documents the actions apply to.

how to re-run CI/CD for EDSC (rebuild/redeploy)

• navigate to the deployment project environment for earthdata search: https://ci.earthdata.nasa.gov/deploy/viewDeploymentProjectEnvironments.action?id=242712580
• each release corresponds to a tag on https://github.com/nasa/earthdata-search/tags
• deployment usually takes roughly 15 minutes

how to change environmental variables (Bamboo variables)

• log into deployment project environment at https://ci.earthdata.nasa.gov/
• open up configuration
• click variables
• edit variables and save
• note: you will have to re-deploy in order to pick up environmental variable changes

how to pin or update dependency versions

• NPM uses semantic versioning
• More about how to specify versions of dependencies: https://docs.npmjs.com/cli/v9/configuring-npm/package-json#dependencies
• open package.json, update the versions in the "dependencies" or "devDependencies" object
• if you want to pin to a specific version, remove the "^" from beginning of the version string
• alternatively, you can run "npm audit fix"
• re-run npm install in order to get a new version of the package-lock.json file
• make sure to commit the package-lock.json file in addition to the package.json file
• note: you can use Snyk's built in "fix this vulnerability" button whereby Snyk will try to update the dependency versions and issue a PR.  it often breaks builds, so it will require modification after the PR is submitted

how to log into Snyk and view vulnerability reports

• https://app.snyk.io/org/esdis-cumulus-core-gibs-cmr-etc.
• In order to navigate to vulnerability reports for specific Pull Requests:
  • go to https://github.com/nasa/earthdata-search/pulls and open up the specific PR on Github
  • scroll down to checks
  • find security/snyk and click details

how to back-port vulnerability fixes:

• Developer's Guide#Backports

how to backup and restore database for EDSC:

• https://wiki.earthdata.nasa.gov/display/EDSC/EDSC+Database+Backup+and+Restore+in+NGAP

how to restore github:

• If a Github repository is hacked / deleted, we can restore it from the corresponding Bitbucket repository, which mirrors the master branch

Scenarios:

• AWS Access Key is compromised.  If an AWS Access Key is compromised, it could give hackers access to some AWS resources.  Here are remediation steps:
  • revoke access key and create a new one
  • replace environmental variables (Bamboo variables) referencing the access key with the new one 
  • re-run CI/CD for EDSC (rebuild/redeploy)
• Malicious NPM Package.  If there is a truly dangerous version of an NPM that we list in our dependencies, we can take the following steps to remediate:
  • how to pin or update dependency versions
  • back-port vulnerability fixes
• Github Hacked.  If our Github repository is hacked, we can take the following steps:
  • Notify security
  • Once we have our access restored, we can restore our source code to a prior version that is mirrored in our Bitbucket repository
• Database Corrupted:
  • If our database is corrupted somehow either though malicious actions or accidentally, we can restore it from a backup using the instructions cited here:
  • https://wiki.earthdata.nasa.gov/display/EDSC/EDSC+Database+Backup+and+Restore+in+NGAP

contact tree (aka phone tree):

• Alicia is the main POC
• Heather is backup to Alicia