Doesn't work when your trying to build with --since origin/master

[tl-NASEEBULLAH-AHMADI]

Hi there!

Great piece of work, and really boosts our CI time. However, unfortunately, I've encountered a problem with our build where the build job doesn't actually work properly? Maybe I'm doing something wrong?

// package.json
{
 ...
  "test": "lerna-parallelism run --stream --concurrency 1 --split $CIRCLE_NODE_TOTAL --partition $CIRCLE_NODE_INDEX test --since $COMMIT",
 ...
}

Our build command involves building typings for each package in our monorepo. The typings should only be built if any of the packages have changed since origin/master.

Can you please help?

[buschtoens]

Glad you like it!

Most likely $COMMIT refers to some commit that’s not checked out locally yet. CircleCI and most other CI services do a shallow clone and only load the branch they’re building for.

If $COMMIT refers to e.g. origin/master, we need to also make sure, that the CI runner also has access to the origin remote.

By default, CircleCI neither has access to remotes nor other branches. Unfortunately there’s also no magic orb (tmk), so we had to come up with this:

commands:
  setup-git:
    description: |
      Configures the local SSH and git client, so that it can be used against
      the GitHub repo.
    steps:
      - add_ssh_keys:
          fingerprints:
            # https://app.circleci.com/settings/project/github/<org>/<repo>/ssh
            - '56:be:2c:…'
      - run:
          name: Add `github.com` SSH fingerprint to `known_hosts`
          command: |
            mkdir -p ~/.ssh
            touch ~/.ssh/known_hosts
            if [ ! "$(ssh-keygen -F github.com)" ]; then
              echo "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> ~/.ssh/known_hosts
            fi
  # Borrowed from: https://circleci.com/developer/orbs/orb/narrativescience/ghpr
  get-pr-info:
    description: |
      Gets and sets the following environment variables:
        * `GITHUB_PR_BASE_BRANCH` - The base branch for the PR.
      Requires `GITHUB_TOKEN` to be set as an environment variable.
    parameters:
      checkout-base-branch:
        type: boolean
        default: true
        description: |
          Whether to create a local branch for
          `$GITHUB_PR_BASE_BRANCH` that tracks the remote branch.
      remote:
        type: string
        default: origin
        description: |
          The remote name to use, when checking out the `$GITHUB_PR_BASE_BRANCH`
          locally.
    steps:
      - run:
          name: Get PR information
          command: |
            PR_NUMBER=$(echo "$CIRCLE_PULL_REQUEST" | sed "s/.*\/pull\///")
            echo "PR_NUMBER: $PR_NUMBER"
            API_GITHUB="https://api.github.com/repos/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME"
            PR_REQUEST_URL="$API_GITHUB/pulls/$PR_NUMBER"
            PR_RESPONSE=$(curl -H "Authorization: token $GITHUB_TOKEN" "$PR_REQUEST_URL")
            # If installed, use jq; otherwise, try Python.
            # Node Docker image has jq.
            # SwiftLint Docker image has Python 2.7.
            if (command -v jq >/dev/null 2>&1)
              then
                PR_BASE_BRANCH=$(echo $PR_RESPONSE | jq \
                  --exit-status \
                  --raw-output \
                  '.base.ref | select(type == "string" and length > 0)'
                )
              else
                PR_BASE_BRANCH=$(echo $PR_RESPONSE | python2.7 \
                  -c "import json, sys; print(json.load(sys.stdin)['base']['ref'])"
                )
            fi
            echo "PR_BASE_BRANCH: $PR_BASE_BRANCH"
            echo "export GITHUB_PR_BASE_BRANCH='$PR_BASE_BRANCH'" >> $BASH_ENV
      - when:
          condition: << parameters.checkout-base-branch >>
          steps:
            - setup-git
            - run:
                name: Checkout PR base branch
                command: |
                  echo "Checking, whether '$GITHUB_PR_BASE_BRANCH' is already checked out."
                  # Even if `parameters.checkout-base-branch` is `true`, only
                  # check out the branch, if there's not already a branch of the
                  # same name checked out locally.
                  # This is the case, e.g. when the base branch is `main`.
                  if (git show-ref --verify --quiet "refs/heads/$GITHUB_PR_BASE_BRANCH")
                    then
                      echo "'$GITHUB_PR_BASE_BRANCH' is already checked out. Skipping."
                    else
                      echo "Checking out '$GITHUB_PR_BASE_BRANCH' from '<< parameters.remote >>'."
                      git fetch << parameters.remote >> "$GITHUB_PR_BASE_BRANCH"
                      git branch "$GITHUB_PR_BASE_BRANCH" "<< parameters.remote >>/$GITHUB_PR_BASE_BRANCH"
                  fi
                  git --no-pager show --summary "$GITHUB_PR_BASE_BRANCH"

jobs:
  node-lint-pr:
    description: Lint JavaScript (PRs)
    executor: node
    steps:
      - attach_workspace:
          at: ~/project
      - get-pr-info
      - run:
          name: Run ESLint
          command: yarn lint:js
      - run:
          name: Check TypeScript types for packages changed in pull request
          command: yarn lerna-run --since $GITHUB_PR_BASE_BRANCH lint:types
      - run:
          name: Check HBS formatting for packages changed in pull request
          command: yarn lerna-run --since $GITHUB_PR_BASE_BRANCH lint:hbs

The great thing about $GITHUB_PR_BASE_BRANCH is that it even works for PRs against branches other than the default one. So for instance, you could merge various PRs into a feature integration branch before merging that to main. Every individual feature PR only gets its true base and thus change set.