Integrating with CircleCI

CircleCI is a CI/CD (Continuous Integration and Deployment) tool that helps development teams release code rapidly and automate the build, test, and deployment of their applications. Using CircleCI, engineers can automate their entire testing suite for new commits, reducing the potential for human error. After source-code repositories (on GitHub, Bitbucket, or GitLab) are authorized and added as a project to CircleCI, every commit triggers CircleCI to run jobs.

An everyday use case for CircleCI is building and testing your application whenever code changes are submitted. To take full advantage of TestRail and manage all your test cases and test results, both manual and automated, in a centralized manner, you can send your automated test results directly from CircleCI to TestRail. This article will explore how to easily use CircleCI and the TestRail CLI to accomplish this task.

Using the TestRail CLI in CircleCI workflows

Automated processes can be configured in CircleCI using workflow configuration files. These are YAML files that allow you to specify your processes in a structured manner and make use of CircleCI for each step of the way to make it easier to configure your execution environment, as well as execute any step you require.

The easiest way to integrate TestRail with CircleCI is by configuring your workflow to use the TestRail CLI to send automated test results to TestRail. Below is a sample of how a workflow file using the TestRail CLI to send test results to TestRail would look like.

version: 2.1
orbs:
  browser-tools: circleci/browser-tools@1.4.1
working_directory: ~/automation-framework-integration
jobs:
  build-and-test:
    docker:
      - image: cimg/openjdk:15.0
    steps:
      - browser-tools/install-browser-tools:
          chrome-version: 109.0.5414.74
      - checkout  # Performs code checkout
      - run:
          name: Build and run the tests
          command: |
            chmod 777 drivers/linux/chromedriver
            mvn clean compile test
      - persist_to_workspace:
          root: ./
          paths:
            - target/TEST-junit-jupiter.xml
  install-trcli-and-upload-results:
    docker:
      - image: cimg/python:3.11
    steps:
     - attach_workspace:
         at: ./automation-framework-integration
     - run:
         name: Install TR CLI and upload the test results to TestRail
         command: |
           pip install trcli
           trcli -y \
             -h https://INSTANCE.testrail.io/ \
             --project "PROJECT NAME" \
             -u USER_EMAIL \
             -p PASSWORD \
             parse_junit \
             --title "Automated Tests from CircleCI workflow" \
             --run-description $CIRCLE_BUILD_URL \
             -f "./automation-framework-integration/target/TEST-junit-jupiter.xml"

# Invoke configured jobs using a workflow:
workflows:
  sample:  # Name of the workflow
    jobs:
      - build-and-test
      - install-trcli-and-parse-results:
         requires:
            - build-and-test

A CircleCI job is a collection of sequential steps. All job steps are executed in a single unit within a fresh container or a virtual machine. Jobs are orchestrated using CircleCI workflows.

Breaking down the file, we notice that the workflow comprises two jobs.

• • In the above-shown YAML file, we have our first job, build-and-test. For this job, we have specified the working directory and the docker image (cimg/openjdk) on which we have to run the job. CircleCI maintains a fleet of convenience images for various programming languages, databases, and operating systems. This job is responsible for checkout, building, and running the tests and persisting data to make it available to the second job.
  • The second job, install-trcli-and-upload-results, runs on a different image (cimg/python), and it installs the TestRail CLI and uploads test results to TestRail. In this case, the second job is a downstream job that runs only after the execution of the first job.

The steps in both jobs define the commands to execute on the docker images. As a whole, these jobs contain the following steps:

1. 1. Install-browser-tools
   2. Checkout
   3. Build and run the tests
   4. Persist workspace
   5. Attach workspace
   6. TestRail CLI upload results

Step 1: Install-browser-tools

This step Installs various browsers and browser-testing tools into any Debian/Ubuntu based Docker image.

Step 2: Checkout

The Checkout step executes the code checkout on the agent for your repository. In the example above, we have chosen the cimg/openjdk image for our execution container.

Step 3: Build and run the tests

In this step, we make the Linux chromedriver executable and run the tests.

- run:
    name: Build and run the tests
    command: |
      chmod 777 drivers/linux/chromedriver
      mvn clean compile test

Step 4: Persist workspace

This step adds files to a persistent workspace to be used by the second job. We need the TEST-junit-jupiter.xml file for the second job to send the test results to TestRail with the help of the TestRail CLI.

- persist_to_workspace:
    root: ./
    paths:
      - target/TEST-junit-jupiter.xml

Step 5: Attach workspace

This step will download the workspace content into the file system of the new image.

install-trcli-and-parse-results:
  docker:
    - image: cimg/python:3.11
  steps:
    - attach_workspace:
        at: ./automation-framework-integration

Step 6: Install the TestRail CLI and upload test results

This step will do two different actions

1. 1. Install the TestRail CLI
   2. Call the TestRail CLI to parse the JUnit report generated by your test automation framework and send the results to TestRail

The TestRail CLI is a Python package hosted on the Python Package Index (PyPI). To install it, we execute the command pip install trcli.

After the TestRail CLI is installed, there are a few mandatory arguments that we need to pass along with trcli -ycommand, such as your TestRail instance address and credentials, the project you want to report to, the title for your test run, and the path to the JUnit report.

For more information about arguments, please check the TestRail CLI documentation.

- run:
    name: Install TR CLI and upload the test results to Testrail
    command: |
      pip install trcli
      trcli -y \
        -h https://INSTANCE.testrail.io/ \
        --project "PROJECT NAME" \
        -u USER_EMAIL \
        -p PASSWORD \
        parse_junit \
        --title "Automated Tests from CircleCI workflow" \
        --run-description $CIRCLE_BUILD_URL \
        -f "./automation-framework-integration/target/TEST-junit-jupiter.xml"

Working example using JUnit5 and Selenium

CircleCI supports many version control tools like GitHub, GitLab, and Bitbucket. In this article, we will walk you through using a GitHub repository.

Below is an example that illustrates how to run a JUnit5 project with selenium tests on CircleCI and then report the results to TestRail.

1. 1. Create a new repository on GitHub
   2. Download the files from the JUnit5-Selenium sample project (you can find the CircleCI config.yml file under the .circleci folder).
   3. Replace TESTRAIL_INSTANCE, PROJECT NAME, USER_EMAIL, and PASSWORD on the CircleCI config.yml file under the .circleci folder (we recommend not replacing the password directly and using the CircleCI environment variables instead).
   4. Upload the project to your GitHub repository.

Executing the workflow

1. 1. On the CircleCI website, click the Log In button.
   2. Log in to CircleCI with your GitHub credentials.
   3. Authorize CircleCI to access your GitHub account.
   4. Set up the code by selecting the Organization, Repository, and config.yml file.
   5. Select your repository and config.yml file, then click Set Up Project.
   6. Once you click on the Set Up Project button, you will navigate to the Dashboard of CircleCI, where you will see that the pipeline is already in Running status.
   7. Once the pipeline status is Success, it will be displayed with green color as displayed in the screenshot below.
   8. You can click on the workflow to see the details. The execution log is divided into two jobs. In the image below, we can see that both of the workflow jobs went well, and hence the pipeline was executed successfully.
   9. By clicking on a specific job, we can see the details for each step.

      

   10. Finally go to your TestRail instance and see your Test Run results.
       

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.