Azure Devops

Automate your CI/CD pipeline executions directly from Azure DevOps using SimplifyQA’s native integration. This integration enables seamless test execution as part of your build and release pipelines.

Prerequisites

Before setting up the integration, ensure the following:

• You have an active SimplifyQA subscription

• An agent is installed and running on the machine for execution

• You have an active Azure DevOps subscription

• An organisation is already created in Azure DevOps

---

Create a Pipeline in SimplifyQA

1. Log in to your SimplifyQA account

2. Navigate to the Pipelines section from the left-hand panel.

1. Click on +Pipeline to create a new one.

1. Enter a meaningful name for your pipeline.

1. Enter the required details and click on save to create the pipeline

Configure the pipeline details:

• Pipeline Type: Choose whether you want to run on Test Case or Suite

• Execution Type: Choose whether you want to run on Local or Cloud

  • Local Execution

    If you choose to run tests locally:

    • Select Local under Execution Type.

    • You must specify the Local Machine Name where the SimplifyQA Agent is installed and running.

  • Cloud Execution

    If you opt for Cloud Execution, you need to register your execution environment on the cloud first. This allows to spin up virtual machines or containers for remote execution. Refer to Cloud Configuration to configure your cloud.
• Test Case/Suite ID: Enter the corresponding ID

• Version & Execution Type: Select the desired versions

• Machine Name: Specify the machine for test execution

• Environment: Choose the target environment

• Release and Sprint: Define these for tracking purposes

---

Install SimplifyQA Extension in Azure DevOps

1. Log in to your Azure DevOps account.

2. From the dashboard, click on the respective project card.

1. Click on Azure Marketplace and search for "SimplifyQA".

1. Click Get it for free.

1. If you already have an organization, proceed to Downloads. If not, create one and then download the extension.

---

Configure Azure DevOps Pipeline

1. Go back to your Azure DevOps homepage.

2. Navigate to Pipelines > Pipelines from the left panel.

1. Click on New Pipeline.

1. Select the code repository you're working with.

1. Authenticate and log in to the repository.

2. Select the specific repository you want to work on.

1. Choose Starter pipeline.

1. Your yaml file will open.

---

Add SimplifyQA Executor to Pipeline

• Click on Show assistant on the right panel.

• Search for SimplifyQA and select "SimplifyQA Pipeline Executor v2".

• Fill in the required details:

  • Pipeline ID: The ID from the pipeline created in SimplifyQA

  • API URL: Your SimplifyQA base URL

  • Threshold: Sets the minimum pass percentage for the test suite. If the actual result falls below this value, the CI/CD pipeline will fail.

    • By default, the threshold is set to 100%.

    • For example, if you set the threshold to 70%, the pipeline will still pass even if up to 30% of the tests fail.

  • Verbose Flag: When set to TRUE, it enables detailed execution logs, which are helpful for debugging and reviewing test steps in depth. If set to FALSE, only minimal or summary-level logs will be displayed.

  • API Key: Your SimplifyQA Machine registration token. Go to Profile > Manage Account > Security

  

Note - If your machine is not registered kindly refer to Register your Machine

• Click Add – the task will be added under the steps section of your YAML.

---

Save and Run the Pipeline

1. Click Save or Save and Run

1. Select the branch and click on Run Pipeline.

2. Go to the Summary tab to monitor the execution.

3. Click on Job to view execution logs, results, and detailed output.

Once integrated:

• Test pipelines from SimplifyQA can be triggered automatically via Azure DevOps.

• Execution statuses and results can be monitored from both platforms.

• You maintain a single source of truth for test validation in your CI/CD workflow.

---

Frequently Asked Questions (FAQs)

What is the benefit of integrating Azure DevOps with SimplifyQA?

The integration enables automated triggering of test pipelines in SimplifyQA directly from your Azure DevOps builds or releases. This ensures quality checks are embedded within your CI/CD workflow, catching bugs earlier in the development cycle.

Do I need to write a YAML file manually?

No, you can use the Starter Pipeline in Azure DevOps, then add the SimplifyQA executor task using the "Show assistant" panel in the pipeline editor.

Can I monitor SimplifyQA test results in Azure DevOps?

Yes. Once the pipeline is triggered, you can:

• View execution logs in the Jobs tab of Azure

• Monitor detailed test results in SimplifyQA

What happens if my test threshold is not met?

If configured, the pipeline will fail when the test success rate falls below the set threshold, enforcing test quality as part of CI/CD validation.