Skip to content

Test case subsetting by environment

As new features are developed and deployed to higher environments, the test suite execution can be time-consuming. To optimize this, we can subset the test cases based on the environment they are intended for. This approach ensures that only the most critical tests are run in each environment, improving efficiency and reducing execution time.

Types of Tests to Run in Different Environments

QA Environment

In the QA environment, we focus on running tests that validate the core functionality of the application. This includes:

  • Unit Tests: Tests that validate individual components or functions of the application.
  • Smoke Tests: Basic tests to ensure that the application is stable and functional.
  • Regression Tests: Tests that cover the main features of the application to ensure that new changes do not break existing functionality.
  • Integration Tests: Tests that verify the interaction between different components of the application.
  • End-to-End Tests: Tests that simulate user scenarios to ensure the application behaves as expected from the user's perspective.
  • High-Risk Area Tests: Tests that focus on areas of the application that are prone to issues or have a history of bugs.

UAT environment

In the UAT (User Acceptance Testing) environment and up to production, we can subset the test cases further to focus on the most critical paths that users will actually take. This includes:

  • Smoke Tests: Basic tests to ensure that the application is stable and functional.
  • Critical Path Tests: Tests that cover the most important user journeys and functionalities that are critical for the business.
  • High-Risk Area Tests: Tests that focus on areas of the application that are prone to issues or have a history of bugs.

Staging Environment

In the Staging environment, we can further refine our test case subset to ensure that we are validating the application in a production-like setting. This includes:

  • Smoke Tests: Basic tests to ensure that the application is stable and functional.
  • Critical Path Tests: Tests that cover the most important user journeys and functionalities that are critical for the business.
  • Performance Tests: Tests that assess the application's performance under load, ensuring it can handle expected user traffic. Before executing these tests, ensure that the environment is configured to mimic production as closely as possible, permissions to run performance tests are granted and risks are assessed and mitigated.

Production Environment

In the Production environment, we focus on ensuring that the application is stable and functional for end users. The test cases run in this environment are limited to:

  • Smoke Tests: Basic tests to ensure that the application is stable and functional after deployment.
  • Health Checks: Tests that verify the overall health of the application, including uptime, response times, and basic functionality.

However, depending on the company's strategy, some organizations may choose to forbid entirely running automated tests in production environments. In such cases, careful manual testing or other easier verification methods may be employed to ensure the stability of the application.

Implementing Test Case Subsetting using Tags

A. Playwright

To effectively subset test cases in Playwright, we can use tags to categorize tests based on the environment they are intended for. This allows us to run only the relevant tests for each environment, optimizing our test execution process.

To implement tags in Playwright, you can use the @tag syntax in your test files. For example:

import { test } from '@playwright/test';

test(
  'example test',
  async ({ page }) => {
    // Test code here
  },
  { tags: ['smoke', 'critical'] },
);

You can then run tests with specific tags using the command line:

npx playwright test --grep @smoke

This command will run only the tests tagged with @smoke. You can combine multiple tags using logical operators to further refine your test selection.

B. Cypress

In Cypress, you can also use tags to subset your test cases. Cypress supports a similar tagging mechanism, allowing you to categorize tests based on the environment they are intended for.

To implement tags in Cypress, you can use the @tag syntax in your test files. For example:

describe('My Test Suite', () => {
  it('example test', { tags: ['smoke', 'critical'] }, () => {
    // Test code here
  });
});

You can then run tests with specific tags using the command line:

npx cypress run --env TAGS='@smoke'

This command will run only the tests tagged with @smoke. You can combine multiple tags using logical operators to further refine your test selection.

C. Selenium

In Selenium, you can implement test case subsetting by using annotations or tags in your test framework (e.g., TestNG or JUnit). This allows you to categorize tests based on the environment they are intended for.

import org.testng.annotations.Test;

public class MySeleniumTests {

  @Test(groups = {"smoke", "critical"})
  public void exampleTest() {
    // Test code here
  }
}

You can then run tests with specific tags using the command line:

mvn test -Dgroups="smoke"

This command will run only the tests tagged with smoke. You can combine multiple tags using logical operators to further refine your test selection.

Considerations in Subsetting Tests

When implementing test case subsetting by environment, consider the following:

  • Test Coverage: Identify the critical functionalities covered in each environment, even if not all tests are run.

  • Environment Configuration: Ensure that each environment is configured correctly to run the relevant tests, including any necessary test data setup or dependencies.

  • Risk Assessment: Assess the risks associated with running a subset of tests in each environment. Ensure that high-risk areas are adequately tested.

  • Continuous Improvement: Regularly review and update the test subsets based on changes in the application, user feedback, and test results. This ensures that the most relevant tests are always being executed in each environment.

  • Automation Framework Support: Ensure that your test automation framework supports tagging or categorization of tests. This is crucial for implementing test case subsetting effectively.

  • Execution Time: Monitor the execution time of tests in each environment. If certain tests consistently take longer than expected, consider whether they are necessary in that environment or if they can be optimized.

  • Feedback Loop: Establish a feedback loop with relevant parties to ensure that the test subsets align with business priorities and user expectations. This helps in maintaining the relevance of the tests being executed.

  • Security and Access Control in Testing: Ensure that test environments are protected by appropriate access controls, limiting who can execute, modify, or view test cases and results. Integrate security-focused tests, such as vulnerability scans and permission checks, into your automated suite to validate that security requirements are met throughout the testing process. Use Role-Based Access Control (RBAC) to restrict sensitive test data and maintain compliance with organizational policies.

Back to top