How to avoid using multiple TestNG suite XML files and Jenkins configuration to support multi-environment combinations?

Identifying the Need

When we have a large number of automation test suites to be supported across each profile/test environment (browsers, OS, regions, etc.), we end up creating multiple TestNG suite XML files. The process reduces the code readability and increases maintenance efforts, especially when two or more automation engineers are collaborating.

Additionally, when we use any CI tool like Jenkins, we end up creating and maintaining one job per profile of pom.xml.

Let us understand the problem in a detailed way

The image below shows multiple TestNG suite XML files considered for each automated test suite – by region and by the browser. In this example, we have two test suites, which support two regions and four browsers, resulting in sixteen (2 * 4 * 2 = 16) TestNG Suite XML files.

multiple TestNG suite XML files

The below image is from the ‘pom.xml’ file, where separate profiles are created for each of the above mentioned TestNG suite XML files. This way, we need to create sixteen profiles for sixteen TestNG suite XML files.

pom.xml file

The image below is from Jenkins, where jobs are created against each profile listed from the ‘pom. xml’ file to trigger executions across multiple environments (regions, browsers). So, we end up having sixteen jobs on the Jenkins server.

Jenkins server

Our Recommendations:

Here are the changes we need to perform:

      1. Re-configure TestNG suite XML file to replace multiple instances
      2. Configuration changes to pom.xml
      3. Parameterization of Jenkins job

Let us understand in detail about each one of these changes at an implementation level

1. Re-configure TestNG suite XML file to replace multiple instances

This can be done by adding * at the end of the ‘package name’ that refers to all the tests inside the package. Please refer to the screenshot below with the configuration changes to have a single TestNG suite XML file (here, ‘TestSuite.xml’ file).

TestNG suite XML


2. Changes to pom.xml:

Dynamic values are picked for environment combinations by using the code, as highlighted in the image below. This helps in creating a single profile (here, ‘Profile1’), which can be used to run all the tests listed under ‘TestSuite.xml’ through Jenkins against multiple supported environments (regions, browsers).

environment combinations by using the code

3. Parameterized Jenkins job

We can reduce Jenkins jobs count from sixteen to one, by using the mentioned details of ‘pom.xml’ profile and configuring a parameterized Jenkins job accordingly. The image below shows a job to execute all tests from the existing package(s) against selected test environments.

configuring a parameterized Jenkins job


For more details on creating parameterized Jenkins job, please refer:
Alternatively, in case you do not have a Jenkins server set up, use ‘Parameters’ in TestNG suite XML file (TestSuite.xml) to specify the required browser or region for execution.