Guide

TestNG Load Generator

Overview.

"TestNG Load Generator" allows you to run TestNG-based tests as load tests.
You need advanced Java programming skills to utilize this load generator.
Also, it would be best if you have at least a basic understanding of the Selenium framework since the actual logic of the load testing scenarios is implemented via calling RemoteWebDriver directly.

All integrations with the Perforator platform are open-sourced.
Please take a look at our GitLab repository "Perforator SDK Java" if you are interested in the internal implementation of the load generator.

Configuration and execution of this load generator is done via a dedicated Maven plugin, while actual logic implementation is done via implementing tests using TestNG framework.
The example below can give you a sense of what a real-world configuration might look like.

Jump Start.

1. Download and unarchive prebuilt example.
2. Start editing pom.xml with your favorite text editor.
3. Update apiClientId property with your own value. Please navigate to the API Settings page to get your API Client ID.
4. Update apiClientSecret property with your own value. Please navigate to the API Settings page to get your API Client Secret.
5. Update projectKey property with your own value. Please refer to the "Projects" chapter to learn more about projects and get your project key.
6. Execute cloud-full-run.sh(on Linux/macOS) or cloud-full-run.cmd(on Windows) using your command line terminal - it should start a lightweight load test with 50 concurrent browsers in the cloud running for 5 minutes.
7. Load generator prints a link to view performance test statistics when all cloud resources are ready, and the actual load test begins. Please copy such a link and open it in your browser.

Note: Please make sure that JDK 11+ and Maven 3 is installed on your computer and available in your terminal PATH.
Execute `mvn --version` in your terminal to validate both.

Maven Build.

Prebuilt example comes with ready-to-use pom.xml where all dependencies and plugins are already configured.
It might be a case when you would like to integrate TestNG Load Generator into an existing maven project or prefer to configure the build from the ground up, so please take a look at the instructions below.

1. All compiled and well-tested artifacts of the Perforator SDK are uploaded to GitLab based maven repo.
Please update your pom.xml by adding pluginRepositories and repositories pointed to our maven repo, so maven can automatically download all required artifacts.

<project>
    ...
    <pluginRepositories>
        ...
        <pluginRepository>
            <id>perforator-gitlab-maven</id>
            <url>https://gitlab.com/api/v4/projects/perforator-labs%2Fperforator-sdk-java/packages/maven</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
            <releases>
                <enabled>true</enabled>
            </releases>
        </pluginRepository>
        ...
    </pluginRepositories>
    ...
    <repositories>
        ...
        <repository>
            <id>perforator-gitlab-maven</id>
            <url>https://gitlab.com/api/v4/projects/perforator-labs%2Fperforator-sdk-java/packages/maven</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
            <releases>
                <enabled>true</enabled>
            </releases>
        </repository>
        ...
    </repositories>
    ...
</project>

2. Add perforator-sdk-load-generator-testng artifact to your dependencies list, so all classes required by the load generator are available during compilation and execution phases.

<project>
    ...
    <dependencies>
        ...
        <dependency>
            <groupId>io.perforator.sdk</groupId>
            <artifactId>perforator-sdk-load-generator-testng</artifactId>
            <version>1.2.1</version>
        </dependency>
        ...
    </dependencies>
    ...
</project>

3. Add perforator-maven-plugin to the build plugins section.

<project>
    ...
    <build>
        ...
        <plugins>
            ...
            <plugin>
                <groupId>io.perforator.sdk</groupId>
                <artifactId>perforator-maven-plugin</artifactId>
                <version>1.2.1</version>
                <configuration>
                    ...
                </configuration>
            </plugin>
            ...
        </plugins>
        ...
    </build>
    ...
</project>

Configuration.

TestNGLoadGenerator configuration and execution are done via a dedicated maven plugin perforator-maven-plugin.

There are four required settings to run this load generator:
- apiClientId - client ID for API authentication. Please navigate to the API Settings page to get your API Client ID.
- apiClientSecret - client Secret for API authentication. Please navigate to the API Settings page to get your API Client Secret.
- projectKey - identifier of the project to create new execution records. Please refer to the "Projects" chapter to learn more about projects and get your project key.
- suiteXmlFile - location of the TestNG suite XML file to execute.


Additionally, the load generator supports many more optional settings allowing you to fine-tune load test execution.
Feel free to review all available options at the Settings tab. There are two major sets of configuration options:
- Settings with the "Scope: loadGenerator" define the overall behavior of the load generator.
- Settings with the "Scope: suite" control behavior of the individual suite.


It might be a case when you would like to apply different load testing scenarios and control such scenarios independently.
Perforator maven plugin allows you to configure multiple suites where every suite is executed independently.
Please take a look at the three configurations below, where the first two options define single suite execution, while the third one allows you to run multiple suites independently.

Overrides.

You can override almost all configuration settings from the command line or via environment variables.

Load generator uses the following order while resolving specific setting values:
1. Command line arguments.
2. Environment variables.
3. pom.xml setting.

This approach allows you to prepare a configuration file once and then execute it with different options.
A real-world example is when you store your project in a version control system like GIT, and you don't want to expose security keys.
So you can configure a new environment variable dedicated to the specific property, and a load generator automatically resolves it.

All Settings mention command-line argument and environment variable where an override is available.
For example, apiClientId has the following overrides:
- Command line arg: -DloadGenerator.apiClientId=...
- Environment variable: LOADGENERATOR_APICLIENTID=...

TestNG Tips.

The main goal of all Perforator-based load tests is to send commands and actions to remote browsers, so such browsers generate an actual load on your website.
You can send all necessary actions to remote browsers using the RemoteWebDriver instance.
Prebuilt example comes with the dedicated TestNG listener PerSuiteRemoteWebDriverManager responsible for starting a new browser session at the beginning of suite instance processing and automatically terminating it at the end.


It might be a case when the default behavior is not desirable, or you would like to tight control when to start and stop a browser session programmatically.
So you can call Perforator.startRemoteWebDriver() directly.
Please don't forget to terminate a browser session by calling RemoteWebDriver.quit() when you are done with your logic.
Otherwise, the browser cloud will become fully occupied with active browsers, and you will not be able to create a new one.


Note: Starting a new browser session takes around 2 seconds under the average load and five or even more seconds when all active browsers in the cloud drain the CPU heavily.
So, it is not suggested to start/stop browsers too frequently. Otherwise, such behavior can have a significant impact on load test results.

Transactions.

TestNG load generator automatically reports suite instance processing as Top-Level transaction, where a transaction name is generated as "suite - ${suite.name}".
Suite name is automatically extracted from suiteXmlFile.
For example, if your TestNG suite has a suite definition as <suite name="Example Load Testing Suite">, then a top-level transaction should be generated as "suite - Example Load Testing Suite".

The load generator additionally generates nested transactions for each test and method execution.
For example, you can expect the following transactions to be automatically generated:
- "test - Example Test".
- "method - com.example.ExampleTest.verify".

Additionally, the prebuilt example automatically starts a new remote browser at the beginning of suite instance processing and terminates it at the end.
Such actions automatically report nested transactions with the names below:
- "Open browser session".
- "Close browser session".

You can also report manual transactions if you would like to measure performance of the specific code blocks.
Such manually reported transactions have the following characteristics:
- Parent Transaction ID is automatically picked from the top-level transaction.
- Parent Transaction Name is automatically picked from the top-level transaction.
- Transaction ID is automatically generated in UUID format.
- Transaction Name is the name of transaction you manually supply.
- Transaction Type is a constant value of "Nested".
- Session ID is automatically picked from the browser session.

Note: Load generator submits transactions to the analytical system only when you execute your load test using browsers in the cloud.
For example, no transactions are reported if webDriverMode = local, which is a case for local.sh and local.cmd scripts.

Please take a look at the examples below on how to report transactions manually:

package com.example;

import io.perforator.sdk.loadgenerator.core.Perforator;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.testng.annotations.Test;

public class ExampleTest {
    
    @Test
    public void verify() throws Exception {
        RemoteWebDriver driver = PerSuiteRemoteWebDriverManager.getRemoteWebDriver();
        
        // Report a transaction for Runnable block
        Perforator.transactionally("Your transaction name #1", () -> {
            driver.navigate().to("https://verifications.perforator.io/");
            awaitToBeVisible(driver, "#async-container");
        });
        
        // Report a transaction for Supplier block
        WebElement container = Perforator.transactionally("Your transaction name #2", () -> {
            driver.navigate().to("https://verifications.perforator.io/");
            return awaitToBeVisible(driver, "#async-container");
        });
        
    }
    
    private static WebElement awaitToBeVisible(RemoteWebDriver driver, String cssSelector) {
        return new WebDriverWait(driver,30).until(
                ExpectedConditions.visibilityOfElementLocated(
                        By.cssSelector(cssSelector)
                )
        );
    }
    
}

Logging.

Load generator uses Log4j logging framework.
All "TestNG" examples come with the prebuilt log4j2.xml configuration file, so you can easily fine-tune the logging of your load test.
Please take a look at the default logging config below.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" strict="true"
    xmlns="http://logging.apache.org/log4j/2.0/config"
    xsi:schemaLocation="http://logging.apache.org/log4j/2.0/config 
    https://raw.githubusercontent.com/apache/logging-log4j2/master/log4j-core/src/main/resources/Log4j-config.xsd"
>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="[%d][%X{X-PERFORATOR-CONTEXT}][%c{1}][%4p] - %m%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console"/>
        </Root>
        <Logger name="org.openqa.selenium" level="error" />
    </Loggers>
</Configuration>
Additionally to the log4j2.xml configuration, the load generator supports the following options:
- logWorkerID - it adds worker ID to the logging context of the current thread.
- logSuiteInstanceID - it adds suite instance ID to the logging context of the current thread.
- logRemoteWebDriverSessionID - it adds browser session ID to the logging context of the current thread.
- logTransactionID - it adds currently active transaction ID to the logging context of the current thread.

Note: It is not suggested to change the logging level to DEBUG or TRACE when you run your load test at full speed because it will log too much information, and process output might become too overloaded, so please use it at your own risk.

Running.

All prebuilt examples come with ready-to-use scripts.
There is no enforced workflow on how to prepare and execute a load test, but here is a suggested one:
1. Make necessary changes in the perforator-maven-plugin configuration.
2. Make necessary changes in your TestNG-based tests.
3. Verify changes using local browsers.
4. Dry run your changes using a small number of remote browsers and ensure that all browser actions are not affected by network delays.
5. Execute your load test at full speed

Please take a look at the scripts below prepared for the suggested workflow.
Linux / macOS:
- local.sh - executes your TestNG suite(s) using Chrome browsers started locally. This script is helpful to verify that everything works as intended visually.
- local-headless.sh - it has the same purpose as local.sh, but valuable for CI/CD environments where a graphical environment is unavailable.
- local-debug.sh - it executes the logic using maven in debug mode, so you can attach external debugger from your IDE.
- cloud-dry-run.sh - it executes your TestNG suite(s) using only 10 browsers started in the cloud. The main idea of this script is to avoid extra charges when you need to ensure that your logic is executed correctly using cloud-based browsers. For example, it is not optimal to run your full-powered test using thousands of browsers and then, in a minute, realize that you have a simple mistake in css selector.
- cloud-full-run.sh - it executes your load test at full speed as defined in maven plugin configuration.

Windows:
- local.cmd - executes your TestNG suite(s) using Chrome browsers started locally. This script is helpful to verify that everything works as intended visually.
- local-headless.cmd - it has the same purpose as local.sh, but valuable for CI/CD environments where a graphical environment is unavailable.
- local-debug.cmd - it executes the logic using maven in debug mode, so you can attach external debugger from your IDE.
- cloud-dry-run.cmd - it executes your TestNG suite(s) using only 10 browsers started in the cloud. The main idea of this script is to avoid extra charges when you need to ensure that your logic is executed correctly using cloud-based browsers. For example, it is not optimal to run your full-powered test using thousands of browsers and then, in a minute, realize that you have a simple mistake in css selector.
- cloud-full-run.cmd - it executes your load test at full speed as defined in maven plugin configuration.

All scripts above are simple wrappers on top of maven.
Feel free to execute maven commands directly in the terminal for more granular control.
For example: `mvn clean test-compile perforator:testng -Dsuite.webDriverMode=cloud -Dsuite.concurrency=10`
Note: Please make sure that JDK 11+ and Maven 3 is installed on your computer and available in your terminal PATH.
Execute `mvn --version` in your terminal to validate both.

Debugging.

Sometimes it is helpful to debug how the load generator processes your TestNG-based tests.
All modern IDEs like IntelliJ IDEA, Eclipse, or Netbeans allow you to attach a debugger to the external process.
You can execute local-debug.sh(on Linux/macOS) or local-debug.cmd(on Windows) prebuilt scripts in your terminal, and it will start the maven process in debug mode so that you can attach an external debugger from your IDE.

Please use the following settings while attaching an external debugger to the maven process:
- transport: dt_socket
- host: localhost
- port: 8000

Note: It is not recommended to attach a debugger when running your suite(s) using browsers in the cloud, i.e., webDriverMode = cloud.
Cloud-based browsers are automatically killed if there is no activity within 60 seconds, so the debugger's extended blockage of the suite processor thread will automatically lead to remote browser termination.

Limitations.

Please keep in mind all the time that the load generator runs locally on your computer or server.
While browsers generating the load on your website are running in the cloud, the load generator itself is executed locally and is only responsible for orchestrating remote browsers.
Such behavior comes with certain requirements applied on local hardware and internet connection speed.

Here are minimal requirements to orchestrate every 500 remote browsers:
- 2 CPU cores
- 2 GB RAM
- 20 Mbps Internet bandwidth

For example, you need the following resources locally to run a load test with 2000 remote browsers:
- CPU cores: 2000 ÷ 500 × 2 Cores = 8 Cores
- RAM: 2000 ÷ 500 × 2GB = 8 GB
- Internet: 2000 ÷ 500 × 20Mbps = 80 Mbps

Warning: You might see very confusing load test statistics and high errors rate if you don't meet minimal hardware and internet bandwidth requirements.