Guide

Embedded Load Generator

Overview.

"Embedded Load Generator" is the most dynamic and flexible load generator integrated with the Perforator platform allowing you to define load testing scenarios directly in code.
You need advanced Java programming skills to utilize this load generator.
Also, it would be best if you have at least a basic understanding of the Selenium framework since the actual logic of the load testing scenarios is implemented via calling RemoteWebDriver directly.

All integrations with the Perforator platform are open-sourced.
Please take a look at our GitLab repository "Perforator SDK Java" if you are interested in the internal implementation of the load generator.

Configuration and execution of this load generator is done via a dedicated Maven plugin, while actual logic implementation is done via extending AbstractSuiteProcessor class.
The example below can give you a sense of what a real-world configuration might look like.

Jump Start.

1. Download and unarchive prebuilt example.
2. Start editing pom.xml with your favorite text editor.
3. Update apiClientId property with your own value. Please navigate to the API Settings page to get your API Client ID.
4. Update apiClientSecret property with your own value. Please navigate to the API Settings page to get your API Client Secret.
5. Update projectKey property with your own value. Please refer to the "Projects" chapter to learn more about projects and get your project key.
6. Execute cloud-full-run.sh(on Linux/macOS) or cloud-full-run.cmd(on Windows) using your command line terminal - it should start a lightweight load test with 50 concurrent browsers in the cloud running for 5 minutes.
7. Load generator prints a link to view performance test statistics when all cloud resources are ready, and the actual load test begins. Please copy such a link and open it in your browser.

Note: Please make sure that JDK 11+ and Maven 3 is installed on your computer and available in your terminal PATH.
Execute `mvn --version` in your terminal to validate both.

Maven Build.

Prebuilt example comes with ready-to-use pom.xml where all dependencies and plugins are already configured.
It might be a case when you would like to integrate Embedded Load Generator into an existing maven project or prefer to configure the build from the ground up, so please take a look at the instructions below.

1. All compiled and well-tested artifacts of the Perforator SDK are uploaded to GitLab based maven repo.
Please update your pom.xml by adding pluginRepositories and repositories pointed to our maven repo, so maven can automatically download all required artifacts.

<project>
    ...
    <pluginRepositories>
        ...
        <pluginRepository>
            <id>perforator-gitlab-maven</id>
            <url>https://gitlab.com/api/v4/projects/32457860/packages/maven</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
            <releases>
                <enabled>true</enabled>
            </releases>
        </pluginRepository>
        ...
    </pluginRepositories>
    ...
    <repositories>
        ...
        <repository>
            <id>perforator-gitlab-maven</id>
            <url>https://gitlab.com/api/v4/projects/32457860/packages/maven</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
            <releases>
                <enabled>true</enabled>
            </releases>
        </repository>
        ...
    </repositories>
    ...
</project>

2. Add perforator-sdk-load-generator-embedded artifact to your dependencies list, so required classes to implement your suite processor are available during compilation and execution phases.

<project>
    ...
    <dependencies>
        ...
        <dependency>
            <groupId>io.perforator.sdk</groupId>
            <artifactId>perforator-sdk-load-generator-embedded</artifactId>
            <version>1.4.2</version>
        </dependency>
        ...
    </dependencies>
    ...
</project>

3. Add perforator-maven-plugin to the build plugins section.

<project>
    ...
    <build>
        ...
        <plugins>
            ...
            <plugin>
                <groupId>io.perforator.sdk</groupId>
                <artifactId>perforator-maven-plugin</artifactId>
                <version>1.4.2</version>
                <configuration>
                    ...
                </configuration>
            </plugin>
            ...
        </plugins>
        ...
    </build>
    ...
</project>

Configuration.

EmbeddedLoadGenerator configuration and execution are done via a dedicated maven plugin perforator-maven-plugin.

There are four required settings to run this load generator:
- apiClientId - client ID for API authentication. Please navigate to the API Settings page to get your API Client ID.
- apiClientSecret - client Secret for API authentication. Please navigate to the API Settings page to get your API Client Secret.
- projectKey - identifier of the project to create new execution records. Please refer to the "Projects" chapter to learn more about projects and get your project key.
- processorClass - fully qualified class name of the suite processor, where a processor should either extend AbstractSuiteProcessor or implement EmbeddedSuiteProcessor.


Additionally, the load generator supports many more optional settings allowing you to fine-tune load test execution.
Feel free to review all available options at the Settings tab. There are two major sets of configuration options:
- Settings with the "Scope: loadGenerator" define the overall behavior of the load generator.
- Settings with the "Scope: suite" control behavior of the individual suite.


It might be a case when you would like to apply different load testing scenarios and control such scenarios independently.
Perforator maven plugin allows you to configure multiple suites where every suite is executed independently.
Please take a look at the three configurations below, where the first two options define single suite execution, while the third one allows you to run multiple suites independently.

Overrides.

You can override almost all configuration settings from the command line or via environment variables.

Load generator uses the following order while resolving specific setting values:
1. Command line arguments.
2. Environment variables.
3. pom.xml setting.

This approach allows you to prepare a configuration file once and then execute it with different options.
A real-world example is when you store your project in a version control system like GIT, and you don't want to expose security keys.
So you can configure a new environment variable dedicated to the specific property, and a load generator automatically resolves it.

All Settings mention command-line argument and environment variable where an override is available.
For example, apiClientId has the following overrides:
- Command line arg: -DloadGenerator.apiClientId=...
- Environment variable: LOADGENERATOR_APICLIENTID=...

Suite Processor.

Suite processor is responsible for executing actual scenarios and commands within remote browser, so it drives the logic of the load test.
Every suite configuration of the embedded load generator requires a dedicated implementation of the suite processor.
You have two options to develop a suite processor: extend your class from AbstractSuiteProcessor or implement it from EmbeddedSuiteProcessor.

Note: Implementing your suite instance from an interface EmbeddedSuiteProcessor provides you absolute freedom managing your own logic, but it comes with certain limitations.
You have to manually open a new RemoteWebDriver session at the beginning of suite instance processing, manually close it at the end, and manually report transactions of such activities.
So, it is suggested to extend from AbstractSuiteProcessor, since it automatically takes care of the "glue" activities.


Extending your suite processor implementation from AbstractSuiteProcessor requires only one method override - processSuite, where you implement your load test scenario logic.
Suite processor automatically passes the following parameters to your method implementation:
- iterationNumber is suite iteration number. It starts from 0 incrementing for every new suite instance execution. Such counter is maintained on a suite level, so counters from different suites are incremented independently.
- suiteInstanceID is an ID of the invoked suite instance.
- suiteConfig is a reference to the currently executed suite instance config.
- remoteWebDriver is a reference to just opened RemoteWebDriver, which is automatically opened before suite instance processing starts and automatically closed once suite instance processing is completed.

Please take a look at example suite processor implementation:

Transactions.

Embedded load generator automatically reports suite instance processing as Top-Level transaction, where a transaction name is generated as "suite - ${suite.name}".
Suite name is automatically generated from processorClass if you don't directly supply suite name.
For example, if you have a suite processor com.example.ExampleSuiteProcessor, then a top-level transaction for suite instance processing should be generated as "suite - com.example.ExampleSuiteProcessor".

Additionally, AbstractSuiteProcessor automatically reports nested transactions related to opening and closing RemoteWebDriver session:
- "Open browser session".
- "Close browser session".

You can also report manual transactions if you would like to measure performance of the specific code blocks.
Such manually reported transactions have the following characteristics:
- Parent Transaction ID is automatically picked from the top-level transaction.
- Parent Transaction Name is automatically picked from the top-level transaction.
- Transaction ID is automatically generated in UUID format.
- Transaction Name is the name of transaction you manually supply.
- Transaction Type is a constant value of "Nested".
- Session ID is automatically picked from the browser session.

Note: Load generator submits transactions to the analytical system only when you execute your load test using browsers in the cloud.
For example, no transactions are reported if webDriverMode = local, which is a case for local.sh and local.cmd scripts.

Please take a look at the examples below on how to report transactions manually:

package com.example;

import io.perforator.sdk.loadgenerator.embedded.AbstractSuiteProcessor;
import io.perforator.sdk.loadgenerator.embedded.EmbeddedSuiteConfig;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class ExampleSuiteProcessor extends AbstractSuiteProcessor {

    @Override
    protected void processSuite(long iterationNumber, String suiteInstanceID, EmbeddedSuiteConfig suiteConfig, RemoteWebDriver driver) {
        // Report a transaction for Runnable block
        transactionally("Your transaction name #1", () -> {
            driver.navigate().to("https://verifications.perforator.io/");
            awaitToBeVisible(driver, "#async-container");
        });
        
        // Report a transaction for Supplier block
        WebElement container = transactionally("Your transaction name #2", () -> {
            driver.navigate().to("https://verifications.perforator.io/");
            return awaitToBeVisible(driver, "#async-container");
        });
    }
    
    private static WebElement awaitToBeVisible(RemoteWebDriver driver, String cssSelector) {
        return new WebDriverWait(driver,30).until(
                ExpectedConditions.visibilityOfElementLocated(
                        By.cssSelector(cssSelector)
                )
        );
    }
    
}
All AbstractSuiteProcessor#transactionally methods are simple wrappers on top of helper methods from Perforator.java.
So, please call Perforator#transactionally to report transactions from classes other than AbstractSuiteProcessor.

Logging.

Load generator uses Log4j logging framework.
All "embedded" examples come with the prebuilt log4j2.xml configuration file, so you can easily fine-tune the logging of your load test.
Please take a look at the default logging config below.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" strict="true"
    xmlns="http://logging.apache.org/log4j/2.0/config"
    xsi:schemaLocation="http://logging.apache.org/log4j/2.0/config 
    https://raw.githubusercontent.com/apache/logging-log4j2/master/log4j-core/src/main/resources/Log4j-config.xsd"
>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="[%d][%X{X-PERFORATOR-CONTEXT}][%c{1}][%4p] - %m%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console"/>
        </Root>
        <Logger name="org.openqa.selenium" level="error" />
    </Loggers>
</Configuration>
Additionally to the log4j2.xml configuration, the load generator supports the following options:
- logWorkerID - it adds worker ID to the logging context of the current thread.
- logSuiteInstanceID - it adds suite instance ID to the logging context of the current thread.
- logRemoteWebDriverSessionID - it adds browser session ID to the logging context of the current thread.
- logTransactionID - it adds currently active transaction ID to the logging context of the current thread.

Note: It is not suggested to change the logging level to DEBUG or TRACE when you run your load test at full speed because it will log too much information, and process output might become too overloaded, so please use it at your own risk.

Running.

All prebuilt examples come with ready-to-use scripts.
There is no enforced workflow on how to prepare and execute a load test, but here is a suggested one:
1. Make necessary changes in the perforator-maven-plugin configuration.
2. Make necessary changes in your suite processor class.
3. Verify changes using local browsers.
4. Dry run your changes using a small number of remote browsers and ensure that all browser actions are not affected by network delays.
5. Execute your load test at full speed

Please take a look at the scripts below prepared for the suggested workflow.
Linux / macOS:
- local.sh - executes your suite processor using Chrome browsers started locally. This script is helpful to verify that everything works as intended visually.
- local-headless.sh - it has the same purpose as local.sh, but valuable for CI/CD environments where a graphical environment is unavailable.
- local-debug.sh - it executes the logic using maven in debug mode, so you can attach external debugger from your IDE.
- cloud-dry-run.sh - it executes your suite processor using only 10 browsers started in the cloud. The main idea of this script is to avoid extra charges when you need to ensure that your logic is executed correctly using cloud-based browsers. For example, it is not optimal to run your full-powered test using thousands of browsers and then, in a minute, realize that you have a simple mistake in css selector.
- cloud-full-run.sh - it executes your load test at full speed as defined in maven plugin configuration.

Windows:
- local.cmd - executes your suite processor using Chrome browsers started locally. This script is helpful to verify that everything works as intended visually.
- local-headless.cmd - it has the same purpose as local.sh, but valuable for CI/CD environments where a graphical environment is unavailable.
- local-debug.cmd - it executes the logic using maven in debug mode, so you can attach external debugger from your IDE.
- cloud-dry-run.cmd - it executes your suite processor using only 10 browsers started in the cloud. The main idea of this script is to avoid extra charges when you need to ensure that your logic is executed correctly using cloud-based browsers. For example, it is not optimal to run your full-powered test using thousands of browsers and then, in a minute, realize that you have a simple mistake in css selector.
- cloud-full-run.cmd - it executes your load test at full speed as defined in maven plugin configuration.

All scripts above are simple wrappers on top of maven.
Feel free to execute maven commands directly in the terminal for more granular control.
For example: `mvn clean test-compile perforator:embedded -Dsuite.webDriverMode=cloud -Dsuite.concurrency=10`
Note: Please make sure that JDK 11+ and Maven 3 is installed on your computer and available in your terminal PATH.
Execute `mvn --version` in your terminal to validate both.

Debugging.

Sometimes it is helpful to debug suite processor implementation using an IDE debugger.
You can execute local-debug.sh(on Linux/macOS) or local-debug.cmd(on Windows) prebuilt scripts in your terminal, and it will start the maven process in debug mode so that you can attach an external debugger from your IDE.

Please use the following settings while attaching an external debugger to the maven process:
- transport: dt_socket
- host: localhost
- port: 8000

Note: It is not recommended to attach a debugger when running your suite(s) using browsers in the cloud, i.e., webDriverMode = cloud.
Cloud-based browsers are automatically killed if there is no activity within 60 seconds, so the debugger's extended blockage of the suite processor thread will automatically lead to remote browser termination.

Limitations.

Please keep in mind all the time that the load generator runs locally on your computer or server.
While browsers generating the load on your website are running in the cloud, the load generator itself is executed locally and is only responsible for orchestrating remote browsers.
Such behavior comes with certain requirements applied on local hardware and internet connection speed.

Here are minimal requirements to orchestrate every 500 remote browsers:
- 2 CPU cores
- 2 GB RAM
- 20 Mbps Internet bandwidth

For example, you need the following resources locally to run a load test with 2000 remote browsers:
- CPU cores: 2000 ÷ 500 × 2 Cores = 8 Cores
- RAM: 2000 ÷ 500 × 2GB = 8 GB
- Internet: 2000 ÷ 500 × 20Mbps = 80 Mbps

Warning: You might see very confusing load test statistics and high errors rate if you don't meet minimal hardware and internet bandwidth requirements.