Verify Web Performance Metrics during test execution

Subscription: Enterprise Edition    |   Availability: Release 6.0

When running Automation tests on web applications, you can enable the validation of Web Performance metrics. This will comprehensively validate key performance metrics related to the user's experience of application performance.

This validation is achieved contextually for every page in your application as a functional automation test is executed. No explicit effort is required from a test development point of view. Since the process is embedded in the functional test cycle, all the pages touched during the automation test are covered for Web Performance automatically. This avoids an otherwise tedious effort to cover various paths and pages involved in an application flow.

What is measured in Web Performance?

When you enable Web Performance, the following performance metrics are collected for the currently loaded page in the application. These metrics include rich “user-centric” parameters that determine how the end user perceives the application performance and not just the time it takes to load the page.

Load Time

Measures the amount of time it took to load the page.

First Contentful Paint (FCP)

First Contentful Paint marks the time at which the first text or image is painted.

Largest Contentful Paint

Largest Contentful Paint marks the time at which the largest text or image is painted.

Total Blocking Time

Sum of all time periods between FCP and Time to Interactive, when task length exceeded 50ms.

Cumulative Layout Shift

Cumulative Layout Shift measures the movement of visible elements within the viewport as the page loads.

Enable Web Performance Validation

Click Non-Functional Testing section in the Run modal and enable the Web Performance. This will automatically kick off the measurement of performance metrics at the beginning of each Action in the test Scenario.

You may also set up a failure threshold as applicable to the environment where the test is running. More about this in subsequent sections of this article.

Establishing Baseline

For the metrics being collected, each page in the test application may define an acceptable baseline. This baseline can be captured during the first time the test is executed with the Web Performance turned on. Once established, subsequent runs will compare against these values. In the test report, simply click on "Establish as Baseline" and update the baseline values.

Each page in your application may have its own set of acceptable values for various metrics.

Execution and Reporting

Once Web Performance is enabled and a baseline is established, performance metrics are compared at the beginning of each step (Action) in the test Scenario. This is executed right after the page synch is completed.

Test Report shows a clear comparison of various metrics between the expected baseline and the actual result.

Specifying SLA threshold

When you run a test on different application environments, there may be a difference in application performance and responsiveness depending on the deployment infrastructure. ACCELQ provides an ability to configure this difference as an "SLA threshold". This threshold is applied on top of the established baseline.

For example, consider the Load time baseline for the Login Page is set as 30 sec. When you are running the test on QA Environment, you may set the SLA threshold as 50%. In this case, the performance step fails only if the Load time exceeds 45 sec (50% above the threshold of 30 sec.). You may also set up a negative threshold if you need to verify that the currently executing environment is faster than the baseline environment.

You can set the SLA Threshold in the Settings on Run modal.