Scenario Insights: Quality and Reliability Analytics

In ACCELQ, a Scenario is a parameterized test that represents a business process on the application under test. Each Scenario can have one or more test cases, where each test case is a data-driven instance of the Scenario representing a specific permutation of that business process.

Scenario Insights is an analytics panel that surfaces the health, stability, and performance characteristics of a Scenario based on its execution history. It appears on the Scenario information page as a side panel, working for both Automation and Manual Scenarios.

In enterprise test environments where Scenarios accumulate hundreds or thousands of executions across multiple browsers, locales, and application environments, identifying patterns in test outcomes becomes impractical through manual inspection. Scenario Insights computes and surfaces these patterns automatically, helping QA teams focus investigation effort where it will have the most impact.

The panel analyzes all test case executions within a 90-day tracking window. Every metric is computed programmatically from execution data. No manual tagging or configuration is required beyond the standard execution parameters and custom fields already captured by ACCELQ. Since all test cases within a Scenario represent the same business process, the panel uses this relationship to distinguish data-level issues (failures in specific test cases) from systemic problems (failures spread broadly).

Accessing Scenario Insights

The panel appears on the right side of the Scenario information page for any Scenario with test cases defined. The header shows three scoping numbers: test case count, total executions in the tracking window, and the window duration (90 days). When no executions exist, a clean empty state is shown with the count of configured test cases.

Health Trends

The two primary indicators of Scenario quality.

Pass Rate is the weighted average pass rate across all test cases, weighted by execution count. Test cases with more executions have proportionally more influence. The delta (▲/▼) shows week-over-week change.

Flakiness measures inconsistency in test outcomes. High flakiness means both passes and failures should be treated with lower confidence. Scenario-level flakiness is the weighted average of individual test case flakiness scores. It requires a minimum number of executions and is hidden until that threshold is met.

Both metrics include a 12-week sparkline. Weeks with no executions appear as gaps. Start and end values are labeled.

Effective Pass Rate appears as a callout when flakiness exceeds 5%. Calculated as Pass Rate x (1 - Flakiness/100), it represents the true confidence in outcomes after accounting for inconsistency. A Scenario at 87% pass rate with 14% flakiness has an effective pass rate of roughly 75%.

Recent Runs

Displays the last 5 executions as compact stacked bars with a detail panel. Each bar shows the proportion of passed (green), failed (red), and skipped (gray) test cases for that run. All execution sources are included: direct runs, suite executions, and CI triggers.

Clicking a bar shows its details. The most recent run is selected by default. The detail panel shows the Job ID (clickable, navigates to full result), relative timestamp, pass/fail/skip counts, and duration.

Partial Runs: When test cases are unchecked before execution, they are excluded from scope entirely and are not counted as skipped. "Skipped" applies only to test cases selected but skipped during execution (for example, due to a dependency failure). Partial runs show "X of Y executed" in the detail panel.

Breakdown by Execution Parameter

Identifies whether failures concentrate in specific execution conditions. This is one of the most actionable sections of the panel: a Scenario that fails only on a particular browser, or only in a specific locale, tells a very different story than one that fails uniformly.

A dropdown lets you select any execution parameter recorded for this Scenario's test case executions. For Automation Scenarios, the standard parameters include Browser/OS, Mobile Device/OS, App Environment, and App Variant, alongside any custom execution parameters defined for the project. For Manual Scenarios, the dropdown contains only custom execution parameters. The table shows every value for the selected parameter with its pass rate, flakiness, and execution count. Values are color-coded based on their health: pass rates and flakiness figures that fall outside healthy thresholds are highlighted to draw attention to problem areas.

Both pass rate and flakiness are shown because they tell different stories. A parameter value with low pass rate and low flakiness points to a real, environment-specific bug that reproduces consistently. A value with low pass rate and high flakiness suggests instability in that environment rather than a deterministic defect. These require different responses.

When no execution parameters have been recorded, the section indicates that environment data requires execution parameters to be configured.

Run Duration

Per-test-case timing computed from passed executions only (failed runs may abort early and would skew timing downward).

Three measures are shown: Average, P50 (median), and P95 (95th percentile). When P95 is trending upward compared to the previous 4-week window, it is highlighted as a potential performance degradation signal.

Below the per-test-case metrics, an Estimated Scenario Time shows the average per-test-case time multiplied by the number of test cases. This is a sequential baseline; actual duration depends on parallelism. The calculation is labeled explicitly (for example, "47 TCs x avg").

Timing metrics require at least one passed execution. When all runs have failed, the section indicates this.

Empty and Partial States

The panel handles missing data gracefully rather than showing misleading values.

• No executions: Full empty state with configured test case count
• Too few executions for flakiness: Pass rate shown alone; flakiness hidden with explanatory note
• All runs failed: Run Duration indicates that timing needs at least one passed execution
• Fewer than 5 runs: Only available bars shown in Recent Runs
• Trend gaps: Sparkline shows a visible break rather than connecting surrounding weeks
• No environment data: Section indicates that execution parameters need to be configured
• No anomalies: Table shows all values performing near the Scenario average
• Partial runs: Bars reflect only in-scope test cases; detail panel shows "X of Y executed"

Reading the Panel

The panel is designed to be read top-to-bottom as a triage flow.

Health Trends gives the headline: is this Scenario healthy, and is it improving or degrading? The pass rate tells you how often the business process succeeds. Flakiness tells you how much to trust that number. A Scenario with high pass rate but rising flakiness is producing unreliable signals. The effective pass rate callout, when it appears, makes this tradeoff concrete.

Recent Runs provides current context. This is especially useful after a deployment or environment change. Look for patterns in the bars: consecutive degradation suggests a regression, while alternating results suggest instability. A fully green bar among otherwise mixed runs may indicate a partial execution that skipped the failing test cases.

Breakdown by Execution Parameter localizes the problem to specific conditions. Start with a quick scan of the table to see if any particular browser, locale, or storefront stands out. A value with low pass rate and high flakiness needs a different response (environment instability) than one with low pass rate and low flakiness (likely a real environment-specific bug).

Run Duration supports pipeline planning and performance regression detection. A rising P95 with stable pass rates may indicate the application is getting slower without actually breaking.

Every section includes a help icon (?) that expands an in-context explanation.