​

Plans

Test Plans let you schedule and organize the execution of multiple test cases at once with comprehensive analytics, smart notifications, and advanced filtering. You can include tests by selecting individual test cases and/or entire folders, with intelligent test selection and flaky test detection.

​

Video overview

​

What are Test Plans?

A Test Plan is a container that groups together:

• Folders: Include all test cases within selected folders (including nested subfolders) with dynamic selection
• Test Cases: Individual tests you select explicitly with advanced filtering by tags
• Scheduling Configuration: When and how often to run with flexible recurrence patterns
• Environment Settings: Which environment to run against
• Smart Notifications: Configurable email alerts with pass rate thresholds
• Analytics & Insights: Comprehensive metrics, trends, and flaky test detection

​

Creating a Test Plan

​

Basic Information

1. Plan Name: A descriptive name for your plan (up to 50 characters)
2. Environment: Select the environment to run against (or choose “No Environment”)

​

Adding Tests

You can add tests to your plan using the advanced test selection interface:

​

Dynamic Folder Selection

Add one or more folders to include all test cases within them. This provides:

• Complete Selection: Includes all current and future tests in the folder
• Automatic Updates: New tests added to folders are automatically included
• Hierarchical Organization: Nested subfolders are included when parent folders are selected

​

Individual Test Case Selection

Add specific test cases for more granular control with advanced filtering:

• Bulk Selection: Select multiple test cases at once
• Partial Selection: Mix folder and individual test selections

​

Test Selection Interface

The new test selection interface provides:

• Tree View: Hierarchical folder structure with expandable nodes
• Smart Selection: Visual indicators for complete vs partial selections
• Bulk Operations: Select all tests in a folder or clear all selections

​

Smart Email Notifications

Configure email notifications with advanced filtering:

1. Email Addresses: Enter up to 10 email addresses (comma or Enter to add)
2. Smart Filtering: Enable “Send only when pass rate falls below threshold”

​

Notification Options

Always Send Notifications (default):

• Plan execution starts
• Plan execution completes
• Any failures occur

Smart Notifications (when enabled):

• Only sends emails when pass rate falls below your threshold (default: 80%)
• Reduces notification noise for stable test plans
• Helps focus attention on significant quality issues

​

Pass Rate Threshold

When smart notifications are enabled:

• Set a pass rate threshold (0-100%)
• Default threshold is 80%
• Notifications only sent when actual pass rate is below threshold
• Helps distinguish between minor issues and significant problems

​

Scheduling

Plans support flexible scheduling using a visual date and time picker:

​

Schedule Options

• One‑time execution: Run once at a specific date and time
• Recurring execution: Daily, weekly, monthly, or yearly cadence
• Custom recurrence: Build complex schedules

​

Schedule Examples

• Daily at 9 AM: Morning smoke tests
• Weekly on Fridays: End‑of‑week regression
• Monthly on the 1st: Comprehensive coverage
• Custom patterns: Every weekday, specific month days, etc.

​

Managing Plans

​

Viewing Plan Details

The enhanced plan details page provides comprehensive insights and management capabilities:

​

Tabs

• Overview: Key metrics, trends, and flaky test analysis
  
• Tests: View the selected folders and test cases for this plan
  
• Runs: Historical data, charts, and detailed run information
  

​

Running Plans

​

Manual Execution

Click Run All to execute the plan immediately against the selected environment. Note: The Run button is disabled if the plan contains no folders or test cases.

​

Scheduled Execution

Plans automatically run according to their configured schedule using the specified environment.

​

Editing Plans

1. Click Edit on the plan details page
2. Modify any configuration (name, schedule, environment, notifications)
3. Click Save to apply changes

​

Managing Test Selection

​

Adding Tests

1. Click Edit Tests on the plan details page
2. Use the advanced test selection interface:
   • Browse Folders: Expand folder tree to see all available tests
   • Filter by Tags: Use tag filters to narrow down test selection
   • Search Tests: Find specific tests by name or content
   • Bulk Selection: Select multiple tests or entire folders at once
3. Click Save to add them to the plan

​

Removing Tests

1. Click Edit Tests on the plan details page
2. In the test selection interface:
   • Deselect Folders: Uncheck folders to remove all their tests
   • Deselect Individual Tests: Uncheck specific test cases
   • Clear All: Remove all current selections
3. Click Save to apply changes

​

Analytics & Insights

Test Plans now include comprehensive analytics to help you understand test performance, identify trends, and spot issues early.

​

Overview Dashboard

The Overview tab provides key metrics at a glance:

​

Key Performance Indicators

• Pass Rate: Current pass rate with trend indicators (↑/↓ from last run)
• Total Tests: Number of tests included with pass/fail breakdown
• Average Duration: Mean execution time with comparison to previous runs
• Reliability Score: Overall test stability metric
• Last Run: When the plan was last executed
• Next Occurrence: When the plan will run next (for scheduled plans)

​

Trend Analysis

Each metric includes:

• Current Value: Real-time performance data
• Trend Indicators: Visual arrows showing improvement or decline
• Delta Values: Exact change from previous run
• Color Coding: Green for improvements, red for declines

​

Test Results Trends

The Test Runs tab provides detailed historical analysis:

​

Results Trend Chart

• Area Chart: Visual representation of pass/fail rates over time
• Date Range Picker: Filter data by custom date ranges
• Environment Context: Shows which environment each run used
• Interactive Tooltips: Hover for detailed run information

​

Duration Analysis

• Duration Trend Chart: Track how execution times change over time
• Average vs Median: Compare mean and median durations
• Performance Insights: Identify tests that are getting slower
• Optimization Opportunities: Spot tests that need attention

​

Flaky Test Detection

​

Top Flaky Tests List

Automatically identifies and ranks the most unreliable tests:

• Flakiness Score: Percentage indicating test instability
• Pass/Fail Counts: Historical success and failure rates
• Last Status: Most recent execution result
• Quick Access: Click to view detailed test information

​

Flakiness Insights

• Top 20 Flaky Tests: Focus on the most problematic tests
• Score Calculation: Based on recent run history
• Actionable Data: Helps prioritize test maintenance efforts
• Trend Monitoring: Track if flakiness is improving or worsening

​

Historical Data

​

Run History Table

• Comprehensive Details: Status, duration, timestamps, and environment
• Sortable Columns: Organize by any metric
• Quick Actions: Direct links to detailed run results
• Status Indicators: Visual status badges for quick scanning

​

Performance Metrics

• Execution Times: Track how long tests take to run
• Success Rates: Monitor pass/fail trends over time
• Environment Performance: Compare results across different environments
• Resource Usage: Understand test execution patterns

​

Best Practices

​

Organizing Plans

• Feature‑based: Group tests by product area or functionality
• Environment‑specific: Separate plans for staging vs production checks
• Priority‑based: Smoke tests vs full regression suites
• Tag‑based: Use consistent tagging for better test organization

​

Test Selection Strategy

• Folder Selection: Use for complete feature coverage that includes future tests
• Individual Selection: Use for critical path tests or specific test combinations
• Mixed Approach: Combine folder and individual selections for maximum flexibility
• Tag Filtering: Leverage tags to quickly select related tests across folders

​

Scheduling Strategy

• Avoid Peak Hours: Schedule heavy runs during low-traffic periods
• Consider Time Zones: Account for recipients’ time zones for notifications
• Balance Frequency: Find the right balance between critical paths and broad coverage
• Environment Rotation: Use different environments for different schedules

​

Notification Management

• Smart Notifications: Enable pass rate thresholds to reduce noise
• Relevant Recipients: Add only stakeholders who need to know about results
• Clear Naming: Use descriptive plan names for easy identification
• Escalation Paths: Establish clear processes for handling failures

​

Analytics Utilization

• Monitor Trends: Regularly check pass rate and duration trends
• Address Flaky Tests: Prioritize fixing tests with high flakiness scores
• Performance Optimization: Use duration charts to identify slow tests
• Environment Comparison: Compare results across environments to spot issues

​

Troubleshooting

​

Common Issues

Plan Won’t Run

• Ensure at least one folder or test case is selected
• Verify the selected environment is available
• Check that no conflicting run is active
• Verify plan is not disabled (check isActive status)

Schedule Not Running

• Verify schedule configuration and timezone settings
• Ensure sufficient worker capacity
• Check if plan is disabled or paused
• Verify environment availability at scheduled times

Missing Notifications

• Verify email formatting and addresses
• Check spam/junk folders
• Ensure notifications are enabled
• Check if smart notifications are filtering out emails due to pass rate threshold
• Verify pass rate threshold is set appropriately

Analytics Not Updating

• Wait for next run to complete for fresh data
• Check if date range filters are applied correctly
• Verify environment selection matches run environment
• Ensure sufficient historical data for trend calculations

Test Selection Issues

• Check if folders contain tests (expand to verify)
• Verify tag filters aren’t excluding all tests
• Ensure test cases aren’t disabled or snoozed
• Check for conflicts between folder and individual selections

Flaky Test Detection

• Ensure sufficient run history for accurate flakiness scores
• Check that tests have run multiple times recently
• Verify environment consistency across runs
• Review test stability improvements over time

​

Getting Help

1. Check Analytics: Review the Overview tab for performance insights
2. Examine Run History: Use the Test Runs tab to analyze execution patterns
3. Review Test Selection: Verify folder and test case selections are correct
4. Check Flaky Tests: Use the flaky test list to identify problematic tests
5. Contact Support: Provide run IDs, plan details, and specific error messages