diff --git a/src/content/docs/project/analytics/images/1-display-settings.png b/src/content/docs/project/analytics/images/1-display-settings.png new file mode 100644 index 00000000..dfab2690 Binary files /dev/null and b/src/content/docs/project/analytics/images/1-display-settings.png differ diff --git a/src/content/docs/project/analytics/images/10-duplicate-chart.png b/src/content/docs/project/analytics/images/10-duplicate-chart.png new file mode 100644 index 00000000..27d11773 Binary files /dev/null and b/src/content/docs/project/analytics/images/10-duplicate-chart.png differ diff --git a/src/content/docs/project/analytics/images/11-chart-timeline.png b/src/content/docs/project/analytics/images/11-chart-timeline.png new file mode 100644 index 00000000..4820b8f3 Binary files /dev/null and b/src/content/docs/project/analytics/images/11-chart-timeline.png differ diff --git a/src/content/docs/project/analytics/images/12-defects-failures.png b/src/content/docs/project/analytics/images/12-defects-failures.png new file mode 100644 index 00000000..fd2ccd65 Binary files /dev/null and b/src/content/docs/project/analytics/images/12-defects-failures.png differ diff --git a/src/content/docs/project/analytics/images/13-linked-items.png b/src/content/docs/project/analytics/images/13-linked-items.png new file mode 100644 index 00000000..8aa63c71 Binary files /dev/null and b/src/content/docs/project/analytics/images/13-linked-items.png differ diff --git a/src/content/docs/project/analytics/images/14-flaky-tests.png b/src/content/docs/project/analytics/images/14-flaky-tests.png new file mode 100644 index 00000000..b0a880ce Binary files /dev/null and b/src/content/docs/project/analytics/images/14-flaky-tests.png differ diff --git a/src/content/docs/project/analytics/images/15-label-statistics-filters.png b/src/content/docs/project/analytics/images/15-label-statistics-filters.png new file mode 100644 index 00000000..0d75d823 Binary files /dev/null and b/src/content/docs/project/analytics/images/15-label-statistics-filters.png differ diff --git a/src/content/docs/project/analytics/images/2-analytics-calendar.png b/src/content/docs/project/analytics/images/2-analytics-calendar.png new file mode 100644 index 00000000..3b827400 Binary files /dev/null and b/src/content/docs/project/analytics/images/2-analytics-calendar.png differ diff --git a/src/content/docs/project/analytics/images/3-automation-coverage.png b/src/content/docs/project/analytics/images/3-automation-coverage.png new file mode 100644 index 00000000..d817c5c6 Binary files /dev/null and b/src/content/docs/project/analytics/images/3-automation-coverage.png differ diff --git a/src/content/docs/project/analytics/images/4-manual-to-automated-widget.png b/src/content/docs/project/analytics/images/4-manual-to-automated-widget.png new file mode 100644 index 00000000..320d34e5 Binary files /dev/null and b/src/content/docs/project/analytics/images/4-manual-to-automated-widget.png differ diff --git a/src/content/docs/project/analytics/images/5-add-custom-chart.png b/src/content/docs/project/analytics/images/5-add-custom-chart.png new file mode 100644 index 00000000..5130ae9b Binary files /dev/null and b/src/content/docs/project/analytics/images/5-add-custom-chart.png differ diff --git a/src/content/docs/project/analytics/images/6-custom-chart-config.png b/src/content/docs/project/analytics/images/6-custom-chart-config.png new file mode 100644 index 00000000..2c68c0d8 Binary files /dev/null and b/src/content/docs/project/analytics/images/6-custom-chart-config.png differ diff --git a/src/content/docs/project/analytics/images/7-1-chart-export-options.png b/src/content/docs/project/analytics/images/7-1-chart-export-options.png new file mode 100644 index 00000000..f9644b23 Binary files /dev/null and b/src/content/docs/project/analytics/images/7-1-chart-export-options.png differ diff --git a/src/content/docs/project/analytics/images/7-custom-chart-output.png b/src/content/docs/project/analytics/images/7-custom-chart-output.png new file mode 100644 index 00000000..5d361499 Binary files /dev/null and b/src/content/docs/project/analytics/images/7-custom-chart-output.png differ diff --git a/src/content/docs/project/analytics/images/8-labels-dropdown.png b/src/content/docs/project/analytics/images/8-labels-dropdown.png new file mode 100644 index 00000000..dcfd6315 Binary files /dev/null and b/src/content/docs/project/analytics/images/8-labels-dropdown.png differ diff --git a/src/content/docs/project/analytics/images/9-color-picker.png b/src/content/docs/project/analytics/images/9-color-picker.png new file mode 100644 index 00000000..f7873f91 Binary files /dev/null and b/src/content/docs/project/analytics/images/9-color-picker.png differ diff --git a/src/content/docs/project/analytics/images/Analytics_Failures.gif b/src/content/docs/project/analytics/images/Analytics_Failures.gif deleted file mode 100644 index 79f13099..00000000 Binary files a/src/content/docs/project/analytics/images/Analytics_Failures.gif and /dev/null differ diff --git a/src/content/docs/project/analytics/images/New_ACit3dNa_2024-08-04.gif b/src/content/docs/project/analytics/images/New_ACit3dNa_2024-08-04.gif deleted file mode 100644 index a87f84db..00000000 Binary files a/src/content/docs/project/analytics/images/New_ACit3dNa_2024-08-04.gif and /dev/null differ diff --git a/src/content/docs/project/analytics/images/New_E5IK2JjS_2024-09-24.png b/src/content/docs/project/analytics/images/New_E5IK2JjS_2024-09-24.png deleted file mode 100644 index 619c9c38..00000000 Binary files a/src/content/docs/project/analytics/images/New_E5IK2JjS_2024-09-24.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/New_Fjgf73qB_2024-07-27.png b/src/content/docs/project/analytics/images/New_Fjgf73qB_2024-07-27.png deleted file mode 100644 index fdb3a82d..00000000 Binary files a/src/content/docs/project/analytics/images/New_Fjgf73qB_2024-07-27.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/New_dMX5JYvp_2024-10-23.gif b/src/content/docs/project/analytics/images/New_dMX5JYvp_2024-10-23.gif deleted file mode 100644 index c0d585e3..00000000 Binary files a/src/content/docs/project/analytics/images/New_dMX5JYvp_2024-10-23.gif and /dev/null differ diff --git a/src/content/docs/project/analytics/images/New_m3fhfYsx_2024-10-01.png b/src/content/docs/project/analytics/images/New_m3fhfYsx_2024-10-01.png deleted file mode 100644 index 29e38e9a..00000000 Binary files a/src/content/docs/project/analytics/images/New_m3fhfYsx_2024-10-01.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/New_mjqIWEbd_2024-09-25.gif b/src/content/docs/project/analytics/images/New_mjqIWEbd_2024-09-25.gif deleted file mode 100644 index 20d6ba16..00000000 Binary files a/src/content/docs/project/analytics/images/New_mjqIWEbd_2024-09-25.gif and /dev/null differ diff --git a/src/content/docs/project/analytics/images/Timeline.png b/src/content/docs/project/analytics/images/Timeline.png deleted file mode 100644 index 14d77c07..00000000 Binary files a/src/content/docs/project/analytics/images/Timeline.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/analytics-overview.png b/src/content/docs/project/analytics/images/analytics-overview.png new file mode 100644 index 00000000..4f76dafa Binary files /dev/null and b/src/content/docs/project/analytics/images/analytics-overview.png differ diff --git a/src/content/docs/project/analytics/images/att2_357.png b/src/content/docs/project/analytics/images/att2_357.png deleted file mode 100644 index 80cde94e..00000000 Binary files a/src/content/docs/project/analytics/images/att2_357.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/att2_4449.png b/src/content/docs/project/analytics/images/att2_4449.png deleted file mode 100644 index 910365cc..00000000 Binary files a/src/content/docs/project/analytics/images/att2_4449.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/att3_357.png b/src/content/docs/project/analytics/images/att3_357.png deleted file mode 100644 index 20074618..00000000 Binary files a/src/content/docs/project/analytics/images/att3_357.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/att3_4449.png b/src/content/docs/project/analytics/images/att3_4449.png deleted file mode 100644 index 58a62fac..00000000 Binary files a/src/content/docs/project/analytics/images/att3_4449.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/att7_357.png b/src/content/docs/project/analytics/images/att7_357.png deleted file mode 100644 index e007f2f6..00000000 Binary files a/src/content/docs/project/analytics/images/att7_357.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/att9_357.png b/src/content/docs/project/analytics/images/att9_357.png deleted file mode 100644 index 82c28e7a..00000000 Binary files a/src/content/docs/project/analytics/images/att9_357.png and /dev/null differ diff --git a/src/content/docs/project/analytics/images/gif2_357.gif b/src/content/docs/project/analytics/images/gif2_357.gif deleted file mode 100644 index b39a01d1..00000000 Binary files a/src/content/docs/project/analytics/images/gif2_357.gif and /dev/null differ diff --git a/src/content/docs/project/analytics/images/set_up_board.gif b/src/content/docs/project/analytics/images/set_up_board.gif deleted file mode 100644 index 1254d844..00000000 Binary files a/src/content/docs/project/analytics/images/set_up_board.gif and /dev/null differ diff --git a/src/content/docs/project/analytics/index.md b/src/content/docs/project/analytics/index.md index 450ef322..6d771a46 100644 --- a/src/content/docs/project/analytics/index.md +++ b/src/content/docs/project/analytics/index.md @@ -12,39 +12,32 @@ head: - tag: meta attrs: name: keywords - content: Testomat.io analytics, test management, automation coverage, custom charts, flaky tests, slow tests, test failures, testing trends, manual tests, automated testing + content: Testomat.io analytics, test management, automation coverage, from manual to automated, custom charts, flaky tests, slow tests, test failures, testing trends, manual tests, automated testing --- -As a test management system, Testomat.io Team intends to provide our users with as much valuable testing data as possible. For this purpose, we developed Analytics Feature. +Testomat.io tracks your automated and manual tests, aggregates their statuses history, analyzes them, defines tests by Analytics categories, and shows them to you. You can configure these metrics. Analytics widgets are updated and supplemented with each completed Test Run. -![Analytics dashboard](./images/att10_357.png) +![What the Analytics dashboard covers: coverage, problem boards, and test health](./images/analytics-overview.png) -## How Does It Work +All available widgets are selected and displayed on the main analytics board by default. Customize your board by selecting/unselecting widgets to meet your specific needs. -Testomat.io tracks your automated and manual tests, aggregates their statuses history, analyzes them, defines tests by Analytics categories, and shows them to you. You can configure these metrics. So you can give a more precise glance to prevent bugs. This also means that Analytics widgets are updated and supplemented with each completed Test Run. +To change the board view: -All available widgets are selected and displayed on the main analytics board by default. However, you can customize your board by selecting/unselecting widgets to meet your specific needs. - -To change the board view, follow these steps: - -1. Go to Settings. +1. Go to **Settings**. 2. Select the widgets you need. -3. Click Save button. +3. Click **Save** button. -![Testomat.io - Analytics board set up](./images/set_up_board.gif) +![Testomat.io - Analytics board set up](./images/1-display-settings.png) To restore all settings to default: -1. Go to Settings. -2. Click Default. -3. Click Save button. - -![Testomat.io - Analytics board default](./images/default_board.gif) - -Let's have more close look at each widget! +1. Go to **Settings**. +2. Click **Default**. +3. Click **Save** button. ## AI Analytics Chat - Chat with Analytics is an AI-powered feature that helps to get specific insights from project analytics data. It works across analytics widgets and metrics, and you can quickly access relevant information without manually exploring dashboards. is specified. Example: ask which tests are most unstable, which areas have the highest failure rate, or how execution trends changed over time. + +Chat with Analytics is an AI-powered feature that helps to get specific insights from project analytics data. It works across analytics widgets and metrics, and you can quickly access relevant information without manually exploring dashboards. :::note @@ -52,191 +45,140 @@ By default, metrics are calculated over a 30-day window unless a different range ::: -## Analytics range +## Filter by Date range -Analytics data loads for the last 4 weeks by default. But you have the option to change the date range to suit your specific needs. +Analytics data loads for the last 4 weeks by default. You have options to change the date range to suit your specific needs: -1. You have the option to specify the range manually by entering a value in the field using a template: `2024-06-30 to 2024-07-27`. -2. Or select the desired date range from the drop-down calendar. The calendar is very flexible and will provide you with accurate analytics for the specified range. -3. You can also choose convenient templates to quickly view your analytics. +1. Specify the range manually by entering a value in the field using a template. +2. Select the desired date range from the drop-down calendar. +3. Choose convenient templates to quickly view your analytics. -![Testomat.io - Analytics range](./images/New_Fjgf73qB_2024-07-27.png) +![Testomat.io - Analytics range](./images/2-analytics-calendar.png) ## Automation Coverage -As a management system for automated tests, Testomat.io team created an Automation Coverage Board where you can track the progress of automation coverage on the project. You can sort your tests by Suite and Automation indicators. - -![Automation Coverage](./images/gif2_357.gif) - -## Custom Charts - -Custom charts are a powerful addition to analytics. They allow you to customize the display of the data most relevant to you — not only for **Tests**, but also for **Test Runs**. - -You can build custom charts using search queries to get a comprehensive view of your testing process, providing visibility into trends, completion metrics, and overall testing performance. - -Custom charts help teams to visualize key testing metrics at a glance: - -- **Environment-Based Metrics**: Track the number of test runs executed on specific platforms or environments, helping you monitor execution across systems. -- **Label-Based Metrics**: Track tests or test runs associated with specific labels, such as a build version or milestone, to gain insights into particular testing contexts. -- **Run Duration Analysis**: Visualize average or total duration of Test Runs over time. -- **Trends Over Time**: With timeline settings, track metrics across a selected period. -- **Widgets**: Any chart can be added as a widget to your dashboard, giving a separate view alongside other analytics. - -### How to Create a Custom Chart +In an **Automation Coverage** board where you can track the progress of automation coverage on the project. You can sort your tests by Suite and Automation indicators. -The flow for creating a custom chart is the same for both **Tests and Test Runs**. The only difference is which **Data Source** you select. - -1. Navigate to the **Analytics** tab in the left sidebar -2. Click **Custom Charts** on the dashboard +![Automation Coverage](./images/3-automation-coverage.png) -![Custom Charts](./images/att1_357.png) +## From Manual to Automated -3. Click the **+** button to open a new chart +The **From Manual to Automated** widget tracks how your test suite progresses from manual to automated. It shows how many tests changed state during the selected date range and updates as you change the supported filters, so automation progress is easy to measure and review. -![Plus button](./images/att2_357.png) - -4. Enter **Title** (required) -5. Toggle **As widget** (optional) +![The From Manual to Automated widget on the Analytics board](./images/4-manual-to-automated-widget.png) :::note -By toggling **As widget**, your custom chart will appear as a separate widget on the Analytics dashboard. This allows you to monitor key metrics continuously alongside other analytics without navigating back to the Custom Charts page. +This widget reports state changes that have already happened. It does not create automated tests and does not change their state. ::: -6. Enter **Description** (optional) -7. Select **Data Source**: (required) - -- **Tests:** for Test data -- **Runs:** for Test Run data - -![Data Source](./images/att3_357.png) - -After selecting the Data Source, additional configuration options appear: - -8. **Chart Type** (required) – choose how data will be visualized (bar, donut, pie, etc.) -9. **Labels** (required) – customize the display of labels (short query, titles, numbers, title and %, etc.) +## Custom Charts -**Timeline settings:** +Custom charts allow to customize the display of the data most relevant to you for **Tests** and **Test Runs**. You can build custom charts using search queries to get a comprehensive view of your testing process, providing visibility into trends, completion metrics, and overall testing performance. -10. **Period** (optional) – enable Timeline to track data changes over a selected period -11. **Extra Line** (optional) – add an additional line to compare metrics within the Timeline -12. **Chart type** (optional) – select the visualization style specifically for the timeline chart -13. Click the **Add Query** button to open the Query Editor: +| Metric | What it gives you | +| --- | --- | +| **Environment-Based Metrics** | The number of test runs executed on specific platforms or environments to monitor execution across systems. | +| **Label-Based Metrics** | Tests or test runs tied to a specific label, such as a build version or milestone, for insight into a particular testing context. | +| **Run Duration Analysis** | Average or total duration of test runs over time. | +| **Trends Over Time** | Metrics across a period you select in the timeline settings. | +| **Widgets** | Any chart added to your dashboard as a widget, shown alongside other analytics. | -Configure queries according to your metrics using supported query variables: +## Create a Custom Chart -- [Tests Variables](https://docs.testomat.io/advanced/tql/#tests-variables) -- [Runs Variables](https://docs.testomat.io/advanced/tql/#runs-variables) +The flow for creating a custom chart is the same for both **Tests and Test Runs**. The only difference is which **Data Source** you select. -![Add Query](./images/att4_357.png) +![Plus button](./images/5-add-custom-chart.png) -14. Click the **Save** button — it will appear on your dashboard or as a widget if selected. +1. Navigate to the **Analytics** tab in the left sidebar. +2. Click **Custom Charts** on the dashboard. +3. Click the **+** button to open a new chart. +4. Enter **Title**. +5. Toggle **As widget** (optional). +6. Enter **Description** (optional). +7. Select **Data Source**: **Tests** for Test data or **Runs** for Test Run data. +8. Click the **Add Query** button to open the Query Editor: +9. Click the **Save** button - it will appear on your dashboard or as a widget if selected. This extended chart functionality enhances your ability to make data-driven decisions by offering insight into both tests and test runs. -![Created Custom Chart](./images/att5_357.png) - -### How To Customize The Chart View - -Custom charts can be tailored to match your reporting needs. In this section, you can adjust how chart information is displayed — by modifying labels, colors, and other visual settings. These options help make your data easier to read and interpret. - -- **Labels** +![Data Source](./images/6-custom-chart-config.png) -During chart creation or in the **Edit** mode, select the **Labels** dropdown to customize the information on the chart to your preference: +:::note -For example, if a query `state == 'manual' and status == 'passed'` has been selected: +By toggling **As widget**, your custom chart will appear as a separate widget on the Analytics dashboard. This allows you to monitor key metrics continuously alongside other analytics without navigating back to the Custom Charts page. -| **Label** | **View** | -| -------------------------- | -------------------------------------------------------- | -| Query | `state == 'manual' and status == 'failed': 9 tests` | -| Short query | `state == 'manual' and status ...: 9 tests` | -| Titles | `state == 'manual' and status == 'failed'` | -| Numbers | `9` | -| Title and items amount | `state == 'manual' and status == 'failed': 9 tests` | -| Title and % | `state == 'manual' and status == 'failed': 1% ` | -| Title, items amounts and % | `state == 'manual' and status == 'failed': 9 tests, 1% ` | +::: -![Labels view](./images/att7_357.png) +### Chart settings -- **Color Customization** +| Setting | Required | What it does | +| --- | --- | --- | +| **Chart type** | Yes | How the data is drawn - bar, donut, pie | +| **Labels** | Yes | What the labels show - short query, title, number, title and % | +| **As widget** | No | Puts the chart on the Analytics dashboard, next to the other widgets | +| **Timeline period** | No | Tracks how the data changes over a period you pick | +| **Timeline extra line** | No | Adds a second line so you can compare two metrics | +| **Timeline chart type** | No | How the timeline itself is drawn | -In addition to customizing labels, you can also personalize the colors of your queries for better data visualization. To do this, follow these steps: +Configure queries according to your metrics using supported query variables [Tests Variables](https://docs.testomat.io/advanced/tql/#tests-variables) and [Runs Variables](https://docs.testomat.io/advanced/tql/#runs-variables). -1. Open a Custom Chart you want to customize -2. Scroll down to the list of queries displayed under the chart +![Created Custom Chart](./images/7-custom-chart-output.png) -![Queries list](./images/att1_4449.png) +## Customize The Chart View -3. Click the color box next to a query to change its color -4. After a color picker appears, select a new color +Custom charts can be tailored to match your reporting needs. You can adjust how chart information is displayed - by modifying labels, colors, and other visual settings. You can export the chart in multiple formats such as PNG, SVG, or CSV, facilitating further analysis or sharing with team members. -- Use the color palette to choose your preferred shade -- Alternatively, enter RGB, HSL, or HEX values manually for precise color selection +![Custom chart export options](./images/7-1-chart-export-options.png) -![Set up color](./images/att2_4449.png) +### Labels for The Chart -Once selected, the chart will automatically reflect the applied color. +During chart creation or in the **Edit** mode, select the **Labels** dropdown to customize the information on the chart to your preference. -5. Click the **Save** button to apply changes +![Labels view](./images/8-labels-dropdown.png) -![Save button](./images/att3_4449.png) +### Color Customization -This feature gives you greater control over the visual presentation of your custom charts by allowing you to select colors for each query. +Personalize the colors of your queries for better data visualization. Use the color palette to choose your preferred shade. Alternatively, enter RGB, HSL, or HEX values manually for precise color selection. -- **Additional Features:** +1. Open a Custom Chart you want to customize. +2. Scroll down to the list of queries displayed under the chart. +3. Click the color box next to a query to change its color. +4. After a color picker appears, select a new color. +5. Click the **Save** button to apply changes. - - **Duplicate Chart**: Quickly create a copy of an existing chart and modify it without starting from scratch +![Set up color](./images/9-color-picker.png) -![Duplicate Chart](./images/att9_357.png) +### Duplicate Chart -- **Download Chart**: Download your chart as **SVG, PNG, or CSV** for reporting or sharing +Quickly create a copy of an existing chart and modify it without starting from scratch -![Export Chart](./images/att8_357.png) +![Duplicate Chart](./images/10-duplicate-chart.png) -- **Sort Queries by Tests/Default**: On Edit Mode, set up the order of queries in the chart +## Sort Queries by Tests/Default -These options give you full control over both the appearance and functionality of your custom charts, making it easier to create professional and insightful visualizations. +On Edit Mode, set up the order of queries in the chart. These options give you full control over both the appearance and functionality of your custom charts, making it easier to create professional and insightful visualizations. ![Sort Queries by Tests/Default](./images/gif1_357.gif) -### Timeline - -**Timeline** for Custom Charts, a feature that allows you to visualize data trends over a specified period. A timeline is a graphical representation of events or data points in chronological order. It helps in understanding trends, patterns, and changes over time by displaying information in a linear format. - -- Search Query-Based Timeline: When creating custom charts, you can now set up a timeline period based on your test search queries. -- Data Collection: Testomat.io will collect data for the specified test search queries over the selected period. -- Timeline Chart View: The collected data is displayed in a timeline chart view, providing a clear visual representation of how your test metrics evolve over time. - -This feature enhances your ability to track progress, identify trends, and make informed decisions based on historical test data. - -![Testomat.io - Timelines for Custom Charts](./images/New_ACit3dNa_2024-08-04.gif) +## Timeline -Each timeline is associated with a unique URL, which can be copied and shared with other members of the project team. +**Timeline** is a graphical representation of events or data points in chronological order. It helps in understanding trends, patterns, and changes over time by displaying information in a linear format. Each timeline is associated with a unique URL and can be copied and shared with the project team. -![Testomat.io - Unique URL for timelines](./images/New_E5IK2JjS_2024-09-24.png) - -:::note - -Timeline feature is also available for other widgets, such as: - -- Automation Coverage -- Failures -- Issues -- Defects -- Ever Failing Tests -- Flaky Tests -- Jira - -::: +| Step | Events | +| --- | --- | +| **Search Query-Based Timeline** | Set the timeline period based on the test search queries. | +| **Data Collection** | Data collection for the test search queries over the period selected. | +| **Timeline Chart View** | Test metrics changed over time. | -![Testomat.io - timeline for widgets](./images/Timeline.png) +![Testomat.io - timeline for widgets](./images/11-chart-timeline.png) ## Failures Board -To make it easier for your team to track what's going on in the project, we’ve added a Failures Board. It shows failures from the latest test runs and lets you quickly jump to the suite, the test run, or the failed test itself. You can now also group and sort failures in the Failures widget to help make sense of the data faster. On top of that, there’s a new Defect column, so you can see any linked issues right away through IMS links like GitHub, Azure DevOps, or Jira. +The Board shows failures from the latest test runs. You can group and sort data in the Failures widget. Defect column allows to see any linked issues right away through IMS links like GitHub, Azure DevOps, or Jira. -![Testomat.io - Failures Board](./images/Analytics_Failures.gif) +![Testomat.io - Failures Board](./images/12-defects-failures.png) ## Issues Board @@ -244,17 +186,15 @@ The Issue Board provides a comprehensive view of all tests, suites and runs asso Visit the [Issues Management Systems](https://docs.testomat.io/integration/issues-management-systems/#_top) page to find out which systems Testomat.io supports and how to connect. -![Testomat.io - Issues Board](./images/New_dMX5JYvp_2024-10-23.gif) +![Testomat.io - Issues Board](./images/13-linked-items.png) ## Defects Board -The Defects report answers a different question from the Defects board: not which bugs are open, but which of your tests are blocked, and by what. Each row is a test case, with its defects nested underneath. Expand a test to see every defect holding it back, each with its severity, status, and a link to the tracker. - -Choose how defects reach a test with the data-source selector: +The Defects report answers a question: which of your tests are blocked, and by what. Each row is a test case, with its defects nested underneath. Expand a test to see every defect holding it back, each with its severity, status, and a link to the tracker. Choose how defects reach a test with the data-source selector: | Data source | What it shows | |-------------|---------------| -| **All Defects** | Both routes merged; a defect is counted once even when it is linked twice. | +| **All Defects** | Both routes merged - a defect is counted once even when it is linked twice. | | **By Tests** | Only defects attached to the test case itself. | | **By Test Runs** | Only defects raised on that test's runs. | @@ -264,25 +204,16 @@ This separates bugs someone filed against a test from bugs the test actually cau ## Flaky Tests -There may be some unstable automated tests on your project that sometimes fail and sometimes pass. - -**How does it work?** - -Flakiness is determined by calculating the average value of run statuses for a given test. The method of calculation can be defined based on specific parameters, including a minimum and maximum success rate threshold. - -- Minimum Success Rate: Defines the lowest acceptable pass rate to be considered within the flakiness range. -- Maximum Success Rate: Defines the highest acceptable pass rate to be considered within the flakiness range. - -Analytics will identify and display tests that have a pass rate falling within the defined range. **The pass rate is calculated based on the last 100 runs.** - -Example: +Flakiness is determined by calculating the average value of run statuses for a given test. The method of calculation can be defined based on specific parameters, including a minimum and maximum success rate threshold. Analytics will identify and display tests that have a pass rate falling within the defined range. **The pass rate is calculated based on the last 100 runs.** -- Minimum Success Rate: 40% -- Maximum Success Rate: 60% +| Setting | Description | Example | +| --- | --- | --- | +| **Minimum Success Rate** | The lowest pass rate that still counts as flaky | 40% | +| **Maximum Success Rate** | The highest pass rate that still counts as flaky | 60% | If a test has been run 14 times and succeeded 7 times, the success rate is calculated as 50%. Since 50% falls within the defined range (40% to 60%), this test would be considered flaky and displayed in the analytics report. -![Testomat.io - Flaky](./images/New_mjqIWEbd_2024-09-25.gif) +![Testomat.io - Flaky](./images/14-flaky-tests.png) ## Slowest Tests @@ -304,30 +235,20 @@ Ever Failing Tests is another useful Analytics widget that will show you automat ## Labels Statistics -Labels Statistics is a feature that allows users to visualize test coverage by labels on an interactive chart. It provides an efficient way to analyze and track the status of testing with the following capabilities: - -- **Label coverage visualization:** you can view information about which tests need to be reviewed, which can be automated, and what impact they have on the system. You can set up your own labels or use the ready-made ones offered by Testomat.io. To learn how to add labels to the project, visit a [dedicated page](https://docs.testomat.io/usage/labels-and-custom-fields/#_top). - -- **Test filtering (1) and search (2):** for convenience, there is filtering by Environments, Tags, Labels, Jira issues, Date Range, Priority, as well as search options to quickly find tests by name or other criteria, making it easier to locate specific tests. - -- **Chart download (3):** the feature allows to export the chart in multiple formats such as PNG, SVG, or CSV, facilitating further analysis or sharing with team members. - -- **Automation coverage analysis:** you can evaluate the level of test automation, helping to identify which parts of the testing process are already automated and where additional efforts are needed. - -This tool enables teams to manage the testing process more efficiently, quickly identify issues, and optimize automation efforts. - -![Testomat.io - Labels Statistics](./images/New_m3fhfYsx_2024-10-01.png) +Labels Statistics is a feature that allows users to visualize test coverage by labels on an interactive chart. -Another big plus is the detailed statistics for a particular label. Click on the label to open another window with options. +View information about which tests need to be reviewed, which can be automated, and what impact they have on the system. You can set up [your own labels](https://docs.testomat.io/usage/labels-and-custom-fields/#_top) or use the ready-made ones offered by Testomat.io. -You will see all tests with this label. You can also filter **(2)** by Tags, Priority and Jira Issues or search **(3)** for specific tests. +You can filter by Environments, Tags, Labels, Jira issues, Date Range, Priority, as well as search options to quickly find tests by name or other criteria, making it easier to locate specific tests. -Additionally, there is a special option **(1)** to display tests: +![Testomat.io - Labels Statistics](./images/15-label-statistics-filters.png) -- **By Tests** - shows all tests created in the project with this label. -- **By Runs** - only shows tests that have a run result. +You can evaluate the level of test automation, helping to identify which parts of the testing process are already automated and where additional efforts are needed. Click on the label to open another window with the detailed statistics for a particular label. There is a special option to display tests: -![Testomat.io - Detailed label information](./images/New_BXpy3mr7_2024-10-14.png) +| View mode | Description | +| --- | --- | +| **By Tests** | All tests in the project that carry this label | +| **By Runs** | Only tests that have a run result | ## Analytics In Run Reports @@ -339,4 +260,4 @@ Overview chart visualizes aggregated tests statuses by suites: Flaky and Slowests tests widgets show the latest 5 tests and navigate to dedicated Analytics pages: -![Flaky and Slowests widgets](./images/att11_357.png) +![Flaky and Slowests widgets](./images/att11_357.png) \ No newline at end of file