Add CoverageReporter for batching code coverage reports

Description

Problem: Large Codebases Cannot Generate Full Coverage Report

Due to memory limitations in CI environments, larger codebases are unable to run all tests as a single testbox run command. Instead, specs are run in a methodical folder-by-folder sequence, separating the testbox run out over many requests and thus working around the Out-Of-Memory exceptions.

While this works, it prevents accurate reporting of code coverage since only a small portion of the tests are executed during any request. The generated code coverage report only shows a tiny fraction of the coverage - say, 2% - and not the whole picture

Solution: Aggregate + Combine Code Coverage

This PR introduces a CoverageReporter component which

  1. runs on every testbox code coverage execution

  2. loads any previous coverage data from a JSON file

  3. combines the previous coverage data with the current execution's coverage data (file by file and line by line)

  4. persists the COMBINED coverage data to a JSON file.

  5. returns the COMBINED coverage data for the CoverageBrowser.cfc to build as an HTML report

When setting url.isBatched=true and executing the batched test runner, the code coverage report will then grow with each sequential testbox run command.

 

Activity

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

Sentry

Created May 10, 2023 at 2:20 PM
Updated May 10, 2023 at 2:24 PM
Resolved May 10, 2023 at 2:24 PM