Add CoverageReporter for batching code coverage reports
Description
Problem: Large Codebases Cannot Generate Full Coverage Report
Due to memory limitations in CI environments, larger codebases are unable to run all tests as a single testbox run command. Instead, specs are run in a methodical folder-by-folder sequence, separating the testbox run out over many requests and thus working around the Out-Of-Memory exceptions.
While this works, it prevents accurate reporting of code coverage since only a small portion of the tests are executed during any request. The generated code coverage report only shows a tiny fraction of the coverage - say, 2% - and not the whole picture
Solution: Aggregate + Combine Code Coverage
This PR introduces a CoverageReporter component which
runs on every testbox code coverage execution
loads any previous coverage data from a JSON file
combines the previous coverage data with the current execution's coverage data (file by file and line by line)
persists the COMBINED coverage data to a JSON file.
returns the COMBINED coverage data for the CoverageBrowser.cfc to build as an HTML report
When setting url.isBatched=true and executing the batched test runner, the code coverage report will then grow with each sequential testbox run command.
Activity
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Problem: Large Codebases Cannot Generate Full Coverage Report
Due to memory limitations in CI environments, larger codebases are unable to run all tests as a single
testbox run
command. Instead, specs are run in a methodical folder-by-folder sequence, separating the testbox run out over many requests and thus working around the Out-Of-Memory exceptions.While this works, it prevents accurate reporting of code coverage since only a small portion of the tests are executed during any request. The generated code coverage report only shows a tiny fraction of the coverage - say, 2% - and not the whole picture
Solution: Aggregate + Combine Code Coverage
This PR introduces a
CoverageReporter
component whichruns on every testbox code coverage execution
loads any previous coverage data from a JSON file
combines the previous coverage data with the current execution's coverage data (file by file and line by line)
persists the COMBINED coverage data to a JSON file.
returns the COMBINED coverage data for the
CoverageBrowser.cfc
to build as an HTML reportWhen setting
url.isBatched=true
and executing the batched test runner, the code coverage report will then grow with each sequentialtestbox run
command.