When running via a browser using DirectoryTestSuite.run(), if something goes wrong in beforeTest(), then an exception is thrown, and testing HALTS. Not just for that test CFC, but completely. This is a right PitA.
MXUnit acknowledge this as a bug:
But it's never been fixed. It's be good if you guys fixed it.
Expected results of fix: tests are still attempted, and if anything a test relies on that didn't get created by the failed beforeTests(), then the test simply bubbles back the error.
Also the current error is not bubbled back correctly. Errors in beforeTests() are reported as occurring within TestBox, eg:
I am not sure if I agree with you. If there is something wrong with the setup of the tests, then it should be reported immediately. Plus, if you have compilation errors, nothing would work anyways. I would disagree that this is a bug but more of a design of the test, where if the setup fails, then it should tell the user immediately that the setup fails.
Sure. The minutiae of how you implement it is neither here nor there, all I am saying is the behaviour of beforeTests() and setup() should be analogous. Currently setup() handles problems gracefully, whereas beforeTests() just crashes. All I'm suggesting is for beforeTests() to deal with errors gracefully.
A case in point: https://gist.github.com/daccfml/8111064
If all that lot is put in the same dir, and run, we get this:
So a break in beforeTest() kills the entire test run.
On the other hand, if we remove the error in TestBeforeTests.cfc so we can even get to TestSetup.cfc, we get this:
IE: setup() deals with its problems; beforeTests() does not.
It's fine for a problem in beforeTests() in one CFC to prevent its own tests from running, but it should not kill the entire test run.
The fix for this is perfect, btw. Cheers.