If beforeTests() errors, testing halts

Description

When running via a browser using DirectoryTestSuite.run(), if something goes wrong in beforeTest(), then an exception is thrown, and testing HALTS. Not just for that test CFC, but completely. This is a right PitA.

MXUnit acknowledge this as a bug:
https://groups.google.com/d/msg/mxunit/tCgs-MXhs64/hEdxfvGQ944J

But it's never been fixed. It's be good if you guys fixed it.

Expected results of fix: tests are still attempted, and if anything a test relies on that didn't get created by the failed beforeTests(), then the test simply bubbles back the error.

Activity

Show:
Adam Cameron
December 23, 2013, 9:48 AM

Also the current error is not bubbled back correctly. Errors in beforeTests() are reported as occurring within TestBox, eg:

Luis Majano
December 23, 2013, 6:08 PM

I am not sure if I agree with you. If there is something wrong with the setup of the tests, then it should be reported immediately. Plus, if you have compilation errors, nothing would work anyways. I would disagree that this is a bug but more of a design of the test, where if the setup fails, then it should tell the user immediately that the setup fails.

Adam Cameron
December 24, 2013, 9:50 AM

Sure. The minutiae of how you implement it is neither here nor there, all I am saying is the behaviour of beforeTests() and setup() should be analogous. Currently setup() handles problems gracefully, whereas beforeTests() just crashes. All I'm suggesting is for beforeTests() to deal with errors gracefully.

Adam Cameron
December 24, 2013, 9:58 AM
Edited

A case in point: https://gist.github.com/daccfml/8111064

If all that lot is put in the same dir, and run, we get this:

So a break in beforeTest() kills the entire test run.

On the other hand, if we remove the error in TestBeforeTests.cfc so we can even get to TestSetup.cfc, we get this:

IE: setup() deals with its problems; beforeTests() does not.

It's fine for a problem in beforeTests() in one CFC to prevent its own tests from running, but it should not kill the entire test run.

Adam Cameron
January 21, 2014, 8:13 AM

The fix for this is perfect, btw. Cheers.

Assignee

Luis Majano

Reporter

Adam Cameron

Labels

None

Components

Affects versions

Priority

Major
Configure