Why I don’t like Jasmine JavaScript unit tests (for MVC web apps)

Update 11-Feb-2013: After some feedback, I would like to clarify what I am talking about here: I am talking from experience in applications not purely written in JavaScript: for example a MVC application written with say C# with JavaScript added to pages.

I’m really not a big fan of Jasmine JavaScript unit tests for applications not written purely in JavaScript. Here’s why:

jasmine logo

Most modern web applications use JavaScript as a way to provide client side validation and interaction. Most of this JavaScript manipulates the Document Object Model (DOM) of the browser, therefore the JavaScript is coupled to the DOM using ids or classes of elements.

As most web pages are dynamically rendered by the server, Jasmine, the JavaScript unit test framework, can’t run against the ‘real’ DOM as it appears in the application, so what it does is duplicates parts of the DOM to test the JavaScript is working properly based upon this duplication.

As there’s no reliance on the real application or its DOM, these tests run super fast, you can run thousands of JavaScript tests in seconds which is pretty amazing.

The problem becomes that over time the real DOM in the application is changed but the Jasmine tests and its own duplicated DOM is not! This means that often I find that all Jasmine tests pass (as they have to for a new version to be deployed to QA) but the JavaScript doesn’t work in the application because an id or class name has been changed and someone ‘forgot’ to update the Jasmine tests.

Tests that don’t pick up valid errors are worthless, and I continually find myself getting annoyed that all the Jasmine tests have passed, but JavaScript is broken and our application doesn’t work.

Another downside is that Jasmine tests are only as good as the JavaScript engine they are run against. We run our Jasmine tests in a headless PhantomJS browser, which is fast, but most of our JavaScript bugs are found in Internet Explorer, particularly earlier versions of it, so Jasmine tests again are worthless.

I don’t know of a way to overcome these downsides to Jasmine that I have mentioned. What we do have is a suite of end to end acceptance tests that run in 5 different ‘real’ browsers that check the JavaScript works correctly. Whilst these take 8 minutes to run per browser, they pick up a lot more JavaScript issues than Jasmine.

I am not still not sure whether it is worth investing the time and effort in Jasmine tests when they don’t provide sufficient confidence that our application ‘works’.

What have others experiences been with Jasmine JavaScript unit testing?

14 thoughts on “Why I don’t like Jasmine JavaScript unit tests (for MVC web apps)

    1. A bit of both.
      I think the issue is when application code is refactored using TDD and the Jasmine tests are not.

    1. Agree in this regard.
      As for mixing in JavaScript into an existing MVC application: this is where DOM duplication is a problem.

  1. I have a feeling that you are mixing different testing types. Unit testing and Acceptance testing. Unit testing doesn’t have to prove that in every possible environment your code will work. Acceptance tests does. Jasmine works great for Unit/Integration testing but not so well for Acceptance testing.
    As for ids dependencies, I have a strong feeling that it is actually more design flow then Jasmine problem. It seems that you have the same variable defined in two different places (javascript and html) no wonder they are becoming inconsistent over time.

    I’ve written short blog post describing my point of view on this subject:
    http://agilefront.blogspot.co.uk/2013/02/one-gun-many-enemies.html

  2. The reason for doing TDD is not to end up with a bunch of tests. Unit testing is just one aspect of QA – it doesn’t guarantee defect-free code.

    TDD is primarily about esign, with the aim being to build classes with clean interfaces and minimal dependencies, making the code easier to maintain, extend and debug.

    If you’re rely on integration-level tests you’ll never be able to get into a red-green-refactor cycle with short enough feedback.

    Unit tests or integration tests alone are never enough – it’s all about getting the balance right.

  3. “JavaScript doesn’t work in the application because an id or class name has been changed and someone ‘forgot’ to update the Jasmine tests.”

    Is this Jasmine fault that someone didn’t update the test? Come on, what kind of argument is that?

    1. The OP clearly stated that he doesn’t like Jasmine *for MVC apps”, not that he doesn’t like it in general.

      I agree with this post. Jasmine tests that test DOM interaction can get much more out of sync than regular unit tests. This is because you are required stub the DOM, while in regular unit tests you don’t have to stub out other methods.

      But, we also have to agree, this is the only way to unit-test interaction with DOM. What you can do is to write a script which would start your application, browse through it, and download necessary HTML for the fixtures in your Jasmine tests. The idea is that every fixture would know how to renew itself. That way every time before pushing the code somewhere you can run that script before running Jasmine tests.

  4. For me, it seems that jasmine is better used for testing discrete chunks of javascript (does this function return what it should), rather than complication DOM interactions. Selenium is better for testing browser based issues, while jasmine is better for javascript library testing. The two used together is best.

  5. “I am talking from experience in applications not purely written in JavaScript: for example a MVC application written with say C# with JavaScript added to pages.”

    Well, there’s your problem.. :P. But seriously, although MVC is quite awesome (especially when for example using ASP.NET MVC instead of ASP.NET Webforms), you still have the problem that HTML is generated using backend code, and then manipulated using Javascript code. On the one hand these are very tightly coupled (changing the View can break the Javascript), but on the other hand the link between the two is usually made using a very unclear way (matching on ids / classes). How are you supposed to know whether some Javascript relies on this id or that class?
    Sure you can have some conventions for this, but it still will remain far from ideal.

    If you can you should try using Angular.js, You would then build your application in Angular.js, and have only a webservice (for example ASP.NET WebAPI (possibly with OData)) as the backend.

    “Another downside is that Jasmine tests are only as good as the JavaScript engine they are run against. We run our Jasmine tests in a headless PhantomJS browser, which is fast, but most of our JavaScript bugs are found in Internet Explorer, particularly earlier versions of it, so Jasmine tests again are worthless.”

    One of the creators of Angular.js created a solution for this: Karma: http://karma-runner.github.io/. Karma is a Javascript testrunner. In short: you can run your Jasmine unit tests (and lots of other kind of tests) with it and specify in which browser the tests should be executed.

  6. “I am talking from experience in applications not purely written in JavaScript: for example a MVC application written with say C# with JavaScript added to pages.”

    Well, there’s your problem.. :P. But seriously, although MVC is quite awesome (especially when for example using ASP.NET MVC instead of ASP.NET Webforms), you still have the problem that HTML is generated using backend code, and then manipulated using Javascript code. On the one hand these are very tightly coupled (changing the View can break the Javascript), but on the other hand the link between the two is usually made using a very unclear way (matching on ids / classes). How are you supposed to know whether some Javascript relies on this id or that class?
    Sure you can have some conventions for this, but it still will remain far from ideal.

    If you can you should try using Angular.js, You would then build your application in Angular.js, and have only a webservice (for example ASP.NET WebAPI (possibly with OData)) as the backend.

    “Another downside is that Jasmine tests are only as good as the JavaScript engine they are run against. We run our Jasmine tests in a headless PhantomJS browser, which is fast, but most of our JavaScript bugs are found in Internet Explorer, particularly earlier versions of it, so Jasmine tests again are worthless.”

    One of the creators of Angular.js created a solution for this: Karma: http://karma-runner.github.io/. Karma is a Javascript testrunner. In short: you can run your Jasmine unit tests (and lots of other kind of tests) with it and specify in which browser the tests should be executed.

  7. I have to disagree on this.
    How we get around this issue is using partial views. In your Jasmine test, load the partial view, and perform your tests on it. Make sure not to use any server code there.

    Yes, you have to create your app in such a away, that the partial view is HTML only.

    Roch

  8. I have evaluated Jasmine recently, and I partially agree with your opinion: Jasmine is not meant to be used for testing code that interacts with the DOM. In this case, it is better to use a tool like Selenium. However Jasmine proves to be useful and extremly valuable for the javascript code that contains some business logic. Thus I believe application developers who write js code should cleanly distinguish these two types of code.

Comments are closed.