UI automation of vendor delivered products always leads to trouble

wise_old_elfI’ve got three darling boys, and they love this show on ABC4Kids called ‘Ben & Holly’s Little Kingdom’. It’s a cartoon from the makers of Peppa Pig about tiny elves and fairies and there’s a character named the wise old elf (pictured) who doesn’t like the fairy magic and whose catchphrase is ‘magic always leads to trouble‘ which has become a little bit of a meme in our household where we replace the word ‘magic’ with something else that’s perilous. This leads me to the point of this article, something I have a strong opinion about:

“UI automation of vendor delivered products always leads to trouble”

Why do I believe that? To be successful in UI automation involves some critical elements which are missing when writing automated tests against a black box vendor delivered products (such as a customized CRM solution). To be successful in UI automation:

  • you need collaboration between testers and developers to write the code needed to write robust and efficient user interface tests;
  • you need opportunities to include testability into the user interface, whether this be test specific navigation controllers, or ensuring that page elements are appropriately identified and structured; and
  • you need to be able to identify areas which can be tested below the UI, whether this be through APIs and web services, or hitting the database directly. Vendors seldom provide services and almost never allow direct access to the database – particularly if it’s a SaaS product.

I strive to advise anyone that it’s a bad idea to write automated tests against the UI of a vendor delivered product. Either that vendor should be doing their own automated testing, or be providing a more robust way to automatically test that changes have been correctly applied to their product.

I also try to avoid any career opportunities that put me in a situation where this is required of me because I don’t believe in it and haven’t seen it been successfully done.

As the wise old elf says: “UI automation of vendor delivered products always leads to trouble”.

3 thoughts on “UI automation of vendor delivered products always leads to trouble

  1. I’ll have to agree strongly with this one. I’ve been on both sides of a ‘vendor’ implementation (once working for the vendor in-house, the other at the client site) trying to do automation. If you are in-house you have a fighting chance to get some things built in to make automation at the GUI level easier, but even still if they use 3rd party objects that are either closed (no native access) or tweaked allover then you’re in a world of hurt. Also, you have to make sure they build in the testability, either API or naming objects properly, to the GUI. Tables/Grid objects are the worst and even more so with embedded objects of their own.

    Now if you are working with a client to try to automate you are at the mercy of the vendor to have implemented much of what I mentioned before. You better hope you can build a working relationship with them and have them ‘bake’ it in. Otherwise you are just screwed. I dealt with this one where certain objects on the GUI were a 3rd party object (grid) that in certain versions of the application you could get to, but in others they used a ‘closed’ compiled library. Meaning some of the ways you can get at native Methods/Properties (.object method in QTP/UFT) that allow access to the objects own methods was closed down. So in one version or application (this was a suite of apps) you had access and others you didn’t.

    It was a nightmare. I was very glad the day I walked away from that project.

  2. Great post.

    Now if only we can boycott/strike and demand our vendor products that allow for integration/incorporation in other products provide testability mechanism in them to start with or else we don’t adopt them, that would be great. Sadly we have no such power or are not willing to do so for such an ideal scenario :(

    But this does bring up some thoughts, how much testability should we expect from a vendor’s plugin product/service under ideal scenario?

    * that their product/service is fully tested and we shouldn’t have to test integration with their product at all, it should “just work”?

    * they should provide below the GUI automation API hooks to integrate with other’s automation solutions?

    * they should provide something like a Selenium page object implementation (or at least a template to model off of for one’s desired language binding) for others to integrate into their UI automation solutions? So as to avoid reinventing the wheel on 3rd party vendor integration UI automation.

    Then you also may consider other aspects of testability beyond UI automation, like performance (testing), security, usability, etc. Like should the 3rd party component be expected to perform well out of the box, or what can vendor provide to facilitate performance testing integration of their product in a customers solution?

  3. Also, another related thought:

    Sometimes a vendor may provide some testability interface, but for the most part, in terms of QA and (Selenium/UI) automation, the QA team doesn’t know about, or have the skills or time to use the testability interface and go with the old fashion route of automating it the traditional way (via the UI with Selenium, etc. in their own code implementation).

    Take Facebook Connect/Login integration with one’s application/website. I see occasional posts on Selenium groups about automating the Facebook login integration. I guess its still valid to test the actual UI of the login widget integration. But assuming that should be robust enough provided by Facebook, I think you can probably test some of the integration and login by use of Facebook’s test APIs:

    https://developers.facebook.com/docs/test_users/

    and after successful login via API, can then resume testing the integration functionality from the UI level (having skipped the login part via the UI, which may offer better speed and reliability in test runs).

Comments are closed.