The sacred cow that is an open source project

Before I start, I want to make it clear that I am huge believer in and supporter of open source software, and I appreciate all the time and effort that people put in to make it so. I want to make open source software better and it is the only reason I write this post.

So…

It all started a couple of months ago when I made what I considered an off hand remark about how I felt the Internet Explorer Webdriver driver was unreliable. Unfortunately this wasn’t taken very well at all by the core IE WebDriver driver developer who found my view unexpected and unfounded. I was surprised but took actions to rectify the issue I had created, such as emailing/tweeting the developer directly with apologies and apologizing to the mailing list where I made the remark. There’s only one Watir developer left working on the Watir IE driver, and I even asked this individual at the time if they would work on improving the Webdriver IE driver but this person didn’t feel confident considering it’s written in C++, and they’re a Ruby developer.

I thought it was all okay, as I hadn’t heard of it again. I didn’t raise any criticisms again (after how they were taken), but then yesterday out of the blue there was this:

And then this one surprised me even more, considering what I did originally:

Update: 19 June 2012: Apparently these comments had nothing to do with me.

Oh boy, Groundhog Day, and I hadn’t even dared to speak of the unreliability again. But that wasn’t the end of it. The Selenium Team caught on and starting retweeting the original tweet, and then adding to the message accusations of trolling, as if anyone who raises any issue, without a reproducible test case, even if they don’t know the underlying technology/language, is apparently a troll.

Oh boy + +

Some background on the original issue

I’m developing a reasonably big set of automated web tests against a large web application behind a corporate firewall. I am writing them for a client, and since they are a Microsoft .NET environment, I am using the C# 4.0 Selenium WebDriver bindings, developing the tests against on Windows.

I have 30 odd end to end scenarios that run perfectly (100% reliably again and again) against Firefox with native events disabled. And then I try to run them in IE. Half of them fail. I rerun them and about a quarter fail, but different ones to the original run. I run them another time and again they fail in different places. What is going on?

The error messages don’t make any sense. The screenshots don’t line up with where the code is failing, as though navigation is failing. After lots of frustration I realize that the driver is silently failing. It thinks it has clicked (but actually hasn’t) so it fails on locating an element that it thinks should be there, but isn’t, because it didn’t click the previous element which was meant to change the page. But what makes matters worse, is it’s inconsistent, one run it works fine, the next it fails, or it fails in a different spot.

I believe it’s native events related, as the IE Driver uses native events to send mouse clicks, and if the browser doesn’t have perfect focus, these events may fail (but think they succeeded). So I make some inquiries to see if I can disable native events, or use synthetic JavaScript events like Firefox in Internet Explorer, but no such support exists.

I look on StackOverflow where I find dozens of questions about this issue dating back over the last year or so (some examples: here, here, here, here and here). There doesn’t seem to be a really solid solution or workaround, but some mentioned include firing every click twice in case one doesn’t work (which obviously introduces other concurrency issues), using sendKeys(“\n”) on elements (strange), refocusing to the webDriver window every time (seems very inefficient to me) and finding the parent of each element using an xpath expression and clicking that (urrrgh). Not really what I was looking for.

I raise it as a criticism: the IE driver is unreliable, but it isn’t taken well. I am very surprised it is the first time the Selenium team have heard of the issue considering all the references (see links above) and workarounds (hacks) I manage to find across the Internet. But it’s very hard to consistently reproduce inconsistent reliability issues.

The problem with reliability issues

The problem with reliability issues is that they’re not easily reproducible. Imagine you’ve got a Volkswagen Passat, and every so often you’re driving down the highway and the engine cuts out. It’s happened to you a dozen times over a few months, but each time has been different (sometimes uphill, down, somethings when braking) so you take it into the Volkswagen dealership. They ask you to take them for a drive to show them the problem, but guess what, it doesn’t happen. Does this mean the problem doesn’t exist? Because you can’t reproduce it? No.

Sporadic reliability issues are notoriously hard to reproduce, but it doesn’t mean they don’t exist, and it certainly doesn’t mean anyone who notices one, or mentions one, is a troll.

The problem with reproducing webdriver issues on client sites

I don’t want seem to be providing excuses for not reproducing issues, but you must realize it’s often very hard to do so. Reproducing inconsistent behavior is not only difficult to begin with, it’s a lot more difficult when working on a corporate application. Firstly, as a consultant, you are legally bound by a non-disclosure agreement to not discuss/share the work/code you are writing, so you must take any issue you’re working on and reproduce it in an external environment completely different from your work at hand. This includes both the client side webdriver code, and the web application code you’re testing, which you’ll have to reproduce yourself. Secondly, the code you’re working on is behind a firewall, often running on a customized desktop Microsoft Windows SOE, not something that can be easily reproduced on a MacBook. And thirdly, in my current case, I don’t have a Visual Studio license or configuration to even work on reproducing the issue.

We need to move pass treating open source projects as sacred cows

I’ve coped a lot of flak over my criticisms of open source projects in the past. I don’t think it’s that fair considering I’ve received a lot of criticisms myself. I have previously shared my views on Capybara which got a lot of people upset. But how are open source projects ever going to get better without criticism? Are open source projects really the sacred cows they make themselves out to be?

It’s quite a sport to criticize commercial software. Think about Microsoft Windows and Internet Explorer: there’s hate sites on the Internet devoted to them. Imagine you worked on the IE core team as a developer. Software developers have also put huge amounts of personal and collective effort into all these commercial products, the only difference is companies have charged for the usage of them. Think about QuickTest Professional (QTP). I lost count of how many times I heard shots against QTP being fired at the Selenium Conference (including those much recited job ad statistics: ‘Selenium’ vs ‘QTP’ on Indeed) and Selenium was even named as a antidote to Mercury poisoning (Mercury Interactive originally made QTP before selling it to HP).

So, does not charging for, or open sourcing software, make it not open for criticism? What does the open in open-source stand for?

We need to recognize that yes, a lot of hard work goes into an open source project, whether it be a simple rubygem, or a large automated testing tool such as Selenium or Watir. But we also need to recognize that just because someone spends their spare time on a such a tool doesn’t instantly make it great and immune to criticism. Greatness comes from usage and feedback.

Sure, we can’t tolerate blatant disrespectful criticism (I believe I may have came across this way initially about the IE Driver issues and I subsequently apologized for it), but we can’t only take criticisms or mention of issues unless accompanied by a self reproducing test case and an offer to dive into unfamiliar technology and fix the problem. I often fix issues with open source software that I write that aren’t accompanied by a reproducible test case: because I’m the expert who wrote the thing and chances are I’ll know what’s going on. An example is this defect someone raised, and I fixed on the same day, last week.

I would personally rather hear about an issue on one of my open source projects without a test case than not hear about it at all, because the person was too afraid of being called a troll to speak up or raise an issue without it being reproducible.

I understand the Watir/Selenium teams have a long, rather non-amicable history together, but I thought that it was getting better with the fantastic Watir-WebDriver library and Selenium Conference, but it seems from this incident we have a long way to go. Here’s hoping we can get over our minor differences and work together to create the best open source projects for ourselves and for our users.

6 thoughts on “The sacred cow that is an open source project

  1. Ah. I’d been hurting for a Safari driver from Apple. I hadn’t had a lot of non-work-around-able IE stuff.

    Is the IE webdriver entirely closed-source? Admittedly, I was recently grumbling a lot about not having sources or debug symbols for IE itself, but isn’t the webdriver a .jar that mostly talks COM to IE?

    1. Actually, the IE WebDriver is a standalone executable written in C++, and there are language bindings that communicate with it via a JSON-over-HTTP protocol. And it’s completely open source. You can see all the gory details at http://code.google.com/p/selenium

  2. I appreciate the views you have on critiquing OSS project for improvement. The only way they can grow properly is through review and critique, even great OSS projects like WATIR. No code is perfect.

  3. Thank you for posting this information.

    I too have developed a very large web-app testing framework that in its previous incarnation (under Watir) worked fine with IE. The new watir-webdriver version works fine with Windows7/FireFox but when using IE9, about half the tests fail, typically because an element can’t be found, or an action (usually a .click) simply is disallowed or has no effect. Of course when THAT action would change the context (i.e., select a different Tab, enable page-elements, or open a dialog/window), attempts to access elements in the new context fail (as they should).

    I have tried numerous workarounds that, like the problems I’ve experienced, are only sporadically effective.

    Unfortunately, IE8/9 is mandated by the users of this web-app (while FF is optional) so I MUST get this working with IE.

    If the Selenium IE driver is the component that must be enhanced/fixed to resolve these difficulties, I’m not sure what course of action is available (short of working with its source code which is beyond my expertise).

    Any suggestions will be greatly appreciated.

  4. I am also working on and with a sizable browser based application testing platform which began with classic Watir and has been moving to Watir-webdriver. The inconsistency between the IE and the Firefox/Chrome support is crippling and requires us to support both technologies. This is painful given our limited resources and our having begun with Ruby 1.8.7, Watir 1.8.1 and not having been able to update either (especially Watir to take advantage of the merging of the Watir api toward that of Watir-webdriver.

    I have built a Cucumber script for IE that runs either classic Watir or Watir-webdriver and run it over a dozen consecutive runs in each. Watir is perfect. Under Watir-webdriver (i.e., the IE driver), it failed 9 of the 12 runs, with three different errors. No changes in the script or the target application were made during any of the 24 runs. Unfortunately this work for a corporate client and I can’t disclose it to provide examples.

    Have been doing development for nigh unto 30 years, including a lot of testing, manual and automated. I know how hard it can be to get something to work reliably, especially when the target moves and hides…

    OSS has to be able to handle criticism without its authors bristling, especially if those authors want to see their work used in the enterprise sphere. And many of us in the enterprise world would much prefer to use OSS where we _can_ see under the covers and maybe help find fixes.

Comments are closed.