Before I start, I want to make it clear that I am huge believer in and supporter of open source software, and I appreciate all the time and effort that people put in to make it so. I want to make open source software better and it is the only reason I write this post.
It all started a couple of months ago when I made what I considered an off hand remark about how I felt the Internet Explorer Webdriver driver was unreliable. Unfortunately this wasn’t taken very well at all by the core IE WebDriver driver developer who found my view unexpected and unfounded. I was surprised but took actions to rectify the issue I had created, such as emailing/tweeting the developer directly with apologies and apologizing to the mailing list where I made the remark. There’s only one Watir developer left working on the Watir IE driver, and I even asked this individual at the time if they would work on improving the Webdriver IE driver but this person didn’t feel confident considering it’s written in C++, and they’re a Ruby developer.
I thought it was all okay, as I hadn’t heard of it again. I didn’t raise any criticisms again (after how they were taken), but then yesterday out of the blue there was this:
And then this one surprised me even more, considering what I did originally:
Update: 19 June 2012: Apparently these comments had nothing to do with me.
Oh boy, Groundhog Day, and I hadn’t even dared to speak of the unreliability again. But that wasn’t the end of it. The Selenium Team caught on and starting retweeting the original tweet, and then adding to the message accusations of trolling, as if anyone who raises any issue, without a reproducible test case, even if they don’t know the underlying technology/language, is apparently a troll.
Oh boy + +
Some background on the original issue
I’m developing a reasonably big set of automated web tests against a large web application behind a corporate firewall. I am writing them for a client, and since they are a Microsoft .NET environment, I am using the C# 4.0 Selenium WebDriver bindings, developing the tests against on Windows.
I have 30 odd end to end scenarios that run perfectly (100% reliably again and again) against Firefox with native events disabled. And then I try to run them in IE. Half of them fail. I rerun them and about a quarter fail, but different ones to the original run. I run them another time and again they fail in different places. What is going on?
The error messages don’t make any sense. The screenshots don’t line up with where the code is failing, as though navigation is failing. After lots of frustration I realize that the driver is silently failing. It thinks it has clicked (but actually hasn’t) so it fails on locating an element that it thinks should be there, but isn’t, because it didn’t click the previous element which was meant to change the page. But what makes matters worse, is it’s inconsistent, one run it works fine, the next it fails, or it fails in a different spot.
I look on StackOverflow where I find dozens of questions about this issue dating back over the last year or so (some examples: here, here, here, here and here). There doesn’t seem to be a really solid solution or workaround, but some mentioned include firing every click twice in case one doesn’t work (which obviously introduces other concurrency issues), using sendKeys(“\n”) on elements (strange), refocusing to the webDriver window every time (seems very inefficient to me) and finding the parent of each element using an xpath expression and clicking that (urrrgh). Not really what I was looking for.
I raise it as a criticism: the IE driver is unreliable, but it isn’t taken well. I am very surprised it is the first time the Selenium team have heard of the issue considering all the references (see links above) and workarounds (hacks) I manage to find across the Internet. But it’s very hard to consistently reproduce inconsistent reliability issues.
The problem with reliability issues
The problem with reliability issues is that they’re not easily reproducible. Imagine you’ve got a Volkswagen Passat, and every so often you’re driving down the highway and the engine cuts out. It’s happened to you a dozen times over a few months, but each time has been different (sometimes uphill, down, somethings when braking) so you take it into the Volkswagen dealership. They ask you to take them for a drive to show them the problem, but guess what, it doesn’t happen. Does this mean the problem doesn’t exist? Because you can’t reproduce it? No.
Sporadic reliability issues are notoriously hard to reproduce, but it doesn’t mean they don’t exist, and it certainly doesn’t mean anyone who notices one, or mentions one, is a troll.
The problem with reproducing webdriver issues on client sites
I don’t want seem to be providing excuses for not reproducing issues, but you must realize it’s often very hard to do so. Reproducing inconsistent behavior is not only difficult to begin with, it’s a lot more difficult when working on a corporate application. Firstly, as a consultant, you are legally bound by a non-disclosure agreement to not discuss/share the work/code you are writing, so you must take any issue you’re working on and reproduce it in an external environment completely different from your work at hand. This includes both the client side webdriver code, and the web application code you’re testing, which you’ll have to reproduce yourself. Secondly, the code you’re working on is behind a firewall, often running on a customized desktop Microsoft Windows SOE, not something that can be easily reproduced on a MacBook. And thirdly, in my current case, I don’t have a Visual Studio license or configuration to even work on reproducing the issue.
We need to move pass treating open source projects as sacred cows
I’ve coped a lot of flak over my criticisms of open source projects in the past. I don’t think it’s that fair considering I’ve received a lot of criticisms myself. I have previously shared my views on Capybara which got a lot of people upset. But how are open source projects ever going to get better without criticism? Are open source projects really the sacred cows they make themselves out to be?
It’s quite a sport to criticize commercial software. Think about Microsoft Windows and Internet Explorer: there’s hate sites on the Internet devoted to them. Imagine you worked on the IE core team as a developer. Software developers have also put huge amounts of personal and collective effort into all these commercial products, the only difference is companies have charged for the usage of them. Think about QuickTest Professional (QTP). I lost count of how many times I heard shots against QTP being fired at the Selenium Conference (including those much recited job ad statistics: ‘Selenium’ vs ‘QTP’ on Indeed) and Selenium was even named as a antidote to Mercury poisoning (Mercury Interactive originally made QTP before selling it to HP).
So, does not charging for, or open sourcing software, make it not open for criticism? What does the open in open-source stand for?
We need to recognize that yes, a lot of hard work goes into an open source project, whether it be a simple rubygem, or a large automated testing tool such as Selenium or Watir. But we also need to recognize that just because someone spends their spare time on a such a tool doesn’t instantly make it great and immune to criticism. Greatness comes from usage and feedback.
Sure, we can’t tolerate blatant disrespectful criticism (I believe I may have came across this way initially about the IE Driver issues and I subsequently apologized for it), but we can’t only take criticisms or mention of issues unless accompanied by a self reproducing test case and an offer to dive into unfamiliar technology and fix the problem. I often fix issues with open source software that I write that aren’t accompanied by a reproducible test case: because I’m the expert who wrote the thing and chances are I’ll know what’s going on. An example is this defect someone raised, and I fixed on the same day, last week.
I would personally rather hear about an issue on one of my open source projects without a test case than not hear about it at all, because the person was too afraid of being called a troll to speak up or raise an issue without it being reproducible.
I understand the Watir/Selenium teams have a long, rather non-amicable history together, but I thought that it was getting better with the fantastic Watir-WebDriver library and Selenium Conference, but it seems from this incident we have a long way to go. Here’s hoping we can get over our minor differences and work together to create the best open source projects for ourselves and for our users.