The current state of iOS automated functional testing

I’ve been working on an iOS project and have been looking for a suitable automated functional test library to drive the iOS GUI. I must say that automated iOS testing feels like automated web testing did about 10 years ago: lots of different tools taking different approaches, and all approaches quite flaky!

Through research I came across quite a few different tools that support iOS automated functional testing but need to work out which one is best.

Whilst I often advocate writing functional tests in the same language as your codebase, in the case of iOS and Objective C, I think using a more lightweight language like Ruby has its own advantages (I don’t really like Objective C).

The one thing I really dislike with a lot of these iOS functional test tools is how they are married to Cucumber and encourage users to write tests like “I tap button x” and “I scroll through list y”. These types of tests are much harder to read as they express implementation over intention and should be avoided.

I will also be writing tests for an Android app so being able to use the same tool-set is a great advantage to me.

I also personally prefer an approach where I don’t need to modify the core behavior of my app to run tests against it (designing for testability is completely different and vital). Some approaches embed some sort of server to receive automation commands, which the user must then remove later because it uses undocumented Apple APIs which will lead to app store rejection. Being able to run tests against the app that you submit to the app store means you can be more confident you have tested it right.

Finally, the iOS app I am working on using embedded web views to display dynamic content from the server, which can be interacted with, and therefore it is vital that these can be interacted with. This feature is actually very rare in an iOS automation framework.

Here’s the list of iOS functional automation tools I am aware of and how they stack up:

  • Tool: Frank
    • Language: Ruby
    • Test Framework: Also supports OSX apps
    • Supports other mobile plaforms: Also supports OSX apps
    • Approach: Requires you to embed a symbiote server in your app and uses undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO (only via JavaScript calls)
  • Tool: KIF
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Modifies your app to use undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Subliminal
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Embeds into your project but uses UIAutomation instead of undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Zucchini
    • Language: Custom DSL (CoffeeScript/Ruby)
    • Test Framework: Custom
    • Supports other mobile plaforms:
    • Approach: Generates UIAutomation JavaScript that is executed against your app
    • Deploy same app to store: YES
    • Supports testing web-views?: NO
  • Tool: Calabash
    • Language: Ruby
    • Test Framework: Cucumber
    • Supports other mobile plaforms: Also supports Android Apps
    • Approach: Requires you to embed a server in your app to control it
    • Deploy same app to store: NO
    • Supports testing web-views?: YES
  • Tool: Appium
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: Also supports Mac OSX apps, Android Firefox OS
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES
  • Tool: ios-driver
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: NO
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES

I’ll let you guess which tool I selected until I write my next blog post about how to get started in that tool.

23 thoughts on “The current state of iOS automated functional testing

  1. 1.) I’m facing a similar situation – needing to support an iOS project, and now an Android version (and eventually even a Windows Phone version) of the same app, with automated testing. I’ve not loved my options so far (Frank and Calabash are what I’ve previously tried). I really appreciate you laying out what you found, especially the emphasis on cross-platform. Thank you!

    2.) Appium looks like a winner to me, but I’ll be looking forward to seeing what you move forward with and how.

        1. fair enough, I don’t find it a big deal. I have it all connected to Grid and it does the orchestration. simples :)

        2. One Ring To Rule Them All™

          seriously though why? chromedriver, IEDriverServer (and soon to be firefox marionette) are all separate tools to drive different browsers but all speak the Selenium WebDriver protocol.

          The idea of needing everything in one magic hammer is a bit silly to me.

          1. That’s a bit different to compare. ChromeDriver and IEDriverServer are simply binaries you keep on system and put in system path or reference the path in code (or startup options) and then use with Selenium.

            With Appium, and I’d assume the other tools, there a bit more setup work involved. For Appium, there’s now Appium.app binary package but it used to be you have to install node.js, then the Appium packages, etc.

            And Appium already supports 2 platforms making it a nice tool to do 2 things whereas ios-driver and Selendroid only do each, so you’ll need both in the latter case to support 2 platforms.

            Like in the real world of hardware tools, unless each specific tool is very good (i.e. the best) in it’s own job, I’d prefer to keep my toolkit small and get the multi-use tools where applicable, sticking with specific single use tool when it does the job best.

            But yest on the WebDriver client side of things, there’s very little to do to switch among tools provided you don’t use tool specific features much.

            1. appium, ios-driver and selendroid are binaries and I don’t think they’re very much different to any other driver. That isn’t where the hassle lies with mobile automation – it’s getting the pre-requisites in place for the simulators etc. That’s the same for all the tools unfortunately but we’re limited by the underlying components/sdks/frameworks which we wrap.

        1. That’s good to know, didn’t know that. So then, especially with the new mobile spec once finalized, I’d assume any of the compatible tools (for iOS and Android) would run fine against Sauce.

          Has Sauce mentioned support for ios-driver? They’ve mostly been touting Appium. It is good to know that another compatible tool can be used in its place on Sauce.

          1. anything will run on sauce which they’re happy to host – ultimately they are providing you access to a Grid, it behaves the same way as doing it yourself but takes the “complication” out of it, though I think that’s often overstated and I generally think people are afraid of grid for some reason. So, they have a bunch of vms with the selenium standalone-server running exposing the capabilities. They can run whatever else they want on those vms as nodes – be that appium [which they obviously favour] or ios-driver or selendroid or anything else for that matter. You may have to request it and they may not advertise it but there’s no reason for them not to do it. You’re paying by usage so I can’t see why they’d exclude themselves from some revenue by not hosting it.

  2. A few comments:

    * KIF does not modify your app. All the undocumented calls are in the unit test suite. We use KIF and our app is in the App Store, with no complaints from Apple.

    * We used to use Zucchini, and in fact I am a contributor to the project. I gave up on it because using Instruments and UIAutomation is such a frustrating experience.

    * Running instruments with UIAutomation is not supported on Xcode server. On the other hand, KIF uses the unit test framework, so all of its tests can run on Xcode server via a Bot.

    * As I mentioned above, using Instruments for testing is very painful, especially on CI servers. It is flakey and has a poor command-line interface. Furthermore, UIAutomation is not a framework I enjoy working with, mainly due to a view hierarchy solely based on accessibility, as well as numerous odd little bugs. This is the main reason I went with KIF, instead of appealing and well-intended solutions like Appium. If I had to go cross-platform…I would still try to avoid Instruments and UIAutomation :-)

    1. I got the information about KIF directly from its homepage (which still states):

      KIF uses undocumented Apple APIs. This is true of most iOS testing frameworks, and is safe for testing purposes, but it’s important that KIF does not make it into production code, as it will get your app submission denied by Apple. Follow the instructions below to ensure that KIF is configured correctly for your project.

      1. Like I said, KIF does not “modify your app” as you stated. That makes it sound dark and scary! It just adds undocumented calls to the test suite target and that code never makes it to the app store.

Comments are closed.