UI Testing Frameworks
opened this issue · comments
Possible UI testing framework and libraries should be analyzed.
In this issue, we are going to analyze 8 UI automation test frameworks in terms of their being Open Source, using with both iOS and Android platforms and real devices, CI, XCode Integration, scripting/testing language, having test recorder.
Appium, Katalon, Calabash, Jest, Detox, Kif, EarlGrey 1 & 2 and XCUITest
I thought this issue includes the swift side UI testing. Appium, katalon etc. are automation tools, is that what we wanna do?
APPIUM
The Appium framework is an open-source testing tool. It supports the automation of the application testing processes on iOS and Android platform devices. The framework provides excellent synchronization of native applications developed specifically for the iOS system. [1]
Appium can work integrated with Cucumber which is a testing framework that supports Behaviour Driven Development (BDD).
Appium offers you to write the test script in different programming languages which include Java, JavaScript, PHP, Ruby, Python, and C#. [2]
You can set up a CI system for Android and IOS native/hybrid apps and how to plugin your Appium tests in the pipeline using Jenkins. [3]
Appium desktop provides an action recorder and code generator. [4]
Sample [1] : Link
KATALON
Katalon Studio is an all-in-one solution providing tools for mobile, Web, and API automation. Test cases may be recorded or created in just a few clicks. Appium is required for black-box mobile automation tests in Katalon Studio. [5]
Katalon Studio offers you to write the test script in Java/Groovy.
It has integration with Popular CI tools e.g. Jenkins, Teamcity. [6]
Katalon Recorder is the Selenium IDE-compatible replacement on the latest Chrome and Firefox that helps record, play, debug, manage automated tests. [7]
Sample [2] : Link
CALABASH (Formerly Microsoft Backed)
Calabash is an iOS automation testing framework that perfectly syncs with hybrid iOS applications. It is important to note that the tests inside this framework are based on Сucumber, in conjunction with BDD methodology, an extensive automatic testing tool. The framework is available on both mobile platforms - iOS and Android. [1]
Test classes will be written in Gherkin which is the format for Cucumber specifications. It is a domain-specific language that helps you to describe business behavior without the need to go into detail of implementation. [9]
One downside is that Microsoft will discontinue contributions to developing Calabash as of iOS 11. But the community will continue to fully adopt and maintain it.
Calabash does not recommend any CI system officially but gives Jenkins and Travis examples for integration.
Playbacks give us great power to record custom user interactions with the UI and replay them. They are created using the calabash-ios console and are recorded on the device. [8]
Sample [1]: Link
JEST (Facebook Backed)
Jest is a delightful JavaScript Testing Framework with a focus on simplicity. It works with projects using: Babel, TypeScript, Node, React, Angular, Vue, and more. [12] Jest does not support native swift / obj-c projects. [14]
We are just passing further analysis because of its not supporting native projects.
DETOX (Wix Backed)
The major problems in mobile UI testing are slowness and flakiness. The tools like Appium are completely black-box and uses a client-server architecture which causes flakiness in the UI tests. Detox is designed to solve the problem of slowness and flakiness for mobile UI testing. [10] The leading native gray box drivers are developed by Google — EarlGrey for iOS and Espresso for Android. Detox does not rely on WebDriver since this is not the web. Detox communicates with its native driver (which extends EarlGrey and Espresso) using a JSON-based reflection mechanism, this allows a common JavaScript implementation to invoke native methods directly on the device. So its test is on Javascript, preferably .jest files. [11]
Detox supports cross-platform testing both Android and iOS, but not real iOS devices. It is mostly preferred by React Native App testing.
It is compatible with the CI platform Travis.
There is not any recorder yet for generating a test script.
Sample : Link
Until here, none of the frameworks have Xcode Integration; now analyze KIF, EarlyGrey, and XCUITest which can be run from XCode itself.
KIF
KIF is iOS only testing platform framework that allows you to drive your UI from your 'unit tests' and reap all the advantages of testing in-process. KIF builds and performs the tests using a standard XCTest testing target. Testing is conducted synchronously in the main thread (running the run loop to force the passage of time) allowing for more complex logic and composition. This also allows KIF to take advantage of the "Xcode Test Navigator", command-line build tools, and Bot test reports. KIF integrates directly into your Xcode project, so there's no need to run an additional web server or install any additional packages. You can easily run a single KIF test with the Test Navigator or kick off nightly acceptance tests with Bots. [13]
Because of its integration with XCode test navigator, we have all power of CI integration power that native has.
Sample Gist: Link
EarlGrey 2
EarlGrey has two versions 1 and 2. Because EarlGrey 2.0 is newer and EarlGrey 2.0 uses a UI Testing Target and not a Unit Testing Target like EarlGrey 1.0, we focus on 2.0.
EarlGrey 2.0 is a native iOS UI automation test framework that combines EarlGrey with XCUITest, Apple's official UI Testing Framework. EarlGrey 2.0 allows you to write clear, concise tests in Objective-C / Swift and enables out of process interactions with XCUITest.
It has the following chief advantages that allow you to query the application under test from your tests and you can use EarlGrey 2.0 natively with Xcode. You can run tests directly from Xcode or xcodebuild. EarlGrey 2.0 currently supports Xcode Projects and building from source for both white and black box testing. [15]
Google is actively using this tool to test their applied programs, including YouTube, Google Calendar, Google Music, and others.
XCUITest is a black-box testing framework, this is not directly possible with EarlGrey 2.0. To fix this, we use eDistantObject (eDO) to allow these white-box interactions.
Sample Gist: Link
Extra: EarlGrey is the ‘Espresso for iOS’ :)
XCUITest
Both XCTest and XCUITest are integral automation frameworks by Apple that are shipped with Xcode for testing iOS applications. XCUITest is a User Interface (UI) testing framework by Apple that is built on the top of XCTest (a unit testing framework). [16]
XCUITest is capable of performing "black-box" UI Testing based on a VoiceOver accessibility feature.
COMPARISON of XCUITest and EarlGrey 2
EarlGrey 2.0 advantages over XCUITest
- Automatic synchronization with Animations, Dispatch Queues, and Network Requests as enumerated here.
- In-built White-Box Testing Support with RMI.
- Better Support for Flakiness Issues.
- Better Control of tests. EarlGrey has a much larger set of matchers.
- EarlGrey performs a pixel-by-pixel check for the visibility of an element.
XCUITest advantages over EarlGrey 2.0
- XCUITest application launches without a delay.
ELEMINATION
Regarding these frameworks, we can group them in 3;
- XCode Integrated Frameworks
- Based on Unit Test
Kif - Based on UI Test
XCUITest, EarlGrey
- Out-of-XCode Frameworks
Appium, Katalon, Calabash, Detox
We can eliminate unit test based XCode integrated frameworks because there are two strong UI test based frameworks. Besides, Kif's one creation reason in those days is not-being-able to create that powerful framework with the UI test base.
We can easily eliminate Detox for not being able to test on real iOS devices. Between the rest 3 one (Appium, Katalon, and Calabash), Appium stands out with flexibility and community size. And remember that Microsoft pulled out its support from Calabash.
PERFORMANCE COMPARISON
Let's compare the remaining ones' (XCUITest, EarlGrey, Appium) performances.
Here is a very valuable blog post about their performance comparisons with 6 sample test in two parts. The blog post 'Thirty Seconds to Pass: Performance Comparison of iOS UI Testing Frameworks' by Roman Zakharov
In the first part, for all 3 samples; we can easily see that EarlGreys (both 1 and 2) have shown greater performance than XCUITest and Appium.
In the second part, for all 3 samples; Although EarlGrey 2, XCUITest, and Appium show similar performance, EarlGrey 1.0 shows significantly better performance on 1 sample.
In light of the detailed information and the given comparison results, my first choice will be EarlGrey, and the second is XCUITest.
[1] - https://applikeysolutions.com/blog/7-best-ios-testing-frameworks-comparison-guide
[2] - https://www.browserstack.com/guide/appium-tutorial-for-testing
[3] - https://confengine.com/appium-conf-2019/proposal/9005/setting-up-jenkins-ci-pipeline-using-appium-tests-for-android-and-ios
[4] - https://www.einfochips.com/blog/understanding-appium-desktop-test-automation/
[5] - http://katalon.com/katalon-studio
[6] - https://www.katalon.com/resources-center/blog/comparison-automated-testing-tools/
[7] - https://docs.katalon.com/katalon-recorder/docs/overview.html
[8] - https://medium.com/@johnwildoran/automated-ui-testing-calabash-7ed8b96b4aa5
[9] - https://www.guru99.com/gherkin-test-cucumber.html
[10] - https://bitbar.com/blog/detox-superfast-e2e-react-native-ui-testing/
[11] - https://medium.com/@selvakumarsubramanian/detox-gray-box-testing-dc6e0a800575
[12] - https://jestjs.io/
[13] - https://github.com/kif-framework/KIF
[14] - https://github.com/facebook/jest
[15] - https://github.com/google/EarlGrey/tree/earlgrey2
[16] - https://www.browserstack.com/guide/top-ios-testing-frameworks
Testing for @adessoTurkey/ios broadcast messaging. This issue will be discussed at Friday's meeting.
We have decided to use EarlGrey 2.0 instead of XCUITest.