Diving in to this Budgeting Sample Application test automation assignment presented me with many new exciting options in determining how to best develop a test plan including a selection of automation tests. My previous software testing background relied mainly on using Java, a small bit of Javascript and Ruby Selenium, Maven, JUnit and Allure for all web application testing.
To meet the requirements (e.g. no Java) for this project, I decided to challenge myself by using Javascript for writing the tests, along with implementing the Node.js tool TestCafe as my end-to-end web-based testing framework. My reasoning behind selecting TestCafe over a Selenium-driven framework for this assignment was based on the following:
- Easy installation and doesn't require WebDriver, ready to run tests after
npm install -g testcafe
(thus no additional installs/configuration/maintenance of plugins, libraries, dependencies, products etc.). - It's built on top of Node.js, which I have more familiarity with, and runs tests quickly in real browsers.
- There's a bit more stability than Selenium, as it automatically waits for page loads and XHR requests, and I'm very happy with the async/wait JS ES2017 feature along with the smart test actions and assertions that wait for page elements to appear.
- The TestCafe API library is clear, concise and offers many possibilities, including full-featured front-end framework-specific libraries of selectors for testing web applications.
- While the features include CI, reporters, concurrent testing and solid debug reporting, there's a growing ecosystem of developers creating custom plugins and extensions that give great enthusiasm.
This Budgeting Sample Application is a contemporary React web application that offers users the ability to enter and track their income and expenses. Additionally, it enables the user with a feature to graphically view all financial activity 'Inflow vs Outflow' and drill down into their 'Spending by Category' with visual bar graphs and pie chart representations of the items entered in on the 'Budget' view.
For the purposes of this assignment, the ultimate goal is to devise a scalable end-to-end test suite that mimics what and how a user would interact with the web application delivering successful expected results. This includes the way in which the application launches, responds to user input and validation that subsequent-displayed data is correct across both the 'Budget' and 'Reports' features.
At a high level, for this product release version, the automation testing scope for this project is the to test all features available to the end user within the 'Budget' and 'Reports' views of the application. The test suite here allows coverage of the application that unit tests and integration test do not cover. The benefits are that the application test suite can be run as frequently as possible, allows timely developer code push and production deployment releases with deeper trust. Lastly, this testing will catch errors missed during manual, unit and integration testing.
The user input field in the application consists of eight main areas to test, where a user will interact with the application. Since the application only allows a user to 'Add' an item, not remove or edit, and the positive or negative 'Value' entered is predetermined by the Category selection (e.g. Income is fixed to a positive numerical value, while all other categories are fixed to a negative numerical value), automating a user's behavior is a bit more straightforward. The eight key areas are:
- Main launch page / clickable Budget button
- Category drop-down selection
- Description entry field
- Amount entry field
- Clickable and Return-key Add button
- Clickable Reports button
- Clickable Spending by Category tab
- Clickable Inflow vs Outflow tab
The application also features graphic and numerical data displays on the 'Budget' and 'Reports' views of the application that will require verification/comparison testing to ensure that the entered data, of text and dollar amounts entered into the 'Budget' inputs correctly persist across both views. This includes:
- The 'Working Balance' numerical display on the bottom of the 'Budget Page'
- The 'Inflow vs Outflow' bar graphs on the 'Reports' view
- The 'Spending by Category' pie charts on the 'Reports' view
The defined success criteria ('Pass') of each item/feature relies on the test case strictly meeting the expected outcome. For this product, each test case has a clear identification of criteria required to deduce a 'Pass' or 'Fail'. The criteria can be further defined by three areas for the testing of this application:
- Suspension - any scenario which causes a blocker to continue the automation testing.
- This could be due to issues within the environment (local and development configuration), server, browser, version release and test harness.
- An example specific to this application is where automation testing fails due to an external browser issue (e.g. the latest Chrome release triggers issue/test failure when launching the application on localhost:3000 due to a new browser feature or as mentioned in the product notes because localhost not on https/SSL.
- Resumption - As in the Chrome browser example scenario above, where the test automation suspension occurred due to a new feature that blocked the successful automation application launch, the issue is identified, automation test code or command is updated, then testing may resume.
- An example in identifying the test case failure issue, parallel browser test execution would be a strong and timely tactic to resolve the automation blocker
- During the development of my automated test scenarios for this application, I discovered that the local Chrome tests were not consistently passing. Running the tests in parallel, across multiple browsers yielded results that showed the test in question passing locally in Firefox and Safari, but not in Chrome. This documented in the Environmental Requirements below, but it led to running the automation testing file command that included a skip Javsascript web page error option to pass the test.
- Approval - the scenario in which each test has met the expected outcome defined in the test case, after re-running the test suite. For this application approval occurred in the Chrome example after resuming the test suite with Chrome tests passing.
The individual test cases below are a set of automate-able tests that will yield the general application and user-behavior results.
Test Case | Expected Outcome | Result |
---|---|---|
TC01 - Launch Application via localhost:3000 | Application should successfully launch | Pass/Fail |
in browser(s) selected | ||
TC02 - Verify that the main/home 'Budget' view basic | All expected columns, column text, text entry | Pass/Fail |
elements are present on page | fields are present | |
TC03 - Verify that all expected images exist on the 'Budget' | Org svg logo, is present on both views | Pass/Fail |
and 'Reports' views | ||
TC04 - Verify that description column contains | Placeholder description items are present | Pass/Fail |
specific item names | (e.g.'Gas', 'Paycheck', 'Trader Joe's food') | |
TC05 - Click 'Reports' button for verification that | 'Inflow vs Outflow' and 'Spending by Category' | Pass/Fail |
'Inflow vs Outflow' and 'Spending by Category' views are present | exist and display the correct text fields | |
TC06 - Verify that the 'Inflow vs Outflow' bar graphs are present | 'Inflow vs Outflow' bar graphs are displaying | Pass/Fail |
on the 'Reports' page | ||
TC07 - Click and verify that the 'Spending by Category' pie chart | 'Spending by Category pie chart is displaying' | Pass/Fail |
is present on the 'Reports' page | ||
TC08 - Click the 'Budget' tab, then click the 'Category' drop-down | 'Category' drop-down give the user 17 options | Pass/Fail |
button and verify that there are 17 options available | to select | |
TC09 - Click on 'Category' drop-down and select option 15 'Income', | A new item 'Paycheck' is created displaying in | Pass/Fail |
then click in to the 'Description' field, enter text 'Paycheck', next | the 'Description' column, with the 'Amount' | |
click in to Value field and enter '4000', then click the Add button | '$4,000' populating the Amount column | |
Additionally, 'Total Inflow' - 'Total Outflow' | ||
= 'Working Balance' is verified and updated | ||
with a positive numerical value increase | ||
then click in to the 'Decscription' field, enter text 'Colombia', next | the 'Description' column, with the 'Amount' | |
Additionally, 'Total Inflow' - 'Total Outflow' | ||
= 'Working Balance' is verified and updated | ||
with a negative numerical value decrease | ||
TC11 - Repeat TC09 and TC10 with various generated Text and Numerical | All added items are present with the correct | |
values for example 'Category' item in the drop-down | text, numerical value and 'Working Balance' | |
correctly reflects the change in item Inflow | ||
and Outflow values (postive or negative) | ||
TC12 - Click on 'Reports' tab and verify that the newly added items | Numerical values should compare and equal | Pass/Fail |
in the 'Budget' view are correctly represented in 'Inflow vs Outflow' | those created in 'Budget' view, newly added | |
items have Category text updated, values | ||
updated and the bar graph is updated including | ||
Inflow/Outflow values | ||
TC13 - Click on the 'Spending by Category' tab under 'Reports' tab and | Numerical values should compare and equal | Pass/Fail |
verify that newly added items in the 'Budget' view are correctly | those created in 'Budget' view, newly added | |
represented in 'Spending by Category' | items have Category text updated, values | |
updated and the pie chart is updated |
Below are the detailed steps/specifications, tools and commands for replicating my selected automation tests on the Budgeting Sample Application.
-
Clone and build my repo version of the Budget Sample Application QA-automation-project-test-cafe that contains the
e2e
folder with a selection of my test scripts to run against the application. -
My browser and hardware environment for this project: Chrome 65.0.3325 / Mac OS X 10.13.4
-
Navigate to the locally saved project directory of the modusCreate-QA-automation-project, and into the
e2e
folder. -
Ensure that Node.js and npm are installed and/or updated to the latest versions.
-
Install TestCafe on the command line:
npm install -g testcafe
-
Install the TestCafe React plugin:
$ npm install testcafe-react-selectors
-
Start up the previously built application:
npm run serve
to serve the app on localhost:3000 -
Open a new terminal tab or command line.
-
The Chrome potential known automation issue execution (as mentioned above in the Pass/Fail Criteria section) with TestCafe can sometimes require that you run my automation test script with following suffix:
--skip-js-errors
-
Enter and run the following on the command line (Chrome can be subbed out for Firefox, Safari etc.):
testcafe chrome doug-major-moduscreate-qa-automation-tests.js --skip-js-errors
-
TestCafe should launch Chrome, localhost:3000, and the automation tests should run successfully.
-
Command line output should return each test in the suite where 1 of the 5 automation tests will fail (Test #2 as detailed in the commented-out explanation above the test code). All other tests should pass successfully.
Other potential dependencies and/or errors known that can affect the automation test script to fail to execute:
- According to the Node.js documentation3, if you are using require in the
doug-major-moduscreate-qa-automation-tests.js
file, the testcafemodule should be in the node_modules folder. Please note that this does not apply to the fs module, because it's a build-in Node.js module.
As such, use a locally installed TestCafe version and type the following command6 in your project folder:
npm link testcafe