There’s more to testing than automation. Here’s why.

Interested in learning more about manual-code, low-code, and no-code testing? Have a look at this 2022 article from the Sofy team!

Since joining Sofy, I am getting a better appreciation of role of automation in software testing. While automation may be the nirvana for any software QA team, 100% automation is impossible.  This also the pragmatic view we took in developing Sofy.

To get a holistic perspective, let me start with the different dimensions of Testing. You can find in many articles on this topic over at Medium.

1. Functional & Non-Functional Testing

Testing can be broadly classified into functional or non-functional testing. Functional testing is app-specific and validates app functionality. So if you have a shopping app, functional tests validate your core functions like product catalog search, shopping cart, shipping and tax calculation, checkout experience, etc. Non-functional testing is app-agnostic and is about behavior of the app independent of the app’s functionality. It tests for app’s performance, UX design, usability, crashes, globalization, localization so that these issues are detected before the app is released to production. Much of non-functional testing is performed manually using exploratory testing given the non-deterministic nature of these tests.

2. Manual & Automation

One of the most important decisions a testing team has to make is to determine which of the above types of testing is needed for their app and how much to automate. Manual testing, as the name suggests, is performed by a QA tester. It offers the benefit of minimal upfront investment and provides an easy path to get started. However, it comes with cost, lack of transparency, need to provide devices each of testers, etc. On the other hand, automated testing solves these problems providing scale and repeatability. Automation requires use of testing frameworks (Appium, selenium, cucumber, etc.), expertise of QA engineers with deep technical skills, and use of multiple tools. However, one must constantly modify these automation scripts to adapt to UI changes, device and OS compatibility.

Types of Automation

There are essentially two ways to automate.

1. Automation using scripts: This is a do-it-yourself way of automation using frameworks like Appium, Selenium, Cucumber, etc. This is the dominant way automation is done today. Most of the commercial automation tool for mobile testing (Sauce Labs, Browserstack) are designed for this scenario. This approach is much better than custom scripting solutions as these frameworks try to abstract some of common functions needed for automation. When considering automation tool for mobile testing, one has deal with variations in device form factors, differences in behavior across OS version within a OS family (Android or IOS) as these are not always handled by the frameworks.
2. No-code or low-code (NCLC) automation is a way to creating automation without writing code. It takes users intent and auto-generates automation scripts based on the frameworks. Some of the automation tool for mobile testing in this category also use AI and ML to address the above challenges that addressed by the frameworks.

Summary

Manual testing is still significant part of any project testing. A very high percentage of testing is performed using manual approach in early stage products. As a product matures, percentage of tests that are automated increases 50% or 60% of total testing.

Today, automation is primarily about writing code/scripts using frameworks such as selenium, Appium, etc. to perform functional testing. Most of the tools in the market target this scenario. No code automation is still far and few.

Exploratory testing is heuristic in nature and is performed predominantly using manual approach. There are very few tools that support automation of non-functional UI testing.