There is more to testing than Automation, here’s why.

Since joining SOFY.AI almost a year ago, I have been immersed in understanding software testing and role of automation in testing.  When I joined SOFY, I had the view full automation is the holy grail of software testing – until I got a holistic perspective.  My views evolved and so let me share my journey and how it shaped how we approach testing in SOFY.AI. 

To put this in context, let me start with different dimensions of Testing.  Much has been written about this topic which can find in many articles in medium website.

1. Functional and Non-Functional Testing

All different types of testing can be broadly classified into functional or non-functional testing.  Functional testing is app-specific and validates app functionality.  So if you have a shopping app, functional tests validate your core functions like product catalog search, shopping cart, shipping and tax calculation, checkout experience, etc. Non-functional testing is app-agnostic and is about behavior of the app independent of the app’s functionality. It tests for app’s performance, UX design, usability, crashes, globalization, localization so that these issues are detected before the app is released to production. Much of non-functional testing is performed manually using exploratory testing given the non-deterministic nature of these tests.

Types of Functional Testing

Types of Non-Functional Testing

2. Manual and Automation

Some of the most important decisions a testing team has to make is to determine which of these test types are needed for their project and whether to automate them or execute them manually. Manual testing, as the name suggests, is performed by a QA tester.  It offers the benefit of minimal upfront investment and provides an easy path to get started. However, it comes with cost, lack of transparency, need to provide devices each of testers, etc. On the other hand, automated testing solves these problems providing scale and repeatability.  Automation requires use of testing frameworks (Appium, selenium, cucumber, etc.), expertise of QA engineers with deep technical skills, and use of multiple tools. However, one must constantly modify these automation scripts to adapt to UI changes, device and OS compatibility.

Types of Automation:

There are essentially 2 ways to automate.

  1. Automation using scripts:  This is a do-it-yourself way of automation using frameworks like Appium, Selenium, Cucumber, etc.  This is the dominant way automation is done today.  Most of the testing tools (Sauce Labs, Browserstack) are optimized for this scenario.  This approach is much better than plan scripting as these frameworks try to abstract some of the elements of underlying OS.  But they have their own challenges and is for another blog.
  2. Nocode or low code (NCLC) automation is a new way to creating automation without writing code and uses Machine Learning to take away the complexity of writing scripts and some of the challenges of writing automation using testing frameworks.

Let me finish this blog with following observations.

  • Manual testing is still significant part of any project testing.  A very high percentage of testing is performed using manual approach in early stage products.  As a product matures, percentage of tests that are automated increases 50% or 60% of total testing.
  • Today, automation is primarily about writing code/scripts using frameworks such as selenium, Appium, etc. to perform functional testing.  Most of the tools in the market target this scenario. 
  • No code automation is still far and few.
  • Exploratory testing is heuristic in nature and is performed predominantly using manual approach. There are very few tools that support automation of non-functional UI testing.

In the next blog, I will talk about how SOFY is addressing these scenarios in a more holistic way.

Leave a Reply