On November 30, 2022, OpenAI released a groundbreaking artificial intelligence (AI) chatbot called ChatGPT. This chatbot quickly went viral, with people showing off its immense power in everything from creative to technical, like writing stories and coding computer programs.
While this technology was not created for the sole purpose of software development and app testing, it comes as no surprise that developers and QA testers alike have latched onto this powerful tool to assist with app development and testing. As a language model designed to interact with humans, ChatGPT can be a helpful tool, especially in the realm of automated testing.
In this blog post, we’ll delve into what ChatGPT is, its advantages and limitations in the context of automated testing, how it can be used for mobile app testing, and some best practices to ensure optimal results.
What is ChatGPT?
ChatGPT uses natural language processing to create human-like conversations. If you ask ChatGPT to define itself, here is what they say:
ChatGPT is a language model developed by OpenAI. It is based on the GPT-3.5 architecture and is designed for natural language understanding and generation. ChatGPT is part of a family of AI models that are trained on a massive amount of text data from the internet, allowing them to generate human-like text in response to prompts or questions.
The primary purpose of ChatGPT is to engage in human-like conversations, answer questions, provide information, generate text, and more. It can be used in a wide range of applications, including chatbots, virtual assistants, content generation, language translation, and more. It’s particularly well-suited for tasks involving natural language understanding and generation, making it a versatile tool for developers and businesses looking to incorporate AI-driven language capabilities into their products and services.
ChatGPT-3.5 vs ChatGPT-4
OpenAI offers Chat GPT-3.5 (their free plan) and Chat GPT-4 (part of their paid ChatGPT Plus plan). If you’re looking to up your AI game, ChatGPT Plus, set at $20/month, is a great option. The major difference between these two plans lies in their capabilities. Chat GPT-4 is advertised as more reliable, creative, collaborative, and more capable of precisely handling nuanced instructions.
ChatGPT-4 can also process both text and image data, offers multilingual capabilities, and provides more control over its responses using a personality you can pick.
Advantages and Limitations of Using ChatGPT for Automated Testing
Advantages
- Natural language interaction:
- Allows for natural language interaction with the testing environment, making it accessible to non-technical team members who may not have coding expertise.
- Generates test cases:
- Can assist in rapidly generating test case scenarios based on textual descriptions and user stories.
- Error message interpretation:
- Interprets error messages, stack traces, and log files, making it easier to identify the root causes of test failures.
- Scalability:
- Handles a large number of inquiries and provides rapid responses, enhancing the scalability of automated testing processes.
- Unfamiliar topics:
- Fills in missing knowledge gaps and can help generate initial code for you as a starting point on unfamiliar features.
- Test case documentation:
- Assists in generating test case documentation, which is crucial for maintaining a comprehensive and up-to-date testing repository.
Limitations
- Lack of contextual awareness:
- ChatGPT lacks deep contextual awareness and may not fully understand complex concepts, which can lead to inaccuracies in test case generation.
- Dependency on training data:
- ChatGPT relies on the quality and diversity of its training data to perform well, which can introduce biases and inaccuracies.
- Makes assumptions without context:
- ChatGPT might guess a user’s intention to develop a response instead of asking clarifying questions.
- Responds to harmful and biased behavior:
- Will sometimes respond to harmful instructions or exhibit biased behavior, despite actions taken to prevent this. They use the Moderation API to warn or block certain unsafe content but still experience false negatives and positives.
- Steep learning curve:
- Extensive knowledge of how to implement the AI into your existing test automation process is required.
Examples of Using ChatGPT for Mobile App Testing
Creating Test Cases
Generating test cases can be a tedious process, requiring time and resources to create an array of test cases. With ChatGPT, testers can rely on the AI to generate test cases quickly by giving natural language descriptions. The AI bot can also come up with multiple scenarios and test cases to cover various app features.
When prompted to create test cases, ChatGPT will create an in-depth plan, with a list of test steps, expected results, required test data, preconditions, and postconditions. Make sure to give ChatGPT details on your app’s functional components and their purpose before asking it to generate a test case.
Example: Create a test case for logging into the automated testing tool Sofy.ai.
Prompt: “Generate an automated test case for logging into Sofy.ai.”
ChatGPT results:
And just like that, ChatGPT created an in-depth test case, with a list of test steps, expected results, required test data, preconditions, and postconditions. For complex test case generation, just be sure to give ChatGPT details on your app’s functional components and their purpose before asking it to generate a test case.
Write Test Scripts
You can also ask ChatGPT to generate test scripts based on natural language descriptions and ask it to produce test scripts using your preferred app testing tool and programming language.
Example: Write a test script for logging into the automated testing tool Sofy.ai.
Prompt: Write a test script for logging into Sofy.ai using Python.
ChatGPT results:
You’ll see that ChatGPT provides a Python sample script to automate the login process for Sofy.ai. It also recommends automated testing tools you can use (Selenium, PyTest, or Unittest) to run and manage test scripts and its resulting data.
Code Review
You can ask ChatGPT to review your test scripts or code and provide recommendations on how to make improvements. The following are some areas in which you can ask ChatGPT to review your code:
- Ensure it follows a consistent style and adheres to any style guidelines or best practices.
- Identify areas in the code that can be modularized and reused.
- Review test suite organization to ensure it follows a logical structure.
- Ensure test assertions are correct, clear, and cover all necessary validation points.
- Review code to ensure it handles potential errors or unexpected conditions.
- Identify areas in code that can be optimized or made more efficient.
Example: Ask ChatGPT to review a test script code for logging in to Sofy.ai.
Prompt: Review the following code and identify if there are areas that need improvements:
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys # Initialize the WebDriver (you need to provide the path to your ChromeDriver executable) driver = webdriver.Chrome(executable_path='path_to_chromedriver') # Navigate to the Sofy.ai login page driver.get('https://www.sofy.ai/login') # Find the username and password input fields username_input = driver.find_element(By.NAME, 'email') # Update 'email' with the actual name attribute password_input = driver.find_element(By.NAME, 'password') # Update 'password' with the actual name attribute # Enter valid login credentials username_input.send_keys('your_username_or_email') password_input.send_keys('your_password') # Submit the login form password_input.send_keys(Keys.RETURN) # Wait for the login to complete (you may need to adjust the sleep duration) import time time.sleep(5) # Check if login was successful by verifying the dashboard page URL or any other element if 'dashboard' in driver.current_url: print('Login Successful') else: print('Login Failed') # Close the browser driver.quit()
ChatGPT results:
The ChatGPT results first point out areas that could be improved in the code, like how it handles errors, web page sleep duration, and more. Finally, it provides an improved version of the code that addresses the listed issues.
Generate Test Data
An important part of test automation is generating test data that encompasses various user scenarios and edge cases. You can use ChatGPT to generate test data for your app by providing the input parameters and their expected output. Additionally, you can ask ChatGPT to format this test data into a table if needed.
Using ChatGPT to generate test data reduces the reliance on manual data creation and increases the efficiency of testing. It might also be helpful to provide ChatGPT with your specific testing requirements if you require more targeted test data.
Example: Generate test data for testing logging in to Sofy.ai.
Prompt: Generate test data for logging into Sofy.ai. Format the data into a table.
ChatGPT results:
With these results, ChatGPT includes both valid and invalid credentials that cover a variety of test cases and edge cases, including long usernames and security testing.
Analyze Test Results
You can use ChatGPT to analyze automated test results and summarize your test results in natural language. You can populate your test results in ChatGPT and ask it questions like “Can you summarize my test results?” or “Why did my test fail?” Or you could simply ask the bot to analyze your test results to get a more general analysis of your results.
Example: Analyze several test runs provided in an XML document.
Prompt: Analyze the following test results:
<?xml version="1.0" encoding="UTF-8"?> <test_result> <build build_id="31" job_id="Test-REST" server_id="12345678"/> <test_runs> <test_run started="STARTED_TS" status="Skipped" duration="14" name="bandTestA" class="BandTest" package="com.mycomp.devops.demoapp" module="webapp"/> <test_run started="STARTED_TS" status="Passed" duration="0" name="bandTestB" class="BandTest" package="com.mycomp.devops.demoapp" module="webapp"/> <test_run started="STARTED_TS" status="Passed" duration="1" name="bandTestC" class="BandTest" package="com.mycomp.devops.demoapp" module="webapp"/> <test_run started="STARTED_TS" status="Passed" duration="1" name="bandTestD" class="BandTest" package="com.mycomp.devops.demoapp" module="webapp"/> <test_run started="STARTED_TS" status="Passed" duration="0" name="bandTestE" class="BandTest" package="com.mycomp.devops.demoapp" module="webapp"/> <test_run started="STARTED_TS" status="Passed" duration="1" name="always_true_A" class="CalcsTest" package="com.mycomp.devops.demoapp" module="webapp"/> <test_run started="STARTED_TS" status="Passed" duration="1" name="always_true_B" class="CalcsTest" package="com.mycomp.devops.demoapp" module="webapp"/> <test_run started="STARTED_TS" status="Passed" duration="1" name="always_true_C" class="CalcsTest" package="com.mycomp.devops.demoapp" module="webapp"/> </test_runs> </test_result>
ChatGPT results:
In these results, ChatGPT provides an analysis of the test results, including the build info, test run and test run details, and the success of the test run.
Best Practices
Here are some tips you can use to overcome any challenges you might face with using ChatGPT for test automation:
- Avoid sharing sensitive info and data.
- Be specific with your keywords and define your desired output.
- Consider possibles biases by ChatGPT.
- Provide context and background info.
- Learn and focus on ChatGPT’s methodology to get appropriate responses.
- Provide feedback if you receive unsatisfactory or incorrect responses.
Incorporating ChatGPT into your mobile app automated testing processes can offer significant advantages, but it’s crucial to be aware of its limitations. With careful implementation and adherence to best practices, you can harness ChatGPT’s natural language processing capabilities to improve the efficiency and effectiveness of your mobile app testing endeavors.
If you’d like to learn more about how AI can improve your mobile app testing experience download our free guide.
