Just press a button and suddenly every part of a new Software Application will be tested and any issues or Bugs will be uncovered. That has been the goal of every Software Development team since the 1950’s.
The reality is a little different, and frequently you have to have a combination of both Automated and Manual Quality Assurance Testing done. In this Blog Article we will discuss different types of QA Testing, and when each one is appropriate.
Testing is the way we can measure the quality of the software, and the results of testing activity represents meaningful information for the team because learning about errors in the project is the better way to improve the process in future projects.
Prior to Automated Testing tools being developed, having a Quality Assurance Engineer manually test the application was the only way to detect Defects or Bugs in an Application. Naturally, this is a labor intensive effort, as you need a person to work with the Application or Module and write down their findings.
Sometimes the number of possible test cases could be infinite depending on the software under testing, and apply random test cases is not efficient, however there are some test strategies (risks based testing, objectives based testing, regulation based testing, etc) and test techniques like boundary value analysis, decision table testing and test transition testing (for black box testing).
The most critical thing with Manual Testing is that the QA Tester has to be extremely detailed in recording what they were doing within the Application and what specifically happened if it Failed or had some other Undetermined event occur. So, it takes time. Because without this information, the Developers won’t be able to reproduce the issue or troubleshoot it, and that will cause a lot of additional work and time.
In addition, there are times where Manual Testing is the best approach.
These QA Test Cases follow what a User would normally do within an application. Thus, we are testing what the User is supposed to do, which is why it is often referred to as the “Happy Path.”
This involves testing where we are trying to replicate a User who goes off of the “Happy Path,” and does something totally unexpected. Examples could include:
It is amazing how many Software Companies do not do this type of testing, or enough of it. And because of this every once in a while you will run into Applications or Websites which simply don’t work as you would expect. Because you have an Edge Use Case that doesn’t occur very often, and thus was never tested for.
Example
Who would think that a US Citizen would be living in another country and not in a US State, but another country's State or Province? And then they make entering a US State a required field and your address can’t be validated either. But this is a good example of an Edge Case, and yes this happened to me yesterday.
This is often done when you ask an Internal Beta Tester or someone not familiar with the Application to simply log in and try and use the application. By having someone unfamiliar with the application and not following a specific Test Plan, they can often find all sorts of things that anyone following a scripted test plan (or automated) might not find.
In essence, you are asking someone to act like a Gorilla and try and make the Application fail or have a problem. The key for doing this type of testing is to make sure the Tester is very detailed in terms of what they did, what did they enter, and what exactly happened when they discovered a problem (or perceived problem).
This is important, because since they are acting like an untrained User, they may receive messages or errors, which are actually correct. But since they are not familiar with the Application or the business use, they may not understand this.
Everyone will say that User Acceptance Testing is absolutely required, because without this you don’t get the sign off for the Release. The biggest problem is how do you get your Subject Matter Experts (SME’s) or Users to actually spend any time testing the new Application.
The Business Executives / Sponsors for the project need to force this. Because otherwise what you will get is only partial testing and by only a few individuals. If the Business Sponsor does not make this a priority, then any of the SME’s won’t make it a priority either.
Automated QA Testing can take many forms. In every case you are going to use some form of a Testing Tool or Suite to first program your test and then run the test against it. Some of this work is traditionally done by Software Developers to test discrete components, while other automated testing of the full application is performed by QA Engineers.
When a Software Developer writes code, they should normally start with writing a short script to test their code. This way they are determining in advance what the code should actually do. And then once they write the actual code, they can run this script against it to ensure that it does what they expected it to do. This is known as a Unit Test, as you are only testing a small portion of the overall Application.
This process is extremely important within Software Development, because the earlier that we can catch errors or Bugs in the software, the easier and cheaper it is to fix. Unfortunately, many of us have seen examples where Developers don’t do this and Bugs make it into production.
Real World Example
Recently I was working on a project which had two different development teams. Unfortunately, the original team would be what we call “Cowboy Coders.” They did zero testing of their own code. The focus was completely on doing things quickly. This led to a lot of production defects and Bugs. There were even “Stubs” that were left in the production Application that actual Customers were using.
So, as a User you might click on something, but nothing would happen. In other cases, when you clicked on something it would crash or give you an unexpected result. If the Developers had taken just a little more time, they could have caught more of these problems. Of course, it is just as important for the Business to understand the need and purpose for performing Test Driven Development.
Behavior Driven Development or BDD has a different approach, because it allows the team to write first the test cases in a human language that can be easily understood for all technical and non-technical staff. This process could be used for Integration and End to end testing.
BDD is a best practice when it is used by implementing automated testing, is an Agile process usually applied by the QA team because it is not focused on the code of the software, instead of that it is just focused on the behavior of it.
This is an example of how to implement BDD:
Scenario: Login to “Testpage”
Given I open browser
And I go to testpage.com
When I add correct credentials
And I press submit button
Then I should be correctly logged to testpage
Automated Testing Tools have been in existence for decades and work to varying degrees. Most of them require a QA Engineer to create a recording of what exactly needs to be tested. And then the QA Engineer provides the Testing Tool with various parameters to use during the tests.
This allows the QA Engineer to run a suite of tests multiple times, using various parameters. And the Testing Tool can capture what errors or Bugs it found during the test. The major benefit is that once you have defined the various Tests you can run them thousands of times and in a fraction of what it would take a person to do the same test. The challenge of course, is in programming the Testing Program, so that the logic makes sense and that it is correct. This takes time and effort, which is why it is important to have a highly trained QA Engineer performing this work.
In the future, much of this work will be performed by Artificial Intelligence systems that can dynamically learn about an application and test things on their own without so much programming. However, we are years or decades from achieving this. Until this happens your team needs to pick a QA Automation Tool that they can use and apply to those areas of the Application that are not changing frequently. Where they can design a standard Test Plan and run it over and over automatically.
Regression Testing is a fancy term for saying that we are going to test the entire Application to ensure that everything works correctly and more importantly that we did not break something. As Information Technology professionals we always have to remember and keep in mind the “Law of Entropy” which is the 2nd Law of Thermodynamics.
The more we build a system up and the more complex it becomes, the greater the chance of a failure.
This is very true with extremely complex Applications with millions of lines of code. Make a simple change and everything can break.
So, it is important before any major or minor release that we run a full Regression Test to ensure that things are working correctly. And the easiest way to do this is to use an Automated Testing Tool. I cannot emphasize enough how important this process is.
Real Example
While working at Universal Technical Institute, we needed to make a very minor change in a specific field in our main database. Just changing it from a 255 character text field to 512 characters. Everyone, including myself thought oh this is no risk, since it is a text field.
Oops!!!
We didn’t do a full Regression Test because of the assumed non-risk. What we didn’t know was that this particular Text Field was being used as a calculated field in Crystal Reports. And in Crystal Reports anytime you go over 256 characters, it becomes a Memo data type and can not be used in calculations. So, by making this simple change we broke 2,500 Reports that the business used every day. It took us hours to figure out what exactly had happened. But lesson learned - the hard way.
Another example of Automated Testing is to use a tool to do a Stress or Load Test against the new Application. You want to do this, when you have an Application that is going to be used by a high volume of users - thousands or millions.
This is especially important for an Application where you may have extraordinary peak loads during the year or even at specific times during a day. And the only way to test this is through automation, having a system send thousands or millions of requests to the Application, in order to see at what point it actually fails.
The question you are trying to answer here is does the application fail at only 100 users? Or a million? Or a billion?
In this Blog Article we have discussed the differences between Manual Quality Assurance testing and Automated Testing, and why you would use each.
We have also covered why both are important, although as AI starts to get better every year, expect that more and more QA testing will become automated. Still in many cases you will need a qualified QA Engineer to train the system in what to do.
We hope that you have enjoyed this article, to learn about User Acceptance testing in quality assurance, click here.
Thank you, David Annis with Maria de la Rosa, QA Engineer.
