Generations of Test Automation Frameworks- Past and Future

January 25, 2019

Generations of Test Automation

In the first publication from the Getting Started Series, we talked about what is a test automation framework and discussed all related terms. Here, we will continue the topic with the FIVE generation of test automation frameworks. They are tons of articles discussing the different types of frameworks. I am not going to give detailed examples. I am going to try to provide you with my point of view why they appeared in the context of the technologies of their time. Also, I am going to connect their evolution with the evolution of the QA engineer profession. At the end of the article, you will know the history of the different types of frameworks and for what you need to look into the future to be up to date.

Generations of Test Automation Frameworks

Before you begin to automate your project you need to know the requirements- what resources do you have, what you need to automate, which are the supported platforms, etc. However, there are different types of types of frameworks and you need to pick carefully one type of them. There many questions you need to answer to complete the process successfully. Here are some of them:

  • How technical are the QA engineers involved?
  • Who will be responsible for maintaining and developing the framework?
  • Do you have someone in your team with lots of experience in designing frameworks?
  • Do you have support from the DEV team?
  • How complex are the functionalities you need to automate?
  • Are the requirements of the applications stable? If no, how often will they be changed?

These are only a few of the questions you need to ask. I will create a separate article about them, discussing everything in more details. My point here is that besides this questions, knowing the history and the context of the different types of test automation frameworks will help you to choose. Especially for new projects you don’t want to live in the past. There are almost no companies developing software waterfall and not using Agile methodologies or embracing continues integration and continues deployment. Also, as we are going to talk the new generations of automation QAs are/should be really good programmers. Sorry, for writing “good” test automation you should be technical, whoever tells you something else is lying or trying to sell you something that costs lots of money.

1st Generation- Record and Playback

I cannot even call this type of automated tests a framework. Since for me a framework should be something almost entirely coded. Some people call this type of automation- Linear Scripting Framework.

Testers don’t need to write code to create functions and the steps are written in a sequential order. In this process, the tester records each step such as navigation, user input, or checkpoints, and then plays the script back automatically to conduct the test.

The most famous example that I can give is Selenium IDE.

Pros and Cons

  • You don’t need to write custom code.
  • You can really fast automate some simple scenarios.
  • Maintaining a large number of these tests is almost impossible.
  • It is a big challenge to automate more complex scenarios.
  • It is a headache to integrate these tests in CI process,get proper reporting, configure it for different working environments.

Context- QA and Technologies

When these tools appeared for the first time most companies were just starting their transition from Waterfall to Agile. Back then there were only a few QA engineers, even if this was the correct name. So most of the testing was manual since the technologies were simpler. The whole web boom was just starting. So, not surprisingly most of this tools support web record and playback preferably. Later appeared similar tools for Desktop automation such as Test Studio. However, since the releases of the new versions didn’t happen so often, it was cheaper to hire more non-technical people and test the apps manually. This is why the coded solution for these folks was not going to work since most of them don’t know how to program. With the later generations, you will see that this started to change and they had to learn new skills to keep their jobs.

2nd Generation- Modular and Data Driven Frameworks

Modular Based Testing Frameworks

In record and playback scripts the data was hard-coded and more complex scenarios were almost impossible to write. This is why most vendors and open source tools started to support export to code option. Where your recorded test is exported to a popular programming language where you can edit it and modify it.

Again the most famous example is with Selenium IDE where you could export your test to Selenium Backend code and later to WebDriver.

Data Driven Testing Frameworks

Since hard-coding the data in the tests was limiting the combination of different steps and making the creation of new tests harder and harder- the data driven frameworks appeared. I think They are just a layer on top of other types of frameworks such as module or library based.

Data Driven Testing Framework helps the user segregate the test script logic and the test data from each other. It lets the user store the test data into an external database. The external databases can be XML files, excel files, text files, CSV files, ODBC repositories etc.

Pros and Cons

  • Since these frameworks are coded, it makes the maintenance a lit bit easier since you can use the power of programming IDEs to fix the tests.
  • Significantly increase the reuse of logic.
  • Since most of the code is auto-generated, you have to rewrite it.
  • The maintainability of the code is still low since the abstraction level is minimal- leads to lots of copy-pasting.
  • With the introduction of data-driven frameworks, the debugging and troubleshooting got harder.
  • Also, since the test logic and test data are separate, the tests readability is worse.

Context- QA and Technologies

With the adoption of Agile, the projects started to get bigger and more complex. People began to feel that the test automation is something can help them to keep the pace. But as we discussed because the applications got more complex a better way of writing tests was needed.

QA folks started to learn the programming languages of their programmers and began to export tests with the hope that with minimal editing can automate something. The introduction of data-driven tests helped to automate things even faster. But the same as completing a new project, in the beginning, everyone tries to finish it no matter what. Which means that nobody knew how hard it would be to keep up with this pace and maintaining the tests. With time the resources for maintaining the tests went bigger and bigger- taking people more time to support the old scripts than writing new ones.

3rd Generation- Library and Keyword Driven Frameworks

Library Architecture Testing Framework

The Library Architecture Testing Framework is fundamentally and foundationally built on Module Based Testing Framework with some additional advantages. Instead of dividing the application under test into test scripts, we segregate the application into functions or rather common functions. The basic fundamental behind the framework is to determine the common steps and group them into functions under a library and call those functions in the test scripts whenever required.

For example, during my time as QA Architect in Telerik, we had back then 3 big websites with 3 shopping carts. As you can imagine, the fastest way for the developers to implement the 2 additional shopping carts was to copy paste the code change only the layout. So, basically the whole workflow stayed the same only the front-end looked different and the web elements had different locators. Back then we had something like a module based testing framework. So, we decided that it will take us much time to maintain the same tests for 2 additional websites. So, we upgraded our module based test framework to library framework- abstracting the whole workflow using abstract facades. Page objects are significant part of this type of framework.

Keyword Driven Testing Framework

In a keyword-driven framework, each function of the application under test is laid out in a table with a series of instructions in consecutive order for each test that needs to be run. In a similar fashion to the data-driven framework, the test data and script logic are separated in a keyword-driven framework, but this approach takes it a step further.

In the table, keywords are stored in a step-by-step fashion with an associated object, or the part of the UI that the action is being performed on. For this approach to work properly, a shared object repository is needed to map the objects to their associated actions.

Number Description Keyword Object
Step 1 Click login link on home page clicklink //*[@id=”menu-main”]/li[3]/a
Step 2 Enter email inputdata id=”email”
Step 3 Enter password inputdata id=”pass”
Step 4 Verify login button enabled verifybutton class=”loginBtn”
Step 5 Log in to the app login anton@gmail.com,12345

Pros and Cons

  • Though the library based frameworks much more test code can be reused.
  • The fixes start to happen in a single place.
  • With time if there are slight modifications of the workflow in some of the apps, it gets harder and harder to keep the abstraction good. Writing more abstract code requires engineers to have more in-depth programming knowledge. Because of the higher abstraction, the tests get even harder to read.
  • Keyword driven frameworks had to ease the tests writing process however they made the maintenance even harder and slowed down the new tests development since the people who write the tests are not the ones responsible for maintaining the keyword repository.

Context- QA and Technologies

The library based testing frameworks were the natural continuation of the module based frameworks. Since the application started to get even more complicated and the release cycles were shortened significantly. So, it became clearer that to keep up with the new functionality development and keep maintaining the old tests, something had to be done. So, a small step forward for faster tests creation and better maintainability were these frameworks.

On another hand, there were still many companies that hadn’t embrace the test automation and continued to hire not technical QA people. However, at some point, they realised that to cope in the new world they have to start embracing the Agile and creating automated tests. However, until their manual QA engineers learn the new required skills something had to be done, this is how the keyword driven testing frameworks appeared. Developers got involved to help, building the “core” keyword repository and providing the QA engineers with a way to write tests in a non-technical way. This way was okay ten years ago, but I don’t know a single company that continues to do this and has healthy test automation.

4th Generation- BDD and Hybrid Frameworks

BDD Testing Frameworks

Behavior Driven Development frameworks allow automation of functional validations in easily readable and understandable format to Business Analysts, Developers, Testers, etc. (or at least this is the marketing message) Such frameworks do not necessarily require the user to be acquainted with the programming language. There are different tools available for BDD like Cucumber, SpecFlow, Gauge etc.

Hybrid Testing Frameworks

As the name suggests, the Hybrid Testing Framework is a combination of more than one above mentioned frameworks. The best thing about such a setup is that it leverages the benefits of all kinds of associated frameworks.

Pros and Cons

  • A hybrid framework can be more easily adapted to get the best test results.
  • Not so readable tests, not so easy to maintain and troubleshoot.
  • Usually built to support only a single platform or technology.
  • BDD frameworks have the same pros and cons as the keyword driven frameworks the only difference between them is that the BDD markups are integrated more tightly in the programming IDEs.

Context- QA and Technologies

Hybrid frameworks or the so-called “custom” testing frameworks started to appear in almost every company. This is maybe the most popular approach nowadays which is entirely understandable. As pointed before, most projects are altogether different, so there couldn’t be a general tool or framework that can automate them all without any kind of modification, this is why the hybrid testing frameworks are so popular.

BDD appeared a continuation of data and keyword driven frameworks. There are still companies with a significant amount of not technical people. It is believed that business people will write tests, but in practice, nobody does that. Just people continue to ignore the fact that this is an unnecessary abstraction that slows down drastically the development time and makes the maintenance of the tests much harder.

4.5th Generation- Selenium Based Frameworks

Selenium Based Testing Frameworks

Until now all testing framework types were more or less focus on how we write the tests. Some people decided that to make the framework “core”, it will be good to share it with the rest of the world and open source it. You can find plenty of examples- Selenide, Atata Framework, EPAM JDI or Objectivity. You can base or library or hybrid framework on them and get some core benefits such as- more readable tests, less brittle tests (usually they include built-in logic for waiting elements), a little better tests maintainability. The primary focus of these frameworks is web automation. You are free to add over them any abstraction you want- keywords or BDD.

Pros and Cons

  • I believe that the open sourced Selenium based testing frameworks are great. Top recognised automation QA engineers build them. So, for sure it is much better to base your hybrid framework on them then reinventing the wheel.
  • Built primarily for web, if you need a solution for API, mobile or desktop you had to create it yourself.
  • All of these frameworks lack proper documentation. If you have a problem in some cases, the code is complicated, so you need to submit an issue. But since these guys are developing the tools in their free time, you don’t know when they will respond.
  • Almost all of these frameworks only wraps WebDriver making the tests more stable without giving so many new features.
  • This is the case since these guys are doing it mostly in their free time, they don’t have the time to make significant development or spend much time in thinking how to make the API more useful, or write unit tests.

Context- QA and Technologies

Since the website automation has been here for the longest, this is why the open source “core” Selenium based testing frameworks appeared. If you check their websites or GitHub pages, you will see that outsourcing companies build them. Their goals is, of course, winning more customers by promising that the projects they create for them will be well tested and that at the end of the project they will get the framework + all tests. Another part of these frameworks is custom frameworks built for some private company that some of their QA engineers want to popularise and show how good he is. There are benefits for the company as well since some new features are added to the framework without paying to someone to do it. And the third type is where someone in his free time decided to build an open source test framework but he continues to work for another company. This is like a hobby project. These folks earn some cash by providing support to some of the companies that decide to use their free tool. Everyone loves free, but sometimes the cheap stuff is much more expensive. (we will talk about that in another article).

5th Generation- Rise of AI, Full-Stack Frameworks

AI Tools- ‘Fake?’ Future

Nowadays, “AI” is a hipe word. Everyone that wants to sell you something includes it in front of their tool or in their marketing materials. There are promises for systems that can write automated tests automatically, codeless tests or effortless/almost 0 tests maintenance. I cannot say that all of it is bullshit there are some decent tools such as- AppliTools (they build tools for visual testing). These tools will have their place in future but they can solve just some edge case scenarios and won’t be able to replace the coded tests.

Full-Stack Test Automation Frameworks- True Future/Present

So, considering the mentioned context what should be the best 5th generation testing framework?

It should not be a hobby project or side project of some outsourcing company. It should be professionally built and tested. For me, test automation product should be treated as a product. Otherwise how you can trust it and rely on it?

Since building a framework is something interesting everyone even people with less experience want to build one. I get it. Nobody wants to have a boring job. But don’t worry no matter which generic framework or tool you use, you still need to integrate it. However, I firmly believe that the toughest part of the test automation framework should be built by an architect that read tons of books about designing APIs, high-quality code and automated tons of applications.

Must have features

  • Documentation – The new generation frameworks should be well documented.
  • Fully tested – The new generation frameworks should be fully tested with all kinds of tests. (which for me questions the whole open source thing)
  • Fully coded – The new generation frameworks should be fully coded- no more keywords, sentences, etc. QA folks should embrace their destiny that they will have to learn to code.
  • Support different platforms – It should be designed to support different platforms- web, mobile, desktop and API, all kinds of resolutions, browser versions, etc.
  • Cross-platform – It should be cross-platform- you don’t have to limit the user which platform they use- Mac, Linux or Windows.
  • Parallel testing – Parallel and distributed testing are hard to implement so this generic framework should give this by default to people.
  • Coding Standards – This is no more art of amateur programmers, test automation frameworks should be treated as production code, so this 5th generation framework should enforce high-quality coding standards.
  • Extend and customize – One of the hardest things to develop is to allow these generic frameworks to be extendable and customizable. All of the mentioned ones lack this property which is crucial if you are going to integrate a framework written by somebody else. Every company and team should be able to modify it and extend it to fit their needs.
  • Troubleshooting – With the increasing tests count and complexity, it will be even more critical the tests be more maintainable. A significant part of this effort is the easier troubleshooting and better support for locating errors.
  • Starter kits – Documentation is not enough. I bet that most of you that have more than few years of experience in the field of automated testing and have a custom framework spent countless hours teaching new colleagues how to write “proper” tests using the team’s framework. I believe that there should be a more automated process allowing people to learn by themselves. I think one way to do it is to utilise the so-called getting started solutions or starter kits. Projects that have examples with explanations on how to write tests and why we use a particular method or not.

Do such frameworks exist? Yes. Some tool vendors already provide complete systems such as SmartBears and IBM. However, their tools don’t always conform to the above statements.

Being one of the best in the region, led me to consult many local companies regarding their automation efforts. In the process, we discovered that most of them needed such 5th generation test automation framework. Building such a tool is a complex task and takes lots of time, needless to say, that it won’t be possible to maintain different versions of the same thing for 10 different companies. This is how BELLATRIX Test Automation Framework was born. It took us almost 2 years to build, but I am pleased with the result. It is a 5th generation framework conforming to all of the above statements. Even if you are not in a search for something new, I still encourage you to try it and install the starter kit. I promise that you will get lots of new ideas how to improve your existing framework (even only reading the docs).

Pros and Cons

For me, AI frameworks are just the new “keyword” and “BDD” beliefs that you can write automated tests without programming. Trying to sell something to QA folks that still refuse to accept that they will need to learn to code. As mentioned they are some excellent tools that solve some particular problems but I haven’t seen any of them be sufficient enough to replace the hybrid frameworks.

The full-stack frameworks have countless benefits and are much better in many ways than the previous generation of frameworks and tools. However, there is always a catch. To use the new generation frameworks, QA engineers need to be much more technical. They need to find time to learn new programming languages and read even more about high-quality code, code design and architecture.

Even having the best generic framework that gives you all of the above benefits won’t solve many of the current QA folks’ automation struggles such as having a ‘clean’ test environment, preparing the ‘right’ test data, designing the ‘right’ tests, how to report the results in the best possible way (yes, I still believe that reporting is not part of the frameworks, you can integrate something but you should have a dedicated tool/system for it) and so on.

Context- QA and Technologies

With the rise of mobile development, microservices and cross-platform technologies the requirements for test automation frameworks got more sophisticated. Now it is not enough your website test is not enough to run okay on latest Firefox on Windows. You have to support the latest 5 versions of 6 different browsers, you have to test the layout of your website on various mobile devices. Also, most companies nowadays have Android and iOS native apps so your framework should be able to test them as well. In order, the application code to be reusable now all companies are shifting in using micro-services which are based on containers and APIs. So, you have to test the APIs as well. Most frameworks nowadays are based on the “standard tools” such as Selenium WebDriver, Appium for mobile, WinAppDriver for Windows desktop apps. Moreover, the period of rushing to write huge amount of automated tests is almost to its end. Nowadays, one by one people starts to realise that it is more important to have less but more stable tests. This is why, the new wave of test automation frameworks will have to answer to this need- providing better tools for troubleshooting, maintainability, readability and coding standards.

Summary

As you saw, there are many different varieties of frameworks. Because of the long past, many companies continue to support some legacy types of frameworks because it is too hard to rewrite them. Is this right or wrong? I cannot say. I know that time, and people are money. It is hard to fire people because they don’t have the right knowledge to support the future systems. As you read, almost during the whole time except the beginning there are always two high-level options- one with code and one without code. The second for the companies that I mentioned here at the beginning of the paragraph, full of people that still need to re-educate themselves. I cannot say that this is something wrong, it is as it is. This is just an observation.

If you have read more than one of my articles, you know that I firmly stay behind the idea of coded tests. And a big part of why I write these series of publications is to show you the future from my perspective. In my opinion, all people that what to do ‘proper’ test automation, have to start to educate themselves, learn to write high-quality code, stop being ‘amateur’ programmers and embrace the future.

Try Now Bellatrix

Build up to 100 tests using full product capabilities.

Download