Motivation: Our QA team is responsible for automation of test cases. As a daily activity, we are supposed to implement and automatically execute tests. At some point, we realized that we had additional tasks such as maintenance, refactoring and analysis of failing tests, which were blocking us from implementation stories. We had too many complaints in our retrospective meetings about this situation. Everyone was sick and tired of maintenance issues and team was not happy.

Problems: We accepted that we had to take these problems more seriously. We started to list the challenges, analyze root causes and discuss possible solutions. Test smells were one of our biggest pain points in our test automation project which caused instabilities in test results and created a huge maintenance burden.

Solutions: In this talk, we provide solutions that we developed to handle test smells and have a more robust test automation environment. They are lessons learnt from our experiences which are applicable to any type of test automation project or may give the audience an idea to develop their own solutions.

We start our talk by emphasizing the importance of test automation and the costs caused by the ignorance of QA. We discuss possible drawbacks and difficulties which are caused by test automation such as inconsistent test results, creation of huge load of test data and test dependency problems. Then, we describe the reasons behind those difficulties that lead us to define the test smells and analyze different test smell types. As our solutions are based on the real-life experiences, we provide insights about the real system under test which is a cloud-based open IoT operating system built on a microservice architecture. It is a big project developed by 600 people in 10 globally distributed sites which brings additional challenges to test process management. Finally, we provide our solutions that we developed to overcome different types of test smells. The solutions we cover in our talk are:

  • Polling mechanisms: For handling asynchronous microservice calls
  • Helper classes: For reusable test codes to decrease the maintenance effort
  • Clean up code: For deleting created objects to decrease the unnecessary load in the system and for eliminating test dependency problems
  • Test annotations: For controlling the order of test executions and group related tests
  • Test execution tools: For scheduling test executions and automated test results reporting
  • Test history: For detection of the flaky tests requiring maintenance
  • Wide test data: For including different values such as corner cases and boundary values
  • Static test code analysis: For increasing the code quality and preventing possible test smells

Results: While we elaborate on each solution, we use real life examples and before-after comparisons to provide concrete results to the audience showing the advantages of our solutions.

Summary: The main objectives of our presentation are:

  • To provide background information to the audience about the definition of test smell and common test smells types
  • To present a set of mitigating actions for test smells within the scope of automated testing of software systems built on microservices architectures
  • To provide real life examples and empirical evidence supporting the advantages of our solutions

To sum up, we develop solutions to overcome the challenges we faced in real life. We aim to create awareness on the audience about possible problems and preventive actions. After all, we think that this talk is of importance in terms of real-life experiences. Instead of theoretical claims, we share the challenges that we experienced and our solutions with before-after situations. We present graphics and evidences to understand the benefits. Instead of what to do, we go over how to do. Take-aways Proposed approaches can be applied by any organization by adapting according to the related work to achieve time and cost reduction. After the talk, key learning will be covered such as:

  • Attendees will be able to understand effects of tests smells in automation projects
  • Attendees will be able to implement ways to achieve robustness of test suite
  • Attendees will be able to reduce manual effort on testing
  • Attendees will be able to realize cost of bugs coming from production environment
  • Attendees will be able to implement some solutions to find bugs in early stages
  • Attendees will have insights for improvement of rapid automation and adaptation

Timeline:

  • 0 – 10: Introduction & Project Background
  • 10 – 20: Test Smells Definition & Types
  • 20 – 40: Solutions to Test Smells • Solutions against flaky tests: Polling algorithms, Test History • Solutions against coverage gaps: Test Data Management • Solutions against code quality: Helper Classes, Static Code Analysis, SonarQube
  • 40 – 80 Demos: • Basics of API & UI Testing: Postman, Test Frameworks, Selenium • Polling mechanisms: For handling asynchronous microservice calls • Helper classes: For reusable test codes to decrease duplication • Annotations, clean up code: For eliminating test dependency problems • CI/CD Practices: Gradle Tasks, Scheduling pipelines, SonarQube Integration • Test history: Report test results to Xray using API • Test Data Management: For including different values such as corner cases and boundary values
  • 80 – 90 Close & Questions