Test Automation

What can I do with BDD?

Behavior driven development (or BDD) is an agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project. It was originally named in 2003 by Dan North^[1] as a response to Test Driven Development, including Acceptance Test or Customer Test Driven Development practices as found in Extreme Programming. It has evolved over the last few years^[2].

On the “Agile Specifications, BDD and Testing eXchange” in November 2009 in London, Dan North^[3] gave the following definition of BDD:

BDD is a second-generation, outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well-defined outputs, resulting in the delivery of working, tested software that matters.

BDD focuses on obtaining a clear understanding of desired software behavior through discussion with stakeholders. It extends TDD by writing test cases in a natural language that non-programmers can read. Behavior-driven developers use their native language in combination with the ubiquitous language of domain driven design to describe the purpose and benefit of their code. This allows the developers to focus on why the code should be created, rather than the technical details, and minimizes translation between the technical language in which the code is written and the domain language spoken by the business, users, stakeholders, project management, etc.

http://en.wikipedia.org/wiki/Behavior_Driven_Development

In theory the purpose of BDD is noble. However in practice I saw that it fails very often. Actually I heard no real success story in BDD applied to automation.

Applied to test automation BDD approach is understood as allowing automation developers to develop framework and QA engineers to create automated tests by writing domain language.

Good aim, but what results?

On one project guys used robot framework to wrap already existing python framework, because PM wanted QA engineers automate test cases.

After implementation of Robot Framework was done, QA engineers refuse working with it. So automation developers should continue automating test case, but now not in python, but on domain specific programming pseudo language.

I understand QA engineers, that refused to create tests in domain language, because that language is actually a programming language, but with another syntax. And in complex tests it does not seem natural, but as just another vendor script.

Also, now you can’t debug whole test case, because it is written in “natural” language(actually new script language) and none thought adding debug function for it.

You can debug runner, however, runner will be wrapping that domain language and you can access to test methods in debug mode only if runner is a part of test suite. It make debug much more complicated and thus make test suite much harder to maintain. And if maintenance in automation fails, then automation fails as well.

So the questions are:

How did you solve problem with debugging on your project with BDD approach?

Where QA engineers eager to write test cases in domain language?

What time QA engineers need to stop asking questions if some keyword exist?

What time QA engineer need to stop asking questions (if they write test case in wrong syntax)?

What time it took for QA engineer to update test case in comparing with update of TA developer?

 

 

 

While implementing and managing automation efforts on various projects I encountered lots of problems connected not with technical issues, but rather organizational.
Every experienced test automation developer knows, that the most important problem in UI automation is automating actually, but supporting test suite in order it to be up to date to AUT.
Most of the modern automation framework and tools address this issue by allowing TA developer to create a map of object identification(AKA windows declarations). SilkTest, QTP, TestComplete, Rational Functional Tester and others have such ability implemented in one or other way. Also in other automation tools, that have no such functionality(like selenium RC, AutoIT, SWTBot, etc.) this functionality is easily created in frames of automation project by automation developers.
Object mapping, custom or self made, greatly decreases time of updating suite to new UI.  My estimate is that time decreases for 50-80%. The bigger the suite, the more effective is mapping.
However, even 80% is often not enough for project, when UI changes very rapidly.
Some time ago I was invited to analyze one automation project. Guys were doing web dev and also making UI test automation. But they feel that they are going wrong way and something could be done much better. It was first time they tried UI automation and they made frequence mistake underestimating complexity of test automation. They thought it would be like making unit tests.  But after few four months automation they realized, that most of the project(test-suite) is   not up to date and they just have no time to do it.
So, they invited me to investigate situation and tell them what they have to do so there test suite will help them rather then slow down. I made several calculations, find out time, need to test all, that was automated manually, then time, that  is needed to update automation suite and average changes in UI after each build and result was, that they should spend 10!!! times more time to update suite then to test all same that covers test suite manually!
What does it mean? It means that on some projects UI test automation will not be effective. Huge UI changes, strong requirements to time of test suite update, bad test suite architecture, small test suite size, no requirements to UI developers.
Some of this issues can be solved easily, some require lot of efforts and some times can’t be addressed.
While thinking past about projects with often changes of whole UI I recalled ways to fix that issue: the main idea is to decrease amount of UI changes, restrict doing it by UI developers or force UI devs to update mapping after UI changes as they update unit tests in case of changes in functionality.

Common ways:

• make UI first, then make functionality;
• regular UI freeze before testing;
• creation special file, where UI identification will be written and updated by UI developers, after that automatically used by automation suite.

Make UI first, then make functionality
•    Advantages: We seriously decrease number of UI changes, even can eliminate them
•    Disadvantages: You have also to agree that with UI developers, who are usually stubborn, do not like anything that limits on their work and often think that their work is more important that work of the others.
•    Summary: hard to implement due to conflict of interests
Regular UI freeze before testing
•    Advantages: We seriously decrease number of UI changes, can update test suite before build
•    Disadvantages: You have also to agree that with UI developers, who are usually stubborn, do not like anything that limits on their work and often think that their work is more important that work of the others.
•    Summary: hard to implement due to conflict of interests
Creation special file, where UI identification will be written and updated by UI developers, after that automatically used by automation suite

• Advantages: We link responsibility of test suite crashes due to UI changes with those who do those changes. This allows us to decrease number of UI changes to necessary and big part of identification will be updated simultaneously.
• Disadvantages: It does not make a big difference with updating of windows mapping itself, because if UI dev do not update something, then you should tell them to update and it will take more time, then update UI declarations. On the other hand it will be their responsibility to update identification and problems will be treated as personal fault. In case of bug UI changes both file and mapping should be updated(double work).
• Summary: this approach often is proposed by UI dev team because this file can be generated or they do not want to learn automation mapping usage.

I thought about idea of creating a way to reduce UI changes in such way, that it would not have strong objections from UI developers and will have same effectiveness as code freeze, etc.
The way is in creating separate branch for the project where all UI changes should be committed.
There is no restrictions on doing UI changes result in creation new controls without altering old.
But if UI dev want to change identification of already existing control, then he should commit it to separate branch.

• Advantages: after every commit in trunk everyone can check code with automated suite, which is impossible if in all other cases except UI code freeze. TA devs can concentrate on developing test suite instead of constant updating because of frequent changes. We will know about branch merge long before it and will have time to plan activities. We can add tests that validate control existing on the page and be automatically executed after each commit(this will alert UI devs if they change something unintentionally).
• Disadvantages: supporting different branches is additional effort for AUT dev team, even it is much less hard to do it then to do UI freeze or make UI first it will create resistance between UI devs.
• Summary: way is better then others but also require good understanding of automation needs by project manager and ability to implement strong decisions

What way do you use? What way do you prefer?

Often, when professionals talk about their area they you specific words, that describe complex processes, activities, items or artifacts.

Test Automation is not an exception. It has its own terminology and conventions. Some of them are common development terms and conventions, some came from business area – QA area, and some are automation specific only.

Automated Testing – testing of a product not manually but via other programs

Architected Automation Approach – a way of design ATS architecture, that includes lot’s of abstraction layers in order to improve maintainability and

AUT (Application under test) – product, for which automated test suite is developed

Automation Framework (AF) – framework that supports creation of windows declarations and UI layer of ATS architecture

ATS (Automated Test Suite) – set of automated test cases for testing some product

Automated Test Case – piece of code or executable file that run test on AUT and verifies test case pass or fail

Automated Test – see. Automated Test Case

Business layer – layer in n-tier ATS that is responsible for abstraction of Business logic

Coverage –percentage of tests in test plan that were automated

Data driven automation – ability of automated test to read data from external sources and thus execute same test cases multiple times on different data sets

Detailed test cases – test cases documentation, where steps are written in so detailed way, that they can be executed by person without QA background (e.g. TA dev). Used to facilitate automation process and make more easy communication between TA dev and TA QA engineers

DB layer – layer in n-tier ATS that is responsible for abstraction of interaction with database, like ORM

Keyword Driven Automation – automation architecture, based on

P dev (product developer(s)) – developers, who develop AUT, that will be tested by ATS

QA Engineers – quality assurance engineers that can test application manually or use ATS for it

Night Runs (night executions or night testing) – preparing environment and launching ATS for execution each night in order to improve ATS reliability or just to get test results

Reliability testing – is continuous repetitive launches of ATS that has aim to catch all bugs in ATS and make it produce less false fails and errors.

Regression Testing – is any type of software testing that seeks to uncover software errors by partially retesting a modified program.

R&R (Record and replay approach) – ineffective automation approach in automation, widely advertized by tool vendors and often used by unskilled beginners

Test Suite architecture – architecture of test suite, how abstraction layers are organized, how test cases are organized and executed, how test data is organized and retrieved

Test Automation – process of ATS architecture design, automating tests, bug fixing and support of ATS.

Test Automation Solution – solution that is able to execute ATS, generate reports, end user- QA engineer

TA QA – test automation quality assurance engineer, who’s role is to run ATS, verify bugs and assign to TA dev or p dev

TA dev (or just dev) – developer, who develops ATS

Test Harness – see. Test Automation Framework

Windows declarations –classes in UI layer that represent AUT windows. They improve ATS maintainability by making easy to update ATS in case of changes in UI in AUT.

Windows Mapping (mapping) – representation AUT UI in easy editable order, that improves maintenance of ATS

UI layer – layer in n-tier ATS that is responsible for abstraction of UI logic (windows, tabs, panels’ representation)

Also here is information about logging messages.

Log messages/statuses –

• Info – information log, represent test case step, main purpose of logging is to have steps to reproduce a bug;
• Pass – log, that informs about verifications that passed successfully
• Warning – log that informs about different statuses in ATS or AUT that require attention and interaction of a (QA, ta dev, p dev). E.g. if ATS can’t identify if it is bug or not, if part of ATS is not implemented or part of AUT functionality is not implemented yet.
• Error – log, that informs about verifications that passed unsuccessfully and thus requires QA attention for assigning a bug.
• Environment – log that informs about problems with environment that do not allow ATS to work correctly or AUT to start/be used correctly.

Of course this list is not final. I will update it if I will recall something missed. Also I could alter it if I found that some terms that I use have by standard another meaning and I like it. )))

Also if I will see in comments questions like “What is BLA-BLA-BLA?” then I will try to add such words to this list.

However I will not try to add all of the IT professional terms here, it will make usage of this list much less useful.

So, please use terms list in case you want.