Architecture Posts

Ideas, Experiences & thoughts worth Sharing!!!

Category Archives: Open source

Behavior Driven Development (BDD) – Best Practices

BDD is a relatively new method for developing software, which is essentially a refined version of test driven development (TDD). Where the TDD uses more traditional tools in developing the tests (QTP, etc), BDD introduces a new paradigm of testing framework & tools. In BDD the tests are written using a language called Gherkin, which follows a high level “Given … When…. Then..” format to define the test (requirement) criteria.

This framework is widely being adopted now and there are tools support in java, .Net and open source technologies. While I had my reservations initially in its practicality, after using it for a few weeks (in a real world financial domain project), it became clear that there is actually value in adopting BDD.

One of striking aspect was the discipline required for it to be effective and re-usable in large enterprises. The discipline comes from well-established best practices and a governance model (ensure compliance) to support the framework & tools. This post doesn’t go into the technical framework and tools which I found mostly covered in a number of blogs. I intend to cover some of the best practices we have created and followed to make this as beneficial as it can be.

Best Practices

Best practices are guidelines that are applicable at a certain time point and can change over a period of time depending on the organization strategy, project context, etc. This section should list the current active best practices at any point of time.

Test pyramid


BDD supports automating component & UI testing. And as a general good practice for maintenance and future enhancement it is recommended that the test pyramid approach is followed in terms of number of tests and the coverage across the spectrum of automated tests. (Unit, Component (Api) and UI tests).



And other numerous web resources.

Example Not applicable.

Test Background


Test background should be limited only to any pre-requisite activity that doesn’t have any individual test specific action or data.

These would activities such as logging in, cleaning up or setting up base data (Not specific to individual tests).



Having test specific data setup likely to change for individual test cases, resulting in complex background feature statements.



Consider a feature (test pack) requiring keying in screens 1 to 5 with 5 difference combination of data (scenarios).

If for all scenarios screens 1-3 remains identical, there would be an urge to move this statement to the background of the feature, hence resulting in less duplication.

But this is not recommended as even though the steps appear identical they may be processing different data set and in future one of these screens might need different action for some of the scenarios.   This will result in rework of the entire feature rather than just modifying the impacted scenarios.

The duplication could be avoided by abstracting at the right level. I.e. instead of having statement to separately key screen 1, screen 2 and screen 3, there could one generic feature statement to enter these details all in one feature statement.

Test Objectives


Establishing the test objectives is the key. In principle each scenario in a feature should cover a single objective. However this need not be strict rule, if it would be more suitable for specific scenarios to cover multiple objectives. (Validation & results for example).

As a guideline UI tests should only cover end to end happy paths. (Warning messages could be included as applicable).

Test objectives not to be confused with test conditions. Each objective can have multiple test conditions (verification points).


Also there is a general myth that each scenario should only have one Then statement(s). There is no valid reason not to have more than one Then statement.   I.e. a scenario could have When <do something> Then <test something> When <do something more> Then <test something else>. While it’s true that we want to keep scenarios small and manageable, having just one then statement as a restriction is not one of the best practices.   It just increases the number of scenarios and increased maintenance overhead.

Rational Separation of concern and easier maintenance.

For a scenario where keying details with missing information and submitting to continue, there are potentially two objectives,

a)       Test the missing information warnings are appearing as expected

b)       Test the submitted details are saved and confirmation displayed.

As the warning is part of the same end to end process where no additional actions are required, it could be included within the same scenario.

However if system required to key in additional details and click separate buttons that are only required for testing the warning and not so for completing the end to end flow, it will be good practice to keep these in two different scenarios.

Process reuse


A library of actions at individual component or UI object level is maintained in order not to repeat the same.

The library should provide the feature statement and details of corresponding actions. The SpecFlow framework (Visual Studio) already provides a natural hierarchy but delivery process must ensure the compliance.

Rational Prevents TA/BAs from writing the same actions in different formats. Since the BDD feature file is written in English like language (Gherkin), same actions could be written in a number of different ways

Must avoid scenarios as below, where product details are entered in multiple feature statements; however the underlying action remains the same.

Scenario: One

When I enter Product details

And I Enter Contact details



Scenario: Two

When I Enter Contact and Product Details click on Continue button



Scenario: Three

When I enter spare parts details


In order to prevent these, a library becomes essential. Visual Studio does provide a lookup like option to support it, but having this library pre-defined would help better management of common actions.

Data driven (inline vs external)


Ensure the feature scenarios are data driven as much as possible. Whenever multiple scenarios are varying only by the data being entered or the validation conditions, it must be designed as data driven scenario.

There are two approaches used for driving the tests using data.

a)       External Data as csv (s/sheet) – Are to be used when large amount of input data to be keyed in for test scenarios (like keying product details, etc)

b)       Inline data tables (as part of Gherkin language) – Recommended for verifying data driven expected results, or minimal data keying.


Avoids duplication of scenarios and allows greater flexibility for maintenance.



Instead of below,


Scenario: Add Spare parts

When I enter product details manually and navigate to results page

Then Verify that Product type is SpareParts



Scenario: Add Grocery

When I enter product details manually and navigate to results page

Then Verify that Product type is Grocery



Scenario Outline: Add Interior decorator test

When I enter product details manually and navigate to results page

Then Verify that product type is Interior Decorator

Use data driven approach as below,

@smoke @AddWorker

Scenario Outline: Add Product Manually

When I enter product details manually and navigate to results page

Then Verify that Product type is '<Product_Type>'



Examples: Add Spare parts

| Product_Type|

| Spare Parts |


Examples: Add Grocery

| Product_Type   |

| Grocery |



Examples: Add Interior decorator test

| Product_Type           |

| Interior Decorator |

This would result in common definition file and page objects hence less maintenance. Also additional tests could be added with ease.



Principles are best practices that are more generic than the best practices. These are applicable across the entire organization and all the time without any time bounds, while the best practices, could be specific to some division and may be applicable only during a certain time period. Best practices also evolve over the time whilst principles are fairly stable.


Description Scripts should be independent of other scripts, so that these could be run on their own.
Rational As the test pack keeps growing, it is important to be able to investigate individual failures & be able to enhance for the future changes. If a complex inter-dependency is created, it would become more difficult and may end up unmanageable for newer project teams.


Description Test pack or individual script should be re-runnable on a target environment and resulting in exact same behaviour as the first run.
Rational It would require running a test pack multiple times in an environment. And having to undo (clear) data updates would prove unmanageable in the longer run. Hence the script should either be designed to clean up its updates, or ensure existence of required state of data as part of background to carry out the test. This will ensure user who may be unfamiliar with a particular functionality is still able to run and analyse the results.
Example Not applicable

Business language

Description The Feature file should be written in business language without reference to technical objects.
Rational The ultimate objective of BDD is to have business understandable specifications are actually used for validating the system built. When these specifications become technical, it defeats the whole purpose of Behaviour Driven test pack.

The following Add Product scenario is entering product details manually by selecting the particular option. The incorrect way of doing this would be,


Scenario: Add product manually

When I Click on Add product image on Admin dashboard

And  I Click on Add product button in Add individually radio option

Then Product Details page is displayed

When Product details and click on Continue button

In this particular instance, reference to the browser page elements like image, button and actions like click are very low level and technical. It makes the understanding complex and also increases maintenance as these details could change over the application lifecycle. Hence this could be written as below.


Scenario: Add product manually

When I select Add product individually option

Then Product details form is presented

When I enter product details and submit

This is more open and not referring any technical objects. The abstraction level is also depends very much on the objective of the test. In this particular instance since the objective is to test the validation, these steps are scripted individually. If the objective is just to test the end to end process, the keying in process could be written in a single feature statement as required.




Open 2 Test Vs Telerik Studio

Recently I had to do a test tool evaluation in trying to establish the right approach for a client. This involved comparing the Open 2 Test framework using Selenium web Driver and the Telerik test suite which the client already had in place for some of their projects. The objective is to aid establishing the key drivers and customizing a solution that achieves the maximum benefit for the investment. This is specifically done for testing web applications, hence the focus is on web functional testing, and however I have also listed other aspects separately below.

Web functional testing



Open 2 Test

Selenium Web Driver

Telerik Test Studio



Free. Open source software based. Uses common office software like MS Excel for scripting tests.

Commercial product licensed per duration. (To check if it supports unlimited users?)

Well open source is not all free, and will require an initial investment to extend and customize the tool as required. So this need to be considered in the right context.

Skills & Learning Requirement

Predefined set of simple keywords used for scripting.

UI based scripting language.

Record & Play mechanism.

Both are intuitive to large extent and may require some amount of learning & basic training.

Integration with TFS

Extensible. Java based, hence could leverage on numerous open source libraries available.

Built-in support

While Telerik supports full integration with Visual Studio & TFS, O2T could be extended & customized to integrate as per the specific requirement. (Logging defects in a specific TFS project, etc)

Multi browser support

Existing support for IE & FF. Could be extended for others, but will require framework enhancement effort.

IDE support only for FF

Built-in support for all major browsers for both recording (scripting) and testing.

Telerik’s feature enables cross browser testing automatically.

Test pack maintainability

Fully hand-crafted scripting, hence relatively easier to update & maintain.

Decouples the page DOM elements using object repository, hence easier to maintain page element changes.

Supports script re-usability.

Recorded scripts hence will require effort to update for maintenance.

Allows in-line script editing.

Allows grouping of tests into a re-usable block

As the scripts are fully hand-crafted and in an easy to use excel format, it would be more effective to manage and maintain the test scripts.


Basic reporting. Being enhanced to provide interactive links.

Fully customizable & extendable.

Basic built-in reporting. Not much option to extend the contents & format of the report

O2T is slightly better off due to the enhancements and customization support for the reporting.


Requires enhancements to the framework.

Built in support


Data driven Testing

Basic support

Full Support



Other Aspects



Open 2 Test

Selenium Web Driver

Telerik Test Studio



Load & Performance test support

Not supported

Built in support albeit with additional licensing cost.


Support for Non-browser based testing

Requires enhancement

Built in support



In Summary, the tools are pretty evenly matched on the attributes measured against. While Open 2 Test (Selenium) scores on the licensing cost and script maintainability, it requires some (one-off) enhancements & customization to the underlying framework to make this fully operational and integrated with the project delivery framework.

The recommendation is to apply weightage to the parameters that are used for comparison depending on the organization context (strategy, goals, immediate and long term delivery plans) and then arrive at a score card.

Open 2 Test

On a proof of concept I was working on for a new prospective client, I have had to get my hands on the Open 2 Test framework. This was an Open Source initiative sponsored by my (NTT DATA) organisation few years back and I never actually bought the value it could bring to an organisation. The main advantage as portrayed by the team is the interoperability across test tools (i.e. you could run the same test script using QTP or Selenium without any(?) changes hence prevents the organisation from any vendor lock-in. But seriously this alone doesn’t convince me to go learn a new scripting syntax, new tool etc. But when I looked at the tool hands on, I realised the true benefits of this tool.

This is a not a tutorial, but just discuss the key benefits of this framework. For detailed tutorial, there are many documentation available on the web site ( ).


The first thing that comes on top is the simplicity of the scripts. It uses a tabular form and works with s/sheet. The following screen shot shows a simple script.

This simple script launches a web browser (configurable), and performs a click and checks for values of couple of div tags. As can be see, it is very self-explanatory once the convention is understood, which is very minimal as well. This is a significant step when compared to the tool specific scripting languages. And the fact this can be done using s/sheet means, even business users could be trained to directly script tests rather than a need for a specific testing tools team.


Another key feature is the ability to extend the keywords and functionalities of testing as required by individual application and environment. For example, on the POC I was working on, there was a requirement to test for a specific text, which gets updated via an Ajax call. The use case was like adding a book to the shopping basket and it invokes an ajax call. And once the book has been added to the basket (at server), the ajax call updates the text on a particular (progress bar like) div tag to Item added to basket. The time lag between invoking the request and finally the text getting updated is unknown due to the nature of the ajax request. This need a wait function that need to wait till a particular text is updated on the div tag. This is supported by the underlying Selenium web driver. Hence I have introduced a new action keyword (waitfortext), which will wait until the element text changes to an expected text. There is an option to specific maximum timeout.

Looking at the source this was very straightforward as below.

It finds the element using one of identifiers or xpath. And then simply invokes the webdriver method to wait for its text value to be the one expected. (with max time limit)

Maintainability & Re-usability

Another key concept it uses is the decoupling of web elements and a logical identifier. In the test script you could refer to the web page elements using logical identifiers and then link these to the specific web elements using another s/sheet called Object repository. This gives a maintainability advantage and also allows re-usability. You maintain a s/sheet as below where you could associate logical names to the web elements.

Then object name would be used in the scripts, which prevents script updates whenever a page element changes. Also this enables re-using the object name across pages and serves as a complete data dictionary.

Open Source

Above all others, the framework is open source and very light weight. This allows complete freedom in terms enabling changes and customising as per specific needs. I don’t have to explain lot about the benefits of open source. However one down side is it doesn’t have a very active community, probably because of less awareness. But there is a small team still actively works on it and publishes periodic updates.


It is quite awesome, but it is due to its age, a bit old school. Now a days there are other open source alternatives available. .Net Canopy being one. But its ability to work with test scripts on s/sheet and be able to load & process data from s/sheet makes it very useful indeed. Perhaps this will require a bit of modernisation to be able to work with latest web technologies.


  • Conceptually strong
  • Open source
  • Others as above. (Simple, Extensible, etc)


  • Bit old style (although this could be improved)
  • The open source code isn’t very clean


ploeh blog

Ideas, Experiences & thoughts worth Sharing!!!

Passionate about data

Ideas, Experiences & thoughts worth Sharing!!!

You’ve Been Haacked

Ideas, Experiences & thoughts worth Sharing!!!

Swaroop C H - India, Startup, Technology, Life Skills

Conning people into thinking I'm intelligent. Since 1982.

Paul Graham: Essays

Ideas, Experiences & thoughts worth Sharing!!!

Martin Fowler

Ideas, Experiences & thoughts worth Sharing!!!

Scott Hanselman's Blog

Ideas, Experiences & thoughts worth Sharing!!!