Ideas, Experiences & thoughts worth Sharing!!!
Category Archives: Automated test
BDD is a relatively new method for developing software, which is essentially a refined version of test driven development (TDD). Where the TDD uses more traditional tools in developing the tests (QTP, etc), BDD introduces a new paradigm of testing framework & tools. In BDD the tests are written using a language called Gherkin, which follows a high level “Given … When…. Then..” format to define the test (requirement) criteria.
This framework is widely being adopted now and there are tools support in java, .Net and open source technologies. While I had my reservations initially in its practicality, after using it for a few weeks (in a real world financial domain project), it became clear that there is actually value in adopting BDD.
One of striking aspect was the discipline required for it to be effective and re-usable in large enterprises. The discipline comes from well-established best practices and a governance model (ensure compliance) to support the framework & tools. This post doesn’t go into the technical framework and tools which I found mostly covered in a number of blogs. I intend to cover some of the best practices we have created and followed to make this as beneficial as it can be.
Best practices are guidelines that are applicable at a certain time point and can change over a period of time depending on the organization strategy, project context, etc. This section should list the current active best practices at any point of time.
BDD supports automating component & UI testing. And as a general good practice for maintenance and future enhancement it is recommended that the test pyramid approach is followed in terms of number of tests and the coverage across the spectrum of automated tests. (Unit, Component (Api) and UI tests).
And other numerous web resources.
Test background should be limited only to any pre-requisite activity that doesn’t have any individual test specific action or data.
These would activities such as logging in, cleaning up or setting up base data (Not specific to individual tests).
Having test specific data setup likely to change for individual test cases, resulting in complex background feature statements.
Consider a feature (test pack) requiring keying in screens 1 to 5 with 5 difference combination of data (scenarios).
If for all scenarios screens 1-3 remains identical, there would be an urge to move this statement to the background of the feature, hence resulting in less duplication.
But this is not recommended as even though the steps appear identical they may be processing different data set and in future one of these screens might need different action for some of the scenarios. This will result in rework of the entire feature rather than just modifying the impacted scenarios.
The duplication could be avoided by abstracting at the right level. I.e. instead of having statement to separately key screen 1, screen 2 and screen 3, there could one generic feature statement to enter these details all in one feature statement.
Establishing the test objectives is the key. In principle each scenario in a feature should cover a single objective. However this need not be strict rule, if it would be more suitable for specific scenarios to cover multiple objectives. (Validation & results for example).
As a guideline UI tests should only cover end to end happy paths. (Warning messages could be included as applicable).
Test objectives not to be confused with test conditions. Each objective can have multiple test conditions (verification points).
Also there is a general myth that each scenario should only have one Then statement(s). There is no valid reason not to have more than one Then statement. I.e. a scenario could have When <do something> Then <test something> When <do something more> Then <test something else>. While it’s true that we want to keep scenarios small and manageable, having just one then statement as a restriction is not one of the best practices. It just increases the number of scenarios and increased maintenance overhead.
|Rational||Separation of concern and easier maintenance.|
For a scenario where keying details with missing information and submitting to continue, there are potentially two objectives,
a) Test the missing information warnings are appearing as expected
b) Test the submitted details are saved and confirmation displayed.
As the warning is part of the same end to end process where no additional actions are required, it could be included within the same scenario.
However if system required to key in additional details and click separate buttons that are only required for testing the warning and not so for completing the end to end flow, it will be good practice to keep these in two different scenarios.
A library of actions at individual component or UI object level is maintained in order not to repeat the same.
The library should provide the feature statement and details of corresponding actions. The SpecFlow framework (Visual Studio) already provides a natural hierarchy but delivery process must ensure the compliance.
|Rational||Prevents TA/BAs from writing the same actions in different formats. Since the BDD feature file is written in English like language (Gherkin), same actions could be written in a number of different ways|
Must avoid scenarios as below, where product details are entered in multiple feature statements; however the underlying action remains the same.
Scenario: One When I enter Product details And I Enter Contact details ... Scenario: Two When I Enter Contact and Product Details click on Continue button ... Scenario: Three When I enter spare parts details ...
In order to prevent these, a library becomes essential. Visual Studio does provide a lookup like option to support it, but having this library pre-defined would help better management of common actions.
Ensure the feature scenarios are data driven as much as possible. Whenever multiple scenarios are varying only by the data being entered or the validation conditions, it must be designed as data driven scenario.
There are two approaches used for driving the tests using data.
a) External Data as csv (s/sheet) – Are to be used when large amount of input data to be keyed in for test scenarios (like keying product details, etc)
b) Inline data tables (as part of Gherkin language) – Recommended for verifying data driven expected results, or minimal data keying.
Avoids duplication of scenarios and allows greater flexibility for maintenance.
Instead of below,
@Carburetor Scenario: Add Spare parts When I enter product details manually and navigate to results page Then Verify that Product type is SpareParts @Rice Scenario: Add Grocery When I enter product details manually and navigate to results page Then Verify that Product type is Grocery @Carpet Scenario Outline: Add Interior decorator test When I enter product details manually and navigate to results page Then Verify that product type is Interior Decorator
Use data driven approach as below,
@smoke @AddWorker Scenario Outline: Add Product Manually When I enter product details manually and navigate to results page Then Verify that Product type is '<Product_Type>' @Carburetor Examples: Add Spare parts | Product_Type| | Spare Parts | @Rice Examples: Add Grocery | Product_Type | | Grocery | @Carpet Examples: Add Interior decorator test | Product_Type | | Interior Decorator |
This would result in common definition file and page objects hence less maintenance. Also additional tests could be added with ease.
Principles are best practices that are more generic than the best practices. These are applicable across the entire organization and all the time without any time bounds, while the best practices, could be specific to some division and may be applicable only during a certain time period. Best practices also evolve over the time whilst principles are fairly stable.
|Description||Scripts should be independent of other scripts, so that these could be run on their own.|
|Rational||As the test pack keeps growing, it is important to be able to investigate individual failures & be able to enhance for the future changes. If a complex inter-dependency is created, it would become more difficult and may end up unmanageable for newer project teams.|
|Description||Test pack or individual script should be re-runnable on a target environment and resulting in exact same behaviour as the first run.|
|Rational||It would require running a test pack multiple times in an environment. And having to undo (clear) data updates would prove unmanageable in the longer run. Hence the script should either be designed to clean up its updates, or ensure existence of required state of data as part of background to carry out the test. This will ensure user who may be unfamiliar with a particular functionality is still able to run and analyse the results.|
|Description||The Feature file should be written in business language without reference to technical objects.|
|Rational||The ultimate objective of BDD is to have business understandable specifications are actually used for validating the system built. When these specifications become technical, it defeats the whole purpose of Behaviour Driven test pack.|
The following Add Product scenario is entering product details manually by selecting the particular option. The incorrect way of doing this would be,
@AddProduct Scenario: Add product manually When I Click on Add product image on Admin dashboard And I Click on Add product button in Add individually radio option Then Product Details page is displayed When Product details and click on Continue button
In this particular instance, reference to the browser page elements like image, button and actions like click are very low level and technical. It makes the understanding complex and also increases maintenance as these details could change over the application lifecycle. Hence this could be written as below.
@AddProduct Scenario: Add product manually When I select Add product individually option Then Product details form is presented When I enter product details and submit
This is more open and not referring any technical objects. The abstraction level is also depends very much on the objective of the test. In this particular instance since the objective is to test the validation, these steps are scripted individually. If the objective is just to test the end to end process, the keying in process could be written in a single feature statement as required.
On a proof of concept I was working on for a new prospective client, I have had to get my hands on the Open 2 Test framework. This was an Open Source initiative sponsored by my (NTT DATA) organisation few years back and I never actually bought the value it could bring to an organisation. The main advantage as portrayed by the team is the interoperability across test tools (i.e. you could run the same test script using QTP or Selenium without any(?) changes hence prevents the organisation from any vendor lock-in. But seriously this alone doesn’t convince me to go learn a new scripting syntax, new tool etc. But when I looked at the tool hands on, I realised the true benefits of this tool.
This is a not a tutorial, but just discuss the key benefits of this framework. For detailed tutorial, there are many documentation available on the web site (http://www.open2test.org ).
The first thing that comes on top is the simplicity of the scripts. It uses a tabular form and works with s/sheet. The following screen shot shows a simple script.
This simple script launches a web browser (configurable), and performs a click and checks for values of couple of div tags. As can be see, it is very self-explanatory once the convention is understood, which is very minimal as well. This is a significant step when compared to the tool specific scripting languages. And the fact this can be done using s/sheet means, even business users could be trained to directly script tests rather than a need for a specific testing tools team.
Another key feature is the ability to extend the keywords and functionalities of testing as required by individual application and environment. For example, on the POC I was working on, there was a requirement to test for a specific text, which gets updated via an Ajax call. The use case was like adding a book to the shopping basket and it invokes an ajax call. And once the book has been added to the basket (at server), the ajax call updates the text on a particular (progress bar like) div tag to Item added to basket. The time lag between invoking the request and finally the text getting updated is unknown due to the nature of the ajax request. This need a wait function that need to wait till a particular text is updated on the div tag. This is supported by the underlying Selenium web driver. Hence I have introduced a new action keyword (waitfortext), which will wait until the element text changes to an expected text. There is an option to specific maximum timeout.
Looking at the source this was very straightforward as below.
It finds the element using one of identifiers or xpath. And then simply invokes the webdriver method to wait for its text value to be the one expected. (with max time limit)
Maintainability & Re-usability
Another key concept it uses is the decoupling of web elements and a logical identifier. In the test script you could refer to the web page elements using logical identifiers and then link these to the specific web elements using another s/sheet called Object repository. This gives a maintainability advantage and also allows re-usability. You maintain a s/sheet as below where you could associate logical names to the web elements.
Then object name would be used in the scripts, which prevents script updates whenever a page element changes. Also this enables re-using the object name across pages and serves as a complete data dictionary.
Above all others, the framework is open source and very light weight. This allows complete freedom in terms enabling changes and customising as per specific needs. I don’t have to explain lot about the benefits of open source. However one down side is it doesn’t have a very active community, probably because of less awareness. But there is a small team still actively works on it and publishes periodic updates.
It is quite awesome, but it is due to its age, a bit old school. Now a days there are other open source alternatives available. .Net Canopy being one. But its ability to work with test scripts on s/sheet and be able to load & process data from s/sheet makes it very useful indeed. Perhaps this will require a bit of modernisation to be able to work with latest web technologies.
- Conceptually strong
- Open source
- Others as above. (Simple, Extensible, etc)
- Bit old style (although this could be improved)
- The open source code isn’t very clean