Justin Hunter invited me to take a look at the new Hexawise Beta site on Friday. Of course I was happy to take him up on the offer! I decided to give it a trial by using some scenarios I used recently when testing an e-commerce site.
To give you some background, one of our developers is making some changes to the way in which Events can (or cannot) be discounted in the shopping basket, however he thinks this may impact other product areas. Broadly speaking the site has three product types – physical products, vouchers & events – which have certain rules that need to be applied in respect of whether or not they can have a discount applied to them by the customer. Although on the surface this is fairly straightforward, the task has been complicated for the developer by some historic code. We decided it was probably worth while checking as many paths through this functionality as possible, particularly since being able to apply discounts is, if not business-critical, fairly high up on the list of capabilities that customers would expect to be working.
The first step when putting together a combinatorial test plan is to identify your parameter and value sets. In the best Blue Peter tradition, here’s some I prepared earlier:
- single discountable product in basket
- multiple discountable products in basket
- discountable product not in basket
- single non-discountable product in basket
- multiple non-discountable products in basket
- non-discountable product not in basket
- mixed discountable and non-discountable products in basket
- in basket BEFORE promo code applied
- in basket AFTER promo code applied
- not in basket
- in basket
- not in basket
- % applied against FULL price
- % applied against LIST price
- GBP applied against FULL price
- GBP applied against LIST price
You can see that what I’ve basically done is identify a range of conditions that we’d like to see tested. Ideally, I want to combine these conditions and verify that for each valid combination of the parameters above, we see the expected result satisfied in the resulting basket state, whether or not the customer then goes on to purchase said items or not.
Having identified the test conditions/parameters, we can start inputting them into the Hexawise front-end. We end up with something resembling the screenshot below.
Once we’ve input the data into Hexawise, we can go straight into creating some tests. Note however the additional (optional) requirements tab. This is new functionality! I’ll explore it in another post. For the time being, let’s just stick to creating that test plan.
Clicking on the Create Tests tab results in some analysis and then a list of 2-way interaction tests, meaning that the application has generated tests that cover interactions between 2 out of the 5 parameters. This isn’t really what I’m looking for though, since I want a set of tests that covers interactions between all 5 parameters. We can do this by changing the Strength setting from 2 to 5-way interactions.
Changing from 2 to 5 way interactions means we end up with a set of 672 tests. These will obviously take some time to implement/execute, even as part of an automated test suite. At this stage, we’d probably want to consider some means of reducing the number of tests, if possible. One way we can do this would be to re-examine the conditions identified and how they fit in with our overall objective. For example, returning to the original purpose of the testing reminds me that the main focus should be the event type of product. We have 3x possible values for this at present – in basket before promo code applied, in basket after promo code applied and not in basket. Since the focus of our tests is the event type of product, it doesn’t make sense to include tests where events aren’t present.
Removing the unnecessary parameter reduces the number of tests to 448. Significant, but not sufficient. The Hexawise application has another solution for this problem though. We can use the value expansion facility to apply some equivalence partitioning to our tests. For example, the product parameter has a number of variations (see bullets earlier) but we can take the view that really, there only 2x variations on the product theme – products that are either in the basket, or products that are not. We can apply this thinking to promo codes as well.
Running this through the analyzer in the Create Tests tab brings full (5-way interaction) coverage down to a much more manageable total of 64 tests.
Where do we go from here? Well for me, I’d probably look to have a discussion with the developer about the list of tests to ensure we’re not missing anything out, and then start to write some automated checks to cover the necessary functionality. Before we can fully implement some tests though, we need to establish what the expected results are. By the looks of things, new Hexawise includes some features to add in expected results to our tests. I’ll take a look at this in my next post.- Simon
P.S If you're interested in learning more about performance testing, checkout my Performance Testing 101 course here.