Justin Hunter invited me to take a look at the new Hexawise Beta site on Friday. Of course I was happy to take him up on the offer! I decided to give it a trial by using some scenarios I used recently when testing an e-commerce site.

To give you some background, one of our developers is making some changes to the way in which Events can (or cannot) be discounted in the shopping basket, however he thinks this may impact other product areas. Broadly speaking the site has three product types – physical products, vouchers & events – which have certain rules that need to be applied in respect of whether or not they can have a discount applied to them by the customer. Although on the surface this is fairly straightforward, the task has been complicated for the developer by some historic code. We decided it was probably worth while checking as many paths through this functionality as possible, particularly since being able to apply discounts is, if not business-critical, fairly high up on the list of capabilities that customers would expect to be working.

The first step when putting together a combinatorial test plan is to identify your parameter and value sets. In the best Blue Peter tradition, here’s some I prepared earlier:

(Physical) product

  • single discountable product in basket
  • multiple discountable products in basket
  • discountable product not in basket
  • single non-discountable product in basket
  • multiple non-discountable products in basket
  • non-discountable product not in basket
  • mixed discountable and non-discountable products in basket


  • in basket BEFORE promo code applied
  • in basket AFTER promo code applied
  • not in basket

Customer (type)

  • normal
  • silver
  • gold
  • platinum


  • in basket
  • not in basket

Promotion code

  • % applied against FULL price
  • % applied against LIST price
  • GBP applied against FULL price
  • GBP applied against LIST price

You can see that what I’ve basically done is identify a range of conditions that we’d like to see tested. Ideally, I want to combine these conditions and verify that for each valid combination of the parameters above, we see the expected result satisfied in the resulting basket state, whether or not the customer then goes on to purchase said items or not.

Having identified the test conditions/parameters, we can start inputting them into the Hexawise front-end. We end up with something resembling the screenshot below.


Once we’ve input the data into Hexawise, we can go straight into creating some tests. Note however the additional (optional) requirements tab. This is new functionality! I’ll explore it in another post. For the time being, let’s just stick to creating that test plan.

Clicking on the Create Tests tab results in some analysis and then a list of 2-way interaction tests, meaning that the application has generated tests that cover interactions between 2 out of the 5 parameters. This isn’t really what I’m looking for though, since I want a set of tests that covers interactions between all 5 parameters. We can do this by changing the Strength setting from 2 to 5-way interactions.

Changing from 2 to 5 way interactions means we end up with a set of 672 tests. These will obviously take some time to implement/execute, even as part of an automated test suite. At this stage, we’d probably want to consider some means of reducing the number of tests, if possible. One way we can do this would be to re-examine the conditions identified and how they fit in with our overall objective. For example, returning to the original purpose of the testing reminds me that the main focus should be the event type of product. We have 3x possible values for this at present – in basket before promo code applied, in basket after promo code applied and not in basket. Since the focus of our tests is the event type of product, it doesn’t make sense to include tests where events aren’t present.

Removing the unnecessary parameter reduces the number of tests to 448. Significant, but not sufficient. The Hexawise application has another solution for this problem though. We can use the value expansion facility to apply some equivalence partitioning to our tests. For example, the product parameter has a number of variations (see bullets earlier) but we can take the view that really, there only 2x variations on the product theme – products that are either in the basket, or products that are not. We can apply this thinking to promo codes as well.


Running this through the analyzer in the Create Tests tab brings full (5-way interaction) coverage down to a much more manageable total of 64 tests.


Where do we go from here? Well for me, I’d probably look to have a discussion with the developer about the list of tests to ensure we’re not missing anything out, and then start to write some automated checks to cover the necessary functionality. Before we can fully implement some tests though, we need to establish what the expected results are. By the looks of things, new Hexawise includes some features to add in expected results to our tests. I’ll take a look at this in my next post.

- Simon

P.S If you're interested in learning more about performance testing, checkout my Performance Testing 101 course here.


  • Phil Kirkham says:

    I’m also evaluating the tool so interesting to read your review

    I am a bit puzzled when you say ” I want a set of tests that covers interactions between all 5 parameters. ” – why do you want that ?

    • Simon Knight says:

      Well, maybe I should say I think that’s what I want. I believe that all 5 parameters can be combined in their various permutations to produce a valid state. Whether that’s a state that we need to test is another question. Hence I’d look to have a discussion with the developer to ensure we’re covering what we need to.

  • Phil Kirkham says:

    Seems I also found a bug in the commenting system – why does my comment above say ‘ByBy’ ?

  • Sean Johnson says:

    Sean here, one of the developers from Hexawise. Thanks for the great write up.

    I agree with Phil that you probably don’t actually want 5-way tests just because you have 5 parameters.

    The way to think about it is… how many things have to happen together to trigger a bug? That many things is how much n-way strength you want.

    So… if a “gold” customer with an “event not in the basked before the promo code” triggers a bug. Then that’s a 2-way bug. Those 2 things together are the key that triggered it, and because you are using Hexawise 2-way tests you are guaranteed that at least 1 of your tests will have “gold” and “event not in the basked before the promo code”.

    If a bug took 3 things happening together to trigger it, then you might get lucky and have those 3 with your 2-way tests, but to be guaranteed to have all 3 you would need to use 3-way tests.

    If you want 5-way coverage you are saying you want to be sure that if there is a bug that exists in your system that is only triggered when 5 things happen together that you are guaranteed to have a test for it. Problem is… 5-way bugs are 3 legged, golden unicorns, named Ralph from Schenectady. That is to say… they don’t really exist, certainly not in numbers that make it worthwhile to hunt for them.

    Unless you are landing on mars or programming a pacemaker, you really have no reason to go beyond 3-way or 4-way coverage.

    Also you have a little typo at the end where you say “Hexaware” rather than Hexawise.


    • Simon Knight says:

      Thanks Sean – I’ll take all those points on board.

      I corrected the typo that Justin pointed out. Didn’t realise there was another! I used to work for Hexaware, so I constantly have to do a mental shift from Hexaware to Hexawise whenever thinking or writing about Hexawise to make sure I’m getting the name right. I’ll try to be more dilligent in this regard! 🙂

  • […] on from my last post on using Hexawise Beta, and taking on board some of the feedback from the Hexawise developers and Justin, I continued with […]

Leave a Reply