Simon Knight

Author: Simon

  • To Automate or not to Automate your Tests?

    To Automate or not to Automate your Tests?

    The degree to which your tests are automated or not, can be a huge factor in how successful your continuous testing strategy is.

    Determining the correct split between automated and manual testing will, to some some extent, make or break your releases.

    If you have too many automated tests, or your tests are too complicated, then they may take too long to run. When things take too long to run, people pay less attention to them. Fast feedback is a critical success factor for teams wishing to deploy high quality software on a frequent basis.

    But, if you don’t have enough automated tests, or if the tests you do have result in intermittent (flaky) outcomes, your team will lack confidence in them; resulting in a higher manual testing burden and extended time to release.

    If your stakeholders are to get the feedback they need, when they need it, so they can make appropriate release decisions, how should you deploy your limited testing resources across manual and automated testing to gain greatest benefit?

    As with anything, there’s no silver bullet. There are some heuristics or general guiding principles you can follow though, which will point you in the right direction. I’ve attempted to compile at least some of them, below.

    While you’re coding

    The most obvious place to start developing your automated testing capability is while the product code is being written in the first place.

    Retrofitting unit tests to legacy code is problematic. If you’re in a situation where code already exists, you may wish to skip this step in favour of tests at some other layer of your technology stack. But where possible to do so, adding unit tests to your testing strategy is going to pay dividends in the long term – providing your team with increased confidence any time they need to add to or change their code.

    If the development team is agreeable, following a TDD (Test Driven Development) approach is pretty much the holy grail of this level of testing. With a TDD approach, your developers will write tests BEFORE the code is produced, helping to guide both the design and the development of your solution, and providing you with the best possible degree of unit test coverage in the process. This is the ultimate win, for all concerned!

    Integration testing

    Sometimes, unit testing won’t be feasible or appropriate though, depending on the technology and architectural choices that have been selected for your project or product. The next best thing is to follow more of an integration testing based approach, focusing on the areas of your project with the highest levels of integration risk.

    Precisely how you approach integration testing will depend on your product architecture. In a classic three-tier (database, business logic, user interface) arrangement, you need to focus on integration between the business logic and database layer, to ensure that correct data is stored or retrieved from your database when specific transactions are performed. For bonus points, start thinking about performance requirements here also, since the transaction times will have some bearing on the success of your project.

    A more popular architectural style these days is the micro services model, wherein the various components of system are implemented as individual services. Developing a product this way makes it highly scalable, since it means those services can be instantiated and deployed as needed across e.g. a cloud based infrastructure. From an integration perspective, it means you need to test the integrations between all those services as part of your automation strategy.

    Focus on high priority areas

    When you’re starting out with your automation strategy, you should work from a set of prioritised tests. The rationale for this is simple: if you’re crunched for time at some point in the future (which of course, is always) – if your tests are prioritised, you can start with the highest priority tests and work down the list of priorities as you have time to do so.

    It’s the same thing for test automation. If you don’t have any automated tests yet, start by automating your highest priority manual tests. Don’t do anything else until you have implemented all your highest priority tests. This ensures you have followed Pareto’s 80/20 rule, which — when applied to automated software testing — means that 20% of your tests will provide you with 80% of the needed confidence.

    Look for product risk

    You might well ask, “how should I prioritise my tests?” And the answer is simple: look for the areas of greatest risk. There are two main kinds of risk we want to consider when implementing an automated testing strategy: product risk and regression risk.

    Product risk is likely to be somewhat contentious in terms of test automation, since not all product risks are easy to test for, which means they also won’t be easy to automate. In my experience, product risk relates to one of three or four key areas:

    • Usability – the solution needs to be a pleasurable experience such that the customer or potential customer wants to keep using the product
    • Fit for purpose – the solution needs to solve the problem for which it was designed.
    • Reliability – the product needs to have the customers confidence, particularly as it relates to accessibility [security] and availability [performance].
    • Business – the product needs to meet the requirements of the organisation that created it; often this means it should generate revenue or turn a profit

    As you might imagine from the list above, it would be very difficult to write an automated test that answered even one of these questions. A more likely outcome is that you have a number of tests, which may be designed to address some facets of those various areas, such that a stakeholder (e.g. a product manager) is able to look at the results of those tests and get a feel for whether or not, the product will be usable or reliable.

    Acceptance test based automation derived from BDD’s (Behaviour Driven Development scenarios) is often used for this purpose, in my experience.

    Look for regression risk

    The other area of risk which you should consider is regression risk; i.e. the risk that something which was working before has now been broken, or does not work in quite the same way, as a result of some change to an area of the product code. Regression testing is a classic win for an automated approach, since by the time you come to automate a regression test, the functionality should be well understood and ripe for automation.

    Catching regression failures by way of test automation is an easy win.

    Take performance into account

    I mentioned performance above, while discussing integration testing, and also as a product risk. Once you have some automated tests in place, it’s actually pretty easy to monitor the performance of your product, simply by capturing and measuring how long it’s taking to run your automated tests over time.

    This does require a baseline of some sort (a set of tests which remains static in terms of scope, the number and execution approach of the tests) – but once you have this, you should be able to take the individual and aggregated time for executing the tests and determine whether the performance of your application has improved, stayed the same, or gotten worse — and take the necessary actions as a result.

    Create levels of automation testing

    Having a baseline set of tests feeds into this next point, which is that it can be helpful to have suites of automated tests which are executed for specific purposes.

    One of the traps I often see teams fall into is that they have a huge number of automated tests, which might take a couple of hours to execute, and they need to execute ALL of those tests in order to determine whether it’s ok to release their product.

    Now sometimes, you’re going to have a huge number of tests, and they’re going to take a long time to run, and there may be no way around that; you HAVE TO run all of those tests for reasons you’ve determined within your team, organisation and context. And that’s fine, but what you don’t want to end up in is a situation where it takes say 2hrs to run a bunch of tests and then find out that because a single high priority test has failed, the whole execution needs to be re-run (once whatever the issue was has been fixed).

    This goes back to the prioritisation point above. If you’ve prioritised your test cases, then you should be able to split-up your test automation jobs into a series of executions that get progressively more advanced, as you need them to. So you could have something like this, for example:

    • Job 1 = Smoke Test {10x HIGHEST priority tests}. If this passes, move onto Job 2:
    • Job 2 = Acceptance Test {50x tests for new features being deployed in the build}. If this passes, move onto Job 3:
    • Job 3 = Regression Test {400x Highest, High priority tests}. If this passes, move onto job 4.
    • Job 4 = Nightly Build {2000x Highest, High, Medium, Low priority tests covering ALL features and functionality}.

    Obviously the specifics and therefore your mileage may vary, but hopefully you get the idea, and the point is this: split up your automation jobs so they increase in scope and duration over time. Don’t START with your longest running tests, FINISH with them. Front-load the risk of your automation failing by running the most important tests first, so you can provide feedback faster in the event something serious is wrong.

    Minimise manual testing effort

    The goal of all your test automation efforts should be to minimise the amount of manual testing effort required to ship your releases. That doesn’t mean you’re going to eliminate manual testing effort entirely. Nor, in my opinion, should you wish to. Instead, you should be aiming to redirect the efforts of your testers to higher value exploratory testing, monitoring and test leadership activities.

    What you don’t want to be in, is a situation where you want to carry out a release but you need weeks of manual feature and regression testing in order to do so. These are exactly the kind of activities that your test automation should be designed to replace. But you should follow the guidelines above, to identify the kinds of tests you SHOULD be aiming to automate, and how to go about doing so.

    Equally, you should follow the guidelines below so as not to fall into the trap of attempting to automate tests which will slow down, reduce confidence in, or outright cause your test automation efforts to spectacularly crash and burn.

    Early stage development

    Trying to automate tests for features still in the early stages of development is a recipe for frustration. You’re much better off waiting until you have something stable to test against. Often, that means the feature will have already been through several iterations of testing and fixing, before it’s ready for an automated test.

    Low priority features

    Don’t waste time and effort trying to automate low value features. Focus on bigger wins. If you’re not sure whether a feature is high or low priority, check-in with your product owner or manager; they should definitely have an answer for you. I know I would!

    Tests that don’t have well defined outcomes

    If you don’t know what the outcome of a test is or should be, you shouldn’t be automating it. Unless it’s a performance test or some other kind of analysis (e.g. a static code or security analysis) where the outcome is a measurement or report which may or may not be acted upon; for those things the rules are slightly different. For a functional test, you should definitely know the outcome, and it should be binary (pass/fail) in nature. Anything else is going to lead to problems.

    Tests which cannot be fully automated

    On the subject of trouble, if your test can only be partially automated, and will as a result need manual intervention to conclude successfully (or unsuccessfully as the case may be) – this is not a good candidate for inclusion in your automated test runs. Use it as a tool, as a script to aid testing by all means, but don’t use it as an automated test.

    To conclude, if you’re trying to put in place a continuous testing strategy, automated testing is going to play a big part. You need to be super careful about what tests you choose to automate, and how you go about tooling them. The guidelines above will help you with that.

    Did you enjoy this article? Watch the video recording below:

  • How to get Product Management Stuff Done in the Face of an Endless Barrage of Other Demands and Firefighting Activities

    How to get Product Management Stuff Done in the Face of an Endless Barrage of Other Demands and Firefighting Activities

    Black and white comic book-style illustration, inspired by classic graphic novels. A product manager battles against a whirlwind of incoming tasks

    One of the questions that frequently comes up within the large team of product managers within which I work is, given all of the other problems, issues, questions and tasks I have to deal with day-to-day as a product manager, how can I find the time to do the high[er] value activities that product managers should be spending time on? Things like:

    • Talking to customers
    • Analysing competitors
    • Researching the market
    • Ideating new products and features

    While I don’t pretend to have all the answers, one of the things I think I’ve been pretty good at over the course of my career is focusing on and prioritizing value-adding activities, so I think I do have some things to say here…

    Prioritise it – in the face of all the other stuff that is asked of you, unless you make something a priority it will always go to the back of the queue and in the worst case, never actually get done. I’m in the habit of planning my work in advance, usually on a week to week basis. But as my role is evolving and I take on more senior/lead responsibilities, I’m recognizing the need to plan even further out – months, even years in advance.

    Timebox it – research is one of those things that can never really be considered complete. There’s always another avenue for investigation; always another question to answer. The easiest way to set some boundaries around the potentially infinite space of whatever research needs to be carried out, is to specify the amount of time you’re going to spend on the activity. A spike, in agile verbiage. Start off with by setting aside a couple of days, and see where you get to. If further time and effort is required, you can plan it in from there.

    Constrain it – to a specific question or hypothesis. If you go into the activity with a clear question or hypothesis in mind, you’re less likely to fall into rabbit holes along the way.

    Distribute (or delegate) it – as the PM, you’re responsible for being the subject matter expert for your product(s), but it doesn’t mean you have to know (or are even capable of knowing) everything. Think about ways you can involve other members of your team in the research activities that need to be performed. I like to think of myself as the spider at the center of a web of information (and business relationships); I spin out the threads, and then need to be sensitive and responsive to tremors along them.

    Automate it – some research activities are relatively mechanical, and as such, are good candidates for being automated. Setting up keyword searches and subscribing to useful sources of information are low hanging fruit. It may be possible for you to automate some other activities too (e.g. I use an R script to pull and collate production metrics into a report, eliminating a few steps of manual effort).

    Document it – find a tool or means of capturing the information you gather so that it’s available to you when you need it. Ideally, you want something that allows you to organise the information in whatever way makes sense for your purposes; I mainly use Evernote for this, but there’s plenty of alternatives.

    Be flexible about it – unlike some of the day-to-day tasks and activities I find myself embroiled in…

    • Acting as a scrummaster and release manager
    • Answering questions about new features during the course of development
    • Handling stakeholders and business needs
    • Supplying information to management, sales, marketing and others as needed
    • Writing documentation and producing other collateral such as webinars and blog posts
    • Generally overseeing the product roadmap and making sure everything is on track and everyone has what they need to do their job

    … Research is an activity that can, to some extent, be carried out anytime and anywhere. I always have my phone with me (I’m writing this blog post on it) and therefore have the capability to read, listen to or watch media related to the objects of my research more or less anywhere I am.

    Of course, not everyone will agree with this point. Some people like to have a much clearer line between work and home life, and I certainly understand that perspective. All I’m saying here is what works (and has worked) for me. Being flexible about when and where I am when I’m doing this kind of work affords me many more opportunities to read, and think, and ultimately to add the kind of value that’s crucial for a PM to really succeed in the areas of their job which can easily fall to the wayside, but which are hugely valuable.

    Even if you’re not willing or able to be flexible in this way, hopefully some of the other suggestions above work for you. And if you have some other thoughts (or just completely disagree with me), I’d love to hear from you!

  • Test-Planning-Simplified-Webinar

    Test-Planning-Simplified-Webinar

    As part of my Product Manager duties with TestRail, I delivered a test planning webinar based on my Test Planning Simplified article.

    You can watch the video below, or via the TestRail site, where you’ll find more webinars in this series and other useful content.

  • Cunning Strategies for Becoming a First Class Noticer

    Cunning Strategies for Becoming a First Class Noticer

    Albert Einstein famously asked the question; “How would it feel to ride on a beam of light?” Why don’t you take a moment and just think about that yourself; how would it feel?

    Nobody knows the answer. We can speculate. Scientists can synthesize information and hypothesize possible answers based on available data. Still, we don’t know. Maybe we never will. You might reasonably argue that the answer doesn’t matter anyway. I mean, who cares – right?

    But isn’t it fun to think about anyway?

    Imagination is more important than knowledge. – Albert Einstein

    The point of the exercise isn’t necessarily to find the answer. Using your imagination to explore unknown realms brings rewards of a different kind. That’s not to say you should give up on the idea of trying to find the answer. After all, your own thought experiment may be an order of magnitude more solvable than knowing what it feels like to ride on a beam of light. Einstein was just a child when he pondered what would turn out to be one of his most fascinating lines of inquiry. It certainly didn’t hurt him or his career.

    Amazing things happen in the brain when you let it roam free. When you allow it to explore the boundaries of the known, and take excursions into things yet to be known.

    This is one of the joys of my work, in the field of professional testing. The exploration of things. Software, systems, artifacts. Always questioning. Always seeking answers. Always trying to dig a little bit deeper.

    I have always loved things. Just things in the world. I love trying to find the shape of things.” – Leonard Cohen

    One of the challenges of software testing is that the focus of your efforts are often somewhat abstract. However, the specific thing you’re currently exploring (or testing) may have elements of physicality. It may have an interface purposely designed for users to interact with, including buttons and displays. It may even be a physical device like a phone, a watch, a robot or a headset.

    If you want to sharpen your noticing powers, it’s worth paying the physical realm some attention. The things you explore with your mind don’t have to be abstract. They can be things right in front of you. Everyday things. Things you sit on. Things you work or play with. Things you put on or into your body. When you explore, you should use as many senses as possible to determine the shape, weight, texture and substance of a thing.

    You need to take an interest in every part to test what you’re working on effectively, and to its fullest extent. You need to show a deep concern for each component and interface. A passion for every detail.

    “Look with all your eyes. Look.” – Jules Verne

    First Class Noticer

    If you’re a tester, you’re basically being paid to observe and to gather information of importance for people who care about a facet of a product. Different aspects of what you do will be for different levels of interest to the people you report to. Ultimately, the whole product is important to somebody, somewhere. For example, the wearer of a smart watch isn’t going to care about the clear, clean user interface if the clasps of their watch are so fragile they can’t leave their home without the device falling into pieces. Nor will they care that it wasn’t your job to test that part. The whole experience is what ultimately counts. In order to give people the experience they’re searching for, you need to become a Noticer of the highest order. An explorer of every aspect of your software, your device, your project.

    First-Class Noticer – noun (coined by Saul Bellow in his novel, The Actual)

    Someone with the ability to spot important details among noise.

    As a tester, it’s your job to become a First Class Noticer. Your ability to identify key issues of concern, separating them out from the surrounding noise, bringing them to the attention of others clearly and persuasively, are skills that separate First Class Noticers from ordinary, or Second Rate Noticers. Testers often joke about having OCD. The ability to notice or observe important details that others miss may come naturally, or it may not. Even if it doesn’t come naturally to you, there is still room for you to build your existing noticing skills from a standing start.

    Here’s Nine ways you can Develop First Class Noticing Skills:

    1. Make sure you are always looking – keep your eyes open and attentive always. Be completely open to the possibility that something of interest is happening either right this very moment, or very soon will be. If you don’t pay attention – you might miss it!Think about ways you can capture more information from the software or systems you’re testing, so you can look in more places at once. Can you monitor the logs while you’re testing? Can you observe network traffic? What about resource utilization on your servers? What other information might you have missed? Did you read all the documentation? Is it up to date? Does it cover everything it needs to?
    2. Deem everything interesting – stay curious! Try to cultivate a sense of wonder, go far beyond the surface. Be deeply interested in the object of your attention. Look at all aspects, elements, components and sub-systems. Keep building upon your understanding. Make detailed notes. Drop and return to them later if you get bored. Slow your work pace down if necessary (and justifiable) so that you can follow a line of inquiry to a conclusion. You never quite know what you may discover and learn along the way.In my experience, it can be useful to occasionally put time into something that isn’t directly related to what you are working on. For example, investigating a new tool may not help you make progress on a deliverable in the moment. However, if it proves to be useful, is likely to realize many benefits in the future, outweighing the cost of time originally spent. That’s what makes the time invested justifiable.
    3. Change course often – don’t allow your brain to get stuck in a rut. Allow your mind to wander and explore many different paths. Focus for a period, then refocus and change your approach. Use heuristics to guide your thinking. Randomize them if this is helpful or necessary to move forward. Your brain is lazy and will happily settle into a groove. It can be difficult to break out of a rut if you allow yourself to stay there for too long. The best thing to do is not to allow your mind to settle in the first place. Using heuristics to guide your thinking down avenues that your brain might otherwise resist, is a great way to disrupt its natural tendency towards idleness.
    4. Observe for long durations – what might you notice if you just maintained your attention a little bit longer? Does the state change over time? Are there details you may have missed previously?Buddhists and other traditions have methods of meditation that lead to a condition they call Jhana – a “state of profound stillness and concentration in which the mind becomes fully immersed and absorbed in the chosen object of attention.” It’s surprisingly easy to enter this state of profound concentration:

    Stare really hard at a thing for 5-10 minutes… Repeat until everything in your peripheral vision gets dark, and only the center of your vision is bright.Whether you use this method or not is up to you. Personally, I’ve found mindfulness meditation to be a useful and reasonably effective tool for practicing increasingly longer periods of focused attention. It’s like the steps above, but without the staring. Or the tunnel vision.

    Sometimes though, it’s just difficult to separate out the distractions, and for those occasions a Pomodoro timer is a useful accessory.

    1. Pay attention to the stories around you – consider the narrative of the situation you’re in, or the application you’re testing. What might be going on in the end-users’ world? How might their story affect the use of the software?Everything around you has a story, a context of some sort. Your job, as a tester, as a First Class Noticer, is to act as an exegetic for the story of your software under test, and for your project. To interpret events, issues, bugs, threats and risks and explain to the people who care about them what they mean, in the context of their occurrence.
    2. Look for patterns and connections – where are the dots and how are they joined? The very act of attempting to identify the patterns and connections between different items is likely to expose gaps in your own understanding, and potentially omissions in the product that has been delivered.This is an area where you can add significant value as a tester. With many teams and projects focusing hard on trying to automate everything, it’s easy to forget that machines, at least for the moment, only do what we’ve programmed them to. They only look where they’ve been told to look, and only see what they’ve been programmed (or trained) to see. The application of your human intelligence, though imperfect, is nonetheless a powerful tool for separating signals from noise.
    3. Document your findings – be a compulsive note taker! Take the mechanics of note-taking seriously. Invest in a quality notebook (I tend to have numerous Moleskin note books lying around for various kinds of notes) and take one everywhere with you. Use it judiciously.Don’t forget to review them occasionally either. Look for insights. See what jumps out at you. What inspired you? What confused or irritated you? When you re-encounter an anomalous behavior, did you make a note of it? Was it timestamped? If you use a tool like Evernote, you can add screenshots, videos, snippets etc and have access to them from whatever device you’re working on!
    4. Don’t judge, be indeterminate – when you judge, what you’re really saying is “I already know as much as I need to about ”. You’re closing your mind to possibilities that lie outside the realm of your current worldview.And that’s fine, I won’t judge you for that. ;-)But, it means some insights may elude you, had you remained open to alternative possibilities or world views. Seek to hear what the object of your focus shows you, instead of imposing answers, trying to shoehorn a situation or problem into a predefined conception, or trying to control the order of things.
    5. Use all your senses – being a First Class Noticer isn’t just about using your eyes. You should always be listening too, and using any of your other senses that are appropriate for the situation. For the purposes of this exercise, you might also consider your emotions as sources of useful information. Do you feel confused? Angry? Offended? What does that tell you about the object of your attention?

    Beyond Software Testing

    In each point above, I’ve applied the principles primarily to software testing activities. Hopefully you realized as you read through them, that these principles can be expanded far beyond software systems and applied elsewhere. For example, you could apply your noticing skills to the people and activities involved with the project your working on. Watching how the product is developed, and how the people who are developing it behave and communicate with each other. Paying deep attention to how people are interacting during meetings, conversations, in emails and tickets, can provide powerful insights into the root causes of bugs and other issues that emerge over the life-cycle of a project.

    Developing your noticing skills can take you much further than software testing. If you aspire to a leadership position, the ability to observe, reflect upon those observations and incorporate them into deeply considered decision making process will serve you incredibly well. While you’re heading in that general direction though, enhancing and applying your noticing skills can, and should, be a key part of your software testing repertoire.

  • Cunning Strategies for Effective Remote Working

    Cunning Strategies for Effective Remote Working

    During my 10 or so years of working as a contractor/freelancer, I think I basically got used to not having much in the way of performance feedback. For me, knowing that I’d performed at least well enough to secure my next gig, was indication enough that I was heading in the right direction.

    Not everyone has the benefit of having had that experience though. So, I think I can add some value by sharing what I’ve learned over the last 4-5 years of working 100% remotely.

    Please find my largely unfiltered advice for successful remote working, below.

    Invest in reliable hardware

    First up, you should be aware that if you’re working remotely, for the vast majority of the time you will be your own IT Support person. If you’re having issues with your technology or workspace, it’s going to be down to you to figure those things out.

    My suggestion = invest in good quality equipment and learn to look after and use it to its maximum potential.

    Have a backup plan

    An issue that will almost inevitably come up is internet connectivity. Whether wired or wireless, if your internet provider (IP) is having issues, you will suffer. And when you’re relying on that connectivity to get your work done, you will suffer even more.

    My suggestion = get a mobile phone plan with unlimited data and tethering. That way, if your internet connection ever does go down (and it will!), you have a backup plan. You can just work off your mobile data, which is sufficiently fast these days to be almost indistinguishable from your internet connection for working on Slack, shared Google docs etc.

    For bonus points, consider extending this line of thinking to having a backup plan for other areas of your work too. What happens if your PC stops working, or your workspace is unavailable for some reason?

    Enjoy your freedom

    Having confidence that you can use your mobile data for work also means you can work from wherever you are. Something I take advantage of quite a lot. As a homeschooling parent, my services are often required to take the kids various places during the daytime. So long as I have my laptop and my phone, I can take them where they need to be, then sit and carry on working.

    My suggestion = take advantage of your remoteness and get comfortable with working from wherever you are.

    Create a dedicated workspace

    Of course, it helps if you do have a dedicated workspace. I love my home office! It’s where I spend the vast majority of my time. Even when I’m not working-working, I’m often working in my space, e.g. on this blogpost, or on a side-gig/hustle, or trying out some new video games, watching a film, reading etc. It’s my Bat-Cave, and I’m very comfortable in it.

    My suggestion = if you’re working 100% remote, try to carve out a dedicated workspace. It’s good to be comfortable.

    Be the host

    Meetings can be problematic as a remote worker. I’ve been quite fortunate over the past few years, that most of the folk I have worked with have also been remote.

    My entire team works remotely at present, over many different timezones. I find that I tend to face more challenges when working with teams who are only partly remote. Since when they’re working together in an office, they forget or are less considerate about the people who are not in the office with them. I suspect this is where some of Lisa’s remote working issues stem from.

    My suggestion = host all your meetings, to the extent possible. I find that if I’m the one hosting the meetings (i.e. calling the shots), it’s a lot easier to make sure people are paying attention to me, if I need them to.

    Use the best software tools for the job

    On that note, you should make sure you have a strong conferencing package for your calls and meetings. Zoom is very good. But there are plenty of other options and many of them are also likely good. Don’t stick with something subpar just because a bean counter somewhere thinks it’s the right choice though. You and your team are the ones who have to use it daily, and dealing with laggy calls and dropped connections is no fun.

    My suggestion = Experiment to find the right choice. Then dig your heels in if necessary.

    Share your work

    A key remote working issue that I see often is not knowing what another person or people are working on. You don’t want to have to ask about this all the time. If I need to know what e.g. my junior product manager has been doing for the last few days, I should be able to see progress without having to go and specifically ask for it.

    Having to ask what’s being worked on right now and how much progress has been made (or not made!) is painful for everyone. It’s much better to just share your work as you’re doing it, so everyone (that wants to) can go look and take whatever actions they need to based on what they see (or not see).

    My suggestion = use Confluence. Use Google Docs. Use Slack with plugins. Use Git and check-in regularly. Use whatever tools you need to. But SHARE. YOUR. WORK. Somewhere everyone can see it. Regularly.

    Share more than just your work

    You can even take it one step further than that. Again, particularly as a remote worker, it can be important to give the rest of the team insights into your personal or family life. I like to do this by sharing occasional pictures of my family and talking about stuff done over the weekend etc. We have a #Random channel also for books/tv/films/video game discussions. Whether everyone or just you is remote, it’s good to contribute to a culture of getting to know each other outside of work.

    My suggestion = share too much information!

    Make like a spider

    Everyone needs feedback, but I’ve found particularly in my more product oriented role, that it’s been helpful to identify a number of different sources of feedback for how we’re doing as an organisation, as a team, and at a product level. I look for feedback on those things in our forums and in our social feeds. I also look for them in our support ticket queues and customer service/success emails. I look at industry news, newsletters and blogs as well. Basically, I’ve always got my eyes open for new and useful sources of information that can provide me with actionable insights as to how we’re doing, at a product kind of level.

    My suggestion = create a web of information sources and monitor them for interesting vibrations.

    Treat everything as feedback

    You can do the same thing internally of course. And I do! I monitor our IM channels for interesting & useful snippets of information. As well as emails and meeting notes or confluence updates. As the facilitator of many meetings, I use them as an opportunity to monitor the pulse of the team also. And let’s not forget one-to-ones and water cooler style conversations also.

    My suggestion = treat everything as a potential source of feedback. Get creative!

    Use systems to support your goals

    Something else that may be worth considering is how you can create your own feedback loops. What systems can you put in place that provide you with the desired feedback? For example, if I want to improve my overall health, I can identify a useful metric(s) – e.g. frequency/distance/speed of runs or cycle rides – and have an expectation that those metrics are going to trend in the desired direction over the course of time. With regards to how I’m doing in a new role, I’d apply the same thinking. What’s a useful KPI? What system is or can I put in place to support it? Is the KPI trending in the right direction? If not, what do I need to tweak?

    My suggestion = use systems to support your feedback goals.

    Remote isn’t for everyone

    I think one of the toughest things about working in a nebulous role like software testing or product management is the sense of imposter syndrome you can have, even after many years of experience, basically because the role is somewhat different and needs to be tweaked for each organisation and team you work with. That sense of being lost or adrift, without a good sense of whether you’re basically heading in the right direction based on useful feedback, can be quite overpowering.

    If you’re working completely remote from your team, it would be very easy for those feelings to become exaggerated. If you let them. And maybe, if you find that happening, remote working isn’t the right thing for you.

    But, I think if you put some of the ideas above in place, working remotely can be a lot easier than it otherwise might be. And if it still isn’t working out, perhaps it’s time to head back to the office!

  • Cunning Strategies for Getting out of a Testing Rut

    Cunning Strategies for Getting out of a Testing Rut

    One of the things you’ve probably noticed when you’ve been software testing for a while, and particularly when you’ve been testing the same product for any length of time, is that your brain starts to settle into some established ways of thinking about the software you’re testing.

    After a while you already know where lots of the more challenging areas of your product are, and when you begin to do some testing you make a beeline for those areas because they’ve already proven themselves to be the most fertile hunting grounds for juicy bugs.

    But how did you get into that area of they system? What route did you follow, and what might you have missed along the way?

    What if instead of making that beeline – you branched out in a completely different direction?

    What might you find instead?

    Getting your groove on

    The brain is a funny thing. Incredibly powerful of course, but also with annoying tendencies to get stuck in ruts or grooves of comfortable thinking.

    In fact – when you’re carrying out your software testing, your brain is probably finding ways to think fast (lazily) rather than slowly (deliberately, analytically, creatively) much the time. And the temptation will be to let it, because, well – that’s just what it does, right?

    In the field of evolutionary psychology there’s a growing body of evidence that serves to demonstrate how our brain can deceive us.The list of cognitive biases (deceitful shortcuts in our thinking) is a long one – but sadly, just being aware of them isn’t a complete solution.

    As professional software testers we have to not only be aware of our biases, we have to take control of them and constantly challenge ourselves to break out of comfortable patterns of thinking.

    Imagine you have a piece of paper and you make marks with a pen on that surface. The surface records the marks accurately. Previous marks do not affect the way a new mark is received. Change the surface to a shallow dish of gelatin. You now put spoonful’s of hot water on to the gelatin. The hot water dissolves the gelatin. In time, channels are formed in the surface. In this case previous information strongly affects the way new information is received. The process is no different from rain falling on a landscape. Streams are formed and then rivers. New rain is channeled along the tracks formed by preceding rain. The gelatin and landscape have allowed the hot water and rain to organise themselves into channels or sequences. – Edward de Bono. “Think!.” Random House, 2009

    Those channels, sequences or grooves that de Bono talks about are exactly what we need to break out of in order to deviate from established paths or sequences when you’re testing a piece of software.

    We have to break out of our biases, old or comfortable ruts – busting out of the established groove and opening up new channels for thinking about and looking at the systems and software we’re paid to test.

    Pattern interrupts

    One way of doing this that de Bono talks about in his same book (Think!), is the use of random words, in order to steer thought away from those established ruts and patterns. His technique is actually very simple:

    • You already have or identify a focal point for your thinking.
    • You choose a random word.
    • You use the random word as a launching point for thinking creatively about the focal point

    So by way of an example – let’s say the focal point of your thinking is a grooming meeting, and you want a new, more creative way of thinking about the stories being presented. At random – you choose the word “Absent”.

    All you need to do now is follow the new direction the word takes your thinking in. So in the case of a grooming meeting, you might start thinking about or asking questions like:

    • What’s missing or absent from this story? Performance requirements? Security requirements?
    • What if a specific piece of data is absent or missed due to, e.g. User error?
    • What if a step is missed or becomes absent?
    • What if a piece of architecture becomes absent (or fails over)?

    You might come up with many more ideas depending on your circumstances, product and team. But hopefully you get the idea.

    The results of using this tool can be very powerful, because now instead of following a familiar route (train of thought) and arriving at the usual destination (conclusion, opinion etc.) – you’re liable to end up somewhere completely different, and probably via a very different route to what you’ve been used to.

    De Bono has a selection of words he suggests you use for this process, nouns typically – like the following: Letter, Barrier, Ear, Tooth, Bomb, Soap.

    But the context he’s using is slightly different. He’s coming from a creativity angle. Testing is absolutely a creative discipline, but perhaps there’s some other words we can use to jog our thinking instead… Heuristics

    Many folk, particularly within the Context Driven Testing community like to talk about heuristics and mnemonics. Powerful reminders that can be used as frames of reference and to steer testing efforts towards areas of risk. Often they’ll come in the form of a checklist, or a mind map, or a cheatsheet.

    For this technique though, we don’t need any of that. Just a list of the keywords will do, like the list of product quality attributes below:

    1. Sequence
    2. Concurrence
    3. Confluence
    4. Synchronisation
    5. Share
    6. Interaction
    7. Continuity
    8. Hierarchy
    9. Priority
    10. Dependency
    11. Repetition
    12. Loop
    13. Parameter
    14. Prerequisite
    15. Configuration
    16. Rule
    17. Customise
    18. Constraint
    19. Resource
    20. Access
    21. Lock
    22. State
    23. History
    24. Rollback
    25. Restore
    26. Refresh
    27. Clone
    28. Temporary
    29. Trace
    30. Batch
    31. Void
    32. Absent
    33. Feedback
    34. Saturate
    35. Sort
    36. Scale
    37. Corrupt
    38. Integrity
    39. Invoke
    40. Timing
    41. Delay
    42. Customers
    43. Information
    44. Developer
    45. Team
    46. Tools
    47. Schedule
    48. Deliverables
    49. Structure
    50. Functions
    51. Data
    52. Platform
    53. Operations
    54. Time
    55. Capability
    56. Reliability
    57. Usability
    58. Scalability
    59. Performance
    60. Compatibility

    Whenever you feel stuck (in an established groove, rut or pattern of thinking), you can just pick one of the words from the list and use it to generate some ideas for new ways to carry out your software testing.

    I’ve been testing a repayment calculator recently. So I’ll use that as the basis for a couple of examples, just to get you started:

    When I looked at the clock, I saw the second’s hand was pointing at 41, Delay:

    • What happens if I delay the user actions? Say I go off to grab a coffee while filling out the various stages? Does my session expire? Do I get logged out? Does the application do anything to protect sensitive information from prying eyes?
    • What happens when the server is under load? Are responses delayed?

    Let’s try a different one. This time, I got 47, Schedule. I’m still working with the same application:

    • The application gives me the option to schedule repayments. I’ll explore that for a bit. Does the schedule work?
    • Can I re-schedule once I’ve submitted a plan?

    When I tried again, I got 11, repetition:

    • What happens if I repeat the calculation? Do I get the same result?
    • What if I keep repeating the same step? Am I able to do so? Should I be able to do so? Do I see any errors?

    And so on and so on. You can use this technique whenever you need to. It’s important that the selection is random though – otherwise your brain will just choose words you feel comfortable with, which is not going to achieve the desired effect of taking you off in a completely new direction.

    Watch the clock

    For randomness you can use another technique straight out of De Bono’s playbook.

    The more observant among you will have noticed there are 60 words. That means you can use the second hand on your watch, or whatever other clock you have to hand. Just glance at the time, make a note of the number of seconds, and use the matching word to bust you out of your testing groove.

  • Systems not Goals

    Systems not Goals

    One of the things folk tend to do around this time of year is think about their goals and targets for the upcoming year, and how well they did with those from the last year. I just did it myself in fact. Fortunately, they’re all in Evernote so it’s very easy to pull them up.

    I had some great objectives for the last year. They were all mind-mapped out (I use XMind for this) and categorised into different areas – health, career, learning, reading etc.

    Some of them I achieved. My health levels have way improved over the last couple of years, although I’ve typically not been very big on exercise.

    Some of them I’ve failed at. I’d planned on submitting abstracts to lots of conferences this year, but that never happened in the end. I’d also set a target for 52 blog posts this year (presumably 1 per week, though the goal doesn’t really state this explicitly.)

    If I go back to my 2014 goals, my track record for non-achievement gets even worse. Or better, depending on which way you look at it!

    Pilates and push ups didn’t fare so well. Neither did journaling, moving house and ramping-up our investment in property.

    So I can’t help but ask myself what the point is? Why bother to set goals when there’s a strong likelihood that a few days later I will have either forgotten or disregarded them?

    Probably – not much. Fortunately though, over the last few months of 2015, I found a much better way of achieving the things I set out to do.

    Systems, not Goals

    Instead of having a specific target or objective, I try to implement a system for actually getting things done instead. I’ll give you a few examples of what I mean.

    Diet

    It’s important to me that my family and I have a relatively healthy diet. So instead of setting a goal for healthy eating and changing our diet completely – I just implement a few rules for the food we purchase, what we cook and how we approach mealtimes, and what we try not to have too much of hanging around in the kitchen. Those rules look something like the below:

    • Don’t eat anything that’s white or that can be made white – bread, potato, pasta, sugar, milk
    • Replace carbs with beans or legumes
    • Eat a high protein breakfast
    • Keep nuts in the house for snacking (if I have nuts, I’m less likely to eat unhealthy alternatives)

    Exercise

    I hate going to the gym, so I just don’t do it. I don’t mind playing racket sports but it’s difficult to find other people to play them with at the times I want to play. So that doesn’t happen too often either.

    I have a fitness tracker, which helps a bit because it means my activities are at least being tracked, and in theory what I pay attention to should improve. The problem is, it’s too easy not to pay any attention. So the system that works for me in terms of exercise is this:

    • Put on trainers, sweat pants etc so I’m ready to exercise
    • Use a running coach (an app on my phone is fine for this)

    Exercise tends to be a bit of a hit and miss affair for me, but the system above is working currently. I’ve been out running more in the past few weeks than I did the entire rest of the year, so I’m doing something right.

    Learning

    I love to learn, so this is a pretty easy one for me. Still, I need to have some way of making sure that I’m learning the right things and that I’m doing it regularly. As such, I have a couple of systems to help me out:

    • Use Wunderlist (or a notepad, or whatever works for you) to track stuff I need to learn about
    • Set aside one day each week to learn new stuff
    • Turn off the TV and read instead

    Wunderlist works great for tracking stuff that I need to do, and stuff I need to learn becomes just another list. Since I have my list and one day a week specifically set aside for learning related activities – it’s just a matter of choosing or continuing with something from the list. Whatever I have energy for at the time.

    Turning off the TV is a more recent addition to my collection of systems. I’m not a big fan of the media generally and try not to listen to, read or watch the News for example (too depressing). Further curtailing my TV watching so it’s even less of a distraction seems like a natural next step.

    Systems work better

    I could go on and on. I have a ton of systems for achieving all of the things I want to do.

    Personally, I think they work way better than goals for several reasons:

    • They’re actionable – systems like the ones I mentioned above are made up of specific activities and things that need to be done. They’re not vague, pie-in-the-sky ideas about things I’d like to do or achieve. They can be done right now!
    • They don’t specify a result – where diet is concerned for example, none of the actions I talked about were tied to weight loss, though that has been a natural consequence of eating more healthily. Along with more energy and an easier life in the kitchen.
    • I don’t have to think about them – having systems made up of specific activities and actions mean I conserve willpower. I don’t need to think about how I’m going to achieve my goals, I just implement the system and trust that it’s moving me in the right direction.

    So here’s the thing. If you’re thinking about some things you’d like to achieve in the New Year… Rather than writing a list of goals, hopes or dreams – try to think about what activities you’re going to need to perform that are going to keep YOU moving in the direction of your goal.

  • The Testing Hero’s Journey

    The Testing Hero’s Journey

    The video of my 2020 Romanian Testing Conference talk, The Testing Hero’s Journey, can be found below. Full disclosure, the talk was originally planned as a 2hr workshop. But then COVID happened. So it got compressed down to a 30min talk instead.

  • Thinking About Product Strategy: Processing Signals from the Changing World

    In my last entry I had narrowed down my view of the Changing World (insofar as I have modelled it) such that it looked like this:

    getting granular with testing

    And what I had established was that in order to meaningfully stay up to speed with changes in the world, you have to place some constraints upon the scope of what you’re going to look at – because otherwise there’s simply too much stuff going on and you’ll be overwhelmed by it all.

    So for my purposes, I want to focus primarily on the software testing and test management industry, which is a sub-class of the software industry:

    the software testing industry is a subclass of the software industry

    That industry (or marketplace in which I’m interested) is comprised of the set of customers and vendors that operate within it. The stuff that I am interested in are the additional factors which may influence activity within the marketplace or industry:

    influences within the marketplace of interest

    Clearly there could be a great number of other factors to take into account, but when modelling you have to stop somewhere, right?

    From my model, it seems apparent that there are three main areas on which I can focus in order to build a picture of what’s happening in the marketplace in which I am interested:

    1. Customers
    2. Competitors
    3. Other influences

    So, how do I find out relevant information about those areas?

    Customers

    For customers, there’s a simple but not easy answer… Talk to them!

    You’d imagine that this would be easy enough. But the challenge I personally experience (and some other PM’s may identify with this) – is that it’s actually quite tough to pin them down to a phone call or a Zoom discussion. And having a one-on-one meeting with the customer is by far the most useful kind of interaction in my experience.

    Other forms of customer interactions are usually by way of surveys, or may be in the form of feedback from other business departments (Customer Support, or Customer Success, or Sales typically) or from other situations such as conferences, meetups, webinars and the like.

    Depending on the technology stack, there’s also the possibility of feedback from within the product itself, by way of user-tracking or other forms of monitoring.

    So, signals from the customer (for me) look like this:

    • Direct interactions (meetings on the phone or Zoom)
    • Survey feedback (NPS or other survey types)
    • Feedback from busines channels
    • Event feedback
    • In product monitoring

    Competitors

    The next big area to try and understand is what competitors are doing in the marketplace.

    For me, this is even tricker and time-consuming than trying to elicit information from customers. Generally speaking, customers are pretty happy to tell you what they’re thinnking about and will often do so in no uncertain terms! Customers have a vested interest in improving the product, particularly if they have already parted with their cash.

    Competitors on the other hand – not so much! They will actively try to hide information so as not to broadcast their product strategy and intent.

    Fortunately, there are some relatively well establiushed mechanisms for analysing competitors in order to glean needed information. Some sources of useful data include the following:

    • Industry publications
    • Case studies
    • Corporate info aggregation sites such as Owler, or Hoovers
    • Press releases
    • Company blogs
    • The competitor product itself (through analysis or reverse engineering)

    Once all that information has been gathered, you can start to turn it into a SWOT (Strengths, Weaknesses, Opportunities, Threats) model. There’s any number of resources on the interweb about what a SWOT is and how to do it, so I won’t dwell to much on it here. Except to mention that once you’ve gathered the necessary SWOT information about any competitors, it can be a good idea to consolidate it into a single view of all that data, so that you can use it formulate attack and defend vectors, as well as to identify potential opportunities (per Steven Haines PM Desktop Reference):

    consolidation of SWOT data to identify attack and defend vectors

    Which makes a lot of sense to me, hence reproducing the model.

    Furthermore, Haines goes on to recommend an additional series of questions for delving deeper into competitor operations:

    • How is the competitor company operated?
    • How does the competitor actually produce their product?
    • Via what channels does the competitor distribute their product?
    • By what means does the competitor promote and sell their product?
    • How does the competitor service and support their customers?
    • What technologies are primarily used in the competitor product?
    • What does the employee situation or culture look like for the competitor?
    • What (if anything) are they communicating to any pertinent regulatory or government bodies?

    Other influences

    There’s a final area, other external influences, which warrants at least a little bit of attention. There’s not really too much I can say about this though, other than to pay attention to the world around your area of focus (remember the earlier narrowing down of that area) in as many ways as makes sense to you.

    Speaking personally, I’m a bit of an information hoover, and will suck up information from anywhere I can find it. But as mentioned previously, that comes at the risk of overwhelm. The challenge is knowing when to stop. Which is what, I hope, the development and application of my model will help me with – once I’ve refined it some more.

    Unfortunately, one thing it won’t help me with, is time.

    Specifically, finding time to do all of the research implied by the various activities above, while still delivering on all the other PM activities expected from me…

  • The Big Leap

    In his book The Big Leap, Gay Hendricks recommends making the jump from your Zone of Excellence to your Zone of Genius. I think this is great advice, so I thought I’d share it, along with some tips for doing so.

    Hendricks basically identifies four modes of living that we move between at various stages or our life and careers.

    1. The Zone of Incompetence – made up of all the activities you’re not good at.
    2. The Zone of Competence – made up of all the activities you’re competent at, but that others can do equally as well a you can.
    3. The Zone of Excellence – activities you can do extremely well and probably make a good living at.
    4. The Zone of Genius – activities which you are uniquely suited to do, as a result of your personality, temperament, DNA etc.

    Most of us who are reasonably successful in our respective careers will have progressed beyond our Zone of Competence into a Zone of Excellence, where we’re utilising skills either developed over the course of time or to some extent, leveraging natural talents and abilities. There’s a good chance that we’re also being paid fairly well for doing so, which is where the danger of staying within our Zone of Excellence creeps in.

    Life just gets a bit too comfortable in the Zone of Excellence. And not just for us [who are operating within it] either. It’s quite possible that family and friends will have a interest in maintaining the status-quo, because while you remain in this space you can provide them with the lifestyle and relationships they’ve become accustomed to.

    Hendricks asserts that making The Big Leap into our Zone of Genius is the “ultimate path to success and satisfaction, and provides the following questions as tools for thinking about what this actually means:

    • What do I love most to do? (I love it so much I can do it for long stretches of time without getting tired or bored.)
    • What work do I do that doesn’t seem like work? (I can do it all day long without ever feeling tired or bored.)
    • In my work, what produces the highest ratio of value and satisfaction to the amount spent? (Even if I do only ten seconds or a few minutes of it, an idea or a deeper connection may spring forth that leads to huge value.)
    • What is my unique ability? (There’s a special skill I’m gifted with. This unique ability, fully realised and put to work, can provide enormous benefits to me and any organisation I serve.)

    I’ve found it massively helpful to work through the questions above and they’ve prompted some surprising answers. I’m quite fortunate in many ways, since the work I love to do is mostly the work I’m already doing. But there are certainly some adjustments that could and should be made. Making those adjustments requires some determination though, since Hendricks suggests that many of us will encounter what he calls The Upper Limit Problem just as we’re rising to success.

    The Upper Limit Problem is typically caused by false beliefs that can often be traced back to early-life experiences:

    • Feeling fundamentally flawed – I can’t expand to my full creative potential because something is fundamentally wrong with me.
    • Disloyalty and abandonment – I can’t expand to my full potential because it would leave me feeling all alone, or would mean being disloyal to my roots or people from my past.
    • Believing that success results in bigger burdens – I can’t expand to my potential because I’d be a bigger burden [to people I care about] than I am now.
    • The crime of outshining – I mustn’t expand to my full potential because if I did I would outshine /<insert name[s] here/> and make them look/feel bad.

    The Upper Limit Problem manifests itself in behaviours that cripple us when we should be riding high:

    1. Worry
    2. Criticism & Blame
    3. Deflecting
    4. Squabbling
    5. Getting sick or hurt
    6. Breaches of integrity

    These are problems that I’m sure many, if not all of us have experienced at times during our lives, both in and out of work. Of course, Hendrick provides some suggestions and tools for dealing with these issues, but above all he recommends an attitude of lighthearted wonder towards our personal shortcomings that resonates deeply with me – perhaps because I recognise so many of them!

    Being a fairly skeptical kind of person (hey, I’m a tester!) I’m naturally suspicious of this kind of material, but it has always made a lot of sense to me personally to keep learning and striving to be better than I am, both at work and in life generally. As a result of reading this book I have a renewed zeal for dealing with the false beliefs and behaviours that cause The Upper Limit problem, and a deep commitment to rising out of my Zone of Excellence (and all the creature comforts that come with it), and into my Zone of Genius.

    I’m ready to make my Big Leap. Are you?

We use cookies in order to give you the best possible experience on our website. By continuing to use this site, you agree to our use of cookies.
Accept
Privacy Policy