Rules

Secret ingredients to quality software

Edit
Info

Rules to Better User Acceptance Tests (UAT) for Bug Management

13 Rules

Since 1990, SSW has supported the developer community by publishing all our best practices and rules for everyone to see.

If you still need help, visit SSW Consulting Servicesand book in a consultant.

  1. Do Your User Stories Include Acceptance Criteria? (aka Never assume automatic Gold Plating)

    Most teams are getting the hang of User Stories that have subtasks. Unfortunately the same can’t be said about acceptance criteria.It is so important because real user stories tell a team when the task is done.

    Also, Product Owners should not get heartburn because ‘obvious’ functionality was not included. All requirements should be specified in the Acceptance Criteria.

    For example, Product Owners should not assume things like:

    • they will get a message that says ‘no records found’ or
    • the grid will support features such as pagination or sorting

    They must be specified in the Acceptance Criteria if required.

    There are 2 parts to getting this right. The Acceptance Criteria, then the Acceptance Tests:

    DevsAndUsers
    Figure: You need a common language to communicate in

    Acceptance Criteria (from the Product Owner) define the exact requirements that must be met for the story to be completed. They answer the question, "How will I know when I' m done with the story?"

    acceptance criteria
    Figure: A User Story with Acceptance Criteria (MSF Agile Template)

    When I enter ‘Adam’ in the search box and click 'Search' I will see all entries starting with 'Adam' in the grid.

    Figure: Bad Example of Acceptance Criteria - Incomplete

    Positive Test -When I enter ‘Adam’ in the Search box and click ‘Search’ I will see all entries starting with Adam in the Grid Negative Test - When I enter ‘zzz’ in the Search box and click ‘Search’ I will see *no* entries in the Grid

    Figure: OK Example of Acceptance Criteria

    Positive Test -When I enter ‘Adam’ in the Search box and click ‘Search’ I will see all entries starting with Adam in the Grid Negative Test - When I enter ‘zzz’ in the Search box and click ‘Search’ I will see *no* entries in the Grid Gold Plating - If no results are retuned show a message box ‘No results found’ Gold Plating – Validation: If no search text is entered, the ‘Search’ button should be disabled Gold Plating – Right-clicking on a column header should provide ‘Sort’ functionality Gold Plating – if a large set of results is returned, display pagination with page numbers and ‘Prev’, ‘Next’ links

    **Figure: Good Example of Acceptance Criteria – Including Gold Plating**For tiny stories, you can omit acceptance criteria. Sometimes you just need a screenshot, or even better a video.

    See screenshot and video. Screenshot – see PDF Video - http://ww w.youtube.com/watch?v=M3FH4D9QuzU

    Figure: Good Example for a simple change – just include a screen capture and/or video

  2. Do you create one Test Plan per Sprint?

    When you use Microsoft Test Manager to create test plans and test suites for your team project, there are several approaches that you can take.

    project test plan bad
    Figure: Bad Example – Only create 1 test plan that you use for all milestones. Add test suites and tests as you progress. This is bad because f you use this approach, you do not have historical data for your test pass rates for previous milestones

    project test plan good
    Figure: Good Example - Create test plans based on your testing goals for a specific sprint

    By creating test plans for each sprint, you can see when a sprint is complete, based on your testing goals. You can also prepare the test plan for the next sprint while you complete your testing for the current sprint.By using this approach, you can track your testing progress for each of your test plans and see that the quality of your application is improving.

    Tip: If you add both manual and automated tests to your test suites, you can view the overall quality based on both of these types of tests for your test suites and test plans.

    Reference: Guidance for Creating Test Plans and Test Suites

  3. Do you know how to create a Test Case with TFS.VisualStudio.com (was TFSPreview)?

    Follow these steps to create a Test Case in TFS.VisualStudio.com:

    create tc 1
    How to create a test case
    Figure: Double click the Product Backlog Item that you want to create a Test Case for to open it
    create tc 2
    How to create a test case
    Figure: Open the "TEST CASES" tab and click on the "New linked work item" button
    create tc 3
    How to create a test case
    Figure: Ensure that the link type is 'Tested By', that the work item type is 'Test Case' and enter a title for the Test Case. Click OK.
    create tc 4
    How to create a test case
    Figure: Select the correct iteration, and update the Status and Details sections. Click on the 'Click here to add a step' and proceed to add the steps required to test the user story
    create tc 5
    How to create a test case
    Figure: After entering each action, along with its expected result, click Save and Close

  4. Do you know how to assign a tester to test configurations?

    This is how assign a tester to test configurations:

    assign tester 1
    How to assign a tester
    Figure: From the Plan menu select the Test Plan, click on the Test Case and then click the Assign button
    assign tester 2
    How to assign a tester
    Figure: Select the rows to assign (Hold down [Ctrl] to select multiple), choose a tester from the available list and click on the Save Assignments button

  5. Do you know how to configure which environments to use for a particular test?

    This how you configure which environments to use for a particular test:

    particular test 1
    How to add a test case to a test plan
    Figure: From the Plan menu choose the Test Suite. Click on the test plan and then the Configurations button
    particular test 2
    How to add a test case to a test plan
    Figure: To view the available configurations, click in the configurations column for the test and then select the arrow at the end of the field. Select configurations and click the Apply button

  6. Do You Run Acceptance Tests?

    Once the coding is done by the developers, the functionality must then be stepped through in the required browsers. You can do this manually or automating it using a great tool like Microsoft Test Manager.

    The 1st step in getting automated tests, is to setup Acceptance Tests:

    run acceptance tests 1
    Figure: Run each 'test case' with a prescribed configuration

    run acceptance tests 2
    Figure: As you progress through each step, 'Pass' or 'Fail' the expected results. Take screen captures or video as appropriate

    run acceptance tests 3
    Figure: Bad Example -After checking all the ‘Expected’ results in your MTM test, do not forget to 'Pass' or 'Fail' the Test Case

    run acceptance tests 4
    Figure: Good example - After all 'Test Steps' have been checked off, choose the overall status for the test. Otherwise it will continue to show as 'Active' on the reports

    run acceptance tests 5
    Figure: Bad Example – No Tests should remain as 'Active' or 'Failed' at the end of a Sprint

    run acceptance tests 6
    Figure: Good Example – every test is 'Passed'

    Tip: You can pass a test from the test list. Select the Test menu, then the Test Suite. Choose the Test Case to pass and then click the green button ‘Pass Test’.

    The next step is to review the Statistics of the Sprint.

  7. Do you know how to check the status and statistics of the current Sprint?

    Developers think they are done when they finish coding and check in.

    Wrong. It is much better to use Microsoft Test Manager (MTM) and step through the Acceptance Tests.

    Once you are doing that, this is how you check the status of the current Sprint:

    check sprint status
    Figure: Good example - This Sprint currently has 2 'Failed' tests (red), and 1 'Active' test (blue). (This 'Results' view is new in MTM 2012)

    Key:

    • The red is work remaining for the developers, and
    • The blue is working remaining for the testers (unfinished testing)
  8. Do you write acceptance tests to verify Acceptance Criteria?

    Acceptance Tests (built by the developers) verify that the Acceptance Criteria are met.

    The goal is for teams to move beyond manual testing and implement automated testing. E.g. CodedUI tests, Telerik Tests etc

    Test cases answer the question, "How do I test and what are the test steps?"

    acceptance criteria test cases
    Figure: Test Cases in a User Story (MSF For Agile Template)

    Positive Test - When I enter ‘Adam’ in the Search box and click ‘Search’ I will see all entries starting with Adam in the Grid
    Negative Test - When I enter ‘zzz’ in the Search box and click ‘Search’ I will see no entries in the Grid
    Gold Plating - If no results are retuned show a message box ‘No results found’
    Gold Plating – Validation: If no search text is entered, the ‘Search’ button should be disabled
    Gold Plating – Validation: If the button is disable and search text is entered, the ‘Search’ button becomes enabled
    Gold Plating – Right clicking on a column header and using the ‘Sort’ functionality, sorts the data by that column
    Gold Plating – if a large set of results is returned, clicking the pagination page numbers shows the correct data
    Gold Plating – if a large set of results is returned and we are on page > 1, clicking the ‘Prev’ button goes to the previous page
    Gold Plating – if a large set of results is returned and we are on page 1, ‘Prev’ button does not error
    Gold Plating – if a large set of results is returned and we are on page < MaxPage, clicking the ‘Next’ button goes to the next page
    Gold Plating – if a large set of results is returned and we are on page = MaxPage, clicking the ‘Next’ button does not error

    Figure: Good example - Acceptance Tests

    test cases
    Figure: The tester sees the Test Cases in Test Manager

    test steps
    Figure: The tester follows each instruction (aka the Test Steps), and gives it a tick or cross

  9. Do you use Exploratory Testing to create Acceptance Tests?

    In an agile team, pre-planning all your tests is not always the most efficient use of time for testers. PBIs can change direction, scope, and priority, and pre-planned tests are likely to change.

    Exploratory testing provides the best way to create repeatable tests from the acceptance criteria - as you need them.

    There are two ways to run an exploratory test in Microsoft Test Manager.

    exploratory 2
    Figure: Bad Example - go to the Test tab, choose Do Exploratory Testing, choose a PBI, then click Explore. Too many steps

    exploratory 1
    Figure: Good Example - Right-click on a requirement in your test suite and choose "Explore requirement"

    Note: You should always run an exploratory test against a PBI. This will automatically relate any bugs and test cases to that PBI (not to mention the exploratory test run).

    When you start an Exploratory test, you don't see any test steps, but you can click on the title of the requirement to see its Acceptance Criteria.

    show criteria
    Figure: Clicking on the title will show you the Acceptance Criteria

    If you find a bug while testing, click the Create bug button to add a bug related to the PBI.

    create bug
    Figure: Creating a bug from exploratory test links to the PBI

    By default, the reproduction steps will be populated with the last 10 actions you took (you can change this and other defaults with configuration). You can cut this down to just the relevant actions by clicking Change steps.

    change bug steps
    Figure: You can change the repro steps captured in the bug very easily

    Now you have a bug, you should create a matching test case so you can verify when the bug is fixed. This also gives you a handy regression test to help ensure the problem isn't reproduced later.

    save create test
    Figure: Click Save and create test to create a matching test case

    Again, the steps are prepopulated from your bug steps.

    create test
    Figure: The test steps are prepopulated from the action recording

  10. Do you do exploratory testing?

    Use Microsoft's Exploratory Testing - Test & Feedback extension - to perform exploratory tests on web apps directly from the browser.

    Capture screenshots, annotate them and submit bugs as you explore your web app - all directly from Chrome (or Firefox) browser. Test on any platform (Windows, Mac or Linux), on different devices. No need for predefined test cases or test steps. Track your bugs in the cloud with Visual Studio Team Services (VSTS).

    **Ravi walks Adam through the exploratory testing extension - You can alsowatch on SSW TV **

    **Ravi Shanker and Adam Cogan talk about the test improvements in Visual Studio Team Services and the Chrome Test & Feedback extension - You can alsowatch on SSW TV **

    Official video from Microsoft Visual Studio channel

    ****

    1. Go to Visual Studio Marketplace and click install.

    exploratory test extension install
    Figure: Microsoft Test & Feedback (was Exploratory Testing) extension

    1. Click "Add to Chrome" to add the extension to the browser on your computer.

    exploratory test extension add
    Figure: Chrome Web Store page for Test & Feedback extension

    1. Go to Chrome.
    2. Start a session by clicking on the Chrome extension and then click start a session.

    exploratory test extension icon
    Figure: Chrome extension icon

    exploratory test extension start
    Figure: Test & Feedback start session button

    1. Upload the screenshot to a PBI.

    pbi screenshot
    Figure: PBI in Visual Studio Team Services (VSTS) showing the screenshot

    Related Links

  11. Do you know how to add a Test Case to a Test Plan in Microsoft Test Manager?

    You organize your Test Cases by adding them to a Test Plan (also called a Test Suite).

    We have one Test Plan per sprint.

    add tc to tp 1
    How to add a test case to a test plan
    Figure: Select Plan from the main menu, and then choose the Test Plan that you would like to add Test Cases to
    add tc to tp 2
    How to add a test case to a test plan
    Figure: Add a filter to make it easy to find your Test Cases. Click the Run button, select the relevant items and click the 'Add Test Cases' button
    add tc to tp 3
    How to add a test case to a test plan
    Figure: Your Test Case has now been added to the suite

  12. Do you know how to run a manual test in Microsoft Test Manager?

    Running tests with MTM allows you to keep track of your testing progress.

  13. Do you use Problem Steps Recorder?

    Problem Steps Recorder is a useful tool that allows anyone to capture details of a bug. Best of all, It's already installed on your Windows PC!

    Bug reports can come from anywhere and anyone - the more people reporting back, the better.

    Once a problem has been discovered, it helps to collect as much information as possible. Although there are many useful tools you can give to your test team, Problem Details Recorder has one big advantage: It's shipped with Windows since Windows 7 so anyone can record details of their issue with no preinstalled software.

    psr1
    Figure: To start Problem Steps recorder, type PSR into the start | run box

    psr2
    Figure: then click 'Start Record'

    Once recording, the user can repeat whatever steps to reproduce an issue.

    psr3
    Figure: User behavior is captured along with full screenshots. This can be saved and attached to a Bug PBI

We open source. This page is on GitHub