End User

Report Testing Guide for End Users

This is a basic testing guide that I prepared after searching fruitlessly for a guide to give the system users at work. Originally posted here.

Please feel free to share your thoughts about how this can be improved, expanded, made more generic, adapted for other uses, etc.


Why do I need to test?

Testing is essential to making sure that any report or system enhancement is accurate and works as expected. The IT team tests as thoroughly as possible, however there are several reasons that testing should be done by the end user as well:

  1. You are much more familiar with your piece of the system than IT is. SAP is a new system for all of us. While you have been learning how your processes function in SAP, we have been providing cut-over support, learning the data structures, and learning how to look up limited pieces of the data.

  2. YOU are the expert in your area of the business. The IT team has limited familiarity with how you use the system. We are constantly learning where various data elements are and how things interact, but we don’t yet know enough about the operations of any business area to consider ourselves experts. YOU are the expert.

  3. Customer satisfaction is important, and YOU are our customer. You are our customer, and it is important to us that the solutions that we provide meet your expectations and needs.

Types of Testing

Testing can vary quite a bit in complexity depending upon the project and the stage of development. There are three essential types of testing for reports:

  1. Interface Review: Validating that the search criteria and columns included in the report are correct and working as expected.

  2. Data Validation: Detailed review of the data presented in the report in comparison to the source system (SAP) to identify any fields that may be incorrect.

  3. Usability Testing: Testing the report as you would normally use it. This testing can help uncover issues in presentation or in the data included in the report that were overlooked in the Interface review and Data Validation.

Identifying Errors

If any problems are found encountered for any of the tests, they need to be documented and sent to the IT team for review.

For problems encountered during Interface Review:

  1. The exact search criteria used
  2. If you received an error: what that error was and instructions for how to reproduce the error. (Screenshots are always welcome!)
  3. If there was a problem with the results, details about how the results differed from expectation.

For problems encountered during Data Validation:

  1. Example transaction
  2. Which column is incorrect
  3. The incorrect data shown
  4. What the correct data should actually be
  5. Where to see the correct data in SAP.

For problems encountered during Usability Testing:

  1. If a data issue is found, then provide the information listed under Data Validation.
  2. If another issue is found, then outline what the issue is and the requested change.

Report Testing Checklist

Interface Review:

#

Test

Pass/Fail

Problems Encountered

1

Are all required columns present?

2

Are all column names correct?

3

Is the sort order for the results correct?

4

Are all required search criteria present?

5

Are all search criteria names correct?

6

Are default search criteria appropriate?

7

Do all search criteria work properly?

8

Are there any other issues or changes needed?

Data Validation:

The following apply to EVERY column.

#

Test

Pass/Fail

Problems Encountered

1

Is the data format correct?

2

Is there data in every column that should contain data?

3

Validate the data shown on the report: If the report contains multiple types of transaction (for example: notification types or billing document types), check at least two of each. Compare the report to what is shown on the transaction(s) in SAP.

4

Validate calculations: If the report contains any calculations (sub-total, days past due, etc), validate that against the other data on the report and against data from SAP.

5

If this report replaces an existing report or data pull that you use, then compare results between the two reports. Do you notice any issues?

6

Are there any other issues or changes needed?

Usability Testing:

Consider how you would use the report in the normal course of business, then test the report as if it were considered live.

Warning: Do not share results from the report or base decisions upon the report unless you have completed Data Validation testing without any issues.

#

Test

Pass/Fail

Problems Encountered

1

If you would provide the report to a customer, then export it and prepare it for the customer. Do you notice any issues?

2

If you would use the report for operational analysis, then export it and work through a sample analysis. Do you notice any issues?

3

If you would use this report to review data periodically (such as checking on the status of a sales order or work order), then use it as needed for a day or two. Do you notice any issues?

Sign Off

When the report has passed all of your tests, it will be ready to be made live. Ultimately, you are as responsible as IT to ensure that your report is accurate prior to us making it live.

Before the report is considered live, we will ask you to approve making it live via email. This will be considered your Sign Off that the report is complete and accurate.

Creative Commons License
End User Testing Guide for Reports by Amanda Thomas is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

An end user testing guide

One of the most interesting (and often frustrating) changes at work since the merger completed has been trying to help the users understand that I am no longer an expert in our systems and processes. It is a complete shift from what we are all used to.

For the past couple months we’ve been working on building reporting with SAP data to replace the reporting that we had before the merger. I’ve learned a LOT about the data and have honed my Google-fu (so much that one of the experienced SAP users recently referred to me as an SAP wizard when I was able to tell him which tables to look to for descriptions of sales document types and billing document types).

The challenge has been testing. Since I used to know the system and processes so well, I used to do the majority of the testing for new reports or when we made system changes. When end users tested system changes, I would give them a checklist of specific things to test for, types of transactions to test with, and types of issues to look out for. When they tested reports, it was mostly about making sure the columns and search criteria they needed were there. But in both cases they didn’t need to test extensively because I’d already worked with the dev team to eliminate every bug I could find (and I am damn proficient at finding bugs, if I do say so myself).

A couple of weeks ago we got to the point where we’d  developed a bunch of reports, most of them were in various stages of testing but a few had been tested by the users, had bug fixes, and been made live. And for two in particular several rounds of fixes had happened after making them live.

I needed to find a quick and relatively hands-off way of getting the users equipped to test reports. It needed to be quick and hands-off because I had ten reports in development with at least as many people (in three time zones) involved in testing, and because I was having a hard time keeping up with the dev team as it was.

After searching for a quick user testing guide with no luck, I decided to make my own. And I decided since this appears to be something that is lacking in general, I thought I’d share.

So without further adieu, here is a quick testing guide for reports (that can be easily tweaked for other testing as well).

UPDATE 13 May 2013: This has received a bit more traffic than I’d anticipated, so I’ve also created a page dedicated to the testing guide. If you have comments, recommendations, questions, etc, please post them on that page. Any future revisions will be made there.


Why do I need to test?

Testing is essential to making sure that any report or system enhancement is accurate and works as expected. The IT team tests as thoroughly as possible, however there are several reasons that testing should be done by the end user as well:

  1. You are much more familiar with your piece of the system than IT is. SAP is a new system for all of us. While you have been learning how your processes function in SAP, we have been providing cutover support, learning the data structures, and learning how to look up limited pieces of the data.

  2. YOU are the expert in your area of the business. The IT team has limited familiarity with how you use the system. We are constantly learning where various data elements are and how things interact, but we don’t yet know enough about the operations of any business area to consider ourselves experts. YOU are the expert.

  3. Customer satisfaction is important, and YOU are our customer. You are our customer, and it is important to us that the solutions that we provide meet your expectations and needs.

Types of Testing

Testing can vary quite a bit in complexity depending upon the project and the stage of development. There are three essential types of testing for reports:

  1. Interface Review: Validating that the search criteria and columns included in the report are correct and working as expected.

  2. Data Validation: Detailed review of the data presented in the report in comparison to the source system (SAP) to identify any fields that may be incorrect.

  3. Usability Testing: Testing the report as you would normally use it. This testing can help uncover issues in presentation or in the data included in the report that were overlooked in the Interface review and Data Validation.

Identifying Errors

If any problems are found encountered for any of the tests, they need to be documented and sent to the IT team for review.

For problems encountered during Interface Review:

  1. The exact search criteria used
  2. If you received an error: what that error was and instructions for how to reproduce the error. (Screenshots are always welcome!)
  3. If there was a problem with the results, details about how the results differed from expectation.

For problems encountered during Data Validation:

  1. Example transaction
  2. Which column is incorrect
  3. The incorrect data shown
  4. What the correct data should actually be
  5. Where to see the correct data in SAP.

For problems encountered during Usability Testing:

  1. If a data issue is found, then provide the information listed under Data Validation.
  2. If another issue is found, then outline what the issue is and the requested change.

Report Testing Checklist

Interface Review:

#

Test

Pass/Fail

Problems Encountered

1

Are all required columns present?

2

Are all column names correct?

3

Is the sort order for the results correct?

4

Are all required search criteria present?

5

Are all search criteria names correct?

6

Are default search criteria appropriate?

7

Do all search criteria work properly?

8

Are there any other issues or changes needed?

Data Validation:

The following apply to EVERY column.

#

Test

Pass/Fail

Problems Encountered

1

Is the data format correct?

2

Is there data in every column that should contain data?

3

Validate the data shown on the report: If the report contains multiple types of transaction (for example: notification types or billing document types), check at least two of each. Compare the report to what is shown on the transaction(s) in SAP.

4

Validate calculations: If the report contains any calculations (subtotal, days past due, etc), validate that against the other data on the report and against data from SAP.

5

If this report replaces an existing report or data pull that you use, then compare results between the two reports. Do you notice any issues?

6

Are there any other issues or changes needed?

Usability Testing:

Consider how you would use the report in the normal course of business, then test the report as if it were considered live.

Warning: Do not share results from the report or base decisions upon the report unless you have completed Data Validation testing without any issues.

#

Test

Pass/Fail

Problems Encountered

1

If you would provide the report to a customer, then export it and prepare it for the customer. Do you notice any issues?

2

If you would use the report for operational analysis, then export it and work through a sample analysis. Do you notice any issues?

3

If you would use this report to review data periodically (such as checking on the status of a sales order or work order), then use it as needed for a day or two. Do you notice any issues?

Sign Off

When the report has passed all of your tests, it will be ready to be made live. Ultimately, you are as responsible as IT to ensure that your report is accurate prior to us making it live.

Before the report is considered live, we will ask you to approve making it live via email. This will be considered your Sign Off that the report is complete and accurate.

Creative Commons License
End User Testing Guide for Reports by Amanda Thomas is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.