Software Development

Bug Report

I just submitted my favorite bug report ever. Since I regularly write bug reports for work, that’s really saying something.

One of the items in the Landmark patch notes today was “Added a feedback message to the player when they submit a bug. Now you know when it successfully sends the bug!”

Well, this evening I came across a bug, and submitted it (like the good little tester I am), and was disappointed to see that I did not receive a bug submission confirmation.

So naturally I proceeded to report another bug: “I submitted a bug report, and didn’t receive the new message confirming that the bug report had been submitted, so I am now submitting a bug report about bug report submission. :)”

When I read this to Hubby, I couldn’t stop laughing… but then a friend asked if I got a message confirming my bug report about not receiving a message about bug reports. And alas I didn’t.

Hopefully the bug is with the confirmation message for the submission of bug reports, and not the bug report submission itself, because it would be a shame if my bug report didn’t make SOMEONE laugh.

Report Testing Guide for End Users

This is a basic testing guide that I prepared after searching fruitlessly for a guide to give the system users at work. Originally posted here.

Please feel free to share your thoughts about how this can be improved, expanded, made more generic, adapted for other uses, etc.


Why do I need to test?

Testing is essential to making sure that any report or system enhancement is accurate and works as expected. The IT team tests as thoroughly as possible, however there are several reasons that testing should be done by the end user as well:

  1. You are much more familiar with your piece of the system than IT is. SAP is a new system for all of us. While you have been learning how your processes function in SAP, we have been providing cut-over support, learning the data structures, and learning how to look up limited pieces of the data.

  2. YOU are the expert in your area of the business. The IT team has limited familiarity with how you use the system. We are constantly learning where various data elements are and how things interact, but we don’t yet know enough about the operations of any business area to consider ourselves experts. YOU are the expert.

  3. Customer satisfaction is important, and YOU are our customer. You are our customer, and it is important to us that the solutions that we provide meet your expectations and needs.

Types of Testing

Testing can vary quite a bit in complexity depending upon the project and the stage of development. There are three essential types of testing for reports:

  1. Interface Review: Validating that the search criteria and columns included in the report are correct and working as expected.

  2. Data Validation: Detailed review of the data presented in the report in comparison to the source system (SAP) to identify any fields that may be incorrect.

  3. Usability Testing: Testing the report as you would normally use it. This testing can help uncover issues in presentation or in the data included in the report that were overlooked in the Interface review and Data Validation.

Identifying Errors

If any problems are found encountered for any of the tests, they need to be documented and sent to the IT team for review.

For problems encountered during Interface Review:

  1. The exact search criteria used
  2. If you received an error: what that error was and instructions for how to reproduce the error. (Screenshots are always welcome!)
  3. If there was a problem with the results, details about how the results differed from expectation.

For problems encountered during Data Validation:

  1. Example transaction
  2. Which column is incorrect
  3. The incorrect data shown
  4. What the correct data should actually be
  5. Where to see the correct data in SAP.

For problems encountered during Usability Testing:

  1. If a data issue is found, then provide the information listed under Data Validation.
  2. If another issue is found, then outline what the issue is and the requested change.

Report Testing Checklist

Interface Review:

#

Test

Pass/Fail

Problems Encountered

1

Are all required columns present?

2

Are all column names correct?

3

Is the sort order for the results correct?

4

Are all required search criteria present?

5

Are all search criteria names correct?

6

Are default search criteria appropriate?

7

Do all search criteria work properly?

8

Are there any other issues or changes needed?

Data Validation:

The following apply to EVERY column.

#

Test

Pass/Fail

Problems Encountered

1

Is the data format correct?

2

Is there data in every column that should contain data?

3

Validate the data shown on the report: If the report contains multiple types of transaction (for example: notification types or billing document types), check at least two of each. Compare the report to what is shown on the transaction(s) in SAP.

4

Validate calculations: If the report contains any calculations (sub-total, days past due, etc), validate that against the other data on the report and against data from SAP.

5

If this report replaces an existing report or data pull that you use, then compare results between the two reports. Do you notice any issues?

6

Are there any other issues or changes needed?

Usability Testing:

Consider how you would use the report in the normal course of business, then test the report as if it were considered live.

Warning: Do not share results from the report or base decisions upon the report unless you have completed Data Validation testing without any issues.

#

Test

Pass/Fail

Problems Encountered

1

If you would provide the report to a customer, then export it and prepare it for the customer. Do you notice any issues?

2

If you would use the report for operational analysis, then export it and work through a sample analysis. Do you notice any issues?

3

If you would use this report to review data periodically (such as checking on the status of a sales order or work order), then use it as needed for a day or two. Do you notice any issues?

Sign Off

When the report has passed all of your tests, it will be ready to be made live. Ultimately, you are as responsible as IT to ensure that your report is accurate prior to us making it live.

Before the report is considered live, we will ask you to approve making it live via email. This will be considered your Sign Off that the report is complete and accurate.

Creative Commons License
End User Testing Guide for Reports by Amanda Thomas is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

An end user testing guide

One of the most interesting (and often frustrating) changes at work since the merger completed has been trying to help the users understand that I am no longer an expert in our systems and processes. It is a complete shift from what we are all used to.

For the past couple months we’ve been working on building reporting with SAP data to replace the reporting that we had before the merger. I’ve learned a LOT about the data and have honed my Google-fu (so much that one of the experienced SAP users recently referred to me as an SAP wizard when I was able to tell him which tables to look to for descriptions of sales document types and billing document types).

The challenge has been testing. Since I used to know the system and processes so well, I used to do the majority of the testing for new reports or when we made system changes. When end users tested system changes, I would give them a checklist of specific things to test for, types of transactions to test with, and types of issues to look out for. When they tested reports, it was mostly about making sure the columns and search criteria they needed were there. But in both cases they didn’t need to test extensively because I’d already worked with the dev team to eliminate every bug I could find (and I am damn proficient at finding bugs, if I do say so myself).

A couple of weeks ago we got to the point where we’d  developed a bunch of reports, most of them were in various stages of testing but a few had been tested by the users, had bug fixes, and been made live. And for two in particular several rounds of fixes had happened after making them live.

I needed to find a quick and relatively hands-off way of getting the users equipped to test reports. It needed to be quick and hands-off because I had ten reports in development with at least as many people (in three time zones) involved in testing, and because I was having a hard time keeping up with the dev team as it was.

After searching for a quick user testing guide with no luck, I decided to make my own. And I decided since this appears to be something that is lacking in general, I thought I’d share.

So without further adieu, here is a quick testing guide for reports (that can be easily tweaked for other testing as well).

UPDATE 13 May 2013: This has received a bit more traffic than I’d anticipated, so I’ve also created a page dedicated to the testing guide. If you have comments, recommendations, questions, etc, please post them on that page. Any future revisions will be made there.


Why do I need to test?

Testing is essential to making sure that any report or system enhancement is accurate and works as expected. The IT team tests as thoroughly as possible, however there are several reasons that testing should be done by the end user as well:

  1. You are much more familiar with your piece of the system than IT is. SAP is a new system for all of us. While you have been learning how your processes function in SAP, we have been providing cutover support, learning the data structures, and learning how to look up limited pieces of the data.

  2. YOU are the expert in your area of the business. The IT team has limited familiarity with how you use the system. We are constantly learning where various data elements are and how things interact, but we don’t yet know enough about the operations of any business area to consider ourselves experts. YOU are the expert.

  3. Customer satisfaction is important, and YOU are our customer. You are our customer, and it is important to us that the solutions that we provide meet your expectations and needs.

Types of Testing

Testing can vary quite a bit in complexity depending upon the project and the stage of development. There are three essential types of testing for reports:

  1. Interface Review: Validating that the search criteria and columns included in the report are correct and working as expected.

  2. Data Validation: Detailed review of the data presented in the report in comparison to the source system (SAP) to identify any fields that may be incorrect.

  3. Usability Testing: Testing the report as you would normally use it. This testing can help uncover issues in presentation or in the data included in the report that were overlooked in the Interface review and Data Validation.

Identifying Errors

If any problems are found encountered for any of the tests, they need to be documented and sent to the IT team for review.

For problems encountered during Interface Review:

  1. The exact search criteria used
  2. If you received an error: what that error was and instructions for how to reproduce the error. (Screenshots are always welcome!)
  3. If there was a problem with the results, details about how the results differed from expectation.

For problems encountered during Data Validation:

  1. Example transaction
  2. Which column is incorrect
  3. The incorrect data shown
  4. What the correct data should actually be
  5. Where to see the correct data in SAP.

For problems encountered during Usability Testing:

  1. If a data issue is found, then provide the information listed under Data Validation.
  2. If another issue is found, then outline what the issue is and the requested change.

Report Testing Checklist

Interface Review:

#

Test

Pass/Fail

Problems Encountered

1

Are all required columns present?

2

Are all column names correct?

3

Is the sort order for the results correct?

4

Are all required search criteria present?

5

Are all search criteria names correct?

6

Are default search criteria appropriate?

7

Do all search criteria work properly?

8

Are there any other issues or changes needed?

Data Validation:

The following apply to EVERY column.

#

Test

Pass/Fail

Problems Encountered

1

Is the data format correct?

2

Is there data in every column that should contain data?

3

Validate the data shown on the report: If the report contains multiple types of transaction (for example: notification types or billing document types), check at least two of each. Compare the report to what is shown on the transaction(s) in SAP.

4

Validate calculations: If the report contains any calculations (subtotal, days past due, etc), validate that against the other data on the report and against data from SAP.

5

If this report replaces an existing report or data pull that you use, then compare results between the two reports. Do you notice any issues?

6

Are there any other issues or changes needed?

Usability Testing:

Consider how you would use the report in the normal course of business, then test the report as if it were considered live.

Warning: Do not share results from the report or base decisions upon the report unless you have completed Data Validation testing without any issues.

#

Test

Pass/Fail

Problems Encountered

1

If you would provide the report to a customer, then export it and prepare it for the customer. Do you notice any issues?

2

If you would use the report for operational analysis, then export it and work through a sample analysis. Do you notice any issues?

3

If you would use this report to review data periodically (such as checking on the status of a sales order or work order), then use it as needed for a day or two. Do you notice any issues?

Sign Off

When the report has passed all of your tests, it will be ready to be made live. Ultimately, you are as responsible as IT to ensure that your report is accurate prior to us making it live.

Before the report is considered live, we will ask you to approve making it live via email. This will be considered your Sign Off that the report is complete and accurate.

Creative Commons License
End User Testing Guide for Reports by Amanda Thomas is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

In their shoes

I’ve really been enjoying work recently.

Last week I got to move from a desk in the customer service area to one in the accounting area. I decided to move because the noise in the customer service area made it very difficult for me to focus. My new desk is in an area with much less foot traffic so I also have fewer unplanned interruptions.

As a bonus, I’m sitting next to a window again, this time on the second floor so I get a nice view of the wetland across the street. Since I moved, I’ve accomplished about twice as much as usual and I’m just generally happier.

Timing couldn’t have been better either. I’m always super busy at work, but in the next few months we have more large projects planned than we did for all of 2010.

I am excited. The type of work we have planned for the next few months is what I love: system implementations and large custom development projects (by large I mean likely to take at least several weeks, I know that isn’t “large” to some). Both of these mean I get to do more analysis, requirements gathering, documentation, and project management than I have for the past couple years, and as a result mean I will be more entertained at work.

One of the implementation projects is to take a piece of software we’ve been using in some of our locations and implement it in our office in England. This software is one that I am very close to; We had another company develop it for us, but I led the team internally for requirements gathering, testing, and implementation.

Even though I know it would be a challenge for me to keep up with my user support role, continue pushing smaller projects through, and project manage this implementation, I was really looking forward to it.

But then today I learned that I may not get to play the role in this implementation that I’d hoped for. My managers are considering having a project manager that we’re working with on another project handle a large portion of the project management for this implementation as well.

They have all the right reasons, and if I were in their shoes I would do the same. Hell, six months from now I might really appreciate that someone else got the assignment.

But not now. Right now I’m just disappointed.

Still looking for that reset button.

Today is already getting better.

Earlier today I tweeted about my day being off to a bad start. It was. I woke up feeling hung over and aching all over (even though last night I only drank water). I had one co-worker upset about a change request I’d sent them after testing an update to one of our internal systems. I’m sure I annoyed the heck out of another with a few emails for little changes on a different project (fortunately, the second co-worker didn’t complain).

(more…)

A glimpse into Sony Online Entertainment

Three weeks ago I made a spur of the moment decision to spend some time in San Diego during my vacation. After I booked my trip (Air miles + PriceLine.com = about $300 for flight plus three nights at a four star hotel), I decided to try to make visiting Sony Online Entertainment (SOE) a reality and the next day I had an appointment to meet Ashlanne during my trip. I was so ecstatic that I did a little dance at my desk (my co-workers thought I had finally gone insane). 

SOE LobbyOne week later, I took a cab from my hotel in the Gaslamp Quarter to the SOE office. I was paranoid about being late and nervous as if I was going to an interview. I arrived half an hour early and fidgeted in the lobby while waiting. Ashlanne was ready for me a few minutes later. 

First we visited the Community Team’s area. I met several members of the community team (advance apologies if I get names mixed up!). Zatozia’s cube is decorated like a torture chamber. I’d seen at least one picture before but it didn’t do it justice (I regret not snapping a couple of my own). 

(more…)

Days like today…

Warning: This post contains venting and lacks quality.

I really dislike the first work day of the month. I work in IT, and every month we have some frustrating system issue come up. Some months they’re expected others they’re surprising. Some months they are easy to fix, others they take a lot of time.

For the past several months without fail we’ve had invoicing issues, but we went into work on April first expecting that we wouldn’t have a problem. We’d found the cause of the problem and fixed it. My husband and I were even joking about sending our boss (we work together) an email from the system saying that invoicing had not worked. It was April Fools day after all.

(more…)