Though we take painstaking effort to maintain the utmost quality on our side, there are things you should keep in mind when writing your tests that will also help keep results quality high and consistent.
The number one driver of consistency and quality is step ambiguity. While there will always be some small chance for human error, in nearly every case of inconsistent or incorrect results we come across, the root cause was step ambiguity.
The good news is it's totally fixable!
What is step ambiguity?
When steps are unclear or open to several interpretations they cause problems for our testers. This is because they may or may not correctly guess which interpretation the test writer had in mind when they wrote the test.
The most common culprit is internal jargon and language —shorthand—that you use when communicating within your team. Words like 'rollbar', 'cta' and questions like 'is the button where it's supposed to be?' or 'does it look right?'. Testers don't have the context your team has, and so have no way to tell how a thing is supposed to look.
Another impact of step ambiguity is that our fraud detection algorithm becomes less effective. This is because we rely heavily on tester quorum to detect quality of results—in other words, if three testers agree on the same outcome, then it's likely to be reliable.
When testers are confused about what they are being asked, the quorum system breaks down because they aren't necessarily answering the same questions. This means that the core signal that the fraud detection algo listens to, quorum (aka. agreement between testers) is now unreliable.
This can lead to fraudulent results getting through to you when they otherwise wouldn't. Fixing the core issue of ambiguous steps will fix this problem.
To fix this problem, refactor steps to be unambiguous to someone who has never seen your product before. This requires you to think through how a new user would see your app, and is a great time to use Preview.
Previewing a test shows things from the tester's perspective, so it's easy to figure out which parts of a step don't make sense to an outsider.
That's it! If you have trouble figuring this out, email the team at firstname.lastname@example.org.