This week we decided to redesign our user study to include a second control group.
Control0: In this group, all the answers to StackOverflow questions will be ordered randomly. This will help validate that the user really is knowledgeable enough to reasonably score answers based on usefulnesses. Without this group, it will be hard to determine the credibility of a user’s ratings since the user might be inclined to rate based on his or her perception or anticipation of what the score should be instead of the actual content.
Control0: In this group, all the answers to StackOverflow questions will be ordered randomly. This will help validate that the user really is knowledgeable enough to reasonably score answers based on usefulnesses. Without this group, it will be hard to determine the credibility of a user’s ratings since the user might be inclined to rate based on his or her perception or anticipation of what the score should be instead of the actual content.