Improvement of the evaluation process
WayfarerMSE-PGO
Posts: 1,515 ✭✭✭✭✭
Since we are still dealing with waves of bots in some parts of Germany where submissions are rejected in rows with absolutely stupid reasons, we would like to present our suggestion for improvement here.
We are aware that not everyone will like it, but we think that this way we can train the judges better and also take the wind out of the sails of all the quick clickers.
@NianticGiffard @NianticTintino We would be happy if this feedback is heard and maybe even put into practice.
We noticed that it takes significantly fewer clicks to reject a suggestion than to accept it. When rejecting, you only select the rejection categories and click on send. Whereas if you accept, you have to give stars in each category before you can click submit.
Here we see an opportunity for improvement.
It would make sense if stars had to be awarded in each individual category even in the case of a complete rejection. This takes time and makes life difficult for the quick clickers.
It should also be obligatory not only to look at the photos briefly, but also to click on them and enlarge them. It is often only when you enlarge the images that you notice that they are actually pixelated because they were photographed from the screen, or even come from third-party sources and therefore cannot be completely enlarged.
If that becomes a necessity before you can click "Next," then, at best, evaluators may be pushed to really look at the individual points of the submission.
Because far too often it happens that you formulate an additional explanation and this is ultimately not read completely because the evaluators do not click on the window to read everything.
If an automatic check takes place here to ensure that every point has really been checked, then that could seriously help.
Please think about it and feel free to bring any suggestions for improvement. In any case, it shouldn't stay the way it is now.
Comments
@WayfarerMSE-PGO I understand why you are frustrated but I think that your recommendation would make the situation worse rather than better.
The massive wave of rejections in Germany has been well-documented as being caused by robots rather than human reviewers. If Niantic changed Wayfarer in the ways that you recommend then the bots would just have code added to jump through the additional hoops required in order to reject a candidate. This is probably a few hours of work that someone would do once, and then the rejection wave would continue unabated.
But... what happens to human reviewers? You would be requiring a lot of additional work for them, which would slow people down. Thus, the whole review process would slow down and it would take longer for new submissions to be approved. That may be an acceptable cost if it wasn't for the fact that it wouldn't solve the problem.
However, what would the human experience be like for reviewers? Speaking as someone who has (legitimately) reviewed 22,248 candidates and counting I can tell you that it would be extremely frustrating. I would be wasting a lot of time clicking on things needlessly in cases where the rejection was blatantly obvious from a very quick glance. Imagine that you are reviewing something like one of these:
I have seen everything on that list more than once. If I had to waste a lot of time every time I came across something that was an automatic rejection I would feel like my time was being wasted and I would stop reviewing completely, and I am sure I am not alone in that.
After reading your reasoning, I have to agree with you. That probably wouldn't make it any better in the long run. Nevertheless, I remain of the opinion that something needs to be changed, in whatever way.
@WayfarerMSE-PGO Thank you for listening to a different viewpoint with an open mind! That seems relatively rare in our world today.
We are in complete agreement that Niantic needs to make Wayfarer more resilient to abuse, and as the resident cranky loudmouthed fuchsia-haired old woman[1] on this forum I've been pretty vociferous in pointing out the ways that the system is broken, and even sometimes suggesting mechanisms for improving it. Our only difference of opinion was about the specific solution you proposed, and that's only because I've built my career on the ability to see unintended consequences and ways that systems can be abused and broken.
Kudos to you for raising the issue and adding your voice to the chorus of people calling for better anti-abuse mechanisms.
[1] OK, middle-aged but old makes for a better caricature. (-: