Ensure the review quality by randomly inserted test proposals

spockym-INGspockym-ING Posts: 13 ✭✭✭
edited March 2022 in April AMA - 2022

From other platforms where user content is reviewed, I know the concept of showing randomly test questions and show feedback to the user. Feedback is the user passed the test or not and on what criteria he should done his decision. If they fail in several consecutively tests, users are blocked for a short time or must redo a training.

So adapted to wayfarer, that would mean that may be one of 50 reviews shown to the wayfarer is a test, with a clear case how to decide. And after the wayfarer had done the test review, the information that this was a test and how he did.

From the other communities I know this is a great experience to see how you are doing on the reviews. And it is good for the quality in the communities because everybody reviews more focused.

Post edited by NianticTintino-ING on
46
46 votes

New · Last Updated

Comments

  • Testonoga-PGOTestonoga-PGO Posts: 98 ✭✭✭

    They're already doing this, but it's limited to rejections because apparently, they haven't found a way for perfectly valid honeypot submissions. If good examples of eligible Wayspots are already in the game, as they should, the test submissions would be instantly rated as duplicates and that wouldn't make much sense. There would have to be some new hidden mechanics.

  • TWVer-INGTWVer-ING Posts: 792 ✭✭✭✭✭

    Where did you get that idea? The test submissions (or honeypots) are just nominations by players that are pre-reviewed by Niantic. They can be acceptances and rejections. Since nobody had the chance to vote on them yet, they won't show as duplicates.

    Would be great if we would get feedback on those after they have reached a descision. Same for appealed nominations.

  • niktero-PGOniktero-PGO Posts: 369 ✭✭✭✭✭

    This has been brought up before but we have still not seen any movement on this. We do have honeypot submissions but after the fiasco of spring 2020 they have much less affect on your rating. Still waiting on honeypots that tell you after the vote if you passed or failed and why.

    July 2020 AMA

    Do you plan to implement some sort of feedback to help educate reviewers on the reasons the rating dropped? Since reviewers have no idea where they are disagreeing with Niantic's assessments, will you provide feedback on what categories reviewers are in most disagreeing with Niantic?

    Thanks for your feedback, we hear you on the frustration you feel around reviewer ratings. We have some changes in store that will help you to better understand your rating as well as the areas which contributed to changes in your rating. We are currently reviewing the manner in which the Niantic selected nominations are processed in the backend to leverage this system to better provide information to Wayfarers not only on where they diverged from our reviews but also to better educate you on what to do next time. Additionally, we’ve rebalanced the ding against your rating for disagreeing with the Niantic review in order to make it less impactful on your overall rating and are continuing to make adjustments to this formula so it doesn’t feel impossible to improve your rating.

  • MargariteDVille-INGMargariteDVille-ING Posts: 2,846 ✭✭✭✭✭
    edited March 2022

    LOL yet another example of Niantic making promises and not keeping them. Like promising to re-review rejected nominations from before the 2020 criteria clarification. Or promising to give rejection reasons on appeals. Take everything they say with a grain of salt.

    But, on this topic - they can't show reviewers what "pre-reviewed" nomination they saw, and if they disagreed, why. There would immediately be a web page collecting "honeypots". So people would just study that, instead of criteria. Cheaters gonna cheat.

    They could only give us feedback if each "honeypot" nomination was only once.

    Maybe, Niantic could generate "honeypots" by getting 5* and 1* nominations from "Trusted Reviewers"... people with 70+% agreement rate... but no one's perfect, so some of them would be wrong.

  • TWVer-INGTWVer-ING Posts: 792 ✭✭✭✭✭

    Of course the feedback shouldn't be given immediately. It should be given after the nomination has been resolved. At that point, the nomination is no longer used as a honeypot. They can't just keep using the same honeypots. At some point these have to be resolved too. It just takes longer for them to be resolved, because they are shown to more reviewers than nominations that weren't pre-reviewed.

    There are no trusted reviewers, and agreement rate says nothing about reviewer quality. So that is a very bad idea.

Sign In or Register to comment.