Dear Niantic: Please consider all new reviewers as "Good" for a while

Hosette-INGHosette-ING Posts: 1,191 ✭✭✭✭✭

To the fine product designers and engineers at Niantic:

I've been a denizen of this forum and one on Facebook for quite a while, and I've been a reviewer since the launch of OPR beta in 2016. I've had the opportunity to see a lot of complaints from reviewers, some legitimate and some misplaced, and there are some that are systemic and quite valid. One of the most commonly-repeated and IMO extremely legitimate complaints involves the experience of new reviewers and their rating dropping immediately. I could find lots of examples to back this up, but this recent post contains an excellent illustration of the issue:

This is a very common complaint: "I was excited to start using Wayfarer! I logged in and spent a couple of hours reviewing, and my rating dropped. Why? I was following the guidelines." This is a poor initial experience for new Wayfarers as their initial exuberance is quickly converted into frustration. Even if someone is doing excellent reviewing there is going to be a lot of noise in their rating at first, and they could drop to poor simply because of the order in which their first few reviews resolve. Please consider assigning new Wayfarers a fixed initial rating, perhaps Good, and keep it locked there until they have enough agreements for their rating to be statistically significant. This would prevent a lot of frustration and probably lower the attrition rate for new reviewers.

(I'm assuming that you are doing the rating calculations "correctly", which is to exclude reviews that are still pending resolution from the rating, and base the rating only on those that have reached a decision. If not, that's an obvious first fix.)



  • 1Max2Max3Max-PGO1Max2Max3Max-PGO Posts: 10 ✭✭

    Thanks for posting, Hosette! I agree entirely with Hosette's proposal. I had the same experience happen to me where I was demoted from "Good" to "Fair." I've since been promoted back to "Good", but I think it's a poor initial experience. Hosette's proposal allows us to make sure new Wayfarers don't get de-motivated just from a low rating that may not deserve.

    I want to see these games improve in smaller communities like mine, and being able to retain more reviewers will help that process - this is a simple low-hanging-fruit edit that helps us get there.

  • Cdk296-PGOCdk296-PGO Posts: 166 ✭✭✭

    I don't think it is a good solution because any cheater or new people not comfortable yet could potentially make votes with a good agreement and help bad proposition make their way to the game.

    Maybe a simpler way would be to just freeze the score and not displaying it until a fix number of reviews has been done and fix number of agreement has been done. And replacing the score by a message saying "waiting for review to display a score" or something like this.

  • Hosette-INGHosette-ING Posts: 1,191 ✭✭✭✭✭

    @Cdk296-PGO That's a reasonable argument, though probably one or two "bad" votes on a submission wouldn't make much difference. I'm pretty sure that happens anyway. Hiding their rating also makes sense. I think I would want their early agreements to count toward badges and upgrades even if the reviewer was technically fair or poor. That "costs" the system very little, and it has the benefit of not making new reviewers feel like their contributions don't count.

  • 1Max2Max3Max-PGO1Max2Max3Max-PGO Posts: 10 ✭✭

    Post upcoming, but @Hosette-ING, thanks for your reply on my previous thread as well as the context. I'm going to not reply on that one as I think it's good to keep this all on one thread and you have a good starter post here.

  • 1Max2Max3Max-PGO1Max2Max3Max-PGO Posts: 10 ✭✭

    @GearGlider-ING - "If you ask people to do free labor, and then tell them their labor is bad and won't actually help anything, of course the laborers are gonna quit." - This line you said is a very powerful line that I think helps tell the story well.

    Between our two posts, I think we have a lot of solutions, and I honestly think any of these could solve the majority of the issue:


    Option 1) Make sure the score considers resolved calculations rather than just submitted (otherwise, someone who does 100 in an hour will likely have about 4/100 as a score, which likely would be listed as Poor or Fair) and either: A) list them as "Good" or B) list them as "Not yet rated" until they hit a certain threshold

    Pros: Simple solution with little technical debt

    Cons: None?

    Option 2) Provide feedback on your past reviews

    Pros: Allows reviewers to understand what is good or bad about their reviews and teaches them to optimize their reviews

    Cons: Heavy technical debt - difficult to implement AI Explainability with a smaller team like Niantic's

    Does anyone else have feedback they would like to add? I think it will be best to just make sure we have a single comprehensive post to cover the consensus opinion.

  • 1Max2Max3Max-PGO1Max2Max3Max-PGO Posts: 10 ✭✭

    @Hosette-ING - Was there any sentiments that we hadn't covered or that you think were missed? Also, any options that you think are logical to present here? I think the best approach is to summarize the problem, but also share a few potential options (rather than just one) that the engineering team can consider.

  • Hosette-INGHosette-ING Posts: 1,191 ✭✭✭✭✭

    @1Max2Max3Max-PGO I think we've done a reasonable job of presenting the problem and offering potential improvements.

  • globusbus-PGOglobusbus-PGO Posts: 1 ✭✭

    I started reviewing recently, with at least 10 - 15 reviews each day, and my ranking got poor. So now I don't review any more, waiting to see if anything happens to my rating.

    At least my numbers of agreements are slowly raising day by day. I hope to start reviewing again within a couple of weeks. There is no other kind of feedback?

    The whole thing seems to be made for the two-reviews-in-a-week reviewers?

  • Mormegil71-INGMormegil71-ING Posts: 133 ✭✭✭

    I constantly have to explain this to new reviewers, and I suppose many of the ones I don't even hear from simply just quit. Setting it to fair or good for the first hundred or so reviews meanwhile also explaining this in the help files would be a good thing.

Sign In or Register to comment.