Dear Niantic: Please consider all new reviewers as "Good" for a while

Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

To the fine product designers and engineers at Niantic:

I've been a denizen of this forum and one on Facebook for quite a while, and I've been a reviewer since the launch of OPR beta in 2016. I've had the opportunity to see a lot of complaints from reviewers, some legitimate and some misplaced, and there are some that are systemic and quite valid. One of the most commonly-repeated and IMO extremely legitimate complaints involves the experience of new reviewers and their rating dropping immediately. I could find lots of examples to back this up, but this recent post contains an excellent illustration of the issue: https://community.wayfarer.nianticlabs.com/discussion/13748/recommendations-to-enhance-initial-ux-experience-of-niantic-wayfarer

This is a very common complaint: "I was excited to start using Wayfarer! I logged in and spent a couple of hours reviewing, and my rating dropped. Why? I was following the guidelines." This is a poor initial experience for new Wayfarers as their initial exuberance is quickly converted into frustration. Even if someone is doing excellent reviewing there is going to be a lot of noise in their rating at first, and they could drop to poor simply because of the order in which their first few reviews resolve. Please consider assigning new Wayfarers a fixed initial rating, perhaps Good, and keep it locked there until they have enough agreements for their rating to be statistically significant. This would prevent a lot of frustration and probably lower the attrition rate for new reviewers.

(I'm assuming that you are doing the rating calculations "correctly", which is to exclude reviews that are still pending resolution from the rating, and base the rating only on those that have reached a decision. If not, that's an obvious first fix.)

Tagged:

Comments

  • 1Max2Max3Max-PGO1Max2Max3Max-PGO Posts: 10 ✭✭

    Thanks for posting, Hosette! I agree entirely with Hosette's proposal. I had the same experience happen to me where I was demoted from "Good" to "Fair." I've since been promoted back to "Good", but I think it's a poor initial experience. Hosette's proposal allows us to make sure new Wayfarers don't get de-motivated just from a low rating that may not deserve.

    I want to see these games improve in smaller communities like mine, and being able to retain more reviewers will help that process - this is a simple low-hanging-fruit edit that helps us get there.

  • Cdk296-PGOCdk296-PGO Posts: 170 ✭✭✭

    I don't think it is a good solution because any cheater or new people not comfortable yet could potentially make votes with a good agreement and help bad proposition make their way to the game.


    Maybe a simpler way would be to just freeze the score and not displaying it until a fix number of reviews has been done and fix number of agreement has been done. And replacing the score by a message saying "waiting for review to display a score" or something like this.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @Cdk296-PGO That's a reasonable argument, though probably one or two "bad" votes on a submission wouldn't make much difference. I'm pretty sure that happens anyway. Hiding their rating also makes sense. I think I would want their early agreements to count toward badges and upgrades even if the reviewer was technically fair or poor. That "costs" the system very little, and it has the benefit of not making new reviewers feel like their contributions don't count.

  • 1Max2Max3Max-PGO1Max2Max3Max-PGO Posts: 10 ✭✭

    Post upcoming, but @Hosette-ING, thanks for your reply on my previous thread as well as the context. I'm going to not reply on that one as I think it's good to keep this all on one thread and you have a good starter post here.

  • 1Max2Max3Max-PGO1Max2Max3Max-PGO Posts: 10 ✭✭

    @Hosette-ING - Was there any sentiments that we hadn't covered or that you think were missed? Also, any options that you think are logical to present here? I think the best approach is to summarize the problem, but also share a few potential options (rather than just one) that the engineering team can consider.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @1Max2Max3Max-PGO I think we've done a reasonable job of presenting the problem and offering potential improvements.

  • globusbus-PGOglobusbus-PGO Posts: 1 ✭✭

    I started reviewing recently, with at least 10 - 15 reviews each day, and my ranking got poor. So now I don't review any more, waiting to see if anything happens to my rating.

    At least my numbers of agreements are slowly raising day by day. I hope to start reviewing again within a couple of weeks. There is no other kind of feedback?

    The whole thing seems to be made for the two-reviews-in-a-week reviewers?

  • Mormegil71-INGMormegil71-ING Posts: 202 ✭✭✭

    I constantly have to explain this to new reviewers, and I suppose many of the ones I don't even hear from simply just quit. Setting it to fair or good for the first hundred or so reviews meanwhile also explaining this in the help files would be a good thing.

  • This system is broken. I know how to review. I constantly read the updates and changes to criteria. However when I started. I just spent days sitting and reviewing until I had no more to review. I hated that I personally had nominations submitted that had been waiting months and months, so I wanted to review what I could for other players. Having no idea how the background personal rating on wayfarer worked. I did so many reviews I broke my rating. The numbers just don't match up and I can never correct it.

    As I can never change my status I just don't review now. Idiot I know but I thought I was helping 😂🤣😂

  • patsufredo-PGOpatsufredo-PGO Posts: 4,213 ✭✭✭✭✭

    @NianticDanbocat your thoughts, please? This kind of issue is one of the reasons which would let people turned away from Wayfarer.

  • MelodyS88Chi-PGOMelodyS88Chi-PGO Posts: 627 ✭✭✭✭✭
    edited November 2021

    Is this really that much of an issue? I never dropped to poor, have been a fairly high volume reviewer from the beginning (averaged out I have done 186 reviews per day since I started on October 7th), and I went to great within a couple days and have stayed there. What would trigger dropping to poor for newer reviewers? Is it disagreements with a low volume of total reviews?

  • Elijustrying-INGElijustrying-ING Posts: 5,482 Ambassador

    The plan to introduce multiple attempts of the start quiz is because Niantic do understand the need for more active reviewers. There have been numerous comments about this. Personally I have an open mind about this - so will wait and see. But a hoped for outcome presumably are new reviewers.

    I can also see that if you come in new and then very quickly see your rating drop, it will be very offputting.

    There are several good suggestions already.

    I think I like the idea of an apprentice level, perhaps either with the great/poor rating hidden, maybe something showing progress towards full reviewing status (so encouragement to continue). I would also put a daily limit or review rate to encourage slow considered reviewing for this level. It might set a good habit but also it might help prevent lots of reviews without resolution. I doubt it would be possible but even giving a higher proportion of nominations that were near resolution might be good not only as these might be resolved soon but also it could pick out if they were reviewing roughly inline with others

    I think if there could be feedback during this stage it would be very good.

    Most of all wouldn’t it be better to know something for certain about the reviewing scoring and the effects of the rating.

    All we have is guess work based on tiny crumbs off the Niantic table. We could have some great ideas but without knowing how it actually works they could be useless.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @DoodlebugzNZ-PGO It looks to me like you approve way too many things... either that or the submitters in your area are incredibly picky and only submit high-quality stuff. You approve 78% of stuff that you review. I'm at the other end of the scale, although always in the great rating category, and I approve about 42%.

  • VladDraco-PGOVladDraco-PGO Posts: 560 ✭✭✭✭

    @DoodlebugzNZ-PGO I have the exact same feeling that @Hosette-ING has.

    I don't know where you are, but after >10k reviewings in different places, I am still in the great rating category and I approve about 40% too.

    About this topic, I am sorry I didn't agree. At the beginning of my reviewing "career" I have made a lot of mistake, so my ratings have gone to Poor. So I start to improve, look closer at each criteria and nomination and after a few weeks of improvement and learning, I've reached great and never be under since.

    I think being not good at reviewing at the beginning is normal, it's part of the learning process. You make some mistake then improve.

  • tp235-INGtp235-ING Posts: 1,383 ✭✭✭✭✭
    edited November 2021

    It would be good to fix it in Good for a certain period of time and until the review is completed, but in that case, the impact of the new auditor's evaluation should be small or not really reflected.

    Therefore, it is rather better to create a training mode.


    Also, I don't understand this, but for some reason new reviewers nowadays tend to use 1 and 5 a lot, and it would be better to specify that such extreme reviews have a negative impact on their own evaluation.

  • MelodyS88Chi-PGOMelodyS88Chi-PGO Posts: 627 ✭✭✭✭✭

    I 1* a lot and have never dropped off of great since reaching it. What I 1* is stuff like photos taken from cars, description mentioning Pokemon Go or Pokestop, K-12 schools, etc..


    If it is a trash nomination I don't think people should be afraid to 1* it and select a reason so that the perhaps the people nominating junk will get a rejection that gives them feedback as to why their nomination was not eligible. They are also easy agreements and help your rating.


    I guess I do 5* less, using 4* for eligible but not especially interesting or unique POI or not great photo and/or description.

Sign In or Register to comment.