Dear Niantic: Please consider all new reviewers as "Good" for a while
To the fine product designers and engineers at Niantic:
I've been a denizen of this forum and one on Facebook for quite a while, and I've been a reviewer since the launch of OPR beta in 2016. I've had the opportunity to see a lot of complaints from reviewers, some legitimate and some misplaced, and there are some that are systemic and quite valid. One of the most commonly-repeated and IMO extremely legitimate complaints involves the experience of new reviewers and their rating dropping immediately. I could find lots of examples to back this up, but this recent post contains an excellent illustration of the issue: https://community.wayfarer.nianticlabs.com/discussion/13748/recommendations-to-enhance-initial-ux-experience-of-niantic-wayfarer
This is a very common complaint: "I was excited to start using Wayfarer! I logged in and spent a couple of hours reviewing, and my rating dropped. Why? I was following the guidelines." This is a poor initial experience for new Wayfarers as their initial exuberance is quickly converted into frustration. Even if someone is doing excellent reviewing there is going to be a lot of noise in their rating at first, and they could drop to poor simply because of the order in which their first few reviews resolve. Please consider assigning new Wayfarers a fixed initial rating, perhaps Good, and keep it locked there until they have enough agreements for their rating to be statistically significant. This would prevent a lot of frustration and probably lower the attrition rate for new reviewers.
(I'm assuming that you are doing the rating calculations "correctly", which is to exclude reviews that are still pending resolution from the rating, and base the rating only on those that have reached a decision. If not, that's an obvious first fix.)
Comments
Or, a Training Mode setting. New reviewers can review away and see how their rating is doing, particularly over the first few hundred reviews. "Training Mode" agreements can be seen but do not register on the players stats until the Reviewer has a) completed a certain minimum number of reviews and b) Switchs off training mode. At this point, the stats are transferred across to the relvant Ingress / PoGo badge scores.
Or, perhaps Niantic could give reviewers some feedback - lots of people have asked for this on a regular basis. Just show us the last 10 reviews where our verdict did not agree with community verdict. This would allow us to improve, and possibly report back to Niantic where genuinely acceptable nominations are being refused, perhaps showing up some of these supposed "1* everything for my PoGo badge" areas.
Whenever the next AMA rolls around, I hope I can hop on it early so I can ask exactly this. This is an issue that has prveented so many people from reviewing. When I started OPR, My rating stayed at "good" for the first couple hundred or so before going up to "great" and I was so worried before that because I had no indication of whether or not I was doing a good job.
If you ask people to do free labor, and then tell them their labor is bad and won't actually help anything, of course the laborers are gonna quit. And SO MANY have so far because of a bad UX choice.
I don't have much to add. Just a screenshot of a beginner's wayfarer rating. They wasn't exactly eager to continue after that.
Thanks for posting, Hosette! I agree entirely with Hosette's proposal. I had the same experience happen to me where I was demoted from "Good" to "Fair." I've since been promoted back to "Good", but I think it's a poor initial experience. Hosette's proposal allows us to make sure new Wayfarers don't get de-motivated just from a low rating that may not deserve.
I want to see these games improve in smaller communities like mine, and being able to retain more reviewers will help that process - this is a simple low-hanging-fruit edit that helps us get there.
I don't think it is a good solution because any cheater or new people not comfortable yet could potentially make votes with a good agreement and help bad proposition make their way to the game.
Maybe a simpler way would be to just freeze the score and not displaying it until a fix number of reviews has been done and fix number of agreement has been done. And replacing the score by a message saying "waiting for review to display a score" or something like this.
@Cdk296-PGO That's a reasonable argument, though probably one or two "bad" votes on a submission wouldn't make much difference. I'm pretty sure that happens anyway. Hiding their rating also makes sense. I think I would want their early agreements to count toward badges and upgrades even if the reviewer was technically fair or poor. That "costs" the system very little, and it has the benefit of not making new reviewers feel like their contributions don't count.
Post upcoming, but @Hosette-ING, thanks for your reply on my previous thread as well as the context. I'm going to not reply on that one as I think it's good to keep this all on one thread and you have a good starter post here.
@GearGlider-ING - "If you ask people to do free labor, and then tell them their labor is bad and won't actually help anything, of course the laborers are gonna quit." - This line you said is a very powerful line that I think helps tell the story well.
Between our two posts, I think we have a lot of solutions, and I honestly think any of these could solve the majority of the issue:
Options
Option 1) Make sure the score considers resolved calculations rather than just submitted (otherwise, someone who does 100 in an hour will likely have about 4/100 as a score, which likely would be listed as Poor or Fair) and either: A) list them as "Good" or B) list them as "Not yet rated" until they hit a certain threshold
Pros: Simple solution with little technical debt
Cons: None?
Option 2) Provide feedback on your past reviews
Pros: Allows reviewers to understand what is good or bad about their reviews and teaches them to optimize their reviews
Cons: Heavy technical debt - difficult to implement AI Explainability with a smaller team like Niantic's
Does anyone else have feedback they would like to add? I think it will be best to just make sure we have a single comprehensive post to cover the consensus opinion.
@Hosette-ING - Was there any sentiments that we hadn't covered or that you think were missed? Also, any options that you think are logical to present here? I think the best approach is to summarize the problem, but also share a few potential options (rather than just one) that the engineering team can consider.
@1Max2Max3Max-PGO I think we've done a reasonable job of presenting the problem and offering potential improvements.
I started reviewing recently, with at least 10 - 15 reviews each day, and my ranking got poor. So now I don't review any more, waiting to see if anything happens to my rating.
At least my numbers of agreements are slowly raising day by day. I hope to start reviewing again within a couple of weeks. There is no other kind of feedback?
The whole thing seems to be made for the two-reviews-in-a-week reviewers?
I constantly have to explain this to new reviewers, and I suppose many of the ones I don't even hear from simply just quit. Setting it to fair or good for the first hundred or so reviews meanwhile also explaining this in the help files would be a good thing.
This system is broken. I know how to review. I constantly read the updates and changes to criteria. However when I started. I just spent days sitting and reviewing until I had no more to review. I hated that I personally had nominations submitted that had been waiting months and months, so I wanted to review what I could for other players. Having no idea how the background personal rating on wayfarer worked. I did so many reviews I broke my rating. The numbers just don't match up and I can never correct it.
As I can never change my status I just don't review now. Idiot I know but I thought I was helping 😂🤣😂
@NianticDanbocat your thoughts, please? This kind of issue is one of the reasons which would let people turned away from Wayfarer.
Is this really that much of an issue? I never dropped to poor, have been a fairly high volume reviewer from the beginning (averaged out I have done 186 reviews per day since I started on October 7th), and I went to great within a couple days and have stayed there. What would trigger dropping to poor for newer reviewers? Is it disagreements with a low volume of total reviews?
The plan to introduce multiple attempts of the start quiz is because Niantic do understand the need for more active reviewers. There have been numerous comments about this. Personally I have an open mind about this - so will wait and see. But a hoped for outcome presumably are new reviewers.
I can also see that if you come in new and then very quickly see your rating drop, it will be very offputting.
There are several good suggestions already.
I think I like the idea of an apprentice level, perhaps either with the great/poor rating hidden, maybe something showing progress towards full reviewing status (so encouragement to continue). I would also put a daily limit or review rate to encourage slow considered reviewing for this level. It might set a good habit but also it might help prevent lots of reviews without resolution. I doubt it would be possible but even giving a higher proportion of nominations that were near resolution might be good not only as these might be resolved soon but also it could pick out if they were reviewing roughly inline with others
I think if there could be feedback during this stage it would be very good.
Most of all wouldn’t it be better to know something for certain about the reviewing scoring and the effects of the rating.
All we have is guess work based on tiny crumbs off the Niantic table. We could have some great ideas but without knowing how it actually works they could be useless.
You are not a new reviewer.
@DoodlebugzNZ-PGO It looks to me like you approve way too many things... either that or the submitters in your area are incredibly picky and only submit high-quality stuff. You approve 78% of stuff that you review. I'm at the other end of the scale, although always in the great rating category, and I approve about 42%.
@DoodlebugzNZ-PGO I have the exact same feeling that @Hosette-ING has.
I don't know where you are, but after >10k reviewings in different places, I am still in the great rating category and I approve about 40% too.
About this topic, I am sorry I didn't agree. At the beginning of my reviewing "career" I have made a lot of mistake, so my ratings have gone to Poor. So I start to improve, look closer at each criteria and nomination and after a few weeks of improvement and learning, I've reached great and never be under since.
I think being not good at reviewing at the beginning is normal, it's part of the learning process. You make some mistake then improve.
It would be good to fix it in Good for a certain period of time and until the review is completed, but in that case, the impact of the new auditor's evaluation should be small or not really reflected.
Therefore, it is rather better to create a training mode.
Also, I don't understand this, but for some reason new reviewers nowadays tend to use 1 and 5 a lot, and it would be better to specify that such extreme reviews have a negative impact on their own evaluation.
I 1* a lot and have never dropped off of great since reaching it. What I 1* is stuff like photos taken from cars, description mentioning Pokemon Go or Pokestop, K-12 schools, etc..
If it is a trash nomination I don't think people should be afraid to 1* it and select a reason so that the perhaps the people nominating junk will get a rejection that gives them feedback as to why their nomination was not eligible. They are also easy agreements and help your rating.
I guess I do 5* less, using 4* for eligible but not especially interesting or unique POI or not great photo and/or description.