As a relatively new reviewer, I’ve noticed that the Wayfinder rating seems to be heavily influenced by the total number of nominations. Once I reviewed 100+ nominations, my rating dropped from ‘Good’ to ‘Fair.’ I suspect this is because most of my reviews haven’t been resolved yet. Shouldn’t the Wayfinder rating be based solely on agreements per resolved nominations?
4 Likes
Welcome @doomsday126
You are quite right to raise questions.
Great to hear you are taking part in reviewing.
This is going to sound a bit strange, but no one (wayfinders) actually knows exactly how the rating system works.
Intuitively you would think we should know so that we can use the information to learn about how we are doing and to use it as a tool to improve.
It is quite basic.
As wayfarer is about judgements and reaching a consensus reaching agreement with the majority is a critical factor. And yes in some areas you will review and what you have reviewed will take time to resolve.
The best advice I can give is to keep answering the questions when reviewing as honestly as you can judging against the criteria. Take it slow and steady and if you are very unsure it is ok to skip.
4 Likes
Thank you for responding! While I understand that nobody knows how exactly the wayfarer rating formula works, I think we can make certain assumptions about the formula that are fairly obvious based on the data we see. My only issue is that the formula is based on how many nominations you’ve reviewed in totality so if someone does not have that many reviews to begin with, the rate of resolves to reviews will always skew the rating negatively. My only option seems to be to let the resolves catch-up to the reviews I’ve done and then continue but that seems counterproductive.
1 Like
Niantic has said repeatedly that number of agreements/disagreements/unresolved does not affect your rating.
Reviewing too quickly can affect it - which is defined not only by speed but also whether you interact with the page. When needed, you can click the title for a google search, click a potential duplicate existing wayspot, click the Google symbol bottom left of the map (or click a place lablel and then open in google maps), click the pictures to enlarge them, scroll the text in description or supplemental. You don’t have to do all this in every nomination - but if you never do any of these things, you are not giving a good review.
Also Niantic uses what we call “honey pots” (that’s slang). What they actually said was “We pre-approve or reject certain Wayspot nominations prior to them entering Wayfarer. Reviewers who disagree with our assessment may see a dip in their performance score.”
3 Likes
Do you happen to have a link to them saying this? I try to save links to anything important like that. In my experience, the few times I have dropped to good, NOT reviewing has made my rating go back up to great.
1 Like
I used to try to keep up with everything, but got burned out and stopped.
For you - I’ll dig thru my archives.
2 Likes
i know it may not exist anymore. we tracked down a niantic statement about reviewer rating affecting the weight of their vote to G+ - and i wasn’t ever in a G+ wayfarer community, even when that existed.
1 Like
This is what is currently being discussed in the rating bug report here:
My last post there, and others as well, has shown that your rating is not just based on your agreements, but there are other factors that we may not fully be aware of. Most ambos even agree with this.
2 Likes
Yes, there were comments about it in G+.
And I saved this link (also gone) https://community.wayfarer.nianticlabs.com/discussion/5007/quality-assurance-and-reviews
From Ingress AMA Archive
Q38: Any way to get out of a poor rating on OPR?
A38: In my experience, people are doing things too quickly, so take your time. Even if you make your decision in a few seconds, just take your time before you actually push that submit button, otherwise you’re probably going to flag something.
(MargariteDVille’s note: By “take your time”, I think he meant mess around on the page some. Click things, etc.)
From Ingress AMA Archive
Q12: I dropped from great to poor in opr exactly a week ago. In that time I have gotten more agreements than new reviews. That means agreements I made while great aren’t being counted while I am poor. Is this fair?
A12: Any agreement that occured while your performance rating was Poor will not count towards your Recon medal. To improve your rating, review the training materials and re-evaluate how you are analyzing candidates based on the criteria.
From Ingress AMA Archive
Q7: Would a mandatory OPR test be something you could explore for those stuck at bad rating? What are the percentage thresholds for ratings or is it more complicated? If so, any additional comments?
A7: This sounds like something worth considering. There are many reasons you may have a poor rating in OPR. The best way to recover is to continue to analyze candidates and follow the criteria and guidelines to the best of your ability.
and
Q41: I went to Panama/Mexico, and did some opr down there… I went from Great to Poor in the matter of a day while maintaining a 70% agreement ratio. I have close to 200 agreements since then…and my needle hasn’t moved…and my stat isn’t counting. My questions are: What gives?
A41: OPR performance takes many factors into account including abuse detection. The best way to ensure your rating stays high is to take time to carefully analyze each candidate.
As an IT professional for many decades, I know that every website you visit, every newsletter email you receive - they all track how far you read, how fast, where you paused, what you clicked, etc. This is generic, easy, packaged code, that I have no doubt Niantic implemented. I’m pretty sure it’s shareware (free).
4 Likes
will save this link - ty
2 Likes
As I’ve said before, I think Niantic has some kind of internal math error with edit reviews.
Niantic Wayfarer says “Note: Reviews of Wayspot Edits are not counted as Agreements at this time.”
I’ll bet edit reviews are added to some internal counters, but not all of them - and some downstream processes uses math with numbers they think were all created the same.
Until the challenge, edits were rare enough that this inconsistency didn’t do anything noticeable. But we got so many edits in the challenge - it tore through with fireworks.
2 Likes
I’ll be honest I did not read these (I HAVE in the past). The longer I’ve been involved in Wayfarer the less confidence I have in anything said by RedSoloCup on Ingress AMA and frankly half the responses we get even from the now dedicated Wayfarer team.
Believe everything at your own risk.
2 Likes