Niantic Question: Are unresolved reviews counted against your rating?

Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

If I sit down today and review 200 submissions, clearly my "Total Nominations Reviewed" statistic will increase by 200. My "Nominations Accepted/Rejected/Duplicated" stats may increase by a small number today, but not by much. The reason for this is that most of the things I review today won't reach consensus today-- they will dribble in over time, and some can take a month or more to reach a final decision.

Is your Wayfinder rating based on the Total Nominations Reviewed number, or is it the (invisible) statistic of Total Nominations Reviewed That Have Reached Consensus?

Comments

  • sogNinjaman-INGsogNinjaman-ING Posts: 3,313 ✭✭✭✭✭

    The latter. It is the total of all "Accepted / Rejected / Duplicates" values, minus any you may have had go through as ""Accepted / Rejected / Duplicates" if and when you were ever in the Red / Poor / Fair categories, but without any seeming addtions to stats for doing location or text edits.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @sogNinjaman-ING I'm willing to believe you, but what is your evidence? Some have suspected that just having a large number of reviews that haven't yet reached consensus could knock you down from Great to Good, Fair, or Poor.

  • AgentX1976-INGAgentX1976-ING Posts: 598 ✭✭✭✭✭

    There really can't be any proof. Since we don't know how many have completed the process there is a missing element. It seems that they don't count against you.

  • Rodensteiner-PGORodensteiner-PGO Posts: 1,705 ✭✭✭✭✭

    that is niantics secret sauce recipe.

    thou if you click too fast, and get honeypot-submissions that Niantic fabricated, you might tank instantly.

  • sogNinjaman-INGsogNinjaman-ING Posts: 3,313 ✭✭✭✭✭

    My evidence is purely empirical from what I have observed over the course of reviewing 43,000 plus nominations and reviews. In the early days of reviewing I kept an eye on things to see how my OPR stats were adding up for my Ingress Medal. Very noticeable if you go into the Red your medal stats stop increasing while you are in the Red.

  • rinkulamuroja-PGOrinkulamuroja-PGO Posts: 2
    edited December 2020

    I think unresolved are actually counted against you. At least I’ve recognized that in my rating. My rating has rose overnight and I can’t find any other explanation than unresolved are counted, too.

    In my case, my rating has dropped since I started some days ago doing reviews. I believe dropping my rating to fair is ok, I’m just learning how to review and how I should evaluate different spots on reviewing scales. But two days ago my rating was poor, which made me feel unsure, what I’ve done wrong. I’m doing my best, and it’s hard to believe that my reviewing is that horrible. At that point I had 544 total nominations reviewed, and 131 (24%) of them accepted/rejected/duplicated. Today when I opened Wayfarer, my rating had rose to fair. 7 more nominations have been a/r/d between my visits and that’s the only reason I can find for the rating rising. I didn’t do any new reviews between, so I couldn’t pass honeyspot test or anything.

    Edit: Here's my rating today. I reviewed 3 more nominations, and my rating dropped again, but one more agreement rose it back. This supports my assumption of unresolved are also counted.


    Post edited by rinkulamuroja-PGO on
  • AScarletSabre-PGOAScarletSabre-PGO Posts: 754 ✭✭✭✭✭

    Yes, as far as I know they are. Wayfarer+ gives me an exact percentage in terms of 'process and agreement'. The other day when I did a bunch of reviews, I started out as agreeing with other reviewers 70% of the time and after doing the reviews I went down to 69%. I can see why this would be an issue for those first starting out reviewing, because their rating might be "great", as mine is and they are being penalized with a lower rating simply by reviewing.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @rinkulamuroja-PGO Thank you! That's interesting evidence although it's not conclusive. Since we don't know how the algorithm for Poor/Fair/Good/Great works it's also possible that it only counts your last N reviews, or the last X% of your reviews, or reviews in the last Q days. If it's an all-time score then someone like me who has five figures worth of agreements would have a huge buffer, and a very low probability of ever dropping from Great except perhaps by mis-reviewing a honeypot. It would mean that the ratings of new reviewers would be much more volatile.

    Looking at your screenshot it's possible that it only counts, oh, the last 100 agreements. If the agreement that dropped off when the new one came in was a bad one then that might be enough to shift your rating.

    My intuition tells me that unresolved reviews are counted. If so, I think that's a bug because it penalizes enthusiastic reviewers unnecessarily... especially new reviewers. I also suspect that your rating is lifetime rather than a trailing window. It's consistent with my experience of having never dropped out of the top tier (green/great) even though I had a couple hundred outstanding reviews at one point. These things are also consistent with the complaints I've heard from new reviewers having their rating drop shortly after they started.

  • NoelleXandria-PGONoelleXandria-PGO Posts: 50 ✭✭

    I'm starting to think that the way to go about reviewing is to try to figure out how other reviewers are likely to vote, and vote in line with them even if you may not agree with it. Early this morning, my rating was fair, even though I was voting honestly. When I started voting how I thought others would vote, even without the A/R/D changing yet, and with my review count going up, my score went up to Good. Then I got smacked with that cooldown thing and had to stop for four hours. That was a couple hours ago.

    Over the last half hour, my rating dropped from Good to Fair even though my total reviews hasn't changed, nor has the A/R/D number. 190, and 8/6/0, if it matters. Nothing but the rating.

    I suspect the rating has as much to do with your ratio of with-the-majority of against-the-majority behind the scenes as it does with the final agreements since that's the only thing that even could have changed for me.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @NoelleXandria-PGO I'm a little confused. What is the difference between with-the-majority and agreements? My understanding is that agreements are cases where you voted with the majority.

  • sogNinjaman-INGsogNinjaman-ING Posts: 3,313 ✭✭✭✭✭

    When something enters into voting, it needs to be reviewed by a certain number of reviewers before a decision is reached. How many this is, Niantic won't say. "Agreements" covers nominations marked as "Accepted, Rejected or Duplicated".

    If your votes "agree" with the majority of the other views from other players for a particular submission, your stats will increase by 1. If you vote "yes" and the majority of other votes also vote "yes", then the submission will be add 1 to your "Nomination Accepted" score. If you vote "yes" but the majority of the other players voting on this one say "No", then your score will not increase.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @sogNinjaman-ING The reason Niantic won't tell us how many is that there is no fixed number. It's not like the decision is made when you get 37 reviews, or 137, or 1337. There's some internal calculation involved in deciding when any particular submission is done with reviewing. The one thing that Niantic has told us is that it's partially dependent on the rating of the reviewers who participate. My intuition is that it also depends upon how closely the reviewers agree. If a large majority agree 5* or 1* then it probably resolves more quickly than if the decisions are split or votes are more in the middle. That is just speculation, though.

  • czerwonytomek-INGczerwonytomek-ING Posts: 18 ✭✭

    There are still editions that do not count towards consistent ratings.

  • 52cucumbers-ING52cucumbers-ING Posts: 225 ✭✭✭✭

    I had a two year break from Ingress, so naturally I wasn't doing OPR reviews either. When I came back to Wayfarer I did a large chunk of reviews and my rating fell to yellow. Over the course of a couple of days it came back up to green on its own. Make of it what you will but I believe unresolved reviews do count.

    Since we don't know how the algorithm for Poor/Fair/Good/Great works it's also possible that it only counts your last N reviews, or the last X% of your reviews, or reviews in the last Q days.

    That is, however, a good point.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @52cucumbers-ING Roughly how many reviews did you do before you took a break?

    Your comment suggests that there might be a time window involved rather than or in addition to a fixed number of reviews. Like maybe it uses a 30 day window to calculate your rating, or maybe it looks for the last N reviews but only looks back X days to find them.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @52cucumbers-ING OK, so doing a few hundred reviews quickly could definitely make a ding in a rating until some resolved. I can see that, given that the resolve time for reviews can be over a month.

Sign In or Register to comment.