Profile and Wayfinder Rating question - what’s the ratio?

Hello!

First time poster here. I went through a lot of the general discussion and couldn’t find these questions addressed.

1) Under the Wayfinder Rating, what is an approximate ratio of reviews vs communal acceptance / agreement for the different levels of the rating? For Poor, Fair, Good and Great?

2) I did some reviews, and then handfuls of reviews over the subsequent days. But only after a few more days did I notice I got communal agreement with my findings. (That is, other people agreed with my review so yay!) Around what approximate timeframe should I anticipate that what I have reviewed, will be seen by others? Days? Weeks?

I had only now found out about the medal, decided I should help out amongst all the other people volunteering to help, but don’t have clear expectations of timeframes to expect. The last I want to do is get flagged because I spent an hour, rip through a lot, and get downgraded to Poor because no one else has seen what I reviewed.

Tagged:

Comments

  • HINABITAMEU-INGHINABITAMEU-ING Posts: 154 ✭✭✭

    1) I expect at least 50% of the agreement rate will be fine.

    My current consent rate is 54%, and it is still great.

    2) I don't know exactly, but it seems to be seen as early as 1 day and at the latest within 2~3 weeks.

  • GotToCatchAMo-PGOGotToCatchAMo-PGO Posts: 6 ✭✭

    1) Thanks!

    2) By chance, have you seen discussion anywhere of this changing, where the rating doesn’t change except by acceptance and rejection?

    That way I wouldn’t have to have any attention on my “pending” rating; only actual.

  • GotToCatchAMo-PGOGotToCatchAMo-PGO Posts: 6 ✭✭
    edited March 2021
  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @GotToCatchAMo-PGO I'm a data/evidence nerd. To be honest, my best assessment right now is "I don't think we know." I have seen some evidence that suggests that the rating is based only on reviews that reached consensus, and some evidence that suggests that pending reviews are not removed from the calculation. The rating can also change based on honeypot submissions that Niantic has inserted into the stream as tests. We're all fairly certain that getting honeypots wrong will negatively impact your rating, which just makes sense. There's much less clarity about how your rating goes up.

  • GotToCatchAMo-PGOGotToCatchAMo-PGO Posts: 6 ✭✭

    @Hosette-ING Based on the answers here alone I can see there isn’t a real clear delineation by Niantic. I couldn’t find anything.

    Well, here’s to hoping.

  • GotToCatchAMo-PGOGotToCatchAMo-PGO Posts: 6 ✭✭

    1) Thanks!

    2) I didn’t look at the spreadsheet, but your summary makes sense. It is what I started considering, so am going to track myself to see. 😁

  • GotToCatchAMo-PGOGotToCatchAMo-PGO Posts: 6 ✭✭

    My experience, though this is sort of vague, is that above about a 50% agreement rate is reasonable. Bear in mind that there's no way to determine from your stats how many reviews are still pending agreement and how many are disagreements. 

    I did some research on exactly this question last year. You can look at my data if you want to, but my experience from about a year ago was that it could take from roughly 1-40 days until most of your reviews reached consensus. That is somewhat sensitive to your specific location and almost certainly will change over time, but the concept of "It can take a few weeks" probably hasn't changed. https://docs.google.com/spreadsheets/d/1XdQRiFqJ0oDIQLlVBFb6J6hGOPTLZ4DoA8bbxY39kME/edit#gid=1767613748

    Thanks!

    And good to know. I have started tracking my stuff as well as I became curious.

  • Babarushki-PGOBabarushki-PGO Posts: 195 ✭✭✭

    Frankly, guesses about “ratio” are just that. There are so many unknowns:

    • How many of your reviewed sites have decisions?
    • When will the sites you’ve reviewed get a ruling?
    • What grants you an agreement? Is it majority rule? Based on how many other reviews?
    • How does Niantic determine agreement for each of the categories (accept/reject/duplicate)?

    The real answer is “We just don’t know, but maybe here are some possibilities.” We’re all just pissing in the wind, spinning scenarios.

    Anecdotally, I know others who are “great” with 35-38% agreement ratio. Ratio is relatively meaningless.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @Babarushki-PGO We understand those things at a conceptual level even if we don't know all of the internal details involved in getting there.

    Niantic has some internal algorithm for knowing when any given submission has an answer. We don't know what that algorithm is, but we do know that there's not a fixed number of reviewers and that it depends partially on the rating of the reviewers who looked at it. In any case, there's a point when Wayfarer says, "OK, this is done reviewing and we have a decision."

    When that review reaches consensus Niantic awards an agreement to each of the reviewers who agreed with the consensus. "Duplicate" is probably the easiest one to understand since it's one clear signal-- if you said it's a duplicate and the consensus of reviewers was that it's a duplicate, applause noises! You have an agreement. If you said it was a duplicate and the consensus was to accept it, well, no agreement for you and sad trombone noise.

    For acceptance and rejection, there's some internal calculation done on your review about whether your overall vote was to accept it. Maybe it's as simple as the star rating for "Should this be a wayspot?" with 3-5 counting as yes and 1-2 counting as no. Maybe it's more complicated and adds up all of your stars... we don't know. But I think we do know that if Niantic's algorithm decided that your overall accept/reject vote agreed with the final decision on that candidate then you rack up an agreement.


    As for 35-38% agreement rate, it would be interesting to know if those rates were from when they had been reviewing a lot and thus had a lot of reviews awaiting consensus, as well as the total number of reviews that they'd done. It seems entirely possible for a relatively new reviewer to be in that situation. I'd raise an eyebrow if someone had done thousands of reviews and a great rating with a 38% agreement rate.

  • DaleTheWolfdog-PGODaleTheWolfdog-PGO Posts: 31 ✭✭

    You are kidding me? If for you 50% agreement rate then explain this.

    I'm having bunch of WF accounts.

    1st - 71% aggrement "FINE from year of being in 69% POOR (before was in the highest possible rank, when update) 5k reviewed"

    2nd - 67% agreement "POOR(again from green) 4,7k reviewed"

    3rd - 75% agreement "GOOD???? 80 reviewed"

    4th - 60% agreement "GREAT?! 230 reviewed"...

    As we can see on my view, WF rating system is broken AF :3 Of course, I'm reviewing everything like it should be reviewed. With WF standards etc. And what? It doesn't work anymore. You want to do OPR medal? Nah, we will drop you to the lowest possible grade and have fun there. Okey, someone can say that I could make a test for getting my grade reseted. But, it doesn't work like that. It will reset your grade for only a while, and after 10 reviewed nominations it will drop you to your old grade.

    And that's my honest opinion about WF rating 🤣

  • HINABITAMEU-INGHINABITAMEU-ING Posts: 154 ✭✭✭

    You may be reviewing something wrong.

    My consent rate has always been between 52 and 54 percent.


  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @HINABITAMEU-ING Wow! That's impressive. You have over nine times as many reviews as I do. This is about a week out of date, but it's still pretty close to current.

    I've run at a 64-65% agreement rate for years. I think it would be about 1% higher if there hadn't been a few periods where "just reject everything" was happening a lot. My percentage always dropped when that was happening.



  • TheFarix-PGOTheFarix-PGO Posts: 5,063 ✭✭✭✭✭

    Your Wayfarer rating has little to do with your overall agreements ratio. First, there are other factors that influance your rating, including your agreements compaired to Niantic pre-reviewed nominations, previous actions taken against an account for participating in abuse, and also the rating only reflect the most recent nominations reviewed instead of the entire lifetime of the Wayfarer account.

  • DaleTheWolfdog-PGODaleTheWolfdog-PGO Posts: 31 ✭✭

    I'm having Ingress and PoGo accounts, so that doesn't count. But rest are my friends accounts, that play pogo. And anyway, anyone doesn't care about that ;) I was talking about this on many other forum posts and I didn't had any problems with that. I'm just pointing everyone that WF system is broken for years when I'm with it.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @DaleTheWolfdog-PGO You're right about Ingress/PoGo. As for using other people's accounts...

    You're right that in PoGo nobody seems to care but it's still technically against the rules.


  • LFX54-PGOLFX54-PGO Posts: 5 ✭✭

    I have no idea how the system works. Just found your post when thinking the same thing. Mine turned to great this morning. I turned level 38 in pogo a few weeks ago but didn’t find out I could also review until a couple of days ago. 25% agreement rate and great lol

  • Ochemist-INGOchemist-ING Posts: 355 ✭✭✭✭✭

    I'd raise an eyebrow if someone had done thousands of reviews and a great rating with a 38% agreement rate.

    Granted it was a different era, but by Sept-October of 2018, if not even later, I had done thousands of reviews, had a 38% agreement rate, and hadn’t dipped below great since I first got there. My rate started steadily creeping up once they began to require a reason for a 1* review that September, as did my acceptance to rejection ratio. Other prolific reviewers in great standing had similar agreement percentages. (It’s now at around 64%.)

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @LFX54-PGO Remember that your total nominations reviewed includes the ones that have reached consensus as well as the ones that still need more reviews before a decision is made. If most of your reviews have been in the last couple of weeks then you probably still have a lot that are undecided. I did a study last year where I did a bunch of reviews and then stopped and just recorded my stats every day. At that point it took almost 40 days before new agreements stopped trickling in, and I still got 2-3 after that until 90 days from my last review.

    We don't know how many of the total are disagreements vs still pending. Niantic doesn't give us enough information to figure it out.

  • Oakes1923-PGOOakes1923-PGO Posts: 419 ✭✭✭✭

    In the early days you're review rating is likely to go up and down on a fairly regular basis. You just wont have the historical data to lean against. However Water tends to seek its own level though so if you are reviewing to the standards you should be fine. I would not worry to much about your rating unless a few months from now it seems to be staying in the fair ranking.

  • Quivittoq-PGOQuivittoq-PGO Posts: 9 ✭✭

    Today I went from Fair to Good

    After starting in Wayfarer I quickly ended up in the Poor category despite thinking knowing all the criteria.

    I could do a retest and set to Great again.

    Unfortunately I know know the retest is just a one-time affair, so I quickly dropped to Fair again...

    With hindsight I should have done this retest only after 500+ reviews or as soon as I noticed my calculated percentage would be stable. Ie. after 506 reviews I had (103 accepted, 50 rejected, 7 duplicates) = 160 / 506 = 0,316 = 32%. It then kept hovering between 30% and ever so slowly climbing on the third digit. Take a day to get all the outstanding reviews out of the system.

    Today I reached 33% and my rating switched from Fair to Good. Probably not exact science but I know I am on the right track now. And 33% and up could be the boundary between Fair and Good. At least my Agreements are counted again for my badge :-)

    I notice my numbers do not change after 48 hours, so there could still be big number of (dis-)agreements in my queue when I read others about months.

  • MargariteDVille-INGMargariteDVille-ING Posts: 2,846 ✭✭✭✭✭

    Lots of nominations are In Voting 6 months to 2 years. Whoever voted on them when they first went into voting won't see that agreement until it resolves.

    Also, edits and photo adds give +1 to your "Total Nominations Reviewed" but do not ever count as agreements. See https://wayfarer.nianticlabs.com/new/help/agreements--upgrades At one point, I was pressing to get double-onyx Recon badge (20,000 agreements) but one out of four reviews was an edit. I got frustrated with the slow pace, and walked away for a while.

Sign In or Register to comment.