Review quality should matter

Obviously there is a flaw in the way ratings are now determined.

Essentially @NianticAaron saying “Wayfarer rating is based on the number of agreements and disagreements.” at: https://community.wayfarer.nianticlabs.com/discussion/37139/rating-dropped

is the same thing as saying rating is based on the number of TRUE and FALSE answers on a TEST.

Shouldn’t the correct answer be a factor? 

If anyone reviewing deservingly rejects ANY submission that should cause the rating to go up, not down.

If Niantic does not have a way to determine if the QUALITY of the review is correct (meaning legitimate rejects do not negatively impact rating) then that is a problem that needs to be corrected.

Left as is you are opening the door to even more abusive submissions and based on recent reviews I have seen and read about here, this is already happening. Is this what you want?

Niantic please get your act together. It stands to reason that long time reviewers with very high reviews and great ratings prior to this change should have earned a little respect and have their opinions valued not insulted by telling us how to improve our reviews.

Comments

  • flatmatt-PGOflatmatt-PGO Posts: 2,137 ✭✭✭✭✭

    You seem to be misunderstanding something here. An agreement occurs when your decision (whether to approve, reject, or mark as a duplicate) aligns with the final community decision for a nomination. These agreements, in all 3 categories, are summarized on your Profile page in Wayfarer.*

    Disagreements (where your decision did not align with the community's final decision) are not displayed on your Profile page at all.

    * There are also agreements/disagreements on Edit suggestions reviewed, but these aren't specifically displayed anywhere.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @claireymc-ING How do you determine the correct answer for every single thing that was submitted? Remember that most of them involve judgement calls. Niantic is basically relying on the wisdom of crowds to determine the "correct" answers, and you are right that there are some issues around that.

    It sounds like you have a specific axe to gri... er, sharpen. What motivated you to post this?

  • claireymc-INGclaireymc-ING Posts: 15 ✭✭
    edited December 2022

    Nah, nothing personal nor misunderstood, just Facts.

    If you can find another post from Niantic clarifying how reviewer "Ratings" are based, by all mean please share.

    Not that it matters, have been reviewing since Wayfarer came out though do not make the time to post in community.

    Post edited by claireymc-ING on
  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    I'm not sure what you mean by "clarifying how reviews are based".

  • patsufredo-PGOpatsufredo-PGO Posts: 4,214 ✭✭✭✭✭

    Then the inverse should be true. If I choose to reject an ineligible nomination but the rest of reviewers accepted it instead, then I won't get the agreement, my ratings went down and that nomination will likely to stay as Schrödinger Wayspot in the database.

    Too bad you're asking now, Niantic are rarely to give detailed clarifications outside of AMA, at which for this time question submission period was already closed.

  • MargariteDVille-INGMargariteDVille-ING Posts: 2,846 ✭✭✭✭✭

    You're hyper-focusing on the first in the list of what NianticAaron said. He ALSO said

    - Take your time and review each segment on its own merit. 

    Interact with the nomination. This includes: expand the pictures, click title for a google search, click the nearby wayspots, click the "Google" icon (bottom left of map) to get more info, switch from streetview to satelite, zoom in and out. EVERY web site tracks how you interact with it - it's standard code. Even your performing arts center email knows if you opened it, how far you scrolled, what links you clicked.

    Also you said "If Niantic does not have a way to determine if the QUALITY of the review is correct (meaning legitimate rejects do not negatively impact rating) then that is a problem that needs to be corrected." They DO check our quality, by throwing in what we call (slang) "Honeypots". Here's where they were introduced: https://nianticlabs.com/news/wayfarer-checkin?hl=en

    taken steps to pre-approve or reject certain Wayspot nominations prior to them entering Wayfarer. Reviewers who disagree with our assessment may see a dip in their performance score.

  • claireymc-INGclaireymc-ING Posts: 15 ✭✭

    Any long standing reviewer knows about Honeypots. I would hope other experienced reviewers not only enlarge the photo submission(s), also double check google maps to verify the object and location, they are no-brainers. So rather than guess about what @NianticAaron really meant, lets hope he or another Niantic representative responds to clarify since they did not in the original post listed above when asked.

  • 26thDoctor-PGO26thDoctor-PGO Posts: 4,914 ✭✭✭✭✭

    With the complete mess Wayfarer has become I'm not sure why any of you are placing any credence on anything Niantic dtay or ask you to do.

  • 26thDoctor-PGO26thDoctor-PGO Posts: 4,914 ✭✭✭✭✭
  • SeaprincessHNB-PGOSeaprincessHNB-PGO Posts: 1,606 Ambassador

    If my vote agrees with the final decision reached by the community (whether that is to accept, reject, or duplicate), my rating goes up.

    If my vote disagrees with the final decision reached by the community, my rating goes down.

    The only part that is not clear is what happens if the community decision is overturned by an Appeal. If I originally got a disagreement and Niantic overturned that, do I now get an agreement? If I originally agreed and Niantic overturned that, do I now get a disagreement?

    I think no because that's a lot of coding work for them to do.

  • MargariteDVille-INGMargariteDVille-ING Posts: 2,846 ✭✭✭✭✭

    Right, backing out agreements would be way too much work. Plus, people could even lose badge levels, when they actually agreed with the majority at the time. THAT would raise a stink.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    I don't think appeals change your rating.

    This question is usually asked in the context of reviewers who rejected something being penalized when it's accepted on appeal. That should NEVER happen. The reason is that there is always a time lag between the original review and the appeal being processed, and the information available for review can change in the intervening time. Specifically, satellite and street view could be updated. If I reject something because it's a playground in the middle of a farmer's field based on the information available to me then my review was correct despite it being a park in satellite view when the appeal is reviewed.

Sign In or Register to comment.