The loss of reviewer feedback in Wayfarer

General Discussion
1

An open letter to @NianticTintino (CC the Wayfarer Community)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Dear @NianticTintino

Since I began my first stint as Wayfarer Ambassador in 2022, one of the areas I have pushed for is for greater feedback for users.

Feedback is central to learning. In education, timely and specific feedback is widely recognised as one of the most important ways students improve their performance. The same principle applies to Wayfarer. Users enter the system as inexperienced submitters and reviewers. Niantic provides information about the criteria, then relies on users to apply those criteria accurately.

So, the question is: How do users know whether they are doing this correctly, and how do they improve if they are not?

For submitting, there is constant feedback provided through outcomes. Nominations get accepted or rejected, and when users appeal they even get a written explanation from Niantic if it doesn’t meet the criteria. This quickly helps users learn what is and is not acceptable.

But what is the feedback for reviewing? Well, there is very limited feedback…

  • Users can see their overall number of agreements, but it isn’t possible to track which votes led to agreements and which didn’t, and whether a low number is due to disagreements or long queue times.
  • Users previously had (but no longer have) access to a Wayfinder Rating which provided a summary of their performance as a reviewer.

image
image764×246 10 KB

The old rating system was not perfect, but it was something. Over the years there have been many forum posts, Reddit posts and Discord conversations where users have reached out seeking help because they had a Poor reviewing status. The community stepped up in each of these times, supported users, and gave advice to help users develop. These organic communities took on the mentorship and educative role which Niantic would not or could not do.

The most recent changes to the Rating System have come with many perks. There is universal praise for the increases to submission and edit caps and accruals. This is an area where Niantic has listened to the community and has over-delivered. Well done.

However, this change also involved hiding one of the only feedback systems Niantic had for reviewers. The new rating system is not a measure of reviewing performance, and it is not designed to be. The new rating system takes into account submitting as well as reviewing. And it is designed to never go backwards, with the exception of abuse sanctions. So it is very possible to have someone who is a ‘Poor’ reviewer who is simultaneously ‘Champion’ on the new ratings scale.

Previously, a reviewer with Poor rating at least received some feedback that could lead to them reaching out and getting help so that they can improve their reviewing. Now, a reviewer may be performing poorly while remaining completely unaware of it. Even if they are well-intentioned, they may continue voting against criteria without understanding that they are doing so incorrectly.

In the FAQ provided in the announcement post ( New Wayfinder Rating System! ), it was stated that Niantic will continue to measure the Wayfinder review activity (i.e. this metric is still tracked), but that users will not be able to see it. It would be one thing if the old rating was being removed entirely and not tracked, however, that does not seem to be the case. This information seems to still exist, but is now hidden. It was not explained why Niantic made the decision to hide the existing information.

In addition to providing feedback to reviewers, the old Wayfinder Rating system was explicitly related to length of cooldowns (Source: Cooldowns Update — Wayfarer ) and number of reviews required for a decision (Source: December 2021 AMA). The FAQ provided in yesterday’s announcement is quick to state that the new Rating System won’t impact these things, but that the internally tracked data will still be used to determine cooldown periods, etc. This suggests that users will still be affected by internally tracked reviewer performance data, but without being able to see the information that affects them.

Overall, I am reaching out because this program is something I have been passionate about for many years. It saddens me that such a fantastic move forward (submission quota) is accompanied by the removal of reviewer feedback systems. @NianticTintino, as you would know, I have shared my suggestions for improvements to feedback systems with you over the last few yearsš. I definitely did not think the old system was perfect. Indeed, it had a lot of room for improvement. But it is disheartening that the existing reviewer feedback system was removed with no replacement in sight.

@NianticTintino, I would be curious to know what Niantic’s rationale was in hiding the old rating system from users. (I’m not asking for the rationale of the new system – I see value in being able to reward users for both reviewing and submitting – but simply why was the old one removed? It would have been technically feasible to show both simultaneously.) I would also like to know whether Niantic has anything planned on its roadmap to improve feedback for reviewers.

As I am not sure whether Niantic will provide a direct response to this matter, I also invite the community to share their thoughts: How would you improve feedback to reviewers? What would you like to see in a future Wayfarer product? Do other users agree with the sentiment of my post, or was the old Reviewer Rating so flawed that entire removal was for the best?

I hope that this sparks some conversations and helps us progress to a better Wayfarer product in the future.

Kind regards,

Tntnnbltn

¹ See for example pp.24–29 of the '2023 Wayfarer Roadmap Input’, shared with you on February 23, 2023.

2

I agree feedback when we are doing well or poorly is really helpful. We recently had training at work on the importance of good and constructive feedback.

3

I think feedback is crucial, so that reviewers who want to be reviewing properly can find the information necessary to know how they are performing.

One problem is that agreeing with the majority decision does not necessarily mean you are reviewing well. Along this, rejecting an ineligible submission despite knowing it would be accepted by the majority would lead to a small downgrade in rating.

This might be a problem Niantic have noticed and could not resolve.

4

I LOVE the new system but I do not understand why the need of old reviewing system need to be removed.

This is actively rewarding bad reviewer because if I am reading correctly

This mean, a bad reviewer can have same voting weight as a good reviewer, which feel really unfair. Hypocritally, if I want to get the platinum medal, the best way is to spam “no” on every submission I want to review. Obvious this is an extreme case, but the fact that it can happen concern me.

5

I read that as meaning the new Rating (up to Platinum) has no bearing on the weighting of reviews, but the hidden internal (old) rating could still be used to weight reviews (Niantic have not said this or said the opposite, meaning it could still be happening).

6

Also here to agree that reviewing feedback is absolutely crucial, especially in a system where we’re only moving forward and anyone who wants it will have hundreds of nominations at their fingertips.

The old rating system depended on the system having more good actors than bad/uninformed doing the voting. Based on what I see being approved, that’s not the case anymore. I report abuse in every single reviewing session. At least abuse gets acted on (mostly). More than that, I see the death of the criteria. Signfarer is going strong, and in most cases, the signs cannot be removed from the game, despite not meeting criteria. Reviewers who approve those things get a good rating (as they agreed with the larger decision) and more pokestops.

A time has come (and frankly long past) where a decision needs to be made by the Wayfarer team about whether the criteria still matter as people run into the world with their hundreds of submissions. Is it Wayfarer? Or Signfarer?

I do love the new update to submitting. But I worry about how much more abuse and just generally non-criteria-meeting submissions will flood the system. Without adjustments to what the criteria currently are or to removal criteria, it’s a mess at the moment.

7

I have seen lots of posts from people suddenly getting the wayfarer medal. That’s great for them, and I have been excited to see those posts, but it made me think about why they were suddenly awarded it? Had their performance been too poor under the old system? And suddenly it is acceptable?

Getting disagreements that hurt my rating when I am certain that I reviewed correctly had me so turned off that I had not been reviewing. (That is a huge part of the issue with not being able to simply reject for “Does not meet criteria.”) Specific feedback on when we got disagreements could be helpful for education purposes, but not if the community decision went against criteria.

I wondered if introducing this new rating system meant Niantic thought the Machine Learning model had been tuned so fine that any nominations that it allowed into voting were most likely things that should be accepted. I did some reviewing after this announcement, and that is definitely not the case! I did a lot of reports of existing Wayspots, too.

I do not know the answer to getting everyone who reviews to review correctly per criteria. When I began reviewing, there was a terrifying test to take that could never be taken again if the answers were incorrect. I don’t want to go back to that. The test that could be taken open book every 15 minutes until passed was too easy to brute force, and canned answers could be found online. But with no feedback, there is no reason to change behaviors.

8

One of the proposed feedback systems which I developed and shared would utilise Niantic reviewers.

Already, some reviews get taken in for Niantic review. Rather than have those solely assessed by Niantic, if they were also assessed in parallel by the community, then this could be used to provide feedback to reviewers. i.e. Did you agree with Niantic or not?

IMG_6392
IMG_63921117×868 75.7 KB

(This would assume that the Niantic reviews are in line with the relevant criteria and are supposed to be the benchmark for what should be accepted or not.)

To more greatly assist users in learning what aspects of their reviewing need improvement, it would be beneficial to show the specific details of the review. Below mockup is a table comparing ‘User’ reviews to ‘Community’ decision, but this could be User vs. Niantic decision if using the above system.

IMG_6394
IMG_63941170×637 100 KB

9

I hope not. A few days ago when reviewing I had several instant rejections for things like street name signs, that previously I would have expected to get filtered out by the ML system. Either people are finding ways to get round ML with low quality submission or the ML system filtering has been downgraded.

10

Based on removal reports that I have made and have seen other people posting about (e.g., multiple disc golf wayspots), I would not be able to trust the Niantic reviews as being in line with criteria any more than the community reviews. I wish I could.

11

Someone on WDD offered this insight:

I think a lot of reviewers were prompted to ask questions and get better educated if their agreements weren’t progressing. Now it seems like there’s no feedback to let them know they aren’t reviewing per the criteria.

Yes, they’ll get e-mails with warnings and ladder punishments, but that feels like going straight to DEFCON 1 if they haven’t been getting other feedback.

I thought it was a really good point. I have seen users highly stressed when receiving a warning email.

12

At least when we had the wayfarer test new Wayfarers had a front and centre explanation of how to review. Now they must read the criteria, but will not know if their methods need adjustment. New reviewers often would be alarmed when their rating dipped and would then come to the community for help.

13

Part of why this matters is that there’s a lot of rhetoric on places like reddit to the tune of “I don’t dare use wayfarer because you WILL get banned for submitting/reviewing.”

To make the system more opaque only feeds into the narrative that the moment you try to wayfarer, you’ll start getting strikes.

Edited to remove direct reddit link, but whoever flagged this post isn’t helping anyone by doing so.

14

I’ll use this as a chance to bring up another form of feedback on reviews I’d like to see. I also can understand why we won’t see it.

We were told how many agreements we earned. However, we don’t know (didn’t know) much about those where we were not on the agreeing side. In statistics there is the concept of type 1 and type 2 error, namely where a correct hypothesis is rejected or an incorrect one is accepted.

For those reviews where we did not earn an agreement, it could be instructive to know if that happened because we voted to accepted what the community rejected, or we rejected what the community decided to accept.

In a perfect world, it would be instructive to know that for each review, but that’s not going to happen. Even on a summary level I feel that could be helpful and instructive to reviewers.

15

We are already seeing the mentality spill over into this forum of “it’s a game, approve everything.” Without education on wrong decisions, that is headed towards the norm. If agreements no longer matter for anything except upgrades, which are no longer needed for the majority of submitters, there is no incentive to learn criteria and review correctly.

And we are already seeing where the folks who know criteria and reviewed correctly got a disagreement anyway.

And where the correct rejection is overturned by the appeals team.

16

wayfare is so way overbloated it’s amazing.

17

I think tracking feedback on all reviews would have a major IT impact, it will require a lot of data storage (maybe it can be included in the monthly email)

It would be nice, if the reviewers still can see that review rating.

18

I don’t understand this comment. Would you like to explain?

19

It shouldn’t be this hard to play the game. wayfarer scares people away from nominating. yes, it’s true, many people around this area have commented on it.. in many nominations, it’s entirely the wording which gets passed/failed. it’s maddening. I nominated 2 routes last week, straight down a road. both stops were STRAIGHT DOWN THE SAME ROAD about 6 blocks apart. verdict? rejected! “inaccessible area”…seriously? something is VERY WRONG.

it’s too hard for no reason. 98% of all posts I see are about rejections and appeals.
This is a construct created by Wayfarer website. And it’s not right. It does not make for happy people.

20

Hi,

Routes are not part of wayfarer, this is a pokemon go issue (no one can help you here with that, though I can tell it is usually easy to appeal these ingame, and get them accepted).

Yes, naturally people come here for appeals and rejection, why would you come to the forum if your nomination was accepted.

People here are keen to help find new wayspots, but yes, we do like them to meet the standard.