I have a theory about "weird" rejection reasons.

Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

It has become very common for people to complain here and in other forums about their wayspot candidates being rejected for "weird" or "wrong" reasons. It's such a frequent refrain that it's impossible to overlook. Rather than jumping to the conclusion that reviewers have become stupider (which is certainly possible, I'll admit) I decided to think about it systemically. What could have changed? Of the possible causes, which ones are most plausible? Which ones best fit the evidence? I came up with a list of four possible causes:

  • Reviewers have deliberately changed their behavior to choose random reasons
  • Niantic has changed something in their UI so that people accidentally click the wrong reason more frequently
  • Niantic has changed the way they decide which rejection reasons to include in email
  • There's a bot or an autoreview script being used that chooses random rejection reasons

Are there any that I'm missing? Maybe. If so let me know what I left out. Here's my analysis of those options.

Reviewers have changed their behavior: I think this is the least likely explanation. If it was limited to a few small geographic areas I would be willing to consider this possibility, but reports are coming in from all over the world and it appears to be a widespread change. It seems entirely implausible that reviewers all over the world have decided in unison to change their behavior.

Niantic has changed the Wayfarer UI: This is a more plausible explanation than reviewers worldwide suddenly losing their minds. If Niantic changed the interface and made it easier for reviewers to misclick/mistap then this change in output of the system wouldn't be surprising. I'm not aware of Niantic having made any changes, but I only review on a desktop computer and can't speak to the mobile experience. Is it plausible? Yes, but only if there have been UI changes.

Niantic generates rejection email differently: This is quite possible. Niantic's system for deciding what goes into rejection email is a magical black box from our perspective. I'm going to make up some numbers as an example. Let's say that a year ago you would get a specific rejection reason included in the email if five reviewers chose it. If Niantic changed the algorithm so that it now only requires two reviewers to pick a reason then people would get a lot more email with "weird" rejection reasons without player behavior changing. Is it plausible? Absolutely, with one underlying assumption: that there have always been reviewers giving silly rejection reasons. Does it fit the evidence? Yes. Remember that Niantic doesn't tell us when they change things under the hood, though we often reverse engineer them by crowd-sourcing information.

Of the above three I think this is by far the most likely as it would be a systemic change and it's one that players wouldn't know happened. In fact, I was pretty convinced that this was the answer... then I thought of a fourth possibility.

Reviewers are using a script or bot: I've heard plenty of talk about local reviewers sharing their submissions with each other and voting in unison on them. I've also heard anecdotal tales of communities that would reject anything that was submitted by someone outside the group. I found a couple of vague references to such a tool but no concrete evidence. It seems certain that such a tool exists, and that a few groups are using it. Is something like this being used widely enough to cause an epidemic of "wrong rejection reasons"? I have no way of knowing. Is it plausible? Yes, though there would have to be a large enough community engagement with it to get this level of outcry. Does it fit the evidence? Yes, modulo the following caveat.

So what's the bottom line? I completely reject the first one. I don't believe that enough reviewers worldwide just started changing their behavior at the same time in a silly way... it doesn't make sense. I reject the second one based on having no evidence of a recent UI change that would cause these sorts of errors... but would re-evaluate if I learned about such a change. Number three seems completely believable to me, though it requires me to accept that there was always some level of rejection-reason noise hidden in the system. Number four seems completely believable to me, though it requires me to believe that there is a system or systems for centralizing data and giving it back to local reviewers. As an engineer I can easily believe this. I can imagine it either giving reviewers instructions for how to vote, or maybe filling in the votes automatically. Do I have hard evidence of a system like this? I do not.

Comments

  • oscarc1-INGoscarc1-ING Posts: 366 ✭✭✭✭✭

    I think it's more likely that Niantic have stuffed up and when a user selects a legitimate rejection reason (eg. "Private Property"), the email displays "Explicit Activity" or something instead.

    Niantic have shown in the past to be incompetent and unable to show accountability or transparency in their methods or processes, so anything could be the case really. Not completely discounting bad reviewers, but with the sheer number and trend of such incorrect rejection reasons, it's more than just some isolated cases, there's something wrong somewhere and the first place to look at would be Niantic.

  • DerWelfe2205-PGODerWelfe2205-PGO Posts: 374 ✭✭✭✭

    The last time I put an URL in the supporting info it got rejected as "URL in title or description"

    (Resubmitted it without the URL and it got accepted lol)

    I think you underestimate the number of incompetent reviewers.

  • sophielab-INGsophielab-ING Posts: 266 ✭✭✭✭

    I see 2 anecdotal stories about stupidity. I believe stupidity is actual a constant in the system. Stupidity neither increased greatly or decreased.

    The increase in rejection reasons not fitting is most likely 3 or 4.

  • WheelTrekker-INGWheelTrekker-ING Posts: 3,387 ✭✭✭✭✭

    For me the fourth option is the one that has lower chances to be related to this issue. In order for that to happen, there should be a script available worldwide that would automatically reject using random reasons.

    I would expect that if such script existed we would be fully aware of it (a script used worldwide, not used only in small communities) because someone should have talked about it.

    I know that there is (or used to be) a website that displays the rating provided by the reviewers that use that website/script, I don't know if it can automatically rate that nomination for other reviewers, but if it does, I would expect it to use the same rejection reasons in order to achieve better agreements. Using random rejection reasons seems counter-productive.

    On the other hand, the first option is the one that I would choose. I fully expect that people might select wrong reasons, sometimes due to a misclick, others due to try to "send a message" to the person that sent the nomination and the last one is "in order to avoid patterns". People are so burned out by the soft-bans in Wayfarer that try to use any way to avoid a ban after rejecting several bad nominations and so they do as "suggested" by Niantic and don't repeat the pattern of previous rejection reasons even if it's just "doesn't meet criteria"

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @DerWelfe2205-PGO I don't underestimate them... I review lots of submissions and those same incompetent reviewers are even more incompetent submitters. In the case you described, the rejection reason shows poor judgment but it's a plausible/understandable one. It's wrong but not nonsensical.

    I just refuse to believe that a significant number of reviewers around the world all just randomly decided at the same time that they were going to start using "live animal", "license plate", "body part" and the like for things that are clearly none of the above.

  • SiIverLyra-PGOSiIverLyra-PGO Posts: 952 ✭✭✭✭✭

    The script/bot option is an interesting one.

    Completely anecdotal, but when trying to apply it to my own ongoing saga with the local group of abusers... What I always found curious is how effective they are. I know they use a mass of alt accounts, but I've always wondered how they find they time to operate so many of them as reviewing tools - which they clearly do, considering the amount of obviously abusive wayspots that they get accepted into the system, and the amount of wayspots they get rejected from areas where people who oppose them play.

    A script/bot would provide some answers. For instance, they could program the bot to reject all - or a large percentage of - nominations from a certain area (using S2 cells, perhaps?). Nominations from their supporters who happen to live in that area possibly get whitelisted beforehand (using the nomination's title? or its coordinates?) - because I do know for fact that those exist and get accepted no problem.

    And then the rest of the nominations in the area get rejected constantly, with seemingly random reasons each time.

    I also know that this group operates bots in a different way - to map existing pokestops and gyms in their area for a public website that they run - so that sets a precedent.

    So if there IS a reviewing script... and recently abusers from all over the world started using it (for easy agreements, maybe?)...

    This is all guesswork - probably wild too - but it does fill some holes that I've been wondering about.

  • Lechu1730-PGOLechu1730-PGO Posts: 537 ✭✭✭✭

    @Hosette-ING Maybe a little bit of all?

    1) Reviewers have deliberately changed their behavior to choose random reasons

    I don't think reviewers themselves have changed, but the average reviewer may have. Don't forget that with the introduction of the platinum badge requirements to reach level 50 a lot of people see the 1500 agreements as something easy to get. That probably added a different type of evaluated to the mix, people that are doing it as a chore and not because they like it. I think these evaluators when bored after seeing too many bad nominations might be more likely to choose a silly rejection reason. When I see live animal, for instance, I wonder if the reviewer is actually insulting the submitter, implying that the animal is to be found behind the camera, holding it.

    2) Niantic has changed something in their UI so that people accidentally click the wrong reason more frequently

    I review almost exclusively on mobile and I don't think you can easily misclick and send a wrong reason. After a long reviewing session I might select an incorrect reason and I have to go back and select it agai, but it doesn't happen often, maybe once every 100 reviews. If I consider a standard error rate in the verification step, it would mean that about 1 in a 500 to 1000 reviews gets a wrong rejection reason.

    Nevertheless, Niantic did change the UI to add more rejection reasons and one of them "Inappropriate Location" is clearly being misunderstood by the community. I think it's wrongly used to signal that the PoI is "generic".

    3) Niantic has changed the way they decide which rejection reasons to include in email

    They might have, but also remember that rejection reasons were only shown in English language emails. Fixing that problem exposed the silly reasons underlying a rejection to a much wider audience.

    4) There's a bot or an autoreview script being used that chooses random rejection reasons

    I agree that a widely used bot seems unlikely, but localized usage as @SiIverLyra-PGO Described is entirely possible.

  • DerWelfe2205-PGODerWelfe2205-PGO Posts: 374 ✭✭✭✭

    That is even worse in my opinion. The general sentiment is "improve your submission to get it accepted" . In my case I had to lower the quality of my submission in order to get it accepted..

    Also a lot of places only recently got rejection reasons included in the mail so the problem didn't randomly appear it simply became visible.

  • Ochemist-INGOchemist-ING Posts: 355 ✭✭✭✭✭

    Are we sure the problem has actually gotten worse? Complaints about bad rejection reasons began pretty much as soon as they started appearing in Prime submissions. If the volume of complaints has increased, could it simply be that, as, e.g. @DerWelfe2205-PGO pointed out, people submitting in languages other than English only recently started getting reasons in their rejection emails?

    I remember begging Niantic to include rejection reasons so that submitters could learn something What a mistake that was -- be careful what you wish for!

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭

    Yeah, rejection reasons have always been weird or misapplied ever since we started receiving rejection reasons in our emails.

    I wouldn’t say reviewers have become worse or “stupider” (as Hosette hypothesized). I think that reviewers had this trend of misapplying rejections ever since OPR, with Reviewing opening to PGO players causing a temporary period where there were more reviewers who were more lax (and some bad stuff got approved, oof). And in recent months we’ve returned to the type of reviewers who stick around with how Niantic operates community things like they did in OPR.

    It could be for a variety of reasons like Hosette and others have stated earlier. Possibly because of botting, collusion/abuse, and my personal fave: uneducated reviewers. Just look at all the PoGO youtuber videos about reviewing when it first came out with tens of thousands of views and look at how bad some of them are.

    The big thing to curb uneducated reviewers is better resources to check criteria and better reviewer education. And a good way to curb botting/abuse (other than existing report methods and bot detection) is to provide better incentives for players to review accurately to get more players into wayfarer.


    TL;DR

    Weird/Poor rejections with bad reasons always happened, we can just see it now. Best solution is to give reviewers better resources to review better and encourage more players to review to counter the bad ones.

  • KetaSkooter-INGKetaSkooter-ING Posts: 177 ✭✭✭
    edited April 2021

    The rejection email always seems to give 3 and at least 2 reasons. I suspect that the reason comical reasons get included is because the system tries to give 3 reasons so if 90% of reviewers select one reason then a few outliers select other reasons you'll see those other reasons in the email. The prevalence of inappropriate location is probably from some misunderstanding what that is for.

  • 0X00FF00-ING0X00FF00-ING Posts: 769 ✭✭✭✭✭

    There are many occasions where the number of rejection is only 0-2 too. Yup, zero. As in you only see: .

    Generally that would imply that the nomination received poor ratings, but reviewers didn't 1* the nomination.

    And sometimes you kinda just know that, when you only get a single irrelevant rejection reason, that it was an "outlier"; as in a single reviewer picked a single rejection reason, once.

  • Rodensteiner-PGORodensteiner-PGO Posts: 1,705 ✭✭✭✭✭

    Wayfarer is broken. We all know that for a long time. Sadly Niantic still has not done much about the general flaws of the system that lead to the system go haywire. I still do not give up hope that niantic just stops letting people make submissions, and then let everyone work off the backlog so that Wayfarer can have a fresh start. Also, the Branding to Wayfarer didnt work. I did get my rejection mail today with an english text. ( Der Vorschlag erfüllt die Akzeptanzkriterien nicht., The real-world location of the nomination cannot be confirmed to exist at the submitted location. )

  • saarstahl-INGsaarstahl-ING Posts: 184 ✭✭✭

    Hypothesis 5: Wayfarer sends fewer reviewers to "poor" status (at least there are not many complaints on this forum any more), so there is a larger pool of not-so-great reviewers.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @saarstahl-ING Oh! That raises an interesting point. We know that if your rating is low it counts less toward the review score. I don't if we know if less = 0 or not, and I'm pretty sure we don't know whether your rejection reason shows up in email if you're fair or poor. I don't even know how to make an educated guess about that, but I have a coin I could toss.

  • Albysf49-INGAlbysf49-ING Posts: 20 ✭✭

    I use both Pogo and Ingress in English since I discovered that rejection emails in English contained the reasons, even if I am Italian. There have always been silly reject reasons, but lately the situation got worse. Out of three reasons, one is almost always "explicit or inappropriate activities", and the two others are very often random.

    This morning a friend's request of a pedestrian place was refused because of "the photo represent one or more people" (the foto was shot at 5 a.m. in order to avoid that), "natural element" (maybe because of the trees?) and "recognizable face" (again, there was nobody in the photo). 3/3 silly reasons.

  • toniukupaoni-INGtoniukupaoni-ING Posts: 41 ✭✭

    There is a possibility, player/s (multi acc) are randomly selecting rejection reasons for random nominations, so they could go through queue fast. Sometimes adding more favourable ratings so that they don't get caught by a cooldown.

    Why would someone do that?

    So that they get through to their (in their area) targeted nominations as fast as they can.

  • David75Chu-PGODavid75Chu-PGO Posts: 23 ✭✭
    edited April 2021

    I am relatively new to the Wayfarer community (300 reviews in) and thus far I have found it an interesting experience, especially once you start reading the forums.

    I believe, like most community endeavours, that in general reviewers are largely trying to do the right thing. There will always be the few that spoil it for the many. There will always be the few who try and game the system, although abusing the process to fast track a nomination that is not guaranteed to be accepted feels like a lot of effort, potentially for very little reward.

    I think some of the wording for making judgements could be clearer, but there is a lot of room for subjective differences of opinion. For example if I say a photo is over exposed but somebody else thinks it is ok or artistic, who is correct?

    Unfortunately you will always get rogue reviewers. You will always get bad nominators, and there are some things that some people find hard to accept. People hate being told they are a bad driver, people hate being told their photos are rubbish, and people really hate their nominations being rejected (because of course it was perfect) :-)

    I'm too new to this to be able to say whether or not there has been an increase in people gaming the system and resorting to auto reject. It feels unlikely.

    However, I have observed a couple of themes in the bits and pieces I have read.

    For instance, some rejections that have invoked the ire of somebody on the forum, frequently look like legitimate cases for rejection, and it would often be easy to improve the nomination, especially as people in the forums are usually offering lots of advice. On the flip side it feels likely that some nominations that should be approved are indeed being rejected for no good reason,

    On a slightly negative note the more I read the forums the more I feel I will get better agreement numbers if I become harsher in my judgements, heaven forbid that I approve a red postbox... :-)

    Joking aside, perhaps the system needs to be more streamlined. Perhaps fast tracking nominations should not be tied to reviews but rather to people's all round persona such as fast track points being awarded for being supportive in the forums or providing detailed feedback. I don't know really, but I have to believe that the community as a whole is generally good and well meaning.

    Post edited by David75Chu-PGO on
  • flatmatt-PGOflatmatt-PGO Posts: 2,137 ✭✭✭✭✭
    edited April 2021

    Do you have any specific evidence that bad rejections have actually increased? Because I can think of several other explanations:

    - More people are finding the Wayfarer forum and other places like it to complain.

    - More people are able to see their rejection reasons since Niantic started providing rejection reasons in non-English emails.

    - There actually hasn't been an increase in bad rejections or complaints, but people perceive an increase regardless.

    Not saying that any of the above are true, just that they are possible explanations for a seeming increase in complaints.

    I'll also add this to the list of possible reasons if, in fact, there has been an increase in bad rejections:

    - Niantic has added new rejection categories (inappropriate/sensitive location in particular) that are not explained properly to reviewers.


    Edit: I see @Ochemist-ING has made some of these same points above. I missed that post in my earlier browse through the thread.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @David75Chu-PGO You are absolutely and unquestionably correct that many "THIS WAS REJECTED INCORRECTLY!!!!1!!!" posts are often from for things where the rejection was correct and the submitter hadn't taken time to understand the guidelines. There are also lots of cases where the thing being submitted was legitimate, but the submitter presented it so badly that reviewers couldn't understand what was going on. (I've written about this at some length.)

    In this particular case I'm looking specifically at rejections that may or may not have been legitimate, but that had rejection reasons that were ludicrous. Imagine a basketball court in a park that came back as rejected for blocking emergency services, body part, and explicit content.

    To answer @flatmatt-PGO's question, I don't have a way to quantify this but my perception is that the rate of people complaining about rejection reasons rather than rejections has increased substantially in the last few months. I see it here and in other forums. My initial reaction to these was, "Oh, c'mon. The key information is that it was rejected and who cares what the reason were" but then I saw so many similar discussions that it felt clear to me that something else was going on.

  • HaramDingo-INGHaramDingo-ING Posts: 1,725 ✭✭✭✭✭

    These theories are very out there, yet seemingly also very possible considering the absolute trash rejection reasons that I get on the daily. Shopping centres for private residential property, playgrounds for séxual activity, statues for harassment, you name it.

    I will purely leave the blame on the increasing amount of foul and selfish reviewers who upgrade nominations that have photos taken from their car or are absolutely ineligible. There was a Reddit post that kind of outlines the following theory:

    ...knowing this about locations, if 5 reviewers out of 50 can change the location of a proposal, we can think that 5 reviewers/trolls out of 50 are able to take down a 5* proposal. This will explain the silly reasons of lots of rejections. We can argue that a troll reviewer should have "poor" performance, so their votes count less, but I don't believe this is true.

    Niantic would not build a system where selecting "Does not meet criteria" send a email to the wayfinder that their nomination had "explicit or séxual activity", despite the broad consensus of incompetence. But despite the majority of my rejections (almost half of all rejects now) for such a stupid reason, I would actually focus the blame on troll reviewers.

    Again, I remain adamant that there is absolutely much more weighting on trolls rejecting for specific reason rather than people just saying "this is acceptable" and give it 3-stars all across the board. I'm more inclined to believe that the "séxual activity" and abuse reasons have far more weighting than just a regular does not meet criteria rejection reason. And if Niantic actually can't see the individual wayfinders or the specific reviews who are voting in such a way to fully investigate it, then that is negligence on their behalf.

    We could analyse and propose theories all the way, but it's just a waste of time if Niantic doesn't even have the tools to do specific analytics themselves.

  • HaramDingo-INGHaramDingo-ING Posts: 1,725 ✭✭✭✭✭

    Oh, and to further pour the salt, a 7-Eleven just outside my local was approved. If a fuel station was deemed as eligible and meets criteria by the community, than what do of other nominations that are seemingly on the fence like murals, playgrounds and the such? Most of the time, reviewers will care only about their immediate area that matters to them, and reject all else.

    I used to think that in general reviewers were doing the right type of thing. But it is actually so much more fun and easier to troll instead. The indicators of poor/good/great have barely any impact or weighing as a Wayfinder, and it is ridiculously simple to stay on top of the best status rating while voting against everything and thinking everything is libidinous.

    For so long as they roam like a pack of wolves, millions of sheep are slaughtered with no recourse. i.e. they're more than welcome to reject great candidates, approve junk like 7-Elevens and donated statues in front of houses and Niantic can only sit and wait until someone posts a massive report for them to start, there are no proactive measures involved.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @HaramDingo-ING I saw that speculation that five reviewers out of fifty could take down a 5* submission, but I think that their logic is faulty there. I've written about it elsewhere, but the TL;dr: I think that the overall submission yes/no is based on a combination of all of the people who reviewed that particular candidate using some magic Niantic formula. Once there's a yes I think Niantic checks for some X% of reviewers to have suggested a new location. I think the threshold for a new location has to be lower than for the overall pass/fail because it seems likely that at least half of reviewers never move the pin. Thus, I think that extrapolating from the number of people it takes to move a pin is faulty logic.

    Do you think there is a large enough increase in troll reviewers, worldwide, to effect this change on their own? That doesn't pass my smell test. I can certainly buy that Niantic changed the system in such a way that the effect of troll reviewers is more visible (that's my third item.)


    I can think of multiple ways that Niantic could address the system of local trolls. In fact, the other night I thought of one that might actually be workable.

Sign In or Register to comment.