A proposal to help Niantic speed up the review pipeline by about 10%.

Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

Based on my personal statistics and recent observations, I think that at least 10% of what I have reviewed in the last year is either a duplicate of an existing wayspot or is perhaps legitimate but is so close an existing wayspot that it will never go live in any games.

We know from recent discussions that there are more things being submitted than there is reviewer capacity to handle them all, and we see that in a lot of the frustration that is expressed here and in other places. People are rightly unhappy that they wait six months or more for a submission to go through the pipeline, and even with upgrades sometimes things don't go into voting for a long time, or they get stuck in voting even with an upgrade. (I'm an old-school Ingress player, and yes I've experienced multi-year wait times. Those weren't fun either.)

So there's maybe 10% of stuff that will never go live in any game. Niantic already knows that a pin is within 20 meters of an existing wayspot. Why not save 10% of reviewers' time and reject those automatically?

Because I think this way, here are my arguments against this proposal:

  • Sometimes the pin is in the wrong place and it will be fine once reviewers correct it. I agree that this will happen occasionally, but probably not very often. This case can be handled with gracefully-worded text in the auto-rejection email.
  • Some dupes have pins placed more than 20 meters apart. This is also true.
  • Why even let people submit things if the system will automatically reject them? I agree with this, but it might be easier to implement this filtering on the review side rather than on the submission side.

My arguments in favor:

  • As I mentioned above, my back-of-the-envelope calculation is that it will speed up the pipeline by about 10%.
  • It will save reviewer frustration, because it's no fun wasting time on something that has no hope.
  • It will get a very quick response to to submitters whose submissions will be blocked by proximity rules.

The bottom line is that I think the cost of having a large number of duplicates and things blocked by the universal 20 meter rule is pretty high, and that saving that much reviewer capacity would be more than worth the rare case where something could have been corrected by reviewers.

Tagged:

Comments

  • Thor3381-INGThor3381-ING Posts: 220 ✭✭✭

    The auto-filter on the 20m rule will handle some cases, but it also has a big downside: If you auto-filter on the 20m rule the misplacement of the pin will get worse. Moving the pin away from the actual subject in order to be 20m from an existing wayspot.

    This is already happening with the (some) PoGo submitters who look at Cell-level

    Currently I spend time in moving the pin to the correct location, even if it's only 5m off (presuming the pin is placed on the location where the picture was taken)

    If an auto-reject is being sent and there are nearby wayspots then I'll have to mark them for wrong location because I will no longer be able to make the distinction if the pin is intentionally 5 to 10m off or not


    The duplicates issue can partly be solved by showing upon submission which nearby wayspots already exist. Many PoGo players are not aware that not all wayspots are in their game.

    The complaints about "my proposal got accepted but I don't see it in the game" could also be handled in the mail that is sent after acceptance. NIA knows using which platform a submission was done and that PoGo doesn't show all existing wayspots. They can easily add a line in the mail noticing the submitter that if there is a nearby wayspot it is possible that their nomination will not make it into the game

  • Thor3381-INGThor3381-ING Posts: 220 ✭✭✭

    Some other ideas

    • A submitters-test cfr the reviewers test could filter-out some cases (but after passing the test you'll still be able to submit the tree in front of your house) But it would make sure that everyone at least knows what can and what cannot be submitted


    • decrease the number of possible submissions per user with a higher limit for active reviewers (one extra per 50 agreements with a max of 10 - you can keep the extra's stacked cfr the upgrades) This is an extra incentive for reviewers a part from the upgrades. The decrease in number of submissions can be temporary, but linking agreements to extra submissions might help us to re-enforce the Reviewer-base (the more we are, the faster it will go)


    • user-priority in the queue depending on your submission-history - higher priority for those who's submissions get accepted, lower priority for the ones with a lot of rejects. Downside of this are the yes-to-all and the no-to-all armies. Also the ones who submit nice but hard to get approved submissions might give up on these gems, which is what we don't want.


    • Some kind of AI in submissions, I've reviewed nominations of elementary schools and kindergardens. There is already some filtering based on words in titles and descriptions. If School is mentionend they could add a pop-up: schools for kids under 12 are not allowed, are you sure you want to continue with this submission


    ps: yes I like reviewing, yes I have more agreements than I need for upgrades

  • Oakes1923-PGOOakes1923-PGO Posts: 419 ✭✭✭✭

    I think if they were to implement this then they would also have to completely revamp the submission tool in games like PokeGo.

    If you have never submitted a stop in Pokemon Go let me tell you it is ROUGH! You are given very little guidance. The "PIN" marker is sort of randomly placed kind of close to your location and you have to find what you think might be the right spot based on a satellite photo and your staring at it from 10 feet away on the ground. When selecting your location you aren't shown anything in game so now you have no idea what you may be close to or if there is another POI that may not be showing up in your game but already exists in ingress. All you have to go on is perhaps another POI 100 ft away that is already in that same cell as the one right next too (which again you don't know exists). Which would make you think that you're fine to submit another stop here but maybe its just over the line for another s17 cell but too close to the POI that doesn't exist in your game (but may be in another). "insert can of worms here"

    @Thor3381-ING I'd also try to gently dissuade you from moving pins by mere feet or meters. For some games it doesn't matter but for PokeGo it might mean the difference between something showing up in game and drawing players to it vs something that will likely be overlooked and never get attention from those folks. Be accurate yes, but to a point. When I review, I will often try to help a submitter out by moving a PIN (within reason) to an open cell its its on the border, rather then have it accepted where it may never be seen by a large base of players. The goal for all should not be to review with an iron fist but to enrich the game play for our communities.

    Its also incredibly hard to actually make sure you are submitting on the right spot without using another app as guidance. For example when I submit I usually have IITC mobile open as well so I can make sure I have the right spot. I usually try to find a tree top or a noticeable color change on the map that I can then reference in PoGo. Most folks will not go to this length. If suddenly you are auto rejecting things based on a 20 meter boundary for something that in game doesn't exist you are punishing players who have no way of knowing unless they start to get super nerdy about it like we all do.

    To fix the problem they need to make reviewing better; I know we heard they are working on a solution, so I don't want knock them before we even see it; but I have my doubts. The also need to come up with a new reward system. I think they need to do away with the agreement award aspect of reviewing. People shouldn't be hunting for agreements, it is a reward system that has formed bad habits. People fall into patterns and reject things regardless of their validity because they are of the mindset "I reject these things". Make rewards something that makes me want to come back and review more often. A weekly task in game for a reward or receiving bonus submissions or something. Drive people to the platform and encourage proper reviewing standards will have more of a positive then making any hard and fast rule for auto-rejections. That will most certainly drive people away, which is the opposite of what the intent is.

    The goal should not be how many things one can wholly reject or accept; and rewarding people for that will continue to hinder the system. Not to pick on you @Hosette-ING , but I know you mentioned in other posts that you are an engineer. I have several close friends that are engineers. They always look at a problem in a very analytical way, which is fine at times. I am in sales and account management so I look at things in a more phycological way, more customer service oriented. What will resonate with my core audience more? What will make them come back to me the next time? Data is great, but data in context is better. The problem isn't rules, or guidance, or systems necessarily; the problem is behaviors. What behaviors do all these rules and systems promote? To me from reading the forums and my personal experience with my own submissions and those of my small group of fellow reviewers the systems in place are driving people away. We can't control submissions from games and there are too many games with varying rules to make sure duplicates don't happen. To your point in another post, we need more review hours. To me the only way to solve that is to get more people interested. You're not gonna accomplish that with auto-rejections.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @Thor3381-ING I think you're right that a few people would deliberately misplace pins.

    However, I would argue that there is a big difference between sophisticated users (who might show up in a forum like this) and the average submitter. If 1% of users start manipulating pins slightly more than before then that still gets (by my calculations) a 9.9% reduction in stuff to review, or if 10% of players do that it would be 9%.

    And yes on showing players things that aren't visible in their specific game. That's a constant source of frustration all-around. I never submit from PoGo personally, but I understand why that's a problem.

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭
    edited March 2021

    I think having a submission auto-rejected would lead to a lot of submitter frustration. Having a nomination you can only use once every two weeks auto-rejected because of something you couldn't see in game (PoGO) or the 20m rule that's not even mentioned in any game or by Wayfarer, could lead to a lot of frustrated players who don't know what went wrong.

    Niantic has mentioned in previous posts that they're working on a solution to not every Wayspot being visible in Pokemon GO. Though we're never given any exact details or timelines, so we can only speculate if and how a solution may be implemented.

    I think @Thor3381-ING covered most of the good alternative ideas and critique that you haven't already. I'm very partial to showing possible nearby duplicates if the pin is placed within X meters of an existing wayspot, since this would let the submitter know that there's a duplicate before burning a submission, and would also help lower the number of duplicate submissions.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @Thor3381-ING Absolutely yes on better education for submitters! I've argued in the past not for a submitter test, but for submitter guidance. Walk the submitter through the process more carefully, give them category-specific guidance about eligibility, and ask them questions that can save them from submitting hopeless things. "Is this on the property of a school for people under the age of 18?"

    A submitter test has the same issues as the current reviewer test in that it measures knowledge at one point in time but doesn't help people keep up with rules changes, it relies on people having excellent memory for knowledge they may not use often, and it's too easy to cheat by getting the answers for other people. I think Niantic should put current information in front of both submitters and reviewers rather than hoping they remember what they (maybe) read once a few years ago.

    @GearGlider-ING Yes on frustration, but do you think it's more frustrating to have it rejected immediately or have it rejected six months later? And yes, as I mentioned it would be better if the rejection happened in-app before you burned a submission. It might be easier to implement it on the back-end, though.

    I have less investment in how it happens, but I think it's super valuable to stop wasting reviewers' time by making them chew through things that can never go live.

  • Randomnessdude2-PGORandomnessdude2-PGO Posts: 10 ✭✭

    I am new to Wayfarer and this community in General. I agree with everything @Oakes1923-PGO said, reviewing in Pogo is ROUGH!! Niantic doesn't need to start auto rejecting requests with another stupid email saying it wasn't valid. They need to start providing Pogo players with much more information.

    Things they could add that would reduce the amount of Pogo duplicates

    • Show us POI in other games when reviewing, this could be a simple overlay from the existing ingress map overtop of the satellite map. This would help by showing what spots already exist but are invisible, as well as would help players be able to judge their location more accurately
    • I know Pogo players are not supposed to know about S2 cells, but that is an old rule that doesn't make any sense when it is crucial to nominating. They don't need to auto reject, just merely say that "This location cannot appear in Pokemon Go due to overcrowding, but it may appear in other games. Would you like to proceed?" I know that this would lead to people moving stops away from their location a meter or two but THAT IS OKAY! If we are following legalese rather than supporting our fellow players in the game we are making everyone experience worse. Nobody is hurt by having a stop not exactly on the middle of a location - Provided it makes sense to have it there.
    • A possibility would be to seperate the two games POI slightly. If a POI in Ingress does not show up in Pogo, it shouldn't exist in Pogo at all. This includes in the reviewing process. Right now they are all connected and it doesn't work. I have a POI near me that if it was moved towards the centre of its location it would show up in Pogo and create a gym, but it cannot be moved because it is on the location, tho the edge of it. If I did not look at the ingress map I would not know it existed and would have already submitted it as a duplicate location - a bad submission and a auto reject as duplication. If the review system was different I could submit only for Pogo. If I know this I could come into wayfarer, and select an option before it goes into reviewing and stop it from showing up in other games and the reviewers would only be shown POI from games the submitter wants it to show up in. A simple Pogo reviewer that doesn't care about wayfarer wouldn't have the option to do this, but a dedicated player who puts time into joining the community would

    I know that any changes would and could be abused. But players who choose to abuse the system will do it anyways. I am putting ideas on how to reduce the amount of innocent duplications and making the players from Pogo have a better submitting experience without having to use third party apps (Ingress or Pogomap and such) or websites like wayfarer to find why they are wasting everyone's time, including their own

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭

    It would be different types of frustration. @Oakes1923-PGO mentioned earlier about how it can have a psychological element.

    The regular review time (6 mo or however many) frustration would come from your nomination being too close to an existing wayspot. I've seen multiple people go through this kind, and it often ends up being frustration that reviewers might have moved it, that there's an existing wayspot already too close to the submission, or also at Niantic for having this system in the first place. Though it often ends with reviewers resolving to do better next time.

    With auto-rejection, you would have submitters frustrated with their nominations being rejection without it having (from what their view would be) a fair chance. If it was auto-rejected right away, they would be frustrated Niantic let them waste a nomination when Niantic knew it would have been rejected anyways. This leads to a lot of the frustration aimed straight at Niantic for not giving their submission a 'fair chance', for having something labelled as a 'rejection' when it is a good candidate (which could also lead to confusion about what is a good candidate or not for less savvy submitters) and for not warning them before they submitted a nomination (especially if it was a hidden POI in PoGO) which could turn players off from submitting and reviewing all together if they think Niantic will just let players waste their time like that.

    In addition, a player who had their nomination auto-rejected by the system might try to get around it by re-submitting it in a less accurate location to try and get it to human reviewers, who then have to waste their time trying to verify a location. Where-as having it approved by human reviewers who can verify location easily, but fail to come online due to proximity, would lead to more submitters taking an "dang, oh well I tried" attitude, while being happier with other submissions made at the time being approved.

    There's also something to be said about letting reviewers review these too-close submissions to gather data about POI that Niantic might not bring online but still use for their own internal purposes, but that's getting way too speculative on what is already a pretty speculative post on my end.

  • WheelTrekker-INGWheelTrekker-ING Posts: 3,387 ✭✭✭✭✭

    I don't think that it's fair that people that have been nominating for years and are fully aware of all the details and gotchas in this process try to state if one situation is more or less frustrating for someone that has just arrived as we're long past that memory.

    IMHO, so many people gets frustrated by waiting even days for their nomination to be evaluated, if right from the start that nomination has 0 chances of being added because it's withing 20 meters of an existing one: Why would it be wrong to tell them that right away instead of waiting for days/weeks/months and then: hey we have approved it, what? you don't see it, umm, maybe we haven't added it to anywhere, but we're not going to tell you that because it's top secret.

    People would try to submit in a slightly misplaced location? Are you trying to tell me that the people that are aware of the 20 meters rule always say: ouch, that's a nice PoI and if I could nominate it 5 meters to the north it would be a valid nomination, but no, that's not OK, I MUST nominate exactly in the right place even if Niantic hasn't ever stated anything about 20 meters.

    So nothing would really change except that people would waste less time and nominations that are going to be "rejected" and reviewers won't waste their time with nominations that won't be added to any platform.

    This is one of the many changes that would improve Wayfarer, so there's little hope that Niantic would listen to this request.

  • Ochemist-INGOchemist-ING Posts: 355 ✭✭✭✭✭

    There's also something to be said about letting reviewers review these too-close submissions to gather data about POI that Niantic might not bring online but still use for their own internal purposes, but that's getting way too speculative on what is already a pretty speculative post on my end.

    I think this is absolutely right; Niantic is trying to build a database of PoIs, not portals or pokethings or inns, per se. It is in their best interest to have as many valid locations as possible, even if they aren't all being currently used. I assume this is why submissions that get Ingress proximity "rejections" are noted simply as being accepted in Wayfarer. They presumably want the flexibility to use anything in a future game, and in fact, when HPWU came out there were reports of live wayspots (inns or the like) that had previously gotten Ingress proximity rejections.

    For this reason alone, I don't see this suggestion being adopted, as much as it might make sense on some levels.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    Do you think people would be less bothered by the rapid "too close" rejection if they got their submission and maybe upgrade back in that case? That would make sense if the submission never had to go to human reviewers.

  • HaramDingo-INGHaramDingo-ING Posts: 1,725 ✭✭✭✭✭

    Not all things within 20m of existing wayspots are accurate. Like you've said, most are already existing wayspots. The 20m rule is primarily an unspoken, but widely known rule, but having this disclosed very early in the process, and then seeing players exclaim when they check on Intel "why are these wayspots so close together!!" will cause further disruption and cast more doubt on the process.

    Niantics does acknowledge that having the email say that it's eligible, but disclosing absolutely nothing about its location other than just a blanket text is a pain point, but those are things that they may be working on.

    There may be so many other things that are already within 20m of existing things, but not being able to capture them in data because they are too close and omitting them from their database just means lost POIs. So the system is not perfect right now, but it's just ok. But there are many nominations that are just nominated too close to existing POIs, but it is very erroneously placed and can comfortably be moved to its more accurate spot and go live.

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭

    If the submitter were credited their nomination back in this case, I imagine they wouldn't be upset about it. It could still lead to the issue of the submitter trying again in less accurate locations or not being able to see dupes in all games, but that's really outside the scope of your suggested improvement and something Niantic really needs to figure out how to handle in a different way.

  • LukeAllStars-INGLukeAllStars-ING Posts: 4,625 Ambassador

    My biggest fear would be the increasing number of abusively placed pins that lead to more location edits also starting to add up.

  • sogNinjaman-INGsogNinjaman-ING Posts: 3,313 ✭✭✭✭✭

    Perhaps just removing the "20 second cooldown limit" for experinced reviewers in "good standing" might have the same effect.

Sign In or Register to comment.