Dear Niantic: You're using valuable compute resources extremely inefficiently

Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

(The first bit may seem irrelevant but stay with me for a moment; I promise this is relevant. If you're familiar with the way large-scale systems work you can skip to the paragraph that starts with "Sooo..."

This got long, so if you just want the highlights reel scroll down to the bullet points.)

My job is involves dealing with capacity/throughput for large-scale distributed systems for a well-known tech company. The cost of compute hardware for a very large website adds up quickly, so my team focuses on efficient use of hardware. We want to serve our customer requests quickly, but to do so without wasting a ton of money on servers... but we need to have enough to handle our traffic. When a large system doesn't have enough servers to handle the traffic that's sent to it one of the first symptoms is that customer requests get slower. A system that normally averages 20 milliseconds per request will start taking 25 ms, then 30, then 50, then 100.. and at some point it just can't handle the traffic and it either builds up a large backlog or it starts dropping requests. The time it takes to process a transaction is called latency, and a system that's slowing down is said to have high latency. A system that doesn't have enough compute resources to handle its workload is described as underscaled.

To make up a simple example, let's say that one computer has the capacity to handle 1000 milliseconds worth of traffic at a time. If each transaction takes 10 milliseconds then it can handle 100 at a time. If each one takes 100 milliseconds it can only handle 10 at a time. If one transaction takes 1000 milliseconds it can only handle one at a time. (Systems folks... yes, I'm oversimplifying. Shush.) If you don't have enough server capacity to handle all of your traffic you either need to make your requests faster, or you need more servers.

Soooo... back to Wayfarer. What are the main compute resources for Wayfarer? Grab a mirror-- it's you. And me. And everyone else who reviews. Yes, there are computers handing us stuff to review, but the main work of reviewing is done by players of Niantic's games. I would argue that reviewers aren't just average players-- in many ways they're your most valuable players because they're willing to invest time not just in playing games but also in doing the "free labor" of building your database of curated real-world locations. In the system called Wayfarer, reviewers are the computing resources.

Is Wayfarer handling its traffic effectively right now? All signs are that it is not. While it used to take a few days to process new submissions we are now seeing a steady stream of complaints about things being in queue or in voting for weeks or months. Today two of my five reviews were submitted sometime before September 2019. Players obviously don't have Niantic's internal data to examine, but we don't need it... it's clear from player reports that there's a backlog building up. Niantic, your compute resources are underscaled for the workload.

Let's take detour to examine history, shall we? When Ingress launched, Niantic was using in-house reviewers (compute resources) for all of the reviews. In 2015 the latency for portal reviews had climbed from a couple of days through a few weeks to well over a year, and the number of submissions that made it through the system had slowed to a trickle. In mid-2015 Niantic shut down portal submissions entirely, or in computing terms it dropped 100% of requests.[1] They launched a new system called Operational Portal Recon in mid-2016 that used players as computing resources rather than in-house reviewers, and the new computing resources started processing the backlog. The new compute platform was running so efficiently that they reopened submissions in late 2017. Niantic has already gone through one cycle of being underscaled to the point where they were dropping 100% of requests.

What we have right now is a classic system that doesn't have enough capacity to handle its traffic, latency is increasing, and a backlog is accumulating. As I mentioned above, the way to handle this is either to add more computing resources (expensive) or make your compute resources more efficient. It's clear to me that Niantic is using its compute resources inefficiently. Players seem to be expected to spend some minimum amount of time per review, regardless of the actual work required. There seems to be a hair-trigger response to players who complete a very short sequence of reviews and average less than whatever time Niantic expects for that number, even when the actual workload presented to them requires much less than Niantic's minimum time. My experience is that a high percentage of the reviews given to me can be handled in just a few seconds. "This thing called Foo Church is right on top of an existing wayspot called Foo Church, and the photos are nearly identical? Duplicate, next." If the queue has a high percentage of quick-review garbage in it then the odds of a player getting several of these in sequence is pretty high, and when they do they get a four-day cooldown... which wastes even more compute resources. Niantic, you have large systemic inefficiency in the use of your reviewers, and you are wasting a lot of your computing resources.

I would argue that the situation right now is even worse than that. Some of the computing resources are overheating and burning out, in this case from the frustration of dealing with a poor-quality workload and then being penalized for handling that workload quickly and effectively. The same workload with fewer resources means that the problem is going to get worse, probably causing more compute resources to burn out, and possibly entering a d-e-a-t-h spiral. Niantic, you seem to be losing compute capacity because of inefficiencies in the system.

There's one other thing going on here, something that I haven't talked about, and that is wasting system resources with unnecessary work. Right now a large percentage of the queue seems to be a low-value workload-- obvious duplicates, things that are clearly ineligible, multiple repeats of the same poor-quality candidates, things that couldn't possibly go live in any game because they're too close to existing wayspots. I've written about this before, and suggested some things that Niantic could do (show players current criteria while they submit so they add less garbage to the queue, automatically reject anything that's too close to go live or at least deprioritize it) and there are others as well. Niantic, you could reduce workload on your compute resources by filtering out low-value requests.

I've seen things suggesting that Niantic is actively trying to add compute capacity (recruit new reviewers), but it doesn't seem to have made much difference from my observations.

If my assessments are correct the situation right now is:

  • There isn't enough reviewer time to keep up with new submissions
  • The latency of the review queue is increasing
  • The workload coming into the queue has a high percentage of low-value requests
  • A great deal of compute capacity is wasted by adding unnecessary waits
  • Compute resources are burning out, further reducing system capacity

I can hear what you're thinking. "OK, Ms. Nerdy McNerdpants Hosette, what would you do about it if you were in charge?" Why thank you for asking! Here's what I would do:

  • Guide submitters through the process so that they understand current criteria (not what they learned three years ago), and ask them a few key questions to weed out things that are automatically disqualified, e.g. "Is this on K-12 school property?"
  • Programmatically filter out or deprioritize low-value requests based on the lowest common denominator density rule in effect (currently that's a minimum of 20 meters between wayspots.)
  • Find a better way to filter out bot reviewers and negligent/malicious human reviewers so you can use reviewer capacity more efficiently and not make them insert wait loops, and definitely stop taking them out of service for four days at a time.
  • Think about ways to reduce repetitious work, i.e. resubmission of the same thing. This probably involves some combination of submitter education, reviewer education, and a backoff mechanism for submissions repeated more than twice.

OK, that's long-winded, but hey... I was planning to review 50-100 things before I went to bed tonight but got a four-day cooldown after my first five reviews because four of them were so blatantly junk that I could dispense with them in a few seconds each. Since Niantic rejected my offer of investing my time in reviews I decided to spend it on attempting to improve the system as a whole. This is my second four-day cooldown in the last two weeks-- the first was right at the end of the India Challenge, and I think it coincided with having my bonus location cleared.

After over 17K reviews this particular computing resource is on the verge of burning out... I love reviewing because it lets me see lots of cool and interesting stuff in a lot of different places, and I'm obviously willing to invest a fair bit of time in it. I'm not getting any other rewards from it since I'm perpetually capped out on upgrades and I got the onyx Recon badge almost a year ago. I review for fun, and because I've had a great deal of fun playing Niantic's games and I want to make them better.

On the other hand, it's not fun to play a constant game of tiptoe-through-the-timeouts, and being banished to review jail repeatedly is pretty disheartening.


[1] Well, technically not quite, but let me have artistic license here.

«1

Comments

  • SiIverLyra-PGOSiIverLyra-PGO Posts: 952 ✭✭✭✭✭

    Brilliantly written, well done.

    One of the things that get me the most about this situation is that... none of this is news, really. These are all issues that have been building up and gradually getting worse and worse over time. Issues users have been constantly pointing out/complaining about (although not always as eloquently as in this post).

    I'd personally add that the lack of coherent abuse-policing is another aspect of the system that fails to be efficient enough and adds strain to the system.

    I know there's a roadmap with near-future and distant-future goals and there's definitely effort going into making things better but I just don't think it's enough right now (although I doubt it's the team's fault - I suspect the Wayfarer team is severely understaffed and/or underbudgeted, but this is just speculation).

    Some Niantic higher-ups seem to fail to understand that a lot of the crucial aspects of Wayfarer just aren't working right now.

  • Thor3381-INGThor3381-ING Posts: 220 ✭✭✭

    A very unpopular measure could be: limit the number of possible submissions.

    Everyone across all games gets 1 submission per week, non cumulative.

    As a submission seems to be handled by about 50 reviewers (guessing) you could state that if you do 50 reviews (while being in good or great) you earn an extra nomination which you can stack up.

    The queue will no longer grow, more people will start reviewing.

    The 1 submission a week could also be a flexible number. When the backlog is small enough it can go back up to 2, 3... This could also be linked to the wayfarer medals

    The ultimate goal is to get extra reviewers, the more reviewers, the bigger the review capacity, the smaller the backlog

  • sophielab-INGsophielab-ING Posts: 266 ✭✭✭✭

    Limiting number of nominations I get to one per cycle doesn't go with how I submit and review. I travel to new area after identifying multiple viable candidates. I submitted a bunch at once and was reviewing in spurts as well. I have most of the stuff in voting or in queue around 10km or more from my house.

    I don't know how many people review for fun or to help clear queue vs to upgrade their nomination. My guess is a significant amount of reviews are done solely to get upgrades and if you can only use one per cycle how many reviews are actually going to get done.

  • kholman1-INGkholman1-ING Posts: 52 ✭✭✭
    edited April 2021

    The current amount of nominations per player isn't the issue. It is the lack of understanding or refusing to look up the rules during onboarding with new submitting players. I have heard people say they will just let it go in and see how it fares without even taking into consideration that they may be making things worse overall. I don't even think some people know who is reviewing and believe it is niantic doing everything and not realizing players are helping niantic with the backlog.

  • ArborAndyYT-PGOArborAndyYT-PGO Posts: 87 ✭✭

    👏👊🏿i unliked this so i could like it again

  • Thor3381-INGThor3381-ING Posts: 220 ✭✭✭

    If you review in spurts you will stack up possible nominations you can use later on - 50 reviews done, 1 extra nomination you can stack.

    If you review for upgrades you'll get the extra nominations as well (2 nominations per earned upgrade if you manage to review at 100 PCT correct)

    If you do let's say 2000 reviews in order to get 14 upgrades for your bunch of submissions you'll stack up 20 new nominations.

    It's only the ones that don't do any reviewing that will be slowed down

  • kholman1-INGkholman1-ING Posts: 52 ✭✭✭

    The number of nominations given to players is not the issue. It is lack of understanding the guidelines or the guidelines being ignored. As it is a small percentage of players are submitting waypoints.

  • kholman1-INGkholman1-ING Posts: 52 ✭✭✭

    This is how I feel. There have been times when it did slow down to the point where the system was barely moving. I find it hard to keep interest when I see locals sending in their private residences for a point of interest. I understand parts of my area do not have a lot of valid objects to nominate but some of these are just bad. Nominating plush toys, an **** TV and other numerous examples just make me lose interest in voting.

  • sogNinjaman-INGsogNinjaman-ING Posts: 3,313 ✭✭✭✭✭

    While I appreciate Niantic want to run a fairly "loose" reviewing system, i.e. not be dragged into a system that requires every little decision to be made by Niantic mods themselves, from a review of these forums it should be fairly obvious to see that there are particular types of submissions that frequently cause "problems". Submitters submit them, reviewers review them against very loose and "open to interpretation" criteria and no one ends up happy.

    Simply by providing some more "official" guidance and more "definite" rulings on exactly what makes an eligible submission for things like "trail markers" and "PRP", you could cut down on delays and "loss of reviewer" incentive. For some things, some well defined and tight criteria are needed. It would be fairly simply to provide a couple of pages of guidelines, with photos, of "eligible / ineligible" submissions from problem areas. Taking a leaf out of Hosettes post - Nianic, you don't want to lose free processing power because reviewers run out of interest in reviewing or simply get P/O.

  • NephilaC-PGONephilaC-PGO Posts: 3 ✭✭

    Exelente post, I woud add another waste of resources. I live in a rural area so there are few players here and few wayspots requests, usually I review on average only 10 requests per week, not because a cool down for reviewing too fast, but because there are not many requets around here.

    As I don't live in my hometown, I set up it's location on settings, but even though my hometown being a larger city, I still get very few requests from there. The only time I was able to review a good amount of requests was recently when I traveled to my hometown, I received requests from all over the region including several neighboring cities, I stayed there for a week, reviewing requests as many as possible every day, and still I couldn't finish the queue.

    So I see it as a great loss of resource, since I can’t review more here while there are places with such a long queue that it’s endless. I like to review requests and I want to, but due to in my opinion a poorly designed system, I just can't do my best.

  • sophielab-INGsophielab-ING Posts: 266 ✭✭✭✭

    I've done over 25,000 reviews. I've done more than my fair share of reviewing. Please keep your hands off my allotment.

  • kholman1-INGkholman1-ING Posts: 52 ✭✭✭

    The problem with trail markers is there no cookie cutter examples on what makes a trail marker. There markers with just visuals, others with wording etc. Having a cookie cutter list actually didn't help and trails in places like national parks were getting rejected while a common walking path in a city was approved because the later had trail name.

  • Rodensteiner-PGORodensteiner-PGO Posts: 1,705 ✭✭✭✭✭

    as i said many times before, the current Wayfarer System is broken. It is not working properly.

    Besides that i also told many times before, that the System needs persons that pre-check all the submissions. I think Hosette and myself could pre-check like thousands of submissions a day. In the end, people would surely only get 50-60 % of those submissions to actually REVIEW.

    Those 40% are submissions of things inside a private home (cats, underpants, cure-posters) OR things in the frontyard OR photos taken from afar, photos of street signs, photos with Watermarks.

    These submissions DO NOT need One Minute to review. With a proper batch software you can easily see from the prime photography that these things are not meant to be reviewed.

    Niantic just needs to test this out.

    Also, please for the sake of our sanity, have first in first out. I have submissions that are out there for nearly 1 year.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @Rodensteiner-PGO I understand where you're coming from, but I don't think that task scales to humans. And no, I couldn't pre-check thousands of submissions per day... my attention span for that would be roughly 50 and after a week of doing that I'd look for a less-tedious hobby, like maybe buying bags of rice and sorting the grains by size.

    Computers are pretty good at this, though. They could easily identify watermarks, reused photos (fake submissions, anyone?), duplicates and things blocked by proximity, photos with prominent people, and some other factors. I proposed recently that Niantic should auto-bounce anything that couldn't go live because of proximity, and probably refund the submission/upgrade when that happened. Ideally they'd stop people from submitting anything blocked but it might be less work to bunce it on the back end. That would probably eliminate 5-10% of things that go into the review queue. Deduping is computationally easy, and handing a person a "hopeless" candidate isn't a good use of human compute resources.

  • Ochemist-INGOchemist-ING Posts: 355 ✭✭✭✭✭

    reused photos (fake submissions, anyone?)


    Nothing wrong with reusing a photo when resubmitting something that had been rejected that the submitter feels is indeed valid. I've done that lots of times and am usually successful on the second try.

  • Rodensteiner-PGORodensteiner-PGO Posts: 1,705 ✭✭✭✭✭

    at least they should implement something that filters out those photos with XIAOMI watermarks. we do get them alot.

  • SeaprincessHNB-PGOSeaprincessHNB-PGO Posts: 1,608 Ambassador

    @NianticAaron @NianticCasey-ING @NianticBrian-ING and whoever else needs to be pinged.

    Y'all just got a free system/process assessment from an Industrial Engineer. I would take a look at it very seriously and be thankful you have people THIS committed to the success of your company. You would be fools to ignore what was outlined here.

  • HaramDingo-INGHaramDingo-ING Posts: 1,725 ✭✭✭✭✭

    Niantic absolutely needs to leverage their community. It seems as if they are specifically engineers and game designers but does not really have experience in creating management systems effectively. Sure, Wayfarer is a great tool but it is not optimised nor robust.

    A pre-check system that IS voluntarily done by the community or by the proposed potential Wayfarer advocates could definitely reduce the amount of foul and dirt poor nominations. Things that do mention any of the games in the description, actual home attempts, photos of children, all those sorts of things that everyone has repeatedly clamoured against.

    Although I don't agree to a major reduction of nominations as some people have proposed, the availability of nominations should be scaled by Wayfarer activity, general accolades (such as medals and badges) and not just by experience/action points. But unless the pain points (such as the 4-day cooldown, incessant cooldowns for rejecting numerous couch attempts in a row, and actually punishing real abusers of the system) are addressed, then it is worthless. You do not want to go through another period of time where you disable nominations so that people can work through a backlog, they will not work if they are not nominating anything for themselves.

    As someone that has developed a compliance-based relationship system, not being able to pick out the Wayfinders that are choosing incorrect reject reasons on PURPOSE (because people think the live animal reject reason is so easy to mistap... yeah right) and being able to create general analyses of people's reviewing patterns or the types of wayspots that are being rejected across different areas (i.e. why is the playground acceptance rate in Australia 90% but the acceptance rate in say, Portugal is 20%?). I have absolutely no idea what you do in these so-called months-long "investigations" (although that's probably in a whole different department) but it is NOT right to continue giving the community the short end of the stick who want to help!

    Steadily, and without any sort of change, your most dedicated reviewers and wayfinders will burn out. You (Niantic) could probably just get rid of the forums and it wouldn't change a thing... to wipe and rid yourselves of the still waters.

    If only some other company can actually compete against you and not just be the sole "gamified-POI global database", then you'd have something to be concerned about.

  • SiIverLyra-PGOSiIverLyra-PGO Posts: 952 ✭✭✭✭✭

    Well said.

    I'll add that - at least in my experience - dedicated reviewers and wayfinders are ALREADY burnt out. A lot of passionate users I know - those who care(d) about the guidelines and about creating quality wayspots, thoughtful reviews, and maintaining the local community - have gotten sick of the system ages ago.

    We're all seeing the results of that, with the purposeful incorrect reject reasons you've mentioned being one example. Review quality is decreasing, nomination quality is decreasing.

  • sogNinjaman-INGsogNinjaman-ING Posts: 3,313 ✭✭✭✭✭
    edited April 2021

    I'd say they are already leaveraging their community, but do not support the reviewing community nearly as well as they could do with regards to things like review feedback and criteria / AMA clarifcations. Certainly all the Niantic staff comments on various issues are scattered throughout these forums, despite many requests to collate all those considered by Niantic to still be valid to be added to the Criteria guidelines.

    As for the idea of "pre-screening by selected reviewers", I honestly think that will just lead to more arguments, abuse and complaints in here. Far too many times we see threads drifting off into arguments about Waypoint removal being driven by factional reasons or "for revenge", along with "demands" for reviewers to be "punished" when submissions are correctly rejected. I think any "pre-screening team" will be subject to a lot of abuse by people making unwarranted claims aling the lines of "he only screened out my sub because he plays Ingress and not PoGo" or similar.

    If Niantic want to leaverage more community support, then they need to engage with the community, and not just this small subset called the "Wayfarer Forum Community". Despite some dissention, the overall the advice proffered to people in here seems generally to be very fair, even if some do not like what they are told. However, one thing is clear, a lot of the wider playing community "out there" have never heard of Wayfarer and have only the vaguest notion of the current criteria. Niantic, or those speaking on their behalf on an "official basis" (Community Advocates anyone) need to be spreading the word and showing submitters the "Official Niantic Criteria" via whatever particular flavour of Social Media they use - Facebook / Telegram / Discord / Twitter etc etc, particularly for the commonest "problem" rejections such as PRP or K-12.

  • Mernie9-INGMernie9-ING Posts: 42 ✭✭✭
    edited April 2021

    I have a feeling that NIA could tweak the algorithm used to make decisions and save a lot of time. Mind you, I have no idea what the algorithm is, but perhaps a three strikes policy should be implemented. Like if you get 3 reviewers that say it's on private or school property, game over. Why continue evaluating for extra assurances?

    I'd estimate that at least a quarter of the things I review are rejected clearly no brainers. Private or school property. Neighborhood or housing development signs. Generic businesses. (Another Starbucks?! The laundromat, really?) If those could disappear quickly based on a smaller number of reviewers, then we could evaluate more things.

    And rejections shouldn't require the full 20 seconds. I can't tell you how much time is wasted just waiting for the 20 second clock to end. I could review more if I weren't twiddling my thumbs.

    Another way to increase efficiency is to sweeten the rewards. Right now we have to have 100 successful evaluations in order to get one upgrade. Would I review more if I could get more upgrades? You bet I would.

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭

    I think there should be rewards besides upgrades, so people will be incentivized to review even if they don't have anything to submit. Some way to transfer reviews to in-game rewards. Or some kind of contest on the wayfarer site with a leaderboard similar to how to the one on OPR.

  • Mernie9-INGMernie9-ING Posts: 42 ✭✭✭

    Good point.

    I should add that for some people that reward is the badge credit. Not sure if they want to change the badge formula though.

    I need upgrades to get anything in my area approved. I have to remind myself that there are a lot of areas where your stuff gets evaluated within a week or two without upgrades.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @Mernie9-ING I have no evidence to back this up, but my intuition is that reviews resolve more quickly when reviewers are more consistent rather than when they're all over the map. I would certainly implement it that way.

    Nothing should require any specific length of time. They are using time as a proxy for review accuracy, and it's a poor proxy for that, and not just because some things are trivially easy rejections. I'm reviewing on a desktop computer with a ton of screen real estate, and it takes a lot less time for me to review in that configuration than if I was using a mobile device.

    I get what you're saying about upgrades, but I think that's a double-edged sword. There's probably a tipping point where people's review accuracy suffers because they are trying to rack up upgrades as quickly as possible. We've seen periods in the past where a lot of people just started rejecting everything because it was the fastest way to get points for a badge, for example.

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭
    edited April 2021

    With Plat in pokemon Go being 1500, that's pretty easy to knock out, (and most Ingressers would have Onyx by now if they wanted it) and not everyone is interested in the badge. I get it's something, but it's not super working as a reward anymore.

    And I'm hitting a point where upgrades tend to hinder anything I submit that isn't an excellent 5* candidate. If it's inside a building and I upgrade it, even with a photosphere, there's a good chance it gets rejected. Hotspots are almost impossible too :/

  • Gendgi-PGOGendgi-PGO Posts: 3,534 Ambassador

    I live/play on the edge of 2 scoring Regions with vastly different turnaround time. I carefully manage nominations between the two cells based on what I'll upgrade. It's pretty sad that a nomination in my "home" region takes 18+ month turnaround and if I go 2 miles west it has taken as little as 3 days from submission to approval.

    I'm almost at 25k on my Recon badge. I actually took a few months off of reviewing when they gave double Recon points because I thought that was silly.

    Anywho, even with nominations taking so long I'd occasionally prefer not having Upgrades forced on me, in the event I'm nominating things I'm sensitive about being upgraded.

Sign In or Register to comment.