The absolutely ridiculous math of April 2022 AMA answers #4 and #6
Warning: This post contains a lot of math, and lots of estimations.
In the April 2022 AMA, questions #4 and #6 asked about appeals. Those questions and answers are below:
4) Why aren't appeals being used as a way to identify, evaluate, and educate poor reviewers?
Great minds think alike! We are waiting to get more appeals processed to start identifying major trends and seeing how we can use this data to support the community as a whole. As of today we have 20K appeals processed, and we think we need about 150K appeals processed to kick off the analysis.
6) What percentage of Appeals have been approved and are you tabulating the rejection reasons?
Love the data questions! We’ve processed about 20K of the appeals submitted since Dec 2021 and we are seeing about a 50/50 split across appeals being accepted and rejected. Also, we are definitely tracking the rejection reasons and we still want to process much more for us to start thinking about how to use the trends to inform what educational materials we hope to create (see #4 response above).
The glaring problem with these answers revolves around the amount of time it took Niantic to review the initial 20k nominations, as well as how long it will take them to process the appeal backlog and the 150k appeals to meet their goal.
Niantic announced they were beginning to review appeals on February 8th, 2022 per this post, and the next day (February 9th, 2022) many users including myself began to have their appeals processed. Going by the date the 20k number was originally revealed on twitter (March 30th, 2022), we also know that it took nearly 2 months for Niantic to review these 20k nominations, a rough average of about 10k per month. This means, to reach the goal of 150K, Niantic intends to spend approximately THIRTEEN MORE MONTHS reviewing the remaining 130k appeals to meet their goal before they actually use that data???
This isn't even considering the fact that the only reason we have this many appeals submitted is because Niantic intentionally allowed (or at least ignored) a bug that proliferated for several months that allowed users to appeal one nomination per day instead of once per month. We also know thanks to this tweet that most of those 20k appeals are from mid-December, telling us Niantic still has several months worth of one-per-day appeals to get to. Niantic activated appeals on December 8th, 2021, so lets estimate that the "appeals from "mid-December" is comprised of the middle 12 days of the month, December 8th through December 20th. As people could submit one appeal per day instead of one per month, this means that the volume of appeals that Niantic received in those estimated 12 days is approximately the same volume they would have expected for an entire year if appeals were given one-per-month as was intended. This means that the 20k nominations reviewed are approximately one year's worth of anticipated appeals. Because a year's worth of anticipated appeals is approximately 20k nominations, had the one-per-day bug not occurred, it may have taken SEVEN AND A HALF YEARS for Niantic to even receive 150k appeals to meet that goal if they stayed at a constant rate. Was the plan to wait seven and a half years to actually make use of appeals data? Does Niantic not understand that by the time they even got halfway to that point, that the older data would be depreciated as the climate of wayfarer changes? It just makes no sense.
Niantic disabled appeals on February 15th, 2022, just a week after they began to process appeals. This means that between when appeals were activated (Dec 8th) and paused (Feb 15th), users could have submitted a max of 69 appeals, with a 70th appeal available once they were recently turned reactivated on March 30th 2021. Continuing our assumption from before, if we go by date, 12 days worth of appeals is approximately 17% of the duration appeals were available. This means that if the same rate of appeals was submitted for the entire duration of appeals (admittedly this is unlikely), we could have approximately 10 more months for Niantic to finish reviewing the current backlog. Because it will take approximately 10 more months to finish the current backlog, we must also account for how long it will take to process the additional appeals from those 10 months, and so on, which compounds to be another 2 months, for a total of approximately 12 months to get rid of the appeal backlog.
The point of the above though, is that while it may take approximately 12 more months for Niantic to review the backlog of appeals at the current rate, as we established at the beginning of this post, based on the current rate of reviewing, it will take about 13 months of reviewing at the current rate for Niantic to meet their goal of 150k appeals processed. While they may feasibly meet the goal a year from now if my backlog estimations were low (although that is still far to long to wait to take action), if my estimations were accurate or high, Niantic will run out of appeals to review before hitting their goal. At that point, the rate of appeals processed with drastically drop in line with how often users can submit appeals. If we again use our estimation that 12 days of appeals produced 20k appeals, that means that approximately 1700 appeals were submitted per day in mid-December, and if the rate stats the same with a one-per-month allocation, Niantic should expect approximately 1700 appeals per month going forward. If my estimations were in fact correct or high, a difference as small as 10k nominations to reach the 150k goal could result in 6-12 months to finish off at a rate of 1700 appeals being submitted per month.
Additionally, going by the exact dates that appeals began to be processed (Feb 9th) and the 20k number was revealed on Twitter (March 30th) we get a total of 50 days difference. This means approximately 400 appeals are being answered every day on average. In practice though it seems there are days when no appeals are processed at all, making the number processed per day a bit higher than that on days that appeals are actually processed. Lets say the appeal reviewers work 5 days a week and accordingly process 560 appeals per day on average. Personally, I can do a few hundred reviews myself in a matter of a few hours (and frequently stream myself doing so on the Wayfarer Discussion Discord), and frankly I would expect a paid and trained Niantic employee to be similarly efficient and produce far more over a full workday. A pair of full time employees should be easily capable of pushing those numbers, much less potentially more. If two people have 8 hours to do 560 appeals, that means they're each doing about 280 per day and they have about 1.7 minutes per appeal. Considering that appeals seem to be outsourced to cheap labor (Appeals only tend to get processed during the hours of the Indian work-day), there really is no excuse for how slowly the process is moving along and how few resources are seemingly being put into it.
Overall, the AMA answers and specifically the goal of reaching 150k nominations processed before utilizing appeal data is unacceptable. Not only is that goal number of 150k absurdly high, its fairly clear Niantic did not have a feasible plan to use data gained from appeals in a meaningful way when they initiated the appeals process, nor have they managed to form one now either. This is especially true given that one-per-day appeals were not intended. The plan for using appeal data absolutely needs to change. Unfortunately, answer #6 showed that when they do reach their data goal, that their only plan is to create new educational tools. This could mean a lot of things, but unfortunately I have a feeling we will just end up with something as useless and horrible as this infamous infographic. This is so tone deaf, mostly because the majority of items seeing success in appeals are things that reviewers should have already known by reading the criteria and passing the test. In many cases, the tools to know these answers already exist, but reviewers are just not reading them or following them. Creating new materials will not magically solve the problems of people not reading or following the rules in the first place. The issue is a failure to keep reviewers educated with existing/updated criteria, a failure to ensure that reviewers (and abusers) are following that criteria, a failure to teach more about what rejection reasons are appropriate for, and a need for better clarification or improvements on problematic rejection reasons. And even though those things are very well known and longstanding issues, it seems like we may have to wait a year or more for you to even look at data to try to fix those issues. I understand that data can lead to better solutions, but there is plenty of existing data and history to go off of now.
Before concluding I just want to point out some potential mathematical caveats that I did consider so we can try to avoid people arguing about them in the comments. While these things are all worth considering and likely effect the math there was not a good or easy way to estimate them, and most of them likely balance each other out anyways:
- The 20k appeals processed number is likely an estimate itself. It could be significantly more or less than 20k, which would mess with almost all of the math. (Personally I would guess it was low and that Niantic rounded up to 20k, which is why I chose to stick to a rough 10k per month processed estimate for that example, as I felt it would be more accurate over time).
- The 20k appeals processed number was stated to be "mostly mid December 2021", implying it includes appeals from outside of mid-December. Personally I would interpret "mid-December" to be the middle third (10 days) of December. By utilizing an estimate of 12 days worth of appeals in my math, I tried to account for that discrepancy.
- The rate at which appeals were submitted when the feature launched likely started high and got lower as time went on because the feature was new and people had a backlog of rejections to appeal.
- Conversely, some people may have appealed more later on than at the launch of the feature as with time the feature itself became more widely known and it became common knowledge that you could submit appeals once per day.
- The frequency at which a user used their once-per-day appeal is likely lower than what how often they will now use their once-per-month appeals, as it was easy to forget to appeal something every day and easier to run out of rejections to appeal.
- Any estimation involving Niantic reaching the end of the backlog or their 150k goal could easily be thrown off by them allowing more frequent appeal accruals, which is something they said they would consider as time goes on.
- Employees reviewing appeals were likely new when this processed started and may have become more efficient over time, or as they learned more nuances of reviewing it could be taking them longer as they become more thorough.
- On average, reviewing an appeal likely takes longer and requires more attention to detail than a regular user review, even when considering the likely possibility that some people appealed their blatantly ineligible coal.
Thanks for reading, sorry in advance if my math sucked.