Cooldown Gedankenexperiment

Lechu1730-PGOLechu1730-PGO Posts: 537 ✭✭✭✭

Ok, let's do a gedankenexperiment to see if we can understand how cooldowns may be working.

We postulate a finite number of frictionless spherical wayfarers, reviewing in the vacuum. They may come in one of two types: the dutiful reviewers, who do enough reviews to upgrade each nomination they make, and the occasional reviewers, which only do enough reviews to upgrade one in ten of their nominations. In all cases they have a 50% agreement rate, so they need 200 reviews to get an upgrade. Each reviewer nominates 5 wayspots per month.

Now you have three sources of nominations: dutiful reviewers nominate 10% of the total, occasional reviewers nominate another 10% and non-reviewers nominate the remaining 80%. Nominations also have two types: 1*, which need 10 reviews to be rejected and 5*, which need 20 reviews to be approved. 60% of the nominations are 1* and 40% are 5*.

So, let's say that in Wakanda there are 1.000 nominations a month. 20 dutiful reviewers submit 100, 20 occasional reviewers submit 100 and the rest are submitted by non-reviewers. Your dutiful reviewers would do 20.000 reviews and the occasional reviewers another 2.000. But you only need 6.000 reviews to reject the 600 1* nominations and 8.000 to approved the 5* ones. So, at some point dutiful reviewers run out of nominations to evaluate, so they get the "thanks for your effort" note

Hey, you would say, but in your example there shouldn't be a backlog of months of PoIs in queue. So let's consider Sokovia.

Unlike Wakanda, Sokovia is full of level 38 PoGo players who only submit pictures of coal lumps, so now 75% of the nominations are rejected. This has driven dutiful reviewers into despair and half of them became occasional reviewers. So in Sokovia you have 1.000 nominations a month, 10 dutiful reviewers submit 50, 30 occasional reviewers 150, and non-reviewers and L38 PoGos 800 nominations more

So in Sokovia you need the 5.000 reviews to process the 5* nominations and you need an additional 7.500 reviews to process the 1* ones. In theory you have 13.000 available so there should be no problems, but because half of the dutiful reviewers were lost, you only have 10 reviewers that go through the entire pile. That's enough to process all 1* nominations but only half of what's needed for approving 5* ones, so they start to spend more time in voting, waiting for the occasional reviewers. Once enough nominations are stuck "in voting", lets say 50, no more nominations enter that state. The queue starts to build up and at the same time dutiful reviewers start running out of new nominations to evaluate. So, once again, they start receiving the "thanks for your effort" note.

So, as you can see, it's perfectly possible to have both a backlog building up and reviewers running out of nominations to evaluate. The only thing needed is for there to be less active reviewers in an area than needed to reach an agreement on the nominations.


 

Comments

  • Hosette-INGHosette-ING Posts: 2,515 ✭✭✭✭✭

    That is an interesting perspective. I would normally work through the math to verify it (because I'm that kind of nerd) but I haven't this time so I'm trusting that your numbers are correct.

    I see where you are coming from and it makes sense, though my intuition tells me that the root cause might be slightly different. My working hypothesis is that submissions where there is strong agreement at one of the poles, either 1* or 5*, resolve more quickly than those where the submission is middle-of-the-road with reviewers, or where there is strong disagreement.

    That doesn't undermine your fundamental hypothesis, though, which is that different areas can see radically different queue behavior due to variations in the player and submitter base. That is certainly true.

  • Lechu1730-PGOLechu1730-PGO Posts: 537 ✭✭✭✭

    I built a small spreadsheet and ran several iterations of the numbers to create those two examples so it would be deeply embarrassing if I had made a mistake.

    My view on evaluations actually matches yours and initially I rated evaluations as 1* and not 1*, but changed that to 5* for clarity. I do believe however that's easier to extract a strong agreement for a rejection than for an approval because of how the evaluation process is designed so on average a rejection would need less reviewers than an approval.

  • Hosette-INGHosette-ING Posts: 2,515 ✭✭✭✭✭

    @Lechu1730-PGO You are my kind of nerd!

    Here's my counter-question: How would you set out to falsify your hypothesis? I can think of a couple of strategies.

  • Lechu1730-PGOLechu1730-PGO Posts: 537 ✭✭✭✭

    Which hypothesis? That rejections need on average less reviewers than approvals?

    If you're in an area that seems to be short on reviewers, don't review for a couple of weeks, do a review session and register your reject to approve ratio. Stop reviewing and track how your rejections and approval agreements change. You should see initially more rejection agreements than your ratio should predict.

    The other way could be called the "brute force" approach and involves several people working together, but I don't want to give anyone ideas that could be used to abuse the system.

  • Hosette-INGHosette-ING Posts: 2,515 ✭✭✭✭✭

    @Lechu1730-PGO I was thinking of a coordinated approach. Work with a few trusted reviewers in the same area as your current area of play. Each person keeps a history of their recent reviews for a period of time. When one person is told that there is nothing to review the other people look to see what is currently available for them to review, and the person who has nothing checks to see if those things are in their recent reviews. If not then there are things in the queue that are potential candidates for that person but they aren't getting shown them.

    It seems to me that someone whos current play area, home location, and bonus location were in significantly different areas should rarely if ever hit the end of the queue. My current setup is that my current play area is a major metropolitan area in California, my bonus location is the largest city in the UK, and my home location is not set. Anecdotally, I get the occasional short-term block but never a long-term one.

  • Lechu1730-PGOLechu1730-PGO Posts: 537 ✭✭✭✭

    @Hosette-ING Your test would probably reveal whether reviewers get the review the entire queue or nominations are broken into different reviewing "groups". I'd say the latter is a sensible approach in order to reduce the impact of "reviewing cabals". However, it may be that reviewers are merely delayed to see part of the nominations but will get them in their queue if the nomination doesn't reach agreement after a certain time.

    I live in a large metropolitan area, 3 millon in the city proper (Buenos Aires) and 10+ million in the suburbs. My bonus location is set in a seaside town (I'm there now) but there's s city of nearly 600k (Mar del Plata) along the coast to the north and another one of 100k (Necochea) to the south. An odd thing I've noticed is that while Buenos Aires is a giant collection of dead cells and nothing gets out of the queue unless upgraded, I rarely get nominations from the city proper or the immediate suburbs. Most of them are from the outskirts of the area. I'm not sure if it's due to abundance of reviewers or lack of nominations. The same happens with Mar del Plata. I've never received a nomination from the central area of the city, only from the outskirts.

  • Hosette-INGHosette-ING Posts: 2,515 ✭✭✭✭✭

    @Lechu1730-PGO Look up posts on @Raachermannl-ING about dead cells if you haven't seen them. They provide a lot of insight into why some cells have very different behavior than others.

  • Lechu1730-PGOLechu1730-PGO Posts: 537 ✭✭✭✭

    Yes, I've read those posts and I understand the mechanic but I would have expected to see upgraded nominations more frequently than I do.

Sign In or Register to comment.