Reviewer quality must be improved!

JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭
edited July 2021 in General Discussion

Hello everyone!

And especially hello @NianticGiffard @NianticCasey-ING @NianticAtlas @NianticAaron @NianticAndres !

For a a while now, me and a lot of locals I know, are having more and more problems with the whole reviewing process and the reviewers' quality. Most of these problems are caused by bad / inexperienced reviewers, but this is always connected to a suboptimal system as well that is not transparent, informative or strict enough. These are my main problems: 

  • reviewers, who clearly have no clue about certain types of wayspots and decline them again and again, although they are mentioned explicitly in the criteria as good waypoints. Some examples: popular restaurants, trail markers, maps, sport fields of a bit more niche sports and a lot more. 
  • reviewers, who think, you must have a star photographer's equipment to make a wayspot photo because they don't have any idea of the photo criteria. 
  • reviewers, who still cannot differentiate between single and multiple family residences
  • reviewers, who give 1* ratings in subcategories because they do not have a clue of the consequences for the rating (especially the 'visually unique' is often completely misinterpreted) 
  • reviewers, who do not make an effort to search for information / click on links etc. because it is more lucrative to decline a wayspot immediately and completely


And the thing is, it would not be that difficult to do something against this! There are a lot of possibilities and adjustments to improve the situation. Here are some of these: 

  • Why having honey pots but do not use them effectively? Why are there no honey pots for these candidates that are always refused mistakenly followed by a pop-up window that informs the user of a good review, if he accepts it and a warning + rating drop, if he refuses it. I don't understand this whole "drop the rating secretly in the background" - mechanism because it does not indicate the mistake at all and the reviewer will continue doing the same mistakes
  • The same thing can be considered for apparently wrong picked rejection reasons and in both cases the users should be corrected but also punished accordingly
  • It must be less lucrative to decline candidates, which is the most profitable way to review right now because it is way faster than distributing the subcategory gradations -> One possibility could be to let the reviewer decline a waypoint completely by setting all of the subcategory ratings to 1 star and moving the first category to the bottom. So - there is no big time difference between rejection and acceptance


And I am not just speaking for myself, but for every good reviewer / submitter. I see discussions about these problems and potential fixes over and over in local communities, but I never saw some officials reacting to these. I also hear a lot about wayfinders completely quitting wayfarer for exactly these problems.

So I hope that these complains are taken seriously and perhaps Niantic can fix some of these issues. Its not that much work.

Tagged:
«1

Comments

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    Sorry, but this should not be about personal rejections that I got. This is a collection of problems that I, but also a lot of other reviewers recognized and therefore I dont want to discuss single cases. This post is more general and should discuss about things that need to be changed to improve the situation for everyone.

    That these issues exist, should be clear. I can underline some of them, but like I said, that is not the main point here.

  • RyuuVanDraco-PGORyuuVanDraco-PGO Posts: 160 ✭✭✭
    edited July 2021

    Yeah, germany somewhat needs honeypots. Especially regarding trail markers, restaurants, post offices.

    Got a restaurant with a successfull history of at lest 31 years, gets rejected every time. Every trail marker got rejected so far. Every Post office. Also artworks of multiple family residents needed more than one try.

    Even pointing out that something is mentioned in the criteria and as much proof as possible leads to nothing but wasted effort. I got proof for these 31 years and every official post office got an entry on the german post page. Ok, mobile users got a huge issue with links as they aren't clickable and around 31% of our community uses mobiles to review.

    Some people admit they reject stuff like that "because it's not interesting" or "it's not like in america" or "it doesn't really connect people". Rather make their own rules than obeying official given ones or adjust to the areas equivalent if it's good enough. It's really discouraging and disappointing being a constant victim of such irresponsible people.

  • tp235-INGtp235-ING Posts: 1,383 ✭✭✭✭✭

    I'm sorry, but that's not how we discuss things.

    There is room in this community, but I think the best way is to check with the high level nominators, who can approve most of the nominations, to see if there are any flaws or errors in your nomination.

    The current vetting process, for better or worse, is decided by majority vote, not by one person's good or bad.

  • tp235-INGtp235-ING Posts: 1,383 ✭✭✭✭✭

    I haven't seen your individual recommendations, so I can only speak in general terms, but living in another country, I feel that the fact that Street View is barely functional in Germany makes recommendations more difficult.

    In the current system, the number of candidates that can be approved based on satellite photos alone is very limited. From the air, you can't put a definite star on a restaurant or a post office. For artwork, 360° photos are a must. In addition, you have to refer to maps from other services to make sure that Google Maps is correct.

  • Kawhinot-INGKawhinot-ING Posts: 189 ✭✭✭

    That's not how we discuss things? LOL

  • tp235-INGtp235-ING Posts: 1,383 ✭✭✭✭✭

    I suggest that you submit your qualified but rejected candidate to Nomination Improvement to see if it is really worth it.

    @JoCze-PGO

  • BleedBoss-PGOBleedBoss-PGO Posts: 269 ✭✭✭

    As i've stated multiple times before, this isn't exclusively a reviewer issue, but also a Niantic issue. They are the ones who entice you with upgrades, and upgrades come from agreements. The general view for Niantic is : If the MAJORITY of the community thinks it's good, it will be accepted. This is a clever way to wash hands of all responsibility, however, the system was built by their team, not the community.


    When rejections have more weight than acceptance, it's no longer about the majority. And we've seen countless posts where people are getting rejection criteria that doesn't match the submission (Explicit location for a Museum? Count me in!). And why does this happen? Because Niantic ALLOWS it to happen. There is literally no drawback to mass rejecting. On the contrary, you will most likely get agreements faster, which is the Carrot on a Stick put in place by the company. Even people that submit continuous fake portals, that tamper with photospheres and whatnot, are still not being properly dealt with.


    Take an example from the real world: If law enforcement didn't exist, the crime rate would skyrocket, because if you have a lot to win and pretty much nothing to lose, the human brain will default towards personal gain rather than moral standards. Same applies here: If there is no enforcement in place to stop blanket rejections and abuse, it's going to continue to happen, and someday, maybe to an extent where using WF just becomes unbearable.


    @NianticAtlas and the crew, might wanna take a deep look into this. And i don't mean a 24H-day-to-night hotfix. Take your time to analyze the reports you have been given, and improve based on that. You could even (and i know this might come as a shock to your team at the office) TALK AND DISCUSS the best course of action with your community, i'm sure many people have very good advice to give you.

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    No, again: All the mentioned problems are self-explanatory and well known in the wayfarer community. Single cases do not have enough weight either because there will always be contrarian opinions about little details and I dont want this thread to become one of these where single cases are talked to deth.

    Or do you disagree in one of the main problems?

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    Yes, that is exactly what I think too. Like I wrote in my statement, the problems are always conntected to Niantic and the system, too. For the fast clicker problem, have a look at my third solution. That could address the issue by making neither acceptance nor rejection more lucrative

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @BleedBoss-PGO writes:

    When rejections have more weight than acceptance

    Do they? Do you have evidence that it takes less than a majority of reviewers to reject something?

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    I think what he means is that complete rejections are much more comfortable for the user than trying to make an adequate review. That is a fact and that should be changed.

    But lets see the case of a wayspot, which has 200 votings in total. 70 hard rejections and 130 ratings, but with lots of fluctuations (2*-5*). In this collection, there are definitely also a lot of reviewers, who rate certain subcategories with 1 or 2 stars and have no clue that this is kind of a rejection as well. Sure I cannot prove that (how could someone, who is not an intern?), but I am pretty sure that such a candidate will be rejected due to averaging reasons only because way too many people reject things hard although that wouldnt be necessary and another group of people dont have necessary knowledge of the system.

    And the system rewards these reviewers because it is the much faster reviewing method.

  • BleedBoss-PGOBleedBoss-PGO Posts: 269 ✭✭✭


    Ask @NianticAtlas and the crew directly, they will tell you (if they have any knowledge of the product). There is a reason why there's so many topics of rejections.

  • RyuuVanDraco-PGORyuuVanDraco-PGO Posts: 160 ✭✭✭
    edited July 2021


    Missing Streetview isn't the core of all these problems. People often post enough proof and often the additional photo is enough proof (as in my cases). Else they link maps, websites, etc. Every Post office has an official entry on the companys webpage whioch teh Google entry often links to. Also people submitting hiking paths often do create a photospeheres or add a geotagged photo. As for restaurants, well, you probably have to trust the submitter and may have a look at their ratings on Google. If it got 4-5 stars, you can be almost sure it's not bad.

    It's indeed the reviewers which are the problem and simply can't accept some things as waypoints or are too lazy to take a closer look.

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    I know, what you mean. There are a lot of bad submissions too. But none of the mentioned problems I pointed out - are connected to single cases. So why does everyone want to speak about these?

    Which of the stated problems isn't a well known problem that affects most of us?

    Which of my suggestions are not easily realizable and a huge improvement?

    So what are we talking about here? I really dont understand this resistance, when I just want to make the system better.

  • Raachermannl-INGRaachermannl-ING Posts: 1,070 ✭✭✭✭✭
    edited July 2021

    Not necessary. This guy knows, what he is doing 😋

    There are no specific examples given, because evry thread, which has them, is then stalled by very special persons, who then simply try to run evrything down as if the criteria wouldnt apply to them.

    Post edited by NianticGray on
  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @JoCze-PGO Which of your suggestions is not a huge improvement?

    One possibility could be to let the reviewer decline a waypoint completely by setting all of the subcategory ratings to 1 star and moving the first category to the bottom. So - there is no big time difference between rejection and acceptance

    If I look at "JC Elementary School" as a title and the photo is clearly of an elementary school with kids walking into it then I can legitimately stop right there and 1* the candidate for being on K-12 property. I've already had this conversation with BochumJules elsewhere, but asking me to rate every field for something that is clearly on Niantic's list of things that must be rejected is a waste of my time. Expecting that I will have to rate every single field 1* is either taking away the 1* rejection for meeting one of Niantic's DISqualifying criteria or requiring me to lie about things. "JC Elementary School" is probably a 100% title, the location has safe pedestrian access, and the pin is probably in the right place. Do I have to lie and rate those things 1* as well in order to ensure that the ineligible candidate is rejected? In your proposal where I have to rate every field 1* in order to reject it how should I fill in each field for "JC Elementary School"?

    The secondary effect of making reviewers do unnecessary busywork is going to be that the queue moves more slowly because reviewers have to waste time. And also because some of them will quit reviewing out of frustration.

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    Yes, it will last a bit longer to make a rejection, but most of the good reviewers already take that time (20 seconds) until they make a complete rejection because otherwise you can run into cooldowns. Only the fast clickers, who want more and more effeciency are slowed down. I could definitely live with that!

    I also contradict the theory that this adjustment will lead to longer queues because that adjustment can lead to more justice for the mentioned "doubtful cases" (which are not doubtful at all, if the reviewers would know the rules), since fast rejecting would not be that much more profitable and in that manner, it will even make the queues smaller because these candidates don't have to be submitted over and over again ... If I have to submit a 100% trail marker by an average of 5-10 times until it is accepted, THAT inflates the queue!

    I can understand the other argument having to "ly" about subcategories, but I am pretty sure that complete rejections are treated exactly like that (1* in all categories and then based on all reviews an average for all categories is formed).

    Furthermore I would not have a problem at all, if you can rate specific subcategories with more than one star and still reject the complete submission at the end. The main point here is just to make the reviewer have to pick all of the categories first.

  • Jtronmoore-PGOJtronmoore-PGO Posts: 1,581 ✭✭✭✭✭

    I’m not giving any resistance tbh. I’m just stating the fact that the majority of people who complain about wrong rejection reasons fail to realize that there nominations are fairly weak or are not eligible to begin with.

    I acknowledge the system needs improvement because there are some people who submit who do unfortunately get rejected for random reasons for something that is obviously eligible. I was merely playing devils advocate in the situation.

    In other threads I have suggested different ways niantic could use the appeal process that would be beneficial to most players. Hopefully someone that runs it can see some suggestions and use the different ideas to help make an effortless and stress free system for users to appeal incorrect rejections :)

  • Jtronmoore-PGOJtronmoore-PGO Posts: 1,581 ✭✭✭✭✭

    Thanks for the shade 😂 but all i try to do is help on nominations or point out obvious flaws.

  • Raachermannl-INGRaachermannl-ING Posts: 1,070 ✭✭✭✭✭
    edited July 2021

    Then do it in the nomination improvement section, where you are asked to do this. So if someone, who is obviously not a newbie to the forum (simply look at the post counter), posts threads with a general problem in General Section you are not asked to do this.

    If you do it nethertheless again and again, then you are undermining the work of the people by watering the topics down, so that Niantic has no pressure to respond or react in some way. That happened to sooooo many threads here, that I'm sick of that. Reading again and again stuff like "This is a bad photo, because I don't like the weather conditions or the perspective" .... this next-level-nitpicking and the crazy search for the fly in the ointment can be constructive in the nomination improvement section, but never ever in a thread like this.

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    Okay, then we agree on the main problem much more than I thought. I also like the idea of appeals for wrong rejections but I did not want to include these improvements that cause much more work for Niantic. I tried to keep the necessary changes as little as possible but still causing big advantages. So, for my suggestions ONE capable programmer could do these changes within a day! I hope that convinces someone responsible.

    And to make that clear. I am not speaking for people who clearly have certain major problems in their submission (regarding criteria, watermarks, bad supporting info/photo and so on).

    Some details are always discussable, but in most cases it should be pretty irrelevant, if a submission has a fantastic photo optimized for all games or if it is just a normal one, where the POI is somehow visible. Or if the description is 500 characters long filled with different interesting facts or only a small but fitting sentence. Or if there are some (partial) sentences in the additional information that someone else would not write that way.

    So, yes, it is always nice to have some feedback and optimizations and I would also prefer the first variants of the mentioned examples in my own submissions, but in many cases, it is presented as if these submissions are rejected correctly only because someone finds a bit upside potential. That is my main problem with these nitpicking case discussions, so I excluded them intentionally.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @JoCze-PGO Having to do more busywork in reviewing would certainly cause some reviewers to review less, and cause some to quit reviewing. Imagine a good reviewer who can turn around reviews in an average of 25 seconds, and who has a spare 15 minutes to review. On average that person will churn out 36 reviews in that 15-minute period.

    That average doesn't tell the whole story, though. I just looked at my last 20 reviews. Two were duplicates. 11 were 1* rejections.

    I estimate that I can turn out a duplicate in ten seconds on average. About half of my 1* rejections are also that fast because they're excruciatingly obvious rejections. So let's call it seven reviews that I have to waste an extra ten seconds on, or 70 seconds of wasted time. That's two candidates that I wasn't able to review during that 15 minute period, or a decrease of eight reviews per hour. That's a 5% reduction in my total throughput. And no, I don't count 20 seconds for each review... I just review at a natural speed. If Niantic decides to give me a timeout it costs them more than it costs me... I don't get any useful benefits from reviewing, and they're losing the value of my labor.

    How much will adding additional busywork improve the quality of my reviews? Zero. I already take plenty of time on the ones that need it. I don't need to waste time checking a bunch of checkboxes on "KinderCare Learning Center", "Scott M. Leaman Elementary School", or "Roseville Honda", all of which were in this sample of 20 reviews. The first two are painfully obvious K-12 rejections, and the third had a main photo of two guys sitting in a breakroom.

    But the 5% doesn't tell the whole story either. If I had to jump through a bunch of stupid hoops for no net benefit then my actual review throughput would fall by 100% since I'd be frustrated by the waste of my time. To calibrate that for you, I did almost 1100 reviews in June 2021.

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    I tried it myself. I needed 10 seconds to put 1 star in each category, scroll back up and chose a rejection reason. The same goes for simply giving 5 stars to all subcategories. If you really think that this bit of a delay is too much to handle or is stopping you from having your top throughput, so be it. But if it helps restricting these fast clickers and gaining more overall reviewing quality these few seconds should not be the reason to run this suggestion down.

  • SlicedPeas-INGSlicedPeas-ING Posts: 336 ✭✭✭✭

    And again, I'll point out that adding "must one star all categories to reject" will just result in that being added as a shortcut to browser plug-ins and so you'll mostly inconvenience those not using those tools. It's a very bad idea.

  • Roli112-PGORoli112-PGO Posts: 2,236 Ambassador

    AGAIN... why would i have to 1* categories that dont deserve 1*... even if its a clear reject?

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    In the same way, you could use a plugin that automatically distribute 5 stars to all categories. So I dont get the point tbh. The idea is to even out rejection and acceptance, so none of these are more lucrative than the other one.

  • JoCze-PGOJoCze-PGO Posts: 54 ✭✭✭

    You read my statement on that?

    I can understand the other argument having to "ly" about subcategories, but I am pretty sure that complete rejections are treated exactly like that (1* in all categories and then based on all reviews an average for all categories is formed).

    Furthermore I would not have a problem at all, if you can rate specific subcategories with more than one star and still reject the complete submission at the end. The main point here is just to make the reviewer have to pick all of the categories first.

    I really dont understand, why this is such a big problem?

Sign In or Register to comment.