Changing the 5 star system to a 4 star system

Would people have an easier time rating and nominating submissions if the star system was simpler?


1* bad

2* ok/neutral

3* good

4* excellent

«1

Comments

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @Rodensteiner-PGO A 2* is a weak disagreement. It doesn't "trash the submission", it just gives it a low score for that specific field.

    Some fields are more important than others. I'm pretty sure the overall "Should this be...?" is the most important and that safe pedestrian access is very important. There's a myth that anything less than three stars in any category is a vote to reject the candidate, but that's not true. (And yes, a bunch of people who believe and perpetuate the myth are going to hit the disagree button.)

  • pkmnsearch2-PGOpkmnsearch2-PGO Posts: 249 ✭✭✭

    i would prefer a yes no maybe

  • pkmnsearch2-PGOpkmnsearch2-PGO Posts: 249 ✭✭✭

    @Hosette-ING how do you explain the reject reasons that appear in the rejection email saying "not visually unique" or "not historical/culturally relevant"?


    please, i am curious.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @pkmnsearch2-PGO Based on the examples I've seen, and I've looked at a LOT of them in various forums, I think what's happening is this: The code that generates the rejection email tries to fill in three rejection reasons. It starts by looking at the 1* hard-rejection reasons. If it doesn't find three from there it starts digging into the fields below "Should this be a Wayspot?" looking for things that scored kinda low and "helpfully" adding them to the emali. I'm pretty sure this is the only way historical/cultural and visually unique show up.

    I'm 99.9% certain that the pass/fail of a candidate is based on an overall score calculated from reviewer ratings for each of the individual fields, using a "secret sauce" formula that gives different fields different weights. I believe this for two reasons. One is that if I think about it from an engineering perspective this is how I would design the system-- it's illogical to me that a candidate that scored 4-5 stars overall but 1-2 stars for a relatively unimportant field would be rejected. That just seems like a dumb design decision for something that isn't mandatory. Second, and more importantly, I've heard a Niantic employee say that during a discussion at an event.

  • Euthanasio2-PGOEuthanasio2-PGO Posts: 272 ✭✭✭

    It is not a myth and I would like to ask people who got a rejection emails with the sole written reason was "no cultural value" or "not visually unique" to provide screenshots

  • pkmnsearch2-PGOpkmnsearch2-PGO Posts: 249 ✭✭✭

    I dont think it's a myth and i dont take it as a fact either but....Better evaluate on the safe side, i think....

    ----------

    "One is that if I think about it from an engineering perspective this is how I would design the system-- it's illogical to me that a candidate that scored 4-5 stars overall but 1-2 stars for a relatively unimportant field would be rejected. "

    This is Niantic that we are talking about. yes, Maybe it is illogical but maybe is that is being applied... illogically.

    -------------

    "Second, and more importantly, I've heard a Niantic employee say that during a discussion at an event."

    Ahm, thank you random player from the internet for this piece of vital information. Apparently only you know...

    ------------

    After knowing that less than 3* is a negative review from my part (not necessarly rejecting it, like you say) i don't think it is a good idea to give less than 3* to something that I overral approve/pass.




    Some others sources:

    https://www.reddit.com/r/IngressOPR/comments/dgetje/historic_or_cultural_significance_rating_causing/

    https://community.wayfarer.nianticlabs.com/discussion/12023/not-historically-significant-or-visually-unique/

    https://community.wayfarer.nianticlabs.com/discussion/comment/58258

  • Edocsil47-PGOEdocsil47-PGO Posts: 150 ✭✭✭✭

    Whether there are thresholds to be met for each category or the scores are combined into a weighted average is semantics. Unless a category is completely omitted from the final decision (which I wouldn't personally bet on), giving any category 2* is almost definitely objectively closer to rejecting the nomination than giving that same category a 5*.

    Maybe voting 2* in "visually unique" and 5* in everything else still makes your vote on the "accept" side of the cutoff, but it's still a less confident vote for the nomination that makes it more likely to be rejected from what other voters do.

    Not saying some nominations don't deserve a low score in other categories.

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭
    edited April 2021

    I've been given rejection reasons for not being visually unique or culturally significant, so rating those sections low can tank a nomination.

    @Euthanasio2-PGO you wanted people to post these right?

    And whether or not 2* is rejecting a submission, there's these Ingress AMA answers:

    Make of those what you will.

    Post edited by GearGlider-ING on
  • HaramDingo-INGHaramDingo-ING Posts: 1,725 ✭✭✭✭✭

    It really is no myth. Many nominations of mine have come back with solely the not culturally significant/visually unique reason, so it is not just looking for the top 3 rejection reasons. Some of these rejected for as such actually have nothing around them, and even tennis courts and playgrounds are getting the short end of the visual stick.

    In relation to OP's question of whether we should change the star system, if you really want less stars, then turn it into a one-star system. Either cast your star and place it on a nice nomination, or retain your star. Kind of reminds me how K-2 teachers say "you'll get a gold star" for doing a good job! And then when it comes to finishing a test, they give out five instead to their favourites or achievers and automatically just give one to the lowest achievers for pity.

    Either one star or bust.

  • KetaSkooter-INGKetaSkooter-ING Posts: 177 ✭✭✭

    The 3* average equals a reject comes from a conclusion when OPR started. Easy enough for niantic to have adjusted things since then but unlikely.


    certain questions should have always been yes no like safety and the initial. I saw a twitch video where the dude 3* every thing including safety saying that was a great approval. , I’m not sure what sorta safe means. Luckily not many have watched his video.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    A lot of the people telling me that I'm wrong are the same people who complain incessantly on this forum about how badly broken Wayfarer is. In the face of that you believe that his one little corner of the system, the one that picks out what rejection reason to send, is flawless and laser-targeted in its accuracy?

    3* is a weak agreement, 2* is a weak rejection. I believe it's always been that way.

    It makes me sad that people keep spreading the falsehood that a 1-2* rating in any category is a vote to reject the entire submission. It's just not. I wish one of the Niantic spokespeople would step in and confirm that so that the myth could die and people would rate submissions accurately rather than based on some rumor that somehow everybody started believing.

  • GearGlider-INGGearGlider-ING Posts: 1,335 ✭✭✭✭✭
    edited April 2021

    @NianticCasey-ING or @NianticGiffard, there seems to be some confusion about the ambiguity of the wayfarer review system in this thread. Would you be able to possibly tell us if rating a single category 1* while rating the other categories highly is a review to reject the nomination for failing to meet that category? I would like to make sure I am not approving anything that might fail to meet the criteria standards.

  • KetaSkooter-INGKetaSkooter-ING Posts: 177 ✭✭✭

    I assume you’re talking about cultural/visual categories because all the others 1* is a rejection. I remember a consensus being that one or the other should be rated high but both do not. The description of each has changed in the last couple years though.

  • TWVer-INGTWVer-ING Posts: 792 ✭✭✭✭✭

    First, 3* is not a weak approval, 4* is a weak approval. 3* is neutral.

    • If you strongly agree, choose 5 stars
    • If you are unsure or have no opinion, choose 3 stars
    • If the answer is definitely no, choose 1 star and select a rejection reason

    Second, we do not know the details of how the rating system works. Maybe a 1* or 2* rating in any category is a reject, maybe it isn't. We don't know. But claiming it certainly isn't is equally as wrong as claiming it is.

    Probability is that many reviewers don't understand both those categories, and rate to low on them.

  • KetaSkooter-INGKetaSkooter-ING Posts: 177 ✭✭✭

    That’s not quite right, we know Niantic wants safe and accurate locations with accurate names and descriptions. If a nomination were to somehow average low in any of those three categories without outright rejections there’s no reason to think it’d pass and go live.

  • pkmnsearch2-PGOpkmnsearch2-PGO Posts: 249 ✭✭✭

    with the information that you we can find online, is hard NOT to consider that by giving less than 3* in each section we are approving something.

    once again, harder to be affirmative about being a myth 100%.

    but you do you.... wayfarer rating doesn't necessarily correlate directly how a great evaluator we are.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @TWVer-ING If you rate a candidate 3* for "Should this be a wayspot?" and it's approved, does that count as an agreement or a disagreement for your rating?

  • TWVer-INGTWVer-ING Posts: 792 ✭✭✭✭✭

    I would assume rating a candidate 3* would not count towards your rating either way.

    This is how I think it works (without taking nuances like how you vote in other categories into account):

    • If a candidate is approved, you get an agreement towards an upgrade and your rating increases if you voted 4* or 5*. Your rating decreases if you voted 1* or 2*.
    • If a candidate is rejected, you get an agreement towards an upgrade and your rating increases if you voted 1* or 2*. Your rating decreases if you voted 4* or 5*.
    • If you rated 3*, the outcome doesn't matter. You do not get an agreement towards an upgrade and your rating remains the same.

    Maybe rating a candidate 1* or 5* has a bigger effect (positive or negative) on your rating, than rating it 2* or 4*.

    But this is all speculation, and we cannot know how it works unless Niantic explains it to us.

  • pkmnsearch2-PGOpkmnsearch2-PGO Posts: 249 ✭✭✭

    I have another question to you.

    When do you give 2* or 1* to any other category if NOT an instant-1* overall ? why give 5* to other categories and then 1* on safe pedestrian if it is the same as giving an instant-1* with "no safe pedestrian acess" reject reason?

    After a while in reviewing, we see so many valid examples that correspond to the criteria that it is a bit automatic process.

    • trail markers
    • gazebos, pavillion, pergolas
    • parks or beaches (with pin in the entrance or a sign/placemarkers)
    • plazas
    • post offices
    • noticeboards
    • political HQ
    • statues
    • playgrounds, outdoor exercise equipment
    • memorial plaques (with descriptive text)
    • maps
    • library, city halls
    • crosses and churches
    • small holy shrines
    • hórreo/granaries
    • wind and water mills
    • mural/urban art
    • fountains and other water features
  • Raachermannl-INGRaachermannl-ING Posts: 1,070 ✭✭✭✭✭

    In general it is a good idea and a common empirical method to switch to a 4 star system to force the people to vote with a tendency.

    But for wayfarer there is the need to solve the problem with the nominations in foreign languages. There are often enough examples, where translation tools dont help and I dont want to judge a nomination. For example

    • when people from Czech Republic or Poland don't use their äddîtiónál châräcter signs, then I as a German dont want to judge these nominations
    • when nominations in foreign languages need google searches to prove them .....

    As long as problems like these are out there, I'm pleased, that there is the possibility to 3* a nomination in all categories.

  • sogNinjaman-INGsogNinjaman-ING Posts: 3,313 ✭✭✭✭✭

    My basic "ok to be a Waypoint" scores for your everyday, no problem submissions like Playgrounds, Post Offices, Pubs etc "33 33 55". I only tend to rate POIS higher for the four questions if they are really outstanding, I can't remember the last time i gave anything "55 55 55".

    Rejection votes are always 1* - why mess about.

    For any future system, the maximum number of points I would want to see would be 3 (No / Accept / Excellent) though I would prefer a simple 2 question "Yes / No" approach would make life easier all round, and do away with a lot of the speculation about "What happens if you rate this question 2*". Much simpler for both submitters and reviewers to understand. At the end of the day it is a simple binar decision, do we accept this waypoint or not?

    As for the questions, do we actually still need the "historic / cultural" question now the main criteria have changed? Certainly keep the "Is this visually unique" question, but clarify what is meant by it for repetative things like identical trail markers every 40 m.

  • Hosette-INGHosette-ING Posts: 3,469 ✭✭✭✭✭

    @pkmnsearch2-PGO asks:

    When do you give 2* or 1* to any other category if NOT an instant-1* overall ? why give 5* to other categories and then 1* on safe pedestrian if it is the same as giving an instant-1* with "no safe pedestrian acess" reject reason?

    Safe pedestrian access is a weird one because it's a primary rejection reason as well as a 1-5*. I would 1* reject something it clearly didn't have safe pedestrian access rather than rating the rest. I wish Niantic would remove this redundancy because it's confusing.

    The two fields that get discussed the most on this topic are historical/cultural significance and visual uniqueness. Neither of these things is a requirement for a wayspot, unlike safe pedestrian access. Niantic's guidance for historic/cultural is to provide an example of one thing being more significant than the other, and "Use your best judgement and rate the nomination accordingly. " In my best judgement, the FooBar Trail Mile 3.5 Marker has no particular historical or cultural significance. Few people know it's there, it's not a destination, and essentially nobody would miss it if it disappeared... a very few hikers might, but they'd just use Mile 3 and Mile 4. My best judgement is that the trail marker should be a wayspot but that it rates very low on historic/cultural significance. Thus it would get 5* for "Should this be?", appropriate marks for title/description and the other fields, and a low rating for historical/cultural significance. Assuming the submission was solid the net result would be a strong approval for the wayspot as a whole even though it doesn't have much historical/cultural significance.

    The pushback I get on this sometimes is, "But hiking is important. Nature is important." I agree with those things. I just don't believe that one specific otherwise-uninteresting trail marker carries the same import as the overall activity.

    (I'm sure I'll get dislikes for this one too.)

  • Mescheryakov-INGMescheryakov-ING Posts: 9 ✭✭
    edited April 2021

    Getting back to the original question, eliminating the possibility for a neutral vote (3* in a five-star system) would make me even less inclined to review than I am now. So I would be discouraged by the four-star proposal by the OP.

  • 52cucumbers-ING52cucumbers-ING Posts: 225 ✭✭✭✭

    Same. I use the 3* when I don't really have an opinion. Since the cultural value of trail markers was mentioned that's a very common instance for me to use it. If there was a 4 star scale I would have to give those 2*. For me the 3* is saying "I'm kinda neutral on this". Another case where I use it often is for the location. If I can't establish wether the location exists (this is VERY common for trail markers) but I think it's likely that it's there 3* is the natural choice. Forcing me to say "yes" or "no" when there is no way for me to be sure would not be a good experience.

  • gazzas89-INGgazzas89-ING Posts: 88 ✭✭✭

    Just going to say, I've had rejections emails that was literally just the historical or visual not bei g good enough, none of the hard rejections

Sign In or Register to comment.