Changing the 5 star system to a 4 star system
erockplatypus91-PGO
Posts: 1 ✭✭
Would people have an easier time rating and nominating submissions if the star system was simpler?
1* bad
2* ok/neutral
3* good
4* excellent
Tagged:
erockplatypus91-PGO
Posts: 1 ✭✭
Would people have an easier time rating and nominating submissions if the star system was simpler?
1* bad
2* ok/neutral
3* good
4* excellent
Comments
1* = No Way
2* = Neutral
3* = Yeh!
This clearly says 1* means NO. There are still people out there that 2* and think that this is OK. Infact, it trashes the submission.
@Rodensteiner-PGO A 2* is a weak disagreement. It doesn't "trash the submission", it just gives it a low score for that specific field.
Some fields are more important than others. I'm pretty sure the overall "Should this be...?" is the most important and that safe pedestrian access is very important. There's a myth that anything less than three stars in any category is a vote to reject the candidate, but that's not true. (And yes, a bunch of people who believe and perpetuate the myth are going to hit the disagree button.)
i would prefer a yes no maybe
@Hosette-ING how do you explain the reject reasons that appear in the rejection email saying "not visually unique" or "not historical/culturally relevant"?
please, i am curious.
I honestly don’t think that a scale from 1-5 or even 1-4 would make any difference in the outcome of the reviews
@pkmnsearch2-PGO Based on the examples I've seen, and I've looked at a LOT of them in various forums, I think what's happening is this: The code that generates the rejection email tries to fill in three rejection reasons. It starts by looking at the 1* hard-rejection reasons. If it doesn't find three from there it starts digging into the fields below "Should this be a Wayspot?" looking for things that scored kinda low and "helpfully" adding them to the emali. I'm pretty sure this is the only way historical/cultural and visually unique show up.
I'm 99.9% certain that the pass/fail of a candidate is based on an overall score calculated from reviewer ratings for each of the individual fields, using a "secret sauce" formula that gives different fields different weights. I believe this for two reasons. One is that if I think about it from an engineering perspective this is how I would design the system-- it's illogical to me that a candidate that scored 4-5 stars overall but 1-2 stars for a relatively unimportant field would be rejected. That just seems like a dumb design decision for something that isn't mandatory. Second, and more importantly, I've heard a Niantic employee say that during a discussion at an event.
5-Star and 3-Star is more intuitive than 4-Star. Very easy to explain and get people to remember the middle star is Neutral. And I like being able to rate 2* for "I dunno for sure, but probably not"
It is not a myth and I would like to ask people who got a rejection emails with the sole written reason was "no cultural value" or "not visually unique" to provide screenshots
Second, and more importantly, I've heard a Niantic employee say that during a discussion at an event.
Earnest as that may be, from you and from the Niantic employee, 1) every Niantic employee on this Wayforum & the Ingress forums has misspoke at least once, 2) the system could be changed at any time, and
3) The argument exists that if a nomination receives 1-2* ratings for cultural/historical (we keep discussing these two the most, but title, safety, and location may play similarly) the nomination may be rejected overall, regardless of the overall "main" rating given. I believe the claim you continue to make is that "There's a myth that anything less than three stars in any category is a vote to reject the candidate, but that's not true." I don't think that's the myth that I have stated but it is perhaps how I have explained it. I believe that enough 1-2* ratings for subcategories will reject the nomination.
A nomination receives 3* on average overall (just factoring "should this be a Wayspot") but 2* average on the subcategories - a rejection. Had said nomination received 3* on average overall but 4* on the subcategories - perhaps that would cause an approval.
In the end, it's possible you're right, possible I'm right, possible neither of us, a combination of both of us, maybe one of us once was right but no longer is or even will be right in the future. I will guarantee that just the members alone who have contributed to the discussion on the Wayforum have spent more time thinking about it and the impact of their votes than those who coded it.
I dont think it's a myth and i dont take it as a fact either but....Better evaluate on the safe side, i think....
----------
"One is that if I think about it from an engineering perspective this is how I would design the system-- it's illogical to me that a candidate that scored 4-5 stars overall but 1-2 stars for a relatively unimportant field would be rejected. "
This is Niantic that we are talking about. yes, Maybe it is illogical but maybe is that is being applied... illogically.
-------------
"Second, and more importantly, I've heard a Niantic employee say that during a discussion at an event."
Ahm, thank you random player from the internet for this piece of vital information. Apparently only you know...
------------
After knowing that less than 3* is a negative review from my part (not necessarly rejecting it, like you say) i don't think it is a good idea to give less than 3* to something that I overral approve/pass.
Some others sources:
https://www.reddit.com/r/IngressOPR/comments/dgetje/historic_or_cultural_significance_rating_causing/
https://community.wayfarer.nianticlabs.com/discussion/12023/not-historically-significant-or-visually-unique/
https://community.wayfarer.nianticlabs.com/discussion/comment/58258
Whether there are thresholds to be met for each category or the scores are combined into a weighted average is semantics. Unless a category is completely omitted from the final decision (which I wouldn't personally bet on), giving any category 2* is almost definitely objectively closer to rejecting the nomination than giving that same category a 5*.
Maybe voting 2* in "visually unique" and 5* in everything else still makes your vote on the "accept" side of the cutoff, but it's still a less confident vote for the nomination that makes it more likely to be rejected from what other voters do.
Not saying some nominations don't deserve a low score in other categories.
I've been given rejection reasons for not being visually unique or culturally significant, so rating those sections low can tank a nomination.
@Euthanasio2-PGO you wanted people to post these right?
And whether or not 2* is rejecting a submission, there's these Ingress AMA answers:
Make of those what you will.
It really is no myth. Many nominations of mine have come back with solely the not culturally significant/visually unique reason, so it is not just looking for the top 3 rejection reasons. Some of these rejected for as such actually have nothing around them, and even tennis courts and playgrounds are getting the short end of the visual stick.
In relation to OP's question of whether we should change the star system, if you really want less stars, then turn it into a one-star system. Either cast your star and place it on a nice nomination, or retain your star. Kind of reminds me how K-2 teachers say "you'll get a gold star" for doing a good job! And then when it comes to finishing a test, they give out five instead to their favourites or achievers and automatically just give one to the lowest achievers for pity.
Either one star or bust.
The 3* average equals a reject comes from a conclusion when OPR started. Easy enough for niantic to have adjusted things since then but unlikely.
certain questions should have always been yes no like safety and the initial. I saw a twitch video where the dude 3* every thing including safety saying that was a great approval. , I’m not sure what sorta safe means. Luckily not many have watched his video.
A lot of the people telling me that I'm wrong are the same people who complain incessantly on this forum about how badly broken Wayfarer is. In the face of that you believe that his one little corner of the system, the one that picks out what rejection reason to send, is flawless and laser-targeted in its accuracy?
3* is a weak agreement, 2* is a weak rejection. I believe it's always been that way.
It makes me sad that people keep spreading the falsehood that a 1-2* rating in any category is a vote to reject the entire submission. It's just not. I wish one of the Niantic spokespeople would step in and confirm that so that the myth could die and people would rate submissions accurately rather than based on some rumor that somehow everybody started believing.
@NianticCasey-ING or @NianticGiffard, there seems to be some confusion about the ambiguity of the wayfarer review system in this thread. Would you be able to possibly tell us if rating a single category 1* while rating the other categories highly is a review to reject the nomination for failing to meet that category? I would like to make sure I am not approving anything that might fail to meet the criteria standards.
I assume you’re talking about cultural/visual categories because all the others 1* is a rejection. I remember a consensus being that one or the other should be rated high but both do not. The description of each has changed in the last couple years though.
First, 3* is not a weak approval, 4* is a weak approval. 3* is neutral.
Second, we do not know the details of how the rating system works. Maybe a 1* or 2* rating in any category is a reject, maybe it isn't. We don't know. But claiming it certainly isn't is equally as wrong as claiming it is.
Probability is that many reviewers don't understand both those categories, and rate to low on them.
That’s not quite right, we know Niantic wants safe and accurate locations with accurate names and descriptions. If a nomination were to somehow average low in any of those three categories without outright rejections there’s no reason to think it’d pass and go live.
with the information that you we can find online, is hard NOT to consider that by giving less than 3* in each section we are approving something.
once again, harder to be affirmative about being a myth 100%.
but you do you.... wayfarer rating doesn't necessarily correlate directly how a great evaluator we are.
@TWVer-ING If you rate a candidate 3* for "Should this be a wayspot?" and it's approved, does that count as an agreement or a disagreement for your rating?
I would assume rating a candidate 3* would not count towards your rating either way.
This is how I think it works (without taking nuances like how you vote in other categories into account):
Maybe rating a candidate 1* or 5* has a bigger effect (positive or negative) on your rating, than rating it 2* or 4*.
But this is all speculation, and we cannot know how it works unless Niantic explains it to us.
I have another question to you.
When do you give 2* or 1* to any other category if NOT an instant-1* overall ? why give 5* to other categories and then 1* on safe pedestrian if it is the same as giving an instant-1* with "no safe pedestrian acess" reject reason?
After a while in reviewing, we see so many valid examples that correspond to the criteria that it is a bit automatic process.
In general it is a good idea and a common empirical method to switch to a 4 star system to force the people to vote with a tendency.
But for wayfarer there is the need to solve the problem with the nominations in foreign languages. There are often enough examples, where translation tools dont help and I dont want to judge a nomination. For example
As long as problems like these are out there, I'm pleased, that there is the possibility to 3* a nomination in all categories.
My basic "ok to be a Waypoint" scores for your everyday, no problem submissions like Playgrounds, Post Offices, Pubs etc "33 33 55". I only tend to rate POIS higher for the four questions if they are really outstanding, I can't remember the last time i gave anything "55 55 55".
Rejection votes are always 1* - why mess about.
For any future system, the maximum number of points I would want to see would be 3 (No / Accept / Excellent) though I would prefer a simple 2 question "Yes / No" approach would make life easier all round, and do away with a lot of the speculation about "What happens if you rate this question 2*". Much simpler for both submitters and reviewers to understand. At the end of the day it is a simple binar decision, do we accept this waypoint or not?
As for the questions, do we actually still need the "historic / cultural" question now the main criteria have changed? Certainly keep the "Is this visually unique" question, but clarify what is meant by it for repetative things like identical trail markers every 40 m.
@pkmnsearch2-PGO asks:
When do you give 2* or 1* to any other category if NOT an instant-1* overall ? why give 5* to other categories and then 1* on safe pedestrian if it is the same as giving an instant-1* with "no safe pedestrian acess" reject reason?
Safe pedestrian access is a weird one because it's a primary rejection reason as well as a 1-5*. I would 1* reject something it clearly didn't have safe pedestrian access rather than rating the rest. I wish Niantic would remove this redundancy because it's confusing.
The two fields that get discussed the most on this topic are historical/cultural significance and visual uniqueness. Neither of these things is a requirement for a wayspot, unlike safe pedestrian access. Niantic's guidance for historic/cultural is to provide an example of one thing being more significant than the other, and "Use your best judgement and rate the nomination accordingly. " In my best judgement, the FooBar Trail Mile 3.5 Marker has no particular historical or cultural significance. Few people know it's there, it's not a destination, and essentially nobody would miss it if it disappeared... a very few hikers might, but they'd just use Mile 3 and Mile 4. My best judgement is that the trail marker should be a wayspot but that it rates very low on historic/cultural significance. Thus it would get 5* for "Should this be?", appropriate marks for title/description and the other fields, and a low rating for historical/cultural significance. Assuming the submission was solid the net result would be a strong approval for the wayspot as a whole even though it doesn't have much historical/cultural significance.
The pushback I get on this sometimes is, "But hiking is important. Nature is important." I agree with those things. I just don't believe that one specific otherwise-uninteresting trail marker carries the same import as the overall activity.
(I'm sure I'll get dislikes for this one too.)
Such a five-star rating system is called a Likert scale. It is widely used in questionnaires. As @TWVer-ING said, according to the original definition and the statements on “Help > Reviewing a Wayspot Nomination,” the definition of 3 stars is neutral, not weak acceptance.
Get back to the question that OP posted. Some researches showed that there would be some issues when we conduct a survey with Likert scales in some context.
The first issue is cultural differences. Lee et al. [1] discovered that different patterns of responses to Likert items might occur in different cultures. For example, members of some cultures might be less willing to select extreme responses.
The second issue is about statistics. Norman [2] showed that we cannot conclude a Likert scale survey by simply calculating its average score because the intensity of choices is not linear. For example, the average score of (1 one-star & 1 three-star) and (2 two-star) are the same, but their meanings are different. The former is one reviewer found strong ineligible evidence, and the other had no opinion. The latter is two reviewers reached a consensus that the nomination is not so good. But the nomination is probability still on the margins.
In my opinion, I prefer a simple yes/no question because it mitigates the effects of cultural differences. In my culture, we are less willing to give 5 stars unless the nomination is extremely good. As a result, the score of an eligible nomination may not be high enough to get accepted in some cases. I found several discussions on the local forum about high-quality nominations being rejected, but almost no discussion about low-quality nominations being accepted. That is different from the forums in other countries.
Since the ratings from reviewers and the algorithms are black boxes, I am not sure that Wayfarer has such issues. But I think a simple yes/no question would be more appropriate than Likert scales.
References:
[1] J. W. Lee, P. S. Jones, Y. Mineyama, and X. E. Zhang, “Cultural differences in responses to a Likert scale,” Research in Nursing & Health, vol. 25, no. 4, pp. 295-306, 2002.
[2] G. Norman, “Likert scales, levels of measurement and the “laws” of statistics,” Advances in health sciences education: theory and practice, vol. 15, pp. 625-632, Feb. 2010.
Getting back to the original question, eliminating the possibility for a neutral vote (3* in a five-star system) would make me even less inclined to review than I am now. So I would be discouraged by the four-star proposal by the OP.
Same. I use the 3* when I don't really have an opinion. Since the cultural value of trail markers was mentioned that's a very common instance for me to use it. If there was a 4 star scale I would have to give those 2*. For me the 3* is saying "I'm kinda neutral on this". Another case where I use it often is for the location. If I can't establish wether the location exists (this is VERY common for trail markers) but I think it's likely that it's there 3* is the natural choice. Forcing me to say "yes" or "no" when there is no way for me to be sure would not be a good experience.
Just going to say, I've had rejections emails that was literally just the historical or visual not bei g good enough, none of the hard rejections