What is the purpose of 2 and 4 star ratings?
tehstone-ING
Posts: 1,037 Ambassador
In the how to review section, 1,3,and 5 star ratings are described as: • If you strongly agree, choose 5 stars • If you are unsure or have no opinion, choose 3 stars • If the answer is definitely no, choose 1 star and select a rejection reason But 2 star and 4 star are not described. What should the 2 star and 4 star ratings be used for?
Comments
I’m very much interested in this, as I’ve heard conflicting answers from multiple people. Especially since we got the clarification last ama that a 1* in some categories isn’t even a rejection.
Isn't it just as simple as 4 stars means agree as opposed to strongly agree or indifferent? Similarly, 2 stars means disagree as opposed to indifferent or strongly disagree?
I have always appreciated that I could chose 2 or 4 stars.
Under the old CAG, there was weird instructions for things like (and I'm not going to fact check myself, here) "trail markers, accept, 5*" and "post offices, accept, 4*"
Why? Why rate something 4* if the desired outcome is to approve? Why the ambiguous 2* that may be an approve or may be a reject?
Giving out ratings that aren't 1 or 5 stars also punishes you for taking a side but not helping it as much as you could have, making you more likely to get a disagreement.
I give 4 stars to those things that do not get high ratings in cultural or historical signficiance. This is in keeping with practices from the Candidate Action Guide were post offices, playgrounds, athletic fields, trail markers, etc. were all given a recommended rating of 4 stars.
Do you have any source to cite on this? I have not heard this before. I use 2 and 4 regularly and get plenty of agreements. I have always been rated "great" on the reviewer rating on the profile page.
I think they're basing it on just theorizing a hypothetical situation where a nomination is almost split outcome between accept & reject and "you" have voted 4* all the way down. It lands on the reject side, but if "you" had given it 5* it would have been accepted.
That's the point I was eluding to: why rate something 4* if it's otherwise acceptable? (Not to mention we see people adamantly insist on only giving 3* ratings for approving nominations.)
Why not just have the review page have an updated "counter" saying "Based on your current scores, you ave voted to accept / not accept this Waypoint". Avoid all the confusion.
Some reading on why 5 star rating schemes aren't the best.
<http://www.evanhamilton.com/star-rating-systems-j-curve/>
<https://www.retailwire.com/discussion/do-five-star-ratings-systems-have-a-positivity-problem/>
<https://techcrunch.com/2009/09/22/youtube-comes-to-a-5-star-realization-its-ratings-are-useless/>
<https://medium.com/@alialghamdi/like-dislike-rating-is-probably-right-for-you-665bf18db887>
In short, such systems trend towards very large numbers of 1 and 5 star reviews and very little in between. Why does Wayfarer even need a 5 star scale? Simplifying to a yes/no or 1-2-3 rating would be an improvement in clarity and usability.
I like the 5* system myself. I feel it's clear enough. 4* likely means "acceptable but may be a bit lacking in something" and 2* likely means "Probably doesn't qualify but isn't clearly ineligible", or something along those lines.
I feel some reviewers (especially new ones) might tend to give predominantly indecisive votes if the Wayfarer system moved to a 1-2-3 system instead of keeping the 5* one. In order to earn the full 3* vote, they might feel compelled to think the submission would have to be perfect and impeccable in every single account before they could award the full 3* rating and might go for the neutral vote instead. So if having 2* and 4* vote options help avoid people giving unnecessary neutral votes, I feel they do have their place in the system.
I personally don't believe 2* and 4* votes affect the results overly much (though I could be wrong). I feel the bigger issue might be the people who, for example, would still like to outright reject eligible candidates for dubious reasons or who still don't quite understand what visually unique really means and similar misunderstandings of the criteria that causes people to rate something lower than just 4*.
Very interesting. Additionally, perhaps Nomination Appeals by Niantic is perhaps a two-step evaluation, approval or denial. What is the difference? Is it necessary? Also I am concerned about the rating system here.
I vote stuff as 4 star because niantic said it doesn't hit your rating as hard of people disagreed with you (as in rejected it). Never use 2 star thiugh
because niantic said
Citation?
I've always taken it as... 1* = Unacceptable, 2* = Poor, 3* = Average or Dont Know, 4* = Good, 5* = Perfect.
For eligible submissions I usually start at a base 4* and if it is a particularly interesting or you can tell the submitter has put effort in then bump it, or for each thing wrong knock a star off.
Can't remember which one it was, but it was an old ama, I believe it was in answer to someone asking how to stop their rating going down
There's 72 responses with the word "rating" and none directly confirm your theory.
https://ingressama.com/search?q=Rating
Why? Why rate something 4* if the desired outcome is to approve? Why the ambiguous 2* that may be an approve or may be a reject?
I hope something along this line gets addressed. There is a binary outcome of voting - your candidate is either accepted or rejected and whether it appears in various games is determined by the platform based on spacing logic. Many users have requested a binary (or at least yes/no/indifferent) voting option rather than the gradient.
A previous AMA even said 2* votes would be a rejection, although many people are interpreting November 2021's answer as a contradiction (which I don't think it is), and elsewhere it was suggested that games might eventually use that rating scale (hmm sure sounds ripe for abuse). But bottom line is, it's a binary outcome and having the 5 scale votes has lead to petty arguments about how to rate acceptable/eligible candidates.
I'm not saying it's a theory, it was from an ama and the idea was you aren't being as extreme in your rating, hence why you don't take as much as a hit. I'm sure someone will get the exact ama because it was spread about when I started (I only started when pogo got nominations)
If you can't find something citing it, it's a theory. The burden of proof is on you. I provided a link to the AMAs and personally didn't find what you suggested.
And, should this be true, my question still stands. Why should somebody vote 4* if they believe it's eligible? If they'd have given it 5*, maybe that extra upvote would have caused it to be approved and there wouldn't need to have been a hit on their rating.
Yeah I can't seem to find it either but it wasn't me who made this up it was said by several people when I first started. But seeing g as you did link the amas, you should have seen they explain the ratings a bit, 4 star is an agreed but not a hard agree, they even explain the 4 star should be used for candidates that aren't amazing or aren't fully explained. My guess is that's where the rsti g thing comes from, as he called it hard agree
I did see those, which is why I've included my additional comments throughout the thread that I hope gets adequately addressed.
Why rate something 4* if the desired outcome is to approve?
I think a lot of us are tired of reviewers who insist 2* or 3* is enough to pass a nomination. If it is, then so be it, and we'll have a Niantic answer to that.
I'd like to see a 4 star system. Make people make a choice.
1) Ineligible for Specific Reason
2) Doesn't reach eligibility in 3 main criteria:
3) Is eligble in one 3 main criteria
4) Is an exceptionally good example of one of the eligibility criteria
Yeah I defo don't agree with people saying 2 and 3 star is an agreement, I literally can't find anything that says a 2 star is an accept, same with 3 star. Everything I see is 2 star being a reject, just not a flat out rejection, and 3 star is no opinion according to the actual wayfarer help reviewing guide
I literally can't find anything that says a 2 star is an accept, same with 3 star.
I thought there was something buried in an AMA saying a 2* was accept (and, of course 3*) but I'm not looking right now and admit I could be incorrectly remembering right now - and I know some of those Ingress forum AMAs flip flopped a lot and I don't think necessarily truly represented the OPR/Wayfarer team as many could have strictly been Mr. Krug's responses. I think a lot of people are possibly incorrectly assuming the 2* is not a reject since the November 2021 response. Anyways, hopefully the answer won't be as disappointing as many of the last AMA responses.
I use 2, 3, and 4 stars a lot.
I rarely use 5 stars.
I think about the number of points a candidate should have if their nomination is perfect.
I think about whether they have 100 scores, 80 scores, or only 60 scores.
So, for example, if a candidate is considered to have only 80 scores, we use 4 stars even if everything is perfect in the nomination.
So there is no need to give 5 stars to a candidate whose original score is low.
On the contrary, I am honestly surprised that some judges would give 5 stars to a candidate whose historical or cultural significance or uniqueness cannot be evaluated.
There is no point in having a 5-star rating then.
Also, if the title or description is wrong, the location information is wrong, or it cannot be found, even if it is an approvable nomination, the corresponding part will be dropped.
The overall rating will also be lowered by about one star.
Giving full marks for inappropriate titles and descriptions, wrong location information, inappropriate photos, etc. sends the wrong message to the players in the field.
It would be nice to be able to correct them and ask them to resubmit, but the current system is different. Therefore, the appropriate thing to do is to lower the rating.
What the reviewers are looking for is a player who submits better nominations.
They don't want players who submit wrong nominations.
And I think the increase in the number of players with extreme yes/no reviews has led to an increase in low quality wayspots, which in turn has led to an increase in players submitting inappropriate nominations.
I use the 2-star rating a lot recently, especially for those kind of nominations, which I get all the time and honestly do not know how to handle correctly:
They simply aren't that unique and do not have the best historicall value (on both subjects, I usually give them one star).
On the other hand, I see in the nearby wayspots lots of them which got accepted (really curious how, but OK)..
Also the title is correct and the object is well visible on the google street view (so in theory that is word 4 or 5 stars).
But this strategy is not getting valued by the system, since my rating dropped in 3 weeks from Great to Poor 😥 (I've been reviewing for 2,5 years now and did over 12K and this is the first time my rating dropped that low..).
I just wanna review in an honest way, which I think I am doing according to the 2-star method. It's just 60% of the nominations I have to do are those freaking Trailmarkers..
So a little AMA clarification on the 2-star rating (and those **** one-of-a-dozen Trailmarkers) would bring back the fun in reviewing for me (or the possibility to skip 50 nominations a day, in stead of 5)..
If 3* means unsure or no opinion, it makes sense that 2* is leaning towards rejection. If you are 2*/rejecting those kinds of trail markers, it makes sense that your rating dropped to poor. You are supposed to accept these. Both of those signs are acceptable. There was recent clarification on this:
https://community.wayfarer.nianticlabs.com/discussion/comment/111961#Comment_111961 https://community.wayfarer.nianticlabs.com/discussion/comment/112676#Comment_112676 https://community.wayfarer.nianticlabs.com/discussion/comment/113993#Comment_113993
Also: Visually unique means that it stands out from it's surroundings, it can easily be found/identified. These markers are designed to stand out. So a 1* rating on visually unique is definitely too low.
Hi, Thanks for helping me. This absolutely makes sense.
The visually unique part still bothers me.. In my country (Holland) we have tons of those markers. Some are exactly the same (number and the same direction arrow). I personally did not see the point of accepting those. But if Niantic made up their mind last fall, who am I to judge ? (this declares why my rating has suddenly dropped since november 2021).
But to double check: if I rate those trailmarkers from now on, with the following stars, you think my score will get up within the next weeks?:
Should this be a wayspot - 3 stars
Title and description - Depends on the nomination/quality of the text
Historic or Cultural Significance 2 or 3 stars
Visually Unique 2 stars
Safe access Either 4/5, or 1 star at 'Should this be a wayspot?', whenever there is no pedestrian access
Location accuracy Depends on the nomination
This is what the help section says about Visual Uniqueness:
Visual Uniqueness
Does the nomination stand out from its surroundings? Wayspots that are easy to locate and visually distinct from the buildings and objects nearby make high-quality Wayspots and should be rated highly. If you think the nomination looks bland and will be hard to locate, give it a lower rating.
It doesn't matter what you can see on the map while reviewing. What you are supposed to look at, is if you are at that location in person, how easy is it to find/identify that specific marker. Sometimes you'll find another one just a little further down the road because there is another intersection there. But most of the time, it'll be pretty easy to find and identify that specific marker at that location.
The way I think we are supposed to rate wayspots, is not by using our personal preferences, but by evaluating wayspots against to criteria. If something certainly doesn't meet criteria, I rate 1*. If I think something doesn't meet criteria, but I am not sure, I rate 2*. If I honestly have no idea whether something meets criteria, I rate 3* If I think something meets criteria, but I am not sure, I rate 4*. And if I am sure something meets criteria, I rate 5*. So I would always rate these trailmarkers 5* on "Should this be a wayspot". But that is my personal interpretation, so I might be wrong on this. But whatever your interpretation, 3* is still not an accept. You should rate at least 4*.
Historical or cultural significance is difficult. It is very unclear what we are supposed to do there.
Visually unique, as explained above, I would usually rate 5* if there isn't another similar looking one nearby, and it doesn't blend into the background. I have yet to see one that I would rate lower than 3*.
I live in Belgium btw, we have the exact same markers.