Dedicated ML Rejection Appeals Forum Section
thenamelesskath-PGO
Posts: 380 ✭✭✭
If Niantic is going to use ML, appeals on such rejections should be unlimited for now, or at the very least a dedicated forum thread created for appealing and improving the model learning.
We're given a meagre 12 appeals annually and it's unreasonable to expect us to waste those to maybe help the model "learn". I say maybe, because appeals reviewers have also been notoriously inconsistent and even outright ignorant for the past year or so. So when they're upholding rejections for non-existent reasons, they're contributing to the ML problems.
It appears there's little recourse once a nomination has been ML rejected and then upheld by appeal staff, as subsequent resubmissions seem to be insta-rejected regardless of any changes made 🤷♀️
This problem needs more direct solutions ASAP, before the ML is further corrupted by ineffective appeal reviewers. Please consider solutions more carefully.
Comments
Meanwhile the ML:
Totally agree. I'd even add a possible thread for appeals that were wrong ly rejected. Though that kne I could see being filled with spam, most people who post their failed appeals on here tend to have majority of forum goers agreeing they should pass
appeals on such rejections should be unlimited for now
this would defeat the whole idea of using filter. What could work - return the appeal IF nomination was filtered by AI and successfully appealed and accepted. And if it was rejected by appeal reviewers - do not return appeal. And perhaps add both accepted and rejected submissions to AI learning base.
The "filter" is still learning, it's still nowhere near accurate enough to have no further recourse than one appeal reviewer, who have also been infamously wrong quite often lately 🤷♀️
If those appeal reviewers are where the buck stops for ML learning, the wheels will be falling off this machine real quick 🤦♀️😅
I would like to have this particular repeated ML-rejected example reviewed by a more experienced Niantic staffer (I know, right? Not holding my breath).
If a dedicated forum section is ever created, feel free to move this there 🤷♀️
As a starting point, according to Niantic, picnic tables are eligible submissions. These examples are just picnic tables without any other distinguishing features and they are perfectly good places to explore and socialise. So, without any other info needed, picnic tables should at least be going through to public review.
Nonetheless, ML pulled the submission (confirmed by the email wording).
Assuming it was the fairly flat colour tone of the image that flagged it, I travelled back within range and resubmitted with a more dynamic primary image. This made it through to public voting, was rejected (still common in community review of picnic tables, due to contextless and largely unchallenged "generic" sentiments), and I appealed for a nonsense rejection that didn't even recognise the object itself as anything more than a "regular bench", with "no significance", despite having explained why it's more significant that many (still eligible) picnic tables.
When I was back in an area that's within submission range yesterday, I resubmitted it for a third time (the area is out bսѕh, down dirt tracks, and we're having a 40°c/104°f heatwave and shouldn't be required to be so inconvenienced just to repeatedly resubmit due to bad rejections).
Again, it was pulled by ML (confirmed by email wording).
So, before we even get into proving that this is 100% a significant recreational area, in a significant historical precinct, in a significant tourism hotspot, in a globally significant natural environment/habitat... We can first agree that picnic tables are eligible nominations that don't require much more than that to be good places to explore and socialise; so long as the other parts of the nomination meet criteria (photos, location, etc).
Need I even provide anything more than that to prove acceptability? I did in my nominations and it seemed relatively pointless, even when it was reviewed by humans. Nonetheless, for more Niantic review purposes, here's just some evidence of the reasons this nomination well exceeds the minimum requirements for acceptability:
I could post all day about the multi-level significance of this site (there's an enormous web presence, which is unusual for many of our local assets)... But a picnic table is suddenly just a "regular bench", it's use for fishing (exercise) and picnicking (socialising) worthless, and everything else about the area it placemarks is for sh!t, according to uneducated ML and some completely unaccountable nameless/faceless appeal reviewer, apparently; and there's no further checks or balances in place to improve the system? 🤦♀️🤦♀️
@thenamelesskath-PGO
Its a subject that the Ambassadors raise .
Machine Learning -ML- Emily (eM L whY are you doing that 😉) as we have been calling it on this forum, is such a mix of great ……stop the very poor nominations to bad….. rejecting some great things.
Using the appeals system is totally wrong in my view. Appeals are a very precious resource.
If you want to get past Emily you need to make changes. Otherwise the only learning that Emily has done is that they were correct to reject so they will be correct again, and this just reinforces the learning.
its a hard nut to crack - to get the balance between rejecting very poor ones that we want it to do (keeps queues shorter and therefore a better wayfinder experience) and allowing appropriate ones to go forward for review.
Ambos are working on it.
Oh haha, I'd noticed, but now I see what you did there 👌😂
I appreciate that you guys are working on it and I'm certainly aware that it's an issue that's regularly been raised. I do think more recourse needs to be in place for these ASAP, though, and a forum section could be a very appreciated useful stop-gap.
While conscious of the fact that I don't see what legitimate coal is being filtered, I am also aware of the vast swathes of casual submitters who won't raise the issue here (many don't even realise it's why they've had rejections), and that I'm personally still seeing about the same amount of mostly the same types of coal for review as before. In fact, the general quality (text, photos, location pin, etc) of even otherwise eligible submissions I'm reviewing has significantly dropped in the last few months, which suggests to me that many of the more experienced submitters have bailed altogether (admittedly, I took a break from submitting for a while, myself, after all the changes and issues they brought) 😣
I can only hope things improve and failsafes are put in place, but I think Niantic does need to be more mindful and respectful of the extra frustration this lack of recourse is causing and the extra effort (and cost) they're asking us to go to in order to get around the issues, for now 🤷♀️
Unlimited appeals would defeat Niantic’s purpose of using the AI in the first place.
They want to reduce their workload and be as hands off as possible with Wayfarer.
Making staff interact with us MORE is not something they would entertain doing.
I know what you're saying, but if they wanted to do that, they could have just as easily left it as it was and not bothered with ML at all 🤷♀️
Unfortunately ML requires time to actually learn how it's expected to behave and that won't be effective unless they're hands on while establishing that, and the more input, the faster that process (you've gotta spend time to save time).
I wasn't suggesting unlimited appeals, period, I was suggesting unlimited appeals only on ML-rejections and only until ML behaviour is more refined. Most people aren't stupid enough to appeal legitimate coal or dodgy submissions, but if it were really a concern, ML appeals could come with a warning about penalties for doing so 🤷♀️
In any case, given Niantic's track record and penchant for years of lip service, I won't hold my breath for much improvement any time in the foreseeable future and I'll just assume the answer is the same tired old "soon" (TM) 🤷♀️
Niantic is perfectly happy to let the AI reject whatever, even legitimate stops. Those are fewer things they have to concern themselves with. I suspect they aren’t really interested in making it more effective.
They just want to interact with Wayfarer as little as possible.
Unlimited appeals would defeat the purpose of using AI.
The Machine Learning that Niantic is using, at the very least in my personal opinion, should only be targeting absolute coal and strictly ineligible candidates. The biggest worry many of the community holds is that it is self feeding based on poor decisions and creating confirmation bias towards what it rejects. Meanwhile, I still see blatant screenshots, watermarks, k-12 schools, and private residences slip through.
To the point of your comments, do you think the bulk of actual ineligible Wayspots people submit (ie the screenshots, houses, etc) would actually be appealed? Or do you think it would mostly be trail markers, picnic tables, and religious buildings as we mostly see shared here?
More appeals of any sort = more work for Niantic.
That is something they are clearly trying to move away from
I think unlimited free appeals are free appeals, and people would mostly appeal everything with "appeal" button, and maybe would also trigger AI intentionally to have quick decisions.
On the other hand, if players use their "regular" appeals and only successfully accepted appeals are "returned", that would 1) motivate players to appeal only submissions which should be accepted; 2) motivate niantic to tune AI to reject strictly obvious garbage.
The biggest worry many of the community holds is that it is self feeding based on poor decisions and creating confirmation bias towards what it rejects
Is there an evidence that AI/ML is "self feeding" and not being spoon-fed (i.e. does not receive both positive and negative processed appeals for example)? I mean, self feeding looks counter-productive, and even in the announcement the process was described as AI/ML trained on pre-screened data.
And who is spoon feeding that data? The very appeals reviewers who have rejected "generic" picnic tables and accepted a dentist office and a Men's Warehouse store in my city?
I think we have different opinions on who would post these types of appeals. I'm not advocating for unlimited community rejections, but the ML 1) defeats the purpose of a community built database and 2) harms prolific submitters the most. In my opinion, the casual submitter who nominates coal as I mentioned above wouldn't be aware of the appeals process enough to take advantage of them. The ML can be used strictly as a deterrent, not a roadblock.
Not saying they aren't - but it is coming at a cost of faith from many of the community. And to create & implement this clearly took a lot of time and effort and they should maximize returns on it. I do think better options can exist than the current appeals system.
I'm glad you have a different opinion on it.
Niantic has clearly made the calculation that legitimate nominations will be rejected by their system and they are comfortable with whatever that percentage is.
This is what is so baffling.
From the very outset, Niantic's algorithm should have included detection of:
Watermarks
Third-party photos
Zoning and property issues
Auto interiors, windshields, mirrors
Fingers
It still requires hand-holding to prevent it behaving like an anti-nature Terminator. If we must use appeals to get our art and nature submissions through, we ought to get those appeals back.
Especially considering the fact that the Appeals responses often look like THEY are coming from the ML.
Sure, real people are still going to use boiler-plates to lighten their work, but they really shouldn't ought to be saying things like "Your custom-built picnic table at a scenic overlook is a regular bench." or "That cairn on a mountain-top is just a normal marker."
If that is human output, it is slipshod in the extreme, revealing hostility to the entire Wayfarer project that is so severe that Human Resources should be called in. You hate your job, you hate your boss and the company for making you do these reviews, you hate the people who keep submitting this stuff? You're terminated.
It's much more likely that Emily was rolled out too soon and doesn't have proper margins to correct itself.
I don't see any reason why we should have to 'pay' for this.
I don't disagree that it's probably a large part of the motivation to implement ML, but it can't be the only reason to do so, because the same could have been accomplished by leaving it as it was and ignoring the community complaints about coal and wait times for resolutions 🤷♀️ It was meant to be a QOL improvement for the Wayfarer community and boost productivity of the system, but if the time isn't taken to make it function as intended, then it could potentially do more harm than good; particularly to the quality of their precious Lightship database.
There is potential for some of that, I guess, and it was just one possible suggestion. However, I'm not sure how anybody would intentionally trigger ML in any way that wouldn't get an appeal rejected, since part of the issue is the complete lack of feedback as to what even triggers it, and most things that might would be clear quality issues or codeword abuses, etc, if used for this purpose 🤷♀️
I agree with the idea of having successful appeals returned and I've said the same for a long time. On the flip side, you seem to have too much faith in appeal reviewers to be 100% black and white accurate and omniscient as far as global cultures and nuance, which they certainly aren't. Even in situations where we do, in fact, have fairly universal black and white answers for acceptability, many of those examples of "submissions that should be accepted" are currently being rejected on appeal for stupid, illogical, and often completely non-existent reasons. This itself has been a problem, so I wouldn't be relying on that system for any real measure of accuracy or authority on the matter 😬
I couldn't agree more and I think most genuine Wayfarers are so fearful of repercussions for even silly things that shouldn't invoke them (eg: genuine valid resubmissions), that any remotely shady behaviour that might get direct Niantic attention from an appeal is much less likely to be a common issue than some would suggest 🤷♀️
I have no problem with ML as a tool in the process, but only if it's going to work with the community to create the best quality database, rather than against us with extreme prejudice and little recourse when it gets it wrong.
I think you've hit the nail on the head suggesting it should only be rejecting absolute coal and ineligible submissions. I'd also go so far as to suggest it shouldn't be working in grey areas at all, because even "generic" businesses and PRP would likely be a problem in this space. I'd rather the opportunity to review these to avoid ML blanket rejecting potentially eligible candidates where there's often global variables, unpredictable exceptions, and inaccurate data available for it to base predictions on (because that's all ML really does, based on the data it's given) 🤷♀️
I am encouraged to know that such insightful, well-spoken, and passionate community ambassadors could potentially affect the way Niantic considers the issues to implement ML in the most effective and least intrusive way possible; I just hope they're really listening to what you guys have to say and using all that invaluable knowledge and insight to best deal with ML so that it might more adeptly walk the line between saving them manpower and allowing the community the opportunity to review local submissions holistically, as intended 🤞🏻
Unfortunately for me, I've had stops that I submitted that were obviously publicly accessible and not on or in a private residence rejected as being in a private residence or farm (it was a community flower garden with a multifaceted abutting large welcome sign, with the description identifying what it was). I've also had a flagpole at a community parking location with multiple pathways and sidewalks leading to the garden and the community lake and lake bridge (all mentioned in the description) rejected as being an offensive image and description, presumably because it had the USA flag on it. Between the ML and the human reviewers, I don't know what to do. There's poor connection in that area, so any submissions typically take multiple attempts just due to cloudiness, etc. Only being able to appeal one every 30 days when the nominations get rejected by seemingly 1,000 people attempting to write Shakespeare on typewriters... The question has to be asked if it is worth it.
Well said and good points made.
i think the appeals doesn't have to be unlimited but certainly higher than one. maybe 5 a month?
I'd settle for 4 a month 🤷♀️
However, the ML issue is completely separate and needs separate recourse; even if an ML appeal only means human eyes for 30 seconds to decide if it is truly absolute coal or ineligible (i.e: banned) and to throw anything that's not back into queue for community voting. As others have said, we shouldn't be paying for those mistakes or using our community rejection appeals on inaccurate ML predictions 🤷♀️
How about this. If the ML's Confidence Level is less than like 95%, it goes to 5 human reviewers. If 4 or 5 of them agree with the rejection, it is rejected. Otherwise it's converted to a "regular" vote (whatever that structure is), with the ML rejection as 1 vote against. Maybe 2.
The ML could be improved by seeing what human reviewers think of their rejections.
I know this would require coding, so is unlikely to get the investment. Also I don't like anything that could put us back to a year+ wait for un-upgraded nominations. But I was playing "what if" - what if we had a hybrid? Could ML and Wayspot results be better?
The NBA instituted a rule change whereby teams can 'challenge' results of a play call. If their challenge is successful because the referees made a mistake, their challenge is returned to them and they can challenge again, with a limit of two challenges per game.
It may be really nice to implement a similar system, whereby instead of one challenge per month users are able to have this limit increased by one or two to account for the fact that Niantic's AI bot isn't actually doing learning and is terrible and bad.
They're interesting thoughts 🤔
I will just throw it out there again, that it is the hardest working high volume submitters that will be penalised most, so any version of the system that significantly limits the recourse on Emily's rejections before it's properly calibrated will be discouraging some of Wayfarer's best unpaid assets.
For example, I'm limiting my submissions to a handful in a week to month now (for my sanity lol), so a couple "challenges" in a month generally won't be an issue for me. But the powerhouse submitters I know of are often running through all their nominations across two apps, so they're statistically likely to have a higher volume of nominations pulled by Emily, so that the ability to challenge only a couple each month will disadvantage them more.
I personally think we shouldn't be tying the hands of the few Wayfarers who are so consistently passionate and energised to do such huge amounts of unpaid labour, and penalising them more than others, no matter how unintentional, is pretty ill-advised 🤷♀️
I don't expect much better from Niantic, though, TBH 😏
I had an appeal available yesterday so I used it on an Emily rejection. I am close enough that I could have resubmitted, I have one other photo from a different angle but it doesn’t look significantly different so doubt the outcome would be any different from Emily. But I thought I would see how the process felt. I did feel very resentful at having to use a precious appeal.
I have had a result in less than 24 hours.
And it’s a rejection …….so now resentment has turned into annoyance.
So the message is that a wood carving that by its very nature is unique, is normal. 😡
Does it make me want to engage in submitting the apples (the acorns and hazelnuts are in the database) - no.
Does it make me feel valued as a Wayfinder - no.
Does it give the impression that someone took time to review my submission in the same way that the community would have done against the current questions - no.
Do I feel motivated to make the effort to resubmit this and the others like it in order to expand the database - no.
Does it make me feel cross that what Emily has learnt is that they were correct and have reinforced that this wrong decision is right - oh yes I feel cross.
That’s a big 👎 from me on this aspect of the Wayfinder experience.
Wow... A normal wood carving? Bugger me dead! It's so ridiculous it's comical (unless you're on the receiving end, of course) 😅😅
I mean, if they said "it looks like a normal lump of wood" and "a better photo might increase the chance of approval", that might be fair enough, but to acknowledge it is what you say it is, and then claim it doesn't meet criteria is completely ridiculous 🤦♀️
Bad appeals and bad Emily might be two largely separate issues, but it does appear likely they'll cause a vicious cycle if neither is fixed soon 😬