Why doesn't Lightship sync continuously?
Hosette-ING
Posts: 3,150 ✭✭✭✭✭
My $DAYJOB is production operations for a large well-known website... if you're reading this it's almost certain that you have used the services I work on. I was meandering through the forum tonight and thinking about Lightship syncing and I found myself wondering why they're doing daily batch syncs to the games.
The Lightship database is pretty small... I just estimated based on the Ingress mission authoring tool and I think that there are something like 15 million portals in the world, though that is far from a precise count. The Lightship database is certainly larger than that as it includes things that don't appear in Ingress, but how much larger? Is it 20 million? I doubt it's more than 50 million, which is a tiny database by modern standards. How many new wayspots are added every day? According to the 2021 Playback there were 1.95M added in 2021, along with 679K edits and 45K reports. That's, oh, let's round up to 2.7 million changes, around 7400 per day, or just over 5 per second. It's probably higher than that in 2022 but I doubt it's more than double. That's a tiny number, just the slow **** of a faucet. The technology for streaming updates at this rate is well-established and inexpensive... there's no need to reinvent the wheel, just pick components off the shelf and add a little bit of glue logic. (For comparison, the systems I work with add several zeros to that number and they do it in near-realtime with a failure rate that's very close to zero.)
It makes me wonder... why is Niantic doing this in such an archaic way? They have access to the massive pool of silicon valley tech workers, and there are thousands of engineers in the valley who could implement continuous sync. I'm sure Niantic has multiple engineers in-house who could do it. Why haven't they? What advantage is there to using batch syncs?
Comments
Is it less costly for them to do daily syncs? I do miss getting an email about an approval and being able to instantly see it on the Ingress Scanner.
There are so many things that Niantic isn't doing correctly and nobody outside can answer why.
Why didn't take action to fix the "daily appeals bug" one day after there were the first reports about them not being monthly?. That's something that doesn't take any engineer to notice, just someone reading the forums and then sending a message: Hey, stop it, it's not working as expected.
If they are unable to do that, continous sync might sound like magic to them.
I agree to this. It even took them several years to notice that gym creation in PoGo worked in a way that was never supposed to be (portal with most photos+upvotes => new gym). Same for the appeal bug, same for many ingame bugs that have been around literally since release.
What surprises me even more: I would understand such things if they happened in the same way through all the games, but they don't. Portals in Ingress are added immediately, while there are specific times in PoGo and there were, if I amm not wrong, even different times in HPWU. That is the thing that makes least sense..
What surprises me even more: I would understand such things if they happened in the same way through all the games, but they don't. Portals in Ingress are added immediately, while there are specific times in PoGo and there were, if I amm not wrong, even different times in HPWU. That is the thing that makes least sense..
Since the Lightship go-live, Ingress portals sync in a similar fashion to Pokemon Go pokestops/gyms, usually about four hours after the Pokemon Go POI appears. Since both sync times occur during my working hours, it saddens me that I will never again experience the magic of watching one of my submitted POI suddenly blink into existence while I'm out playing the game. A continuous sync would restore some of that fun to the process!
The real question is why did they disable the appeal button when it less code adjusting was needed to change the appeal time from 1 day to 30 days than remove the option...
And I totally agree with OP. If you know a bit out large scale database distribution, CDNs, load balancing etc, it makes no sense how it's been done now compared to how it was.
Im guessing someone at Niantic during a meeting brought up limits of how it was done at the time and suggested this huge project that required many staff under them to complete and be future proof, throwing in many marketing terms the boss's like to hear. When in fact only a few small changes really needed to be made to the workflow to support other games, now and future, with all their individual needs.
The fact there appears to be no monitoring at all of any of the process's involved worries me too. Nothing to alarm people if a sync of any kind fails. Im Sure if there is network monitoring it's nothing more than than a simple ping watch to see if it fails, rather than watching latency or SNMP monitoring.
With Niantic it always seems the bare minimum is done.
How many times they **** up Pokémon go events is beyond belief too. So many makeup days because they forgot to active the shiny they said would happen for that event or whatever. The amount of mistakes is phenomenal.
So I think the issue is deeper than just the unnecessary syncing when it comes to the IT and infrastructure there.
I would assume that it was because of all of the "Day One" abusers that took advantage of the bug and clogged up the entire system, and it will take time to clear it out. Thus, it wasn't disabled to fix the bug but to allow Niantic staff to catch up. Of course, this bug would have been addressed sooner had Niantic not delayed in reviewing the appeals for the first three months.
Remember Niantic’s motto is if we can create code and make totally bizarre decisions, especially those that don’t advantage players, then that is the way we will go.
Because they only have enough backup tape cartridges to do one backup per day, and the guy who swaps the tapes only pops in once per day at 17.30 PST? They can only do the synch after the tape finishes?
(Then again, perhaps I am thinking computing processes in the days when I used to try to sell people "massive" 750Mb (Yes - Mb) disc drives that were the size of a large washing machine for "only" $50,000 each........... 🤣)
I had four appeals in before they shut them down, which is more than I would have had if the process was working as intended - but I don't consider myself an abuser. My first appeal was in good faith on day one. The second to test the "hey we get one a day not one a month" rumor I'd seen here. The other two appeals were only submitted after NUMEROUS attempts to elicit a response from Niantic about whether it was a bug or intended functionality. They had every opportunity to just say "hey, it's a bug, please don't submit more than one per month till we get it fixed" and I know I would not have been the only one to comply with that request. Some of that backlog was caused by Niantic inactivity and poor communication, not willful abuse.
Actually, that method of Gym creation sounds like it works exactly how it was designed to work, it's a nice simple way of working out which Stop should become a Gym on a "random" basis. What Niantic didn't do was figure in the fact the community would work out how the process was controlled, then use it to game the system.
I can't remember who said this, but it was somewhere mentioned that the instant syncs to Ingress (which were implemented I think around 11/11/2021) were changed because continuously obtaining data and changes from the Lightship database were considered a source of the Ingress latency lag (i.e. deploying resonators or recharging).
Now that Ingress is on a daily sync, look at how effective it is now. Still laggy as hell.
@HaramDingo-ING That's either a ridiculous explanation or a ridiculous design.
Or both. My money is on both.
I agree to this. It even took them several years to notice that gym creation in PoGo worked in a way that was never supposed to be (portal with most photos+upvotes => new gym). Same for the appeal bug, same for many ingame bugs that have been around literally since release.
@Juleswtal-PGO does it not work like that anymore? If not, is it known how it works now?
I wonder why the 2021 Playback says "1.95M Wayspots approved" here: https://community.wayfarer.nianticlabs.com/discussion/27079/2021-playback-and-roadmap-update
but Niantic's blog says says "3,843,336 new wayspots added in 2021" here: https://nianticlabs.com/blog/niantic-recap-2021/?hl=en
And yeah, either way, that's not many transactions by today's standards. UNLESS maybe because the data corresponds to maps... Google rolls out changes in batches.
Even if it's the lower 1.95M in 2021 - when you add in other years, and seed sources (Post offices, libraries, lots of history databases)... I feel like it would be enough to feed machine language and create Artificial Intelligence. The AI could then pre-review nominations to resolve the obvious yes or no, kicking out anything where it has less than, say, 70% confidence.
The problem with AI review is that its susceptible to being manipulated.
It's not hard to imagine people reverse-engineering the AI parameters and submitting things that push all its buttons.
Some of it was probably the 4square imports in India https://community.wayfarer.nianticlabs.com/discussion/26534/new-wayspots-in-india#latestand the imports in Turkey https://community.wayfarer.nianticlabs.com/discussion/comment/109916 and then there was the bug when pikmin bloom went live which resulted in a net increase in pokestops
I'm pretty sure niantic just has one programmer on staff dedicated to wayfarer and they're too busy plugging holes to actually improve anything
Is it Giffard?
@ftzzghzuhdidu-ING No, it isn't Giffard. I know one of the engineers.
You should ask them why Lightship doesn't sync continuously 😂
(-:
@tehstone-ING I could, but I generally don't want to bring work into our discussions.
Have you thought about asking this on the Ingress community forum?
Apologies for lack of knowing how it should be set up, but do you think Lightship pushes on a schedule or Ingress pulls on a schedule? I would have written this off as an Ingress decision rather than Wayfarer one, but push/pull it "makes sense" to me that Lightship sends a daily sync package rather than waits to be asked.
However, to my understanding PoGO sync time changed with DST but Ingress didn't, which implies a game-side decision rather than Lightship arbitrarily treating the schedule for one game differently.
So it's that kind of relation rrrrrrr
@Gendgi-PGO An excellent observation. I've never paid attention to sync times.
It's a Niantic decision in terms of defining the architecture and I have no insight into whether it is push or pull. In a perfect world, any scheduling would be done in UTC.