Why doesn't Lightship sync continuously?

Hosette-INGHosette-ING Posts: 3,150 ✭✭✭✭✭

My $DAYJOB is production operations for a large well-known website... if you're reading this it's almost certain that you have used the services I work on. I was meandering through the forum tonight and thinking about Lightship syncing and I found myself wondering why they're doing daily batch syncs to the games.

The Lightship database is pretty small... I just estimated based on the Ingress mission authoring tool and I think that there are something like 15 million portals in the world, though that is far from a precise count. The Lightship database is certainly larger than that as it includes things that don't appear in Ingress, but how much larger? Is it 20 million? I doubt it's more than 50 million, which is a tiny database by modern standards. How many new wayspots are added every day? According to the 2021 Playback there were 1.95M added in 2021, along with 679K edits and 45K reports. That's, oh, let's round up to 2.7 million changes, around 7400 per day, or just over 5 per second. It's probably higher than that in 2022 but I doubt it's more than double. That's a tiny number, just the slow **** of a faucet. The technology for streaming updates at this rate is well-established and inexpensive... there's no need to reinvent the wheel, just pick components off the shelf and add a little bit of glue logic. (For comparison, the systems I work with add several zeros to that number and they do it in near-realtime with a failure rate that's very close to zero.)

It makes me wonder... why is Niantic doing this in such an archaic way? They have access to the massive pool of silicon valley tech workers, and there are thousands of engineers in the valley who could implement continuous sync. I'm sure Niantic has multiple engineers in-house who could do it. Why haven't they? What advantage is there to using batch syncs?


Sign In or Register to comment.