2022-03-10 Server Load

[Haxus]

Server Load
Servers have been very heavily loaded managing 270k solar systems. That happened fast.

All of my attention this week has been on this problem.

I made a significant change that has potential to cause problems. I am monitoring the servers closely.

I made database backups and backups of the current running server software. This change can be undone completely if it's a total disaster.

[Seathre]

Crossing fingers and knocking on wood in hopes of no issues!

[Athena]

Why don't systems that aren't in use simply get unloaded until someone enters them, or if loading times are a problem, enters a neighboring system? Even decay wouldn't be necessary if systems and cities that entered the decaying state just stopped being loaded.

[HankHelium]

Please don't simulate 270 thousand systems consecutively when not in use. I know you hate shortcuts but culling exists for a reason.

[martianant]

I feel like new style cities have room to be optimized for processing power. With the introduction of per building based inventory, now processes have to check every building's inventory across the entire system which can be many hundreds. Before with a city wide inventory that was less and at most the same number of planets in the system * 3. Additionally, each process could be drastically simplified.
For example, cities generally reach a sort of equilibrium after some amount of time provided a player isn't actively fiddling with it. Once that equilibrium is reached processes could be simplified. Treat processes as a +X resource per Y time (city cycle?). So instead of calculating 500 smelting processes each, just try to add 5000 metal to the inventory. If nothing changed between last city cycle and now, no need to check everything again. I feel like most cities will be in this equilibrium state so I can only imagine how drastically this would help with server load.

Also, 270k systems? There is no way they all need to be loaded and calculated right? There are 414 avatars and 110 officers. If we assume all are in separate systems, that's only 524 systems with something in them. Why are the rest loaded and taking up processing time? We must be misunderstanding what you meant by "managing" 270k systems...

[Trenial]

First restart today, we came back and everything was fine. This recent restart, and now its back to horrendous.

[zephar]

Wierd server started up good. Now it seems that my systems are disapearing. The system my ship was in wont respond and I am now stuck on the starter planet.

[Haxus]

First restart today there were no changes. While waiting for the servers to shut down, I was testing on the debug servers. There were serious problems so I just restarted the servers without updating them.

Server load is definitely better now.

Tomorrow I will go after the various issues that have been reported.

[Zensras]

Seems to be a lot of random limboing happening as well.


One thing I've noticed: One avatar limbos, my other one gets limbo'd as well.

[Haxus]

The number of worker servers increased from 9 to 17.

Eight of the servers, the R815s and R820s, have four ethernet ports. On each of those, I split the CPU cores between two server processes. Each of them uses a different ethernet port to divide up the traffic.

This was an incidental change. I don't know if it had much affect on the server load. We'll see.