2022-02-03 Server Status Update

[Haxus]

The final rack server has a problem. The network interface is not working. They are sending a new NIC; that should arrive on Tuesday.

Things are coming together. It still looks possible to have this thing back online by the end of this month. That is my goal.

[Rockinsince87]

Total power consumption by the rack is now just over 10kW. That emits about 34k BTUs per hour.

With only five of the PE1950s turned on, the room quickly warmed up to 95F and their fans were screaming like hair dryers.

This forced me to install a CRAC unit (computer room air conditioner) at a cost of $7,500. It is mostly installed and it should be ready to run on Monday.

The 10kW rack with the 10kW CRAC will eat up about 14,400kW hours of electricity per month.
Electricity around here goes for about $.10 per kW hour, for a monthly electric bill that could exceed $1400 per month.

Dang. Yeah that's the problem with the PE1950s. They are very power hungry. We had a large number of IBM x260 servers, 7U and 4 power supplies each. I hated them. They consumed a monstrous amount of power. Standing behind those racks was brutal. After we upgraded those to newer servers after we finally migrated away from the solution running on them , we cut power consumption in half while gaining more than 10x the compute power at 1/4 the foot print.  

I would encourage slowly replacing the PE1950s at a rate your comfortable with over time. It should help with the power consumption long term. 

Nice call on the CRAC and it sounds like you got a decent deal actually.

The final rack server has a problem. The network interface is not working. They are sending a new NIC; that should arrive on Tuesday.

Things are coming together. It still looks possible to have this thing back online by the end of this month. That is my goal.

Awesome! Very exciting news!

[resonkinetic]

The final rack server has a problem. The network interface is not working. They are sending a new NIC; that should arrive on Tuesday.

Things are coming together. It still looks possible to have this thing back online by the end of this month. That is my goal.

Just in time to for me to cope with undesirable circumstances I've created for myself in real life.

[Haxus]

Web server is now moved over to the new servers.

Scene servers are up. They are not open to the public yet.

[jakbruce2012]

I saw "Preparing the depths of space." and got maybe a little too excited.
Have the servers been behaving well so far?

[Greydog]

Does this update Starship as well?

[Haxus]

Have the servers been behaving well so far?

Servers are coming together nicely.

I received the new NIC. It installed easily and works fine.

One of the other new R820s reported a failing DIMM. I am currently waiting for a replacement. Then that worker server will join the crowd. It is currently off.

The R815s were reporting only 96GB of memory. They all have 128GB. This led me to a BIOS setting that caused them to reserve 25% of their memory as spare, to hot replace any memory that might fail. wtf

This drove me crazy for a while until I discovered the BIOS setting. I don't recall ever messing with that setting though I do recall one of those servers persistently only reporting 96GB of memory, once, years ago. I don't recall the solution back then.

I reconfigured each of the R815s to not reserve memory for spare. They all report 128GB of memory now. Yippee. It's the little things in life that keep it spicy.

The CRAC unit is working great, keeping my little computer room a nice comfortable 72F. This is much better than sweating while standing in front of the keyboard and monitor in a tiny 95F room. It may have been the first time in all my career that I perspired in a computer room. Usually you're wearing a sweater or hoodie, maybe gloves if you have to sit and work for a long time.

When the room was very hot, the servers were screaming like hair dryers. The sound was quite audible outside the room. I got nervous looks and a few queries from the people in the neighboring offices.

We installed sound proofing material on two of the walls, to placate the neighbors. We used 1lb Mass Loaded Vinyl, which is designed to stop sound. It is very heavy. At 1lb per square foot, a 25'x4' roll weighs 100lb. It is easy to work with. It is intended to be applied beneath the drywall but it works hanging on the wall too, just looks ugly, like tar paper.

Total site prep cost was about $13,000.

[Haxus]

Does this update Starship as well?

No updates have been posted.

When I do post an update, it will likely include an update to Starship. All the Hazeron client programs build side-by-side. I routinely switch between Starship and SoH when testing my changes.

[Haxus]

I saw "Preparing the depths of space." and got maybe a little too excited.

I got kind of excited too. I'm a little curious why no other messages appeared. It may be because the game universe is almost empty. It only contains 21 galaxies and each of them is empty of sectors. I have not tried logging in yet which will create a bunch of sectors when my home world is created.

There was a peculiar message when the master server initialized the universe for the first time. It doesn't happen very often. I may take a few minutes to find out what went wrong the first time. The symptom is that the galaxies didn't get created or didn't get stored in the db.

The problem corrected itself when I restarted the server. The missing galaxies were created and stored properly. That's where the servers were left last.

[Greydog]

Does this update Starship as well?

No updates have been posted.

When I do post an update, it will likely include an update to Starship. All the Hazeron client programs build side-by-side. I routinely switch between Starship and SoH when testing my changes.

I though as much, I only ask because there has been a strange bug show up since last night. 


https://discordapp.com/channels/22645666...1054167040