Fatal Error: Steamcmd needs to be online to update.
Sylver-Dragon opened this issue · comments
I am attempting to create a dedicated game server based on the cm2network/steamcmd docker container. I have attempted to use both the "latest" and "root" tagged versions. The container images were fetched from dockerhub via docker pull cm2network/steamcmd:latest and docker pull cm2network/steamcmd:root.
Environment
Host OS: Ubuntu 18.04.5 LTS (Bionic Beaver)
docker version
Client:
Version: 19.03.6
API version: 1.40
Go version: go1.12.17
Git commit: 369ce74a3c
Built: Fri Dec 18 12:21:44 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.6
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: 369ce74a3c
Built: Thu Dec 10 13:23:49 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.3-0ubuntu1~18.04.4
GitCommit:
runc:
Version: spec: 1.0.1-dev
GitCommit:
docker-init:
Version: 0.18.0
GitCommit:
Problem Description
Then container is started via:
docker run -it --rm --name=steamcmd cm2network/steamcmd:root /bin/bash
Once inside the container I attempt to just run:
./steamcmd.sh
This results in the output:
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/root/Steam/logs/stderr.txt'
[ 0%] Checking for available updates...
[----] Downloading update (0 of 16258 KB)...
[ 0%] Downloading update (0 of 16258 KB)...
[ 0%] Downloading update (0 of 16258 KB)...
[ 0%] Downloading update (404 of 16258 KB)...
[ 2%] Downloading update (0 of 16258 KB)...
[----] Verifying installation...
[ 0%] Downloading update...
[ 0%] Checking for available updates...
[----] Downloading update (0 of 16258 KB)...
[ 0%] Downloading update (0 of 16258 KB)...
[ 0%] Downloading update (0 of 16258 KB)...
[ 0%] Downloading update (73 of 16258 KB)...
[ 0%] Downloading update (683 of 16258 KB)...
[ 4%] Downloading update (0 of 16258 KB)...
[ 0%] !!! Fatal Error: Steamcmd needs to be online to update. Please confirm your network connection and try again.
src/tier0/threadtools.cpp (3553) : Assertion Failed: Illegal termination of worker thread 'Thread(0x0x588388f0/0x0xf7c27b'
This is consistent if I use either the 'root' or 'latest' tagged image. Capturing network traffic, it appears that the update attempt
starts but is ultimately reset by the server (see attached pcap file steamcmd.zip).
Expected result
steamcmd update completes successfully.
As best I can tell, the container is online (the download does start); but, it is stopped before it completes. Because the update fails, I am unable to download any sort of application to begin the process of standing up a server.
Have you verified that doing anything network-related inside containers on that host works? Have you excluded that the machine's docker network can connect & resolve external hosts?
Easy test-case; Use debian:latest and try to curl github.com, does that work?
Yes, the same host has sever containers which host services on the internet. Also, using the 'root' tagged image, I installed iputils-ping so that I could try a simple ping out to the internet.
root@64d970a62cf9:/home/steam# ping media.steampowered.com
PING a1843.b.akamai.net (23.221.227.32) 56(84) bytes of data.
64 bytes from a23-221-227-32.deploy.static.akamaitechnologies.com (23.221.227.32): icmp_seq=1 ttl=52 time=16.3 ms
64 bytes from a23-221-227-32.deploy.static.akamaitechnologies.com (23.221.227.32): icmp_seq=2 ttl=52 time=17.9 ms
64 bytes from a23-221-227-32.deploy.static.akamaitechnologies.com (23.221.227.32): icmp_seq=3 ttl=52 time=17.5 ms
64 bytes from a23-221-227-32.deploy.static.akamaitechnologies.com (23.221.227.32): icmp_seq=4 ttl=52 time=14.4 ms
^C
--- a1843.b.akamai.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 14.379/16.520/17.876/1.367 ms
This would indicate that the container is both online and able to get DNS resolution. Also, in the pcap I included, I do see some data being downloaded from the that host (media.steampowered.com).
installing and then trying curl https://github.com does result in the main page of github.com being downloaded.
Hmm, thought as much.
Unable to find image 'cm2network/steamcmd:root' locally
root: Pulling from cm2network/steamcmd
45b42c59be33: Already exists
6678b2c9f2da: Pull complete
Digest: sha256:7a647438a26773b88368c3a1b76de426297b15fd4f85a3af7e6e7e96665afadf
Status: Downloaded newer image for cm2network/steamcmd:root
root@ac453fa6ffdf:/home/steam/steamcmd# ./steamcmd.sh
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/root/Steam/logs/stderr.txt'
/tmp/dumps is not owned by us - delete and recreate
[ 0%] Checking for available updates...
[----] Downloading update (0 of 16258 KB)...
[ 0%] Downloading update (2391 of 16258 KB)...
[ 14%] Downloading update (4169 of 16258 KB)...
[ 25%] Downloading update (5397 of 16258 KB)...
[ 33%] Downloading update (6743 of 16258 KB)...
[ 41%] Downloading update (7984 of 16258 KB)...
[ 49%] Downloading update (9181 of 16258 KB)...
[ 56%] Downloading update (10544 of 16258 KB)...
[ 64%] Downloading update (11789 of 16258 KB)...
[ 72%] Downloading update (13021 of 16258 KB)...
[ 80%] Downloading update (14339 of 16258 KB)...
[ 88%] Downloading update (15522 of 16258 KB)...
[ 95%] Downloading update (16258 of 16258 KB)...
[100%] Download complete.
[----] Installing update...
[----] Extracting package...
[----] Extracting package...
[----] Extracting package...
[----] Extracting package...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Installing update...
[----] Cleaning up...
[----] Update complete, launching Steamcmd...
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/root/Steam/logs/stderr.txt'
[ 0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...src/clientdll/applicationmanager.cpp (4255) : Assertion Failed: CApplicationManager::GetMountVolume: invalid index
src/clientdll/applicationmanager.cpp (4255) : Assertion Failed: CApplicationManager::GetMountVolume: invalid index
src/clientdll/applicationmanager.cpp (4420) : Assertion Failed: m_vecInstallBaseFolders.Count() > 0
OK.
Steam>
I'm having trouble replicating this. It works on my (bare-metal) server.
This part of your log is really weird, it seems like it's downloading a piece, then failing then trying to resume? Could it be that it can't write to the underlying disk? (Keep in mind that this steamcmd image always allocates a fresh volume.
[ 0%] Checking for available updates... [----] Downloading update (0 of 16258 KB)... [ 0%] Downloading update (0 of 16258 KB)... [ 0%] Downloading update (0 of 16258 KB)... [ 0%] Downloading update (404 of 16258 KB)... [ 2%] Downloading update (0 of 16258 KB)... [----] Verifying installation... [ 0%] Downloading update... [ 0%] Checking for available updates... [----] Downloading update (0 of 16258 KB)... [ 0%] Downloading update (0 of 16258 KB)... [ 0%] Downloading update (0 of 16258 KB)... [ 0%] Downloading update (73 of 16258 KB)... [ 0%] Downloading update (683 of 16258 KB)... [ 4%] Downloading update (0 of 16258 KB)... [ 0%] !!! Fatal Error: Steamcmd needs to be online to update. Please confirm your network connection and try again. src/tier0/threadtools.cpp (3553) : Assertion Failed: Illegal termination of worker thread 'Thread(0x0x588388f0/0x0xf7c27b'
I'm really confused...
EDIT:
I know this thread is a bit outdated and is referring to desktop Steam but maybe see if this solution works for you:
It at least seems to point to some sort of routing issue.
I also just noticed that you shouldn't run the steamcmd.sh directly using the root image. Can you try to switch to the steam user first and the execute ./steamcmd.sh again?
su steam
./steamcmd.sh
I think you are right that is has something to do with writing to the container's volume. Running as the user steam ran into the same issue. However, I was able to get steamcmd.sh to run successfully by circumventing the update process. From looking at the pcap file, it seems that the update process first downloads the file http://media.steampowered.com/client/steam_cmd_linux. Then based on the information on that file, it downloads updates as zipvz files (at least, that's what's listed in the manifest).
So, I spun up a fresh container using docker run -it --rm --name=steamcmd cm2network/steamcmd:root /bin/bash, ran su steam and then curl those files into /home/steam/steamcmd/package/. After the downloads finished, I ran /home/steam/steamcmd/steamcmd.sh, which extracted and installed the updates successfully and ended up in the steamcmd interface.
For good measure, I also tried downloading a game sever:
/home/steam/steamcmd/steamcmd.sh +login anonymous +force_install_dir /home/steam/valheimserver +app_update 896660 validate +quit
This ran successfully.
As a possible direction for me to investigate, when the normal update runs, does it place the downloaded files directly in /home/steam/steamcmd/package/ or, do they get cached somewhere while downloading?
This issue appears unique to me, and I have a valid workaround. I appreciate the help and would be willing to troubleshoot further, if you feel it's warranted. Otherwise, please close out the issue and I'll just use the workaround.
I just noticed your edit with the link to the other thread. Unfortunately, it didn't seem to work for me. media.steampowered.com resolves to one of two addresses:
root@7c4067885110:/home/steam/steamcmd# nslookup media.steampowered.com
Server: 1.0.0.1
Address: 1.0.0.1#53
Non-authoritative answer:
media.steampowered.com canonical name = cdn.akamai.steamstatic.com.edgesuite.net.
cdn.akamai.steamstatic.com.edgesuite.net canonical name = a1843.b.akamai.net.
Name: a1843.b.akamai.net
Address: 23.221.227.43
Name: a1843.b.akamai.net
Address: 23.221.227.32
Using the hosts file to force the client to use one or the other results in no change in behavior.
This issue appears unique to me, and I have a valid workaround. I appreciate the help and would be willing to troubleshoot further, if you feel it's warranted. Otherwise, please close out the issue and I'll just use the workaround.
I'll leave this open for now, maybe someone will have the same issue in the future. It seems to be quiet specific.
Hoping to bring some closure on this. I pulled the latest docker image for cm2network/steamcmd:latest this afternoon and tried the exact same process. Today the steamcmd.sh ran through the update process without error. Whatever was causing the issues seems to be gone for now.