Different node ip can't start k3s

[steveb05]

Hi, I am experiencing a problem with k3s v1.25.5+k3s1 and embedded etcd on single node. Currently we are using a second provider than the one that provides us the dedicated server for protection from ddos attacks, where they give us a protected ip, and to make everything work we have a service on our server that uses route tables to redirect the traffic of the whole dedicated server (0.0.0.0/1) through this ip. The problem is that k3s fails to start, looking at the logs I think it is because it cannot connect to the etcd datastore.

k3s logs:

Feb 10 21:47:53 GGameServer-H1 k3s[1362]: {"level":"info","ts":"2023-02-10T21:47:53.914+0100","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes":10473472,"current-db-size":"10 MB","current-db-size-in-use-bytes":7360512,"current-db-size-in-use":"7.4 MB"}
Feb 10 21:47:53 GGameServer-H1 k3s[1362]: {"level":"info","ts":"2023-02-10T21:47:53.951+0100","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes-diff":-3371008,"current-db-size-bytes":7102464,"current-db-size":"7.1 MB","current-db-size-in-use-bytes-diff":-266240,"current-db-size-in-use-bytes":7094272,"current-db-size-in-use":"7.1 MB","took":"36.896624ms"}
Feb 10 21:47:53 GGameServer-H1 k3s[1362]: {"level":"info","ts":"2023-02-10T21:47:53.951+0100","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
Feb 10 21:47:53 GGameServer-H1 k3s[1362]: time="2023-02-10T21:47:53+01:00" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [ggameserver-h1-bcffbb90=https://<node-public-ip>:2380], expect: ggameserver-h1-bcffbb90=https://<node-protected-ip>:2380"
Feb 10 21:47:53 GGameServer-H1 k3s[1362]: time="2023-02-10T21:47:53+01:00" level=info msg="Containerd is now running"
Feb 10 21:47:53 GGameServer-H1 k3s[1362]: time="2023-02-10T21:47:53+01:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"

I tried to ask the second provider if they could do anything on their end and they said no. Is there a solution that would work without having to reset the etcd datastore?
Thank you.

[brandond]

Did you just copy the files over from the other node, or what? Etcd stores the member addresses inside the datastore. If you change the server's IP, you need to start K3s with --cluster-reset to reset the etcd cluster membership to the current node name and IP, after which you should be able to start it again normally.

[brandond]

{"level":"warn","ts":"2023-02-15T16:10:13.376+0100","caller":"snap/db.go:88","msg":"failed to find [SNAPSHOT-INDEX].snap.db","snapshot-index":9951035,"snapshot-file-path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/000000000097d73b.snap.db","error":"snap: snapshot file doesn't exist"}

Hmm, it looks like the etcd datastore has been corrupted somehow; critical files are not present on disk. Was the node shut down unexpectedly at some point?

[steveb05]

Now that I think about it there was recently an occasion when the node network interface became unresponsive and we had to reboot it, it usually runs a regular shutdown process, that's the only case I can think of.

[steveb05]

@brandond i was able to check and the node was forcibly turned off (equivalent of long pressing the power button). Is there anything I can do to fix it?

[brandond]

Restore from an etcd snapshot.