Node changed state to OFFLINE without any intervention #953

New issue

Closed

opened 2025-12-28 17:21:17 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:21:17 +00:00

Owner

Originally created by @anikeev-yadro on GitHub (Jan 16, 2023).

Originally assigned to: @carpawell on GitHub.

Node changed state to OFFLINE in period of time from 13.01.2023 to 16.01.2023
I saw only following lines in log:

root@glagoli:/tmp# journalctl |grep -i "online"
Jan 13 10:05:48 tatlinobject kernel: smartpqi 0000:19:00.0: Online Firmware Activation not supported by controller
Jan 13 10:05:49 tatlinobject systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems.
Jan 13 10:05:49 tatlinobject systemd[1]: Starting Remove Stale Online ext4 Metadata Check Snapshots...
Jan 13 10:05:49 tatlinobject systemd[1]: Finished Remove Stale Online ext4 Metadata Check Snapshots.
Jan 13 10:06:02 tatlinobject systemd[1]: Reached target Network is Online.
Jan 13 10:25:00 glagoli neofs-node[17354]: 2023-01-13T10:25:00.840Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "OFFLINE"}
Jan 13 10:29:17 glagoli neofs-node[18166]: 2023-01-13T10:29:17.259Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "OFFLINE"}
Jan 13 10:33:49 glagoli neofs-node[18166]: 2023-01-13T10:33:49.124Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 10:54:06 glagoli neofs-node[18166]: 2023-01-13T10:54:06.011Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 11:17:45 glagoli neofs-node[18166]: 2023-01-13T11:17:45.711Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 11:38:02 glagoli neofs-node[18166]: 2023-01-13T11:38:02.480Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 11:58:19 glagoli neofs-node[18166]: 2023-01-13T11:58:19.263Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 12:18:36 glagoli neofs-node[18166]: 2023-01-13T12:18:36.033Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 12:38:52 glagoli neofs-node[18166]: 2023-01-13T12:38:52.809Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 13:01:49 glagoli neofs-node[18166]: 2023-01-13T13:01:49.599Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 13:26:46 glagoli neofs-node[18166]: 2023-01-13T13:26:46.339Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 14:03:03 glagoli neofs-node[18166]: 2023-01-13T14:03:03.015Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 14:23:19 glagoli neofs-node[18166]: 2023-01-13T14:23:19.743Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 14:43:36 glagoli neofs-node[18166]: 2023-01-13T14:43:36.601Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 15:03:53 glagoli neofs-node[18166]: 2023-01-13T15:03:53.398Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 15:24:21 glagoli neofs-node[18166]: 2023-01-13T15:24:21.030Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 15:44:27 glagoli neofs-node[18166]: 2023-01-13T15:44:27.096Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 13 17:05:38 glagoli neofs-node[18166]: 2023-01-13T17:05:38.067Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "OFFLINE"}
Jan 13 18:30:09 glagoli neofs-node[18166]: 2023-01-13T18:30:09.823Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "OFFLINE"}
Jan 15 03:10:18 glagoli systemd[1]: Starting Online ext4 Metadata Check for All Filesystems...
Jan 15 03:10:18 glagoli systemd[1]: Finished Online ext4 Metadata Check for All Filesystems.

Than I changed state to ``ONLINE```:

Jan 16 06:47:02 glagoli sudo[1809669]:     root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json control set-status --status online
Jan 16 06:52:08 glagoli sudo[1809807]:     root : TTY=pts/0 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json control set-status --status online
Jan 16 06:52:11 glagoli sudo[1809819]:     root : TTY=pts/0 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json control set-status --status online
Jan 16 06:57:40 glagoli neofs-node[1816249]: 2023-01-16T06:57:40.801Z        info        neofs-node/netmap.go:296        initial network state        {"epoch": 313, "state": "online"}
Jan 16 06:57:41 glagoli neofs-node[1816249]: 2023-01-16T06:57:41.813Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 07:12:54 glagoli neofs-node[1816249]: 2023-01-16T07:12:54.194Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 07:33:11 glagoli neofs-node[1816249]: 2023-01-16T07:33:11.124Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 07:53:36 glagoli neofs-node[1816249]: 2023-01-16T07:53:36.772Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 08:13:52 glagoli neofs-node[1816249]: 2023-01-16T08:13:52.495Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 08:34:18 glagoli neofs-node[1816249]: 2023-01-16T08:34:18.746Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 09:37:59 glagoli neofs-node[1882278]: 2023-01-16T09:37:59.949Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "OFFLINE"}
Jan 16 09:56:50 glagoli neofs-node[1882278]: 2023-01-16T09:56:50.977Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 10:17:06 glagoli neofs-node[1882278]: 2023-01-16T10:17:06.710Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 10:37:28 glagoli neofs-node[1882278]: 2023-01-16T10:37:28.176Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 10:57:39 glagoli neofs-node[1882278]: 2023-01-16T10:57:39.450Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}
Jan 16 11:54:53 glagoli neofs-node[1941679]: 2023-01-16T11:54:53.095Z        info        neofs-node/netmap.go:296        initial network state        {"epoch": 337, "state": "online"}
Jan 16 11:55:23 glagoli neofs-node[1941679]: 2023-01-16T11:55:23.791Z        info        neofs-node/config.go:888        bootstrapping with online state        {"previous": "ONLINE"}

Please triage what was the reason that node was in offline state.

Logs:

Version:

NeoFS Storage node
Version: v0.34.0-149-ge5f1af53-dirty
GoVersion: go1.18.4

Your Environment

Server setup and configuration:
HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

Originally created by @anikeev-yadro on GitHub (Jan 16, 2023). Originally assigned to: @carpawell on GitHub. Node changed state to OFFLINE in period of time from 13.01.2023 to 16.01.2023 I saw only following lines in log: ``` root@glagoli:/tmp# journalctl |grep -i "online" Jan 13 10:05:48 tatlinobject kernel: smartpqi 0000:19:00.0: Online Firmware Activation not supported by controller Jan 13 10:05:49 tatlinobject systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Jan 13 10:05:49 tatlinobject systemd[1]: Starting Remove Stale Online ext4 Metadata Check Snapshots... Jan 13 10:05:49 tatlinobject systemd[1]: Finished Remove Stale Online ext4 Metadata Check Snapshots. Jan 13 10:06:02 tatlinobject systemd[1]: Reached target Network is Online. Jan 13 10:25:00 glagoli neofs-node[17354]: 2023-01-13T10:25:00.840Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "OFFLINE"} Jan 13 10:29:17 glagoli neofs-node[18166]: 2023-01-13T10:29:17.259Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "OFFLINE"} Jan 13 10:33:49 glagoli neofs-node[18166]: 2023-01-13T10:33:49.124Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 10:54:06 glagoli neofs-node[18166]: 2023-01-13T10:54:06.011Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 11:17:45 glagoli neofs-node[18166]: 2023-01-13T11:17:45.711Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 11:38:02 glagoli neofs-node[18166]: 2023-01-13T11:38:02.480Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 11:58:19 glagoli neofs-node[18166]: 2023-01-13T11:58:19.263Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 12:18:36 glagoli neofs-node[18166]: 2023-01-13T12:18:36.033Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 12:38:52 glagoli neofs-node[18166]: 2023-01-13T12:38:52.809Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 13:01:49 glagoli neofs-node[18166]: 2023-01-13T13:01:49.599Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 13:26:46 glagoli neofs-node[18166]: 2023-01-13T13:26:46.339Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 14:03:03 glagoli neofs-node[18166]: 2023-01-13T14:03:03.015Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 14:23:19 glagoli neofs-node[18166]: 2023-01-13T14:23:19.743Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 14:43:36 glagoli neofs-node[18166]: 2023-01-13T14:43:36.601Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 15:03:53 glagoli neofs-node[18166]: 2023-01-13T15:03:53.398Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 15:24:21 glagoli neofs-node[18166]: 2023-01-13T15:24:21.030Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 15:44:27 glagoli neofs-node[18166]: 2023-01-13T15:44:27.096Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 13 17:05:38 glagoli neofs-node[18166]: 2023-01-13T17:05:38.067Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "OFFLINE"} Jan 13 18:30:09 glagoli neofs-node[18166]: 2023-01-13T18:30:09.823Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "OFFLINE"} Jan 15 03:10:18 glagoli systemd[1]: Starting Online ext4 Metadata Check for All Filesystems... Jan 15 03:10:18 glagoli systemd[1]: Finished Online ext4 Metadata Check for All Filesystems. ``` Than I changed state to ``ONLINE```: ``` Jan 16 06:47:02 glagoli sudo[1809669]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json control set-status --status online Jan 16 06:52:08 glagoli sudo[1809807]: root : TTY=pts/0 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json control set-status --status online Jan 16 06:52:11 glagoli sudo[1809819]: root : TTY=pts/0 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/neofs-cli --endpoint 127.0.0.1:8091 -w /etc/neofs/storage/wallet.json control set-status --status online Jan 16 06:57:40 glagoli neofs-node[1816249]: 2023-01-16T06:57:40.801Z info neofs-node/netmap.go:296 initial network state {"epoch": 313, "state": "online"} Jan 16 06:57:41 glagoli neofs-node[1816249]: 2023-01-16T06:57:41.813Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 07:12:54 glagoli neofs-node[1816249]: 2023-01-16T07:12:54.194Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 07:33:11 glagoli neofs-node[1816249]: 2023-01-16T07:33:11.124Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 07:53:36 glagoli neofs-node[1816249]: 2023-01-16T07:53:36.772Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 08:13:52 glagoli neofs-node[1816249]: 2023-01-16T08:13:52.495Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 08:34:18 glagoli neofs-node[1816249]: 2023-01-16T08:34:18.746Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 09:37:59 glagoli neofs-node[1882278]: 2023-01-16T09:37:59.949Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "OFFLINE"} Jan 16 09:56:50 glagoli neofs-node[1882278]: 2023-01-16T09:56:50.977Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 10:17:06 glagoli neofs-node[1882278]: 2023-01-16T10:17:06.710Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 10:37:28 glagoli neofs-node[1882278]: 2023-01-16T10:37:28.176Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 10:57:39 glagoli neofs-node[1882278]: 2023-01-16T10:57:39.450Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} Jan 16 11:54:53 glagoli neofs-node[1941679]: 2023-01-16T11:54:53.095Z info neofs-node/netmap.go:296 initial network state {"epoch": 337, "state": "online"} Jan 16 11:55:23 glagoli neofs-node[1941679]: 2023-01-16T11:55:23.791Z info neofs-node/config.go:888 bootstrapping with online state {"previous": "ONLINE"} ``` Please triage what was the reason that node was in offline state. ## Logs: ## Version: ``` NeoFS Storage node Version: v0.34.0-149-ge5f1af53-dirty GoVersion: go1.18.4 ``` ## Your Environment Server setup and configuration: HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw Operating System and version (uname -a): linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux