neofs-storage.service gets continuously restarting when internal interface is down #797

New issue

Closed

opened 2025-12-28 17:20:43 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:20:43 +00:00

Owner

Originally created by @ikovalev1 on GitHub (Oct 13, 2022).

Originally assigned to: @fyrchik on GitHub.

Expected Behavior

If a node loses internal network, it should continiue to respond on data network via gRPC/HTTP/S3

Current Behavior

neofs storage gets restarting endlessly every 5 sec (systemd configuration):
systemctl status neofs-storage.service ... Active: activating (auto-restart) (Result: exit-code) since Thu 2022-10-13 12:14:49 UTC; 3s ago

Steps to Reproduce (for bugs)

there are 4 nodes with 2 interfaces (int and data)
create a public container with 1 replication, put an object in the container:
neofs-cli -r node1:8080 -w wallet.json container create --policy "REP 1 IN Node1 SELECT 1 FROM N1 AS Node1 FILTER Node EQ node1 AS N1" --await --basic-acl public-read-write neofs-cli -r node1.data:8080 -w wallet.json object put --file /home/service/test.txt --no-progress --cid Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK Enter password > [/home/service/test.txt] Object successfully stored OID: hR4kM3aPD4icZuPPFKmoocCogHep4HfJjMfcTG2spUY CID: Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK
Verify that you can read the object fron other nodes via data network:
neofs-cli -r node1.data:8080 -w wallet.json object get --cid Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK --oid hR4kM3aPD4icZuPPFKmoocCogHep4HfJjMfcTG2spUY --file /tmp/get_object.sample --no-progress
bring down the internal interface on node1
ifdown int
Try to read the object again:
oot@buky:/home/service# neofs-cli -r node1.data:8080 -w /etc/neofs/storage/wallet.json object put --file /home/service/test.txt --no-progress --cid Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK Enter password > create session: can't fetch network info: write request: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.26.161.238:8080: connect: connection refused"
Look if neofs-storage is running on node1:
root@az:/home/service# systemctl status neofs-storage.service Active: activating (auto-restart) (Result: exit-code) since Thu 2022-10-13 12:14:49 UTC; 3s ago

Your Environment

sbercloun 4 VM with 2 interfaces
root@az:/home/service# uname -a
Linux az 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

root@az:/home/service# neofs-cli --version
NeoFS CLI
Version: v0.32.0-34-g76cfcc24
GoVersion: go1.18.4

root@az:/home/service# neofs-node --version
NeoFS Storage node
Version: v0.32.0-34-g76cfcc24
GoVersion: go1.18.4

Originally created by @ikovalev1 on GitHub (Oct 13, 2022). Originally assigned to: @fyrchik on GitHub. ## Expected Behavior If a node loses internal network, it should continiue to respond on data network via gRPC/HTTP/S3 ## Current Behavior neofs storage gets restarting endlessly every 5 sec (systemd configuration): `systemctl status neofs-storage.service ... Active: activating (auto-restart) (Result: exit-code) since Thu 2022-10-13 12:14:49 UTC; 3s ago ` ## Steps to Reproduce (for bugs) 0. there are 4 nodes with 2 interfaces (int and data) 1. create a public container with 1 replication, put an object in the container: `neofs-cli -r node1:8080 -w wallet.json container create --policy "REP 1 IN Node1 SELECT 1 FROM N1 AS Node1 FILTER Node EQ node1 AS N1" --await --basic-acl public-read-write neofs-cli -r node1.data:8080 -w wallet.json object put --file /home/service/test.txt --no-progress --cid Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK Enter password > [/home/service/test.txt] Object successfully stored OID: hR4kM3aPD4icZuPPFKmoocCogHep4HfJjMfcTG2spUY CID: Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK` 3. Verify that you can read the object fron other nodes via data network: `neofs-cli -r node1.data:8080 -w wallet.json object get --cid Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK --oid hR4kM3aPD4icZuPPFKmoocCogHep4HfJjMfcTG2spUY --file /tmp/get_object.sample --no-progress` 4. bring down the internal interface on node1 `ifdown int` 5. Try to read the object again: `oot@buky:/home/service# neofs-cli -r node1.data:8080 -w /etc/neofs/storage/wallet.json object put --file /home/service/test.txt --no-progress --cid Fk89aSd6YErcmGyf9UMvKx4bAwXbGBJCSvG4Zza7U5nK Enter password > create session: can't fetch network info: write request: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.26.161.238:8080: connect: connection refused" ` 6. Look if neofs-storage is running on node1: `root@az:/home/service# systemctl status neofs-storage.service Active: activating (auto-restart) (Result: exit-code) since Thu 2022-10-13 12:14:49 UTC; 3s ago ` ## Your Environment sbercloun 4 VM with 2 interfaces root@az:/home/service# uname -a Linux az 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux root@az:/home/service# neofs-cli --version NeoFS CLI Version: v0.32.0-34-g76cfcc24 GoVersion: go1.18.4 root@az:/home/service# neofs-node --version NeoFS Storage node Version: v0.32.0-34-g76cfcc24 GoVersion: go1.18.4