Can't get object when one of the node became offline while put was running #847

New issue

Closed

opened 2025-12-28 17:20:53 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:20:53 +00:00

Owner

Originally created by @vkarak1 on GitHub (Oct 26, 2022).

Originally assigned to: @fyrchik on GitHub.

Can't get object when one of the node became unavailable while put was running

Expected Behavior

Object get should be finished ok.

Current Behavior

Get fails with:

root@az:/etc/neofs/storage# neofs-cli object get --cid 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk -w wallet.json -r node1.neofs:8080 -t 10m --file new_object1 --oid 3AWojWxqDX2fvmCKL7T4sK1fYkeK4edCvab4GYTfHuPF
Enter password >
 187904192 / 524288000 [============================================================>------------------------------------------------------------------------------------------------------------]  35.84% 2s
rpc error: copy payload: status: code = 2049 message = object not found

Steps to Reproduce (for bugs)

Create container with REP1 (node1):
neofs-cli container create --basic-acl public-read-write -r node1.neofs:8080 -w /etc/neofs/storage/wallet.json -p "REP 1"
Run container nodes (node1):

root@az:/etc/neofs/storage# neofs-cli container nodes  -w wallet.json -t 10m -r node1.neofs:8080 --cid  8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk
Enter password >
Rep 1
        Node 1: 02665dba76c7326948536ff98bdf936ebf7e8f7fe5f35622e40ace93ea48775836 ONLINE /dns4/node1.neofs/tcp/8080
                Continent: Europe
                Country: Russia
                CountryCode: RU
                Deployed: YACZROKH
                Location: Moskva
                Node: node1
                Price: 10
                SubDiv: Moskva
                SubDivCode: MOW
                UN-LOCODE: RU MOW
        Node 2: 02df711d9f69027f4e299fa64fa87e984435d36eb5076146d9677ecd1d1d7cfe2f ONLINE /dns4/node4.neofs/tcp/8080
                Continent: Europe
                Country: Finland
                CountryCode: FI
                Deployed: YACZROKH
                Location: Helsinki (Helsingfors)
                Node: node4
                Price: 10
                SubDiv: Uusimaa
                SubDivCode: 18
                UN-LOCODE: FI HEL
        Node 3: 03d464fed5dfa77da0773758a2a6783a59f2fcc05095139e3a3b5f941a1e95f35a ONLINE /dns4/node2.neofs/tcp/8080
                Continent: Europe
                Country: Russia
                CountryCode: RU
                Deployed: YACZROKH
                Location: Saint Petersburg (ex Leningrad)
                Node: node2
                Price: 10
                SubDiv: Sankt-Peterburg
                SubDivCode: SPE
                UN-LOCODE: RU LED

Allocate 500M object and issue put command (node1):

root@az:/etc/neofs/storage# neofs-cli object put --cid 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk -w wallet.json -r node1.neofs:8080 -t 10m --file object1
Enter password >
 524288000 / 524288000 [======================================================================================================================================================================] 100.00% 3m47s
[object1] Object successfully stored
  OID: 3AWojWxqDX2fvmCKL7T4sK1fYkeK4edCvab4GYTfHuPF
  CID: 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk

While step3 is running issue commands below to node4:
journalctl -xeu neofs-storage.service > /home/service/before_kernel_node4.txt && echo 1 > /proc/sys/kernel/sysrq && echo "c" > /proc/sysrq-trigger
Reboot node4 and I have saved the log after reboot:
journalctl -xeu neofs-storage.service > /home/service/log_reboot_after_kp_node4.txt
Run get command (node1):

root@az:/etc/neofs/storage# neofs-cli object get --cid 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk -w wallet.json -r node1.neofs:8080 -t 10m --file new_object1 --oid 3AWojWxqDX2fvmCKL7T4sK1fYkeK4edCvab4GYTfHuPF
Enter password >
 187904192 / 524288000 [============================================================>------------------------------------------------------------------------------------------------------------]  35.84% 2s
rpc error: copy payload: status: code = **2049 message = object not found**
root@az:/etc/neofs/storage# journalctl -xeu neofs-storage.service > /home/service/**log_attemp_to_get_object.node1**.txt

Attached logs

Your Environment

Linux az 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

NeoFS Storage node
Version: v0.33.0-31-ge1be0180
GoVersion: go1.18.4

Server setup and configuration:
cloud, 4 VMs, 4 SN, 4 http qw, 4 s3 gw

Originally created by @vkarak1 on GitHub (Oct 26, 2022). Originally assigned to: @fyrchik on GitHub. Can't get object when one of the node became unavailable while put was running ## Expected Behavior Object get should be finished ok. ## Current Behavior Get fails with: ``` root@az:/etc/neofs/storage# neofs-cli object get --cid 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk -w wallet.json -r node1.neofs:8080 -t 10m --file new_object1 --oid 3AWojWxqDX2fvmCKL7T4sK1fYkeK4edCvab4GYTfHuPF Enter password > 187904192 / 524288000 [============================================================>------------------------------------------------------------------------------------------------------------] 35.84% 2s rpc error: copy payload: status: code = 2049 message = object not found ``` ## Steps to Reproduce (for bugs) 1. Create container with REP1 (node1): `neofs-cli container create --basic-acl public-read-write -r node1.neofs:8080 -w /etc/neofs/storage/wallet.json -p "REP 1"` 2. Run container nodes (node1): ``` root@az:/etc/neofs/storage# neofs-cli container nodes -w wallet.json -t 10m -r node1.neofs:8080 --cid 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk Enter password > Rep 1 Node 1: 02665dba76c7326948536ff98bdf936ebf7e8f7fe5f35622e40ace93ea48775836 ONLINE /dns4/node1.neofs/tcp/8080 Continent: Europe Country: Russia CountryCode: RU Deployed: YACZROKH Location: Moskva Node: node1 Price: 10 SubDiv: Moskva SubDivCode: MOW UN-LOCODE: RU MOW Node 2: 02df711d9f69027f4e299fa64fa87e984435d36eb5076146d9677ecd1d1d7cfe2f ONLINE /dns4/node4.neofs/tcp/8080 Continent: Europe Country: Finland CountryCode: FI Deployed: YACZROKH Location: Helsinki (Helsingfors) Node: node4 Price: 10 SubDiv: Uusimaa SubDivCode: 18 UN-LOCODE: FI HEL Node 3: 03d464fed5dfa77da0773758a2a6783a59f2fcc05095139e3a3b5f941a1e95f35a ONLINE /dns4/node2.neofs/tcp/8080 Continent: Europe Country: Russia CountryCode: RU Deployed: YACZROKH Location: Saint Petersburg (ex Leningrad) Node: node2 Price: 10 SubDiv: Sankt-Peterburg SubDivCode: SPE UN-LOCODE: RU LED ``` 3. Allocate 500M object and issue put command (node1): ``` root@az:/etc/neofs/storage# neofs-cli object put --cid 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk -w wallet.json -r node1.neofs:8080 -t 10m --file object1 Enter password > 524288000 / 524288000 [======================================================================================================================================================================] 100.00% 3m47s [object1] Object successfully stored OID: 3AWojWxqDX2fvmCKL7T4sK1fYkeK4edCvab4GYTfHuPF CID: 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk ``` 4. While step3 is running issue commands below to node4: `journalctl -xeu neofs-storage.service > /home/service/before_kernel_node4.txt && echo 1 > /proc/sys/kernel/sysrq && echo "c" > /proc/sysrq-trigger ` 5. Reboot node4 and I have saved the log after reboot: `journalctl -xeu neofs-storage.service > /home/service/log_reboot_after_kp_node4.txt ` 6. Run get command (node1): ``` root@az:/etc/neofs/storage# neofs-cli object get --cid 8sRdhwH6ERxjC4cpXEiNcVYaeYudGynpzxsD6wfCJSGk -w wallet.json -r node1.neofs:8080 -t 10m --file new_object1 --oid 3AWojWxqDX2fvmCKL7T4sK1fYkeK4edCvab4GYTfHuPF Enter password > 187904192 / 524288000 [============================================================>------------------------------------------------------------------------------------------------------------] 35.84% 2s rpc error: copy payload: status: code = **2049 message = object not found** root@az:/etc/neofs/storage# journalctl -xeu neofs-storage.service > /home/service/**log_attemp_to_get_object.node1**.txt ``` [Attached logs](https://github.com/nspcc-dev/neofs-node/files/9867657/KP_node4_put_get.zip) ## Your Environment Linux az 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux NeoFS Storage node Version: v0.33.0-31-ge1be0180 GoVersion: go1.18.4 Server setup and configuration: cloud, 4 VMs, 4 SN, 4 http qw, 4 s3 gw