object hash command fails with "context deadline exceeded" error #824

New issue

Open

opened 2025-12-28 17:20:48 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:20:48 +00:00

Owner

Originally created by @vkarak1 on GitHub (Oct 20, 2022).

I faced issue when node1 was attemted to start but due to issue with ports couldn't:

Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing pprof service...
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:78        pprof service has been successfully initialized
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing prometheus service...
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:78        prometheus service has been successfully initia>
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing tree service...
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:78        tree service has been successfully initialized
Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z        info        neofs-node/main.go:76        initializing control service...
Oct 20 09:29:11 az neofs-node[1051]: 2022/10/20 09:29:11 listen tcp 127.0.0.1:8091: bind: address already in use
Oct 20 09:29:11 az systemd[1]: neofs-storage.service: Main process exited, code=exited, status=1/FAILURE
Oct 20 09:29:11 az systemd[1]: neofs-storage.service: Failed with result 'exit-code'.

And I tried to issue object hash command to node2 and got the following error:

root@buky:/etc/neofs/storage# neofs-cli object hash -r node2.neofs:8080 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSjrPsqv --oid FqWtXTxQx83bqVNr54nR1AZPh3Mew                  mEmH1TJYbv8eCWh --range 0:11 -w wallet.json
Enter password >
rpc error: read payload hashes via client: write request: context deadline exceeded

When I issued "systemctl restart neofs-storage.service" to node2 the problem had been sorted out.

Expected Behavior

Node2 should process command while node1 was trying to start.

Current Behavior

Object hash fails on node2 while node1 is unavailable.

Possible Solution

run systemctl restart neofs-storage.service

Steps to Reproduce (for bugs)

node1 create container:
neofs-cli --rpc-endpoint node1.neofs:8080 --wallet wallet.json container create --name test --policy "REP 2 IN X CBF 1 SELECT 2 FROM * AS X" --basic-acl public-read-write --await
node1: allocate object1:
dd if=/dev/urandom of=object1 bs=1M count=1
node1: put object1:
neofs-cli --rpc-endpoint node1.neofs:8080 -w wallet.json object put --file object1 --cid 5oFBZsSdAvRKUsL2FknAnNqjyFj6UjvXW8QfWQeEtG3z --no-progress
node1: shards list:
neofs-cli --endpoint localhost:8091 -w /etc/neofs/storage/wallet.json control shards list | grep Shard && curl -s localhost:6672 | rg neofs_node_object_counter | sed 1,2d
I was working on validating the following issue , and I performed to node1:
systemctl restart neofs-storage.service
to double check that shards list and neofs_node_object_counter display the same shard ids
And got the problem that node1 was trying to start:
rpc error: read payload hashes via client: write request: context deadline exceeded
node2: created container and put 10 bytes object:

neofs-cli --rpc-endpoint node2.neofs:8080 --wallet wallet.json container create --name rep1 --policy "REP 1 " --basic-acl public                  -read-write   --await
neofs-cli --rpc-endpoint node2.neofs:8080 -w wallet.json object put --file object1 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSj

node2: object hash:

root@buky:/etc/neofs/storage# neofs-cli object hash -r node2.neofs:8080 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSjrPsqv --oid FqWtXTxQx83bqVNr54nR1AZPh3Mew                  mEmH1TJYbv8eCWh --range 0:11 -w wallet.json
Enter password >
rpc error: read payload hashes via client: write request: context deadline exceeded

Your Environment

Linux buky 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

NeoFS Storage node
Version: v0.33.0-6-g4c63be66
GoVersion: go1.18.4

Attached logs

Originally created by @vkarak1 on GitHub (Oct 20, 2022).  I faced issue when node1 was attemted to start but due to issue with ports couldn't: ``` Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z info neofs-node/main.go:76 initializing pprof service... Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z info neofs-node/main.go:78 pprof service has been successfully initialized Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z info neofs-node/main.go:76 initializing prometheus service... Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z info neofs-node/main.go:78 prometheus service has been successfully initia> Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z info neofs-node/main.go:76 initializing tree service... Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z info neofs-node/main.go:78 tree service has been successfully initialized Oct 20 09:29:11 az neofs-node[1051]: 2022-10-20T09:29:11.463Z info neofs-node/main.go:76 initializing control service... Oct 20 09:29:11 az neofs-node[1051]: 2022/10/20 09:29:11 listen tcp 127.0.0.1:8091: bind: address already in use Oct 20 09:29:11 az systemd[1]: neofs-storage.service: Main process exited, code=exited, status=1/FAILURE Oct 20 09:29:11 az systemd[1]: neofs-storage.service: Failed with result 'exit-code'. ``` And I tried to issue object hash command to node2 and got the following error: ``` root@buky:/etc/neofs/storage# neofs-cli object hash -r node2.neofs:8080 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSjrPsqv --oid FqWtXTxQx83bqVNr54nR1AZPh3Mew mEmH1TJYbv8eCWh --range 0:11 -w wallet.json Enter password > rpc error: read payload hashes via client: write request: context deadline exceeded ``` When I issued "systemctl restart neofs-storage.service" to node2 the problem had been sorted out. ## Expected Behavior Node2 should process command while node1 was trying to start. ## Current Behavior Object hash fails on node2 while node1 is unavailable. ## Possible Solution run `systemctl restart neofs-storage.service ` ## Steps to Reproduce (for bugs) 1. node1 create container: `neofs-cli --rpc-endpoint node1.neofs:8080 --wallet wallet.json container create --name test --policy "REP 2 IN X CBF 1 SELECT 2 FROM * AS X" --basic-acl public-read-write --await` 2. node1: allocate object1: `dd if=/dev/urandom of=object1 bs=1M count=1` 3. node1: put object1: `neofs-cli --rpc-endpoint node1.neofs:8080 -w wallet.json object put --file object1 --cid 5oFBZsSdAvRKUsL2FknAnNqjyFj6UjvXW8QfWQeEtG3z --no-progress` 4. node1: shards list: `neofs-cli --endpoint localhost:8091 -w /etc/neofs/storage/wallet.json control shards list | grep Shard && curl -s localhost:6672 | rg neofs_node_object_counter | sed 1,2d` 5. I was working on validating the following [issue ](https://github.com/nspcc-dev/neofs-node/pull/1887), and I performed to node1: `systemctl restart neofs-storage.service` to double check that shards list and neofs_node_object_counter display the same shard ids 6. And got the problem that node1 was trying to start: `rpc error: read payload hashes via client: write request: context deadline exceeded` 7. node2: created container and put 10 bytes object: ``` neofs-cli --rpc-endpoint node2.neofs:8080 --wallet wallet.json container create --name rep1 --policy "REP 1 " --basic-acl public -read-write --await neofs-cli --rpc-endpoint node2.neofs:8080 -w wallet.json object put --file object1 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSj ``` 8. node2: object hash: ``` root@buky:/etc/neofs/storage# neofs-cli object hash -r node2.neofs:8080 --cid AFyW1jGfA4XhtKy9h92WE4FhSwJ9ks6xR3kTnSjrPsqv --oid FqWtXTxQx83bqVNr54nR1AZPh3Mew mEmH1TJYbv8eCWh --range 0:11 -w wallet.json Enter password > rpc error: read payload hashes via client: write request: context deadline exceeded ``` ## Your Environment Linux buky 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux NeoFS Storage node Version: v0.33.0-6-g4c63be66 GoVersion: go1.18.4 [Attached logs](https://github.com/nspcc-dev/neofs-node/files/9832077/node2_object_hash_issue.zip)

sami added the

bug

neofs-storage

labels

2025-12-28 17:20:48 +00:00

sami commented

2025-12-28 17:20:49 +00:00

Author

Owner

@vkarak1 commented on GitHub (Oct 20, 2022):

Also would like to add that when I issued "systemctl restart neofs-storage.service" to node2 it took about 1.5 min.
Also before the error message in step#8 displayed the command was running about 15 seconds.

@vkarak1 commented on GitHub (Oct 20, 2022): Also would like to add that when I issued "systemctl restart neofs-storage.service" to node2 it took about 1.5 min. Also before the error message in step#8 displayed the command was running about 15 seconds.