Errors "context deadline exceeded" with not so much gRPC PUT load #950

New issue

Closed

opened 2025-12-28 17:21:16 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:21:16 +00:00

Owner

Originally created by @anikeev-yadro on GitHub (Jan 13, 2023).

Originally assigned to: @carpawell on GitHub.

Have errors "context deadline exceeded" with not so much gRPC PUT load.

I have started following k6 load on nodes around 14:00:

./k6 run -e DURATION=3000 -e WRITERS=20 -e READERS=0 -e DELETERS=0 -e DELETE_AGE=10 -e SLEEP_DELETE=10 -e REGISTRY_FILE=/tmp/grpc_reg_130123-1 -e WRITE_OBJ_SIZE=150000  -e GRPC_ENDPOINTS=10.78.70.118:8080,10.78.70.119:8080,10.78.70.120:8080,10.78.70.121:8080 -e PREGEN_JSON=/home/service/k6_new/grpc_130123-1.json scenarios/grpc.js

While load was running I saw 90-95% CPU load on nodes.
But in about 18 min I saw a lot of errors in k6:

INFO[3005] {"cid":"E3Kc2Z78VQfZFUxAuGnSjS5jvXCDgUZWMeVUbh58cDv2","error":"context deadline exceeded"}  source=console

At the same time, the load on the processor dropped to 2%.

Logs:

journal_az.tar.gz
journal_buki.tar.gz
journal_vedi.tar.gz
journal_glagoli.tar.gz

Versions:

NeoFS Storage node
Version: v0.34.0-149-ge5f1af53-dirty
GoVersion: go1.18.4

Your Environment

Server setup and configuration:
HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

Originally created by @anikeev-yadro on GitHub (Jan 13, 2023). Originally assigned to: @carpawell on GitHub. Have errors "context deadline exceeded" with not so much gRPC PUT load. I have started following k6 load on nodes around 14:00: ``` ./k6 run -e DURATION=3000 -e WRITERS=20 -e READERS=0 -e DELETERS=0 -e DELETE_AGE=10 -e SLEEP_DELETE=10 -e REGISTRY_FILE=/tmp/grpc_reg_130123-1 -e WRITE_OBJ_SIZE=150000 -e GRPC_ENDPOINTS=10.78.70.118:8080,10.78.70.119:8080,10.78.70.120:8080,10.78.70.121:8080 -e PREGEN_JSON=/home/service/k6_new/grpc_130123-1.json scenarios/grpc.js ``` While load was running I saw 90-95% CPU load on nodes. But in about 18 min I saw a lot of errors in k6: ``` INFO[3005] {"cid":"E3Kc2Z78VQfZFUxAuGnSjS5jvXCDgUZWMeVUbh58cDv2","error":"context deadline exceeded"} source=console ``` At the same time, the load on the processor dropped to 2%. ## Logs: [journal_az.tar.gz](https://github.com/nspcc-dev/neofs-node/files/10413489/journal_az.tar.gz) [journal_buki.tar.gz](https://github.com/nspcc-dev/neofs-node/files/10413490/journal_buki.tar.gz) [journal_vedi.tar.gz](https://github.com/nspcc-dev/neofs-node/files/10413491/journal_vedi.tar.gz) [journal_glagoli.tar.gz](https://github.com/nspcc-dev/neofs-node/files/10413492/journal_glagoli.tar.gz) ## Versions: ``` NeoFS Storage node Version: v0.34.0-149-ge5f1af53-dirty GoVersion: go1.18.4 ``` ## Your Environment Server setup and configuration: HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw Operating System and version (uname -a): linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux