Tree service requests sometimes fail during high load #372

New issue

Closed

opened 2025-12-28 17:37:10 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:37:10 +00:00

Owner

Originally created by @cthulhu-rider on GitHub (Jul 13, 2023).

when xk6 loads NeoFS through S3 gateway with PutObject / GetObject stream, some Tree service requests fail

Jul 13 09:54:41 neofs-s3-gw[580041]: 2023-07-13T09:54:41.245Z        error        handler/util.go:29        call method        {"status": 500, "request_id": "5b93dd2a-d244-476d-8d0a-ae1b225ad6c6", "method": "PutObject", "bucket": "48969b24-1e7c-4a29-8a40-8962542605fb", "object": "4dad9708-4309-4630-906e-ed49e851ce09", "description": "could not upload 
object", "error": "couldn't add new verion to tree service: failed to add node by path: rpc error: code = Unavailable desc = error reading from server: read tcp [<ipv6>]:57298->[<ipv6>]:8080: use of closed network connection", "body close errors": []}

Jul 13 09:54:40 neofs-s3-gw[580041]: 2023-07-13T09:54:40.927Z        error        handler/util.go:29        call method        {"status": 500, "request_id": "64848ab0-bd31-4e37-b7a5-b048395bc1f5", "method": "GetObject", "bucket": "8d80a780-b5f0-4ab2-9c68-e85f6501381d", "object": "8d04d4e2-e2a9-4b17-896e-1ef96e8807b4", "description": "could not find ob
ject", "error": "failed to get node by path: rpc error: code = Unavailable desc = error reading from server: read tcp [<ipv6>]:44100->[<ipv6>]:8080: use of closed network connection"}

this happens periodically during load test

we need to explore how this (attempt to use closed TCP conn) happens and which side - client or server - should be fixed. I searched a bit for similar cases, they are mostly caused by premature server shutdown, but it doesn't seem to be our case.

Originally created by @cthulhu-rider on GitHub (Jul 13, 2023). when xk6 loads NeoFS through S3 gateway with `PutObject` / `GetObject` stream, some Tree service requests fail ``` Jul 13 09:54:41 neofs-s3-gw[580041]: 2023-07-13T09:54:41.245Z error handler/util.go:29 call method {"status": 500, "request_id": "5b93dd2a-d244-476d-8d0a-ae1b225ad6c6", "method": "PutObject", "bucket": "48969b24-1e7c-4a29-8a40-8962542605fb", "object": "4dad9708-4309-4630-906e-ed49e851ce09", "description": "could not upload object", "error": "couldn't add new verion to tree service: failed to add node by path: rpc error: code = Unavailable desc = error reading from server: read tcp [<ipv6>]:57298->[<ipv6>]:8080: use of closed network connection", "body close errors": []} Jul 13 09:54:40 neofs-s3-gw[580041]: 2023-07-13T09:54:40.927Z error handler/util.go:29 call method {"status": 500, "request_id": "64848ab0-bd31-4e37-b7a5-b048395bc1f5", "method": "GetObject", "bucket": "8d80a780-b5f0-4ab2-9c68-e85f6501381d", "object": "8d04d4e2-e2a9-4b17-896e-1ef96e8807b4", "description": "could not find ob ject", "error": "failed to get node by path: rpc error: code = Unavailable desc = error reading from server: read tcp [<ipv6>]:44100->[<ipv6>]:8080: use of closed network connection"} ``` this happens periodically during load test we need to explore how this (attempt to use closed TCP conn) happens and which side - client or server - should be fixed. I searched a bit for similar cases, they are mostly caused by premature server shutdown, but it doesn't seem to be our case.