Replicator repeatedly can't get data from RPC

sami commented

2025-12-28 17:23:07 +00:00

Owner

Originally created by @roman-khimov on GitHub (Aug 7, 2025).

Expected Behavior

Reconnect of some kind solving the problem.

Current Behavior

Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}
Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}
Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}
Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}

Possible Solution

Reconnect, continue operation.

Steps to Reproduce (for bugs)

Unknown. It just happened.

Context

But it happened on mainnet.

Regression

Unknown.

Your Environment

Version used: 0.48.1
Server setup and configuration:

fschain:
  endpoints:
  - wss://rpc.morph.fs.neo.org/ws

Originally created by @roman-khimov on GitHub (Aug 7, 2025). ## Expected Behavior Reconnect of some kind solving the problem. ## Current Behavior ``` Aug 07 05:55:51 metis1 neofs-node[2289135]: error policer/check.go:85 could not get container {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"} Aug 07 05:55:51 metis1 neofs-node[2289135]: error policer/check.go:85 could not get container {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"} Aug 07 05:55:51 metis1 neofs-node[2289135]: error policer/check.go:85 could not get container {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"} Aug 07 05:55:51 metis1 neofs-node[2289135]: error policer/check.go:85 could not get container {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"} ``` ## Possible Solution Reconnect, continue operation. ## Steps to Reproduce (for bugs) Unknown. It just happened. ## Context But it happened on mainnet. ## Regression Unknown. ## Your Environment * Version used: 0.48.1 * Server setup and configuration: ``` fschain: endpoints: - wss://rpc.morph.fs.neo.org/ws ```

sami

2025-12-28 17:23:07 +00:00

closed this issue
added the
bug

neofs-storage

S4

I4

U1
labels

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

For whatever reason all four nodes were stuck this way.

@roman-khimov commented on GitHub (Aug 7, 2025): For whatever reason all four nodes were stuck this way.

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

9 @ 0x47942e 0x410b25 0x410777 0xd79b1b 0xea22ca 0xea221f 0xc4fa8a 0x480f81
#       0xd79b1a        github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch+0xfa    github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247
#       0xea22c9        main.initLocalStorage.func1+0x89                                                                        github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36
#       0xea221e        main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1+0x1e               github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75
#       0xc4fa89        github.com/panjf2000/ants/v2.(*goWorker).run.func1+0x89                                                 github.com/panjf2000/ants/v2@v2.9.0/worker.go:67
...
1 @ 0x47942e 0x456cd7 0xbb9156 0xbad9aa 0xbb45f0 0xbb3827 0xb99a4d 0xbc6265 0xbc763a 0xbe9435 0xbe9388 0xe77983 0xea6808 0xe77ac6 0xe5f139 0xe566b0 0xaaeac6 0xa59d96 0xa5ef88 0xa5773f 0x480f81
#       0xbb9155        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).makeWsRequest+0x315               github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:767
#       0xbad9a9        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).performRequest+0x149                github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/client.go:185
#       0xbb45ef        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).invokeSomething+0x24f               github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/rpc.go:894
#       0xbb3826        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).InvokeFunction+0x1a6                github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/rpc.go:827
#       0xb99a4c        github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker.(*Invoker).Call+0xac                  github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/invoker/invoker.go:161
#       0xbc6264        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).Call+0x44                    github.com/nspcc-dev/neofs-node/pkg/morph/client/client.go:74
#       0xbc7639        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).TestInvoke+0x39              github.com/nspcc-dev/neofs-node/pkg/morph/client/client.go:333
#       0xbe9434        github.com/nspcc-dev/neofs-node/pkg/morph/client.StaticClient.TestInvoke+0x1f4          github.com/nspcc-dev/neofs-node/pkg/morph/client/static.go:222
#       0xbe9387        github.com/nspcc-dev/neofs-node/pkg/morph/client/container.(*Client).GetEACL+0x147      github.com/nspcc-dev/neofs-node/pkg/morph/client/container/eacl.go:25
#       0xe77982        main.newCachedEACLStorage.func1+0x82                                                    github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:229
#       0xea6807        main.(*ttlNetCache[...]).get+0x287                                                      github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:109
#       0xe77ac5        main.(*ttlEACLStorage).GetEACL+0x85                                                     github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:237
#       0xe5f138        github.com/nspcc-dev/neofs-node/pkg/services/object/acl.(*Checker).CheckEACL+0x358      github.com/nspcc-dev/neofs-node/pkg/services/object/acl/acl.go:144
#       0xe566af        github.com/nspcc-dev/neofs-node/pkg/services/object.(*Server).SearchV2+0x26f            github.com/nspcc-dev/neofs-node/pkg/services/object/server.go:1837
#       0xaaeac5        github.com/nspcc-dev/neofs-sdk-go/proto/object._ObjectService_SearchV2_Handler+0x1a5    github.com/nspcc-dev/neofs-sdk-go@v1.0.0-rc.13.0.20250715070617-c7038b450691/proto/object/service_grpc.pb.go:762
#       0xa59d95        google.golang.org/grpc.(*Server).processUnaryRPC+0x1035                                 google.golang.org/grpc@v1.70.0/server.go:1400
#       0xa5ef87        google.golang.org/grpc.(*Server).handleStream+0xb87                                     google.golang.org/grpc@v1.70.0/server.go:1810
#       0xa5773e        google.golang.org/grpc.(*Server).serveStreams.func2.1+0x7e                              google.golang.org/grpc@v1.70.0/server.go:1030

1 @ 0x47942e 0x456cd7 0xbcb425 0x480f81
#       0xbcb424        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).closeWaiter+0x64     github.com/nspcc-dev/neofs-node/pkg/morph/client/multi.go:62

1 @ 0x47942e 0x456cd7 0xbd271d 0x480f81
#       0xbd271c        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).routeNotifications+0x1dc     github.com/nspcc-dev/neofs-node/pkg/morph/client/notifications.go:239
...
1 @ 0x47942e 0x456cd7 0xd8bc09 0x480f81
#       0xd8bc08        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForHeaders+0xc8       github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:198

1 @ 0x47942e 0x456cd7 0xd8c045 0x480f81
#       0xd8c044        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotary+0x184       github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:230

1 @ 0x47942e 0x456cd7 0xd8c3b9 0xd8b5a5 0xd8b388 0x4969eb 0xd8b312 0xd8b2d9 0xe8cdeb 0xe8995e 0xe8cd7f 0xea54a6 0xea56b4 0x480f81
#       0xd8c3b8        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotifications+0xd8 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:250
#       0xd8b5a4        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listen+0x1e4                github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:145
#       0xd8b387        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError.func1+0x27  github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:128
#       0x4969ea        sync.(*Once).doSlow+0xaa                                                                sync/once.go:78
#       0xd8b311        sync.(*Once).Do+0x51                                                                    sync/once.go:69
#       0xd8b2d8        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError+0x18        github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:127
#       0xe8cdea        main.listenMorphNotifications.func1.1+0x2a                                              github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:71
#       0xe8995d        main.runAndLog+0x9d                                                                     github.com/nspcc-dev/neofs-node/cmd/neofs-node/main.go:158
#       0xe8cd7e        main.listenMorphNotifications.func1+0x5e                                                github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:70
#       0xea54a5        main.(*workerFromFunc).Run+0x25                                                         github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:22
#       0xea56b3        main.startWorkers.func1+0x33                                                            github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:30

@roman-khimov commented on GitHub (Aug 7, 2025): ``` 9 @ 0x47942e 0x410b25 0x410777 0xd79b1b 0xea22ca 0xea221f 0xc4fa8a 0x480f81 # 0xd79b1a github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch+0xfa github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 # 0xea22c9 main.initLocalStorage.func1+0x89 github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 # 0xea221e main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1+0x1e github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 # 0xc4fa89 github.com/panjf2000/ants/v2.(*goWorker).run.func1+0x89 github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 ... 1 @ 0x47942e 0x456cd7 0xbb9156 0xbad9aa 0xbb45f0 0xbb3827 0xb99a4d 0xbc6265 0xbc763a 0xbe9435 0xbe9388 0xe77983 0xea6808 0xe77ac6 0xe5f139 0xe566b0 0xaaeac6 0xa59d96 0xa5ef88 0xa5773f 0x480f81 # 0xbb9155 github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).makeWsRequest+0x315 github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:767 # 0xbad9a9 github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).performRequest+0x149 github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/client.go:185 # 0xbb45ef github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).invokeSomething+0x24f github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/rpc.go:894 # 0xbb3826 github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).InvokeFunction+0x1a6 github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/rpc.go:827 # 0xb99a4c github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker.(*Invoker).Call+0xac github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/invoker/invoker.go:161 # 0xbc6264 github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).Call+0x44 github.com/nspcc-dev/neofs-node/pkg/morph/client/client.go:74 # 0xbc7639 github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).TestInvoke+0x39 github.com/nspcc-dev/neofs-node/pkg/morph/client/client.go:333 # 0xbe9434 github.com/nspcc-dev/neofs-node/pkg/morph/client.StaticClient.TestInvoke+0x1f4 github.com/nspcc-dev/neofs-node/pkg/morph/client/static.go:222 # 0xbe9387 github.com/nspcc-dev/neofs-node/pkg/morph/client/container.(*Client).GetEACL+0x147 github.com/nspcc-dev/neofs-node/pkg/morph/client/container/eacl.go:25 # 0xe77982 main.newCachedEACLStorage.func1+0x82 github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:229 # 0xea6807 main.(*ttlNetCache[...]).get+0x287 github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:109 # 0xe77ac5 main.(*ttlEACLStorage).GetEACL+0x85 github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:237 # 0xe5f138 github.com/nspcc-dev/neofs-node/pkg/services/object/acl.(*Checker).CheckEACL+0x358 github.com/nspcc-dev/neofs-node/pkg/services/object/acl/acl.go:144 # 0xe566af github.com/nspcc-dev/neofs-node/pkg/services/object.(*Server).SearchV2+0x26f github.com/nspcc-dev/neofs-node/pkg/services/object/server.go:1837 # 0xaaeac5 github.com/nspcc-dev/neofs-sdk-go/proto/object._ObjectService_SearchV2_Handler+0x1a5 github.com/nspcc-dev/neofs-sdk-go@v1.0.0-rc.13.0.20250715070617-c7038b450691/proto/object/service_grpc.pb.go:762 # 0xa59d95 google.golang.org/grpc.(*Server).processUnaryRPC+0x1035 google.golang.org/grpc@v1.70.0/server.go:1400 # 0xa5ef87 google.golang.org/grpc.(*Server).handleStream+0xb87 google.golang.org/grpc@v1.70.0/server.go:1810 # 0xa5773e google.golang.org/grpc.(*Server).serveStreams.func2.1+0x7e google.golang.org/grpc@v1.70.0/server.go:1030 1 @ 0x47942e 0x456cd7 0xbcb425 0x480f81 # 0xbcb424 github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).closeWaiter+0x64 github.com/nspcc-dev/neofs-node/pkg/morph/client/multi.go:62 1 @ 0x47942e 0x456cd7 0xbd271d 0x480f81 # 0xbd271c github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).routeNotifications+0x1dc github.com/nspcc-dev/neofs-node/pkg/morph/client/notifications.go:239 ... 1 @ 0x47942e 0x456cd7 0xd8bc09 0x480f81 # 0xd8bc08 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForHeaders+0xc8 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:198 1 @ 0x47942e 0x456cd7 0xd8c045 0x480f81 # 0xd8c044 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotary+0x184 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:230 1 @ 0x47942e 0x456cd7 0xd8c3b9 0xd8b5a5 0xd8b388 0x4969eb 0xd8b312 0xd8b2d9 0xe8cdeb 0xe8995e 0xe8cd7f 0xea54a6 0xea56b4 0x480f81 # 0xd8c3b8 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotifications+0xd8 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:250 # 0xd8b5a4 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listen+0x1e4 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:145 # 0xd8b387 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError.func1+0x27 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:128 # 0x4969ea sync.(*Once).doSlow+0xaa sync/once.go:78 # 0xd8b311 sync.(*Once).Do+0x51 sync/once.go:69 # 0xd8b2d8 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError+0x18 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:127 # 0xe8cdea main.listenMorphNotifications.func1.1+0x2a github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:71 # 0xe8995d main.runAndLog+0x9d github.com/nspcc-dev/neofs-node/cmd/neofs-node/main.go:158 # 0xe8cd7e main.listenMorphNotifications.func1+0x5e github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:70 # 0xea54a5 main.(*workerFromFunc).Run+0x25 github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:22 # 0xea56b3 main.startWorkers.func1+0x33 github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:30 ```

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

It reproduced on m3, but restored eventually.

info client/notifications.go:230 RPC connection lost, attempting reconnect

@roman-khimov commented on GitHub (Aug 7, 2025): It reproduced on m3, but restored eventually. > info client/notifications.go:230 RPC connection lost, attempting reconnect

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

But before reconnect it's

error	neofs-node/netmap.go:211	could not update node state on new epoch after retries	{"epoch": 33473, "retries": 25, "error": "failed to write JSON request (invokefunction / 4): write tcp 135.181.79.214:33480->65.109.234.149:443: write: broken pipe: before sending the request"}
info	neofs-node/netmap.go:250	new epoch event, requesting new network map to sync SN connection caches...	{"epoch": 33473}

@roman-khimov commented on GitHub (Aug 7, 2025): But _before_ reconnect it's ``` error neofs-node/netmap.go:211 could not update node state on new epoch after retries {"epoch": 33473, "retries": 25, "error": "failed to write JSON request (invokefunction / 4): write tcp 135.181.79.214:33480->65.109.234.149:443: write: broken pipe: before sending the request"} info neofs-node/netmap.go:250 new epoch event, requesting new network map to sync SN connection caches... {"epoch": 33473} ```

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

So it fails requests, but receives an event. Pretty strange combination.

@roman-khimov commented on GitHub (Aug 7, 2025): So it fails requests, but receives an event. Pretty strange combination.

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

It's as if writer thread is done in WSClient, but reader is still active. Which shouldn't happen, broken connection is broken for both sides. We also have some ping-ponging.

@roman-khimov commented on GitHub (Aug 7, 2025): It's as if writer thread is done in WSClient, but reader is still active. Which shouldn't happen, broken connection is broken for both sides. We also have some ping-ponging.

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

Another case:

1 @ 0x47942e 0x410b25 0x410777 0xbb6014 0xbb8926 0xbb7945 0x480f81
#       0xbb6013        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*headerOfAddedBlockReceiver).TrySend+0xd3    github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:202
#       0xbb8925        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).notifySubscribers+0x285           github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:707
#       0xbb7944        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).wsReader+0x5e4                    github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:598
...
1 @ 0x47942e 0x43cfd7 0x478645 0x4f3627 0x4f491a 0x4f4908 0x5f8225 0x607fa5 0x68ccdb 0x5210d8 0x68cebe 0x689fcf 0x6907a5 0x6907a6 0x6d9ba3 0x6d9cd3 0xba3ec5 0xba5ead 0xba6d8d 0xba7cc5 0xbb749d 0x480f81
#       0x478644        internal/poll.runtime_pollWait+0x84                                     runtime/netpoll.go:351
#       0x4f3626        internal/poll.(*pollDesc).wait+0x26                                     internal/poll/fd_poll_runtime.go:84
#       0x4f4919        internal/poll.(*pollDesc).waitRead+0x279                                internal/poll/fd_poll_runtime.go:89
#       0x4f4907        internal/poll.(*FD).Read+0x267                                          internal/poll/fd_unix.go:165
#       0x5f8224        net.(*netFD).Read+0x24                                                  net/fd_posix.go:55
#       0x607fa4        net.(*conn).Read+0x44                                                   net/net.go:194
#       0x68ccda        crypto/tls.(*atLeastReader).Read+0x3a                                   crypto/tls/conn.go:809
#       0x5210d7        bytes.(*Buffer).ReadFrom+0x97                                           bytes/buffer.go:211
#       0x68cebd        crypto/tls.(*Conn).readFromUntil+0xdd                                   crypto/tls/conn.go:831
#       0x689fce        crypto/tls.(*Conn).readRecordOrCCS+0x3ce                                crypto/tls/conn.go:629
#       0x6907a4        crypto/tls.(*Conn).readRecord+0x144                                     crypto/tls/conn.go:591
#       0x6907a5        crypto/tls.(*Conn).Read+0x145                                           crypto/tls/conn.go:1385
#       0x6d9ba2        bufio.(*Reader).fill+0x102                                              bufio/bufio.go:113
#       0x6d9cd2        bufio.(*Reader).Peek+0x52                                               bufio/bufio.go:152
#       0xba3ec4        github.com/gorilla/websocket.(*Conn).read+0x24                          github.com/gorilla/websocket@v1.5.3/conn.go:371
#       0xba5eac        github.com/gorilla/websocket.(*Conn).advanceFrame+0x6c                  github.com/gorilla/websocket@v1.5.3/conn.go:809
#       0xba6d8c        github.com/gorilla/websocket.(*Conn).NextReader+0xac                    github.com/gorilla/websocket@v1.5.3/conn.go:1009
#       0xba7cc4        github.com/gorilla/websocket.(*Conn).ReadJSON+0x24                      github.com/gorilla/websocket@v1.5.3/json.go:50
#       0xbb749c        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).wsReader+0x13c    github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:539
...
1 @ 0x47942e 0x456cd7 0xbb82b7 0x480f81
#       0xbb82b6        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).wsWriter+0x156    github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:661
...

@roman-khimov commented on GitHub (Aug 7, 2025): Another case: ``` 1 @ 0x47942e 0x410b25 0x410777 0xbb6014 0xbb8926 0xbb7945 0x480f81 # 0xbb6013 github.com/nspcc-dev/neo-go/pkg/rpcclient.(*headerOfAddedBlockReceiver).TrySend+0xd3 github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:202 # 0xbb8925 github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).notifySubscribers+0x285 github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:707 # 0xbb7944 github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).wsReader+0x5e4 github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:598 ... 1 @ 0x47942e 0x43cfd7 0x478645 0x4f3627 0x4f491a 0x4f4908 0x5f8225 0x607fa5 0x68ccdb 0x5210d8 0x68cebe 0x689fcf 0x6907a5 0x6907a6 0x6d9ba3 0x6d9cd3 0xba3ec5 0xba5ead 0xba6d8d 0xba7cc5 0xbb749d 0x480f81 # 0x478644 internal/poll.runtime_pollWait+0x84 runtime/netpoll.go:351 # 0x4f3626 internal/poll.(*pollDesc).wait+0x26 internal/poll/fd_poll_runtime.go:84 # 0x4f4919 internal/poll.(*pollDesc).waitRead+0x279 internal/poll/fd_poll_runtime.go:89 # 0x4f4907 internal/poll.(*FD).Read+0x267 internal/poll/fd_unix.go:165 # 0x5f8224 net.(*netFD).Read+0x24 net/fd_posix.go:55 # 0x607fa4 net.(*conn).Read+0x44 net/net.go:194 # 0x68ccda crypto/tls.(*atLeastReader).Read+0x3a crypto/tls/conn.go:809 # 0x5210d7 bytes.(*Buffer).ReadFrom+0x97 bytes/buffer.go:211 # 0x68cebd crypto/tls.(*Conn).readFromUntil+0xdd crypto/tls/conn.go:831 # 0x689fce crypto/tls.(*Conn).readRecordOrCCS+0x3ce crypto/tls/conn.go:629 # 0x6907a4 crypto/tls.(*Conn).readRecord+0x144 crypto/tls/conn.go:591 # 0x6907a5 crypto/tls.(*Conn).Read+0x145 crypto/tls/conn.go:1385 # 0x6d9ba2 bufio.(*Reader).fill+0x102 bufio/bufio.go:113 # 0x6d9cd2 bufio.(*Reader).Peek+0x52 bufio/bufio.go:152 # 0xba3ec4 github.com/gorilla/websocket.(*Conn).read+0x24 github.com/gorilla/websocket@v1.5.3/conn.go:371 # 0xba5eac github.com/gorilla/websocket.(*Conn).advanceFrame+0x6c github.com/gorilla/websocket@v1.5.3/conn.go:809 # 0xba6d8c github.com/gorilla/websocket.(*Conn).NextReader+0xac github.com/gorilla/websocket@v1.5.3/conn.go:1009 # 0xba7cc4 github.com/gorilla/websocket.(*Conn).ReadJSON+0x24 github.com/gorilla/websocket@v1.5.3/json.go:50 # 0xbb749c github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).wsReader+0x13c github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:539 ... 1 @ 0x47942e 0x456cd7 0xbb82b7 0x480f81 # 0xbb82b6 github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).wsWriter+0x156 github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:661 ...

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

1 @ 0x47942e 0x410b25 0x410777 0xbd27e9 0xbd27c6 0x480f81
#       0xbd27e8        github.com/nspcc-dev/neofs-node/pkg/morph/client.handleEv[...]+0x2a8                    github.com/nspcc-dev/neofs-node/pkg/morph/client/notifications.go:324
#       0xbd27c5        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).routeNotifications+0x285     github.com/nspcc-dev/neofs-node/pkg/morph/client/notifications.go:243
...
1 @ 0x47942e 0x47aa79 0x47aa59 0x496185 0xc4f6a5 0xc4f165 0xe7f6e2 0xd8cc92 0xd8c339 0xd8b5a5 0xd8b388 0x4969eb 0xd8b312 0xd8b2d9 0xe8cdeb 0xe8995e 0xe8cd7f 0xea54a6 0xea56b4 0x480f81
#       0x47aa58        sync.runtime_notifyListWait+0x138                                                               runtime/sema.go:597
#       0x496184        sync.(*Cond).Wait+0x84                                                                          sync/cond.go:71
#       0xc4f6a4        github.com/panjf2000/ants/v2.(*Pool).retrieveWorker+0xc4                                        github.com/panjf2000/ants/v2@v2.9.0/pool.go:360
#       0xc4f164        github.com/panjf2000/ants/v2.(*Pool).Submit+0x24                                                github.com/panjf2000/ants/v2@v2.9.0/pool.go:222
#       0xe7f6e1        main.initContainerService.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func10+0xc1     github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:74
#       0xd8cc91        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).parseAndHandleNotification+0x6d1    github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:318
#       0xd8c338        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotifications+0x58         github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:262
#       0xd8b5a4        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listen+0x1e4                        github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:145
#       0xd8b387        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError.func1+0x27          github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:128
#       0x4969ea        sync.(*Once).doSlow+0xaa                                                                        sync/once.go:78
#       0xd8b311        sync.(*Once).Do+0x51                                                                            sync/once.go:69
#       0xd8b2d8        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError+0x18                github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:127
#       0xe8cdea        main.listenMorphNotifications.func1.1+0x2a                                                      github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:71
#       0xe8995d        main.runAndLog+0x9d                                                                             github.com/nspcc-dev/neofs-node/cmd/neofs-node/main.go:158
#       0xe8cd7e        main.listenMorphNotifications.func1+0x5e                                                        github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:70
#       0xea54a5        main.(*workerFromFunc).Run+0x25                                                                 github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:22
#       0xea56b3        main.startWorkers.func1+0x33                                                                    github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:30

Looks like we can't process an event here.

@roman-khimov commented on GitHub (Aug 7, 2025): ``` 1 @ 0x47942e 0x410b25 0x410777 0xbd27e9 0xbd27c6 0x480f81 # 0xbd27e8 github.com/nspcc-dev/neofs-node/pkg/morph/client.handleEv[...]+0x2a8 github.com/nspcc-dev/neofs-node/pkg/morph/client/notifications.go:324 # 0xbd27c5 github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).routeNotifications+0x285 github.com/nspcc-dev/neofs-node/pkg/morph/client/notifications.go:243 ... 1 @ 0x47942e 0x47aa79 0x47aa59 0x496185 0xc4f6a5 0xc4f165 0xe7f6e2 0xd8cc92 0xd8c339 0xd8b5a5 0xd8b388 0x4969eb 0xd8b312 0xd8b2d9 0xe8cdeb 0xe8995e 0xe8cd7f 0xea54a6 0xea56b4 0x480f81 # 0x47aa58 sync.runtime_notifyListWait+0x138 runtime/sema.go:597 # 0x496184 sync.(*Cond).Wait+0x84 sync/cond.go:71 # 0xc4f6a4 github.com/panjf2000/ants/v2.(*Pool).retrieveWorker+0xc4 github.com/panjf2000/ants/v2@v2.9.0/pool.go:360 # 0xc4f164 github.com/panjf2000/ants/v2.(*Pool).Submit+0x24 github.com/panjf2000/ants/v2@v2.9.0/pool.go:222 # 0xe7f6e1 main.initContainerService.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func10+0xc1 github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:74 # 0xd8cc91 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).parseAndHandleNotification+0x6d1 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:318 # 0xd8c338 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotifications+0x58 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:262 # 0xd8b5a4 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listen+0x1e4 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:145 # 0xd8b387 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError.func1+0x27 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:128 # 0x4969ea sync.(*Once).doSlow+0xaa sync/once.go:78 # 0xd8b311 sync.(*Once).Do+0x51 sync/once.go:69 # 0xd8b2d8 github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError+0x18 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:127 # 0xe8cdea main.listenMorphNotifications.func1.1+0x2a github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:71 # 0xe8995d main.runAndLog+0x9d github.com/nspcc-dev/neofs-node/cmd/neofs-node/main.go:158 # 0xe8cd7e main.listenMorphNotifications.func1+0x5e github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:70 # 0xea54a5 main.(*workerFromFunc).Run+0x25 github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:22 # 0xea56b3 main.startWorkers.func1+0x33 github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:30 ``` Looks like we can't process an event here.

sami commented

2025-12-28 17:23:08 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 7, 2025):

And it's GC blocking things because previous iteration hasn't finished yet.

1 @ 0x47942e 0x457d7d 0x457d54 0x47a865 0x497e28 0xd58ea5 0x480f81
#       0x47a864        sync.runtime_SemacquireWaitGroup+0x24                                                   runtime/sema.go:110
#       0x497e27        sync.(*WaitGroup).Wait+0x47                                                             sync/waitgroup.go:118
#       0xd58ea4        github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*gc).listenEvents+0xe4  github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:114

@roman-khimov commented on GitHub (Aug 7, 2025): And it's GC blocking things because previous iteration hasn't finished yet. ``` 1 @ 0x47942e 0x457d7d 0x457d54 0x47a865 0x497e28 0xd58ea5 0x480f81 # 0x47a864 sync.runtime_SemacquireWaitGroup+0x24 runtime/sema.go:110 # 0x497e27 sync.(*WaitGroup).Wait+0x47 sync/waitgroup.go:118 # 0xd58ea4 github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*gc).listenEvents+0xe4 github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/gc.go:114 ```

Rows
Columns

Replicator repeatedly can't get data from RPC #1474

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Regression

Your Environment