Properly await WS connections finalization on RPC server shutdown #1416

New issue

Open

opened 2025-12-28 17:16:22 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:16:22 +00:00

Owner

Originally created by @AnnaShaleva on GitHub (Nov 6, 2024).

Current Behavior

Described in https://github.com/nspcc-dev/neo-go/pull/3661#issuecomment-2457035557. Stacktrace of improperly reused RPC server mutex shows that WS-handler routines are still active after server shutdown:

oct‚ 22 13:51:15 node1 neo-go[1183338]: fatal error: sync: Unlock of unlocked RWMutex
oct‚ 22 13:51:15 node1 neo-go[1183338]: goroutine 538 [running]:
oct‚ 22 13:51:15 node1 neo-go[1183338]: sync.fatal({0xf83d64?, 0xc001085880?})
oct‚ 22 13:51:15 node1 neo-go[1183338]:         runtime/panic.go:1007 +0x18
oct‚ 22 13:51:15 node1 neo-go[1183338]: sync.(*RWMutex).Unlock(0xc00019a4c8)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         sync/rwmutex.go:208 +0x45
oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).dropSubscriber(0xc00019a2c8, 0xc000a77740)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:825 +0xce
oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).handleWsReads(0xc00019a2c8, 0xc0034478c0, 0xc000af5f80, 0xc000a7
7740)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:810 +0x266
oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).handleHTTPRequest(0xc00019a2c8, {0x11c3900, 0xc003437dc0}, 0xc00
31945a0)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:582 +0x54a
oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.HandlerFunc.ServeHTTP(0x471779?, {0x11c3900?, 0xc003437dc0?}, 0xc000943b68?)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:2171 +0x29
oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.serverHandler.ServeHTTP({0xc000a77680?}, {0x11c3900?, 0xc003437dc0?}, 0x6?)
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:3142 +0x8e
oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.(*conn).serve(0xc0032030e0, {0x11c5220, 0xc000a76960})
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:2044 +0x5e8
oct‚ 22 13:51:15 node1 neo-go[1183338]: created by net/http.(*Server).Serve in goroutine 534
oct‚ 22 13:51:15 node1 neo-go[1183338]:         net/http/server.go:3290 +0x4b4

Expected Behavior

All goroutines spawned during Server's lifetime are properly finilized and awaited in the call to Server's Shutdown().

Possible Solution

It's not a critical bug, especially after https://github.com/nspcc-dev/neo-go/pull/3661 merge, but we'd better keep the proper behaviour and await all WS-related routines in server shutdown.

Originally created by @AnnaShaleva on GitHub (Nov 6, 2024). ## Current Behavior Described in https://github.com/nspcc-dev/neo-go/pull/3661#issuecomment-2457035557. Stacktrace of improperly reused RPC server mutex shows that WS-handler routines are still active after server shutdown: ``` oct‚ 22 13:51:15 node1 neo-go[1183338]: fatal error: sync: Unlock of unlocked RWMutex oct‚ 22 13:51:15 node1 neo-go[1183338]: goroutine 538 [running]: oct‚ 22 13:51:15 node1 neo-go[1183338]: sync.fatal({0xf83d64?, 0xc001085880?}) oct‚ 22 13:51:15 node1 neo-go[1183338]: runtime/panic.go:1007 +0x18 oct‚ 22 13:51:15 node1 neo-go[1183338]: sync.(*RWMutex).Unlock(0xc00019a4c8) oct‚ 22 13:51:15 node1 neo-go[1183338]: sync/rwmutex.go:208 +0x45 oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).dropSubscriber(0xc00019a2c8, 0xc000a77740) oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:825 +0xce oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).handleWsReads(0xc00019a2c8, 0xc0034478c0, 0xc000af5f80, 0xc000a7 7740) oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:810 +0x266 oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv.(*Server).handleHTTPRequest(0xc00019a2c8, {0x11c3900, 0xc003437dc0}, 0xc00 31945a0) oct‚ 22 13:51:15 node1 neo-go[1183338]: github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/server.go:582 +0x54a oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.HandlerFunc.ServeHTTP(0x471779?, {0x11c3900?, 0xc003437dc0?}, 0xc000943b68?) oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http/server.go:2171 +0x29 oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.serverHandler.ServeHTTP({0xc000a77680?}, {0x11c3900?, 0xc003437dc0?}, 0x6?) oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http/server.go:3142 +0x8e oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http.(*conn).serve(0xc0032030e0, {0x11c5220, 0xc000a76960}) oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http/server.go:2044 +0x5e8 oct‚ 22 13:51:15 node1 neo-go[1183338]: created by net/http.(*Server).Serve in goroutine 534 oct‚ 22 13:51:15 node1 neo-go[1183338]: net/http/server.go:3290 +0x4b4 ``` ## Expected Behavior All goroutines spawned during Server's lifetime are properly finilized and awaited in the call to Server's `Shutdown()`. ## Possible Solution It's not a critical bug, especially after https://github.com/nspcc-dev/neo-go/pull/3661 merge, but we'd better keep the proper behaviour and await all WS-related routines in server shutdown.