FSTree panic #1477

New issue

Closed

opened 2025-12-28 17:23:08 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:23:08 +00:00

Owner

Originally created by @roman-khimov on GitHub (Aug 11, 2025).

Originally assigned to: @End-rey on GitHub.

Expected Behavior

No panics.

Current Behavior

Aug 11 12:58:57 metis3 neofs-node[3545588]: panic: runtime error: index out of range [1] with length 1
Aug 11 12:58:57 metis3 neofs-node[3545588]: goroutine 8307 [running]:
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.parseCombinedPrefix(...)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:432
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).extractHeaderAndStream(0xc000240a80, {0x94, 0x61, 0xd2, 0x54, 0x5e, 0x64, 0x73, 0xc9, 0xe9, ...}, ...)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/head.go:116 +0x6c5
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).getObjectStream(0xc000240a80, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...})
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/head.go:55 +0x246
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Head(0xc00141c480?, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...})
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/head.go:33 +0x45
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).Head(0xc0018aa060, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...}, ...)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/head.go:74 +0x797
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Head(0xc000240a10, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...}, ...)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/head.go:40 +0x2db
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*storageEngineWrapper).get(0x474a18dfa51a07d0?, 0x60000001f7395c8?)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/util.go:202 +0x24d
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).executeLocal(0xc02b233960)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/local.go:14 +0x3c
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).execute(0xc02b233960)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:101 +0x3c
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).get(0xc02b042ae0, {0x1463b30, 0xc00dba7200}, {{0x145aaf8, 0xc00dba7350}, 0xc023c2a800, {{0x23, 0xea, 0x75, 0xd2, ...}, ...}, ...}, ...)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:92 +0x17b
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).Head(0x40?, {0x1463b30?, 0xc00dba7200?}, {{{0x145aaf8, 0xc00dba7350}, 0xc023c2a800, {{0x23, 0xea, 0x75, 0xd2, ...}, ...}, ...}})
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:70 +0x88
Aug 11 12:58:57 metis3 neofs-node[3545588]: main.(*objectSvc).Head(0x14688b0?, {0x1463b30?, 0xc00dba7200?}, {{{0x145aaf8, 0xc00dba7350}, 0xc023c2a800, {{0x23, 0xea, 0x75, 0xd2, ...}, ...}, ...}})
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/object.go:77 +0x5b
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object.(*Server).Head(0xc00016adc0, {0x1463b30, 0xc00dba7200}, 0xc023c2a680)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-node/pkg/services/object/server.go:618 +0x485
Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-sdk-go/proto/object._ObjectService_Head_Handler({0x11e7060, 0xc00016adc0}, {0x1463b30, 0xc00dba7200}, 0xc020142600, 0x0)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         github.com/nspcc-dev/neofs-sdk-go@v1.0.0-rc.14/proto/object/service_grpc.pb.go:733 +0x1a6
Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ee800, {0x1463b30, 0xc00dba7170}, 0xc02407b440, 0xc02b03dc50, 0x1e62678, 0x0)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         google.golang.org/grpc@v1.70.0/server.go:1400 +0x1036
Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc.(*Server).handleStream(0xc0001ee800, {0x1464a48, 0xc01756e340}, 0xc02407b440)
Aug 11 12:58:57 metis3 neofs-node[3545588]:         google.golang.org/grpc@v1.70.0/server.go:1810 +0xb88
Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc.(*Server).serveStreams.func2.1()
Aug 11 12:58:57 metis3 neofs-node[3545588]:         google.golang.org/grpc@v1.70.0/server.go:1030 +0x7f
Aug 11 12:58:57 metis3 neofs-node[3545588]: created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 566
Aug 11 12:58:57 metis3 neofs-node[3545588]:         google.golang.org/grpc@v1.70.0/server.go:1041 +0x11d

Possible Solution

Fix the damn thing.

Steps to Reproduce (for bugs)

Unknown.

Context

Mainnet.

Regression

Unknown.

Your Environment

Version used: 0.48.2
Server setup and configuration: mainnet

Originally created by @roman-khimov on GitHub (Aug 11, 2025). Originally assigned to: @End-rey on GitHub. ## Expected Behavior No panics. ## Current Behavior ``` Aug 11 12:58:57 metis3 neofs-node[3545588]: panic: runtime error: index out of range [1] with length 1 Aug 11 12:58:57 metis3 neofs-node[3545588]: goroutine 8307 [running]: Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.parseCombinedPrefix(...) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:432 Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).extractHeaderAndStream(0xc000240a80, {0x94, 0x61, 0xd2, 0x54, 0x5e, 0x64, 0x73, 0xc9, 0xe9, ...}, ...) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/head.go:116 +0x6c5 Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).getObjectStream(0xc000240a80, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...}) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/head.go:55 +0x246 Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Head(0xc00141c480?, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...}) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/blobstor/fstree/head.go:33 +0x45 Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard.(*Shard).Head(0xc0018aa060, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...}, ...) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/shard/head.go:74 +0x797 Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Head(0xc000240a10, {{0x23, 0xea, 0x75, 0xd2, 0x1, 0x6a, 0x26, 0xaa, 0x64, ...}, ...}, ...) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/head.go:40 +0x2db Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*storageEngineWrapper).get(0x474a18dfa51a07d0?, 0x60000001f7395c8?) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get/util.go:202 +0x24d Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).executeLocal(0xc02b233960) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get/local.go:14 +0x3c Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*execCtx).execute(0xc02b233960) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:101 +0x3c Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).get(0xc02b042ae0, {0x1463b30, 0xc00dba7200}, {{0x145aaf8, 0xc00dba7350}, 0xc023c2a800, {{0x23, 0xea, 0x75, 0xd2, ...}, ...}, ...}, ...) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:92 +0x17b Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).Head(0x40?, {0x1463b30?, 0xc00dba7200?}, {{{0x145aaf8, 0xc00dba7350}, 0xc023c2a800, {{0x23, 0xea, 0x75, 0xd2, ...}, ...}, ...}}) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:70 +0x88 Aug 11 12:58:57 metis3 neofs-node[3545588]: main.(*objectSvc).Head(0x14688b0?, {0x1463b30?, 0xc00dba7200?}, {{{0x145aaf8, 0xc00dba7350}, 0xc023c2a800, {{0x23, 0xea, 0x75, 0xd2, ...}, ...}, ...}}) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/cmd/neofs-node/object.go:77 +0x5b Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object.(*Server).Head(0xc00016adc0, {0x1463b30, 0xc00dba7200}, 0xc023c2a680) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-node/pkg/services/object/server.go:618 +0x485 Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-sdk-go/proto/object._ObjectService_Head_Handler({0x11e7060, 0xc00016adc0}, {0x1463b30, 0xc00dba7200}, 0xc020142600, 0x0) Aug 11 12:58:57 metis3 neofs-node[3545588]: github.com/nspcc-dev/neofs-sdk-go@v1.0.0-rc.14/proto/object/service_grpc.pb.go:733 +0x1a6 Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ee800, {0x1463b30, 0xc00dba7170}, 0xc02407b440, 0xc02b03dc50, 0x1e62678, 0x0) Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc@v1.70.0/server.go:1400 +0x1036 Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc.(*Server).handleStream(0xc0001ee800, {0x1464a48, 0xc01756e340}, 0xc02407b440) Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc@v1.70.0/server.go:1810 +0xb88 Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc.(*Server).serveStreams.func2.1() Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc@v1.70.0/server.go:1030 +0x7f Aug 11 12:58:57 metis3 neofs-node[3545588]: created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 566 Aug 11 12:58:57 metis3 neofs-node[3545588]: google.golang.org/grpc@v1.70.0/server.go:1041 +0x11d ``` ## Possible Solution Fix the damn thing. ## Steps to Reproduce (for bugs) Unknown. ## Context Mainnet. ## Regression Unknown. ## Your Environment * Version used: 0.48.2 * Server setup and configuration: mainnet

sami

2025-12-28 17:23:08 +00:00

closed this issue
added the
bug

neofs-storage

S4

I4

U1
labels

sami commented

2025-12-28 17:23:09 +00:00

Author

Owner

@roman-khimov commented on GitHub (Aug 11, 2025):

Can be related to
3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV/GNRwRi1HmvwdsxrJXwx4Usb7FRUj88zH4tKGoyALz7pr
3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV/B11P8GQ54rMR7UjNU99Tr33AeDX1QwvFmN4GWhNhUPvM
3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV/AzDrV87hiHWwCoNR92WYRUUkpmJPPUSpNsNvbyCfPK2W

(that's what others requested from node)

@roman-khimov commented on GitHub (Aug 11, 2025): Can be related to 3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV/GNRwRi1HmvwdsxrJXwx4Usb7FRUj88zH4tKGoyALz7pr 3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV/B11P8GQ54rMR7UjNU99Tr33AeDX1QwvFmN4GWhNhUPvM 3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV/AzDrV87hiHWwCoNR92WYRUUkpmJPPUSpNsNvbyCfPK2W (that's what others requested from node)