Explore Reed-Solomon coder options for optimizations #1468

New issue

Open

opened 2025-12-28 17:23:06 +00:00 by sami · 2 comments

sami commented

2025-12-28 17:23:06 +00:00

Owner

Originally created by @cthulhu-rider on GitHub (Aug 5, 2025).

https://pkg.go.dev/github.com/klauspost/reedsolomon#New is used for EC encoding/decoding. It provides various options incl. optimizing ones, but we do not use any of them

Describe the solution you'd like

look around the options, choose fitting ones (if any)
prepare benchmarks
make an improvement

Additional context

#3457

Originally created by @cthulhu-rider on GitHub (Aug 5, 2025). ## Is your feature request related to a problem? Please describe. https://pkg.go.dev/github.com/klauspost/reedsolomon#New is used for EC encoding/decoding. It provides various options incl. optimizing ones, but we do not use any of them ## Describe the solution you'd like 1. look around the options, choose fitting ones (if any) 2. prepare benchmarks 3. make an improvement ## Additional context #3457

sami added the

enhancement

neofs-storage

performance

labels

2025-12-28 17:23:06 +00:00

sami commented

2025-12-28 17:23:06 +00:00

Author

Owner

@cthulhu-rider commented on GitHub (Sep 1, 2025):

@cthulhu-rider commented on GitHub (Sep 1, 2025): see also https://pkg.go.dev/github.com/klauspost/reedsolomon#NewStream

sami commented

2025-12-28 17:23:06 +00:00

Author

Owner

@cthulhu-rider commented on GitHub (Sep 25, 2025):

explore

panic: runtime error: index out of range [3] with length 2

goroutine 349 [running]:
github.com/klauspost/reedsolomon.(*reedSolomon).reconstruct(0x0?, {0xc0011a60c0?, 0x0?, 0x4}, 0x69?, {0xc000010478?, 0x111d120?, 0xc000222001?})
	github.com/klauspost/reedsolomon@v1.12.5/reedsolomon.go:1456 +0xfc6
github.com/klauspost/reedsolomon.(*reedSolomon).ReconstructSome(0x2?, {0xc0011a60c0?, 0x0?, 0x0?}, {0xc000010478?, 0x2045e10?, 0x0?})
	github.com/klauspost/reedsolomon@v1.12.5/reedsolomon.go:1417 +0x57
github.com/nspcc-dev/neofs-node/internal/ec.DecodeRange({0x40?, 0x1a?}, 0x1, 0x1, {0xc0011a60c0, 0x4, 0x4})
	github.com/nspcc-dev/neofs-node/internal/ec/ec.go:102 +0x103
github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).copyECObjectRangeByRule(0xc0002823d8, {0x15c2890, 0xc0012c6ba0}, {0x7c2fc7056198, 0xc0012c6d50}, {0xdb, 0x2e, 0x13, 0xc4, 0x5e, ...}, ...)
	github.com/nspcc-dev/neofs-node/pkg/services/object/get/ec.go:725 +0x2afe
github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).copyECObjectRange(...)
	github.com/nspcc-dev/neofs-node/pkg/services/object/get/ec.go:377
github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).GetRange(0xc0002823d8, {0x15c2890, 0xc0012c6ba0}, {{{0x15b9240, 0xc0012c6d50}, 0xc000c95240, {{0xdb, 0x2e, 0x13, 0xc4, ...}, ...}, ...}, ...})
	github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:81 +0x847
main.(*objectSvc).GetRange(0x15c7720?, {0x15c2890?, 0xc0012c6ba0?}, {{{0x15b9240, 0xc0012c6d50}, 0xc000c95240, {{0xdb, 0x2e, 0x13, 0xc4, ...}, ...}, ...}, ...})
	github.com/nspcc-dev/neofs-node/cmd/neofs-node/object.go:97 +0x58
github.com/nspcc-dev/neofs-node/pkg/services/object.(*Server).GetRange(0xc00017afd0, 0xc000c950c0, {0x15cb730, 0xc0012824c0})
	github.com/nspcc-dev/neofs-node/pkg/services/object/server.go:1309 +0x447
github.com/nspcc-dev/neofs-sdk-go/proto/object._ObjectService_GetRange_Handler({0x132eac0, 0xc00017afd0}, {0x15c7a80, 0xc0010d44b0})
	github.com/nspcc-dev/neofs-sdk-go@v1.0.0-rc.14.0.20250909155603-306163ef973e/proto/object/service_grpc.pb.go:783 +0x110
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00034ea00, {0x15c2890, 0xc0012c6ae0}, 0xc0000a1b60, 0xc000194960, 0x20632e0, 0x0)
	google.golang.org/grpc@v1.75.1/server.go:1722 +0x12e8
google.golang.org/grpc.(*Server).handleStream(0xc00034ea00, {0x15c3798, 0xc0000c7a00}, 0xc0000a1b60)
	google.golang.org/grpc@v1.75.1/server.go:1846 +0xb47
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	google.golang.org/grpc@v1.75.1/server.go:1061 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 374
	google.golang.org/grpc@v1.75.1/server.go:1072 +0x11d

this occured when ReconstructSome() is called with required = make([]true, rule.DataPartNum). Although this should not be a problem. Passing make([]true, rule.DataPartNum+rule.ParityPartNum) fixes. Lib bug?

@cthulhu-rider commented on GitHub (Sep 25, 2025): explore ``` panic: runtime error: index out of range [3] with length 2 goroutine 349 [running]: github.com/klauspost/reedsolomon.(*reedSolomon).reconstruct(0x0?, {0xc0011a60c0?, 0x0?, 0x4}, 0x69?, {0xc000010478?, 0x111d120?, 0xc000222001?}) github.com/klauspost/reedsolomon@v1.12.5/reedsolomon.go:1456 +0xfc6 github.com/klauspost/reedsolomon.(*reedSolomon).ReconstructSome(0x2?, {0xc0011a60c0?, 0x0?, 0x0?}, {0xc000010478?, 0x2045e10?, 0x0?}) github.com/klauspost/reedsolomon@v1.12.5/reedsolomon.go:1417 +0x57 github.com/nspcc-dev/neofs-node/internal/ec.DecodeRange({0x40?, 0x1a?}, 0x1, 0x1, {0xc0011a60c0, 0x4, 0x4}) github.com/nspcc-dev/neofs-node/internal/ec/ec.go:102 +0x103 github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).copyECObjectRangeByRule(0xc0002823d8, {0x15c2890, 0xc0012c6ba0}, {0x7c2fc7056198, 0xc0012c6d50}, {0xdb, 0x2e, 0x13, 0xc4, 0x5e, ...}, ...) github.com/nspcc-dev/neofs-node/pkg/services/object/get/ec.go:725 +0x2afe github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).copyECObjectRange(...) github.com/nspcc-dev/neofs-node/pkg/services/object/get/ec.go:377 github.com/nspcc-dev/neofs-node/pkg/services/object/get.(*Service).GetRange(0xc0002823d8, {0x15c2890, 0xc0012c6ba0}, {{{0x15b9240, 0xc0012c6d50}, 0xc000c95240, {{0xdb, 0x2e, 0x13, 0xc4, ...}, ...}, ...}, ...}) github.com/nspcc-dev/neofs-node/pkg/services/object/get/get.go:81 +0x847 main.(*objectSvc).GetRange(0x15c7720?, {0x15c2890?, 0xc0012c6ba0?}, {{{0x15b9240, 0xc0012c6d50}, 0xc000c95240, {{0xdb, 0x2e, 0x13, 0xc4, ...}, ...}, ...}, ...}) github.com/nspcc-dev/neofs-node/cmd/neofs-node/object.go:97 +0x58 github.com/nspcc-dev/neofs-node/pkg/services/object.(*Server).GetRange(0xc00017afd0, 0xc000c950c0, {0x15cb730, 0xc0012824c0}) github.com/nspcc-dev/neofs-node/pkg/services/object/server.go:1309 +0x447 github.com/nspcc-dev/neofs-sdk-go/proto/object._ObjectService_GetRange_Handler({0x132eac0, 0xc00017afd0}, {0x15c7a80, 0xc0010d44b0}) github.com/nspcc-dev/neofs-sdk-go@v1.0.0-rc.14.0.20250909155603-306163ef973e/proto/object/service_grpc.pb.go:783 +0x110 google.golang.org/grpc.(*Server).processStreamingRPC(0xc00034ea00, {0x15c2890, 0xc0012c6ae0}, 0xc0000a1b60, 0xc000194960, 0x20632e0, 0x0) google.golang.org/grpc@v1.75.1/server.go:1722 +0x12e8 google.golang.org/grpc.(*Server).handleStream(0xc00034ea00, {0x15c3798, 0xc0000c7a00}, 0xc0000a1b60) google.golang.org/grpc@v1.75.1/server.go:1846 +0xb47 google.golang.org/grpc.(*Server).serveStreams.func2.1() google.golang.org/grpc@v1.75.1/server.go:1061 +0x7f created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 374 google.golang.org/grpc@v1.75.1/server.go:1072 +0x11d ``` this occured when `ReconstructSome()` is called with `required = make([]true, rule.DataPartNum)`. Although this should not be a problem. Passing `make([]true, rule.DataPartNum+rule.ParityPartNum)` fixes. Lib bug?