client: Cache small data chunks before PUTing them #246

New issue

Open

opened 2025-12-28 18:07:33 +00:00 by sami · 0 comments

sami commented

2025-12-28 18:07:33 +00:00

Owner

Originally created by @cthulhu-rider on GitHub (Dec 5, 2024).

https://pkg.go.dev/github.com/nspcc-dev/neofs-sdk-go/client#DefaultObjectWriter.Write sends at least one message to the underlying stream when chunk is non-empty regardless of its size. Each message requires preparation incl. pretty heavyweight cryptography. The server also verifies each message. Even besides app processing, in general, the more messages the slower the stream

thus, unintentional transition to granular chunks (e.g. when using io.Copy) can significantly slow down the process

Describe the solution you'd like

bufferize smaller chunks, and flush the buffer to the stream when it's full. This would minimize the number of messages

the disadvantage of this approach will be an additional buffer for each operation. Diff with the current approach will require careful benchmark testing

there is also a nuance with dynamic buffer allocation: how to know if the intended data volume (unknown in general) is less than the limit so to not overallocated the buffer? In special cases, we can make a hypothesis:

if the object is sewn, look at the payload length
add an initialization option (which ofc will not be used by everyone)

Describe alternatives you've considered

keep as is
leave the number of messages, but remove the signing of chunk requests. This will improve performance, but will require an alternative integrity check (e.g. checksum)
expose recommended buffer size for users so some of them (but not all ofc) could stream their data the most optimal way

Additional context

Client testing

Originally created by @cthulhu-rider on GitHub (Dec 5, 2024). ## Is your feature request related to a problem? Please describe. https://pkg.go.dev/github.com/nspcc-dev/neofs-sdk-go/client#DefaultObjectWriter.Write sends at least one message to the underlying stream when chunk is non-empty regardless of its size. Each message requires preparation incl. pretty heavyweight cryptography. The server also verifies each message. Even besides app processing, in general, the more messages the slower the stream thus, unintentional transition to granular chunks (e.g. when using `io.Copy`) can significantly slow down the process ## Describe the solution you'd like bufferize smaller chunks, and flush the buffer to the stream when it's full. This would minimize the number of messages the disadvantage of this approach will be an additional buffer for each operation. Diff with the current approach will require careful benchmark testing there is also a nuance with dynamic buffer allocation: how to know if the intended data volume (unknown in general) is less than the limit so to not overallocated the buffer? In special cases, we can make a hypothesis: - if the object is sewn, look at the payload length - add an initialization option (which ofc will not be used by everyone) ## Describe alternatives you've considered 0. keep as is 1. leave the number of messages, but remove the signing of chunk requests. This will improve performance, but will require an alternative integrity check (e.g. checksum) 2. expose recommended buffer size for users so some of them (but not all ofc) could stream their data the most optimal way ## Additional context `Client` testing