Optimize node memory consumption

sami commented

2025-12-28 17:19:52 +00:00

Owner

Copy link

Originally created by @fyrchik on GitHub (May 19, 2022).

There is a number of cases where we can optimize the memory consumption.

After SDK move all IDs should be passed by value where possible
#1418
#1460
#1260
Optimize object transformer pipelines: for a single object we consume C * size(object), ideally we'd like to have C = 1, using streaming interfaces, each separate transformer could be profiled
Restrict the amount of concurrent service connections, return ErrBusy
#1483
#1501
#1473
#1482
#1516

Anything else?

Related #1319 .

Originally created by @fyrchik on GitHub (May 19, 2022). There is a number of cases where we can optimize the memory consumption. - [ ] After SDK move all `ID`s should be passed by value where possible - [x] #1418 - [x] #1460 - [ ] #1260 - [ ] Optimize object transformer pipelines: for a single object we consume `C * size(object)`, ideally we'd like to have `C = 1`, using streaming interfaces, each separate transformer could be profiled - [ ] Restrict the amount of concurrent service connections, return `ErrBusy` - [x] #1483 - [x] #1501 - [x] #1473 - [x] #1482 - [x] #1516 Anything else? Related #1319 .

sami

2025-12-28 17:19:52 +00:00

closed this issue
added the
I3

epic

U3

S3

performance
labels

sami commented

2025-12-28 17:19:53 +00:00

Author

Owner

Copy link

@carpawell commented on GitHub (Jul 4, 2022):

Anything else?

Optimize GetRange of the FSTree? Now the blobstor reads the whole object via Get method, decompresses it, and only after that cuts the desired part of an object. Get method reads all the payload in the memory even if only a small part is required.

Optimization may require moving compression on the blobovnicza and FSTree levels and decompressing the object only if it was really compressed.

There is already some kind of thoughts about compression on the bloboznicza side: nspcc-dev/neofs-node@f699e82ea7/pkg/local_object_storage/blobstor/blobovnicza.go (L607-L610)

@nspcc-dev/neofs-core

@carpawell commented on GitHub (Jul 4, 2022): > Anything else? Optimize `GetRange` of the FSTree? Now the blobstor [reads the whole object](https://github.com/nspcc-dev/neofs-node/blob/f699e82ea7c47857fdcba8190effcc3339126a6a/pkg/local_object_storage/blobstor/get_range_big.go#L33) via `Get` method, *decompresses* it, and only after that cuts the desired part of an object. `Get` method [reads](https://github.com/nspcc-dev/neofs-node/blob/f699e82ea7c47857fdcba8190effcc3339126a6a/pkg/local_object_storage/blobstor/fstree/fstree.go#L243) all the payload in the memory even if only a small part is required. Optimization may require moving compression on the blobovnicza and FSTree levels and [decompressing](https://github.com/nspcc-dev/neofs-node/blob/master/pkg/local_object_storage/blobstor/get_range_big.go#L44) the object only if it [was really compressed](https://github.com/nspcc-dev/neofs-node/blob/f699e82ea7c47857fdcba8190effcc3339126a6a/pkg/local_object_storage/blobstor/blobovnicza.go#L831). There is already some kind of thoughts about compression on the bloboznicza side: https://github.com/nspcc-dev/neofs-node/blob/f699e82ea7c47857fdcba8190effcc3339126a6a/pkg/local_object_storage/blobstor/blobovnicza.go#L607-L610 @nspcc-dev/neofs-core

sami commented

2025-12-28 17:19:53 +00:00

Author

Owner

Copy link

@roman-khimov commented on GitHub (Sep 23, 2025):

Obsolete, unspecific and largely irrelevant now to me. Many optimizations were made and we're handling memory much better now. There is still potential for improvement, but more specific issues are welcome for this.

@roman-khimov commented on GitHub (Sep 23, 2025): Obsolete, unspecific and largely irrelevant now to me. Many optimizations were made and we're handling memory much better now. There is still potential for improvement, but more specific issues are welcome for this.

No Branch/Tag specified