Improve combined fstree format #1440

New issue

Open

opened 2025-12-28 17:23:00 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:23:00 +00:00

Owner

Originally created by @roman-khimov on GitHub (Jun 25, 2025).

I'm always frustrated when reads from FSTree take more time than they could in case of combined format. We do not have any OID: [offset, size] map there, so reader has to Seek() through to get the object he needs. For 128 objects in a pack it could take some time.

Describe the solution you'd like

A table can be stored at the end of the file with this data, it's trivial to do for writer and then reader can:

open file, read a chunk
try the first object, try others already read, if found --- OK
if not, seek to length-MAX_TABLE_SIZE of the file
read data
find the number of entries stored in the last bytes
jump backwards through the data to find the object needed
Seek to it directly, read

This works for files generated on-the-fly from processed PUT requests. But if we have a write cache and use direct batched writing this can be further optimized by placing the table at the beginning of the file (format should support both). Then reader can get and use it immediately after initial read.

Describe alternatives you've considered

Not known.

Additional context

https://github.com/nspcc-dev/neofs-node/pull/2814
https://github.com/nspcc-dev/neofs-node/issues/2925

Originally created by @roman-khimov on GitHub (Jun 25, 2025). ## Is your feature request related to a problem? Please describe. I'm always frustrated when reads from FSTree take more time than they could in case of combined format. We do not have any `OID: [offset, size]` map there, so reader has to `Seek()` through to get the object he needs. For 128 objects in a pack it could take some time. ## Describe the solution you'd like A table can be stored at the end of the file with this data, it's trivial to do for writer and then reader can: * open file, read a chunk * try the first object, try others already read, if found --- OK * if not, seek to length-MAX_TABLE_SIZE of the file * read data * find the number of entries stored in the last bytes * jump backwards through the data to find the object needed * Seek to it directly, read This works for files generated on-the-fly from processed PUT requests. But if we have a write cache and use direct batched writing this can be further optimized by placing the table at the beginning of the file (format should support both). Then reader can get and use it immediately after initial read. ## Describe alternatives you've considered Not known. ## Additional context https://github.com/nspcc-dev/neofs-node/pull/2814 https://github.com/nspcc-dev/neofs-node/issues/2925