Store objects in "header+payload" format #1253

New issue

Open

opened 2025-12-28 17:22:20 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:22:20 +00:00

Owner

Originally created by @roman-khimov on GitHub (Aug 28, 2024).

Originally assigned to: @End-rey on GitHub.

I'm always frustrated when I look at how we allocate a lot of memory to fetch data from storage. #2316, #2178, #1398. #2814 also reminds about it.

Describe the solution you'd like

#1932 has some valid input for these problems, although it's localized to CLI, whereas I'm more concerned about SN. We can store/fetch objects in a bit different manner, serializing header and payload separately right at the FSTree level. Then object get operation would return a header and an io.ReaderCloser pointing to the file (if it's not too small). This can then be streamed to the outside as needed (but likely API changes are required as well).

Describe alternatives you've considered

Keep pulling MBs of data from the storage is no fun.

Originally created by @roman-khimov on GitHub (Aug 28, 2024). Originally assigned to: @End-rey on GitHub. ## Is your feature request related to a problem? Please describe. I'm always frustrated when I look at how we allocate a lot of memory to fetch data from storage. #2316, #2178, #1398. #2814 also reminds about it. ## Describe the solution you'd like #1932 has some valid input for these problems, although it's localized to CLI, whereas I'm more concerned about SN. We can store/fetch objects in a bit different manner, serializing header and payload separately right at the FSTree level. Then object get operation would return a header and an `io.ReaderCloser` pointing to the file (if it's not _too_ small). This can then be streamed to the outside as needed (but likely API changes are required as well). ## Describe alternatives you've considered Keep pulling MBs of data from the storage is no fun.