nspcc-dev/neofs-node

Fork

You've already forked neofs-node

mirror of https://github.com/nspcc-dev/neofs-node.git synced 2026-03-01 04:29:10 +00:00

Code Issues 189 Projects Packages Wiki Activity

Add an option to disable homomorhic hashing in storage nodes #510

New issue

Closed

opened 2025-12-28 17:19:42 +00:00 by sami · 11 comments

sami commented

2025-12-28 17:19:42 +00:00

Owner

Copy link

Originally created by @alexvanin on GitHub (Apr 6, 2022).

Originally assigned to: @carpawell on GitHub.

There is a need to toggle calculation of objects' payload homomorphic hash in NeoFS environment. Disabling can speed up object formation.

Solution

API

add bool homomorphic_hashing_disabled to Container message of the NeoFS API V2 protocol
create constant key to the corresponding value in network configuration stored in Morph chain (can it be defined in contract repo and imported to SDK from its packages?)
add function to modify container according to global network configuration
func (*Container) ApplyNetworkConfig(netmap.NetworkConfig)
add function to assert network configuration for the container
func (Container) AssertNetworkConfig(netmap.NetworkConfig) bool
add function to sync container with network state using API client
func SyncContainerWithNetwork(context.Context, *container.Container, *Client) error
P.S. to begin with, I propose leaving Client.ContainerPut to accept prepared Container structure, and perform all preparation stages outside.

Inner ring

Container approval

make checkPutContainer method of Container processor to:

receive network config using Morph client
assert network configuration for the stored container using AssertNetworkConfig
return an error on false return

Data audit

make data audit routine to not validate homomorphic hash of the objects if homomorphic_hashing_disabled flag is set in the container of the storage group under audit (maybe Context is the best place to store this info).

Storage node

make payloadSizeLimiter to calculate homomorphic hash of the payload if homomorphic_hashing_disabled is set
make Object Put service to pass homomorphic_hashing_disabled to payloadSizeLimiter from the Container of the request context (as an optimization we can try to provide a quick access to the container for Object service e.g. from the context.Context)
do not validate homomorphic hash during object validation if homomorphic_hashing_disabled is set in the corresponding container (maybe this isn't checked at all)

NeoFS CLI

make container create command to call SyncContainerWithNetwork for the container before storing it
make storagegroup put command to not calculate homomorphic hash field if flag is set.

Originally created by @alexvanin on GitHub (Apr 6, 2022). Originally assigned to: @carpawell on GitHub. There is a need to toggle calculation of objects' payload homomorphic hash in NeoFS environment. Disabling can speed up object formation. ## Solution ### API - [x] add `bool homomorphic_hashing_disabled ` to [Container](https://github.com/nspcc-dev/neofs-api/blob/650c367529c6fe6b215ea8d5f183b07b4ac8f9a9/container/types.proto#L15) message of the NeoFS API V2 protocol - [x] create constant key to the corresponding value in network configuration stored in Morph chain (can it be defined in contract repo and imported to SDK from its packages?) - [x] add function to modify container according to global network configuration `func (*Container) ApplyNetworkConfig(netmap.NetworkConfig)` - [x] add function to assert network configuration for the container `func (Container) AssertNetworkConfig(netmap.NetworkConfig) bool` - [x] add function to sync container with network state using API client `func SyncContainerWithNetwork(context.Context, *container.Container, *Client) error` P.S. to begin with, I propose leaving `Client.ContainerPut` to accept prepared `Container` structure, and perform all preparation stages outside. ### Inner ring #### Container approval - [x] make [checkPutContainer](https://github.com/nspcc-dev/neofs-node/blob/1fc9351f4feb83ba90fca247fd8ac77865caf5a2/pkg/innerring/processors/container/process_container.go#L65) method of Container processor to: 1. receive network config using Morph client 2. assert network configuration for the stored container using `AssertNetworkConfig` 3. return an error on false return #### Data audit - [x] make data audit [routine](https://github.com/nspcc-dev/neofs-node/blob/606dfa341437795b56a46bb6234b3a017cc5fff8/pkg/services/audit/auditor/por.go#L38) to not validate homomorphic hash of the objects if `homomorphic_hashing_disabled` flag is set in the container of the storage group under audit (maybe `Context` is the best place to store this info). ### Storage node - [x] make [payloadSizeLimiter](https://github.com/nspcc-dev/neofs-node/blob/1fc9351f4feb83ba90fca247fd8ac77865caf5a2/pkg/services/object_manager/transformer/transformer.go#L131) to calculate homomorphic hash of the payload if `homomorphic_hashing_disabled` is set - [x] make Object Put [service](https://github.com/nspcc-dev/neofs-node/blob/606dfa341437795b56a46bb6234b3a017cc5fff8/pkg/services/object/put/streamer.go#L91) to pass `homomorphic_hashing_disabled` to `payloadSizeLimiter` from the `Container` of the request context (as an optimization we can try to provide a quick access to the container for Object service e.g. from the `context.Context`) - [x] do not validate homomorphic hash during object validation if `homomorphic_hashing_disabled` is set in the corresponding container (maybe this isn't checked at all) ### NeoFS CLI - [x] make container [create](https://github.com/nspcc-dev/neofs-node/blob/606dfa341437795b56a46bb6234b3a017cc5fff8/cmd/neofs-cli/modules/container.go#L139) command to call `SyncContainerWithNetwork` for the container before storing it - [x] make storagegroup [put](https://github.com/nspcc-dev/neofs-node/blob/606dfa341437795b56a46bb6234b3a017cc5fff8/cmd/neofs-cli/modules/storagegroup.go#L163) command to not calculate homomorphic hash field if flag is set.

sami

2025-12-28 17:19:42 +00:00

closed this issue
added the
enhancement

neofs-cli

neofs-storage

neofs-ir

U2

epic
labels

sami commented

2025-12-28 17:19:42 +00:00

Author

Owner

Copy link

@fyrchik commented on GitHub (Apr 6, 2022):

What does this option do, exactly? For example, when I put an object via such node with a session token, would it be put without homomorphic hash?

@fyrchik commented on GitHub (Apr 6, 2022): What does this option do, exactly? For example, when I put an object via such node with a session token, would it be put without homomorphic hash?

sami commented

2025-12-28 17:19:42 +00:00

Author

Owner

Copy link

@alexvanin commented on GitHub (Apr 6, 2022):

There are two primary use cases of the tz-hash lib in NeoFS:

calculate homomorphic hash at object.Put request (when receiving incomplete object),
use homomorphic properties during data audit in IR.

I think both should be disabled if we consider global network option.

@alexvanin commented on GitHub (Apr 6, 2022): There are two primary use cases of the tz-hash lib in NeoFS: 1. calculate homomorphic hash at object.Put request (when receiving incomplete object), 2. use homomorphic properties during data audit in IR. I think both should be disabled if we consider global network option.

sami commented

2025-12-28 17:19:42 +00:00

Author

Owner

Copy link

@cthulhu-rider commented on GitHub (Apr 7, 2022):

If network config, then static, right? Or at least it should be per-epoch value in order to prevent undefined behaviors.

It can also be a container config.

UPD: network config can't be static same as any other data stored in smart contract by design.

@cthulhu-rider commented on GitHub (Apr 7, 2022): If network config, then static, right? Or at least it should be per-epoch value in order to prevent undefined behaviors. It can also be a container config. UPD: network config can't be static same as any other data stored in smart contract by design.

sami commented

2025-12-28 17:19:42 +00:00

Author

Owner

Copy link

@alexvanin commented on GitHub (Apr 7, 2022):

Container is immutable so it makes sense, network configuration might be changed any time.

@alexvanin commented on GitHub (Apr 7, 2022): Container is immutable so it makes sense, network configuration might be changed any time.

sami commented

2025-12-28 17:19:43 +00:00

Author

Owner

Copy link

@realloc commented on GitHub (Apr 12, 2022):

I'd suggest using both network settings and container attribute. When network setting is toggled, it affects the attribute set in container.

@realloc commented on GitHub (Apr 12, 2022): I'd suggest using both network settings and container attribute. When network setting is toggled, it affects the attribute set in container.

sami commented

2025-12-28 17:19:43 +00:00

Author

Owner

Copy link

@carpawell commented on GitHub (Apr 13, 2022):

@realloc, what do you mean by saying "affects the attribute set in container"? Literally changing its value?

@carpawell commented on GitHub (Apr 13, 2022): @realloc, what do you mean by saying "affects the attribute set in container"? Literally changing its value?

sami commented

2025-12-28 17:19:43 +00:00

Author

Owner

Copy link

@realloc commented on GitHub (Apr 13, 2022):

@realloc, what do you mean by saying "affects the attribute set in container"? Literally changing its value?

Doesn't allow to create a container with the attribute conflicting current network settings. For example if there is network setting DisableTZHash=True, then all containers without DisableTZHash=True attribute considered invalid.

@realloc commented on GitHub (Apr 13, 2022): > @realloc, what do you mean by saying "affects the attribute set in container"? Literally changing its value? Doesn't allow to create a container with the attribute conflicting current network settings. For example if there is network setting `DisableTZHash=True`, then all containers without `DisableTZHash=True` attribute considered invalid.

sami commented

2025-12-28 17:19:43 +00:00

Author

Owner

Copy link

@alexvanin commented on GitHub (Apr 13, 2022):

@realloc can you elaborate the concept of invalid containers? As for now, we check container validity in Alphabet nodes during container creation. Therefore container always considered as valid after approval.

Does the scheme stays the same in your proposal? Or you consider having invalid containers after creation (created before DisableTZHash setting toggle). If so, then describe how storage and inner ring nodes should behave with requests of invalid containers.

@alexvanin commented on GitHub (Apr 13, 2022): @realloc can you elaborate the concept of _invalid_ containers? As for now, we check container validity in Alphabet nodes during container creation. Therefore container _always considered as valid_ after approval. Does the scheme stays the same in your proposal? Or you consider having _invalid_ containers after creation (created before `DisableTZHash` setting toggle). If so, then describe how storage and inner ring nodes should behave with requests of _invalid_ containers.

sami commented

2025-12-28 17:19:43 +00:00

Author

Owner

Copy link

@realloc commented on GitHub (Apr 13, 2022):

The proposed scheme just adds a step at IR container validity check. All containers created should be valid, but TZHash operations must be applied to their objects according to the attribute settings.

@realloc commented on GitHub (Apr 13, 2022): The proposed scheme just adds a step at IR container validity check. All containers created should be valid, but TZHash operations must be applied to their objects according to the attribute settings.

sami commented

2025-12-28 17:19:43 +00:00

Author

Owner

Copy link

@cthulhu-rider commented on GitHub (Apr 14, 2022):

using both network settings and container attribute

Final solution.

IR denies container if flag differs with network config
SN processes container flag only
client side sets flag according to NetworkInfo rpc response

@cthulhu-rider commented on GitHub (Apr 14, 2022): > using both network settings and container attribute Final solution. - IR denies container if flag differs with network config - SN processes container flag only - client side sets flag according to `NetworkInfo` rpc response

sami commented

2025-12-28 17:19:43 +00:00

Author

Owner

Copy link

@cthulhu-rider commented on GitHub (May 16, 2022):

Implemented in https://github.com/nspcc-dev/neofs-node/tree/feat/optional-homomorphic-hashing.

@cthulhu-rider commented on GitHub (May 16, 2022): Implemented in https://github.com/nspcc-dev/neofs-node/tree/feat/optional-homomorphic-hashing.

No Branch/Tag specified

master

put-ants-pool

lens-meta-resync

policer-boost-mode

fix-shard-evacuation-for-ec

fstree-head-buffer

initial-placement

test-metadata-contract

head-proxy-buffer

feat/meta-data-native-contract

local-head-buffer

head-local-bin

test/notary-lock

feat/meta-side-chain

notary-set-attribute

search-1000

ec-put-parallel

fix/incorrect-TS-placement

new-object-structure

get-forwarding

test-bolt

logs

decrease-prototick

bugfix/listen-maxobjsize-changes

2684-api-server-optimize-sig-verify

v0.51.1

v0.51.0

v0.50.2

v0.50.1

v0.50.0

v0.49.1

v0.49.0

v0.48.3

v0.48.2

v0.48.1

v0.48.0

v0.47.1

v0.47.0

v0.46.1

v0.46.0

v0.45.2

v0.45.1

v0.45.0

v0.44.2

v0.44.1

v0.44.0

v0.43.0

v0.42.1

v0.42.0

v0.41.1

v0.41.0

v0.40.1

v0.40.0

v0.39.2

v0.39.1

v0.39.0

v0.38.1

v0.38.0

v0.37.0

v0.36.1

v0.36.0

v0.35.0

v0.34.0

v0.33.0

v0.32.0

v0.31.0

v0.30.2

v0.30.1

v0.30.0

v0.29.0

v0.28.3

v0.28.2

v0.28.1

v0.28.0

v0.28.0-rc.3

v0.27.7

v0.27.6

v0.28.0-rc.2

v0.28.0-rc.1

v0.27.5

v0.27.4

v0.27.3

v0.27.2

v0.27.1

v0.27.0

v0.27.0-rc.1

v0.26.1

v0.26.0

v0.25.1

v0.25.0

v0.24.1

v0.24.0

v0.23.1

v0.23.0

v0.22.3

v0.22.2

v0.22.1

v0.22.0

v0.21.1

v0.21.0

v0.20.0

v0.19.0

v0.18.0

v0.17.0

v0.16.0

v0.15.0

v0.14.3

v0.14.2

v0.14.1

v0.14.0

v0.14.0-rc.1

v0.13.2

v0.13.1

v0.13.0

v0.13.0-rc.1

v0.12.1

v0.12.0

v0.12.0-rc3

v0.12.0-rc2

v0.12.0-rc1

v0.11.0

v0.10.0

Labels

Clear labels

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

nspcc-dev/neofs-node#510

Reference in a new issue

Repository

nspcc-dev/neofs-node

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns