Fast object upload policy in gateway / per request #285

New issue

Closed

opened 2025-12-28 17:36:51 +00:00 by sami · 2 comments

sami commented

2025-12-28 17:36:51 +00:00

Owner

Originally created by @alexvanin on GitHub (Jul 18, 2022).

Usually NeoFS containers have placement policies with replica value > 1. It doesn't make much sense to use single replica, because NeoFS can't provide reliable storage this way (no erasure codes yet, so the system stores full object replicas).

When client sends the data to storage node, by default it waits until storage node store all replicas according to placement policy. This behaviour can be changed by setting copies_number parameter in the request. Setting this value to 1 allows client to wait only for one successful replica upload. Storage node will replicate the object asynchronously later.

It allows to increase object upload rate, but decrease reliability: storage node may lose the data before it will be replicated, which gonna lead to data loss. However, sometimes it worth the risk, especially for pure benchmarks.

Allow S3 Gateway clients to switch between reliable and fast upload.

Gateway setting

Client setting

https://github.com/nspcc-dev/neofs-s3-gw/issues/637

/cc @realloc @KirillovDenis @masterSplinter01

Originally created by @alexvanin on GitHub (Jul 18, 2022). Usually NeoFS containers have placement policies with replica value > 1. It doesn't make much sense to use single replica, because NeoFS can't provide reliable storage this way (no erasure codes yet, so the system stores full object replicas). When client sends the data to storage node, by default it waits until storage node store all replicas according to placement policy. This behaviour can be changed by setting [copies_number](https://github.com/nspcc-dev/neofs-api/blob/10f18e4b69ec79991b1a2b83e212d99203f3ee1a/object/service.proto#L304-L306) parameter in the request. Setting this value to 1 allows client to wait only for one successful replica upload. Storage node will replicate the object asynchronously later. It allows to increase object upload rate, but decrease reliability: storage node may lose the data before it will be replicated, which gonna lead to data loss. However, sometimes it worth the risk, especially for pure benchmarks. --- Allow S3 Gateway clients to switch between reliable and fast upload. ## Gateway setting - [x] https://github.com/nspcc-dev/neofs-sdk-go/issues/302 - [x] https://github.com/nspcc-dev/neofs-s3-gw/issues/634 ## Client setting - [x] https://github.com/nspcc-dev/neofs-s3-gw/issues/637 /cc @realloc @KirillovDenis @masterSplinter01

sami

2025-12-28 17:36:51 +00:00

closed this issue
added the
U3

epic
labels

sami commented

2025-12-28 17:36:52 +00:00

Author

Owner

@KirillovDenis commented on GitHub (Jul 18, 2022):

We can use user-defined metadata https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata.

in sdk (e.g. java) it will look like (use Builder.metadata):

Map<String,String> metadata = new HashMap<String,String>();
metadata.put("key", "value");

PutObjectRequest putObjectRequest = PutObjectRequest.builder()
    .bucket("bucketName")
    .key("obj")
    .metadata(metadata)
    .build();

@KirillovDenis commented on GitHub (Jul 18, 2022): We can use user-defined metadata https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata. in sdk (e.g. java) it will look like (use [Builder.metadata](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/PutObjectRequest.Builder.html#metadata-java.util.Map-)): ```java Map<String,String> metadata = new HashMap<String,String>(); metadata.put("key", "value"); PutObjectRequest putObjectRequest = PutObjectRequest.builder() .bucket("bucketName") .key("obj") .metadata(metadata) .build(); ```

sami commented

2025-12-28 17:36:52 +00:00

Author

Owner

@alexvanin commented on GitHub (Sep 1, 2022):

All done

@alexvanin commented on GitHub (Sep 1, 2022): All done