nspcc-dev/neofs-s3-gw

Fork 0

mirror of https://github.com/nspcc-dev/neofs-s3-gw.git synced 2026-03-01 04:29:15 +00:00

1G object PUT performance #541

New issue

Open

opened 2025-12-28 17:37:44 +00:00 by sami · 10 comments

sami commented

2025-12-28 17:37:44 +00:00

Owner

Originally created by @cthulhu-rider on GitHub (Oct 6, 2025).

i measured write performance using NeoFS DevEnv and AWS CLI

Current Behavior

default:

$ time aws s3api put-object --bucket bkt1 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl --key big --body ../fs/testfile_1g
{
    "ETag": "a0d1a4e7c4f884445bdd5dc069070ec41df820c90a131f732049de2acf6ba582"
}

real	2m2.932s

directly to NeoFS:

$ time neofs-cli object put -r s01.neofs.devenv:8080 -w ../devenv/wallets/wallet.json --cid 54ZdsMjxiPqv43PafNzuvVAyEDrpfzAQ2j39DYh2Grtz --file testfile_1g --timeout 2m
Enter password > 
 1073741824 / 1073741824
[testfile_1g] Object successfully stored
  OID: GUxcCexYNkrmo4gQ5ysKTh3w5qWjxgD8WLaUNDUqoWWi
  CID: 54ZdsMjxiPqv43PafNzuvVAyEDrpfzAQ2j39DYh2Grtz

real	0m54.286s

w/ slicer:

$ time aws s3api put-object --bucket bkt2 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl --key big --body ../fs/testfile_1g
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "ETag": "a0d1a4e7c4f884445bdd5dc069070ec41df820c90a131f732049de2acf6ba582"
}

real	1m18.491s

it should be noted that i cleaned up the env storage before each probe

Expected Behavior

i suggest to analyze these results to explore whether setup w/o slicer may be improved

Steps to Reproduce

issue creds for aws

neofs-auth issue-secret --wallet wallets/wallet.json  --peer s01.neofs.devenv:8080  --gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf --bearer-rules bearer.json --session-tokens sessions.json

create bucket

$ aws s3api create-bucket --bucket bkt1 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl

put object

$ time aws s3api put-object --bucket bkt2 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl --key big --body ../fs/testfile_1g

Your Environment

Version of the product used: https://github.com/nspcc-dev/neofs-s3-gw/releases/tag/v0.41.2
Server setup and configuration files used: nspcc-dev/neofs-dev-env@4cfdcaef77

$ aws --version
aws-cli/2.8.2 Python/3.9.11 Linux/6.8.0-84-generic exe/x86_64.ubuntu.22 prompt/off

Originally created by @cthulhu-rider on GitHub (Oct 6, 2025). i measured write performance using NeoFS DevEnv and AWS CLI ## Current Behavior default: ``` $ time aws s3api put-object --bucket bkt1 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl --key big --body ../fs/testfile_1g { "ETag": "a0d1a4e7c4f884445bdd5dc069070ec41df820c90a131f732049de2acf6ba582" } real 2m2.932s ``` directly to NeoFS: ``` $ time neofs-cli object put -r s01.neofs.devenv:8080 -w ../devenv/wallets/wallet.json --cid 54ZdsMjxiPqv43PafNzuvVAyEDrpfzAQ2j39DYh2Grtz --file testfile_1g --timeout 2m Enter password > 1073741824 / 1073741824 [testfile_1g] Object successfully stored OID: GUxcCexYNkrmo4gQ5ysKTh3w5qWjxgD8WLaUNDUqoWWi CID: 54ZdsMjxiPqv43PafNzuvVAyEDrpfzAQ2j39DYh2Grtz real 0m54.286s ``` w/ slicer: ``` $ time aws s3api put-object --bucket bkt2 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl --key big --body ../fs/testfile_1g urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.neofs.devenv'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings { "ETag": "a0d1a4e7c4f884445bdd5dc069070ec41df820c90a131f732049de2acf6ba582" } real 1m18.491s ``` it should be noted that i cleaned up the env storage before each probe ## Expected Behavior i suggest to analyze these results to explore whether setup w/o slicer may be improved ## Steps to Reproduce 1. issue creds for aws ``` neofs-auth issue-secret --wallet wallets/wallet.json --peer s01.neofs.devenv:8080 --gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf --bearer-rules bearer.json --session-tokens sessions.json ``` 2. create bucket ``` $ aws s3api create-bucket --bucket bkt1 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl ``` 3. put object ``` $ time aws s3api put-object --bucket bkt2 --endpoint-url https://s3.neofs.devenv:8080 --no-verify-ssl --key big --body ../fs/testfile_1g ``` ## Your Environment * Version of the product used: https://github.com/nspcc-dev/neofs-s3-gw/releases/tag/v0.41.2 * Server setup and configuration files used: https://github.com/nspcc-dev/neofs-dev-env/commit/4cfdcaef77b0f6656f76ce35c3a6c4c58b0a0ec1 ``` $ aws --version aws-cli/2.8.2 Python/3.9.11 Linux/6.8.0-84-generic exe/x86_64.ubuntu.22 prompt/off ```

sami added the

bug

performance

labels

2025-12-28 17:37:44 +00:00

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@cthulhu-rider commented on GitHub (Oct 6, 2025):

new try:

slicer 1m14s
no slicer 4m59s

may be an env issue

@cthulhu-rider commented on GitHub (Oct 6, 2025): new try: - slicer `1m14s` - no slicer `4m59s` may be an env issue

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@roman-khimov commented on GitHub (Oct 6, 2025):

@roman-khimov commented on GitHub (Oct 6, 2025): Related to https://github.com/nspcc-dev/neofs-sdk-go/issues/729

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@roman-khimov commented on GitHub (Oct 8, 2025):

@smallhive, try smaller (like 256K) buffer here first.

@roman-khimov commented on GitHub (Oct 8, 2025): @smallhive, try smaller (like 256K) buffer here first.

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@smallhive commented on GitHub (Oct 8, 2025):

Latest AIO, latest gate master. 1GB object. Each time a new container for an object
without slicer

real    1m5,610s
user    0m5,506s
sys     0m0,761s

with slicer

real    1m14,014s
user    0m5,647s
sys     0m0,569s

The buffer size has no effect

@smallhive commented on GitHub (Oct 8, 2025): Latest AIO, latest gate master. 1GB object. Each time a new container for an object without slicer ``` real 1m5,610s user 0m5,506s sys 0m0,761s ``` with slicer ``` real 1m14,014s user 0m5,647s sys 0m0,569s ``` The buffer size has no effect

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@roman-khimov commented on GitHub (Oct 8, 2025):

AIO vs dev evn? 1 vs 4 nodes?

@roman-khimov commented on GitHub (Oct 8, 2025): AIO vs dev evn? 1 vs 4 nodes?

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@smallhive commented on GitHub (Oct 8, 2025):

In some way, comparison is not fair, but for the gate, it doesn't matter how many (NeoFS) nodes it has; it uses a pool and stores objects in the same way.
I will try to check with dev-env one more time

@smallhive commented on GitHub (Oct 8, 2025): In some way, comparison is not fair, but for the gate, it doesn't matter how many (NeoFS) nodes it has; it uses a pool and stores objects in the same way. I will try to check with dev-env one more time

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@smallhive commented on GitHub (Oct 9, 2025):

$ aws --version
aws-cli/2.25.3 Python/3.12.9 Linux/5.4.0-216-generic exe/x86_64.linuxmint.20

$ docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                    PORTS     NAMES
686eaa6236d3   nspccdev/neofs-storage:0.48.3   "neofs-node --config…"   19 minutes ago   Up 19 minutes (healthy)             s04
f2232dff32a1   nspccdev/neofs-storage:0.48.3   "neofs-node --config…"   19 minutes ago   Up 19 minutes (healthy)             s01
b2797b91a2ba   nspccdev/neofs-storage:0.48.3   "neofs-node --config…"   19 minutes ago   Up 19 minutes (healthy)             s03
fe9bda604586   nspccdev/neofs-storage:0.48.3   "neofs-node --config…"   19 minutes ago   Up 19 minutes (healthy)             s02
628cca6969f5   nspccdev/neofs-ir:0.48.3        "neofs-ir --config /…"   21 minutes ago   Up 21 minutes (healthy)             ir01
7ea6365e90f3   nspccdev/neo-go:0.110.0         "/usr/bin/privnet-en…"   21 minutes ago   Up 21 minutes (healthy)             main_chain
6d99fe93bbdf   debian:10                       "/bin/sleep infinity"    21 minutes ago   Up 21 minutes                       bastion

s3 latest master
dev-env latest master

Script to upload 1GB file

export S3HOST=http://localhost:19080
export BUCKET=heh$(date +%s); echo "export BUCKET=$BUCKET"
export KEY=m$(date +%s); echo "export KEY=$KEY"

aws s3api create-bucket --bucket $BUCKET --endpoint $S3HOST
time aws s3api put-object --endpoint $S3HOST --bucket $BUCKET --key $KEY --body ./1gb.dat

REP-3, slicer off
real    1m17,015s

REP-3, slicer on
real    1m25,727s

REP-1, slicer off
real    1m14,013s

REP-1, slicer on
real    1m6,835s

@smallhive commented on GitHub (Oct 9, 2025): ```shell $ aws --version aws-cli/2.25.3 Python/3.12.9 Linux/5.4.0-216-generic exe/x86_64.linuxmint.20 ``` ```shell $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 686eaa6236d3 nspccdev/neofs-storage:0.48.3 "neofs-node --config…" 19 minutes ago Up 19 minutes (healthy) s04 f2232dff32a1 nspccdev/neofs-storage:0.48.3 "neofs-node --config…" 19 minutes ago Up 19 minutes (healthy) s01 b2797b91a2ba nspccdev/neofs-storage:0.48.3 "neofs-node --config…" 19 minutes ago Up 19 minutes (healthy) s03 fe9bda604586 nspccdev/neofs-storage:0.48.3 "neofs-node --config…" 19 minutes ago Up 19 minutes (healthy) s02 628cca6969f5 nspccdev/neofs-ir:0.48.3 "neofs-ir --config /…" 21 minutes ago Up 21 minutes (healthy) ir01 7ea6365e90f3 nspccdev/neo-go:0.110.0 "/usr/bin/privnet-en…" 21 minutes ago Up 21 minutes (healthy) main_chain 6d99fe93bbdf debian:10 "/bin/sleep infinity" 21 minutes ago Up 21 minutes bastion ``` s3 [latest master](https://github.com/nspcc-dev/neofs-s3-gw/commit/7ac95b67bac6961c1a7878563bd2427cf558e217) dev-env [latest master](https://github.com/nspcc-dev/neofs-dev-env/commit/4cfdcaef77b0f6656f76ce35c3a6c4c58b0a0ec1) Script to upload 1GB file ```shell export S3HOST=http://localhost:19080 export BUCKET=heh$(date +%s); echo "export BUCKET=$BUCKET" export KEY=m$(date +%s); echo "export KEY=$KEY" aws s3api create-bucket --bucket $BUCKET --endpoint $S3HOST time aws s3api put-object --endpoint $S3HOST --bucket $BUCKET --key $KEY --body ./1gb.dat ``` ``` REP-3, slicer off real 1m17,015s REP-3, slicer on real 1m25,727s REP-1, slicer off real 1m14,013s REP-1, slicer on real 1m6,835s ```

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@cthulhu-rider commented on GitHub (Oct 15, 2025):

s3 latest master

sad, i mentioned that i used release version. May be this affects nothing, but still

@cthulhu-rider commented on GitHub (Oct 15, 2025): > s3 [latest master](https://github.com/nspcc-dev/neofs-s3-gw/commit/7ac95b67bac6961c1a7878563bd2427cf558e217) sad, i mentioned that i used release version. May be this affects nothing, but still

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@cthulhu-rider commented on GitHub (Oct 15, 2025):

$ aws --version
aws-cli/2.8.2 Python/3.9.11 Linux/6.8.0-85-generic exe/x86_64.ubuntu.22 prompt/off

my new run:

REP 3, slicer on
real	1m55.686s

REP 3, slicer off
real	7m25.788s

@cthulhu-rider commented on GitHub (Oct 15, 2025): ``` $ aws --version aws-cli/2.8.2 Python/3.9.11 Linux/6.8.0-85-generic exe/x86_64.ubuntu.22 prompt/off ``` my new run: ``` REP 3, slicer on real 1m55.686s REP 3, slicer off real 7m25.788s ```

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@roman-khimov commented on GitHub (Oct 15, 2025):

Dropping it from 0.41.3, it requires more time for debugging and observation.

@roman-khimov commented on GitHub (Oct 15, 2025): Dropping it from 0.41.3, it requires more time for debugging and observation.