nspcc-dev/neofs-node

Fork 0

mirror of https://github.com/nspcc-dev/neofs-node.git synced 2026-03-01 04:29:10 +00:00

Complex object sometimes can't be fetched from certain nodes when bearer token is attached #875

New issue

Closed

opened 2025-12-28 17:20:59 +00:00 by sami · 2 comments

sami commented

2025-12-28 17:20:59 +00:00

Owner

Originally created by @alexvanin on GitHub (Nov 10, 2022).

Originally assigned to: @carpawell on GitHub.

I am trying to upload and download object with bearer token. I use wallets from neofs-dev-env:

wallets/wallet.json as container owner,
services/s3_gate/wallet.json as request sender.

Expected Behavior

All storage nodes in the system return complex object.

Current Behavior

Some container nodes return object not found error.

neofs-cli object get --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \
  --oid BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B \
  -w services/s3_gate/wallet.json \
  --bearer ./b.signed.json \
  -r s03.neofs.devenv:8080 --file ./a
Enter password > 
 1000000 / 10485760 [==========>-----------------------------------------------------]   9.54% 0s
rpc error: copy payload: status: code = 2049 message = object not found

When bearer token is not attached, object is fetched (from public container).

neofs-cli object get --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \
  --oid BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B \
  -w services/s3_gate/wallet.json \
  -r s03.neofs.devenv:8080 --file ./a
Enter password > 
 10485760 / 10485760 [===============================================================] 100.00% 0s
[./a] Object successfully saved
ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B
CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE
Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt
...

Steps to Reproduce (for bugs)

Start dev-env. I also reduce max object size to 1MB so it works faster.

make up/basic && make update.max_object_size val=1000000 && make prepare.ir

Create container where container size is bigger than number of replicas.

I used REP 2 IN X CBF 1 SELECT 3 FROM * AS X so I had:

one node out of container
three nodes in container
two replicas for each object in these three nodes

neofs-cli container create -p "REP 2 IN X CBF 1 SELECT 3 FROM * AS X" \
  --basic-acl eacl-public-read-write \
  -r s01.neofs.devenv:8080 \
  -w wallets/wallet.json

Sign bearer token for all containers. Bearer token for services/s3_gate/wallet.json is attached to this issue.

neofs-cli util sign bearer-token --from b.json --to b.signed.json --json -w wallets/wallet.json

Upload complex object into container

neofs-cli object put --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \
  --file ./10m \
  -w services/s3_gate/wallet.json \
  -r s01.neofs.devenv:8080 \
  --bearer ./b.signed.json

Try to get complex object from all available nodes using bearer token

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s01.neofs.devenv:8080 --file ./a 
Enter password > 
 0 / 10485760 [----------------------------------------------------------------]   0.00% 0s
rpc error: copy payload: status: code = 2049 message = object not found

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json  -r s02.neofs.devenv:8080 --file ./a
Enter password > 
 10485760 / 10485760 [===============================================================] 100.00% 0s
[./a] Object successfully saved
ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B
CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE
Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s03.neofs.devenv:8080 --file ./a 
Enter password > 
 1000000 / 10485760 [==========>-----------------------------------------------------]   9.54% 0s
rpc error: copy payload: status: code = 2049 message = object not found

neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s04.neofs.devenv:8080 --file ./a 
Enter password > 
 10485760 / 10485760 [===============================================================] 100.00% 0s
[./a] Object successfully saved
ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B
CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE
Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt

Context

GET requests are failed on container nodes which contain some split information.

# S01 FAIL
neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s01.neofs.devenv:8080 --ttl 1
Found 1 objects.
BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B

# S02 OK [ OUT OF CONTAINER ]
$ neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s02.neofs.devenv:8080 --ttl 1
Found 0 objects.

# S03 FAIL
neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s03.neofs.devenv:8080 --ttl 1
Found 1 objects.
BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B

# S04 OK
$ neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s04.neofs.devenv:8080 --ttl 1
Found 0 objects.

These nodes try to send inter-container requests to fetch the one of the child objects from container nodes, but they fail.

2022-11-10T15:58:07.349Z        debug   get/remote.go:14        processing node...      {"component": "Object.Get service", "request": "GET", "address": "BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE/H2zVhS6sUSdLyoaNUuAVM1fdxw4dDidfwbJFCKvjUEKx", "raw": false, "local": false, "with session": false, "with bearer": true}
2022-11-10T15:58:07.351Z        debug   get/remote.go:34        remote call failed      {"component": "Object.Get service", "request": "GET", "address": "BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE/H2zVhS6sUSdLyoaNUuAVM1fdxw4dDidfwbJFCKvjUEKx", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "read object header: status: code = 2048 message = access to object operation denied"}

See more logs in attachment below.

Node sends new requests to container nodes and attaches bearer token from original request. Bearer token integrity check fails there.

nspcc-dev/neofs-node@d8d3588e1b/pkg/services/object/acl/acl.go (L252-L255)

Possible Solution

Do not attach bearer token for inter-container communication. It should work fine considering that only container nodes are assembling the objects right now (see https://github.com/nspcc-dev/neofs-node/issues/838)

Your Environment

Version used: neofs-node v0.34.0-28-g01a226b3 (support branch)

b.json
s03-get-logs.txt

Originally created by @alexvanin on GitHub (Nov 10, 2022). Originally assigned to: @carpawell on GitHub. I am trying to upload and download object **with bearer token**. I use wallets from neofs-dev-env: - `wallets/wallet.json` as container owner, - `services/s3_gate/wallet.json` as request sender. ## Expected Behavior All storage nodes in the system return complex object. ## Current Behavior Some container nodes return `object not found` error. ``` neofs-cli object get --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \ --oid BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B \ -w services/s3_gate/wallet.json \ --bearer ./b.signed.json \ -r s03.neofs.devenv:8080 --file ./a Enter password > 1000000 / 10485760 [==========>-----------------------------------------------------] 9.54% 0s rpc error: copy payload: status: code = 2049 message = object not found ``` When bearer token **is not attached**, object is fetched (from public container). ``` neofs-cli object get --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \ --oid BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B \ -w services/s3_gate/wallet.json \ -r s03.neofs.devenv:8080 --file ./a Enter password > 10485760 / 10485760 [===============================================================] 100.00% 0s [./a] Object successfully saved ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt ... ``` ## Steps to Reproduce (for bugs) 1) Start dev-env. I also reduce max object size to 1MB so it works faster. ``` make up/basic && make update.max_object_size val=1000000 && make prepare.ir ``` 2) Create container where container size is bigger than number of replicas. I used `REP 2 IN X CBF 1 SELECT 3 FROM * AS X` so I had: - one node out of container - three nodes in container - two replicas for each object in these three nodes ``` neofs-cli container create -p "REP 2 IN X CBF 1 SELECT 3 FROM * AS X" \ --basic-acl eacl-public-read-write \ -r s01.neofs.devenv:8080 \ -w wallets/wallet.json ``` 3) Sign bearer token for all containers. Bearer token for `services/s3_gate/wallet.json` is attached to this issue. ``` neofs-cli util sign bearer-token --from b.json --to b.signed.json --json -w wallets/wallet.json ``` 4) Upload complex object into container ``` neofs-cli object put --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE \ --file ./10m \ -w services/s3_gate/wallet.json \ -r s01.neofs.devenv:8080 \ --bearer ./b.signed.json ``` 5) Try to get complex object from **all** available nodes using bearer token ``` neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s01.neofs.devenv:8080 --file ./a Enter password > 0 / 10485760 [----------------------------------------------------------------] 0.00% 0s rpc error: copy payload: status: code = 2049 message = object not found neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s02.neofs.devenv:8080 --file ./a Enter password > 10485760 / 10485760 [===============================================================] 100.00% 0s [./a] Object successfully saved ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s03.neofs.devenv:8080 --file ./a Enter password > 1000000 / 10485760 [==========>-----------------------------------------------------] 9.54% 0s rpc error: copy payload: status: code = 2049 message = object not found neofs-cli object get --cid [..] --oid [..] -w services/s3_gate/wallet.json --bearer b.signed.json -r s04.neofs.devenv:8080 --file ./a Enter password > 10485760 / 10485760 [===============================================================] 100.00% 0s [./a] Object successfully saved ID: BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B CID: BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE Owner: NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt ``` ## Context GET requests are failed on container nodes which contain some split information. ``` # S01 FAIL neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s01.neofs.devenv:8080 --ttl 1 Found 1 objects. BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B # S02 OK [ OUT OF CONTAINER ] $ neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s02.neofs.devenv:8080 --ttl 1 Found 0 objects. # S03 FAIL neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s03.neofs.devenv:8080 --ttl 1 Found 1 objects. BJitbXrk4h5mM2C9uLBvvaeuNcvj2hPnHXbarWtyHH1B # S04 OK $ neofs-cli object search --root --cid BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE -g -r s04.neofs.devenv:8080 --ttl 1 Found 0 objects. ``` These nodes try to send inter-container requests to fetch the one of the child objects from container nodes, but they fail. ``` 2022-11-10T15:58:07.349Z debug get/remote.go:14 processing node... {"component": "Object.Get service", "request": "GET", "address": "BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE/H2zVhS6sUSdLyoaNUuAVM1fdxw4dDidfwbJFCKvjUEKx", "raw": false, "local": false, "with session": false, "with bearer": true} 2022-11-10T15:58:07.351Z debug get/remote.go:34 remote call failed {"component": "Object.Get service", "request": "GET", "address": "BYnFpg7z9MEYXwaAocaoHXAcYU3ZJ8JX4CJq2ex54zpE/H2zVhS6sUSdLyoaNUuAVM1fdxw4dDidfwbJFCKvjUEKx", "raw": false, "local": false, "with session": false, "with bearer": true, "error": "read object header: status: code = 2048 message = access to object operation denied"} ``` See more logs in attachment below. Node sends **new** requests to container nodes and attaches bearer token from original request. Bearer token integrity check fails there. https://github.com/nspcc-dev/neofs-node/blob/d8d3588e1b824af37c34f16bef89e5a2af71f70b/pkg/services/object/acl/acl.go#L252-L255 ## Possible Solution Do not attach bearer token for inter-container communication. It should work fine considering that only container nodes are assembling the objects right now (see https://github.com/nspcc-dev/neofs-node/issues/838) ## Your Environment * Version used: neofs-node v0.34.0-28-g01a226b3 (support branch) [b.json](https://github.com/nspcc-dev/neofs-node/files/9862070/b.json.txt) [s03-get-logs.txt](https://github.com/nspcc-dev/neofs-node/files/9982676/s03-get-logs.txt)

sami

2025-12-28 17:20:59 +00:00

closed this issue
added the
bug

neofs-storage

U2
labels

sami commented

2025-12-28 17:21:00 +00:00

Author

Owner

@carpawell commented on GitHub (Nov 10, 2022):

@fyrchik, @acid-ant, @cthulhu-rider, depending on the "speed of the fix", solution for #838, and some other preferences i see two main possibilities:

Drop bearer token when spawning a new GET/HEAD request if a node IS sure it is a container node (in fact, i am surprised why the token is even attached);
Add a session token for every child object (or for a full container) to any GET request (strange but i do not see any other opportunity to be able to get all the parts of a big object with a bearer token).

@carpawell commented on GitHub (Nov 10, 2022): @fyrchik, @acid-ant, @cthulhu-rider, depending on the "speed of the fix", solution for #838, and some other preferences i see two main possibilities: 1. Drop bearer token when spawning a new GET/HEAD request if a node IS sure it is a container node (in fact, i am surprised why the token is even attached); 2. Add a session token for every child object (or for a full container) to any GET request (strange but i do not see any other opportunity to be able to get all the parts of a big object with a bearer token).

sami commented

2025-12-28 17:21:00 +00:00

Author

Owner

@fyrchik commented on GitHub (Nov 12, 2022):

Closed via #2045

@fyrchik commented on GitHub (Nov 12, 2022): Closed via #2045