HTTP headers vs. NeoFS attributes and associated logic #99

New issue

Open

opened 2025-12-28 17:36:09 +00:00 by sami · 1 comment

sami commented

2025-12-28 17:36:09 +00:00

Owner

Originally created by @roman-khimov on GitHub (May 25, 2023).

Suppose you're trying to make a real send.fs.neo.org, not a fake one, but the one that really uses the underlying technologies. So you set up a container that looks like this:

container ID: 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC
owner ID: NantTUz5ZATfykShmSjY5v6DW3Mqkdtm2k
basic ACL: fbfbfff (eacl-public-read-write)
       RangeHASH    Range      Search     Delete     Put        Head       Get
0 0    1 1 1 1      1 0 1 1    1 1 1 1    1 0 1 1    1 1 1 1    1 1 1 1    1 1 1 1
X F    U S O B      U S O B    U S O B    U S O B    U S O B    U S O B    U S O B
  X-Sticky F-Final U-User S-System O-Others B-Bearer
created: 2023-05-24 16:51:33 +0300 MSK
attributes:
        Timestamp=1684936293
placement policy:
REP 2 IN X
CBF 2
SELECT 2 FROM F AS X
FILTER Deployed EQ NSPCC AS F

It's sowewhat open for the basic ACL, but it's got an extended one as well that fixes it:

{
  "version": {
    "major": 2,
    "minor": 13
  },
  "containerID": {
    "value": "WjCnuT1qPenbAEvs869WjnzJuIi7A0ZEgmIKZ7bxlWE="
  },
  "records": [
    {
      "operation": "PUT",
      "action": "DENY",
      "filters": [],
      "targets": [
        {
          "role": "OTHERS",
          "keys": []
        }
      ]
    },
    {
      "operation": "DELETE",
      "action": "DENY",
      "filters": [],
      "targets": [
        {
          "role": "OTHERS",
          "keys": []
        }
      ]
    }
  ]
}

Basically, no one but the owner can do anything with it, an attempt to upload any object will fail miserably:

$ ./bin/neofs-cli object put --file README.md -g --cid 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC -r grpcs://st3.t5.fs.neo.org:8082
rpc error: client failure: status: code = 2048 message = access to object operation denied: access to operation OBJECT_PUT is denied by extended ACL check: denied by rule

And that's what we want, because our mighty neofs-oauthz will issue proper token for us with a very simple constraint:

	hashedEmail := fmt.Sprintf("%x", sha256.Sum256([]byte(email)))

	t := eacl.CreateTable(b.config.ContainerID)
	// order of rec is important
	rec := eacl.CreateRecord(eacl.ActionAllow, eacl.OperationPut)
	rec.AddObjectAttributeFilter(eacl.MatchStringEqual, "Email", hashedEmail)
	eacl.AddFormedTarget(rec, eacl.RoleOthers)
	t.AddRecord(rec)
	rec2 := eacl.CreateRecord(eacl.ActionDeny, eacl.OperationPut)
	eacl.AddFormedTarget(rec2, eacl.RoleOthers)
	t.AddRecord(rec2)

One can upload, but only if the uploaded object has an Email attribute of very specific kind (hash of the e-mail received via OAuth). All good. There is nspcc-dev/neofs-oauthz#29, but let's consider it to be fixed for the moment.

The way send.fs.neo.org works is it uploads objects via HTTP gateway (that's why we're here!), so frontend initiates upload with

		api('POST', `/gate/upload/${environment.containerID}`, formData, {
			'X-Attribute-NEOFS-Expiration-Epoch': String(Number(epoch) + Number(lifetime)),
			'Authorization': `Bearer ${user.XBearer}`,
			'X-Attribute-Email': user.XAttributeEmail,
			'Content-Type': 'multipart/form-data',
		})

Notice X-Attribute-Email and Authorization, that's exactly what HTTP gateway needs to get a bearer token and an Email attribute for the object. But what we receive as a reply for this request is not a CID/OID pair, but

could not store file in neofs: init writing on API client: client failure: status: code = 2048 message = access to object operation denied

How could it happen? If we're to extract the bearer token and email hash from the browser and try going via CLI, that'd be like:

$ ./bin/neofs-cli object put -g --cid 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC -e 15400 -r grpcs://st3.t5.fs.neo.org:8082 --file README.md --attributes Email=... --bearer brr
[README.md] Object successfully stored
  OID: EyocojwDTaBdBWK2vycVkmYFLAttAUUUsW2bjsCuqbww
  CID: 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC

And it works just fine (thanks, NeoFS). But if we're to

$ curl -F 'file=@README.md;filename=README.md' -H 'X-Attribute-Email: ...' -H 'Authorization: Bearer ...' https://http.t5.fs.neo.org/upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC
could not store file in neofs: init writing on API client: client failure: status: code = 2048 message = access to object operation denied

It still doesn't work. Now let's remember that the setup is a little more complicated than that and in fact we've got an nginx in-between the HTTP gateway and the client. It does some proxy_pass for requests with slight URL adjustments, caches GETs, so it's useful there. But it passes the headers as well.

If we're to try to push the same request to the gateway directly (right from the appropriate host):

# curl -F 'file=@/usr/bin/sed;filename=sed' -H 'X-Attribute-NEOFS-Expiration-Epoch: 6500' -H 'X-Attribute-Email: ...' -H 'Authorization: Bearer ...' http://localhost:8888/upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC

It all suddenly works again! But it's not via the nginx. So nginx can be easily blamed for this, but in fact it's still a little more complicated than that. If you're to carefully look at http-gw debug logs you may notice that the gateway doesn't use any bearer token at all for the upsteam request, no wonder it's denied. But if we're to also try tcpdump -A -i lo port 8888 to see what's going on on the wire there are expected headers there:

E..2e.@.@............."..G.k2...P....'..POST /upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC HTTP/1.1
Host: localhost:8888
Connection: close
Content-Length: 539
user-agent: curl/7.66.0
accept: */*
x-attribute-neofs-expiration-epoch: 6502
x-attribute-email: ...
authorization: Bearer ...
content-type: multipart/form-data; boundary=------------------------b402e2161cfe5a35

So why can't HTTP gateway take the token and use it? The answer to that is this attempt:

# curl -F 'file=@/usr/bin/sed;filename=sed' -H 'X-Attribute-NEOFS-Expiration-Epoch: 6500' -H 'X-Attribute-Email: ...' -H 'authorization: Bearer ...' http://localhost:8888/upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC
could not store file in neofs: init writing on API client: client failure: status: code = 2048 message = access to object operation denied

Which suddenly doesn't work at all. And the only difference with the successful one is lowercased authorization header.

RFC 2616 says:

Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive.

So one might argue that the header is OK, but there is something wrong with the l33t fasthttp server we're using. Yes and no, we set it up this way:
nspcc-dev/neofs-http-gw@8321181b29/app.go (L120)

And this knob is documented as "Header names are passed as-is without normalization if this option is set" (hi, #125 also). Why do we have it then? Well, NeoFS (unlike HTTP) has case-sensitive attributes and gateway is documented to accept them from HTTP headers:

all "X-Attribute-*" headers get converted to object attributes with "X-Attribute-" prefix stripped, that is if you add "X-Attribute-Ololo: 100500" header to your request the resulting object will get "Ololo: 100500" attribute

Obviously if you want Ololo you have to have this passed as Ololo as ololo is a different attribute.

Maybe nginx can be fixed to not change attributes? No, because it only cares about HTTP and it's absolutely compatible with the 1.1 specification while HTTP 2.0 is even more interesting:

Just as in HTTP/1.x, header field names are strings of ASCII
characters that are compared in a case-insensitive fashion. However,
header field names MUST be converted to lowercase prior to their
encoding in HTTP/2. A request or response containing uppercase
header field names MUST be treated as malformed (Section 8.1.2.6).

Our http gateway is 1.1 only, but nginx obviously works with any HTTP version and does what's more appropriate for it to do (lowercases everything). And if we're to consider switching to HTTP 2.0 then the whole attribute scheme breaks completely as we'll never receive Ololo and couldn't send X-Attribute-Ololo in response as well (like we do now).

What options do we have?

Switch DisableHeaderNamesNormalizing back to false? This seems like an easy solution, email becomes Email again, Authorization is always Authorization irrespective of how it's encoded in the request, all good. Except that the example from the README with X-Attribute-FilePath breaks immediately, it'll be Filepath, sorry.
Handle Authorization specially, iterate through all headers manually and do case-insensitive comparisons to find it? Yeah, but this may then also be required for fasthttp.HeaderDate and doesn't solve object attribute problem (we may switch to email for send.fs.neo.org, but in general the problem still remains).
Pass bearer token via cookie? May work, still date and object attributes are a problem.
Drop nginx from the scheme? Not really possible for send.fs.neo.org, it has to handle two containers and special endpoints.
Completely change the way we accept/return attributes? Current one is convenient, doing anything else may be less so. Current scheme may even work for custom software working directly with the gateway. What alternative can we have?
Anything else?

I tend to think we might do 2 as a quick fix, but this problem may eventually bite us in some other setting. Opinions are welcome.

Originally created by @roman-khimov on GitHub (May 25, 2023). Suppose you're trying to make a real send.fs.neo.org, not a fake one, but the one that really uses the underlying technologies. So you set up a container that looks like this: ``` container ID: 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC owner ID: NantTUz5ZATfykShmSjY5v6DW3Mqkdtm2k basic ACL: fbfbfff (eacl-public-read-write) RangeHASH Range Search Delete Put Head Get 0 0 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 X F U S O B U S O B U S O B U S O B U S O B U S O B U S O B X-Sticky F-Final U-User S-System O-Others B-Bearer created: 2023-05-24 16:51:33 +0300 MSK attributes: Timestamp=1684936293 placement policy: REP 2 IN X CBF 2 SELECT 2 FROM F AS X FILTER Deployed EQ NSPCC AS F ``` It's sowewhat open for the basic ACL, but it's got an extended one as well that fixes it: ``` { "version": { "major": 2, "minor": 13 }, "containerID": { "value": "WjCnuT1qPenbAEvs869WjnzJuIi7A0ZEgmIKZ7bxlWE=" }, "records": [ { "operation": "PUT", "action": "DENY", "filters": [], "targets": [ { "role": "OTHERS", "keys": [] } ] }, { "operation": "DELETE", "action": "DENY", "filters": [], "targets": [ { "role": "OTHERS", "keys": [] } ] } ] } ``` Basically, no one but the owner can do anything with it, an attempt to upload any object will fail miserably: ``` $ ./bin/neofs-cli object put --file README.md -g --cid 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC -r grpcs://st3.t5.fs.neo.org:8082 rpc error: client failure: status: code = 2048 message = access to object operation denied: access to operation OBJECT_PUT is denied by extended ACL check: denied by rule ``` And that's what we want, because our [mighty neofs-oauthz](https://github.com/nspcc-dev/neofs-oauthz) will issue proper token for us with a very simple constraint: ``` hashedEmail := fmt.Sprintf("%x", sha256.Sum256([]byte(email))) t := eacl.CreateTable(b.config.ContainerID) // order of rec is important rec := eacl.CreateRecord(eacl.ActionAllow, eacl.OperationPut) rec.AddObjectAttributeFilter(eacl.MatchStringEqual, "Email", hashedEmail) eacl.AddFormedTarget(rec, eacl.RoleOthers) t.AddRecord(rec) rec2 := eacl.CreateRecord(eacl.ActionDeny, eacl.OperationPut) eacl.AddFormedTarget(rec2, eacl.RoleOthers) t.AddRecord(rec2) ``` One can upload, but only if the uploaded object has an `Email` attribute of very specific kind (hash of the e-mail received via OAuth). All good. There is nspcc-dev/neofs-oauthz#29, but let's consider it to be fixed for the moment. The way send.fs.neo.org works is it uploads objects via HTTP gateway (that's why we're here!), so frontend initiates upload with ``` api('POST', `/gate/upload/${environment.containerID}`, formData, { 'X-Attribute-NEOFS-Expiration-Epoch': String(Number(epoch) + Number(lifetime)), 'Authorization': `Bearer ${user.XBearer}`, 'X-Attribute-Email': user.XAttributeEmail, 'Content-Type': 'multipart/form-data', }) ``` Notice `X-Attribute-Email` and `Authorization`, that's exactly what HTTP gateway needs to get a bearer token and an `Email` attribute for the object. But what we receive as a reply for this request is not a CID/OID pair, but ``` could not store file in neofs: init writing on API client: client failure: status: code = 2048 message = access to object operation denied ``` How could it happen? If we're to extract the bearer token and email hash from the browser and try going via CLI, that'd be like: ``` $ ./bin/neofs-cli object put -g --cid 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC -e 15400 -r grpcs://st3.t5.fs.neo.org:8082 --file README.md --attributes Email=... --bearer brr [README.md] Object successfully stored OID: EyocojwDTaBdBWK2vycVkmYFLAttAUUUsW2bjsCuqbww CID: 754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC ``` And it works just fine (thanks, NeoFS). But if we're to ``` $ curl -F 'file=@README.md;filename=README.md' -H 'X-Attribute-Email: ...' -H 'Authorization: Bearer ...' https://http.t5.fs.neo.org/upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC could not store file in neofs: init writing on API client: client failure: status: code = 2048 message = access to object operation denied ``` It still doesn't work. Now let's remember that the setup is a little more complicated than that and in fact we've got an nginx in-between the HTTP gateway and the client. It does some `proxy_pass` for requests with slight URL adjustments, caches `GET`s, so it's useful there. But it passes the headers as well. If we're to try to push the same request to the gateway directly (right from the appropriate host): ``` # curl -F 'file=@/usr/bin/sed;filename=sed' -H 'X-Attribute-NEOFS-Expiration-Epoch: 6500' -H 'X-Attribute-Email: ...' -H 'Authorization: Bearer ...' http://localhost:8888/upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC ``` It all suddenly works again! But it's not via the nginx. So nginx can be easily blamed for this, but in fact it's still a little more complicated than that. If you're to carefully look at `http-gw` debug logs you may notice that the gateway doesn't use any bearer token at all for the upsteam request, no wonder it's denied. But if we're to also try `tcpdump -A -i lo port 8888` to see what's going on on the wire there are expected headers there: ``` E..2e.@.@............."..G.k2...P....'..POST /upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC HTTP/1.1 Host: localhost:8888 Connection: close Content-Length: 539 user-agent: curl/7.66.0 accept: */* x-attribute-neofs-expiration-epoch: 6502 x-attribute-email: ... authorization: Bearer ... content-type: multipart/form-data; boundary=------------------------b402e2161cfe5a35 ``` So why can't HTTP gateway take the token and use it? The answer to that is this attempt: ``` # curl -F 'file=@/usr/bin/sed;filename=sed' -H 'X-Attribute-NEOFS-Expiration-Epoch: 6500' -H 'X-Attribute-Email: ...' -H 'authorization: Bearer ...' http://localhost:8888/upload/754iyTDY8xUtZJZfheSYLUn7jvCkxr79RcbjMt81QykC could not store file in neofs: init writing on API client: client failure: status: code = 2048 message = access to object operation denied ``` Which suddenly doesn't work at all. And the only difference with the successful one is lowercased `authorization` header. [RFC 2616 says](https://www.rfc-editor.org/rfc/rfc2616#section-4.2): > Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive. So one might argue that the header is OK, but there is something wrong with the l33t fasthttp server we're using. Yes and no, we set it up this way: https://github.com/nspcc-dev/neofs-http-gw/blob/8321181b2973e0a300a86ef2c4326cc8ce7f371d/app.go#L120 And this knob is [documented](https://pkg.go.dev/github.com/valyala/fasthttp#Client) as "Header names are passed as-is without normalization if this option is set" (hi, #125 also). Why do we have it then? Well, NeoFS (unlike HTTP) has case-sensitive attributes and gateway [is documented](https://github.com/nspcc-dev/neofs-http-gw#uploading) to accept them from HTTP headers: > all "X-Attribute-*" headers get converted to object attributes with "X-Attribute-" prefix stripped, that is if you add "X-Attribute-Ololo: 100500" header to your request the resulting object will get "Ololo: 100500" attribute Obviously if you want `Ololo` you have to have this passed as `Ololo` as `ololo` is a different attribute. Maybe nginx can be fixed to not change attributes? No, because it only cares about HTTP and it's absolutely compatible with the 1.1 specification while HTTP 2.0 is [even more interesting](https://www.rfc-editor.org/rfc/rfc7540#section-8.1.2): > Just as in HTTP/1.x, header field names are strings of ASCII characters that are compared in a case-insensitive fashion. However, header field names MUST be converted to lowercase prior to their encoding in HTTP/2. A request or response containing uppercase header field names MUST be treated as malformed (Section 8.1.2.6). Our http gateway is 1.1 only, but nginx obviously works with any HTTP version and does what's more appropriate for it to do (lowercases everything). And if we're to consider switching to HTTP 2.0 then the whole attribute scheme breaks completely as we'll _never_ receive `Ololo` and couldn't send `X-Attribute-Ololo` in response as well (like we do now). What options do we have? 1. Switch `DisableHeaderNamesNormalizing` back to `false`? This seems like an easy solution, `email` becomes `Email` again, `Authorization` is always `Authorization` irrespective of how it's encoded in the request, all good. Except that the example from the README with `X-Attribute-FilePath` breaks immediately, it'll be `Filepath`, sorry. 2. Handle `Authorization` specially, iterate through all headers manually and do case-insensitive comparisons to find it? Yeah, but this may then also be required for `fasthttp.HeaderDate` and doesn't solve object attribute problem (we may switch to `email` for send.fs.neo.org, but in general the problem still remains). 3. Pass bearer token via cookie? May work, still date and object attributes are a problem. 4. Drop nginx from the scheme? Not really possible for send.fs.neo.org, it has to handle two containers and special endpoints. 5. Completely change the way we accept/return attributes? Current one is convenient, doing anything else may be less so. Current scheme may even work for custom software working directly with the gateway. What alternative can we have? 6. Anything else? I tend to think we might do 2 as a quick fix, but this problem may eventually bite us in some other setting. Opinions are welcome.