Transport of stably serialized messages over `gRPC` #68

New issue

Open

opened 2025-12-28 18:12:26 +00:00 by sami · 2 comments

sami commented

2025-12-28 18:12:26 +00:00

Owner

Originally created by @cthulhu-rider on GitHub (Mar 10, 2022).

Context

In current implementation we define type per message in NeoFS API in order to:

do not tie the transport logic to gRPC lib
implement stable serialization (Protocol Buffers with direct field order)

Given the need to serialize messages for signatures and the like, we can try to skip the extra conversion+serialization step (ToGRPCMessage methods) of the gRPC library utilities and pass the messages directly in binary form (our serialization follows Protocol Buffers).

Proposal

Research direct transmission of message types on the example of some request and pay attention on the optimality criterion:

memory overhead on extra conversion
network (and any other) overhead without an extra conversion
if the rejection of additional conversion will show a decent performance gain, how much will it be necessary to change and maintain the library API

It is worth mentioning that the results directly depend on the selected versions of the libraries: current and gRPC.

For experimentation, I propose to implement a test scenario based on a fake connection (net.Conn), which will allow you to calculate the amount of transmitted traffic (at least from the application side).

type fakeConn struct {
  net.Conn
  traffic int
}

func (x *fakeConn) Write(p []byte) (int, error) {
  n := len(p)
  x.traffic += n
  return n, nil
}

Originally created by @cthulhu-rider on GitHub (Mar 10, 2022). ## Context In current implementation we define type per message in NeoFS API in order to: - do not tie the transport logic to `gRPC` lib - implement stable serialization (Protocol Buffers with direct field order) Given the need to serialize messages for signatures and the like, we can try to skip the extra conversion+serialization step (`ToGRPCMessage` methods) of the `gRPC` library utilities and pass the messages directly in binary form (our serialization follows Protocol Buffers). ## Proposal Research direct transmission of message types on the example of some request and pay attention on the optimality criterion: 1. memory overhead on extra conversion 2. network (and any other) overhead without an extra conversion 3. if the rejection of additional conversion will show a decent performance gain, how much will it be necessary to change and maintain the library API It is worth mentioning that the results directly depend on the selected versions of the libraries: current and `gRPC`. For experimentation, I propose to implement a test scenario based on a fake connection (`net.Conn`), which will allow you to calculate the amount of transmitted traffic (at least from the application side). ``` type fakeConn struct { net.Conn traffic int } func (x *fakeConn) Write(p []byte) (int, error) { n := len(p) x.traffic += n return n, nil } ```

sami added the

discussion

labels

2025-12-28 18:12:26 +00:00

sami commented

2025-12-28 18:12:26 +00:00

Author

Owner

@fyrchik commented on GitHub (Mar 12, 2022):

This will definitely lower the memory consumption.

The simplest way to gain control over what is transmitted over wire is to implement all methods from https://github.com/protocolbuffers/protobuf-go/blob/master/runtime/protoiface/methods.go#L17 . I am not sure that this methods are always called instead of the default implementation, though. This helps to transmit stably-serialized messages. However there is still a need for ProtoReflect method and there are some Super-tricky comments for the current implementation https://github.com/protocolbuffers/protobuf-go/blob/master/internal/impl/pointer_unsafe.go#L144 ( MessageStateOf is called from ProtoReflect).

As for having some interface for transmitting raw bytes, I am not sure this is possible at all, without reimplementing parts of gRPC from scratch.

Another approach is to implement stable marshalers directly on protobuf-generated structures. I have implemented simple protobuf plugin for generating them https://github.com/fyrchik/neofs-api-go/tree/marshal . Here we lose the benefit (arguable) of not tying the transport logic to gRPC, though I am not sure I understand what it means. However, it is easily extendable, fully-automatic and can help to enforce consistent naming.

The downside is that types are not so rich, but we don't have many places where types inferred from *.proto files are inconvenient (the only case that comes to mind is that we use uint32 for enums). protoc-gen-go implementation is not big protocolbuffers/protobuf-go@fb30439f55/cmd/protoc-gen-go, so we can even fork it and add necessary code right there (stable marshaling, uint32 for enums, pointer-less slices) without an additional plugin.

@fyrchik commented on GitHub (Mar 12, 2022): This will definitely lower the memory consumption. The simplest way to gain control over what is transmitted over wire is to implement _all_ methods from https://github.com/protocolbuffers/protobuf-go/blob/master/runtime/protoiface/methods.go#L17 . I am not sure that this methods are always called instead of the default implementation, though. This helps to transmit stably-serialized messages. However there is still a need for `ProtoReflect` method and there are some `Super-tricky` comments for the current implementation https://github.com/protocolbuffers/protobuf-go/blob/master/internal/impl/pointer_unsafe.go#L144 ( `MessageStateOf` is called from `ProtoReflect`). As for having some interface for transmitting raw bytes, I am not sure this is possible at all, without reimplementing parts of gRPC from scratch. Another approach is to implement stable marshalers directly on protobuf-generated structures. I have implemented simple protobuf plugin for generating them https://github.com/fyrchik/neofs-api-go/tree/marshal . Here we lose the benefit (arguable) of not tying the transport logic to gRPC, though I am not sure I understand what it means. However, it is easily extendable, fully-automatic and can help to enforce consistent naming. The downside is that types are not so rich, but we don't have many places where types inferred from `*.proto` files are inconvenient (the only case that comes to mind is that we use `uint32` for enums). `protoc-gen-go` implementation is not big https://github.com/protocolbuffers/protobuf-go/tree/fb30439f551a7e79e413e7b4f5f4dfb58e117d73/cmd/protoc-gen-go, so we can even fork it and add necessary code right there (stable marshaling, `uint32` for enums, pointer-less slices) without an additional plugin.

sami commented

2025-12-28 18:12:26 +00:00

Author

Owner

@alexvanin commented on GitHub (Mar 17, 2022):

Another approach is to implement stable marshalers directly on protobuf-generated structures. I have implemented simple protobuf plugin for generating them https://github.com/fyrchik/neofs-api-go/tree/marshal . Here we lose the benefit (arguable) of not tying the transport logic to gRPC, though I am not sure I understand what it means. However, it is easily extendable, fully-automatic and can help to enforce consistent naming.

That is very nice! I propose to test this approach in protobuf messages of control service in NeoFS Node. This service is not a part of the API so it can be a good place to test new structures generated by forked protoc-gen-go without affecting core protocol.

@alexvanin commented on GitHub (Mar 17, 2022): > Another approach is to implement stable marshalers directly on protobuf-generated structures. I have implemented simple protobuf plugin for generating them https://github.com/fyrchik/neofs-api-go/tree/marshal . Here we lose the benefit (arguable) of not tying the transport logic to gRPC, though I am not sure I understand what it means. However, it is easily extendable, fully-automatic and can help to enforce consistent naming. That is very nice! I propose to test this approach in protobuf messages of `control` service in NeoFS Node. This service is not a part of the API so it can be a good place to test new structures generated by forked `protoc-gen-go` without affecting core protocol.