Simpler HTTP client to send requests #15

New issue

Open

opened 2025-12-28 18:11:54 +00:00 by sami · 0 comments

sami commented

2025-12-28 18:11:54 +00:00

Owner

Originally created by @roman-khimov on GitHub (Dec 13, 2020).

We're using viper at the moment and it's just made for a different problem, it manages connection pool to optimize connections to different servers while we only have one target server. It spends some time on URL parsing to see where the request should be sent, but we don't need that. It can have multiple connections to one server, while we don't need that either, we run multiple instances for that.

What we really need as a "worker" is a dumb TCP connection with minimal HTTP 1.1 handling on top of it. One worker is one permanent connection, POST and get response. Simple as that. Unfortunately, I wasn't able to find any real HTTP library for request creation/result parsing and it's very annoying. Of course our requests are rather simple and we can use some static header, but responses need to be parsed properly anyway and it's better to use some library for that.

It'd be nice to rework this part, I think it'll give us more predictable behavior (as we had some problems with workers already) and minimize bench overhead (giving more resources to nodes).

Originally created by @roman-khimov on GitHub (Dec 13, 2020). We're using viper at the moment and it's just made for a different problem, it manages connection pool to optimize connections to different servers while we only have one target server. It spends some time on URL parsing to see where the request should be sent, but we don't need that. It can have multiple connections to one server, while we don't need that either, we run multiple instances for that. What we really need as a "worker" is a dumb TCP connection with minimal HTTP 1.1 handling on top of it. One worker is one permanent connection, POST and get response. Simple as that. Unfortunately, I wasn't able to find any real HTTP library for request creation/result parsing and it's very annoying. Of course our requests are rather simple and we can use some static header, but responses need to be parsed properly anyway and it's better to use some library for that. It'd be nice to rework this part, I think it'll give us more predictable behavior (as we had some problems with workers already) and minimize bench overhead (giving more resources to nodes).