Since the release of reactor-netty 0.9.0.RELEASE, both the server and the client will now collect several groups of metrics (TCP/UPD/Conenction Pool etc). These are enabled by invoking a .metrics() on the server/client.
This is one of the related issue on reactor-netty (https://github.com/reactor/reactor-netty/issues/797)
Should we update the configuration of the netty web server to enable these metrics by default? Is there a way to enable these metrics by default as well on all web clients?
The default metrics library used by reactor-netty is micrometer.
HttpServer Docs: https://projectreactor.io/docs/netty/release/api/index.html?reactor/netty/http/server/HttpServer.html TcpClient Docs: https://projectreactor.io/docs/netty/release/api/index.html?reactor/netty/tcp/TcpClient.html HttpClient Docs: https://projectreactor.io/docs/netty/release/api/reactor/netty/http/client/HttpClient.html
Comment From: bclozel
Note: we should wait for Reactor Netty 0.9.3.RELEASE, because of reactor/reactor-netty#931 and reactor/reactor-netty#926.
Comment From: violetagg
@bclozel Currently we have maximumAllowableTags for uri by default 100
Do you need a way for customising it?
https://github.com/reactor/reactor-netty/blob/master/src/main/java/reactor/netty/http/MicrometerHttpMetricsRecorder.java#L53
Comment From: michaelmcfadyen
For reference, the existing client and server metrics created by spring boot have a max uri tag value that also defaults to 100. The developer can then override this value via properties.
If we're following the same pattern here, then there would need to be a way of customising that value.
Comment From: violetagg
@michaelmcfadyen https://github.com/reactor/reactor-netty/issues/950
Comment From: michaelmcfadyen
I believe this is now unblocked by the release of reactor-netty 0.9.5.RELEASE
Comment From: michaelmcfadyen
I've been trying to think of some acceptance criteria for this issue and have came up with the following.
HttpServer: - metrics enabled by default (maybe using a NettyServerCustomizer) - can be disabled via an config prop - NettyServerCustomizer will be auto configured or not based on the config prop
HttpClient: - unsure if this can be defaulted in spring boot as WebClient.Builder and ReactorClientHttpConnector are in spring framework. These classes control the default configuration of the netty HttpClient.
Thoughts?
Comment From: michaelmcfadyen
Noticed this issue raised around a memory issue metrics in reactor netty. This may be another blocker before integrating into spring boot. https://github.com/reactor/reactor-netty/issues/1010
Comment From: bclozel
Indeed, the current arrangement is storing data in unbounded collections and using tags (raw URIs) that are likely to create cardinality explosion.
Comment From: bclozel
See https://github.com/reactor/reactor-netty/issues/1075
Comment From: bclozel
I've just closed this issue, here are a few notes about the current state of metrics support.
First, we've only enabled TCP and buffer allocation metrics in Reactor Netty (so no HTTP metrics) for several reasons:
- similar metrics are already recorded at the WebFlux level (client and server) so we should not duplicate data here
- HTTP metrics are all tagged with URI tag information which causes cardinality explosion if we don't provide a function that transforms expanded URIs into templated ones. At the WebFlux level, this infrastructure doesn't make any sense - this is only useful for direct Reactor Netty users
- the two points above show that producing HTTP metrics would quickly reach limits (a metrics filter is configured to avoid memory issues) and is of limited use in the Spring Boot context
Also, metrics support in Reactor Netty is quite new and there might still be performance issues/bugs (see reactor/reactor-netty#1075). Right now, Reactor Netty does not allow to publish metrics to a custom registry, but rather to the global registry only.
Comment From: violetagg
@bclozel Please check whether the fix https://github.com/reactor/reactor-netty/issues/1076 for https://github.com/reactor/reactor-netty/issues/1075 is relevant for the integration that is committed for this issue. We are still discussing but most probably REMOTE_ADDRESS will be removed for the HTTP server metrics and also the metrics on protocol level will be disabled. I think that you enabled metrics on protocol level for the server.
Comment From: bclozel
Thanks for the heads up @violetagg
1076 doesn't seem to affect our support here; having less tags is easier than removing them once they've been published, so we're good here.
Comment From: violetagg
https://github.com/spring-projects/spring-boot/commit/956afdcf972a3af41bf580fe7a54f20a8602f8d1#diff-fc6a2fac46dd5d9f6d194074364855baR42 here you enable TCP level (i.e. protocol) metrics, with a lot of HTTP clients this will lead to a cardinality explosion as they have REMOTE_ADDRESS tag. Wdyt?
Comment From: bclozel
I thought it was fixed by #1076 already? Why cardinality explosion doesn't apply for raw TCP communications as well?
Comment From: bclozel
Reopening as it doesn't seem we can use the TCP metrics either.
Comment From: bclozel
I'm declining this enhancement feature since it seems that it's not really useful for Spring Boot applications.
On top of my previous comment, it seems that TCP metrics are only meant for raw TCP clients/servers and that there is no allocator metrics specific API (which is provided by Netty). I'm declining this enhancement as a result.