How can I attach a profiler or look at other JMX metrics of a running Flink cluster that was deployed in Kubernetes with Ververica Platform?
Note: This section applies to Flink 1.8 - 1.11 with Ververica Platform 2.0 - 2.2.
A Ververica Platform deployment, once started, will create a Flink cluster consisting of a JobManager (JM) pod and a few TaskManager (TM) pods. If you want to connect to these via JMX, e.g. from JConsole, VisualVM, or Mission Control, you need to allow these connections.
The simplest solution is to forward the JMX port from your Kubernetes pod to your local machine with
kubectl port-forward. This way makes it possible to debug any Java pod via JMX without having to publicly expose JMX via a Kubernetes service.
Change your Flink configuration
Add the following JVM options to your Deployment yaml (or set these via the UI accordingly):
spec: template: spec: flinkConfiguration: env.java.opts: >- -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Djava.rmi.server.hostname=127.0.0.1
- The same port is used as
jmxremote.rmi.port. This is needed to forward one port only.
127.0.0.1should be passed as the RMI server hostname. This is needed for the JMX connection to work via port-forwarding.
Find the pod to connect to
<deploymentId> from your deployment and the Kubernetes
<namespace> it is running in, you can retrieve all relevant Kubernetes pods via:
kubectl -n <namespace> get pods -l deploymentId=<deploymentId>
Forward the JMX port to your local machine
kubectl -n <namespace> port-forward <pod> 1099
Important: Do not redirect the port to a different local port; as JMX will verify the port and deny access.
Open JMX connection to your local port
Note: Since we have not configured SSL in this setup, you will have to allow an insecure connection.
Similarly, you can configure VisualVM, Mission Control, or the JMX tool of your choice to connect to a remote connection on