I seem to have performance problems with Ververica Platform or see unexpected CPU spikes in the Ververica Platform pod. How can I connect a profiler/JMX to the containers in this pod to get a CPU sampling snapshot for diagnosis?
Note: This section applies to Ververica Platform 2.3-2.6.
In order to connect JMX to a container inside the Ververica Platform pod just like connecting JMX to the Flink cluster, you need to set a few properties in the JVM to accept JMX connections. These properties, however, cannot be set by the helm template, so we have to employ some manual work to get it done.
Add JMX settings
In order to configure the JVMs used by Ververica Platform, you can pass custom Java options via the `JAVA_TOOL_OPTIONS` environment variable and configure that in the Ververica Platform deployment resource. In the example below, we do so for the `appmanager` container, but if you suspect your problem in any of the other containers, they can be changed accordingly.
- Find the deployment to configure:
kubectl -n <namespace> get deployments -l component=ververica-platform
- Set the `JAVA_TOOL_OPTIONS` environment variable to allow unauthenticated, unencrypted JMX connections (don't worry, we won't expose these ports publicly):
kubectl -n <namespace> set env deployments <deployment> -c appmanager \Note: If you want to introspect more than one container at a time, you should use a separate port for each of them.
Find the pod to connect to
If you don't know the Ververica Platform pod name yet, you can find it via
kubectl -n <namespace> get pods -l component=ververica-platform
You should actually see it restarting due to the change above. If It isn't, you can restart manually via
kubectl -n <namespace> delete pod <pod>
Forward the JMX port to your local machine
With the pod name from above, you can forward the port that you configured for JMX:
kubectl -n <namespace> port-forward <pod> 1099
Open JMX connection to your local port
Note: Since we have not configured SSL in this setup, you will have to allow an insecure connection.
Similarly, you can configure VisualVM, Mission Control, or the JMX tool of your choice to connect to a remote connection on
127.0.0.1:1099 and gather statistics and CPU samples for debugging.