Skip to main content

How to Connect a Profiler/JMX to Ververica Platform containers

Permanently deleted user
  • Updated

Question

I seem to have performance problems with Ververica Platform or see unexpected CPU spikes in the Ververica Platform pod. How can I connect a profiler/JMX to the containers in this pod to get a CPU sampling snapshot for diagnosis?

Answer

Note: This section applies to Ververica Platform 2.3 or later.

In order to connect JMX to a container inside the Ververica Platform pod just like connecting JMX to the Flink cluster, you need to set a few properties in the JVM to accept JMX connections. These properties, however, cannot be set by the helm template, so we have to employ some manual work to get it done.

Add JMX settings

In order to configure the JVMs used by Ververica Platform, you can pass custom Java options via the `JAVA_TOOL_OPTIONS` environment variable and configure that in the Ververica Platform deployment resource. In the example below, we do so for the `appmanager` container, but if you suspect your problem in any of the other containers, they can be changed accordingly.

  1. Find the deployment to configure: 
    kubectl -n <namespace> get deployments -l component=ververica-platform
  2. Set the `JAVA_TOOL_OPTIONS` environment variable to allow unauthenticated, unencrypted JMX connections (don't worry, we won't expose these ports publicly): 
    kubectl -n <namespace> set env deployments <deployment> -c appmanager \
      JAVA_TOOL_OPTIONS="-Dcom.sun.management.jmxremote \
        -Dcom.sun.management.jmxremote.authenticate=false \
        -Dcom.sun.management.jmxremote.ssl=false \
        -Dcom.sun.management.jmxremote.local.only=false \
        -Dcom.sun.management.jmxremote.port=1099 \
        -Dcom.sun.management.jmxremote.rmi.port=1099 \
        -Djava.rmi.server.hostname=127.0.0.1"
    Note: If you want to introspect more than one container at a time, you should use a separate port for each of them.

Find the pod to connect to

If you don't know the Ververica Platform pod name yet, you can find it via

kubectl -n <namespace> get pods -l component=ververica-platform

You should actually see it restarting due to the change above. If It isn't, you can restart manually via

kubectl -n <namespace> delete pod <pod>

Forward the JMX port to your local machine

With the pod name from above, you can forward the port that you configured for JMX:

kubectl -n <namespace> port-forward <pod> 1099

Open JMX connection to your local port

jconsole 127.0.0.1:1099

Note: Since we have not configured SSL in this setup, you will have to allow an insecure connection.

Similarly, you can configure VisualVM, Mission Control, or the JMX tool of your choice to connect to a remote connection on 127.0.0.1:1099 and gather statistics and CPU samples for debugging.

 

Related Information

Related articles