Question
How do I run Flink Jobs in Ververica Platform using custom images?
Answer
Note: This applies to Ververica Platform 2.0 and later.
In order to use your custom images as default images for an Apache Flink version and to configure the web user interface accordingly, you need to update the Flink version metadata in the platform’s configuration.
Warning: The internal structure of the official images is not considered a public, stable API and may change in the future.
Warning: For custom images, we cannot guarantee compatibility between all possible base images, custom dependencies and Ververica Platform components. Therefore, we are only able to support custom images on a best-effort basis. Application Manager and Apache Flink themselves, as well as their integration with Kubernetes, remain supported.
You may want to build your own Docker images of Ververica Platform components. In most cases, it will suffice to base your own images on the official Ververica Platform images. If this is not an option (e.g. if you require different base images), we recommend using Docker’s multi-stage build feature to extract the Ververica Platform components from the official Ververica Platform images. Access to the official registry is only needed during build time.
Examples
Application Manager
The following Dockerfile specifies a CentOS 8 based image for Ververica Platform 2.5. Note that the platform uses uid 999 and gid 999 by default. Certain upstream container images may conflict with this.
FROM centos:centos8 RUN set -ex; \ mkdir -p /vvp && chown 999:999 /vvp; RUN yum install -y java-11-openjdk COPY --from=registry.ververica.com/v2.5/vvp-appmanager:2.5.0 /vvp /vvp USER 999:999 VOLUME ["/vvp/etc", "/vvp/data", "/vvp/secrets"]
CMD ["java", "-XX:MaxRAMPercentage=50.0", "-XX:MaxMetaspaceSize=256m", "-Djava.security.egd=file:/dev/./urandom", "-cp", "/vvp/app:/vvp/app/lib/*", "com.dataartisans.appmanager.apiserver.AppManagerApplication"]
Note: The default configuration included in the package at `/vvp/etc`. Ververica Platform persist its data to the directory `/vvp/data`. On systems such as Kubernetes, you must ensure this directory is a mounted persistent volume.
Warning: Ververica Platform requires `libtcnative-1` and `sqlite3`. Make sure you have them installed in your base image.
Apache Flink
The following Dockerfile specifies a Java 11 based Flink 1.13.1 image including additional custom dependencies
FROM registry.ververica.com/v2.5/flink:1.13.1-stream1-scala_2.12-java11
COPY my-custom-library.jar /flink/lib
Note: From Ververica Platform 2.5.0 onwards, you can use java 8 or 11. You can select your desired java version image (`-java8` or `-java11`) accordingly.
Warning: For custom images based on Flink 1.10 onwards, it is important to copy filesystem implementations only to the `/flink/plugins/` directory to avoid class loading issues. More information about that topic can be found here.
Warning: From Flink 1.10 onwards we use OpenSSL as the default SSL Provider. For this, we assume that netty-tcnative is available in the image and set up as described in the Flink documentation. If netty-tcnative is not available, the following error is shown in your Flink logs: `ClassNotFoundException: org.apache.flink.shaded.netty4.io.netty.internal.tcnative.SSL`. Setting up netty-tcnative is not required if you set the SSL provider to `security.ssl.provider: JDK`.
Warning: From Flink 1.10.3 and Flink 1.11.3 (comes with Ververica Platform 2.3.3) onwards, we use `jemalloc` as the memory allocator which has been shown to reduce memory fragmentation. If your base image does not include `jemalloc`, you would need to build & install it, and append the full path of `libjemalloc.so` to the environment variable `LD_PRELOAD`
The following Dockerfile specifies a CentOS 8 based image for Flink 1.13.1
FROM centos:centos8
RUN groupadd -r flink \
&& adduser -r -g flink flink -d /flink
RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
RUN yum -y install java-11-openjdk
ENV PATH=/flink/bin:$PATH
COPY --from=registry.ververica.com/v2.5/flink:1.13.1-stream1-scala_2.12-java11 --chown=flink:flink /flink /flink
USER flink:flink
WORKDIR /flink/bin
ENTRYPOINT ["docker-entrypoint.sh"]
EXPOSE 6123 8081
Related Information
Ververica Platform Docker Images