Question
How can I run my Flink JAR-based deployment in VVP with latest Scala 3 or Scala 2.13 version? I am using Flink 1.15 or newer version as it is no longer coupled to older version of Scala 2.12.
Answer
Note: This section applies to Flink 1.15 or newer with Ververica Platform 2.12.x or newer.
1. Assemble your application JAR file. It may include all dependencies which your application requires.
Standard Scala libraries, such as "scala-library-2.13.*.jar" or "scala-library-3-*.jar", can be packaged into the same JAR as well, but we recommend to keep Scala standard library files as VVP Deployment additional dependencies.
In order to assemble your application JAR file, we recommend to use SBT plugin assembly.
2. Create new VVP Deployment and use your application JAR in the Jar URI field.
3. Set Additional dependencies to the standard Scala libraries, in case you did not package them into the application JAR already:
For Scala 3:
For Scala 2.13:
Note: Scala 3 requires both libraries, i.e. 2.13.* and 3-*. In case you use Scala 2.13 only, then Scala 3 library is not required.
Note: minor and patch versions, such as *.12 for Scala 2 and *.3.1. for Scala 3, are not mandatory, they can be any latest numbers you see for Scala currently available.
4. Remove scala package from classloader.parent-first-patterns.default Flink's configuration property.
Default property value in Flink configuration:
classloader.parent-first-patterns.default: java.;scala.;org.apache.flink.;com.esotericsoftware.kryo;org.apache.hadoop.;javax.annotation.;org.xml;javax.xml;org.apache.xerces;org.w3c;org.rocksdb.;org.slf4j;org.apache.log4j;org.apache.logging;org.apache.commons.logging;ch.qos.logback
To explicitly remove the Scala package, add the following classloader.parent-first-patterns.default config to your Deployment.:
classloader.parent-first-patterns.default: >-
java.;org.apache.flink.;com.esotericsoftware.kryo;org.apache.hadoop.;javax.annotation.;org.xml;javax.xml;org.apache.xerces;org.w3c;org.rocksdb.;org.slf4j;org.apache.log4j;org.apache.logging;org.apache.commons.logging;ch.qos.logback
Tip: this change allows Flink to load Scala standard library from user supplied libraries rather than from Flink's lib internal folder.
Note: your classloader.parent-first-patterns.default may have other default packages removed or added from/in it. The main idea is that we remove scala package. Other changes in this property depend on your use cases and preferences. Feel free to modify it based on your needs.