flink和spark的类装载器
文章讨论了Flink和Spark在处理依赖关系时的类加载问题。Flink配置中使用`child-first`类加载策略,并指定某些包使用`parent-first`模式。然而,在运行过程中出现了`LinkageError`错误,原因是不同类加载器对`com.codahale.metrics.Histogram`类有不同的定义。文章还提到Spark通过配置额外的类路径来解决类似问题。
2025-7-3 00:18:52
Author: dyrnq.com(查看原文)
阅读量:27
收藏
flink
classloader.resolve-order: child-first # parent-first
#classloader.parent-first-patterns.default
classloader.parent-first-patterns.additional: com.codahale.metrics
java.lang.LinkageError: loader constraint violation: when resolving method 'void org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper.<init>(com.codahale.metrics.Histogram)' the class loader org.apache.flink.util.ChildFirstClassLoader @1f4f7e8c of the current class, org/apache/iceberg/flink/sink/IcebergStreamWriterMetrics, and the class loader 'app' for the method's defining class, org/apache/flink/dropwizard/metrics/DropwizardHistogramWrapper, have different Class objects for the type com/codahale/metrics/Histogram used in the signature (org.apache.iceberg.flink.sink.IcebergStreamWriterMetrics is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @1f4f7e8c, parent loader 'app'; org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper is in unnamed module of loader 'app')
at org.apache.iceberg.flink.sink.IcebergStreamWriterMetrics.<init>(IcebergStreamWriterMetrics.java:54)
at org.apache.iceberg.flink.sink.IcebergStreamWriter.open(IcebergStreamWriter.java:56)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:107)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.base/java.lang.Thread.run(Unknown Source)
spark
--conf "spark.driver.extraClassPath="
--conf "spark.executor.extraClassPath="
文章来源: https://dyrnq.com/classloader-for-flink-and-spark/
如有侵权请联系:admin#unsafe.sh