Skip to content

[Feature-13052][Task Plugin] Support Apache Seatunnel Connector-V2 #13086

New issue

Have a question about this project? No Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “No Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? No Sign in to your account

Merged
merged 3 commits into from
Apr 8, 2023

Conversation

shangeyao
Copy link
Contributor

Purpose of the pull request

close #13052

Brief change log

support flink-v2, spark-v2, seatunnel-engine

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

@codecov-commenter
Copy link

codecov-commenter commented Dec 3, 2022

Codecov Report

Merging #13086 (5d17b81) into dev (fa3e2a5) will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev   #13086      +/-   ##
============================================
+ Coverage     39.20%   39.22%   +0.01%     
- Complexity     4273     4274       +1     
============================================
  Files          1069     1069              
  Lines         40311    40311              
  Branches       4634     4634              
============================================
+ Hits          15804    15811       +7     
+ Misses        22729    22722       -7     
  Partials       1778     1778              
Impacted Files Coverage Δ
...e/dolphinscheduler/remote/NettyRemotingClient.java 52.08% <0.00%> (-0.70%) ⬇️
...r/plugin/task/sqoop/parameter/SqoopParameters.java 52.38% <0.00%> (+1.19%) ⬆️
...dolphinscheduler/remote/future/ResponseFuture.java 83.60% <0.00%> (+1.63%) ⬆️
...erver/master/processor/queue/TaskEventService.java 80.35% <0.00%> (+10.71%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sonarqubecloud
Copy link

sonarqubecloud bot commented Dec 3, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@EricGao888 EricGao888 added this to the 3.2.0 milestone Dec 5, 2022
@EricGao888 EricGao888 added feature new feature 3.2.0 for 3.2.0 version labels Dec 5, 2022
@caishunfeng
Copy link
Contributor

Hi @zhuangchong @CalvinKirs Do you have time to review this pr?

@CalvinKirs
Copy link
Member

Could you provide a test? or your local running screenshot?

Copy link

@chaplinthink chaplinthink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have view and test the code. Find a bug need to fix, when save seatunnel process definition.

Class SeatunnelParameters need add follow code for json parse.

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, visible = true, property = "engine")
@JsonSubTypes({
        @JsonSubTypes.Type(value = SeatunnelFlinkParameters.class, name = "FLINK"),
        @JsonSubTypes.Type(value = SeatunnelFlinkParameters.class, name = "FLINK_V2"),
        @JsonSubTypes.Type(value = SeatunnelSparkParameters.class, name = "SPARK"),
        @JsonSubTypes.Type(value = SeatunnelSparkParameters.class, name = "SPARK_V2"),
        @JsonSubTypes.Type(value = SeatunnelEngineParameters.class, name = "SEATUNNEL_ENGINE")
})

@chaplinthink
Copy link

The screenshot that i run in my machine. It can work for seatunnel engine to run ETL job. @CalvinKirs

image

image

@chaplinthink
Copy link

@caishunfeng @CalvinKirs Hi, This PR hasn't been merged for a long time, is there any consideration? I would be happy to help with anything.

@jobmission
Copy link

@shangeyao This is a great PR, very useful. Can you continue to complete it?

@jobmission
Copy link

jobmission commented Mar 29, 2023

@shangeyao 你好,我在使用的时候,我用的seatunnel、flink版本不支持run-mode参数,可否在创建任务页面及构造job 配置的任务增加一个run-mode枚举none ,不拼加run-mode参数,这样使用范围更广一些,感谢

image

image

@jobmission
Copy link

jobmission commented Mar 30, 2023

@shangeyao @caishunfeng @CalvinKirs @chaplinthink Hi ,生成seatunnel 脚本的时候,是否可以加上任务名称,现在提交到fink的job名字都是默认的seatunnel 看着很难受,seatunnel可以接受--name参数,可以把工作流的名称拼接上

Copy link
Contributor

@davidzollo davidzollo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
LGTM

@davidzollo davidzollo merged commit 4439c1e into apache:dev Apr 8, 2023
@davidzollo
Copy link
Contributor

@chaplinthink, considering this PR hasn't been merged for a long time, I merged it and you can submit a new PR to fix it, thx
@jobmission, you can continue to submit a PR to complete what you said.

No Sign up for free to join this conversation on GitHub. Already have an account? No Sign in to comment
Labels
backend document feature new feature UI ui and front end related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Task Plugin] Support Apache Seatunnel Connector-V2
9 participants