分布式部署 DolphinScheduler原创

# 分布式部署 DolphinScheduler

DolphinScheduler-官方部署文档 (opens new window) (3.1.5)

# 1. 前期准备

JDK：下载JDK (opens new window) (1.8+)，安装并配置 JAVA_HOME 环境变量，并将其下的 bin 目录追加到 PATH 环境变量中。
二进制包：DolphinScheduler (opens new window) (3.1.5)，下载地址 (opens new window)
数据库：MySQL (opens new window) (5.7+)， MySQL 则需要 JDBC Driver 8.0.16（mysql-connector-java-8.0.16.jar (opens new window) ）
注册中心：ZooKeeper (opens new window) (3.6.3)，下载地址 (opens new window)
外部jar包：hadoop-cos-3.3.3.jar (opens new window)、cos_api-bundle-5.6.69.jar (opens new window)、hadoop-shaded-guava-1.1.1.jar (opens new window)（配置资源中心支持COS存储）
外部配置文件：core-site.xml （配置资源中心支持COS存储）

# 2 部署DolphinScheduler

[localhost]$ cd /datadrive
[localhost]$ wget https://archive.apache.org/dist/dolphinscheduler/3.1.5/apache-dolphinscheduler-3.1.5-bin.tar.gz
[localhost]$ tar -zxvf apache-dolphinscheduler-3.1.5-bin.tar.gz
[localhost]$ mv apache-dolphinscheduler-3.1.5-bin.tar.gz dolphinscheduler

1
2
3
4

# 3 启动zookeeper

注意

若使用云zookeeper，则跳过操作 3

进入zookeeper 的安装目录，将 zoo_sample.cfg 配置文件复制到 conf/zoo.cfg，并将 conf/zoo.cfg 中 dataDir 中的值改成 dataDir=./tmp/zookeeper

[localhost]$ ./bin/zkServer.sh start

# 4 初始化MySQL

# 4.1 建库

[localhost]$ mysql -uroot -p
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

1
2

# 4.2 添加MySQL驱动

下载mysql-connector-java-8.0.16.jar 放至 /datadrive/dolphinscheduler/

[localhost]$ cd /datadrive/dolphinscheduler/
[localhost]$ cp mysql-connector-java-8.0.16.jar api-server/libs/
[localhost]$ cp mysql-connector-java-8.0.16.jar alert-server/libs/
[localhost]$ cp mysql-connector-java-8.0.16.jar master-server/libs/
[localhost]$ cp mysql-connector-java-8.0.16.jar worker-server/libs/
[localhost]$ cp mysql-connector-java-8.0.16.jar tools/libs/

1
2
3
4
5
6

# 4.3 修改 dolphinscheduler_env.sh 文件

[localhost]$ cd /datadrive/dolphinscheduler/bin/env
[localhost]$ vim dolphinscheduler_env.sh
    
### 修改以下配置
    export JAVA_HOME=${JAVA_HOME:-~/jdk1.8.0_151}
    
    export DATABASE=${DATABASE:-mysql}
    export SPRING_PROFILES_ACTIVE=${DATABASE}
    export SPRING_DATASOURCE_URL="jdbc:mysql://10.75.16.4:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
    export SPRING_DATASOURCE_USERNAME=root
    export SPRING_DATASOURCE_PASSWORD=123456
    
    export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
    export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
    
### 其他配置按需添加

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# 4.4 同步dolphinscheduler_env.sh文件

[localhost]$ cd /datadrive/dolphinscheduler/
[localhost]$ cp bin/env/dolphinscheduler_env.sh api-server/conf/
[localhost]$ cp bin/env/dolphinscheduler_env.sh alert-server/conf/
[localhost]$ cp bin/env/dolphinscheduler_env.sh master-server/conf/
[localhost]$ cp bin/env/dolphinscheduler_env.sh worker-server/conf/

1
2
3
4
5

# 4.5 初始化数据库建表

[localhost]$ cd /datadrive/dolphinscheduler/
[localhost]$ ./tools/bin/upgrade-schema.sh

1
2

# 5 配置资源中心存储COS

# 5.1 修改 common.properties 文件

[localhost]$ cd /datadrive/dolphinscheduler/
[localhost]$ cp master-server/conf/common.properties ./
[localhost]$ vim common.properties
    
###
    data.basedir.path=/datadrive/dolphinscheduler/tmp
    resource.storage.type=HDFS
    resource.storage.upload.base.path=/dolphinscheduler
    
    resource.hdfs.root.user=hadoop
    resource.hdfs.fs.defaultFS=cosn://qianyi-los-1305502582
###

1
2
3
4
5
6
7
8
9
10
11
12

# 5.2 同步common.properties 文件

[localhost]$ cd /datadrive/dolphinscheduler/
[localhost]$ cp common.properties api-server/conf/
[localhost]$ cp common.properties alert-server/conf/
[localhost]$ cp common.properties master-server/conf/
[localhost]$ cp common.properties worker-server/conf/

1
2
3
4
5

# 5.3 同步core-site.xml

[localhost]$ cd /datadrive/dolphinscheduler/
[localhost]$ cp core-site.xml api-server/conf/
[localhost]$ cp core-site.xml alert-server/conf/
[localhost]$ cp core-site.xml master-server/conf/
[localhost]$ cp core-site.xml worker-server/conf/

1
2
3
4
5

# 5.4 同步jar包

[localhost]$ cp hadoop-cos-3.3.3.jar api-server/lib/
[localhost]$ cp hadoop-cos-3.3.3.jar alert-server/lib/
[localhost]$ cp hadoop-cos-3.3.3.jar master-server/lib/
[localhost]$ cp hadoop-cos-3.3.3.jar worker-server/lib/
    
[localhost]$ cp cos_api-bundle-5.6.69.jar api-server/lib/
[localhost]$ cp cos_api-bundle-5.6.69.jar alert-server/lib/
[localhost]$ cp cos_api-bundle-5.6.69.jar master-server/lib/
[localhost]$ cp cos_api-bundle-5.6.69.jar worker-server/lib/

[localhost]$ cp hadoop-shaded-guava-1.1.1.jar api-server/lib/
[localhost]$ cp hadoop-shaded-guava-1.1.1.jar alert-server/lib/
[localhost]$ cp hadoop-shaded-guava-1.1.1.jar master-server/lib/
[localhost]$ cp hadoop-shaded-guava-1.1.1.jar worker-server/lib/

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# 6 启停伪分布式集群

[localhost]$ cd /datadrive/dolphinscheduler/bin
# 启停 Master
[localhost]$ ./dolphinscheduler-daemon.sh stop master-server
[localhost]$ ./dolphinscheduler-daemon.sh start master-server
    
# 启停 Worker
[localhost]$ ./dolphinscheduler-daemon.sh start worker-server
[localhost]$ ./dolphinscheduler-daemon.sh stop worker-server
    
# 启停 Api
[localhost]$ ./dolphinscheduler-daemon.sh start api-server
[localhost]$ ./dolphinscheduler-daemon.sh stop api-server
    
# 启停 Alert
[localhost]$ ./dolphinscheduler-daemon.sh start alert-server
[localhost]$ ./dolphinscheduler-daemon.sh stop alert-server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# 7 登录 DolphinScheduler

URL：http://localhost:12345/dolphinscheduler/ui

用户名/密码：admin/dolphinscheduler123

# 8 集群部署

[localhost]$ cd /datadrive
[localhost]$ tar -zcvf dolphinscheduler.tar.gz dolphinscheduler
    
 ### 将dolphinscheduler.tar.gz 分发到部署服务器
[localhost]$ tar -zxvf dolphinscheduler.tar.gz
    
### 启动对应进程即可部署完成
参考 [操作6：启停伪分布式集群]

1
2
3
4
5
6
7
8

成功部署

#DolphinScheduler

上次更新: 2024/06/28, 14:46:16

← XXL-JOB Dolphinschedul集成COS异常→