Skip to content
This repository was archived by the owner on Jan 8, 2020. It is now read-only.

Commit c0d4caa

Browse files
committed
Update README.md
1 parent 750e479 commit c0d4caa

File tree

1 file changed

+13
-35
lines changed

1 file changed

+13
-35
lines changed

Diff for: README.md

+13-35
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,7 @@ This project is used for [flume-ng](https://github.com/apache/flume) to communic
55

66
Current sql database engines supported
77
-------------------------------
8-
9-
- MySQL
10-
- PostgreSQL
11-
- SQLite
12-
- Microsoft SQLServer
13-
- Oracle
14-
15-
8+
- After the last update the code has been integrated with hibernate, so all databases supported by this technology should work.
169

1710
Compilation and packaging
1811
----------
@@ -38,34 +31,27 @@ $ wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.
3831
$ tar xzf mysql-connector-java-5.1.35.tar.gz
3932
$ cp mysql-connector-java-5.1.35-bin.jar $FLUME_HOME/plugins.d/lib/sql-source/libext
4033
```
41-
##### Oracle
42-
TODO
43-
##### Derby
44-
TODO
34+
4535
##### Microsoft SQLServer
4636
Download the official Microsoft 4.1 Sql Server jdbc driver and copy in libext flume plugins directory:
4737
Download URL: https://www.microsoft.com/es-es/download/details.aspx?id=11774
4838
```
4939
$ tar xzf sqljdbc_4.1.5605.100_enu.tar.gz
5040
$ cp sqljdbc_4.1/enu/sqljdbc41.jar $FLUME_HOME/plugins.d/lib/sql-source/libext
5141
```
52-
##### DB2
53-
TODO
54-
##### Sysbase IQ
55-
TODO
5642

5743
Configuration of SQL Source:
5844
----------
5945
Mandatory properties in <b>bold</b>
6046

6147
| Property Name | Default | Description |
6248
| ----------------------- | :-----: | :---------- |
49+
| <b>channels</b> | - | Connected channel names |
6350
| <b>type</b> | - | The component type name, needs to be org.keedio.flume.source.SQLSource |
6451
| <b>connection.url</b> | - | Url to connect with the remote Database |
6552
| <b>user</b> | - | Username to connect with the database |
6653
| <b>password</b> | - | Password to connect with the database |
6754
| <b>table</b> | - | Table to export data |
68-
| <b>incremental.colum.name</b> | - | Autoincremental column name |
6955
| <b>status.file.name</b> | - | Local file name to save last row number readed |
7056
| status.file.path | /var/lib/flume | Path to save the status file |
7157
| incremental.value | 0 | Start value to import data |
@@ -77,14 +63,12 @@ Mandatory properties in <b>bold</b>
7763

7864
Custom Query
7965
-------------
80-
A custom query is supported to bring the possibility of use entire SQL languaje. This is powerfull, but risky, be carefull with the custom queries.
81-
To use is needed an special character '@' to indicate where the incremental value should be.
66+
A custom query is supported to bring the possibility of use entire SQL languaje. This is powerfull, but risky, be carefull with the custom queries used.
67+
8268
Example:
8369
```
84-
agent.sources.sql-source.custom.query = SELECT field1,field2 FROM table1 WHERE field1='test' AND @ ORDER BY field2;
70+
agent.sources.sql-source.custom.query = SELECT field1,field2 FROM table1 WHERE field1='test'
8571
```
86-
Will run the query:
87-
```SELECT field1,field2 FROM table1 WHERE field1='test' AND incrementalColumn > 0 ORDER BY field2;```
8872

8973
Configuration example
9074
--------------------
@@ -94,22 +78,19 @@ Configuration example
9478
agent.sources = sql-source
9579
agent.sources.sql-source.type = org.keedio.flume.source.SQLSource
9680

97-
# URL to connect to database (currently only mysql is supported)
81+
# URL to connect to database
9882
agent.sources.sql-source.connection.url = jdbc:mysql://host:port/database
9983

10084
# Database connection properties
10185
agent.sources.sql-source.user = username
10286
agent.sources.sql-source.password = userpassword
10387
agent.sources.sql-source.table = table
104-
agent.sources.sql-source.database = database
10588

10689
# Columns to import to kafka (default * import entire row)
10790
agent.sources.sql-source.columns.to.select = *
10891

109-
# Increment column properties
110-
agent.sources.sql-source.incremental.column.name = id
11192
# Increment value is from you want to start taking data from tables (0 will import entire table)
112-
agent.sources.sql-source.incremental.value = 0
93+
agent.sources.sql-source.incremental.value = 0
11394

11495
# Query delay, each configured milisecond the query will be sent
11596
agent.sources.sql-source.run.query.delay=10000
@@ -119,20 +100,17 @@ agent.sources.sql-source.status.file.path = /var/lib/flume
119100
agent.sources.sql-source.status.file.name = sql-source.status
120101

121102
# Custom query
122-
agent.sources.sql-source.custom.query = SELECT * FROM table WHERE something AND @;
103+
agent.sources.sql-source.custom.query = SELECT * FROM table WHERE something;
123104
agent.sources.sql-source.batch.size = 1000;
124105
agent.sources.sql-source.max.rows = 10000;
125106

126-
127-
#
107+
# Connected channel names
108+
agent.sources.sql-source.channels = memoryChannel
128109

129110
```
130111

131-
Testing the source
132-
---------------------
133-
To test this source take a look to https://github.com/keedio/flume-stress-test-tools
134-
135-
Thanks!!
112+
Special thanks
113+
---------------
136114

137115
I used flume-ng-kafka to guide me (https://github.com/baniuyao/flume-ng-kafka-source.git).
138116
Thanks to [Frank Yao](https://github.com/baniuyao).

0 commit comments

Comments
 (0)