|
1 | 1 | # Hdfs output plugin for Embulk |
2 | 2 |
|
3 | | -TODO: Write short description here and build.gradle file. |
| 3 | +A File Output Plugin for Embulk to write HDFS. |
4 | 4 |
|
5 | 5 | ## Overview |
6 | 6 |
|
7 | | -* **Plugin type**: output |
| 7 | +* **Plugin type**: file output |
8 | 8 | * **Load all or nothing**: no |
9 | 9 | * **Resume supported**: no |
10 | | -* **Cleanup supported**: yes |
| 10 | +* **Cleanup supported**: no |
11 | 11 |
|
12 | 12 | ## Configuration |
13 | 13 |
|
14 | | -- **property1**: description (string, required) |
15 | | -- **property2**: description (integer, default: default-value) |
| 14 | +- **config_files** list of paths to Hadoop's configuration files (array of strings, default: `[]`) |
| 15 | +- **config** overwrites configuration parameters (hash, default: `{}`) |
| 16 | +- **output_path** the path finally stored files. (string, default: `"/tmp/embulk.output.hdfs_output.%Y%m%d_%s"`) |
| 17 | +- **working_path** the path temporary stored files. (string, default: `"/tmp/embulk.working.hdfs_output.%Y%m%d_%s"`) |
16 | 18 |
|
17 | 19 | ## Example |
18 | 20 |
|
19 | 21 | ```yaml |
20 | 22 | out: |
21 | 23 | type: hdfs |
22 | | - property1: example1 |
23 | | - property2: example2 |
| 24 | + config_files: |
| 25 | + - /etc/hadoop/conf/core-site.xml |
| 26 | + - /etc/hadoop/conf/hdfs-site.xml |
| 27 | + - /etc/hadoop/conf/mapred-site.xml |
| 28 | + - /etc/hadoop/conf/yarn-site.xml |
| 29 | + config: |
| 30 | + fs.defaultFS: 'hdfs://hdp-nn1:8020' |
| 31 | + dfs.replication: 1 |
| 32 | + mapreduce.client.submit.file.replication: 1 |
| 33 | + fs.hdfs.impl: 'org.apache.hadoop.hdfs.DistributedFileSystem' |
| 34 | + fs.file.impl: 'org.apache.hadoop.fs.LocalFileSystem' |
| 35 | + formatter: |
| 36 | + type: csv |
| 37 | + encoding: UTF-8 |
24 | 38 | ``` |
25 | 39 |
|
26 | 40 |
|
|
29 | 43 | ``` |
30 | 44 | $ ./gradlew gem |
31 | 45 | ``` |
| 46 | + |
| 47 | +## Development |
| 48 | + |
| 49 | +``` |
| 50 | +$ ./gradlew classpath |
| 51 | +$ bundle exec embulk run -I lib example.yml |
| 52 | +``` |
0 commit comments