Skip to content

Commit 5572639

Browse files
committed
Update README.md
1 parent b70fb7c commit 5572639

File tree

1 file changed

+28
-7
lines changed

1 file changed

+28
-7
lines changed

README.md

Lines changed: 28 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,40 @@
11
# Hdfs output plugin for Embulk
22

3-
TODO: Write short description here and build.gradle file.
3+
A File Output Plugin for Embulk to write HDFS.
44

55
## Overview
66

7-
* **Plugin type**: output
7+
* **Plugin type**: file output
88
* **Load all or nothing**: no
99
* **Resume supported**: no
10-
* **Cleanup supported**: yes
10+
* **Cleanup supported**: no
1111

1212
## Configuration
1313

14-
- **property1**: description (string, required)
15-
- **property2**: description (integer, default: default-value)
14+
- **config_files** list of paths to Hadoop's configuration files (array of strings, default: `[]`)
15+
- **config** overwrites configuration parameters (hash, default: `{}`)
16+
- **output_path** the path finally stored files. (string, default: `"/tmp/embulk.output.hdfs_output.%Y%m%d_%s"`)
17+
- **working_path** the path temporary stored files. (string, default: `"/tmp/embulk.working.hdfs_output.%Y%m%d_%s"`)
1618

1719
## Example
1820

1921
```yaml
2022
out:
2123
type: hdfs
22-
property1: example1
23-
property2: example2
24+
config_files:
25+
- /etc/hadoop/conf/core-site.xml
26+
- /etc/hadoop/conf/hdfs-site.xml
27+
- /etc/hadoop/conf/mapred-site.xml
28+
- /etc/hadoop/conf/yarn-site.xml
29+
config:
30+
fs.defaultFS: 'hdfs://hdp-nn1:8020'
31+
dfs.replication: 1
32+
mapreduce.client.submit.file.replication: 1
33+
fs.hdfs.impl: 'org.apache.hadoop.hdfs.DistributedFileSystem'
34+
fs.file.impl: 'org.apache.hadoop.fs.LocalFileSystem'
35+
formatter:
36+
type: csv
37+
encoding: UTF-8
2438
```
2539
2640
@@ -29,3 +43,10 @@ out:
2943
```
3044
$ ./gradlew gem
3145
```
46+
47+
## Development
48+
49+
```
50+
$ ./gradlew classpath
51+
$ bundle exec embulk run -I lib example.yml
52+
```

0 commit comments

Comments
 (0)