Skip to content

Conversation

@jo8937
Copy link

@jo8937 jo8937 commented Oct 6, 2022

Thank you for your great work. embulk-output-hdfs is very cool plugin.
I want to add some feature - kerberos keytab auth.
use like this.

  • command line auth
kinit -kt user.keytab user
  • embulk yaml file
out:
  ....
  type: hdfs
  path_prefix: ./testdata
  file_ext: tsv
  config_files:
    - ./hadoop-conf/core-site.xml
    - ./hadoop-conf/hdfs-site.xml
  doas: user
  keytab_config:
    keytab_path: ./env/user.keytab
    keytab_principal: [email protected]
    krb5_config_path: ./env/krb5.conf
  config:
    fs.defaultFS: 'webhdfs://my-hadoop-server:9870'
    fs.webhdfs.impl: 'org.apache.hadoop.hdfs.web.WebHdfsFileSystem'
    hadoop.security.authentication: 'kerberos'
    hadoop.security.authorization: 'true'
    dfs.client.use.datanode.hostname: 'true'
    dfs.namenode.kerberos.principal.pattern: 'hdfs/*@CLOUDERA.COM'    
   ....

This pull request's implementation is working fine in my environment with "webhdfs"

@jo8937 jo8937 closed this Oct 6, 2022
@jo8937
Copy link
Author

jo8937 commented Oct 6, 2022

Sorry for invalid PR.

@jo8937 jo8937 reopened this Oct 6, 2022
@jo8937
Copy link
Author

jo8937 commented Oct 6, 2022

The keytab feature has some errors with hdfs scheme, but It works fine with webhdfs scheme
maybe it is useful for some user

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant