Splunk token invalid when using environment variables

[mentalblock]

When using environment variables to interpolate the fluent.conf configuration file, the Splunk token somehow becomes invalid. When I put the token statically into the configuration file without interpolation, the Splunk message is successfully accepted.

This doesn't work correctly when ensuring the token is in this variable:

token "#{ENV["FLUENTD_SPLUNK_TOKEN"]}"

This works fine

token my-token-statically-written-out

I am deploying using a derivative of the the alpine/v1.1.0-onbuild. The derivative is an image I built following the instructions in https://hub.docker.com/r/fluent/fluentd/ for the sole purpose of installing the Splunk gem. Can anyone confirm this problem?

[brycied00d]

The first diagnostic step should be to turn on debug logging and check those for clues. Does the token appear correctly when fluentd starts and prints its configuration? Worked for me...

$ FLUENTD_SPLUNK_TOKEN=HelloWorld fluentd -c ~/fluent-testing/test.conf -vvvv
2018-03-01 18:45:03 -0500 [info]: fluent/log.rb:316:info: parsing config file is succeeded path="/home/brycec/fluent-testing/test.conf"
2018-03-01 18:45:03 -0500 [trace]: fluent/log.rb:275:trace: registered output plugin 'splunk-http-eventcollector'
2018-03-01 18:45:03 -0500 [trace]: fluent/log.rb:275:trace: splunk-http-eventcollector(initialize) called
2018-03-01 18:45:03 -0500 [trace]: fluent/log.rb:275:trace: registered buffer plugin 'memory'                        
2018-03-01 18:45:03 -0500 [warn]: fluent/log.rb:336:warn: 'time_format' specified without 'time_key', will be ignored
2018-03-01 18:45:03 -0500 [trace]: fluent/log.rb:275:trace: splunk-http-eventcollector(configure) called 
2018-03-01 18:45:03 -0500 [trace]: fluent/log.rb:275:trace: registered filter plugin 'record_transformer'
2018-03-01 18:45:03 -0500 [trace]: fluent/log.rb:275:trace: registered input plugin 'forward'
2018-03-01 18:45:03 -0500 [info]: fluent/log.rb:316:info: using configuration file: <ROOT>
  <system>           
    log_level trace                                                                                                                     
    suppress_config_dump false                                                                                                           
  </system>                                                                                                       
  <source>                                                                                                                     
    @type forward                                                                                      
    port 24224                                                                                           
  </source>                                                                                          
  <match **>                                                                                                                               
    @type splunk-http-eventcollector  
    protocol "rest"                                                                                                                        
    server "splunk.example.com:8089"                           
    token "HelloWorld"                                                                                                                     
    sourcetype "log"                                    
    time_format none                                                                                                                       
    format text                                                              
    <buffer>                                                                                                                               
      flush_mode interval                                                            
      retry_type exponential_backoff                                                                                                       
    </buffer>                                                               
    <parse>                                                                                                                                
      time_format none                                 
    </parse>                                                                                                                               
    <inject>                                             
      time_format none                                                                                                                     
    </inject>                                                           
    <format>                                                                                                                               
      @type text                               
    </format>                                                                                                                              
  </match>                                                   
</ROOT>                                                                                                                                    
2018-03-01 18:45:03 -0500 [info]: fluent/log.rb:316:info: starting fluentd-1.0.2 pid=27747 ruby="2.4.2"

[mentalblock]

Yes. Whether I use a variable or statically hardcode the token, it is correctly printed when Fluentd starts. The only difference is that when using a variable, Fluentd prints in the log that the Splunk Event Collector replied 403 Forbidden with a JSON reply that the token is invalid. Can you confirm the problem on your end?

[brycied00d]

I don't have a Splunk instance to test with (much less the time), but I added a quick log.info "Token: [#{@token}] to def configure and confirmed that the token is set identically whether using a static string or variable interpolation.

My next step would be to get a packet capture (tcpdump) of the traffic on the wire and compare what's actually being written to the socket.

[mentalblock]

Capturing the packets for the token is not an option for I am using HTTPS to connect to the event collector. I suppose I can change it. I'll do more digging.