sincedb default should not use ENV["HOME"]

[Michael Doyle]

Looks like by default sincedb will try to use $HOME:

https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L274

This is problematic, because $HOME may not be set. Per the POSIX standard, this variable is set by the login program:

HOME
The system shall initialize this variable at the time of login to be a pathname
of the user's home directory. See <pwd.h>.
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html

In my case, this failed when I ran logstash as an upstart job with the logstash service account.

This leads to the following:

  Error: can't convert nil into String
  Exception: TypeError
  Stack: org/jruby/RubyFile.java:2023:in `join'
org/jruby/RubyFile.java:861:in `join'
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-3.1.1/lib/logstash/inputs/s3.rb:276:in `sincedb_file'
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-3.1.1/lib/logstash/inputs/s3.rb:263:in `sincedb'
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-3.1.1/lib/logstash/inputs/s3.rb:102:in `list_new_files'

What should be used instead? Perhaps, $LOGSTASH_HOME?

[Jordan Sissel]

For now, please set HOME.

We could use the new Settings feature in LS 5.0 to query the path.data directory and use that.

[Michael Doyle]

Your proposal sounds good - FWIW, I ended up just setting the sincedb_path directly in my logstash config.

[Günter Grodotzki]

Still investigating why my s3-input since 2.4 to 5.3 ELK upgrade has not been working - suspecting this might be the case. Seems like the s3-input plugin works differently because I used to have 6 sincedb files, and ever since the upgrade I had only 3 under $HOME ?

Also why is the convention of environment variables being used over having it simply in the config? Why not just make the config file reader to support environment variables?