TCP Support
CyborgMaster opened this issue · comments
Is there anyway this could support sending overt the TCP Protocol (instead of UDP)? I want to take advantage of the larger message size.
The concept is sure possible, but it runs into 2 major considerations which don't exist with UDP:
- the sender needs to buffer until the receiver acknowledges the write. That means this logger needs to stash messages in a buffer of some kind. There is no fire-and-forget with TCP.
- any buffering brings other requirements: capping the buffer size so it doesn't grow unbounded, draining it (possibly concurrently), possibly locking, etc.
Given these requirements, I'm not sure there are many cases where this library is a better solution than papertrail/remote_syslog. Take a look at it and see whether remote_syslog works for you, and if doesn't, let me know why. It supports TCP with and without TLS, and a file makes a pretty decent buffer.
On Feb 24, 2014, at 8:17 AM, CyborgMaster notifications@github.com wrote:
Is there anyway this could support sending overt the TCP Protocol (instead of UDP)? I want to take advantage of the larger message size.
—
Reply to this email directly or view it on GitHub.
I've looked into the remote_syslog gem and read a fair amount of it's code, but I would prefer not to use it if possible. Our application is deployed inside of AWS Elastic Beanstalk, which makes changes to the system (outside of application code) rather difficult. We chose the remote_syslog_logger gem because it could be deployed inside the application without modifying the system (installing gems, setting up config files, etc).
I would be happy to help with the implementation (it's my top priority today). Could a similar approach, using event_machine be used?
Regarding using remote_syslog from Elastic Beanstalk, I think it might be
easier - a lot easier - than you described. It doesn't require a custom
system image or really much of anything.
http://help.papertrailapp.com/kb/hosting-services/aws-elastic-beanstalk
https://gist.github.com/4710601#file-01papertrail-config
Given how easy that is and how much cleaner it is to use the filesystem as
a buffer, it seems more elegant on its face - even if remote_syslog_logger
already supported TCP. That said, it's not my time, so if you review those
drop-in scripts and think its still a good use of yours, feel free.
With regard to event_machine, it doesn't solve the buffering issues.
Something still has to hold events that can't be sent quickly enough,
regardless of how they're eventually transmitted.
Unrelated to that, remote_syslog itself uses EM and I've been disappointed
with how difficult it is to support across different Ruby VM versions and
distributor packages. I don't pretend to have a better suggestion (other
than using a language which compiles to something closer to the OS, which
is what we're doing for it). I do think it changes the tradeoffs a user
would face when deciding whether to use remote_syslog_logger, though. I
wouldn't jump at introducing EM, an EM reactor, and an intermediate
buffer into my app merely for logging, and if I did introduce it, I
wouldn't use it for Rails.logger.
On Monday, February 24, 2014, CyborgMaster notifications@github.com wrote:
I've looked into the remote_syslog gem and read a fair amount of it's
code, but I would prefer not to use it if possible. Our application is
deployed inside of AWS Elastic Beanstalk, which makes changes to the system
(outside of application code) rather difficult. We chose the
remote_syslog_logger gem because it could be deployed inside the
application without modifying the system (installing gems, setting up
config files, etc).I would be happy to help with the implementation (it's my top priority
today). Could a similar approach, using event_machine be used?Reply to this email directly or view it on GitHubhttps://github.com//issues/4#issuecomment-35907832
.
Thanks for the feedback. I'm fine, of course, using the best tool for the job, whichever it happens to be. I leaned toward remote_syslog_logger, because I thought it was going to be simpler (and it was at the time), but if setting up a remote_syslog installation (using the scripts you linked to) is easier than modifying remote_syslog_logger to support TCP, then I'll go for it.
The last thing I would like to ask (as it does appear that you are a papertrail employee), is about TSL requirements. There appears to be conflicting documentation on the papertrail knowledge base. Some say that TLS is required if you use TCP, others say that if you contact papertrail support they can enable plaintext TCP access. However I see a checkbox on my endpoint configuration to allow plaintext TCP access, and some of the tutorials for remote_syslog indicate that TLS is optional.
I am leaning away from using TLS for the same reason that I was leaning away from remote_syslog: it appears to require the installation of both a root cert and client side certs, which again, can be relatively difficult inside of elastic beanstalk.
So the question boils down to: is TLS required if I use the TCP protocol? If not, is the "plaintext" checkbox on the endpoint config page enough to configure it, or do I need to contact support? If TLS is required, is there an easy way to install the needed certs inside Elastic Beanstalk (similar to the scripts you linked to for the remote_syslog configuration)?
Thank you for all of your help, I've been very impressed with the rapid and very informed response. You've made a loyal papertrail customer for life.
You're very welcome for the help. I just went through our knowledge base articles and fixed 1 document which incorrectly said that TCP w/o TLS wasn't supported, and made a second document clearer. This makes it easier to answer:
is TLS required if I use the TCP protocol?
.. with a conclusive "no" :-) All that's required is enabling the checkbox you saw. Let us know if we can help, too, either with TCP or just r_s in general.
Great! Thanks for the info. I'm very glad that this interaction has cleared up a couple knowledge base articles to help other who have the same question. Again, thank you for your help, I'm implementing the Elastic Beanstalk remote_syslog configuration you suggested. I'll let you know how it goes.
Side Note:
I just looked at one of the articles that I referred to in my last message (http://help.papertrailapp.com/kb/configuration/troubleshooting-remote-syslog-reachability) and it still has the following in it:
Sending over TCP
Papertrail accepts messages over TCP with TLS. Unencrypted TCP is not generally supported but can be enabled by contacting support.
It could be that the changes you mention in your message aren't live or are still cached, in which case, disregard this comment.
Ack, nice catch! Corrected. The downside of putting information in the places where it seems most relevant is, well, this ;-)
Perfect. Now I see the updated information. Much clearer (if it had been like this in the beginning, there wouldn't have been any confusion). Thanks!
I just wanted to report back. I've got logging working great using the remote_syslog gem over tcp. Thanks again! I did have to fork and modify your gist a bit though and thought I'd share what I had to do:
https://gist.github.com/CyborgMaster/9202227
The most important was that I had to include the json gem. I'm not sure if that was unique to my setup somehow, but my deploys to Elastic Beanstalk all failed until I did it. I also simplified the script somewhat by hardcoding the hostname instead of running another deploy command to load it from an environment variable.
A side comment, there is one small piece of functionality that the remote_syslog_logger gem had that the remote_syslog gem does not. With the logger, I could set the syslog tag to anything I wanted (using the program: option). In remote_syslog it is hard coded to the log filename. Not really important, but I wanted to report the feature disparity.
This is wonderful. I'm away from a computer today, but wanted to let you
know that we received this and are going to work it into the original gist,
then reply again.
Also, you're totally correct about the program/file naming difference.
However, you can get the same result as remote_syslog_logger by creating a
symlink and pointing remote_syslog at that. remote_syslog will se the name
of the symlink as the program (AKA syslog "tag") value.
On Feb 25, 2014, at 11:01 AM, CyborgMaster
<notifications@github.comjavascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:
I just wanted to report back. I've got logging working great using the
remote_syslog gem over tcp. I did have to fork and modify your gist a bit
though and thought I'd share what I had to do:
https://gist.github.com/CyborgMaster/9202227
The most important was that I had to include the json gem. I'm not sure if
that was unique to my setup somehow, but my deploys to Elastic Beanstalk
all failed until I did it. I also simplified the script somewhat by
hardcoding the hostname instead of running another deploy parameter to load
it from an environment variable.
A side comment, there is one small piece of functionality that the
remote_syslog_logger gem had that the remote_syslog gem does not. With the
logger, I could set the syslog tag to anything I wanted (using the
program:option). In remote_syslog it is hard coded to the log
filename. Not really
important, but I wanted to report the feature disparity.
Reply to this email directly or view it on
GitHubhttps://github.com//issues/4#issuecomment-36044462
.
Cool. I'm glad to know my efforts might help the next guy to do this.
Thanks again, @CyborgMaster. Your gist is now linked as a second example on http://help.papertrailapp.com/kb/hosting-services/aws-elastic-beanstalk. You're famous ;-)
I appreciate the extra care you put into making it public and am very glad to reciprocate by helping it save others time. If you need to delete or abandon the gist down the road, just let us know and we'll move it elsewhere.
Cool. Thanks!
FYI: I've actually been having trouble with remote_syslog loosing track of log files in the middle of night and I have to bounce the service to get it to start uploading again. I think it's a conflict with logrotate (although it appears to be using the copy truncate option, so I'm not sure why). I haven't had a chance to really dig into it. As soon as I get it figured out I'll open another issue and let you know.
Thanks, I'd appreciate that. Do you have any interest in beta testing our remote_syslog Go rewrite? Aside from being easier to deploy (binary), it almost certainly does not have that problem (though it may have other ones.. ;-).
Maybe, this is running the backbone logging for our business so it does need to be rock solid. Just curious, why did you decide a rewrite was necessary?
3 reasons:
- many systems don't have a Ruby VM installed, and many more have something funky with it (using rvm, using an ancient version which shipped with their distro, have the VM but not rubygems, have the VM and rubygems but not libs & headers to compile gems with native code)
- anyone who doesn't use Ruby doesn't understand any of this, any more than if you dropped a developer of Language X into a Language Y environment and asked them to setup the toolchain. Not Ruby's fault and wouldn't be any different with any other scripting language, but is completely different with a compiled binary.
- the eventmachine library, and modules that remote_syslog relies on, have some bugs (or just undocumented interactions) with the kernel that we'd need to repro and track down. This is as time-intensive as it sounds, especially when different VM and library versions are mixed in. Being able to make more-or-less libc calls is really handy to prevent that.
You make some very good points. I program in Ruby a ton, so these things seem natural to me, but I could see how they might be problematic for people from different backgrounds. I haven't had a chance to look at Go myself yet (my first introduction to it was tonight as I looked up some information), but it does seem like the right tool for the job. The compile to native plus built in concurrency and asynchronous primitives seem to match remote_syslog's needs perfectly.
Do you think the Go implementation might solve my logrotate problem? I'm using a vanilla Elastic Beanstalk Rails AMI to deploy our app, so nothing should be weird there. Specifically, my logs will stop uploading some nights (not all), at 3am UTC. That is exactly when logrotate is scheduled to kick in on the AMI, and I have confirmed that the inode is changing for my log file, so that should be the problem. I have to now pin down why the inode is changing. I don't think it should, I've found what I believe to the be the logrotate config for my Rails logs:
/var/app/support/logs/*.log {
daily
rotate 10
missingok
compress
copytruncate
}
copytrucate there should make the inode stay constant, so I'll have to figure out what's going on.
If you think the Go implimentation would handle the changing inode, then I'm all for trying it out.
Do you think the Go implementation might solve my logrotate problem?
Yes. It may introduce new problems (it's beta), but we'll fix those. Email me at support@pt if you want a copy of the beta bits.
copytruncate
I agree, this should work as-is. (We actually recommend that config in general, regardless of whether someone uses remote_syslog or anything else.)
So I tried to email you this morning. I sent an email to support@pt.com and now I realize that probably wasn't the right place to send it. I then looked on papertrail's support site to try and find a way to contact you, but all I found were ways to open up public conversations.
Long story short, I can't figure out to get you a private message lol. Sorry, where should I send it?
Sorry, that was shorthand for our domain name :-) support@papertrailapp.com or http://help.papertrailapp.com/discussions
Thanks, I re-sent you the email at support@papertrailapp.com.
@troy How I can get my hands on to the new remote_syslog Go rewrite?
thanks
It's at https://github.com/papertrail/remote_syslog2. Important notes:
- it's a rewrite of https://github.com/papertrail/remote_syslog, not the
remote_syslog_logger repo which this issue was opened in - expect to need to upgrade once or twice in the next month to be on a
supported, stable version.
On Thursday, May 15, 2014, Rimas Mocevicius notifications@github.com
wrote:
@troy https://github.com/troy How I can get my hands on to the new
remote_syslog Go rewrite?thanks
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4#issuecomment-43197494
.