Getting benchmark behavior I cannot explain (triggers malware detection)
kswope opened this issue · comments
I've been benchmarking many minimalist frameworks in rust/node/go/elixir/etc for a while now on my MBA M1. I discovered crystal and I'm giving it a try.
I'm comparing kemal cut-n-past
require "kemal"
get "/" do
"Hello World!"
end
Kemal.run
to a cut-n-past of the stdlib http
require "http/server"
server = HTTP::Server.new do |context|
context.response.content_type = "text/plain"
context.response.print "Hello world! The time is #{Time.local}"
end
address = server.bind_tcp 8000
puts "Listening on http://#{address}"
server.listen
I'm using k6 for benchmarking
import http from "k6/http";
export const options = {
vus: 10,
duration: "1m",
};
export default function () {
http.get("http://127.0.0.1:8000");
}
Both verified to be on the same ip/port
~Development/Crystal/scratch master lsof -Pn -i4|grep 8000 ✔
scratch 35557 kevin 8u IPv4 0x2659e688e9bbed09 0t0 TCP 127.0.0.1:8000 (LISTEN)
~Development/Crystal/scratch master lsof -Pn -i4|grep 8000 ✔
scratch 35909 kevin 8u IPv4 0x2659e688e9a88b51 0t0 TCP 127.0.0.1:8000 (LISTEN)
Now the unexplained behavior. The kemal version is running at about 1/20 the performance of the std http. But not only that, its triggering my anti malware app
named BlockBlock to run at almost 100% CPU. All this app allegedly does is watch startup directories on osx and blocks anything snuck in there.
BlockBlock is not giving me any info, its just running like crazy only during the kemal benchmark session.
I've been an obsessive benchmarker for years and I've never seen this before. Maybe its a crystal thing (but doesn't effect std http), or maybe its just kemal. I have no idea, I have one day experience with crystal.
closing this issue for now. I found that BlockBlock does somehow watch something, like binding to sockets. For example I did notice blockblock slowing the connection of a node server in cluster mode (slows the binding of all the forked children), but once the bindings were done, it ran fine. Something is odd about the kemal example in that it might have been continuously binding to the socket. Maybe the example was overlooking an important configuration.
Hey thanks for benchmarking.
Please do turn off the logging like the following and report the results
require "kemal"
logging false
get "/" do
"Hello World!"
end
Kemal.run
Logging off: 105890 /s
Logging on: 2823 /s <-- !?!?
Here's go/chi
Logging off: 81173 /s
Logging on: 23921 /s
So what is weird with kemal logging?. (note I haven't tried any other logging with crystal, so I don't know if its a kemal thing or a crystal thing)
I discovered a few more things. With blockblock on, things like opening a new terminal tab is a few seconds slower. And when running node.js in cluster mode, it takes about a second each for each child to spawn (and I think bind to socket). So blockblock is either wacky, or its using something in the OS that's acting wacky (I don't remember this happening a year ago). Something with apples "Endpoint Security Framework"?
But besides all that, why is kemal logging bogging down while I'm not seeing the same thing in logging for any framework/lang I've ever tried?
@kswope Also keep in mind Kemal does routing while the stdlib example does not. I.e. Kemal would error if you requested a diff route, or used a diff method than GET. The stdlib benchmark would happily return the same data regardless of its path/method since it does not have routing built in.
Kemal logs to stdout by default, which is usually connected to a terminal. A terminal console is comparatively slow - when you run thousands of requests per seconds simply stalls Kemal.
Kemal logs to stdout by default, which is usually connected to a terminal. A terminal console is comparatively slow - when you run thousands of requests per seconds simply stalls Kemal.
Lots of frameworks log to stdout, its part of the "12 factor app", I've never seen a 98% drop in performance because of it, whether it was connected to a terminal or not. And I've never seen it spike a malware detection app to 100% CPU while it was running either.
You're right. Such a high impact is indeed unexpected. Not sure why that is. May be something else at play there.
Testing the hello world example with wrk -d1m I get the following on an i7 4770:
- Logging to console: 5801 req/s
- Logging to file: 22263 req/s
- Logging disabled: 35817 req/s
Those differences seem reasonable, accounting for the respective logging overhead.
So the big difference you're observing might be somehow specific to your system. Maybe someone else could try this with an M1?
I won't speculate on behaviour of a malware scanner. They might just be doing anything, really. Unless there is substantial evidence that Kemal does something wrong, I would assume it's 100% their fault.
Also, if you want honest performance benchmarks, you must not let an external tool affect the programs being tested.
I would assume it's 100% their fault.
I agree, but we're all familiar sometimes a bug can expose other bugs. But maybe the bug is in all the other frameworks. Maybe they are all doing something wrong, and kemal is doing something right.
I just noticed that golang is now triggering blockblock during compilation.
I've emailed the creator of blockblock and maybe I'll hear something back. Its not OSS so I can't whine about it in github like I'm doing here.
The normal HTTP server doesn't trigger blockblock? If you disable blockblock how's the relative performance? You could sample profile kemal to see where it seems to be "sticking" https://crystal-lang.org/reference/1.3/guides/performance.html GL!
The normal HTTP server doesn't trigger blockblock?
nope, the only thing that seems to trigger blockblock is doing things like starting a cluster of node workers, each one is very slow to start. Or even starting a new terminal (shell maybe?). Blockblock seems to slow down things like that, which might have something to do with a recent update to the OSX "Endpoint Security Framework". But why is it being triggered by Kemal logging? No logging in any other web framework I've benched is doing that, its very strange.
If you disable blockblock how's the relative performance?
Very good, better than go and node
I'm curious, any update on this? Like what might be triggering?
I'm curious, any update on this? Like what might be triggering?
No updates here. I was just experimenting crystal but after getting these bizarre results I just went back to Go. If I were to completely make up some crazy theory I'd say that kemal/crystal was doing something unconventional with sockets/networking and that was triggering something in some OSX security subsystem and blockblock was "listening" to those "events" and was being flooded.
What I keep coming back to is that running blockblock slowed down a node cluster starting up (in other words, each node would take half a second or more to start, like each binding to the network socket took its time) which became obvious when you were starting 8 of them on a MBA, and it seemed to blip up blockblock in the activity monitor, but once it was running, there was no problem. But on the other hand, the same benchmark with kemal was just flustering blockblock, as if each connection was doing the same thing node did, but not only when it started. Possibly a http keep-alive issue?
I'd be curious to hear if there's any update on this @kswope 📈 If not I'd like to close the issue