Why Goliath server is running slower in staging environment than on my dev machine?

[Larry Zhao]

I just setup my api server with Goliath and Grape, it's working well on my dev machine (running Mac OSX 10.9.2, 16GB RAM), and I deploy it to my staging server (Ubuntu 12.04, 2GB RAM), both running Ruby 2.1.1 and I found by calling the same api endpoint, the response time on the staging server doubled comparing with the response time on my dev machine.

Following is what I see from the Goliath log:

api 1:
on staging server: [19676:INFO] 2014-04-18 18:47:24 :: Status: 200, Content-Length: 34006, Response Time: 552.24ms
on my dev machine: [47227:INFO] 2014-04-18 18:47:58 :: Status: 200, Content-Length: 44241, Response Time: 267.36ms

api 2:
on staging server: [19676:INFO] 2014-04-18 18:47:10 :: Status: 200, Content-Length: 11610, Response Time: 59.07ms
on my dev machine: [47227:INFO] 2014-04-18 18:48:18 :: Status: 200, Content-Length: 11634, Response Time: 34.88ms

On the staging server, Goliath is running behind a nginx server, here's my nginx configuration:

upstream goliath {
        server 127.0.0.1:8100 fail_timeout=0;
}

server {
        client_max_body_size 20M;

        listen       80;
        server_name  api.mydomain.com;
        root /home/deployer/deploy/myapi/current/;

        access_log  logs/myapi.access.log power_log;

        location / {
                proxy_redirect off;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Server $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                if (!-f $request_filename) {
                        proxy_pass http://goliath;
                }
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
                root   html;
        }
}

Why is that? which part should I look at?

Thank you very much.

[Ilya Grigorik]

-e prod mode vs development? In production mode we don't hot-reload the code (which requires an fstat of the file on every request). And if that's not it, try profiling your code... Add some log statements / grab rubyprof / use strace and take a look at where the time is going.

[Larry Zhao]

Thanks @igrigorik . Yes, respond time doubled in the log comparing development vs prod. What tool/approach would you recommend to do profiling with Goliath, I am really a newbie in the concurrency world. Is https://github.com/tmm1/perftools.rb the right way to go or there's something better?

[Ilya Grigorik]

perftools and strace are probably the two I use most... plus debug print statements ;)

[Larry Zhao]

Thanks @igrigorik for the recommendations, I tried perftools and print statements :).

The structure of my project is: Grape on top of Goliath, and inside Grape API ActiveRecord is used to query the data, and I use jbuilder to render json with the help of gem grape-jbuilder.

The funny thing is I tried to measure the time on local machine and on the server using --environment prod for both, and the response time on server is still twice as long as on my local Mac. I took sample, and found out that the Grape API part(including database query) is really fast, it's the render JSON part doubled on the server, averge 207ms on local Mac and 492ms on Ubuntu Server.

I could understand that rendering JSON really takes more time but twice slow on the server than the local machine is really confusing me ( I've also tried grape-entity, it's the same. ) Do you have any thought on this?

And one more thing, what would you use to render JSON on a Goliath API server?

Thanks a lot.

[Christian Kakesa]

Try YAJL http://lloyd.github.io/yajl/. I use it with Goliath, Sinatra and Rails. It woks fine.

[Larry Zhao]

Thanks @FeNicks, I tried that today, it's a little faster. But I still would like to know why it's much more faster on my local Mac than on the server.

[Christian Kakesa]

What is your Mac and server CPU model, frequency ?

[Larry Zhao]

Hi, @FeNicks , I list them below, would it be the difference?

Mac: Intel(R) Core(TM) i5-2500S CPU @ 2.70GHz

Here is server:

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 13
model name  : QEMU Virtual CPU version (cpu64-rhel6)
stepping    : 3
microcode   : 0x1
cpu MHz     : 2599.998
cache size  : 4096 KB
fpu     : yes
fpu_exception   : yes
cpuid level : 4
wp      : yes
flags       : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm up nopl pni cx16 hypervisor lahf_lm
bogomips    : 5199.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual

[Christian Kakesa]

The "QEMU Virtual CPU version (cpu64-rhel6)" is less powerful than a real CPU. The CPU behind the "QEMU Virtual CPU" is probably shared with other VM. That's why you have less power to process requests than your local Mac CPU who is dedicated to you.

[Larry Zhao]

Thanks @FeNicks