Strange jq performance problem

[Payam Nab]

I have a bash script which analyze 100 miners data using jq on ip range sets, recently noticed its running slower than before so after lots of tests noticed if another jq script is running they both run faster.

If I run two of my script on two different ip range at the same time, they both complete in 20 seconds, if I run them one after the other they complete in about 50 seconds which will be 100 seconds instead of 20.

To check I ran this test:

time bash -c 'for i in `seq 1 1000` ; do jq -n . > /dev/null; done'
    real    0m4.087s
    user    0m2.785s
    sys     0m1.323s

Then in another terminal run the same in a loop:

while true ; do time bash -c 'for i in `seq 1 1000` ; do jq -n . > /dev/null; done' ;done

Now run the first test and times are less than half:

time bash -c 'for i in `seq 1 1000` ; do jq -n . > /dev/null; done'
real    0m1.651s
user    0m1.364s
sys     0m0.279s

So the problem is when I run one instance of any script with jq, jq runs really slow and its like some bottlenecks but when use 2 jq scripts it runs really fast.

Any idea where should I look for this problem and how to solve it?

[Mattias Wadman]

Tried to reproduce with macOS 13.6.1 macbook 3,5 GHz Dual-Core Intel Core i7. In my case the result is the opposite, time bash -c 'for i in seq 1 1000 ; do jq -n . > /dev/null; done' got a tiny bit slower when while true ; do time bash -c 'for i in seq 1 1000 ; do jq -n . > /dev/null; done' ;done is running at the same time.

I suspect this might be the effect of some caching or CPU throttling/energy saving thingy?

[Emanuele Torre]

You cannot time a program that runs very quickly like that, you will get unstable results, and get some sample that run very quickly and some that run very slowly compared to the average and mess up the result.

I do not get consistently faster results all the times with this technique.

[Payam Nab]

As I said earlier if I run my monitoring bash script in two sets they both finish in 20 seconds, but in normal run it takes 50 seconds

I have no power saving , It shouldnt be cache because I tested 2 different sets of scripts, if its throttling it should throtle when 2 jq is running as well, right?

[Mattias Wadman]

Sorry with throttling i meant one cpu energy saving technique. Can you reproduce this with something else than jq? ex true, stat $HOME (simulate some light IO), node -p 123 etc?

[Payam Nab]

Sorry with throttling i meant one cpu energy saving technique. Can you reproduce this with something else than jq? ex true, stat $HOME (simulate some light IO), node -p 123 etc?

Can you please tell me what exactly you want me to run, and also do you want me to run in two terminals or one?

time bash -c 'node -p 123'
123

real    0m0.340s
user    0m0.328s
sys     0m0.012s

time stat $HOME
  File: /home/payam
  Size: 12288           Blocks: 24         IO Block: 4096   directory
Device: 813h/2067d      Inode: 8126465     Links: 31
Access: (0755/drwxr-xr-x)  Uid: ( 1000/   payam)   Gid: ( 1000/   payam)
Access: 2023-12-11 13:34:48.424829700 +0330
Modify: 2023-12-11 14:22:13.767689419 +0330
Change: 2023-12-11 14:22:13.767689419 +0330
 Birth: -

real    0m0.007s
user    0m0.001s
sys     0m0.006s

[Mattias Wadman]

Sorry with throttling i meant one cpu energy saving technique. Can you reproduce this with something else than jq? ex true, stat $HOME (simulate some light IO), node -p 123 etc?

Can you please tell me what exactly you want me to run, and also do you want me to run in two terminals or one?

time bash -c 'node -p 123'
123

real    0m0.340s
user    0m0.328s
sys     0m0.012s

Same setup but replace jq -n . with true, state $HOME and node -p 123. Possibly with different number of iterations if it's to slow or to fast

[Payam Nab]

So it looks like it happens with stats $HOME as well

while true ; do time bash -c 'for i in `seq 1 1000` ; do stat $HOME  . > /dev/null; done' ;done

real    0m3.053s
user    0m1.522s
sys     0m1.586s

real    0m2.472s
user    0m1.411s
sys     0m1.093s

real    0m2.215s
user    0m1.313s
sys     0m0.926s

"run another in other session"
real    0m1.818s
user    0m1.146s
sys     0m0.700s

real    0m0.802s
user    0m0.640s
sys     0m0.164s

real    0m0.799s
user    0m0.669s
sys     0m0.134s

real    0m0.803s
user    0m0.645s
sys     0m0.156s

"stopped other session"
real    0m2.222s
user    0m1.391s
sys     0m0.850s

real    0m2.152s
user    0m1.325s
sys     0m0.845s

real    0m2.224s
user    0m1.402s
sys     0m0.841s

real    0m2.151s
user    0m1.334s
sys     0m0.834s

[Mattias Wadman]

Ok that i guess rules out that this is caused by jq. You sure there isn't some kind of power management thing going on? that would be my guess

[Payam Nab]

And same as above if I run stats $HOME in one terminal and jq -n in another

[Payam Nab]

Its power settings shows nothing:

[Mattias Wadman]

You might need to use some CLI tool etc and it might even be specific to what hardware your using. I know very little about linux (?) power management so maybe have a look on stackoverflow etc

[Payam Nab]

Thanks a lot man, you are real life saver ...
Was pulling my hairs to find the problem, now I know where to look

acpitool -c
  CPU type               : Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz 
  Min/Max frequency      : 400/4700 MHz
  Current frequency      : 784 MHz
  Frequency governor     : powersave

[Mattias Wadman]

👍 good luck

[Payam Nab]

👍 good luck

So the problem was intel_pstate, disabled it in grub and set cpu governor to performance.
All is good and much better than multiple instances

while true ; do time bash -c 'for i in `seq 1 1000` ; do jq -n . > /dev/null; done' ;done
real    0m1.506s
user    0m1.249s
sys     0m0.355s

Thanks again for your help.