Async submit gradually taking more time
abin-mathew opened this issue · comments
I have noticed that my sms service slows down under high traffic. I am using async submit, and to debug this I have added a log to print the time taken to do an async submit. From the log I can see that the time taken for async submit increases gradually from about 2 millisec to almost 400 millisecs. Under what situation can this happen? I though since the submit is async the time taken should be constant. My logging code is :
long start = Calendar.getInstance().getTimeInMillis();
ZdAsyncSmppSession zdSession = smppSessionPoolContainer.getSession();
SmppSession session = zdSession.getSession();
session.sendRequestPdu(submit, 30000, false);
long end = Calendar.getInstance().getTimeInMillis();
long timeTaken = end - start;
log.debug("time taken for 1 async submit by "+zdSession.getName()+" : "+timeTaken);
SMPP is a windowed protocol. The idea being that you can asynchronously submit up to X requests that have not yet been responded to / acknowledged. Unless you've entirely turned off the windowing feature, sendRequestPDU will find / wait until a slot opens up in the window before it asynchronously sends the PDU onto the wire.
If you find the time is going up for sendRequestPDU -- my guess is that you are somewhat overloading your upstream SMSC and the windowing feature is doing what it is supposed to do -- slow down your submits. In this case, if you find it maxes out at 400 ms, then ultimately your SMSC is taking 400 ms to respond / accept submit requests. The math is pretty easy when you think about it. If your window is set to 20 requests and your SMSC really takes 400 ms to respond to them, then you'll pretty quickly find that your app overloads the SMSC and sendRequestPDU almost becomes synchronous in some regards since it's waiting for a slot to fill up.
It sort of is what it is -- the problem really is the other side -- SMSCs don't handle unlimited throughput.
One more thing. You can easily print out the window stats to confirm this is what is likely happening.
The sample for performance client may help.
session.getSendWindow() will return the window object that you can print out stats against as your app runs.
Initially I thought it s a windowing issue. But changing the window size didn't have any effect on this delay. To confirm this I have added logs to print window stats as well. The log I added is
Window<Integer, PduRequest, PduResponse> window = session.getSendWindow();
log.debug("Window data. max size : "+window.getMaxSize()+" size : "+window.getSize()+" free : "+window.getFreeSize()+" pending : "+window.getPendingOfferCount()+" session : "+zdSession.getName()+" time : "+timeTaken);
The logs printed for a single session at three points during the execution are
12:32:30.618[DEBUG][taskExecutor-23] [smpp] [ZdAsyncMessageSender.java:97] - Window data. max size : 50 size : 1 free : 49 pending : 0 session : Account_2000140116_36 time : 24
12:32:31.649[DEBUG][taskExecutor-43] [smpp] [ZdAsyncMessageSender.java:97] - Window data. max size : 50 size : 3 free : 47 pending : 0 session : Account_2000140116_36 time : 133
12:32:33.850[DEBUG][taskExecutor-44] [smpp] [ZdAsyncMessageSender.java:97] - Window data. max size : 50 size : 3 free : 47 pending : 0 session : Account_2000140116_36 time : 461
From the log it appears that even when the window is not full the async submit time gradually increases. Is there anything else that I am missing?