[BUG]: NullReferenceException when placing Order in Alpaca SDK

Question

[BUG]: NullReferenceException when placing Order in Alpaca SDK

ObaidaAlhaasan opened this issue a year ago · comments

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Our trading strategy is implemented in a .NET Core application and uses the Alpaca SDK to place Buy/Sell orders on Alpaca's Paper Trading environment. On June 23, 2023, while running our strategy, we encountered 9 NullReferenceExceptions. The exceptions occurred while trying to place orders for different stocks. The exceptions were thrown within the Alpaca SDK. Attached a sample stack trace from one of the exceptions

Expected Behavior

We expected the Alpaca SDK to handle our Buy/Sell orders without throwing any exceptions. If there was a problem with a specific order, we expected an error message related to the specifics of that order, not a NullReferenceException which suggests a programming error within the SDK.that

Steps To Reproduce

Run our trading strategy on the Alpaca Paper Trading environment.
The strategy attempts to place Buy/Sell orders for various stocks.
Observe that NullReferenceExceptions are thrown from within the Alpaca SDK.

Environment

SDK Version: 6.2.1 Latest
OS (version, bitness): Windows 10 Pro
.NET SDK (version): Asp Net Core 7
target process .NET version/bitness:

Anything else?

 Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Net.Http.HttpConnection.WriteAsciiStringAsync(String s, Boolean async)
   at System.Net.Http.HttpConnection.WriteHeadersAsync(HttpHeaders headers, String cookiesFromContainer, Boolean async)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Alpaca.Markets.ThrottleParameters.CustomHttpHandler.sendAsyncWithTimeout(HttpRequestMessage request, CancellationToken cancellationToken)
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Polly.Wrap.AsyncPolicyWrapEngine.<>c__DisplayClass2_0`1.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Polly.Wrap.AsyncPolicyWrapEngine.ImplementationAsync[TResult](Func`3 func, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext, IAsyncPolicy outerPolicy, IAsyncPolicy`1 innerPolicy)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Alpaca.Markets.ThrottleParameters.CustomHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Alpaca.Markets.HttpClientExtensions.callAndDeserializeAsync[TApi,TJson](HttpClient httpClient, HttpRequestMessage request, CancellationToken cancellationToken)
   at Alpaca.Markets.HttpClientExtensions.callAndDeserializeAsync[TApi,TJson,TContent](HttpClient httpClient, HttpMethod method, Uri endpointUri, TContent content, CancellationToken cancellationToken)
   at AlpacaStockTrader.<>c__DisplayClass7_0.<<PostOrderAsync>b__0>d.MoveNext() in 
   
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at AlpacaStockTrader.PostOrderAsync(Guid tradeActionId, StockId stockId, Decimal units, OrderActionType orderActionType) in

Oleg Rakhmatulin · Answer 1 · Mon Jun 26 2023 17:43:38 GMT+0800 (China Standard Time)

Thanks a lot, @ObaidaAlhaasan for reporting this issue. As you can see the real exception happened deep inside the .NET HTTP client infrastructure intertwined with Polly resending logic. But I still want to investigate this problem because it can signal some higher-level issues specific to ASP.NET scenarios. So, please, answer the following questions - it will help me to understand the situation better:

This problem never happened before last Friday, right? Have you changed any other dependency except the Alpaca SDK before that happened?
Did this exception happen for all orders placed by your strategy last Friday or just for some orders? Do these orders have something in common (type, duration, etc.), or all of them are different?
What type of authorization do you use? You probably use the basic one (with API key/secret pair), but it would be better to explicitly confirm it.
Can you reproduce the same problem in a more simple scenario with all dependencies/libraries but without real code that processes trading signals? Something like a basic ASP.NET app that sends several random orders and some/all of them failed with this exception.

In the meantime, I'll try to find a bug report in the .NET/Polly GitHub repositories with similar stack traces. My current understanding is that this problem is not in Alpaca SDK itself but something related to its dependencies.

Oleg Rakhmatulin · Answer 2 · Mon Jun 26 2023 19:14:40 GMT+0800 (China Standard Time)

And one additional thing. AFAIU from call stack you @ObaidaAlhaasan have HTTP log enabled in your application, right? Could you please confirm that these problematic orders were rejected by Alpaca REST API before the exception occurred? Looks like normal order placement works fine but if we try to re-send POST messages we've had a problem. Maybe we shouldn't apply retry policies for POST/PUT messages at all.

Obaida Alhassan · Answer 3 · Tue Jun 27 2023 00:49:42 GMT+0800 (China Standard Time)

Hi @OlegRa

Yes, you're correct. The problem first occurred last Friday. That was the first day we execute orders against Alpaca Paper trading API and before that, we were simulating the order executions on our end. We did not make any other significant changes to our dependencies, including the Alpaca SDK.
The exception was not universal across all orders. From the hundreds of orders we placed that day, the exception occurred 9 times. We examined the orders that threw exceptions and found no obvious commonalities in terms of type, duration, or any other specific parameters. The exceptions appeared to be sporadic, occurring amidst a diverse range of orders.
We are indeed using the basic type of authorization, employing the API key/secret pair for authentication with the Alpaca SDK.
In terms of reproducing the issue, it seems to be more complex than a simple scenario. The exceptions are occurring under high-volume conditions, specifically when executing hundreds of orders at several-minute intervals for various stocks, including both buy and sell orders for stocks previously bought. However, we understand the value of reproducible scenarios for debugging, and if it would be helpful, we are willing to create a separate application that mimics these conditions as closely as possible.

We appreciate your time and effort in investigating this issue. We understand that the root of the exception lies deep within the .NET HTTP client infrastructure and Polly resending logic, so any insights you can provide are immensely helpful.

Obaida Alhassan · Answer 4 · Tue Jun 27 2023 00:59:53 GMT+0800 (China Standard Time)

Yes we have HTTP log enabled, and I took a look and found when the NullRefException is thrown only the Start processing HTTP request POST is shown in the logs but the Received HTTP response wasn't shown. Therefreo I cannot confirm if those orders rejected from Alpaca first or not

`2023-06-23T23:23:05.3011550+03:00 [INF] (PID: 22772|System.Net.Http.HttpClient.IAlpacaTradingClient.LogicalHandler) Start processing HTTP request POST https://paper-api.alpaca.markets/v2/orders

But for other exceptions like insufficient qty available for order both HTTP request and HTTP response shown in logs

Failed to place Sell order. Exception: Alpaca.Markets.RestClientErrorException: insufficient qty available for order

Oleg Rakhmatulin · Answer 5 · Tue Jun 27 2023 16:21:03 GMT+0800 (China Standard Time)

@ObaidaAlhaasan, thanks a lot for your prompt answers. Unfortunately, I didn't find anything related to this problem in .NET and Polly issue trackers. But I still believe that the problem's root cause is in the SDK retry policy although I can't confirm that right now.

My suggestion is to build a pre-release version of SDK with a single change - the retry policy will work only for the HTTP GET request. In all other cases (PUT, POST, PATCH, DELETE) any error returned by the server will be immediately reported to the caller side.

This change is backward compatible at the API level but will change the behavior of existing code and I don't want to make it widely available until I'll be sure it helps in your case. The main difference will be in error handling - you should implement a retry policy (if you need it) and service-unavailable conditions handling on the caller side.

If you are OK with this approach and have time/ability to test this pre-release version using your application I can make this release later today.

Obaida Alhassan · Answer 6 · Sat Jul 08 2023 22:03:12 GMT+0800 (China Standard Time)

Hi @OlegRa , thanks a lot for your reply

I was monitoring the app last week and it seems the exception wasn't thrown anymore "Not sure why since nothing changed"

I'll monitor it next week and confirm if the issue still exists then we try the approach you mentioned otherwise will close the issue

Thank you