401 (Unauthorized) error on Azure
mayerwin opened this issue · comments
I am getting the following error on Azure:
Flurl.Http.FlurlHttpException: Request to https://query1.finance.yahoo.com/v7/finance/download/MCD?period1=1436745600&period2=1499817600&interval=1d&events=history&crumb= failed with status code 401 (Unauthorized). at Flurl.Http.Configuration.FlurlMessageHandler.d__1.MoveNext() in C:\projects\flurl\src\Flurl.Http.Shared\Configuration\FlurlMessageHandler.cs:line 59 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.d__58.MoveNext()
It works flawlessly for a day or two with requests made every few hours, then the above exception starts being thrown. It looks like the crumb parameter is empty? What could cause that?
@mayerwin the unauthorised exception is thrown sometimes because yahoo does not include a cookie occasionally and no crumb is generated. There's a similar issue for quantmod joshuaulrich/quantmod@d54e593 .To solve it, I will implement a retry logic similar to quantmod in these few days, it will be updated later, thanks :)
Noted, I'll look forward to seeing the fix, thanks.
Hello guys, I will just share some of my slim version of flurlClient with repeat logic for generic FlurlHttpException. Maybe you will find it usefull:
`
public class YahooFlurlClient
{
private static string _crumb;
private const string QueryUrl = "https://query1.finance.yahoo.com/v7/finance/download";
private const string CrumbUrl = "https://query1.finance.yahoo.com/v1/test/getcrumb";
private const string Period1Tag = "period1";
private const string Period2Tag = "period2";
private const string IntervalTag = "interval";
private const string EventsTag = "events";
private const string CrumbTag = "crumb";
private static readonly IDictionary<Period, string> PeriodMap = new Dictionary<Period, string>
{
{Period.Daily, "d"},
{Period.Weekly, "w"},
{Period.Monthly, "m"}
};
public static async Task<Stream> GetResponseStreamAsync(string symbol, DateTime? startTime, DateTime? endTime,
Period period = Period.Daily, string events= "history")
{
var yahooClient = YahooClientFactory.GetYahooClient;
//Example URI
//https://query1.finance.yahoo.com/v7/finance/download/
//SBUX?period1=1451602800&period2=1496140683&interval=1d&events=history&crumb=.BprRZODzhT
var url = QueryUrl
.AppendPathSegment(symbol)
.SetQueryParam(Period1Tag, (startTime ?? new DateTime(1970, 1, 1)).ToUnixTimestamp())
.SetQueryParam(Period2Tag, (endTime ?? DateTime.Now).ToUnixTimestamp())
.SetQueryParam(IntervalTag, $"1{PeriodMap[period]}")
.SetQueryParam(EventsTag, events)
.SetQueryParam(CrumbTag, YahooClientFactory.Crumb);
try
{
return await yahooClient.EnableCookies()
.WithUrl(url)
.GetAsync(CancellationToken.None)
.ReceiveStream();
}
catch (Exception ex) when (!(ex is FlurlHttpException))
{
return await yahooClient.EnableCookies()
.WithUrl(url)
.GetAsync(CancellationToken.None)
.ReceiveStream();
}
}
}
public static class YahooClientFactory
{
public static IFlurlClient GetYahooClient => yahooClientInstance;
public static string Crumb { get; }
private static readonly IFlurlClient yahooClientInstance;
private const string CookieUrl = "https://finance.yahoo.com";
private static string userAgent = "User-Agent";
private static string headerString =
"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
private const string CrumbUrl = "https://query1.finance.yahoo.com/v1/test/getcrumb";
static YahooClientFactory()
{
yahooClientInstance =
new FlurlClient()
.WithHeader(userAgent, headerString)
.EnableCookies();
yahooClientInstance.WithUrl(CookieUrl)
.GetAsync(CancellationToken.None)
.Result
.EnsureSuccessStatusCode();
Crumb = yahooClientInstance
.WithUrl(CrumbUrl)
.GetAsync(CancellationToken.None)
.ReceiveString()
.Result;
}
}
public enum Period
{
Daily,
Weekly,
Monthly
}
public static class UtilsExtension
{
private static readonly DateTime DefaultUnixDateTime = new DateTime(1970, 1, 1);
public static string Name<T>(this T @enum)
{
string name = @enum.ToString();
if (typeof(T).GetMember(name).First().GetCustomAttribute(typeof(EnumMemberAttribute))
is EnumMemberAttribute attr && attr.IsValueSetExplicitly)
name = attr.Value;
return name;
}
public static T GetValue<T>(this string str)
=> (T)Convert.ChangeType(str, typeof(T));
public static string ToUnixTimestamp(this DateTime dateTime)
=> ((DateTimeOffset)dateTime).ToUnixTimeSeconds().ToString();
}
}
`
@delvier thanks for the contribution. I have taken some code & idea from your code sample in the commit, look nice to have a factory taken out from the main class :)
@mayerwin i have pushed a new package to nuget, retry/re-establishment is added. It should lower the probability of getting unauthorized error, please help check if it's less prone to throwing unauthorize exception, thanks :)
Thanks a lot! It seems to work much better already, I'll let you know if I still encounter issues.