401 (Unauthorized) error on Azure

[mayerwin]

I am getting the following error on Azure:
Flurl.Http.FlurlHttpException: Request to https://query1.finance.yahoo.com/v7/finance/download/MCD?period1=1436745600&period2=1499817600&interval=1d&events=history&crumb= failed with status code 401 (Unauthorized). at Flurl.Http.Configuration.FlurlMessageHandler.d__1.MoveNext() in C:\projects\flurl\src\Flurl.Http.Shared\Configuration\FlurlMessageHandler.cs:line 59 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.d__58.MoveNext()

It works flawlessly for a day or two with requests made every few hours, then the above exception starts being thrown. It looks like the crumb parameter is empty? What could cause that?

[Karl Wan]

@mayerwin the unauthorised exception is thrown sometimes because yahoo does not include a cookie occasionally and no crumb is generated. There's a similar issue for quantmod joshuaulrich/quantmod@d54e593 .To solve it, I will implement a retry logic similar to quantmod in these few days, it will be updated later, thanks :)

[mayerwin]

Noted, I'll look forward to seeing the fix, thanks.

[pljanot]

Hello guys, I will just share some of my slim version of flurlClient with repeat logic for generic FlurlHttpException. Maybe you will find it usefull:

`
public class YahooFlurlClient
{
private static string _crumb;

    private const string QueryUrl = "https://query1.finance.yahoo.com/v7/finance/download";
    private const string CrumbUrl = "https://query1.finance.yahoo.com/v1/test/getcrumb";

    private const string Period1Tag = "period1";
    private const string Period2Tag = "period2";
    private const string IntervalTag = "interval";
    private const string EventsTag = "events";
    private const string CrumbTag = "crumb";

    private static readonly IDictionary<Period, string> PeriodMap = new Dictionary<Period, string>
    {
        {Period.Daily, "d"},
        {Period.Weekly, "w"},
        {Period.Monthly, "m"}
    };

    public static async Task<Stream> GetResponseStreamAsync(string symbol, DateTime? startTime, DateTime? endTime,
        Period period = Period.Daily, string events= "history")
    {
        var yahooClient = YahooClientFactory.GetYahooClient;

        //Example URI
        //https://query1.finance.yahoo.com/v7/finance/download/
        //SBUX?period1=1451602800&period2=1496140683&interval=1d&events=history&crumb=.BprRZODzhT
        var url = QueryUrl
            .AppendPathSegment(symbol)
            .SetQueryParam(Period1Tag, (startTime ?? new DateTime(1970, 1, 1)).ToUnixTimestamp())
            .SetQueryParam(Period2Tag, (endTime ?? DateTime.Now).ToUnixTimestamp())
            .SetQueryParam(IntervalTag, $"1{PeriodMap[period]}")
            .SetQueryParam(EventsTag, events)
            .SetQueryParam(CrumbTag, YahooClientFactory.Crumb);

        try
        {
            return await yahooClient.EnableCookies()
                .WithUrl(url)
                .GetAsync(CancellationToken.None)
                .ReceiveStream();
        }
        catch (Exception ex) when (!(ex is FlurlHttpException))
        {
            return await yahooClient.EnableCookies()
                .WithUrl(url)
                .GetAsync(CancellationToken.None)
                .ReceiveStream();
        }
    }
}

public static class YahooClientFactory
{
    public static IFlurlClient GetYahooClient => yahooClientInstance;
    public static string Crumb { get; }

    private static readonly IFlurlClient yahooClientInstance;
    private const string CookieUrl = "https://finance.yahoo.com";

    private static string userAgent = "User-Agent";
    private static string headerString =
        "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
    private const string CrumbUrl = "https://query1.finance.yahoo.com/v1/test/getcrumb";

    static YahooClientFactory()
    {
        yahooClientInstance =
            new FlurlClient()
                .WithHeader(userAgent, headerString)
                .EnableCookies();

        yahooClientInstance.WithUrl(CookieUrl)
            .GetAsync(CancellationToken.None)
            .Result
            .EnsureSuccessStatusCode();

        Crumb = yahooClientInstance
            .WithUrl(CrumbUrl)
            .GetAsync(CancellationToken.None)
            .ReceiveString()
            .Result;
    }
}

public enum Period
{
    Daily,
    Weekly,
    Monthly
}

public static class UtilsExtension
{

    private static readonly DateTime DefaultUnixDateTime = new DateTime(1970, 1, 1);

    public static string Name<T>(this T @enum)
    {
        string name = @enum.ToString();
        if (typeof(T).GetMember(name).First().GetCustomAttribute(typeof(EnumMemberAttribute))
                is EnumMemberAttribute attr && attr.IsValueSetExplicitly)
            name = attr.Value;
        return name;
    }

    public static T GetValue<T>(this string str)
        => (T)Convert.ChangeType(str, typeof(T));

    public static string ToUnixTimestamp(this DateTime dateTime)
        => ((DateTimeOffset)dateTime).ToUnixTimeSeconds().ToString();

}

}
`

[Karl Wan]

@delvier thanks for the contribution. I have taken some code & idea from your code sample in the commit, look nice to have a factory taken out from the main class :)

@mayerwin i have pushed a new package to nuget, retry/re-establishment is added. It should lower the probability of getting unauthorized error, please help check if it's less prone to throwing unauthorize exception, thanks :)

[mayerwin]

Thanks a lot! It seems to work much better already, I'll let you know if I still encounter issues.