does not get the full content of the page

Question

does not get the full content of the page

drma-tech opened this issue 3 years ago · comments

there are some pages (probably generate their content through javascript) that only work if I run this link through the browser.
https://api.allorigins.win/get?url=https://countryeconomy.com/national-minimum-wage

but if i run this code, it brings me json, but with the page incomplete. i know this code doesn't specifically run javascript, but i'm calling a json api, so theoretically it should come complete.

using (HttpClient client = new HttpClient())
{
    using (var requestMessage = new HttpRequestMessage(HttpMethod.Get, "https://api.allorigins.win/get?url=" + url))
    {
        var response = await client.SendAsync(requestMessage);
        using HttpContent content = response.Content;

        var sourcedata = await content.ReadAsStringAsync();

        var obj = System.Text.Json.JsonSerializer.Deserialize<allorigins>(sourcedata);

        return obj?.contents ?? "";
    }
}

I found some people commenting that this should make it work if I was accessing the page directly (which isn't the case), but it didn't work anyway.
client.DefaultRequestHeaders.Add("User-Agent", "C# console program");

this is the return, which is incomplete.
{"contents":"<!DOCTYPE html><html lang=\\"en\\" dir=\\"ltr\\" prefix=\\"content: http://purl.org/rss/1.0/modules/content/ dc: http://purl.org/dc/terms/ foaf: http://xmlns.com/foaf/0.1/ rdfs: http://www.w3.org/2000/01/rdf-schema# sioc: http://rdfs.org/sioc/ns# sioct: http://rdfs.org/sioc/types# skos: http://www.w3.org/2004/02/skos/core# xsd: http://www.w3.org/2001/XMLSchema# og: http://ogp.me/ns# schema: http://schema.org/ \\"><head><meta charset=\\"utf-8\\" />\\n<link rel=\\"canonical\\" href=\\"…ent(\\"script\\");s.setAttribute(\\"data-gdpr-applies\\", \\"${gdpr}\\");s.setAttribute(\\"data-consent-string\\", \\"${gdpr_consent}\\");s.src=\\"https://static.sunmedia.tv/integrations/55128703-3dfc-4c42-ba25-2b30788cc412/55128703-3dfc-4c42-ba25-2b30788cc412.js\\",s.async=!0,n.document.body.appendChild(s)}\\n}\\n</script></body></html>","status":{"url":"https://countryeconomy.com/national-minimum-wage","content_type":"text/html; charset=UTF-8","content_length":178622,"http_code":200,"response_time":1113}}