[Question] Are webworkers taken into account?

[Jogai]

So I'm loading a certain page, but the actual data that I want to scrape is set by a script trough a webworker. It listens to a stream on a different endpoint to get the actual values.

Question1: Are webworkers supposed to run? (probably not).

Question2 if answer#1 is no: Is it possible to inject the script to the page to try and execute it.

Question3 if answer#2 is no: what would need to be done to make webworkers run? I might take a shot at it.

For the curious, my code so far:

            IConfiguration angleSharpConfig = Configuration.Default
                    .WithDefaultLoader(new LoaderOptions { IsResourceLoadingEnabled = true })
                    .WithDefaultCookies().WithJs();

            try
            {
                string cookieValue = String.Empty;
                IBrowsingContext angleSharpContext = BrowsingContext.New(angleSharpConfig);

                    string baseUrl = Regex.Match(sensor.Url, @"^https?:\/+([\d+|\.])+\/").Groups[0].ToString();
                    string loginUrl = $@"{baseUrl}cgi/login?username={usr}&password={pass}";

                    using (WebClient client = new WebClient())
                    {
                        string loginInfo = client.DownloadString(loginUrl);
                        cookieValue = loginInfo.Split(new[] { "\n", Environment.NewLine },
                                StringSplitOptions.RemoveEmptyEntries)
                            .FirstOrDefault(sx => sx.Contains("="));
                        angleSharpContext.SetCookie(new Url(baseUrl), cookieValue);
                    }

                IDocument document = await angleSharpContext.OpenAsync(sensor.Url).WaitUntilAvailable();

                IHtmlCollection<IElement> t = document.QuerySelectorAll("#m1_page1 [data-sid]");
                //At this point the elements have default values and not yet set by the script despite using WaitUntilAvailable
            }

[Florian Rappl]

I've moved the question as Web Workers are a JS feature - they have nothing to do with the core.

[Jogai]

Thanks. Sorry, I had both repos open when I creted the issue. Must have mixed up the tabs.

[Florian Rappl]

Question1: Are webworkers supposed to run? (probably not).

Right now there is no WebWorker support. But I think adding them is possible (and is a good addition).

Question2 if answer#1 is no: Is it possible to inject the script to the page to try and execute it.

Well, a web worker would not work in standard DOM. Usually, the scripts behind web workers only work in the worker context, as they'd use the worker specific APIs for communication and computation.

So I guess "no" again.

Question3 if answer#2 is no: what would need to be done to make webworkers run? I might take a shot at it.

I think a start would be to add them to the exposed DOM objects. An example of a similar (yet quite different) object is the DOMParser (https://github.com/AngleSharp/AngleSharp.Js/blob/master/src/AngleSharp.Js/Dom/DomParser.cs). For the implementation the WHATWG spec can be followed: https://html.spec.whatwg.org/multipage/workers.html#dedicated-workers-and-the-worker-interface

Loading the referenced script is possible by going against the services of the associated browsing context. In there the resource loader should be used.