skrapeit / skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Home Page:https://docs.skrape.it

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] mode = Mode.DOM seems to be not working anymore

dewijones92 opened this issue · comments

Describe the bug
https://docs.skrape.it/docs/dsl/extract-client-side-rendered-data
this says how to run the client side javascript

Code Sample

  val scrapedData = skrape {
       url = "http://some.url"
       mode = Mode.DOM // <--- here's the magic
       extract { 
           div {
               withClass = "dynamic"
               findFirst { text }
           }
       }
   }
   println(scrapedData)

Expected behavior
i get this error

None of the following functions can be called with the arguments supplied.
skrape(BlockingFetcher<TypeVariable(R)>, suspend Scraper<TypeVariable(R)>.() → TypeVariable(T))   where R = TypeVariable(R), T = TypeVariable(T) for    fun <R, T> skrape(fetcher: BlockingFetcher<R>, init: suspend Scraper<R>.() → T): T defined in it.skrape.fetcher
skrape(NonBlockingFetcher<TypeVariable(R)>, suspend Scraper<TypeVariable(R)>.() → TypeVariable(T))   where R = TypeVariable(R), T = TypeVariable(T) for    suspend fun <R, T> skrape(fetcher: NonBlockingFetcher<R>, init: suspend Scraper<R>.() → T): T defined in it.skrape.fetcher

Additional context
Is there a working repo where I can see this javascript example? thanks

Sorry for inconvenience, the docs are outdate for this example.

This is how you can archive it with the latest version

https://github.com/skrapeit/skrape.it#scrape-a-client-side-rendered-page

Hi @christian-draeger . Found this issue while trying to play with Mode.DOM also.
I had already tried with BrowserFetcher and it still seems the dom isn't loaded when the fetch happens.
The url I'm testing to fetch is https://www.fantrax.com/player/00454/pr6b2ivjlmccxj5j and printing the htmlDocument I can see that the whole <app-root> is empty (where all the content should be)

Any suggestion?