skrape{it} is a Kotlin-based HTML/XML testing and web scraping library that can be used seamlessly in Spring-Boot, Ktor, Android or other Kotlin-JVM projects. The ability to analyze and extract HTML including client-side rendered DOM trees and all other XML-related markup specifications such as SVG, UML, RSS,... makes it unique. It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. First and foremost skrape{it} aims to be a testing tool (not tied to a particular test runner), but it can also be used to scrape websites in a convenient fashion.
- Deserialization of HTML/XML from websites, local html files and html as string to data classes / POJOs.
- Designed to deserialize HTML but can handle any XML-related markup specifications such as SVG, UML, RSS or XML itself.
- DSL to select html elements as well as supporting CSS query-selector syntax by string invocation.
- Http-Client without verbosity and ceremony to make requests and corresponding request options like headers, cookies etc in a fluent style interface.
- Pre-configure client regarding auth and other request settings
- Can handle client side rendered web pages. Javascript execution results can optionally be considered in the response body.
- Easy to use, idiomatic and type-safe DSL to ensure a high level of readability.
- Build-in matchers/assertions based on infix functions to archive a very high level of readability.
- DSL is behaving like a Fluent-Api to make data extraction/scraping as comfortable as possible.
- Not bind to a specific test-runner, framework or whatever.
- Open to use any other assertion library of your choice.
In addition, extensions for well-known testing libraries are provided to extend them with the mentioned skrape{it} functionality. Currently available:
You'll always find the latest documentation, release notes and examples regarding official releases at https://docs.skrape.it. The README file you are reading right now provides example related to the latest master. Just use it if you won't wait for latest changes to be released. If you don't want to read that much or just want to get a rough overview on how to use skrape{it}, you can have a look at the Documentation by Example section.
All our official/stable releases will be published to mavens central repository.
Gradle
dependencies {
implementation("it.skrape:skrapeit-core:1.0.0-alpha6")
}Maven
<dependency>
<groupId>it.skrape</groupId>
<artifactId>skrapeit-core</artifactId>
<version>1.0.0-alpha6</version>
</dependency>We are offering snapshot releases via jitpack. Thereby you can install every commit and version you want. But be careful, these are non official releases and may be unstable as well as breaking changes can occur at any time.
If you want be a bit more safe you can use a certain commit instead of referencing skrape{it}'s master branch as a dependency to avoid sudden breaking changes. ! Please make sure to go to https://jitpack.io/#skrapeit/skrape.it/ click the commits tab and the "Get it"-button to force a build of a certain commit.
Gradle
repositories {
maven { url = uri("https://jitpack.io") }
}
dependencies {
implementation("com.github.skrapeit:skrape.it:master-SNAPSHOT")
// or use a certain commit to avoid sudden breaking changes
implementation("com.github.skrapeit:skrape.it:<commit-hash-short>")
}Maven
<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
</repository>
</repositories>
...
<dependency>
<groupId>com.github.skrapeit</groupId>
<artifactId>skrape.it</artifactId>
<version>master-SNAPSHOT</version>
</dependency>@Test
fun `can read and return html from String`() {
htmlDocument("""
<html>
<body>
<h1>welcome</h1>
<div>
<p>first p-element</p>
<p class="foo">some p-element</p>
<p class="foo">last p-element</p>
</div>
</body>
</html>""") {
h1 {
findFirst {
text toBe "welcome"
}
p {
withClass = "foo"
findFirst {
text toBe "some p-element"
className toBe "foo"
}
}
p {
findAll {
text toContain "p-element"
}
findLast {
text toBe "last p-element"
}
}
}
}
}data class MyDataClass(
var httpStatusCode: Int = 0,
var httpStatusMessage: String = "",
var paragraph: String = "",
var allParagraphs: List<String> = emptyList(),
var allLinks: List<String> = emptyList()
)
class HtmlExtractionService {
fun extract() {
val extracted = skrape(HttpFetcher) {
request {
url = "http://localhost:8080/"
}
extractIt<MyDataClass> {
it.httpStatusCode = statusCode
it.httpStatusMessage = statusMessage.toString()
htmlDocument {
it.allParagraphs = p { findAll { eachText }}
it.paragraph = p { findFirst { text }}
it.allLinks = a { findAll { eachHref }}
}
}
}
print(extracted)
// will print:
// MyDataClass(httpStatusCode=200, httpStatusMessage=OK, paragraph=i'm a paragraph, allParagraphs=[i'm a paragraph, i'm a second paragraph], allLinks=[http://some.url, http://some-other.url])
}
}@Test
fun `dsl can skrape by url`() {
skrape(HttpFetcher) {
request {
url = "http://localhost:8080/example"
}
expect {
htmlDocument {
// all official html and html5 elements are supported by the DSL
div {
withClass = "foo" and "bar" and "fizz" and "buzz"
findFirst {
text toBe "div with class foo"
// it's possible to search for elements from former search results
// ⚠️ this is only available in jitpack version for now!
// e.g. search all matching span elements within the above div with class foo etc...
span {
findAll {
// do something
}
}
}
findAll {
toBePresentExactlyTwice
}
}
// can handle custom tags as well
"a-custom-tag" {
findFirst {
toBePresentExactlyOnce
text toBe "i'm a custom html5 tag"
text
}
}
// can handle custom tags written in css selctor query syntax
"div.foo.bar.fizz.buzz" {
findFirst {
text toBe "div with class foo"
}
}
// can handle custom tags and add selector specificas via DSL
"div.foo" {
withClass = "bar" and "fizz" and "buzz"
findFirst {
text toBe "div with class foo"
}
}
}
}
}
}fun getDocumentByUrl(urlToScrape: String) = skrape(BrowserFetcher) { // <--- pass Browser fetcher to include rendered JS
request { url = urlToScrape }
extract { htmlDocument { this } }
}
fun main() {
// do stuff with the document
println(getDocumentByUrl("https://docs.skrape.it").eachLink)
}class ExampleTest {
val myPreConfiguredClient = skrape(HttpFetcher) {
// url can be a plain url as string or build by #urlBuilder
request {
url = urlBuilder {
protocol = UrlBuilder.Protocol.HTTPS
host = "skrape.it"
port = 12345
path = "/foo"
queryParam = mapOf("foo" to "bar")
}
timeout = 5000 // optional -> defaults to 5000ms
followRedirects = true // optional -> defaults to true
userAgent = "some custom user agent" // optional -> defaults to "Mozilla/5.0 skrape.it"
cookies = mapOf("some-cookie-name" to "some-value") // optional
headers = mapOf("some-custom-header" to "some-value") // optional
}
preConfigured
}
@Test
fun `can use preconfigured client`() {
myPreConfiguredClient.expect {
status { code toBe 200 }
// do more stuff
}
// slightly modify preconfigured client
myPreConfiguredClient.apply {
request {
followRedirects = false
}
}.expect {
status { code toBe 301 }
// do more stuff
}
}
}If you need help, have questions on how to use skrape{it} or want to discuss features or bugs please raise issues on GitHub. If you want to discuss or ask more generel things or possible implementations as well as feature ideas you can join the #skrape-it channel on the Kotlin Slack.
- Issues: You can discuss and raise issues on GitHub.
- Slack: Join the #skrape-it channel on the Kotlin Slack.
- Twitter: Follow @skrape_it on Twitter for updates and release notifications.
- Stasckoverflow: post or search issues on Stackoverflow
Skrape{it} is and always will be free and open-source. I try to reply to everyone needing help using these projects. Obviously, the development, maintenance takes time.
However, if you are using this project and be happy with it or just want to encourage me to continue creating stuff or fund the caffeine and pizzas that fuel its development, there are few ways you can do it :-
- Starring and sharing the project 🚀 to help make it more popular
- Giving proper credit when you use skrape{it}, tell your friends and others about it 😃
- Sponsor Skrape{it} with a one-time donations via PayPal by just click this button →
or use the GitHub sponsors programm to support on a monthly basis 💖