Villane / shttpc

Simple HTTP Client, wraps Apache HttpClient 4.0 to provide an easy to use Scala API

Simple HTTP Client

Wraps Apache HttpClient 4.0 to provide an easy to use Scala API. Java users have some benefits as well.

Copyright © 2009, Erkki Lindpere
All rights reserved.

This software is distributed under the MIT license. See the file License.txt for the full text.

The project name shttpc is pronounced like ShittyPC.

A Maven repository at villane.org contains both sources and binaries.

Maven note: you will have to add SHTTPC’s optional Maven dependencies to your own POM, they won’t get transitively resolved (they are marked optional because it’s annoying to transitively get a bunch of libraries that you don’t need)

Usage examples (Scala)

GET with params Map, comprehensions with NekoHTML parsed to Scala’s XML Nodes

import org.villane.shttpc._

object MetacriticTest {
  val http = new Http

  case class Result(name: String, url: String, score: Int)
  
  def feelingLucky(query: String) = {
    val html = http.get("http://www.metacritic.com/search/process" ? Map(
      "sort" -> "relevance",
      "termtype" -> "all",
      "ts" -> query,
      "ty" -> "0"
    )).asHtml

    val res = for {
      div <- html \ "body" \ "div"
      if (div \ "@id" == "pagecontainer")
      col <- div \ "div" \ "table" \ "tbody" \ "tr" \ "td"
      if (col \ "@id" == "rightcolumn")
      p <- col \ "p"
      if (p.text.startsWith("1."))
    } yield p

    Result(res \ "a" text, res \ "a" \ "@href" text, (res \ "span" text).toInt)
  }

  def main(sgra: Array[String]) = {
    println(feelingLucky("Brutal Legend").score)
  }
}

Alternate comprehension with SHTTPC’s XPath support

Note that XPath support is very limited, basically only expressions like the one below are supported.

	import BetterXPath._
    val res = for {
      p <- html / "body/div[@id='pagecontainer']/div/table/tbody/tr/td[@id='rightcolumn']/p"
      if (p.text.startsWith("1."))
    } yield p

Usage Examples (Java)

Similar Example to the above, but in Java.

import org.villane.shttpc.Http;
import org.villane.shttpc.IterableNodeList;
import org.villane.shttpc.SimpleHttpResponse;
import org.villane.shttpc.SimpleXPath;
import org.w3c.dom.Document;
import org.w3c.dom.Node;

public class MetacriticTestJava {

	public static class Result {
		public final String name;
		public final String url;
		public final Integer score;

		public Result(String name, String url, Integer score) {
			this.name = name;
			this.url = url;
			this.score = score;
		}
	}

	public static Result feelingLucky(String query) throws Exception {
		Http http = new Http();
		SimpleHttpResponse response = http.get(	"http://www.metacritic.com/search/process"
				+ "?sort=relevance&termtype=all&ts={}&ty=0", query);
		Document html = response.asDomHtml();
		IterableNodeList ps = SimpleXPath.evalNodeList(html,
				"/HTML/BODY/DIV[@id='pagecontainer']/DIV/TABLE/TBODY/TR/TD[@id='rightcolumn']/P");
		Node p = null;
		for (Node node : ps) {
			if (node.getTextContent().startsWith("1.")) {
				p = node;
				break;
			}
		}
		return new Result(
				SimpleXPath.evalNode(p, "A").getTextContent(),
				SimpleXPath.evalString(p, "A/@href"),
				SimpleXPath.evalNumber(p, "SPAN/text()").intValue()
		);
	}

	public static void main(String[] args) throws Exception {
		System.out.println(feelingLucky("Brutal Legend").score);
	}
}