medcl / gopa-abandoned

GOPA, a spider written in Go.(NOTE: this project moved to https://github.com/infinitbyte/gopa )

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Gopa

https://github.com/medcl/gopa

[狗爬], A Spider Written in Go.

Travis Go Report Card

Build Gopa

Mac/Linux: Run make build to build the Gopa

Windows: go with this wiki - How to build GOPA on windows

Download

Release

Start Gopa

After download/build the binary file, run ./gopa to start the Gopa

Run ./gopa -h to get the full list of commandline options

  • -log option : logging level,can be set to trace,debug,info,warn,error ,default is info
  • -daemon option : run in background as daemon
  • -pprof option : enable and setup pprof/expvar service, eg: localhost:6060 , the endpoint will be: http://localhost:6060/debug/pprof/ and http://localhost:6060/debug/vars
  • -cpuprofile option : write cpu profile to this file
  • -memprofile option : write memory profile to this file

Stop Gopa

It's safety to press ctrl+c stop the current running Gopa, Gopa will handle the rest,saving the checkpoint, you may restore the job later,the world is still in your hand.

If you are running Gopa as daemon, you can stop it like this:

 kill -QUIT `pgrep gopa`

UI

Visit http://127.0.0.1:9001/ for more details.

License

Released under the Apache License, Version 2.0 .

About

GOPA, a spider written in Go.(NOTE: this project moved to https://github.com/infinitbyte/gopa )

License:Other


Languages

Language:Go 91.9%Language:HTML 6.8%Language:JavaScript 0.9%Language:Makefile 0.4%Language:Batchfile 0.0%Language:CSS 0.0%Language:Shell 0.0%