whrss9527/go-template

English | 简体中文

Project Overview:

go-template

A Go microservice template project

Built with the go-kratos framework for quick project building

Go + Kratos + Gorm + GRPC + Docker + Jenkins + K8S

Features:

Getting Started

1. Install dependencies & plugins

Clone the project to your local machine, then execute the following command

$ sudo make init

2. Start services

$ kratos run

3. Test

$ cd internal/biz  # go to your test dirs
$ go test -v

Project Structure

├── api
│    └── template_proj 
│       ├── v1
│       │   └── xxx.proto (biz, generated by make api)
│       └── errors
│            └── errors.proto (error enum, generate by make errors)
├── bin
│    └── server (generated by make build)
├── cmd
│    └── server
│         ├── main.go (main function)
│         ├── wire.go (wire)
│         └── wire_gen.go (generate by wire)
├── configs
│     └── config.yaml (config file)
├── internal
│       ├── biz
│       │    ├── biz.go (define biz ProviderSet)
│       │    ├── main_test.go (test main function)
│       │    ├── template.go (biz logic, inclouding template usecase definition)
│       │    ├── template_test.go (test case for template.go)
│       │    └── usecase_manager.go (usecase manager, using for calling each other between modules)
│       ├── data
│       │    ├── model
│       │    │    └── user.go (msyql model)
│       │    ├── mysql
│       │    │    ├── mysql.go (define mysql ProviderSet)
│       │    │    └── user.go (mysql dao)
│       │    └── redis
│       │         ├── redis.go (define redis ProviderSet)
│       │         └── user.go (redis dao)
│       ├── conf
│       │    ├── conf.proto (config struct proto)
│       │    └── conf.pb.go (generate by make config)
│       ├── pkg
│       │    ├── db
│       │    │    └── db.go 
│       │    ├── redis
│       │    │    └── redis.go
│       │    └── redis_sync
│       │         └── redis_sync.go
│       ├── server
│       │    ├── grpc.go 
│       │    ├── http.go (http config, including middleware like jwt...)
│       │    └── server.go (define server ProviderSet)
│       └── util (some utils)
├── proto
│    ├── template_proj
│    │    ├── v1
│    │    │    └── xxx.proto (orginal proto file)
│    │    └── errors
│    │         └── errors.proto (orginal error enum)
│    └── third_party (some third party proto file)
├── go.mod  
├── go.sum (generate by make init)
├── Makefile
├── Jenkinsfile
├── Dockerfile
└── openapi.yaml (swagger file, generate by make api)

We divide the project structure into three layers: api, biz, and model.

The api is defined in "proto/template_proj/v1/xxx.proto". This api is the external interface, meaning that our business logic is exposed to the outside world through this interface.

After the definition is complete, use the "make api" command to generate the corresponding interface in "api/template_proj/v1/xxx.go".

Then implement these interfaces in "service/xxx.go". The "service" here is the entry point for our business logic.

The wire framework is used for dependency injection in the overall project.

ProviderSets are defined in cmd/server/wire.go. These ProviderSets are the objects used for dependency injection.

ProviderSets are defined in biz, mysql, redis, and server, and these ProviderSets are injected in cmd/server/wire.go.

The dependency injection chain: api -> service -> biz -> data

After implementing the interface in "service/xxx.go", we can implement the specific business in "biz/template.go" for "xxx.go" to call.

After implementing the business logic in "biz/template.go", we can implement the corresponding database operation in "data/mysql/user.go".

Inject "UseCase" in "service", inject "Repo" in "UseCase", and inject "DB" in "Repo".

API Documentation:

Provide API interface documentation for other developers to understand the API interface of this microservice. After defining proto, use the "make api" command to generate the "openapi.yaml" file in the project root directory. You can use the "swagger" tool to view the interface documentation or directly import the interface documentation into tools such as Postman.

Testing:

1. Mock DB

In unit testing, it is crucial to mock the database, and the database needs to be in a clean initial state and cannot be slow every time it runs.

(1) Mysql's Mock

The approach used here is borrowed from this big brother's method How to conduct Fake Testing for MySQL

Initialization of DB
In the db directory

type Config struct {
   DSN             string // write data source name.
   MaxOpenConn     int    // open pool
   MaxIdleConn     int    // idle pool
   ConnMaxLifeTime int
}

var DB *gorm.DB

// InitDbConfig initializes Db
func InitDbConfig(c *conf.Data) {
   log.Info("Initializing Mysql")
   var err error
   dsn := c.Database.Dsn
   maxIdleConns := c.Database.MaxIdleConn
   maxOpenConns := c.Database.MaxOpenConn
   connMaxLifetime := c.Database.ConnMaxLifeTime
   if DB, err = gorm.Open(mysql.Open(dsn), &gorm.Config{
      QueryFields: true,
      NamingStrategy: schema.NamingStrategy{
         //TablePrefix:   "",   // Table name prefix
         SingularTable: true, // Use singular table name
      },
   }); err != nil {
      panic(fmt.Errorf("Failed to initialize the database: %s \n", err))
   }
   sqlDB, err := DB.DB()
   if sqlDB != nil {
      sqlDB.SetMaxIdleConns(int(maxIdleConns))                               // Number of idle connections
      sqlDB.SetMaxOpenConns(int(maxOpenConns))                               // Maximum number of connections
      sqlDB.SetConnMaxLifetime(time.Second * time.Duration(connMaxLifetime)) // Unit: seconds
   }
   log.Info("Mysql: initialization completed")
}

Initialization and Injection of fake-mysql
In the fake_mysql directory

var (
   dbName    = "mydb"
   tableName = "mytable"
   address   = "localhost"
   port      = 3380
)

func InitFakeDb() {
   go func() {
      Start()
   }()
   db.InitDbConfig(&conf.Data{
      Database: &conf.Data_Database{
         Dsn:             "no_user:@tcp(localhost:3380)/mydb?timeout=2s&readTimeout=5s&writeTimeout=5s&parseTime=true&loc=Local&charset=utf8,utf8mb4",
         ShowLog:         true,
         MaxIdleConn:     10,
         MaxOpenConn:     60,
         ConnMaxLifeTime: 4000,
      },
   })
   migrateTable()
}

func Start() {
   engine := sqle.NewDefault(
      memory.NewMemoryDBProvider(
         createTestDatabase(),
         information_schema.NewInformationSchemaDatabase(),
      ))

   config := server.Config{
      Protocol: "tcp",
      Address:  fmt.Sprintf("%s:%d", address, port),
   }

   s, err := server.NewDefaultServer(config, engine)
   if err != nil {
      panic(err)
   }

   if err = s.Start(); err != nil {
      panic(err)
   }

}

func createTestDatabase() *memory.Database {
   db := memory.NewDatabase(dbName)
   db.EnablePrimaryKeyIndexes()
   return db
}

func migrateTable() {
// Generate a user table to fake mysql
   err := db.DB.AutoMigrate(&model.User{})
   if err != nil {
      panic(err)
   }
}

Call InitFakeDb() at the beginning of unit testing.

func setup() {
   fake_mysql.InitFakeDb()
}

(2) Redis's Mock

We use miniredis here, and the Redis Client that matches it is go-redis/redis/v8. Invoke InitTestRedis() to inject it.

// RedisClient redis client  
var RedisClient *redis.Client  
  
// ErrRedisNotFound not exist in redisconst ErrRedisNotFound = redis.Nil  
  
// Config redis config
type Config struct {  
   Addr         string  
   Password     string  
   DB           int  
   MinIdleConn  int  
   DialTimeout  time.Duration  
   ReadTimeout  time.Duration  
   WriteTimeout time.Duration  
   PoolSize     int  
   PoolTimeout  time.Duration  
   // tracing switch  
   EnableTrace bool  
}  
  
// Init instantiates a Redis client.  
func Init(c *conf.Data) *redis.Client {  
   RedisClient = redis.NewClient(&redis.Options{  
      Addr:         c.Redis.Addr,  
      Password:     c.Redis.Password,  
      DB:           int(c.Redis.DB),  
      MinIdleConns: int(c.Redis.MinIdleConn),  
      DialTimeout:  c.Redis.DialTimeout.AsDuration(),  
      ReadTimeout:  c.Redis.ReadTimeout.AsDuration(),  
      WriteTimeout: c.Redis.WriteTimeout.AsDuration(),  
      PoolSize:     int(c.Redis.PoolSize),  
      PoolTimeout:  c.Redis.PoolTimeout.AsDuration(),  
   })  
  
   _, err := RedisClient.Ping(context.Background()).Result()  
   if err != nil {  
      panic(err)  
   }  
  
   // hook tracing (using open telemetry)  
   if c.Redis.IsTrace {  
      RedisClient.AddHook(redisotel.NewTracingHook())  
   }  
  
   return RedisClient  
}  
  
// InitTestRedis instantiates a Redis client for unit testing.  
func InitTestRedis() {  
   mr, err := miniredis.Run()  
   if err != nil {  
      panic(err)  
   }  
   // Uncomment the following command to test the case where the link is closed.  
   // defer mr.Close()  
  
   RedisClient = redis.NewClient(&redis.Options{  
      Addr: mr.Addr(),  
   })  
   fmt.Println("mini redis addr:", mr.Addr())  
}

2. Unit Testing

After comparison, I chose the unit testing framework goconvey because it is much easier to use than the native go testing framework. goconvey also provides many useful features:

Multi-level nested testing
Rich assertions
Clear test results
Support for native go test

Use

go get github.com/smartystreets/goconvey

func TestLoverUsecase_DailyVisit(t *testing.T) {  
   Convey("Test TestLoverUsecase_DailyVisit", t, func() {  
      // clean  
      uc := NewLoverUsecase(log.DefaultLogger, &UsecaseManager{})  
  
      Convey("ok", func() {  
         // execute  
         res1, err1 := uc.DailyVisit("user1", 3)  
         So(err1, ShouldBeNil)  
         So(res1, ShouldNotBeNil)  
         // the n (>=2)times visit，should return nil
         res2, err2 := uc.DailyVisit("user1", 3)  
         So(err2, ShouldBeNil)  
         So(res2, ShouldBeNil)  
      })  
   })  
}

As you can see, the function signature is consistent with the original go test. The testing is nested in two levels of Convey, with the outer layer newing the parameters required by the inner-layer Convey. The inner layer calls the function and performs assertion on the return value.

Assertions can also compare return values like this So(x, ShouldEqual, 2) or judge the length, etc. So(len(resMap), ShouldEqual, 2)

The nesting of Convey can be flexible and multi-level, extending like a multi-branch tree, which can meet the needs of business simulation.

3. TestMain

Add a TestMain to serve as a unified entry for all cases

import (  
"os"  
"testing"  
  
. "github.com/smartystreets/goconvey/convey"  
)  
  
func TestMain(m *testing.M) {  
   setup()  
   code := m.Run()  
   teardown()  
   os.Exit(code)
}
// init fake db
func setup() {  
   fake_mysql.InitFakeDb()  
   redis.InitTestRedis()
}

Some Issues Encountered

1. Issue with Proto Interface Definitions

When defining proto interfaces with fields named in the a_b format, they are automatically converted to aB when generating OpenAPI documentation. This inconsistency in field names between the interface documentation and the actual code can be resolved by using the json_name tag. Here is an example:

message LoginReq {
  // Account
  string account = 1;
  string account_type = 2 [json_name = "account_type"];
}

2. Lack of Native Support for Content-Types Other than JSON

The system does not natively support Content-Types other than JSON, such as XML or HTML. To handle other Content-Types, you need to implement a custom ResponseEncoder and register it in the http.go file. Here's an example of a custom ResponseEncoder in Go:

func ResponseEncoder(w http.ResponseWriter, r *http.Request, data interface{}) error {
	
	respContentType := r.Header.Get("Response-Content-Type")
	if respContentType != "" && respContentType != "application/json" {
		switch respContentType {
		case "application/xml":
			w.Header().Set("Content-Type", "application/xml")

		case "application/x-protobuf":
			w.Header().Set("Content-Type", "application/x-protobuf")
		default:
			w.Header().Set("Content-Type", "application/json")
		}
		body := data.(v1.HttpBody)
		w.Write(data.Data)
	} else {
		// Serialize the data to JSON
		jsonRes, err := json.Marshal(data)
		if err != nil {
			return err
		}
		w.Header().Set("Content-Type", "application/json")
		w.Write(jsonRes)
	}
	return nil
}

To register this custom ResponseEncoder in http.go, you can use the following code:

var opts = []http.ServerOption{
    // Custom response data structure
    http.ResponseEncoder(middleware.ResponseEncoder), // Replace the default response data structure
    http.Middleware(),
}

3. Limited ORM Support

The system's support for Object-Relational Mapping (ORM) is relatively weak, and many features need to be implemented manually. However, this provides flexibility and room for custom implementation according to your specific requirements.

Contribution

Pull requests and/or issues are welcome.

License

MIT License

Author

Gmail: whrss9527@gamil.com

whrss9527 / go-template