Go gRPC Clean architecture microservice with Prometheus, Grafana monitoring and Jaeger opentracing ⚡️
Let’s try to create closer to real world gRPC microservice with tracing and monitoring 👋:
Source code u can find here
Core tools used what will be used: 🚀
PostgreSQL as database
Redis for sessions and caching
Jaeger open source, end-to-end distributed tracing
Prometheus monitoring and alerting
Grafana for to compose observability dashboards with everything from Prometheus
First we need to start all necessary infrastructure containers:
run make localUI interfaces will be available on ports:
After sending any requests, you are able to monitoring of metrics at the dashboard of Prometheus.
For Prometheus go to http://localhost:9090/graph type gRPC and choose one.👨💻
You can import Grafana dashboard templates from the grafana directory, default login/password is **admin** and password **admin**
I like to use evans for simple testing gRPC.
In cmd folder let’s init all dependencies and start the app.
Viper is very good and common choice as complete configuration solution for Go applications.
We use here config-local.yml file approach.
https://gist.github.com/b7bc0f4d32cf177dcdd1acf119947cd4
Next let’s create logger, here i used Uber’s Zap under the hood, important here is to create Logger interface for be able to replace logger in the future if it’s need.
https://gist.github.com/5381d61ad4a6e789a4ad60db2ceb6d8d
Setup postgres and redis
Usually production SQL db standard solution for these days is combination of sqlx and pgx.
Good Redis Go clients is go-redis and redigo, i used first.
https://gist.github.com/b1c2f0b3ec88718cd8c37b3ad160792d
And let’s set up Jaeger:
https://gist.github.com/80c528ac6bf35e551e7fc4d5f29a5c37
And add the global tracer to our application:
https://gist.github.com/b3fd528eb1057975e28b188bea242806
Prometheus has have 4 types of metrics: Counter, Gauge, Histogram, Summary
To expose Prometheus metrics in a Go application, you need to provide a /metrics HTTP endpoint.
You can use the prometheus/promhttp library’s HTTP Handler as the handler function.
https://gist.github.com/d0335fe64d093eb5175a24ae24c22a4d
user.proto file
In gRPC documentation we can find good practice recommendations and naming conventions for writing proto files.
As you can see, each field in the message definition has a unique number. These field numbers are used to identify your fields in the message binary format, and should not be changed once your message type is in use. Note that field numbers in the range 1 through 15 take one byte to encode, including the field number and the field’s type (you can find out more about this in Protocol Buffer Encoding).
https://gist.github.com/f8d18b968a177a465be2e22a1e56d762
generate your user. proto file 🤓
https://gist.github.com/db2ea11aaabbf350b66bae31bf9fb149
it creates user.pb.go file with server and client interfaces what need to implement in our microservice:
https://gist.github.com/3eaa95b529868988255dbd4b1216bf4b
Then in server.go initialize the repository, use cases, metrics and so on then start gRPC server:
https://gist.github.com/393d749d12f0e34fb177c8f34da03599
I found this is very good gRPC Middleware repository, but we easy can create our own, for example logger interceptor:
https://gist.github.com/b9e6ac6acbd83c6b694aa962c762df2e
We can access grpc metadata in service handlers too, for example here we extract and validate session_id which client must send in the request context:
https://gist.github.com/50d1716a6f5e7346a8658abd961f02c8
So let’s create unary service handler for creating the new user:
https://gist.github.com/ea1e1ae622902fb2e84c1ceb15fc6aa9
On the first lines start tracing span. The “span” is the primary building block of a distributed trace.
Each component of the distributed system contributes a span — a named, timed operation representing a piece of the workflow.
opentracing
https://gist.github.com/5ed0313543e2dc4f2788b751a5c0d864
let’s check how it’s look in Jaeger:
open http://localhost:16686/
Then we usually have to validate request input, for errors gRPC has packages status and codes
I found good practice to parse and log errors in handler layer, here i use ParseGRPCErrStatusCode method, which parse err and returns matched gRPC code.
Validator is good solution for validation.
https://gist.github.com/51987e1d3016cadcc5a61cfee15910f6
After request input validation call use case method which contains business logic and works with users repository: https://gist.github.com/b08ed9515e8b26fd0f924dfdede2231b
Inside a.user.Register(ctx, user) method we start new tracing span and call user repository methods:
https://gist.github.com/a13e60f6fb5dea275d572ecc4c1e9d69
In user repository Create method we again start new tracing span and run our query
Important note here:
Good practice is always wrap err with some additional information, it’s will make debugging much easier in the future 👍
On Repository and UseCase levels usually we don’t log errors, only wrap with the message and returns, because we logging errors on top layer in handlers. 👨💻
So we don’t need log the one error multiple time, already warped it with debug message whats went wrong.
https://gist.github.com/4c50d3f7856730bf4799e51936b61ec7
Finally, service handler must return response object generated by proto, here usually we need to create helpers for map our internal business logic models to response object for return it.
https://gist.github.com/aefbcfa26c36c11b0b5ce76c10623302
Every app must be covered by tests, I didn’t completely cover all code this one, but wrote some test of course. For testing and mocking testify and gomock is very good tools.
Source code and list of all used tools u can find here 👨💻 :) I hope this article is usefully and helpfully, I’ll be happy to receive any feedbacks or questions :)
Originally published at http://github.com.
