Recent Posts

Debuging Session

Today we encountered an intriguing production resource leak that deserved to be shared. The Observation: In our production server, the CPU and Memory usage continued to increase while the new request came in. In the development server where no requests were served, the resource usage is stable. The server code was written in Go. Two suspicions were drawn from the observation right away. The problematic code resides in the hot path, ie.
read more

Metaverse

The word Metaverse still bugs me. I have learned the word metadata means the data of data, and the word metaprogramming means the programing of programing (macros, annotations, code generations etc). In Reactive programing I have also seen the word metastream, meaning a stream of streams. Given the word Metaverse, shouldn’t it just be the (uni)verse of (uni)verse? In my understanding metaverse, should it exist, can only be found in a higher dimension than the universe we know about.
read more

查看Pod重启的原因

在Kubernetes中，有时候Pod会异常重启。事后发现的时候错误原因在控制台面板里已经看不到了。 实际上Kubernetes提供了相关的工具。我们可以在不可恢复的异常发生时拦截异常，将其写入/dev/termination-log。 package main import ( "fmt" "os" ) func main() { defer func() { if r := recover(); r != nil { _ = os.WriteFile( "/dev/termination-log", []byte(fmt.Sprintf("panic: %s", r)), os.ModePerm, ) panic(r) } }() // ... } 查看的时候，我们可以在edit pod时在lastState当中找到错误信息： apiVersion:v1kind:Pod...lastState:terminated:containerID:...exitCode:0finishedAt:...message:| panic: goroutine 1 [running]:main.main.func1()/Users/donew/src/kitty/main.go:18+0x131panic(0x5192e20,0x5640e10)/usr/local/Cellar/go/1.16/libexec/src/runtime/panic.go:965+0x1b9main.main()/Users/donew/src/kitty/main.go:21+0x5b... 也可以直接用kubectl 查看。 kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" 写入的路径可以在terminationMessagePath中更改。 除此之外，如果项目不方便拦截错误，还可以将terminationMessagePolicy设置为FallbackToLogsOnError。 此时使用容器日志输出的最后一块作为终止消息。 日志输出限制为 2048 字节或 80 行，以较小者为准。 实际操作中，FallbackToLogsOnError的长度限制导致有时候会截取太少，丢掉重要信息。 这里提供一个简单的最佳实践：将正常的日志打印到stdout中，将致命错误打印到stderr(Go里的panic默认就是stderr，不用拦截了)，然后将terminationMessagePath设置为/dev/stderr，即可精确的获取Pod重启原因。 参考阅读：https://kubernetes.io/zh/docs/tasks/debug-application-cluster/determine-reason-pod-failure/
read more

An Elegant Way to Bootstrap Go App

The twelve-factor methodology has proven its worth over the years. Since its invention many fields in technology have changed, many among them are shining and exciting. In the age of Kubernetes, service mesh and serverless architectures, the twelve-factor methodology has not faded away, but rather has happened to be a good fit for nearly all of those powerful platforms. Scaffolding a twelve-factor go app may not be a difficult task for experienced engineers, but certainly presents some challenges to juniors.
read more

Keys to Fast Development

I have been on both sides. Sometimes people are amazed at our project grounding up from zero so soon, and some other times people are questioning why our project progress is that slow. It was said that compiled/strong-typed languages were faster at runtime, but slower to code. While it is true to a certain degree, it rarely matters in commercial applications. There was also a common belief that enforcing code qualities by means of TDD/Code review/testing-in-general would increase the time to launch.
read more

More

All Posts