One of the issues you’ll have to solve is to place metrics and health checks.
You need a simple way to tell if the app is up or down. If your service uses HTTP than you can just use /health and /metrics. If the service can be accessed by external users than you might want not to expose external metrics outside.
If you have an HTTP routing service in front of your app (like Nginx) you can block access to health and metrics from outside with router. But you might someday have rewrites that will rewrite URL from outside to the root of some app. In the case of a separate app that process tags you might expose internal data to the users of your site. Url /tags/health will send to the app’s /health. You can’t block with regexp .*/health$ as it still might be a valid resource for the site. In this case, you can use /healthz and no resources about health will be harmed.
But what if there is no HTTP-router to hide your internal URLs? In this case, I’d hide it to another private port. You still need to check the app’s public port (you app private port might be available but the public might have problems). Thus I’d add /healthz also to public port but with empty content to check only response codes - no internal data exposed to users.
What we can do with 2 ports in Kubernetes? Liveness probe can check /healthz at private port and readiness probe can check /healthz at the public port.
You can also add API for the private port to have the ability to hide instance from balancer by switching public /healthz response code to other than 200. It might be useful to debug the app.
- Monitorama PDX 2016 - Kelsey Hightower - healthz: Stop reverse engineering applications and start monitoring from the inside;
- PromLens - visual editor for Prometheus requests.