Operating web applications can be complicated. Luckily, a number of open source tools are available today to make our lives easier. They give insights into deployed systems by providing solutions for logging, monitoring, distributed tracing and visualization.
In our migration to Kubernetes we (re-)deployed those solutions. In the past, we have secured access to such internal services with basic authentication configured in HAproxy. This approach was suboptimal because it needed tedious manual work to update the list of users.
Instead, it would be nice to use an existing identity provider. We are using GitHub to host our source code. Therefore it seemed natural to use oAuth2 with GitHub as the resource server.
For the first service, we tried Bitly’s oauth2_proxy. Unfortunately, the project is no longer maintained and only allows a single upstream. Of course it is possible to expose multiple services behind an nginx reverse proxy but this setup seemed quite cumbersome to use.
We wanted to achieve two things:
We found a solution by chance. When evaluating different API gateway solutions, we ended up choosing Ambassador. It is built on the envoy proxy, which we already use in our service mesh istio. Ambassador’s configuration is decentralized. This allows service teams to autonomously expose their services, without the help of an operations team. Adding an annotation to a service is enough to expose it.
apiVersion: v1 kind: Service metadata: name: my-service annotations: getambassador.io/config: | --- apiVersion: ambassador/v1 kind: Mapping ambassador_id: production-gateway name: my-service-mapping prefix: /my-service service: my-service.prod:80 spec: ...
This configuration will expose
https://api.example.com/my-service, given that the ambassador instance
production-gateway is exposed on that particular domain.
Ambassador provides a flexible mechanism to provide custom authorization for your API. All requests must pass through this service before they reach their intended destination. A
200 OK response from the
AuthService indicates successful authentication and makes Ambassador forward it to the actual service.
annotations: getambassador.io/config: | --- apiVersion: ambassador/v1 kind: AuthService name: authentication ambassador_id: internal-gateway auth_service: "ambassador-github-oauth.internal-gateway:3000" proto: http --- apiVersion: ambassador/v1 kind: Mapping name: login_mapping ambassador_id: internal-gateway prefix: /auth/login rewrite: /auth/login service: ambassador-github-oauth.internal-gateway:3000 bypass_auth: true
bypass_auth directive to disable authentication for requests to the
Because nothing similar existed, we wrote a small service which provides GitHub OAuth2 authentication for Ambassador.
We deployed a separate Ambassador instance named
internal-gateway and exposed it on a
A GitHub OAuth App is registered to point to this domain.
Certmanager manages automatic certificate issuance and renewals:
apiVersion: certmanager.k8s.io/v1alpha1 kind: Certificate metadata: name: internal-gateway-tls namespace: internal-gateway spec: secretName: internal-gateway-tls dnsNames: - internal.yourdomain.com issuerRef: name: letsencrypt kind: ClusterIssuer acme: config: - dns01: provider: <dns-provider-name> domains: - internal.yourdomain.com
With the above in place, all development teams could expose their internal services with minimal configuration:
annotations: getambassador.io/config: | --- apiVersion: ambassador/v1 kind: TLSContext name: grafana_context ambassador_id: internal-gateway hosts: - grafana.yourdomain.com secret: grafana-tls.monitoring --- apiVersion: ambassador/v1 kind: Mapping name: grafana_mapping ambassador_id: internal-gateway host: grafana.yourdomain.com prefix: / service: monitoring-grafana.monitoring:80