gruchalski.com

Istio, cert-manager, Let’s Encrypt and HTTPS redirect

cert-manager ACME Let’s Encrypt with working catch-all HTTPS redirect, the how and the why
 5115  25 mins

I’m running Istio 1.16.1 with cert-manager 1.11.0 ClusterIssuer pointed at Let’s Encrypt using HTTP-01 challenge. I have a gateway with a virtual service and I would like to automatically redirect all HTTP traffic to HTTPS. Except of the Let’s Encrypt challenge. Because, after Let’s Encrypt documentation[1]:

The HTTP-01 challenge can only be done on port 80. Allowing clients to specify arbitrary ports would make the challenge less secure, and so it is not allowed by the ACME standard.

Istio documentation suggests using the ServerTLSSettings[2] httpRedirect: true but this has a very nasty side effect. All traffic, even challenge solving is redirected to HTTPS. No certificate will be issued or renewed until httpsRedirect is set to false. This can be observed in cert-manager logs (broken into multiple lines for readability):

E0708 15:04:16.187682       1 sync.go:190] cert-manager/challenges
    "msg"="propagation check failed"
    "error"="failed to perform self check GET request 'http://test.svcs.sh/.well-known/acme-challenge/KVPhulYn7MtBxU9rPnFumBfh8mutovrVi50UGzaEOCA': Get \"https://test.svcs.sh/.well-known/acme-challenge/KVPhulYn7MtBxU9rPnFumBfh8mutovrVi50UGzaEOCA\": EOF"
    "dnsName"="test.svcs.sh"
    "resource_kind"="Challenge"
    "resource_name"="test-certificate-mfjkm-1639063799-1954195061"
    "resource_namespace"="istio-system"
    "resource_version"="v1"
    "type"="HTTP-01"

The challenge needs to be done over HTTP but due to the httpRedirect=true, it gets redirected to HTTPS, which cannot work. The EOF is intriguing. Someone closes the connection!

The fix is relatively simple and if you are interesting in that part only, jump right fixing the problem. The rest of this post is an investigation for a supporting evidence why it is working and why the httpsRedirect: true doesn’t work.

§istio configuration

Here’s my baseline configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod-cluster
  namespace: istio-system
spec:
  acme:
    email: email@address
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-cluster
    solvers:
    - http01:
        ingress:
          class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-certificate
  namespace: istio-system
spec:
  secretName: test-certificate-secret
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
  - server auth
  - client auth
  dnsNames:
  - "test.svcs.sh"
  issuerRef:
    name: letsencrypt-prod-cluster
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-svcs-sh-gtw
  namespace: test
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "test.svcs.sh"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: test-certificate-secret
    hosts:
    - "test.svcs.sh"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: test-svcs-sh-vs
  namespace: test
spec:
  hosts:
  - "test.svcs.sh"
  gateways:
  - test-svcs-sh-gtw
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 3000
        host: some-backend-service

§fixing the problem

The solution can be found in one of the comments to this GitHub issue[3]. It’s really simple:

  1. Keep the httpRedirect set to false in the Gateway.
  2. Use a match.uri.prefix with scheme.exact: http and redirect.scheme: https to handle the redirect selectively in a VirtualService.

We arrive at the following definition:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-svcs-sh-gtw
  namespace: test
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "test.svcs.sh"
    tls:
      httpsRedirect: false # Handle redirect in the VirtualService
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: test-certificate-secret
    hosts:
    - "test.svcs.sh"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: test-svcs-sh-vs
  namespace: test
spec:
  hosts:
  - "test.svcs.sh"
  gateways:
  - test-svcs-sh-gtw
  http:
  - match:
    - uri:
        prefix: /
      scheme:
        exact: http
    redirect:
      scheme: https
      redirectCode: 302
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 3000
        host: some-backend-service

Problem fixed. The certificate is issued and the redirect works as expected:

1

curl -v http://test.svcs.sh/

*   Trying 167.235.105.89:80...
* Connected to test.svcs.sh (167.235.105.89) port 80 (#0)
> GET / HTTP/1.1
> Host: test.svcs.sh
> User-Agent: curl/7.86.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: https://test.svcs.sh/
< date: Sat, 08 Jul 2023 15:06:07 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host test.svcs.sh left intact

§what’s the difference?

The obvious question is: what’s different? It works but why? Let’s do some digging. I’m using Istio Gateway so the component to start investigating from is the istio-ingressgateway which is installed into the namespace where Istio installs to.

1

kubectl get pods -l app=istio-ingressgateway -n istio-system

NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-77cf7f984b-smhhn   1/1     Running   0          2d2h

Istio does many interesting things. Most of those features are supported by the Envoy proxy. Istio uses and wraps Envoy extensively. As soon as the Gateway is deployed, an Envoy listener is created, it remains the same during the whole investigation and there’s nothing interesting in it so let’s not focus too much on it.

1
2
3
4

istioctl proxy-config listeners \
    istio-ingressgateway-77cf7f984b-smhhn.istio-system \
    | grep test
0.0.0.0 8443  SNI: test.svcs.sh     Route: https.443.https.test-svcs-sh-gtw.test

The HTTPS listener is built from this part of the Gateway declaration:

1
2
3
4
5
6
7
8
9

  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: test-certificate-secret
    hosts:
    - "test.svcs.sh"

§how istio interacts with envoy proxy

  • Envoy configuration starts with a listener.
  • A listener contains a list of filters.
  • One of the filters deals with HTTP protocol traffic.
  • A filter uses a router do distribute the traffic inside of the mesh.
  • A router contains a list of virtual hosts.
  • A virtual host contains individual routes.

From Istio perspective: there’s always going to be one Envoy router dealing with HTTP traffic for each Istio ingress gateway component. Istio Gateway maps to Envoy’s virtual host. Individual routes are constructed from Istio VirtualService.

The relevant Istio code dealing with Envoy[4].

§gateway httpsRedirect in simple terms

I can use the istioctl proxy-config routes command to look at Envoy’s virtual hosts and routes.

I’m investigating HTTP traffic redirection to HTTPS so the point at which HTTP (port 80) traffic is accepted seems like a good place to start. The name of the virtual host is http.<Envoy’s listener bind port>, which is usually 8080. I’m looking for test.svcs.sh traffic handling Let’s look at the routing configuration for test.svcs.sh:80.

Here’s a good starting point to get the hand of some of those commands[5] used later in this post.

1
2
3
4

istioctl proxy-config \
    routes istio-ingressgateway-77cf7f984b-smhhn.istio-system \
    -o yaml \
    | yq '. | filter(.name == "http.8080") | .[0].virtualHosts | filter(.name == "test.svcs.sh:80")'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

- domains:
    - test.svcs.sh
  includeRequestAttemptCount: true
  name: test.svcs.sh:80
  requireTls: ALL
  routes:
    - decorator:
        operation: test-svcs-sh-vs:80/*
      match:
        caseSensitive: true
        headers:
          - name: :scheme
            stringMatch:
              exact: http
        prefix: /
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/test/virtual-service/test-svcs-sh-vs
      redirect:
        pathRedirect: ""
        responseCode: FOUND
        schemeRedirect: https
    - decorator:
        operation: some-backend-service.test.svc.cluster.local:3000/*
      match:
        caseSensitive: true
        prefix: /
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/test/virtual-service/test-svcs-sh-vs
      route:
        cluster: outbound|3000||some-backend-service.test.svc.cluster.local
        maxGrpcTimeout: 0s
        retryPolicy:
          hostSelectionRetryMaxAttempts: "5"
          numRetries: 2
          retriableStatusCodes:
            - 503
          retryHostPredicate:
            - name: envoy.retry_host_predicates.previous_hosts
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
          retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
        timeout: 0s

We can see the requireTls: ALL property.

Okay, let’s set the Gateway httpsRedirect to false and apply the change. Repeat the last istioctl command. This time the output is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

- domains:
    - test.svcs.sh
  includeRequestAttemptCount: true
  name: test.svcs.sh:80
  routes:
    - decorator:
        operation: test-svcs-sh-vs:80/*
      match:
        caseSensitive: true
        headers:
          - name: :scheme
            stringMatch:
              exact: http
        prefix: /
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/test/virtual-service/test-svcs-sh-vs
      redirect:
        pathRedirect: ""
        responseCode: FOUND
        schemeRedirect: https
    - decorator:
        operation: some-backend-service.test.svc.cluster.local:3000/*
      match:
        caseSensitive: true
        prefix: /
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/test/virtual-service/test-svcs-sh-vs
      route:
        cluster: outbound|3000||some-backend-service.test.svc.cluster.local
        maxGrpcTimeout: 0s
        retryPolicy:
          hostSelectionRetryMaxAttempts: "5"
          numRetries: 2
          retriableStatusCodes:
            - 503
          retryHostPredicate:
            - name: envoy.retry_host_predicates.previous_hosts
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
          retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
        timeout: 0s

Exactly the same except that the requireTls: ALL is gone. Envoy’s requireTls: ALL setting[6] has exactly the same meaning as httpsRedirect: true in Istio Gateway,:

⁣All requests must use TLS. If a request is not using TLS, a 301 redirect will be sent telling the client to use HTTPS.

This can be observed, with httpsRedirect: true:

1

curl -v http://test.svcs.sh/

*   Trying 167.235.105.89:80...
* Connected to test.svcs.sh (167.235.105.89) port 80 (#0)
> GET / HTTP/1.1
> Host: test.svcs.sh
> User-Agent: curl/7.86.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< location: https://test.svcs.sh
< date: Sat, 08 Jul 2023 15:46:40 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host test.svcs.sh left intact

§scheme redirect

The fix to the catch-all redirect and working ACME challenge uses the scheme redirect. The scheme redirect has been added as part of this feature request[7].

§why the eof

Indeed, this is a good question. At the end of the day, it shouldn’t matter if httpRedirect: true, or an individual route redirect is in use. Both should work 🤞.

From the perspective of the client the redirect is 301, 302, or any other 3xx supported by Istio.

Something else must be going on.

§cert-manager ingress and solver

What happens when cert-manager enters the challenge solving state?

  • An Ingress resource is created, as instructed by the ClusterIssuer .spec.acme.solvers.
  • This Ingress points at a solver Service resource.
  • The Service points at the solver pod.

Relevant cert-manager source code[8].

§clean investigation: investigation.svcs.sh

Let’s pick a new domain name and start from the non-working condition by requesting a certificate for investigation.svcs.sh while httpsRedirect is true. This immediately triggers the behavior:

E0708 16:32:22.491714       1 sync.go:190] cert-manager/challenges
    "msg"="propagation check failed"
    "error"="failed to perform self check GET request 'http://investigation.svcs.sh/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0': Get \"https://investigation.svcs.sh/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0\": EOF"
    "dnsName"="investigation.svcs.sh"
    "resource_kind"="Challenge"
    "resource_name"="test-certificate-p7zmc-788698794-972312837"
    "resource_namespace"="istio-system"
    "resource_version"="v1"
    "type"="HTTP-01"

§the ingress

cert-manager creates a new Ingress in the namespace where Istio is installed:

1

kubectl get ingress -n istio-system

NAME                        CLASS    HOSTS           ADDRESS   PORTS   AGE
cm-acme-http-solver-7djvq   <none>   investigation.svcs.sh             80      22m

1

kubectl get ingress cm-acme-http-solver-7djvq -n istio-system -o yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  creationTimestamp: "2023-07-08T07:17:27Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    acme.cert-manager.io/http-domain: "586089708"
    acme.cert-manager.io/http-token: "1089211970"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-7djvq
  namespace: istio-system
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: test-certificate-p7zmc-788698794-972312837
    uid: 94e8a8ca-3ce1-44a5-b08e-b0dccef48e70
  resourceVersion: "57527723"
  uid: a0a4d83b-aa5b-4fad-93c1-29a79fc00a77
spec:
  rules:
  - host: investigation.svcs.sh
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-dw2ws
            port:
              number: 8089
        path: /.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

§the service

The Ingress use a Service for its backend. The Service:

1

kubectl get service cm-acme-http-solver-dw2ws -n istio-system -o yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

apiVersion: v1
kind: Service
metadata:
  annotations:
    auth.istio.io/8089: NONE
  creationTimestamp: "2023-07-08T07:17:27Z"
  generateName: cm-acme-http-solver-
  labels:
    acme.cert-manager.io/http-domain: "586089708"
    acme.cert-manager.io/http-token: "1089211970"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-dw2ws
  namespace: istio-system
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: test-certificate-p7zmc-788698794-972312837
    uid: 94e8a8ca-3ce1-44a5-b08e-b0dccef48e70
  resourceVersion: "57527721"
  uid: 19863c57-d8c1-4a5d-b608-4689825f39cf
spec:
  clusterIP: 10.43.44.72
  clusterIPs:
  - 10.43.44.72
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 31493
    port: 8089
    protocol: TCP
    targetPort: 8089
  selector:
    acme.cert-manager.io/http-domain: "586089708"
    acme.cert-manager.io/http-token: "1089211970"
    acme.cert-manager.io/http01-solver: "true"
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

§pods

Since there’s a service, there are also pods, let’s find them with one of the Service selectors:

1

kubectl get pods -l "acme.cert-manager.io/http01-solver=true" -n istio-system

NAME                        READY   STATUS    RESTARTS   AGE
cm-acme-http-solver-djwm7   1/1     Running   0          31m

§the ingress is picked up by envoy

Because of the ClusterIssuer .spec.acme.solvers having an istio solver in the list, the solver route is picked up by Envoy:

1
2
3

istioctl proxy-config \
    routes istio-ingressgateway-77cf7f984b-smhhn.istio-system -o yaml \
    | yq '. | filter(.name == "http.8080") | .[0].virtualHosts | filter(.name == "investigation.svcs.sh:80")'

Produces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

- domains:
    - investigation.svcs.sh
  includeRequestAttemptCount: true
  name: investigation.svcs.sh:80
  requireTls: ALL
  routes:
    - decorator:
        operation: cm-acme-http-solver-dw2ws.istio-system.svc.cluster.local:8089/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0
      match:
        caseSensitive: true
        path: /.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/virtual-service/investigation-svcs-sh-cm-acme-http-solver-7djvq-istio-autogenerated-k8s-ingress
      route:
        cluster: outbound|8089||cm-acme-http-solver-dw2ws.istio-system.svc.cluster.local
        maxGrpcTimeout: 0s
        retryPolicy:
          hostSelectionRetryMaxAttempts: "5"
          numRetries: 2
          retriableStatusCodes:
            - 503
          retryHostPredicate:
            - name: envoy.retry_host_predicates.previous_hosts
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
          retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
        timeout: 0s
    - decorator:
        operation: test-svcs-sh-vs:80/*
      match:
        caseSensitive: true
        headers:
          - name: :scheme
            stringMatch:
              exact: http
        prefix: /
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/test/virtual-service/test-svcs-sh-vs
      redirect:
        pathRedirect: ""
        responseCode: FOUND
        schemeRedirect: https
    - decorator:
        operation: some-backend-service.test.svc.cluster.local:3000/*
      match:
        caseSensitive: true
        prefix: /
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/test/virtual-service/test-svcs-sh-vs
      route:
        cluster: outbound|3000||some-backend-service.test.svc.cluster.local
        maxGrpcTimeout: 0s
        retryPolicy:
          hostSelectionRetryMaxAttempts: "5"
          numRetries: 2
          retriableStatusCodes:
            - 503
          retryHostPredicate:
            - name: envoy.retry_host_predicates.previous_hosts
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
          retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
        timeout: 0s

Exactly the same as in case of test.svcs.sh before but with a new route:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

    - decorator:
        operation: cm-acme-http-solver-dw2ws.istio-system.svc.cluster.local:8089/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0
      match:
        caseSensitive: true
        path: /.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0
      metadata:
        filterMetadata:
          istio:
            config: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/virtual-service/investigation-svcs-sh-cm-acme-http-solver-7djvq-istio-autogenerated-k8s-ingress
      route:
        cluster: outbound|8089||cm-acme-http-solver-dw2ws.istio-system.svc.cluster.local
        maxGrpcTimeout: 0s
        retryPolicy:
          hostSelectionRetryMaxAttempts: "5"
          numRetries: 2
          retriableStatusCodes:
            - 503
          retryHostPredicate:
            - name: envoy.retry_host_predicates.previous_hosts
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
          retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
        timeout: 0s

Explanation:

  • Istio Gateway and a VirtualService are created for the domain name investigation.svcs.sh.
  • Istio creates virtual host routes in Envoy for the domain name.
  • The certificate resource for the domain name is requested, the cert-manager ClusterIssuer starts the dance.
    • cert-manager creates an Ingress for the investigation.svcs.sh domain name.
    • This gets picked up by Envoy and the route is rolled into an already existing Envoy virtual host.

We see that the route is configured to forward the traffic to the solver service:

1
2

      route:
        cluster: outbound|8089||cm-acme-http-solver-dw2ws.istio-system.svc.cluster.local

But the pod doesn’t receive anything. The log just sits there waiting for something to come through:

1

kubectl logs --follow cm-acme-http-solver-djwm7 -n istio-system

I0708 17:17:28.613173       1 solver.go:39] cert-manager/acmesolver 
    "msg"="starting listener"
    "expected_domain"="investigation.svcs.sh"
    "expected_key"="bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0.fXDK-85QmDd5AVhxsp1Aj3jeez2KozCIcoubbYfervY"
    "expected_token"="bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0"
    "listen_port"=8089

Nothing ever comes through.

§cert-manager.io/issue-temporary-certificate

One of the comments from the GitHub issue suggests using the cert-manager.io/issue-temporary-certificate: true annotation. Let’s see how that goes. I change the Certificate manifest to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-certificate
  namespace: istio-system
  annotations:
    cert-manager.io/issue-temporary-certificate: "true"
spec:
  secretName: test-certificate-secret
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
  - server auth
  - client auth
  dnsNames:
  - "investigation.svcs.sh"
  issuerRef:
    name: letsencrypt-prod-cluster
    kind: ClusterIssuer
    group: cert-manager.io

cert-manager now logs:

E0708 17:22:18.802362       1 sync.go:190] cert-manager/challenges
    "msg"="propagation check failed"
    "error"="wrong status code '404', expected '200'"
    "dnsName"="investigation.svcs.sh"
    "resource_kind"="Challenge"
    "resource_name"="test-certificate-wq6ft-788698794-972312837"
    "resource_namespace"="istio-system"
    "resource_version"="v1"
    "type"="HTTP-01"

This can be replicated with curl:

1
2

curl --insecure -I -L -v \
    http://investigation.svcs.sh/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0

*   Trying 167.235.105.89:80...
* Connected to investigation.svcs.sh (167.235.105.89) port 80 (#0)
> HEAD /.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0 HTTP/1.1
> Host: investigation.svcs.sh
> User-Agent: curl/7.86.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< location: https://investigation.svcs.sh/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0
location: https://investigation.svcs.sh/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0
< date: Sat, 08 Jul 2023 17:25:31 GMT
date: Sat, 08 Jul 2023 17:25:31 GMT
< server: istio-envoy
server: istio-envoy
< transfer-encoding: chunked
transfer-encoding: chunked

<
* Connection #0 to host investigation.svcs.sh left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://investigation.svcs.sh/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0'
*   Trying 167.235.105.89:443...
* Connected to investigation.svcs.sh (167.235.105.89) port 443 (#1)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: serialNumber=1234567890
*  start date: Jul  8 17:21:47 2023 GMT
*  expire date: Oct  6 17:21:47 2023 GMT
*  issuer: CN=cert-manager.local
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: HEAD]
* h2h3 [:path: /.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0]
* h2h3 [:scheme: https]
* h2h3 [:authority: investigation.svcs.sh]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15a811e00)
> HEAD /.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0 HTTP/2
> Host: investigation.svcs.sh
> user-agent: curl/7.86.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 404
HTTP/2 404
< x-dns-prefetch-control: on
x-dns-prefetch-control: on
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< etag: "wz51d2hkyd1sp"
etag: "wz51d2hkyd1sp"
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 2329
content-length: 2329
< vary: Accept-Encoding
vary: Accept-Encoding
< date: Sat, 08 Jul 2023 17:25:31 GMT
date: Sat, 08 Jul 2023 17:25:31 GMT
< x-envoy-upstream-service-time: 11
x-envoy-upstream-service-time: 11
< server: istio-envoy
server: istio-envoy

<
* Connection #1 to host investigation.svcs.sh left intact

Hmm, this doesn’t work either but… it’s no longer an EOF, it’s a HTTP 404. This is significant in the context of cert-manager code responsible for the handling of the challenge[9]. Especially this part:

1
2
3
4
5
6
7
8
9

		TLSClientConfig: &tls.Config{
			// If we're following a redirect, it's permissible for it to be HTTPS and
			// its certificate may be invalid (they are trying to get a certificate, after all!)
			// See: https://letsencrypt.org/docs/challenge-types/#http-01-challenge
			// > When redirected to an HTTPS URL, it does not validate certificates (since
			// > this challenge is intended to bootstrap valid certificates, it may encounter
			// > self-signed or expired certificates along the way).
			InsecureSkipVerify: true,
		},

Aha, cert-manager will accept any certificate as long as there is one.

  • cert-manager issued a temporary certificate and stored it in the secret expected by the Istio Gateway.
  • The request on port 80 is redirected to HTTPS at the httpsRedirect level.
  • The request arrives in the mesh and is matched against the VirtualService https match for the /* path prefix.
  • cert-manager ignores invalid certificate and accepts the redirect.
  • The redirect leads to the HTTPS endpoint but the response is HTTP 404. This error comes from some-backend-service.

§enough evidence for final conclusions

Frankly, the virtual host output with cert-manager solver route gives it all away but let’s break it down.

§why httpsRedirect: true cannot work

  • The original request from cert-manager self-check arrives on port 80.
  • Envoy immediately redirects it to HTTPS with the HTTP 301 status code.
  • cert-manager follows the redirect. There are two possibilities:
    1. The ClusterIssuer uses the cert-manager.io/issue-temporary-certificate: true annotation.
      • Because a temporary certificate exists, the Istio Gateway accepts a HTTPS request and Envoy doesn’t trip over as the redirected request was HTTPS.
      • The request enters the mesh via the Gateway port 443 and is dealt with inside of the route with the decorator.operation: some-backend-service.test.svc.cluster.local:3000/*.
      • It is forwarded to the backend service but backend service doesn’t known anything about /.well-known/acme-challenge/… path, so returns HTTP 404.
    2. The ClusterIssuer doesn’t use a temporary certificate.
      • The request enters the mesh via the Gateway port 443 and is dealt with inside of the route with the decorator.operation: some-backend-service.test.svc.cluster.local:3000/*.
      • There’s no certificate under the secret configured for the Gateway, Istio closes the connection.
      • cert-manager HTTP client receives an EOF error.

§why httpsRedirect with an explicit route redirect works

This is really obvious in its non-obviousness:

  • The route for the cert-manager solver sits before other routes defined by the VirtualService.
  • There is no redirect, the handling of the solver request happens before VirtualService.

§verify the final conclusion

To verify the final conclusion, I can do the following:

  • While cert-manager continues failing the self-check:
    • Enable trace logging on the Istio ingressgateway pod.
    • Disable httpsRedirect.
    • Observe the log.
§enable trace logging on istio ingressgateway
1

istioctl proxy-config log istio-ingressgateway-77cf7f984b-smhhn.istio-system --level http:trace
§observe the log
1

kubectl logs --follow istio-ingressgateway-77cf7f984b-smhhn -n istio-system

There’s a lot of output so I need to be fast. Among other messages, I can observe:

2023-07-08T18:43:59.970913Z	trace	envoy http	[C8] parsed 75 bytes
2023-07-08T18:44:00.372329Z	trace	envoy http	[C144420] parsing 264 bytes
2023-07-08T18:44:00.372364Z	trace	envoy http	[C144420] message begin
2023-07-08T18:44:00.372372Z	debug	envoy http	[C144420] new stream
2023-07-08T18:44:00.372392Z	trace	envoy http	[C144420] completed header: key=Host value=investigation.svcs.sh
2023-07-08T18:44:00.372402Z	trace	envoy http	[C144420] completed header: key=User-Agent value=cert-manager-challenges/v1.11.0 (linux/amd64) cert-manager/2a0ef53b06e183356d922cd58af2510d8885bef5
2023-07-08T18:44:00.372408Z	trace	envoy http	[C144420] completed header: key=Accept-Encoding value=gzip
2023-07-08T18:44:00.372415Z	trace	envoy http	[C144420] onHeadersCompleteBase
2023-07-08T18:44:00.372456Z	trace	envoy http	[C144420] completed header: key=Connection value=close
2023-07-08T18:44:00.372464Z	trace	envoy http	[C144420] Server: onHeadersComplete size=4
2023-07-08T18:44:00.372475Z	trace	envoy http	[C144420] message complete
2023-07-08T18:44:00.372488Z	debug	envoy http	[C144420][S18069518382980252164] request headers complete (end_stream=true):
':authority', 'investigation.svcs.sh'
':path', '/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0'
':method', 'GET'
'user-agent', 'cert-manager-challenges/v1.11.0 (linux/amd64) cert-manager/2a0ef53b06e183356d922cd58af2510d8885bef5'
'accept-encoding', 'gzip'
'connection', 'close'

2023-07-08T18:44:00.372497Z	debug	envoy http	[C144420][S18069518382980252164] request end stream
2023-07-08T18:44:00.372602Z	trace	envoy http	[C144420][S18069518382980252164] decode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:00.372618Z	trace	envoy http	[C144420][S18069518382980252164] decode headers called: filter=istio.alpn status=0
2023-07-08T18:44:00.372623Z	trace	envoy http	[C144420][S18069518382980252164] decode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:00.372626Z	trace	envoy http	[C144420][S18069518382980252164] decode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:00.372634Z	trace	envoy http	[C144420][S18069518382980252164] decode headers called: filter=istio.stats status=0
2023-07-08T18:44:00.372642Z	debug	envoy http	[C144420][S18069518382980252164] Sending local reply with details direct_response
2023-07-08T18:44:00.372672Z	trace	envoy http	[C144420][S18069518382980252164] encode headers called: filter=istio.stats status=0
2023-07-08T18:44:00.372678Z	trace	envoy http	[C144420][S18069518382980252164] encode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:00.372681Z	trace	envoy http	[C144420][S18069518382980252164] encode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:00.372687Z	trace	envoy http	[C144420][S18069518382980252164] encode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:00.372696Z	debug	envoy http	[C144420][S18069518382980252164] closing connection due to connection close header
2023-07-08T18:44:00.372708Z	debug	envoy http	[C144420][S18069518382980252164] encoding headers via codec (end_stream=true):
':status', '301'
'location', 'https://investigation.svcs.sh/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0'
'date', 'Sat, 08 Jul 2023 17:44:00 GMT'
'server', 'istio-envoy'
'connection', 'close'

This is coming from the httpsRedirect: true setting.

§disable the httpsRedirect

Change the httpsRedirect to false and watch the log closely. cert-manager doesn’t waste time. The following happens almost immediately:

2023-07-08T18:44:09.969969Z	trace	envoy http	[C6] parsed 75 bytes
2023-07-08T18:44:10.771798Z	trace	envoy http	[C144427] parsing 264 bytes
2023-07-08T18:44:10.771840Z	trace	envoy http	[C144427] message begin
2023-07-08T18:44:10.771849Z	debug	envoy http	[C144427] new stream
2023-07-08T18:44:10.771876Z	trace	envoy http	[C144427] completed header: key=Host value=investigation.svcs.sh
2023-07-08T18:44:10.771888Z	trace	envoy http	[C144427] completed header: key=User-Agent value=cert-manager-challenges/v1.11.0 (linux/amd64) cert-manager/2a0ef53b06e183356d922cd58af2510d8885bef5
2023-07-08T18:44:10.771896Z	trace	envoy http	[C144427] completed header: key=Accept-Encoding value=gzip
2023-07-08T18:44:10.771905Z	trace	envoy http	[C144427] onHeadersCompleteBase
2023-07-08T18:44:10.771908Z	trace	envoy http	[C144427] completed header: key=Connection value=close
2023-07-08T18:44:10.771917Z	trace	envoy http	[C144427] Server: onHeadersComplete size=4
2023-07-08T18:44:10.771930Z	trace	envoy http	[C144427] message complete
2023-07-08T18:44:10.771945Z	debug	envoy http	[C144427][S18087745377321120087] request headers complete (end_stream=true):
':authority', 'investigation.svcs.sh'
':path', '/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0'
':method', 'GET'
'user-agent', 'cert-manager-challenges/v1.11.0 (linux/amd64) cert-manager/2a0ef53b06e183356d922cd58af2510d8885bef5'
'accept-encoding', 'gzip'
'connection', 'close'

2023-07-08T18:44:10.771965Z	debug	envoy http	[C144427][S18087745377321120087] request end stream
2023-07-08T18:44:10.772238Z	trace	envoy http	[C144427][S18087745377321120087] decode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:10.772273Z	trace	envoy http	[C144427][S18087745377321120087] decode headers called: filter=istio.alpn status=0
2023-07-08T18:44:10.772313Z	trace	envoy http	[C144427][S18087745377321120087] decode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:10.772336Z	trace	envoy http	[C144427][S18087745377321120087] decode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:10.772350Z	trace	envoy http	[C144427][S18087745377321120087] decode headers called: filter=istio.stats status=0
2023-07-08T18:44:10.772969Z	trace	envoy http	[C144427][S18087745377321120087] decode headers called: filter=envoy.filters.http.upstream_codec status=4
2023-07-08T18:44:10.773192Z	trace	envoy http	[C144427][S18087745377321120087] decode headers called: filter=envoy.filters.http.router status=1
2023-07-08T18:44:10.773377Z	trace	envoy http	[C144427] parsed 264 bytes
2023-07-08T18:44:10.773609Z	trace	envoy http	[C144427][S18087745377321120087] continuing filter chain: filter=0x55d94218c000
2023-07-08T18:44:10.778127Z	trace	envoy http	[C144428] parsing 256 bytes
2023-07-08T18:44:10.778160Z	trace	envoy http	[C144428] message begin
2023-07-08T18:44:10.778180Z	trace	envoy http	[C144428] completed header: key=Cache-Control value=no-cache, no-store, must-revalidate
2023-07-08T18:44:10.778193Z	trace	envoy http	[C144428] completed header: key=Date value=Sat, 08 Jul 2023 17:44:10 GMT
2023-07-08T18:44:10.778200Z	trace	envoy http	[C144428] completed header: key=Content-Length value=87
2023-07-08T18:44:10.778208Z	trace	envoy http	[C144428] onHeadersCompleteBase
2023-07-08T18:44:10.778212Z	trace	envoy http	[C144428] completed header: key=Content-Type value=text/plain; charset=utf-8
2023-07-08T18:44:10.778221Z	trace	envoy http	[C144428] status_code 200
2023-07-08T18:44:10.778225Z	trace	envoy http	[C144428] Client: onHeadersComplete size=4
2023-07-08T18:44:10.778331Z	trace	envoy http	[C144427][S18087745377321120087] encode headers called: filter=istio.stats status=0
2023-07-08T18:44:10.778389Z	trace	envoy http	[C144427][S18087745377321120087] encode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:10.778423Z	trace	envoy http	[C144427][S18087745377321120087] encode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:10.778494Z	trace	envoy http	[C144427][S18087745377321120087] encode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:10.778513Z	debug	envoy http	[C144427][S18087745377321120087] closing connection due to connection close header
2023-07-08T18:44:10.778533Z	debug	envoy http	[C144427][S18087745377321120087] encoding headers via codec (end_stream=false):
':status', '200'
'cache-control', 'no-cache, no-store, must-revalidate'
'date', 'Sat, 08 Jul 2023 17:44:10 GMT'
'content-length', '87'
'content-type', 'text/plain; charset=utf-8'
'x-envoy-upstream-service-time', '5'
'server', 'istio-envoy'
'connection', 'close'

followed by a bunch of requests similar to:

2023-07-08T18:44:11.968591Z	trace	envoy http	[C8] parsed 75 bytes
2023-07-08T18:44:13.019511Z	trace	envoy http	[C144430] parsing 264 bytes
2023-07-08T18:44:13.019601Z	trace	envoy http	[C144430] message begin
2023-07-08T18:44:13.019611Z	debug	envoy http	[C144430] new stream
2023-07-08T18:44:13.019637Z	trace	envoy http	[C144430] completed header: key=Host value=investigation.svcs.sh
2023-07-08T18:44:13.019649Z	trace	envoy http	[C144430] completed header: key=User-Agent value=cert-manager-challenges/v1.11.0 (linux/amd64) cert-manager/2a0ef53b06e183356d922cd58af2510d8885bef5
2023-07-08T18:44:13.019658Z	trace	envoy http	[C144430] completed header: key=Accept-Encoding value=gzip
2023-07-08T18:44:13.019667Z	trace	envoy http	[C144430] onHeadersCompleteBase
2023-07-08T18:44:13.019670Z	trace	envoy http	[C144430] completed header: key=Connection value=close
2023-07-08T18:44:13.019680Z	trace	envoy http	[C144430] Server: onHeadersComplete size=4
2023-07-08T18:44:13.019694Z	trace	envoy http	[C144430] message complete
2023-07-08T18:44:13.019710Z	debug	envoy http	[C144430][S11783251202062423115] request headers complete (end_stream=true):
':authority', 'investigation.svcs.sh'
':path', '/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0'
':method', 'GET'
'user-agent', 'cert-manager-challenges/v1.11.0 (linux/amd64) cert-manager/2a0ef53b06e183356d922cd58af2510d8885bef5'
'accept-encoding', 'gzip'
'connection', 'close'

2023-07-08T18:44:13.019723Z	debug	envoy http	[C144430][S11783251202062423115] request end stream
2023-07-08T18:44:13.019858Z	trace	envoy http	[C144430][S11783251202062423115] decode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:13.019880Z	trace	envoy http	[C144430][S11783251202062423115] decode headers called: filter=istio.alpn status=0
2023-07-08T18:44:13.019887Z	trace	envoy http	[C144430][S11783251202062423115] decode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:13.019893Z	trace	envoy http	[C144430][S11783251202062423115] decode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:13.019903Z	trace	envoy http	[C144430][S11783251202062423115] decode headers called: filter=istio.stats status=0
2023-07-08T18:44:13.020260Z	trace	envoy http	[C144430][S11783251202062423115] decode headers called: filter=envoy.filters.http.upstream_codec status=4
2023-07-08T18:44:13.020367Z	trace	envoy http	[C144430][S11783251202062423115] decode headers called: filter=envoy.filters.http.router status=1
2023-07-08T18:44:13.020400Z	trace	envoy http	[C144430] parsed 264 bytes
2023-07-08T18:44:13.020544Z	trace	envoy http	[C144430][S11783251202062423115] continuing filter chain: filter=0x55d941dc5710
2023-07-08T18:44:13.023215Z	trace	envoy http	[C144431] parsing 256 bytes
2023-07-08T18:44:13.023309Z	trace	envoy http	[C144431] message begin
2023-07-08T18:44:13.023360Z	trace	envoy http	[C144431] completed header: key=Cache-Control value=no-cache, no-store, must-revalidate
2023-07-08T18:44:13.023385Z	trace	envoy http	[C144431] completed header: key=Date value=Sat, 08 Jul 2023 17:44:13 GMT
2023-07-08T18:44:13.023403Z	trace	envoy http	[C144431] completed header: key=Content-Length value=87
2023-07-08T18:44:13.023445Z	trace	envoy http	[C144431] onHeadersCompleteBase
2023-07-08T18:44:13.023464Z	trace	envoy http	[C144431] completed header: key=Content-Type value=text/plain; charset=utf-8
2023-07-08T18:44:13.023487Z	trace	envoy http	[C144431] status_code 200
2023-07-08T18:44:13.023569Z	trace	envoy http	[C144431] Client: onHeadersComplete size=4
2023-07-08T18:44:13.023705Z	trace	envoy http	[C144430][S11783251202062423115] encode headers called: filter=istio.stats status=0
2023-07-08T18:44:13.023770Z	trace	envoy http	[C144430][S11783251202062423115] encode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:13.023791Z	trace	envoy http	[C144430][S11783251202062423115] encode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:13.023817Z	trace	envoy http	[C144430][S11783251202062423115] encode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:13.023861Z	debug	envoy http	[C144430][S11783251202062423115] closing connection due to connection close header
2023-07-08T18:44:13.023898Z	debug	envoy http	[C144430][S11783251202062423115] encoding headers via codec (end_stream=false):
':status', '200'
'cache-control', 'no-cache, no-store, must-revalidate'
'date', 'Sat, 08 Jul 2023 17:44:13 GMT'
'content-length', '87'
'content-type', 'text/plain; charset=utf-8'
'x-envoy-upstream-service-time', '3'
'server', 'istio-envoy'
'connection', 'close'

and another bunch of requests like this one:

2023-07-08T18:44:21.968492Z	trace	envoy http	[C3] parsed 75 bytes
2023-07-08T18:44:22.749799Z	trace	envoy http	[C144440] parsing 265 bytes
2023-07-08T18:44:22.749841Z	trace	envoy http	[C144440] message begin
2023-07-08T18:44:22.749850Z	debug	envoy http	[C144440] new stream
2023-07-08T18:44:22.749875Z	trace	envoy http	[C144440] completed header: key=Host value=investigation.svcs.sh
2023-07-08T18:44:22.749884Z	trace	envoy http	[C144440] completed header: key=User-Agent value=Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
2023-07-08T18:44:22.749890Z	trace	envoy http	[C144440] completed header: key=Accept value=*/*
2023-07-08T18:44:22.749898Z	trace	envoy http	[C144440] completed header: key=Accept-Encoding value=gzip
2023-07-08T18:44:22.749904Z	trace	envoy http	[C144440] onHeadersCompleteBase
2023-07-08T18:44:22.749907Z	trace	envoy http	[C144440] completed header: key=Connection value=close
2023-07-08T18:44:22.749914Z	trace	envoy http	[C144440] Server: onHeadersComplete size=5
2023-07-08T18:44:22.749925Z	trace	envoy http	[C144440] message complete
2023-07-08T18:44:22.749939Z	debug	envoy http	[C144440][S3714970174883588932] request headers complete (end_stream=true):
':authority', 'investigation.svcs.sh'
':path', '/.well-known/acme-challenge/bgLcgdeOvQ4nbDaD4wENG-qR76dxR8GkZKe2Wp1msX0'
':method', 'GET'
'user-agent', 'Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)'
'accept', '*/*'
'accept-encoding', 'gzip'
'connection', 'close'

2023-07-08T18:44:22.749956Z	debug	envoy http	[C144440][S3714970174883588932] request end stream
2023-07-08T18:44:22.750082Z	trace	envoy http	[C144440][S3714970174883588932] decode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:22.750731Z	trace	envoy http	[C144440][S3714970174883588932] decode headers called: filter=istio.alpn status=0
2023-07-08T18:44:22.750789Z	trace	envoy http	[C144440][S3714970174883588932] decode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:22.750842Z	trace	envoy http	[C144440][S3714970174883588932] decode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:22.750865Z	trace	envoy http	[C144440][S3714970174883588932] decode headers called: filter=istio.stats status=0
2023-07-08T18:44:22.751403Z	trace	envoy http	[C144440][S3714970174883588932] decode headers called: filter=envoy.filters.http.upstream_codec status=4
2023-07-08T18:44:22.751436Z	trace	envoy http	[C144440][S3714970174883588932] decode headers called: filter=envoy.filters.http.router status=1
2023-07-08T18:44:22.751443Z	trace	envoy http	[C144440] parsed 265 bytes
2023-07-08T18:44:22.751507Z	trace	envoy http	[C144440][S3714970174883588932] continuing filter chain: filter=0x55d941e44900
2023-07-08T18:44:22.754621Z	trace	envoy http	[C144441] parsing 256 bytes
2023-07-08T18:44:22.754641Z	trace	envoy http	[C144441] message begin
2023-07-08T18:44:22.754656Z	trace	envoy http	[C144441] completed header: key=Cache-Control value=no-cache, no-store, must-revalidate
2023-07-08T18:44:22.754666Z	trace	envoy http	[C144441] completed header: key=Date value=Sat, 08 Jul 2023 17:44:22 GMT
2023-07-08T18:44:22.754671Z	trace	envoy http	[C144441] completed header: key=Content-Length value=87
2023-07-08T18:44:22.754710Z	trace	envoy http	[C144441] onHeadersCompleteBase
2023-07-08T18:44:22.754720Z	trace	envoy http	[C144441] completed header: key=Content-Type value=text/plain; charset=utf-8
2023-07-08T18:44:22.754731Z	trace	envoy http	[C144441] status_code 200
2023-07-08T18:44:22.754734Z	trace	envoy http	[C144441] Client: onHeadersComplete size=4
2023-07-08T18:44:22.754827Z	trace	envoy http	[C144440][S3714970174883588932] encode headers called: filter=istio.stats status=0
2023-07-08T18:44:22.754935Z	trace	envoy http	[C144440][S3714970174883588932] encode headers called: filter=envoy.filters.http.cors status=0
2023-07-08T18:44:22.755045Z	trace	envoy http	[C144440][S3714970174883588932] encode headers called: filter=envoy.filters.http.fault status=0
2023-07-08T18:44:22.755092Z	trace	envoy http	[C144440][S3714970174883588932] encode headers called: filter=istio.metadata_exchange status=0
2023-07-08T18:44:22.755185Z	debug	envoy http	[C144440][S3714970174883588932] closing connection due to connection close header
2023-07-08T18:44:22.755289Z	debug	envoy http	[C144440][S3714970174883588932] encoding headers via codec (end_stream=false):
':status', '200'
'cache-control', 'no-cache, no-store, must-revalidate'
'date', 'Sat, 08 Jul 2023 17:44:22 GMT'
'content-length', '87'
'content-type', 'text/plain; charset=utf-8'
'x-envoy-upstream-service-time', '3'
'server', 'istio-envoy'
'connection', 'close'

Let’s Encrypt talks over port 80. We just observed the solving of the challenge, the certificate has been issued.

§final conclusion

  1. Keep the httpRedirect set to false in the Gateway.
  2. Use a match.uri.prefix with scheme.exact: http and redirect.scheme: https to handle the redirect selectively in a VirtualService.

That’s the best way to handle catch-all HTTPS redirect and I was able to fnd an explanation for it.

  1. Let’s Encrypt HTTP-01 challenge 

  2. Istio ServerTLSSettings 

  3. httpsRedirect breaks cert-manager acme challenge, which was working in Istio 1.6 

  4. Istio sources: Envoy interaction 

  5. Debugging Envoy and Istiod 

  6. Envoy config.route.v3.VirtualHost.TlsRequirementType 

  7. Envoy: New feature: Scheme redirect in RedirectAction 

  8. Ingress handling cert-manager source code. 

  9. cert-manager ACME HTTP-01 solver testReachability function 

See Also: