AWS NLB + NGINX Ingress + веб-сокеты == 502 Bad Gateway or Bad Handshake

Конфигурация контроллера входа (без изменений в файле ниже)

https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/aws/deploy.yaml

Сервис

kind: Service
apiVersion: v1
metadata:
  name: websockets-service
  namespace: development
  annotations:
    argocd.argoproj.io/sync-wave: "10"
spec:
  selector:
    app: project-websocket
  ports:
  #- name: http
  #  protocol: TCP
  #  port: 80
  #  targetPort: 9000
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
  type: ClusterIP

Ingress

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: websockets-ingress
  namespace: development
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"

    # nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/websocket-services: websockets-service
    nginx.org/websocket-services: websockets-service

    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header HOST $host;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_pass_request_headers on;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
    argocd.argoproj.io/sync-wave: "10"
spec:
  tls:
    - hosts:
      - backend-dev.project.com
      secretName: tls-secret
  rules:
  - host: backend-dev.project.com
    http:
      paths:
      - path: /ws/
        backend:
          serviceName: websockets-service
          servicePort: 443

пробуя различные клиенты веб-сокета CLI:

➜ ws --insecure wss://backend-dev.project.com/ws/table/123
websocket: bad handshake

➜ websocat -k  wss://backend-dev.project.com/ws/table/123
websocat: WebSocketError: Received unexpected status code (502 Bad Gateway)
websocat: error running

Все работает, если я проксирую порт из сервиса:

k port-forward svc/websockets-service 8443:443
Forwarding from 127.0.0.1:8443 -> 443
Forwarding from [::1]:8443 -> 443
Handling connection for 8443

➜ websocat -k wss://localhost:8443/ws/table/123
{"connect": {"channel_name": "specific.239f96e4a93a470688a12fc0bc8d0374!9d92256f6f65446386fb17b59d1aac57", "table_token": "123"}}

and in logs I see

127.0.0.1:50070 - - [21/Nov/2020:05:11:50] "WSCONNECTING /ws/table/123" - -
127.0.0.1:50070 - - [21/Nov/2020:05:11:50] "WSCONNECT /ws/table/123" - -

Информация об SSL

➜ curl -vvI https://backend-dev.project.com
*   Trying 44.235.41.143:443...
* Connected to backend-dev.project.com (44.235.41.143) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.project.com
*  start date: Oct 28 00:00:00 2020 GMT
*  expire date: Oct 28 23:59:59 2021 GMT
*  subjectAltName: host "backend-dev.project.com" matched cert's "*.project.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5579ce4a9af0)
> HEAD / HTTP/2
> Host: backend-dev.project.com
> user-agent: curl/7.69.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
HTTP/2 200 
< date: Sat, 21 Nov 2020 05:13:22 GMT
date: Sat, 21 Nov 2020 05:13:22 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 143552
content-length: 143552
< vary: Accept-Encoding
vary: Accept-Encoding
< server-timing: TimerPanel_utime;dur=31.268000000011398;desc="User CPU time", TimerPanel_stime;dur=0.0;desc="System CPU time", TimerPanel_total;dur=31.268000000011398;desc="Total CPU time", TimerPanel_total_time;dur=31.340599060058594;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
server-timing: TimerPanel_utime;dur=31.268000000011398;desc="User CPU time", TimerPanel_stime;dur=0.0;desc="System CPU time", TimerPanel_total;dur=31.268000000011398;desc="Total CPU time", TimerPanel_total_time;dur=31.340599060058594;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: DENY
x-frame-options: DENY
< vary: Cookie, Origin
vary: Cookie, Origin
< set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains

< 
* Connection #0 to host backend-dev.project.com left intact

nginx-ingress amazon-eks websocket aws-nlb
person DmitrySemenov    schedule 21.11.2020    source источник
comment
Я столкнулся с той же проблемой. Вы нашли решение?   -  person Stephan_Berlin    schedule 19.02.2021

Ответы (1)


arrow_upward
0
arrow_downward

Веб-сокеты и SSE (события, отправленные сервером) - это проблема AWS. Я обнаружил, что для SSE правильно работают только ALB.

Хорошие новости: в AWS есть входной контроллер ALB
Плохие новости: вот и все. Нет интеграции с nginx ingress

Что вы можете сделать, так это использовать сочетание alb-ingress-controller и nginx-ingress-controller:
Все сводится к установке alb-ingress-controller и ingress-nginx (голая диаграмма), а затем созданию входящего ресурса, который указывает на alb-ingress-controller в службу NodePort ingress-nginx.

Так что-то вроде:

kind: Ingress
metadata:
  name: "alb-ingress-to-nginx-ingress"
  labels:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/target-type: "ip"
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:yyy:certificate/xxxx
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
spec:
  rules:
  - host: 'toto.edu'
    http:
      paths:
      - backend:
          serviceName: nginx-controller
          servicePort: http

Ресурсы: средняя статья репозиторий github

person Arthur Woimbée    schedule 01.03.2021