PJSIP TLS handshake fails with "wrong curve" -- Grandstream HT801v2, Asterisk 22.8.2 in Docker

Hi all,
Asterisk 22.8.2 in Docker (andrius/asterisk:latest, Debian, OpenSSL 3.x) with PJSIP TLS on 5061. Grandstream HT801v2 as ATA. Was working fine until ~2 days ago, broke without intentional changes.
WARNING: pjproject: SSL SSL_ERROR_SSL (Handshake): err: <error:0A00017A:SSL routines::wrong curve>

Already tried (none helped): re-issuing the cert, explicit cipher suites, removing cipher restrictions, full Docker rebuild, container restart.

Questions:

  1. Anyone seen error:0A00017A (wrong curve) with recent Asterisk 22.x / pjproject builds?

  2. Is ca_list_file required on the transport? Could the “Error reading CA certificates from buffer” be causing the handshake to fail?

  3. Known incompatibilities between pjproject TLS and Grandstream ATAs?

Thanks.

Did you do an update a few days ago to Asterisk?

Did you do a firmware update to the HY801 a few days ago?

Does it work if you turn off TLS?

Did you do an update a few days ago to Asterisk?

Did you do a firmware update to the HY801 a few days ago?

Does it work if you turn off TLS?

  • I did not upgrade the firmware before the bug happened.
  • I upgraded the firmware to the latest version to solve the issue but it had no effect.
  • Yes it works if I turn off TLS.

Did you contact Grandstream’s tech support?

This is the OpenSSL 3.x ECDH curve validation change — I’ve run into this on a few Asterisk boxes after base image rebuilds. The “wrong curve” error means your certificate’s elliptic curve key doesn’t match what the HT801 is offering during the TLS handshake.

**What’s happening:** OpenSSL 3.x got stricter about validating that the server certificate’s EC key curve is in the client’s supported groups list. OpenSSL 1.1.x was lenient about this — it would negotiate around mismatches. 3.x just refuses. This is [documented in OpenSSL issue #18280]("wrong curve" error during mutual authentication since upgrade from 1.1.0l to 1.1.1n · Issue #18280 · openssl/openssl · GitHub).

The Grandstream HT801v2 likely only supports `secp256r1` (P-256) for ECDH, maybe `secp384r1` depending on firmware. If your certificate was generated with a different curve, or if the ECDH negotiation picks a curve the HT801 doesn’t speak, you get “wrong curve.”

**Quick diagnosis — check your cert’s key type:**

```bash

openssl x509 -in /etc/asterisk/keys/asterisk.crt -text -noout | grep -A 2 “Public Key”

```

If it shows `secp384r1` or `secp521r1`, that’s your problem. If it’s RSA, the issue is in the ECDHE key exchange negotiation instead.

**The fix — regenerate with P-256:**

```bash

# Generate P-256 key (universally compatible with SIP devices)

openssl ecparam -genkey -name prime256v1 -out /etc/asterisk/keys/asterisk.key

# Self-signed cert

openssl req -new -x509 -key /etc/asterisk/keys/asterisk.key \

-out /etc/asterisk/keys/asterisk.crt \

-days 3650 -subj “/CN=asterisk.local”

# Combine for Asterisk

cat /etc/asterisk/keys/asterisk.key /etc/asterisk/keys/asterisk.crt \

/etc/asterisk/keys/asterisk.pem

# Reload

asterisk -rx “module reload res_pjsip.so”

```

**Or use RSA if you want to sidestep curves entirely:**

```bash

openssl req -new -x509 -nodes -newkey rsa:2048 \

-keyout /etc/asterisk/keys/asterisk.key \

-out /etc/asterisk/keys/asterisk.crt \

-days 3650 -subj “/CN=asterisk.local”

```

**Docker-specific note:** since you’re running in Docker, this probably broke because a base image rebuild pulled in a newer OpenSSL. Your certs are likely mounted from the host and were generated with the old OpenSSL. The fix is the same — regenerate with P-256 — but you might also want to pin your base image digest so this doesn’t silently break again.

**For your pjsip transport, make sure you have:**

```ini

[tls-transport]

type = transport

protocol = tls

bind = 0.0.0.0:5061

cert_file = /etc/asterisk/keys/asterisk.crt

priv_key_file = /etc/asterisk/keys/asterisk.key

method = tlsv1_2

```

Also `method = tlsv1_2` is worth setting explicitly — the HT801 almost certainly doesn’t do TLS 1.3 and you avoid a whole other class of negotiation issues.

Re: your `ca_list_file` question — those CA read errors are a red herring. `ca_list_file` is only needed if you’re doing mutual TLS (verifying client certs) or verifying against a CA chain. For self-signed setups with `verify_client = no`, you can omit it entirely.

I put together a deeper writeup on this whole OpenSSL 3 + PJSIP TLS situation at Fix Asterisk PJSIP TLS Wrong Curve Error After OpenSSL 3 Upgrade | ViciStack Blog if you want the full picture of what changed and why.

If this works please post here, and also please feed this back to Grandstream’s tech support!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.