Protocol Security Violation C11: Fingerprint mismatch does not immediately tear down the media session

[songxpu]

1. Problem Overview

this is a serious vulnerability.

libpeer computes the SHA-256 fingerprint of the DTLS peer certificate and compares it with the fingerprint declared in the SDP. When the two do not match, the code only returns -1 from the handshake function, while the PeerConnection remains in the CONNECTED state and the already-formed DTLS/SRTP session state is not torn down immediately. This behavior violates the mandatory requirement to terminate the media session immediately when fingerprint identity verification fails.

2. Specification Requirements

RFC 8842 Section 5.1 requires that the certificate received during the DTLS handshake must match the fingerprint declared in the SDP fingerprint attribute. If the fingerprints do not match, the endpoint must tear down the media session immediately. This is mandatory behavior, not optional.

"The certificate received during the DTLS handshake [RFC6347] MUST match a certificate fingerprint received in SDP "fingerprint" attributes according to the procedures defined in [RFC8122]. If fingerprints do not match the hashed certificate, then an endpoint MUST tear down the media session immediately (see [RFC8122])."

3. Code Analysis

In the dtls_srtp_handshake function, the mismatch branch only returns -1 and performs no cleanup or state transition. The caller peer_connection_loop only enters PEER_CONNECTION_COMPLETED on a successful handshake; on failure it does not transition the PeerConnection to FAILED or CLOSED. The relevant code is as follows:

/* src/dtls_srtp.c:445-454 — dtls_srtp_handshake */
const mbedtls_x509_crt* remote_crt;
if ((remote_crt = mbedtls_ssl_get_peer_cert(&dtls_srtp->ssl)) != NULL) {
  dtls_srtp_x509_digest(remote_crt, dtls_srtp->actual_remote_fingerprint);

  if (strncmp(dtls_srtp->remote_fingerprint,
              dtls_srtp->actual_remote_fingerprint,
              DTLS_SRTP_FINGERPRINT_LENGTH) != 0) {
    LOGE("Actual and Expected Fingerprint mismatch: %s %s",
         dtls_srtp->remote_fingerprint,
         dtls_srtp->actual_remote_fingerprint);
    return -1;
  }
}

/* src/peer_connection.c:304-317 — peer_connection_loop */
case PEER_CONNECTION_CONNECTED:
  if (dtls_srtp_handshake(&pc->dtls_srtp, NULL) == 0) {
    LOGD("DTLS-SRTP handshake done");
    if (pc->config.datachannel) {
      LOGI("SCTP create socket");
      sctp_create_association(&pc->sctp, &pc->dtls_srtp);
      pc->sctp.userdata = pc->config.user_data;
    }
    STATE_CHANGED(pc, PEER_CONNECTION_COMPLETED);
  }
  break;

peer_connection_loop(CONNECTED)
  -> dtls_srtp_handshake
     -> fingerprint mismatch -> return -1
        -> no reset/dealloc, no transition to FAILED/CLOSED

The return -1 branch does not call dtls_srtp_reset_session(), does not free the SRTP contexts, and does not transition the PeerConnection to FAILED/CLOSED. The caller only handles the success case, so after a mismatch the connection remains in the CONNECTED state and the media session is not actively torn down.

4. Impact Analysis

This is a direct security vulnerability: the code does not clean up the DTLS/SRTP state immediately after fingerprint verification fails, and performs no state transition to prevent subsequent operations. A session that should have been terminated may still be used, exposing the communicating users to man-in-the-middle attack risk.