WebSocket Enhancements: Proxies and Keepalives for ARI and Media Outbound Websockets.

See the notes below for high-level descriptions of the new features.

* Proxies

Outbound/forward HTTP proxies are now supported and configurable in
websocket_client.conf. You can specify a host:port plus optional proxy_username
and proxy_password. Because WebSockets aren't consistently supported among
proxies (specifically passing through UPGRADEs), the CONNECT method is always
used to establish a TCP tunnel through the proxy. This is required if a TLS
session is to be established with the WebSocket server anyway.  It's important
to understand that that negotiation with the proxy is ALWAYS unsecured. Once
the proxy establishes the tunnel, the TLS session will be negotiated directly
with the remote WebSocket server via the tunnel.

* Keepalives

Both TCP-level and WebSocket PING/PONG keepalives can be configured and are
available with either the curl or tcptls client implementations. The TCP
keepalives are handled entirely by the operating system and require no
resources from Asterisk but by their very nature, they can't traverse proxies.
WebSocket PING/PONGs are implemented in the Asterisk websocket code and require
a scheduler thread to keep track of them so they're a bit more complicated but
they do traverse proxies.  Which one is used is completely up to the admin.
You could use both.

* Other Changes

A few changes were needed to res/ari/ari_websockets and
res/res_aeap/transport_websocket to add explicit calls to ast_websocket_close.
They had been assuming that the websocket session destructor would close the
websocket when it unreffed it but the keepalive process now holds a reference
so the destructor wouldn't actually run without the call to ast_websocket_close
to stop the keepalives.

A few new methods were added to tcptls.c to allow switching an existing
connection from unsecured to TLS.  These were required because the initial
connection and handshake with a proxy is always unsecured but then needs
to be switched to TLS if required for the remote WebSocket server.

There was a bug in sorcery.h where the ast_sorcery_register_uint macro
was referencing _stringify (which doesn't exist) instead of _sorcery_stringify.

Resolves: #1881
Resolves: #1933

UserNote: Forward/outbound proxies can now be specified for outbound websockets.
See the websocket_client.conf.sample file for configuration information.

UserNote: TCP-level or WebSocket PING/PONG keepalives can now be enabled on
outbound websockets.  They can help detect network failures even when a
persistent connection is idle. See the websocket_client.conf.sample
file for configuration information.

DeveloperNote: The addition of the proxy and keepalive configuration parameters
pushed the websocket client parameter count over 32. This necessitated changing
the size of the ast_ws_client_fields enum from a 32 bit bitfield to a 64-bit
bitfield with a corresponding change to the ast_websocket_client structure.

configs/samples/websocket_client.conf.sample

@@ -49,3 +49,111 @@
 ;verify_server_hostname = no   ; Verify that the hostname in the server's certificate
                                ; matches the hostname in the URI configured above.
                                ; Default: yes
+;
+; Outbound/Forward HTTP Proxy
+;
+; If you need to use a proxy to reach the websocket server, you can configure it below.
+; Because WebSockets aren't consistently supported among proxies (specifically passing
+; through UPGRADEs), the CONNECT method is always used to establish a TCP tunnel
+; through the proxy. This is required if a TLS session is to be established with the
+; WebSocket server anyway.  It's important to understand that that negotiation with the
+; proxy is ALWAYS unsecured. Once the proxy establishes the tunnel, the TLS session
+; will be negotiated directly with the remote WebSocket server via the tunnel.
+;
+;proxy_host = proxy:8080       ; The <hostname>:<port> for the forward proxy.
+                               ; Default: none
+;proxy_username = username     ; An authentication username for the proxy.
+                               ; Default: none
+;proxy_password = password     ; An authentication password for the proxy.
+                               ; Default: none
+
+;
+; Detecting active connection failures.
+;
+; Generally speaking, if the websocket server process dies, the host operating
+; system will automatically close the TCP connection.  Asterisk will detect
+; that immediately and attempt to re-establish the connection using the configured
+; connection_timeout, reconnect_interval and reconnect_attempts parameters.
+;
+; However if the host server itself dies or packets are being dropped in the
+; network, Asterisk won't be able to detect this until the next time it tries
+; to send data.  To address this, there are two capabilities available:
+; TCP Keepalives and WebSocket PING/PONG.
+;
+; Which method you choose is entirely up to you and your environment.  Nothing
+; prevents you from activating both but sending a PING frame will reset the
+; TCP keepalive idle timeout so, depending on the intervals configured, the
+; connection may never be idle long enough to trigger sending keepalives.
+;
+; * TCP Keepalives
+;
+; TCP keepalives are implemented by the Linux kernel and once configured, require no
+; resources from Asterisk. The same can be said for the server side as replying to
+; the keepalives is also handled entirely by the operating system without any
+; involvement from application servers.  Once the configured number of replies are
+; missed, the kernel will close the socket which notifies Asterisk and causes an
+; attempt to reconnect using the configured connection_timeout, reconnect_interval
+; and reconnect_attempts parameters.
+;
+; The one downside to TCP keepalives is that they won't traverse a proxy so
+; you'll really only be testing if you can reach the proxy, not the websocket
+; server itself.  However, if the proxy itself can be configured to do TCP
+; keepalives to the websocket server, you could still use TCP keepalives
+; between Asterisk and the proxy to create an effective end-to-end test. 
+;
+;enable_tcp_keepalives = yes  ; Enable TCP keepalives on outgoing WebSockets.
+                              ; Default: no
+
+;tcp_keepalive_time = 20      ; Start sending keepalives when no data has been sent on
+                              ; the connection for this many seconds.
+                              ; Default: 20 seconds
+
+;tcp_keepalive_interval = 20  ; Send keepalives at this interval in seconds.
+                              ; If a reply isn't received by the time the next keepalive
+                              ; is due to be sent, it's considered missed so this option
+                              ; also controls how long it takes to detect a failure.
+                              ; See below.
+                              ; Default: 20 seconds
+
+;tcp_keepalive_probes = 3     ; Close the connection after this many missed replies.
+                              ; The time to detect a failure will be between
+                              ; (probes * interval) and ((probes + 1) * interval) seconds.
+                              ; If the connection closes and reconnect_interval and
+                              ; reconnect_attempts are set, a new connection will be
+                              ; attempted using those parameters.
+                              ; Default: 3 probes
+;
+; * WebSocket PING/PONG frames
+;
+; WebSocket PING/PONG frames are another way to detect failures.  When configured,
+; Asterisk will send a WebSoocket PING frame to the server at the configured interval.
+; All WebSocket server implementation are REQUIRED to respond with a PONG frame.
+; Once the configured number of PONG replies are missed, Asterisk will close the
+; connection and attempt to reconnect using the configured connection_timeout,
+; reconnect_interval and reconnect_attempts parameters.  Unlike TCP heepalives
+; however, Asterisk must create a scheduler thread and manage the process so it's
+; not without overhead, however small.
+;
+; In contrast to TCP keepalives, WebSocket PING/PONGs will traverse a proxy.
+;
+;enable_pingpongs = yes ; Enable Websocket PING/PONGs
+                        ; Default: no
+
+;pingpong_interval = 20 ; Send WebSocket PINGs at this interval in seconds.
+                        ; If a reply isn't received by the time the next PING is due
+                        ; to be sent, it's considered missed so this option also controls
+                        ; how long it takes to detect a failure.  See below.
+                        ; Default: 20 seconds
+
+;pingpong_probes = 3    ; Close the connection after this many missed PONG replies.
+                        ; The time to detect a failure is will be between
+                        ; (probes * interval) and ((probes + 1) * interval)
+                        ; seconds.  If the connection closes and reconnect_interval and
+                        ; reconnect_attempts are set, a new connection will be attempted using
+                        ; those parameters.
+                        ; Default: 3 probes
+;
+; Also be aware...many websocket server implementations can send PING frames
+; _to_ Asterisk which will automatically generate PONG replies.  These won't
+; interfere with Asterisk sending its own PINGs.
+;

include/asterisk/http_websocket.h

@@ -75,6 +75,7 @@ enum ast_websocket_opcode {
 	AST_WEBSOCKET_OPCODE_PONG = 0xA,         /*!< Response to a ping */
 	AST_WEBSOCKET_OPCODE_CLOSE = 0x8,        /*!< Connection is being closed */
 	AST_WEBSOCKET_OPCODE_CONTINUATION = 0x0, /*!< Continuation of a previous frame */
+	AST_WEBSOCKET_OPCODE_UNKNOWN = 0xf,      /*!< Error */
 };
 
 /*! \brief Websocket Status Codes from RFC-6455 */
@@ -460,6 +461,7 @@ enum ast_websocket_result {
 	WS_WRITE_ERROR,
 	WS_CLIENT_START_ERROR,
 	WS_UNAUTHORIZED,
+	WS_TLS_ERROR,
 };
 
 /*!
@@ -480,9 +482,15 @@ enum ast_websocket_result {
  * \param protocols a comma separated string of supported protocols
  * \param tls_cfg secure websocket credentials
  * \param result result code set on client failure
+ *
  * \return a client websocket.
  * \retval NULL if object could not be created or connected
  * \since 13
+ *
+ * \warning The returned websocket must be closed with \ref ast_websocket_close
+ * and its reference count decremented with \ref ast_websocket_unref when
+ * it's no longer needed.
+ *
  */
 AST_OPTIONAL_API(struct ast_websocket *, ast_websocket_client_create,
 		 (const char *uri, const char *protocols,
@@ -517,9 +525,29 @@ struct ast_websocket_client_options {
 	 * Secure websocket credentials
 	 */
 	struct ast_tls_config *tls_cfg;
-	const char *username;          /*!< Auth username */
-	const char *password;          /*!< Auth password */
+	const char *username;          /*!< WebSocket server auth username */
+	const char *password;          /*!< WebSocket server auth password */
+
 	int suppress_connection_msgs;  /*!< Suppress connection log messages */
+	/*!
+	 * Forward proxy
+	 */
+	const char *proxy_host;      /*!< Proxy server host:port */
+	const char *proxy_username;  /*!< Proxy server auth username */
+	const char *proxy_password;  /*!< Proxy server auth password */
+	/*!
+	 * TCP Keepalives
+	 */
+	int tcp_keepalives;                   /*!< Enable TCP keepalives */
+	unsigned int tcp_keepalive_time;      /*!< Start sending when connection has been idle for this many seconds */
+	unsigned int tcp_keepalive_interval;  /*!< Send keepalives at this interval in seconds */
+	unsigned int tcp_keepalive_probes;    /*!< Close connection after this many missed responses */
+	/*!
+	 * WebSocket PING/PONG
+	 */
+	int pingpongs;                   /*!< Enable Websocket PING/PONGs */
+	unsigned int pingpong_interval;  /*!< Send PING messages at this interval in seconds */
+	unsigned int pingpong_probes;    /*!< Close connection after this many missed responses */
 };
 
 /*!
@@ -536,6 +564,10 @@ struct ast_websocket_client_options {
  *
  * \return a client websocket.
  * \retval NULL if object could not be created or connected
+ *
+ * \warning The returned websocket must be closed with \ref ast_websocket_close
+ * and its reference count decremented with \ref ast_websocket_unref when
+ * it's no longer needed.
  */
 AST_OPTIONAL_API(struct ast_websocket *, ast_websocket_client_create_with_options,
 	(struct ast_websocket_client_options *options,

include/asterisk/sorcery.h

@@ -1727,7 +1727,7 @@ extern int ast_sorcery_update_or_create_on_update_miss;
  */
 #define ast_sorcery_register_uint(object, structure, option, field, def_value) \
 	ast_sorcery_object_field_register(sorcery, #object, #option, \
-		_stringify(def_value), OPT_UINT_T, PARSE_IN_RANGE, \
+		_sorcery_stringify(def_value), OPT_UINT_T, PARSE_IN_RANGE, \
 		FLDSET(struct structure, field), 0, UINT_MAX)
 
 /*!

include/asterisk/tcptls.h

@@ -218,6 +218,47 @@ void ast_tcptls_server_start(struct ast_tcptls_session_args *desc);
  */
 void ast_tcptls_server_stop(struct ast_tcptls_session_args *desc);
 
+/*!
+ * \brief Set up an SSL client
+ *
+ * \since 20.21.0
+ * \since 22.11.0
+ * \since 23.5.0
+ *
+ * \note This function only needs to be called if an unsecured tcptls session is
+ * already established and you need to switch it to TLS. This function doesn't
+ * actually start any negotiation.  It just reads any certificates, key files
+ * and options from the config and creates the SSL context.
+ *
+ * \param cfg Configuration for the SSL client
+ *
+ * \retval 1 Success
+ * \retval 0 Failure
+ */
+int ast_ssl_setup_client(struct ast_tls_config *cfg);
+
+/*!
+ * \brief Start TLS negotiation on an existing unsecured connection
+ *
+ * \since 20.21.0
+ * \since 22.11.0
+ * \since 23.5.0
+ *
+ * \note This function only needs to be called if an unsecured tcptls session is
+ * already established and you need to switch it to TLS. This function performs
+ * the handshake and validation.  \ref ast_ssl_setup_client must be called first
+ * to create the SSL context.
+ *
+ * \param tcptls_session The existing unsecured session
+ *
+ * \warning If this function fails, it will automatically dereference tcptls_session
+ * and close the connection so don't attempt to dereference it again.
+ *
+ * \retval tcptls_session on Success
+ * \retval NULL Failure
+ */
+struct ast_tcptls_session_instance *ast_tcptls_start_tls(struct ast_tcptls_session_instance *tcptls_session);
+
 /*!
  * \brief Set up an SSL server
  *

include/asterisk/websocket_client.h

@@ -22,36 +22,56 @@
 #include "asterisk/http_websocket.h"
 #include "asterisk/sorcery.h"
 
+/*
+ * Using 1ULL is important as it forces the enum to be 64 bits.
+ */
 enum ast_ws_client_fields {
 	AST_WS_CLIENT_FIELD_NONE =                   0,
-	AST_WS_CLIENT_FIELD_URI =                    (1 << 0),
-	AST_WS_CLIENT_FIELD_PROTOCOLS =              (1 << 1),
-	AST_WS_CLIENT_FIELD_USERNAME =               (1 << 3),
-	AST_WS_CLIENT_FIELD_PASSWORD =               (1 << 4),
-	AST_WS_CLIENT_FIELD_TLS_ENABLED =            (1 << 7),
-	AST_WS_CLIENT_FIELD_CA_LIST_FILE =           (1 << 8),
-	AST_WS_CLIENT_FIELD_CA_LIST_PATH =           (1 << 9),
-	AST_WS_CLIENT_FIELD_CERT_FILE =              (1 << 10),
-	AST_WS_CLIENT_FIELD_PRIV_KEY_FILE =          (1 << 11),
-	AST_WS_CLIENT_FIELD_CONNECTION_TYPE =        (1 << 13),
-	AST_WS_CLIENT_FIELD_RECONNECT_INTERVAL =     (1 << 14),
-	AST_WS_CLIENT_FIELD_RECONNECT_ATTEMPTS =     (1 << 15),
-	AST_WS_CLIENT_FIELD_CONNECTION_TIMEOUT =     (1 << 16),
-	AST_WS_CLIENT_FIELD_VERIFY_SERVER_CERT =     (1 << 17),
-	AST_WS_CLIENT_FIELD_VERIFY_SERVER_HOSTNAME = (1 << 18),
+	AST_WS_CLIENT_FIELD_URI =                    (1ULL << 0),
+	AST_WS_CLIENT_FIELD_PROTOCOLS =              (1ULL << 1),
+	AST_WS_CLIENT_FIELD_USERNAME =               (1ULL << 3),
+	AST_WS_CLIENT_FIELD_PASSWORD =               (1ULL << 4),
+	AST_WS_CLIENT_FIELD_TLS_ENABLED =            (1ULL << 7),
+	AST_WS_CLIENT_FIELD_CA_LIST_FILE =           (1ULL << 8),
+	AST_WS_CLIENT_FIELD_CA_LIST_PATH =           (1ULL << 9),
+	AST_WS_CLIENT_FIELD_CERT_FILE =              (1ULL << 10),
+	AST_WS_CLIENT_FIELD_PRIV_KEY_FILE =          (1ULL << 11),
+	AST_WS_CLIENT_FIELD_CONNECTION_TYPE =        (1ULL << 13),
+	AST_WS_CLIENT_FIELD_RECONNECT_INTERVAL =     (1ULL << 14),
+	AST_WS_CLIENT_FIELD_RECONNECT_ATTEMPTS =     (1ULL << 15),
+	AST_WS_CLIENT_FIELD_CONNECTION_TIMEOUT =     (1ULL << 16),
+	AST_WS_CLIENT_FIELD_VERIFY_SERVER_CERT =     (1ULL << 17),
+	AST_WS_CLIENT_FIELD_VERIFY_SERVER_HOSTNAME = (1ULL << 18),
+	AST_WS_CLIENT_FIELD_PROXY_HOST =             (1ULL << 19),
+	AST_WS_CLIENT_FIELD_PROXY_USERNAME =         (1ULL << 20),
+	AST_WS_CLIENT_FIELD_PROXY_PASSWORD =         (1ULL << 21),
+	AST_WS_CLIENT_FIELD_TCP_KEEPALIVES =         (1ULL << 22),
+	AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_TIME =     (1ULL << 23),
+	AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_INTERVAL = (1ULL << 24),
+	AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_PROBES =   (1ULL << 25),
+	AST_WS_CLIENT_FIELD_PINGPONGS =              (1ULL << 26),
+	AST_WS_CLIENT_FIELD_PINGPONG_INTERVAL =      (1ULL << 27),
+	AST_WS_CLIENT_FIELD_PINGPONG_PROBES =        (1ULL << 28),
 	AST_WS_CLIENT_NEEDS_RECONNECT = AST_WS_CLIENT_FIELD_URI | AST_WS_CLIENT_FIELD_PROTOCOLS
 		| AST_WS_CLIENT_FIELD_CONNECTION_TYPE
 		| AST_WS_CLIENT_FIELD_USERNAME | AST_WS_CLIENT_FIELD_PASSWORD
 		| AST_WS_CLIENT_FIELD_TLS_ENABLED | AST_WS_CLIENT_FIELD_CA_LIST_FILE
 		| AST_WS_CLIENT_FIELD_CA_LIST_PATH | AST_WS_CLIENT_FIELD_CERT_FILE
 		| AST_WS_CLIENT_FIELD_PRIV_KEY_FILE | AST_WS_CLIENT_FIELD_VERIFY_SERVER_CERT
-		| AST_WS_CLIENT_FIELD_VERIFY_SERVER_HOSTNAME,
+		| AST_WS_CLIENT_FIELD_VERIFY_SERVER_HOSTNAME
+		| AST_WS_CLIENT_FIELD_PROXY_HOST | AST_WS_CLIENT_FIELD_PROXY_USERNAME
+		| AST_WS_CLIENT_FIELD_PROXY_PASSWORD
+		| AST_WS_CLIENT_FIELD_TCP_KEEPALIVES
+		| AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_TIME | AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_INTERVAL
+		| AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_PROBES
+		| AST_WS_CLIENT_FIELD_PINGPONGS | AST_WS_CLIENT_FIELD_PINGPONG_INTERVAL
+		| AST_WS_CLIENT_FIELD_PINGPONG_PROBES,
 };
 
 /*
- * The first 23 fields are reserved for the websocket client core.
+ * The first 29 fields are reserved for the websocket client core.
  */
-#define AST_WS_CLIENT_FIELD_USER_START 24
+#define AST_WS_CLIENT_FIELD_USER_START 30
 
 struct ast_websocket_client {
 	SORCERY_OBJECT(details);
@@ -75,6 +95,16 @@ struct ast_websocket_client {
 	int verify_server_cert;              /*!< Verify server certificate */
 	int verify_server_hostname;          /*!< Verify server hostname */
 	AST_STRING_FIELD_EXTENDED(uri_params); /*!< Additional URI parameters */
+	AST_STRING_FIELD_EXTENDED(proxy_host);  /*!< Proxy server URI */
+	AST_STRING_FIELD_EXTENDED(proxy_username);  /*!< Proxy username */
+	AST_STRING_FIELD_EXTENDED(proxy_password);  /*!< Proxy password */
+	int tcp_keepalives;                  /*!< Enable TCP Keepalives */
+	unsigned int tcp_keepalive_time;     /*!< Start sending when connection has been idle for this many seconds */
+	unsigned int tcp_keepalive_interval; /*!< Send keepalives at this interval in seconds */
+	unsigned int tcp_keepalive_probes;   /*!< Close connection after this many missed responses */
+	int pingpongs;                   /*!< Enable WebSocket PING/PONGs */
+	unsigned int pingpong_interval;  /*!< Send WebSocket PINGs at this interval in seconds */
+	unsigned int pingpong_probes;    /*!< Close connection after this many missed PONG responses */
 };
 
 /*!
@@ -134,6 +164,10 @@ void ast_websocket_client_observer_remove(
  *               result of the connection attempt.
  *
  * \return A pointer to the ast_websocket structure on success, or NULL on failure.
+ *
+ * \warning The returned websocket must be closed with \ref ast_websocket_close
+ * and its reference count decremented with \ref ast_websocket_unref when
+ * it's no longer needed.
  */
 struct ast_websocket *ast_websocket_client_connect(struct ast_websocket_client *wc,
 	void *lock_obj, const char *display_name, enum ast_websocket_result *result);

main/tcptls.c

@@ -130,48 +130,13 @@ static void write_openssl_error_to_log(void)
 }
 #endif
 
-/*! \brief
-* creates a FILE * from the fd passed by the accept thread.
-* This operation is potentially expensive (certificate verification),
-* so we do it in the child thread context.
-*
-* \note must decrement ref count before returning NULL on error
-*/
-static void *handle_tcptls_connection(void *data)
+struct ast_tcptls_session_instance *ast_tcptls_start_tls(struct ast_tcptls_session_instance *tcptls_session)
 {
-	struct ast_tcptls_session_instance *tcptls_session = data;
 #ifdef DO_SSL
 	SSL *ssl;
 #endif
 
-	/* TCP/TLS connections are associated with external protocols, and
-	 * should not be allowed to execute 'dangerous' functions. This may
-	 * need to be pushed down into the individual protocol handlers, but
-	 * this seems like a good general policy.
-	 */
-	if (ast_thread_inhibit_escalations()) {
-		ast_log(LOG_ERROR, "Failed to inhibit privilege escalations; killing connection from peer '%s'\n",
-			ast_sockaddr_stringify(&tcptls_session->remote_address));
-		ast_tcptls_close_session_file(tcptls_session);
-		ao2_ref(tcptls_session, -1);
-		return NULL;
-	}
-
-	/*
-	 * TCP/TLS connections are associated with external protocols which can
-	 * be considered to be user interfaces (even for SIP messages), and
-	 * will not handle channel media.  This may need to be pushed down into
-	 * the individual protocol handlers, but this seems like a good start.
-	 */
-	if (ast_thread_user_interface_set(1)) {
-		ast_log(LOG_ERROR, "Failed to set user interface status; killing connection from peer '%s'\n",
-			ast_sockaddr_stringify(&tcptls_session->remote_address));
-		ast_tcptls_close_session_file(tcptls_session);
-		ao2_ref(tcptls_session, -1);
-		return NULL;
-	}
-
-	if (tcptls_session->parent->tls_cfg) {
+	if (tcptls_session->parent->tls_cfg && tcptls_session->parent->tls_cfg->enabled) {
 #ifdef DO_SSL
 		if (ast_iostream_start_tls(&tcptls_session->stream, tcptls_session->parent->tls_cfg->ssl_ctx, tcptls_session->client) < 0) {
 			SSL *ssl = ast_iostream_get_ssl(tcptls_session->stream);
@@ -270,6 +235,51 @@ static void *handle_tcptls_connection(void *data)
 #endif /* DO_SSL */
 	}
 
+	return tcptls_session;
+}
+
+/*! \brief
+* creates a FILE * from the fd passed by the accept thread.
+* This operation is potentially expensive (certificate verification),
+* so we do it in the child thread context.
+*
+* \note must decrement ref count before returning NULL on error
+*/
+static void *handle_tcptls_connection(void *data)
+{
+	struct ast_tcptls_session_instance *tcptls_session = data;
+
+	/* TCP/TLS connections are associated with external protocols, and
+	 * should not be allowed to execute 'dangerous' functions. This may
+	 * need to be pushed down into the individual protocol handlers, but
+	 * this seems like a good general policy.
+	 */
+	if (ast_thread_inhibit_escalations()) {
+		ast_log(LOG_ERROR, "Failed to inhibit privilege escalations; killing connection from peer '%s'\n",
+			ast_sockaddr_stringify(&tcptls_session->remote_address));
+		ast_tcptls_close_session_file(tcptls_session);
+		ao2_ref(tcptls_session, -1);
+		return NULL;
+	}
+
+	/*
+	 * TCP/TLS connections are associated with external protocols which can
+	 * be considered to be user interfaces (even for SIP messages), and
+	 * will not handle channel media.  This may need to be pushed down into
+	 * the individual protocol handlers, but this seems like a good start.
+	 */
+	if (ast_thread_user_interface_set(1)) {
+		ast_log(LOG_ERROR, "Failed to set user interface status; killing connection from peer '%s'\n",
+			ast_sockaddr_stringify(&tcptls_session->remote_address));
+		ast_tcptls_close_session_file(tcptls_session);
+		ao2_ref(tcptls_session, -1);
+		return NULL;
+	}
+
+	if (ast_tcptls_start_tls(tcptls_session) == NULL) {
+		return NULL;
+	}
+
 	if (tcptls_session->parent->worker_fn) {
 		return tcptls_session->parent->worker_fn(tcptls_session);
 	} else {
@@ -579,6 +589,11 @@ int ast_ssl_setup(struct ast_tls_config *cfg)
 	return __ssl_setup(cfg, 0, 0);
 }
 
+int ast_ssl_setup_client(struct ast_tls_config *cfg)
+{
+	return __ssl_setup(cfg, 1, 1);
+}
+
 void ast_ssl_teardown(struct ast_tls_config *cfg)
 {
 #ifdef DO_SSL

res/ari/ari_websockets.c

@@ -733,7 +733,7 @@ static int session_update(struct ari_ws_session *ari_ws_session,
 			general->write_timeout);
 	}
 
-	ao2_ref(ast_ws_session, +1);
+	ast_websocket_ref(ast_ws_session);
 	ari_ws_session->ast_ws_session = ast_ws_session;
 	ao2_lock(ari_ws_session);
 	for (i = 0; i < AST_VECTOR_SIZE(&ari_ws_session->message_queue); i++) {
@@ -853,6 +853,7 @@ static void websocket_established_cb(struct ast_websocket *ast_ws_session,
 			upgrade_headers, ari_ws_session->app_name, msg);
 		ast_json_unref(msg);
 	}
+	ast_websocket_close(ast_ws_session, AST_WEBSOCKET_STATUS_GOING_AWAY);
 	ari_ws_session->connected = 0;
 
 	SCOPE_EXIT("%s: Websocket closed\n", remote_addr);
@@ -1003,6 +1004,8 @@ static void *outbound_session_handler_thread(void *obj)
 		 * We only want to send "ApplicationRegistered" events in the
 		 * case of a reconnect.  The initial connection will have already sent
 		 * the events when outbound_register_apps() was called.
+		 *
+		 * Note: session_update() bumps astws.
 		 */
 		session_update(session, astws, !already_sent_registers);
 		already_sent_registers = 0;
@@ -1022,6 +1025,8 @@ static void *outbound_session_handler_thread(void *obj)
 			session->thread = 0;
 			session->connected = 0;
 			ast_websocket_close(astws, 1000);
+			/* Clean up the reference held by session_update() */
+			ast_websocket_unref(astws);
 			session->ast_ws_session = NULL;
 			break;
 		}
@@ -1042,6 +1047,8 @@ static void *outbound_session_handler_thread(void *obj)
 		}
 
 		session->connected = 0;
+		ast_websocket_close(session->ast_ws_session, AST_WEBSOCKET_STATUS_GOING_AWAY);
+		/* Clean up the reference held by session_update() */
 		ast_websocket_unref(session->ast_ws_session);
 		session->ast_ws_session = NULL;
 		if (session->closing) {

res/ari/internal.h

@@ -96,13 +96,17 @@ struct ari_conf_user {
 	struct ast_acl_list *acl;
 };
 
+/*
+ * Using 1ULL is important as it forces the enum to be 64 bits to match
+ * the size of enum ast_ws_client_fields.
+ */
 enum ari_conf_owc_fields {
 	ARI_OWC_FIELD_NONE =                    0,
-	ARI_OWC_FIELD_WEBSOCKET_CONNECTION_ID = (1 << AST_WS_CLIENT_FIELD_USER_START),
-	ARI_OWC_FIELD_APPS =                    (1 << (AST_WS_CLIENT_FIELD_USER_START + 1)),
-	ARI_OWC_FIELD_LOCAL_ARI_USER =          (1 << (AST_WS_CLIENT_FIELD_USER_START + 2)),
-	ARI_OWC_FIELD_LOCAL_ARI_PASSWORD =      (1 << (AST_WS_CLIENT_FIELD_USER_START + 3)),
-	ARI_OWC_FIELD_SUBSCRIBE_ALL =           (1 << (AST_WS_CLIENT_FIELD_USER_START + 4)),
+	ARI_OWC_FIELD_WEBSOCKET_CONNECTION_ID = (1ULL << AST_WS_CLIENT_FIELD_USER_START),
+	ARI_OWC_FIELD_APPS =                    (1ULL << (AST_WS_CLIENT_FIELD_USER_START + 1)),
+	ARI_OWC_FIELD_LOCAL_ARI_USER =          (1ULL << (AST_WS_CLIENT_FIELD_USER_START + 2)),
+	ARI_OWC_FIELD_LOCAL_ARI_PASSWORD =      (1ULL << (AST_WS_CLIENT_FIELD_USER_START + 3)),
+	ARI_OWC_FIELD_SUBSCRIBE_ALL =           (1ULL << (AST_WS_CLIENT_FIELD_USER_START + 4)),
 	ARI_OWC_NEEDS_RECONNECT = AST_WS_CLIENT_NEEDS_RECONNECT
 	| ARI_OWC_FIELD_WEBSOCKET_CONNECTION_ID | ARI_OWC_FIELD_LOCAL_ARI_USER
 	| ARI_OWC_FIELD_LOCAL_ARI_PASSWORD,

res/res_aeap/transport_websocket.c

@@ -60,8 +60,7 @@ static int websocket_disconnect(struct aeap_transport *self)
 	struct aeap_transport_websocket *transport = (struct aeap_transport_websocket *)self;
 
 	if (transport->ws) {
-		ast_websocket_unref(transport->ws);
-		transport->ws = NULL;
+		ast_websocket_close(transport->ws, AST_WEBSOCKET_STATUS_NORMAL);
 	}
 
 	return 0;
@@ -69,11 +68,12 @@ static int websocket_disconnect(struct aeap_transport *self)
 
 static void websocket_destroy(struct aeap_transport *self)
 {
-	/*
-	 * Disconnect takes care of cleaning up the websocket. Note, disconnect
-	 * was called by the base/dispatch interface prior to calling this
-	 * function so nothing to do here.
-	 */
+	struct aeap_transport_websocket *transport = (struct aeap_transport_websocket *)self;
+
+	if (transport->ws) {
+		ast_websocket_unref(transport->ws);
+		transport->ws = NULL;
+	}
 }
 
 static intmax_t websocket_read(struct aeap_transport *self, void *buf, intmax_t size,

res/res_websocket_client.c

@@ -221,6 +221,128 @@ verify_server_hostname = no
 					</since>
 					<synopsis>If set to true, verify that the server's hostname matches the common name in it's certificate. (optional)</synopsis>
 				</configOption>
+				<configOption name="proxy_host">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Proxy host including port for outbound proxy if required. (optional)</synopsis>
+					<description>
+						<para>
+							If an outbound proxy is required to reach the websocket server,
+							specify a host in the form <literal>&lt;host&gt;:&lt;port7gt;</literal>.
+							Currently only http (non-TLS) proxies are supported although the tunnelled
+							connection to the websocket server can have TLS enabled.
+						</para>
+					</description>
+				</configOption>
+				<configOption name="proxy_username">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Proxy authentication username if required. (optional)</synopsis>
+				</configOption>
+				<configOption name="proxy_password">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Proxy authentication password if required. (optional)</synopsis>
+				</configOption>
+				<configOption name="enable_tcp_keepalives">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Enable TCP Keepalives. (optional)</synopsis>
+				</configOption>
+				<configOption name="tcp_keepalive_time">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Start sending keepalives when no data has been sent for this many seconds. (optional)</synopsis>
+				</configOption>
+				<configOption name="tcp_keepalive_interval">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Send keepalives at this interval in seconds. (optional)</synopsis>
+					<description>
+						<para>
+							If a reply isn't received by the time the next keepalive is due
+							to be sent, it's considered missed so this option also controls
+							how long it takes to detect a failure.
+						</para>
+					</description>
+				</configOption>
+				<configOption name="tcp_keepalive_probes">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Close the connection after this many missed replies. (optional)</synopsis>
+					<description>
+						<para>
+							If a reply isn't received by the time the next keepalive is due
+							to be sent, it's considered missed.  The time to detect a failure
+							is therefore between (probes * interval) and
+							((probes + 1) * interval) seconds. If the connection closes
+							and reconnect_interval reconnect_attempts are set, a new connection
+							will be attempted using those parameters.
+						</para>
+					</description>
+				</configOption>
+				<configOption name="enable_pingpongs">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Enable WebSocket PING/PONGs.. (optional)</synopsis>
+				</configOption>
+				<configOption name="pingpong_interval">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Send WebSocket PINGs at this interval in seconds. (optional)</synopsis>
+					<description>
+						<para>
+							If a reply isn't received by the time the next PING is due
+							to be sent, it's considered missed so this option also controls
+							how long it takes to detect a failure.
+						</para>
+					</description>
+				</configOption>
+				<configOption name="pingpong_probes">
+					<since>
+						<version>20.21.0</version>
+						<version>22.11.0</version>
+						<version>23.5.0</version>
+					</since>
+					<synopsis>Close the connection after this many missed PONG replies. (optional)</synopsis>
+					<description>
+						<para>
+							If a reply isn't received by the time the next PING is due
+							to be sent, it's considered missepingd.  The time to detect a failure
+							is therefore between (probes * interval) and
+							((probes + 1) * interval) seconds. If the connection closes
+							and reconnect_interval reconnect_attempts are set, a new connection
+							will be attempted using those parameters.
+						</para>
+					</description>
+				</configOption>
 			</configObject>
 		</configFile>
 	</configInfo>
@@ -276,6 +398,16 @@ struct ast_websocket *ast_websocket_client_connect(struct ast_websocket_client *
 			.password = wc->password,
 			.timeout = wc->connect_timeout,
 			.suppress_connection_msgs = 1,
+			.proxy_host = wc->proxy_host,
+			.proxy_username = wc->proxy_username,
+			.proxy_password = wc->proxy_password,
+			.tcp_keepalives = wc->tcp_keepalives,
+			.tcp_keepalive_time = wc->tcp_keepalive_time,
+			.tcp_keepalive_interval = wc->tcp_keepalive_interval,
+			.tcp_keepalive_probes = wc->tcp_keepalive_probes,
+			.pingpongs = wc->pingpongs,
+			.pingpong_interval = wc->pingpong_interval,
+			.pingpong_probes = wc->pingpong_probes,
 			.tls_cfg = NULL,
 		};
 
@@ -383,6 +515,21 @@ static void *wc_alloc(const char *id)
 		return NULL;
 	}
 
+	if (ast_string_field_init_extended(wc, proxy_host) != 0) {
+		ao2_cleanup(wc);
+		return NULL;
+	}
+
+	if (ast_string_field_init_extended(wc, proxy_username) != 0) {
+		ao2_cleanup(wc);
+		return NULL;
+	}
+
+	if (ast_string_field_init_extended(wc, proxy_password) != 0) {
+		ao2_cleanup(wc);
+		return NULL;
+	}
+
 	ast_debug(2, "%s: Allocated websocket client config\n", id);
 	return wc;
 }
@@ -438,6 +585,29 @@ static int wc_apply(const struct ast_sorcery *sorcery, void *obj)
 		res = -1;
 	}
 
+	if (!ast_strlen_zero(wc->proxy_host)) {
+		char *host = NULL;
+		char *port = NULL;
+		char *s = ast_strdupa(wc->proxy_host);
+		if (!ast_sockaddr_split_hostport(s, &host, &port, PARSE_PORT_REQUIRE)) {
+			ast_log(LOG_WARNING, "%s: proxy_host '%s' is missing a port\n", id, wc->proxy_host);
+			res = -1;
+		}
+	}
+
+	if (wc->tcp_keepalives) {
+		if (!wc->tcp_keepalive_time || !wc->tcp_keepalive_interval || !wc->tcp_keepalive_probes) {
+			ast_log(LOG_WARNING, "%s: tcp_keepalive_time, tcp_keepalive_interval and tcp_keepalive_probes must all be non-zero\n", id);
+			res = -1;
+		}
+	}
+
+	if (wc->pingpongs) {
+		if (!wc->pingpong_interval || !wc->pingpong_probes) {
+			ast_log(LOG_WARNING, "%s: pingpong_interval and pingpong_probes must be non-zero\n", id);
+			res = -1;
+		}
+	}
 	if (res != 0) {
 		ast_log(LOG_WARNING, "%s: Websocket client configuration failed\n", id);
 	} else {
@@ -526,6 +696,26 @@ enum ast_ws_client_fields ast_websocket_client_get_field_diff(
 			changed |= AST_WS_CLIENT_FIELD_VERIFY_SERVER_CERT;
 		} else if (ast_strings_equal(v->name, "verify_server_hostname")) {
 			changed |= AST_WS_CLIENT_FIELD_VERIFY_SERVER_HOSTNAME;
+		} else if (ast_strings_equal(v->name, "proxy_host")) {
+			changed |= AST_WS_CLIENT_FIELD_PROXY_HOST;
+		} else if (ast_strings_equal(v->name, "proxy_username")) {
+			changed |= AST_WS_CLIENT_FIELD_PROXY_USERNAME;
+		} else if (ast_strings_equal(v->name, "proxy_password")) {
+			changed |= AST_WS_CLIENT_FIELD_PROXY_PASSWORD;
+		} else if (ast_strings_equal(v->name, "enable_tcp_keepalives")) {
+			changed |= AST_WS_CLIENT_FIELD_TCP_KEEPALIVES;
+		} else if (ast_strings_equal(v->name, "tcp_keepalive_time")) {
+			changed |= AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_TIME;
+		} else if (ast_strings_equal(v->name, "tcp_keepalive_interval")) {
+			changed |= AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_INTERVAL;
+		} else if (ast_strings_equal(v->name, "tcp_keepalive_probes")) {
+			changed |= AST_WS_CLIENT_FIELD_TCP_KEEPALIVE_PROBES;
+		} else if (ast_strings_equal(v->name, "enable_pingpongs")) {
+			changed |= AST_WS_CLIENT_FIELD_PINGPONGS;
+		} else if (ast_strings_equal(v->name, "pingpong_interval")) {
+			changed |= AST_WS_CLIENT_FIELD_PINGPONG_INTERVAL;
+		} else if (ast_strings_equal(v->name, "pingpong_probes")) {
+			changed |= AST_WS_CLIENT_FIELD_PINGPONG_PROBES;
 		} else {
 			ast_debug(2, "%s: Unknown change %s\n", new_id, v->name);
 		}
@@ -599,6 +789,16 @@ static int load_module(void)
 	ast_sorcery_register_int(websocket_client, ast_websocket_client, connection_timeout, connect_timeout, 500);
 	ast_sorcery_register_int(websocket_client, ast_websocket_client, reconnect_attempts, reconnect_attempts, 4);
 	ast_sorcery_register_int(websocket_client, ast_websocket_client, reconnect_interval, reconnect_interval, 500);
+	ast_sorcery_register_sf(websocket_client, ast_websocket_client, proxy_host, proxy_host, "");
+	ast_sorcery_register_sf(websocket_client, ast_websocket_client, proxy_username, proxy_username, "");
+	ast_sorcery_register_sf(websocket_client, ast_websocket_client, proxy_password, proxy_password, "");
+	ast_sorcery_register_bool(websocket_client, ast_websocket_client, enable_tcp_keepalives, tcp_keepalives, "no");
+	ast_sorcery_register_uint(websocket_client, ast_websocket_client, tcp_keepalive_time, tcp_keepalive_time, 20);
+	ast_sorcery_register_uint(websocket_client, ast_websocket_client, tcp_keepalive_interval, tcp_keepalive_interval, 20);
+	ast_sorcery_register_uint(websocket_client, ast_websocket_client, tcp_keepalive_probes, tcp_keepalive_probes, 3);
+	ast_sorcery_register_bool(websocket_client, ast_websocket_client, enable_pingpongs, pingpongs, "no");
+	ast_sorcery_register_uint(websocket_client, ast_websocket_client, pingpong_interval, pingpong_interval, 20);
+	ast_sorcery_register_uint(websocket_client, ast_websocket_client, pingpong_probes, pingpong_probes, 3);
 
 	ast_sorcery_load(sorcery);