Reverse SSH support

Owned by Shibshankar Talukder
Last updated: Mar 13, 2024
9 min read

General

Adds capability to OpenSync nodes to accept SSH connections and provide remote Shell access through pre-defined configuration and for a pre-defined duration of time.

By default, access through Secure Shell (SSH) to an OpenSync node is not supported for security reasons. The “RevSSH” feature adds capability to OpenSync nodes to accept SSH connections and provide remote Shell access through pre-defined configuration and for a pre-defined duration of time.

Shell access to an OpenSync node can be established through a “Jump Server” (​What is a Jump Server?) setup and run time configurations (IP address, Port, credentials, certificates etc.) through Northbound API.

Theory of operation

  • After a successful configuration through a controller an OpenSync node would:

    • Instantiate a temporary SSH server instance on the node with temporary credentials allowing local connections only

    • Set up an outbound connection to the Jump Server with reverse SSH tunnel / port forwarding enabled

    • When a user logs in to the Jump Server and connects to pre-configured port, the connection is forwarded over the secure channel back to OpenSync node -- establishing a connection to the temporary SSH server instance that was instantiated on the node

    • This reverse SSH tunnel will be established for a configurable max time duration only to abide best security practices.

    • The OpenSync node will issue audit logs and we encourage you to have them also on RevSSH jump server.

  • OpenSync node can be of gateway or extender type and user can initiate multiple sessions at the same time on a single node

  • At a minimum, the following parameters will need to be configured by the user (corresponding to RevSSH OVSDB schema): server_host, server_port, server_user, server_pub_key, tunnel_remote_bind_port, tunnel_local_port. Other parameters may be left at default or they might need to be set.

  • If OpenSync node fails to SSH to the jump RevSSH server due to authentication failure, it retries the connection every 5 seconds, for max 3 minutes.

  • Status will be reported through RevSSH->tunnel_status

  • A RevSSH session is automatically timed-out and is destroyed after the configured maximum time regardless of activity (default 8h) and/or after a configured idle/inactivity timeout (default 10 minutes)

Northbound API

OVSDB schema

Name

Type

Description

Name

Type

Description

server_host

string

RevSSH server address: IPv4, IPv6 or FQDN.

server_port

integer

[optional]

RevSSH server port.

Default: 22

server_user

string

Specifies the user for the node to log in as on the jump RevSSH server.

server_pubkey

ARRAY of string [4]

1 or more (max 4) public keys of a RevSSH server user(s).

Note: The design allows only up to 1 RevSSH tunnel session at any given time (max 1 RevSSH row allowed).

However, the same RevSSH tunnel can be reused:

  • The same user logs in 2 or more times,

  • and/or: multiple users log in at the same time (providing that all users have their corresponding pubkey configured here). Max 4 different public keys can be configured.

The public keys configured here will be temporarily (just for the duration of a RevSSH session) added to node’s authorized_keys list.

In the SSH public key format - RFC 4253.

tunnel_remote_bind_addr

string

[optional]

Remote bind address.

Optional, By default, if not set (recommended), TCP listening sockets on the server will be bound to the loopback interface only.

Default: 127.0.0.1

tunnel_remote_bind_port

integer

Remote bind port

TCP listening socket on the server bind port.

Care must be taken to not use conflicting ports, i.e. to not use a port number that may already be in use by another service (or another RevSSH session) on the RevSSH service. It is advised to not used well-known ports (0-1023) – those can be bound to only by a privilleged user anyway.

tunnel_local_addr

string

[optional]

Local bind address.

And address to forward the TCP connection to.

IPv4 or IPv6 address.

Temporary SSH server instance will be started bound to this address. TCP connections will be forwarded to this local address on the node.

Default: 127.0.0.1

tunnel_local_port

integer

[optional]

Local bind port.

And local port to forward the TCP connection to.

Temporary SSH server instance will be started bound to this port. TCP connections will be forwarded to this local port on the node.

Care must be taken to not use a conflicting port that may already be open on the node. It is advised to avoid well-known ports (0-1023).

Default: 30022

idle_timeout

integer

[optional]

Idle timeout per user SSH session. [minutes]

If a user SSH session is idle (inactive) for this amount of time, it is automatically closed.

Note: More then 1 concurrent SSH sessions per 1 RevSSH tunnel are possible – from the same or from different users (providing that server_pubkey array has public keys for all users configured).

If there are no more active user SSH sessions, the whole RevSSH session is destroyed and resources cleaned up.

OpenSync default, if not set: 10 minutes

session_max_time

integer

[optional]

Maximum time allowed for the whole RevSSH session (all user SSH sessions) regardless if the session is active or inactive.

In minutes.

If this time is reached, regardless if user session is active or not, the whole RevSSH session is destroyed and resources cleaned up.

Note: Also, session is destroyed, when the row is deleted.

OpenSync default, if not set: 8h (480 minutes)

node_gen_key_type

enum

[optional]

{ "rsa", "ecdsa", "ed25519" }

Optional: Specifies the type of (temporary node) key to create.

Default, if not set: Do not create temporary node key. Use node’s static keypair, whatever its type is (usually rsa).

Note: Depending on the SSH keytool (or how it was built) on the node not all keytypes may be supported.

The temporary node key generation is optional but encouraged as it is arguable more secure (even if public key is left in RevSSH’s server authorized_keys list, it’s not a problem, since it’s valid only temporary for the duration of the RevSSH session).

node_gen_key_bits

integer

[optional]

Specifies the number of bits in the (temporary node) key to create.

Must be set if RevSSH->node_gen_key_type is set.

Allowed values:

  • RSA: minimum size is 1024 bits. Default: 3072 bits

  • ECDSA: 256, 384 or 521 bits

  • Ed25519: fixed size of 256 bits

Default: Do not create temporary node key. Use node’s static keypair, whatever its type and bit length is.

node_pubkey

string

[State, reported by the node – not to be set]

Public key of the node reported here [i.e. this field is NOT a CONFIG item, but State/Status]

This will be either node’s “static” public key, or if node_gen_key_type/node_gen_key_bits are set, it will be public key of the node's temporarily-generated public-private key pair.

RevSSH user should then add this public key to RevSSH server user authorized_keys list.

In the SSH public key format - RFC 4253.

tunnel_status

enum

[State, reported by the node – not to be set]

  • unknown: or unset: Tunnel status unknown. Usually just at the beginning shortly after RevSSH configured.

  • init_error: Error before even trying to connect to RevSSH server. Usually this would mean that anykind of initialization failed, or the temporary SSH server failed to start on the node (maybe a bind error / address already in used or similar).

  • connect_failure: Node failed connecting with error other then authentication failure. Usually this would mean that the RevSSH server was not reachable via the configured FQDN or IP address or port number, is refusing connections or some other network error.

  • preauth_failure: Node established a connection to RevSSH server, but failed in the preauthorization phase: the client and the server could not agree on the key algorithms and/or signatures schemes.

    • See also bellow: Notes on “preauth failure”.

  • connecting: Node is trying to connect to RevSSH server every 5 seconds for max 3 minutes, and is succeeding in TCP connection establishment, but each time failing only with public key authorization failure (server did not authorize the client).

    • RevSSH user is expected to add RevSSH->node_pub_key to RevSSH server's authorized_keys list. When added, the node is expected to successfully connect/SSH to RevSSH server and establish a reverse SSH tunnel with remote port forwarding.

  • auth_failure: Node eventually gave up after several authentication failures (after trying @ref REVSSH_CLIENT_AUTH_RETRY_MAX times every @ref REVSSH_CLIENT_AUTH_RETRY_INTERVAL seconds)

  • remote_fwd_failed: Remote port forwarding failed. This indicates an error on the remote (RevSSH server) side. Usually this would happen because a conflicting (in use) port was used on the RevSSH server side and although client successfully connected, a local TCP socket could not be created.

  • established: node SSH-ed into RevSSH server, reverse SSH tunnel with remote port forwarding established. User can now login to RevSSH server and then SSH to the node through the established reverse SSH tunnel.

  • active: There is at least 1 reverse SSH user session detected – i.e. at least 1 SSH user is currently logged in to the node through the established reverse SSH tunnel. Implies established.

    • Note: Depending on the SSH server/client agent and its configuration (for instance dropbear version or built-time configuration), the “active” status may actually not be possible to be determined. So in that case, the tunnel status would stay in “established" state regardless if any user actually logged in or not.

  • disconnected_idle: The whole RevSSH session automatically disconnected due to idle (inactivity) timeout.

  • disconnected_maxtime: The whole RevSSH session automatically disconnected due to max session time reached.

  • disconnected: Tunnel disconnected for any other reason.

log_last_err_msg

string

Not used

Notes on “preauth_failure”

  • OpenSSH deprecated RSA signatures using the SHA-1 hash algorithm in version 8.8 and such a SSH server would, by default, refuse authentications from (old) clients supporting only ssh-rsa (RSA with SHA-1).

  • The solution is either to make the client use rsa-sha2-256 (RSA with SHA-256) or the server to
    allow ssh-rsa. Note: Since RSA keys are not dependent on the choice of hash function the new public key algorithms reuse the “ssh-rsa” public key format. All aspects of the public key format are kept, including the encoded string “ssh-rsa". This allows existing RSA keys to be used with the new public key algorithms, without requiring re-encoding or affecting already trusted key fingerprints.

  • In the RevSSH server logs you would typically see messages such as: sshd[432238]: Unable to negotiate with 80.222.23.16 port 60738: no matching host key type found. Their offer: ssh-rsa [preauth]

  • The solution for "preauth_failure" in the case where the node’s SSH client is offering only ssh-rsa (RSA with SHA-1) but the RevSSH server is rejecting it could be: On the RevSSH server side, add the following to /etc/ssh/ssh_config:

    • HostKeyAlgorithms +ssh-rsa
      PubkeyAcceptedAlgorithms +ssh-rsa

  • For the reverse part of the tunnel: SSH-ing back through the established tunnel, you may also get errors such as “Unable to negotiate with 127.0.0.1 port 30337: no matching host key type found. Their offer: ssh-rsa" in which case you would need to add -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa to the SSH client command

  • To summarize: The solutions are conceptually 2: either server/client mitigations to allow ssh-rsa as mentioned above, or – better, also from security perspective – to upgrade the old SSH clients/servers. Usually the problem is an old Dropbear version on the OpenSync device, so the best option is to upgrade Dropbear on such devices.

Southbound API

  • OpenSync supports Dropbear, however one can use different SSH server/client libraries of their choice

revssh_t *revssh_new(void); bool revssh_server_params_set(revssh_t *self, const char *host, int port, const char *user); bool revssh_authorized_keys_add(revssh_t *self, const char *pubkey); bool revssh_tunnel_params_set( revssh_t *self, osn_ipany_addr_t *remote_bind_addr, int remote_bind_port, osn_ipany_addr_t *local_addr, int local_port); bool revssh_timeout_set(revssh_t *self, int session_max_time, int idle_timeout); bool revssh_notify_status_callback_set(revssh_t *self, revssh_status_fn_t *status_fn_cb); bool revssh_tmpkeygen_set(revssh_t *self, enum revssh_keytype type, int bits); bool revssh_start(revssh_t *self); bool revssh_node_pubkey_get(revssh_t *self, char *pubkey, size_t len); bool revssh_del(revssh_t *self); void revssh_cleanup_dangling_sessions(void);

 

Requirements

SSH Server, Client and tools (key generation and copy tools etc.) packages (e.g. dropbear)

Limitations

  • Dropbear does not support remote TCP forward parameters addresses to be IPv6 addresses (specified in square brackets): So the following ssh foo@$REVSSH_SERVER -R[::1]:30337:[::1]:40337 -p 2222 does not work. Hence, the OpenSync dropbear backend implementation does not support TCP forward parameters with IPv6 addresses

  • Dropbear versions such as v2017.75 or v2019.78 support ssh-rsa signatures and not rsa-sha2-256. Support for these signature algorithms was added in Dropbear version 2020.79

Example configuration and usage

Example with a minimal number of parameters set: Use RevSSH server, specify user, port and user's public key and tunnel_remote_bind_port: (all other parameters will use default values)

ovsh i RevSSH \ server_host:=revssh-opensync.io \ server_user:=opensync_user \ server_port:=2244 \ server_pubkey:="ssh-rsa BBAAB3NzaC1yc2EAAAADAQABAAABAQDYcuU1XfHNO181R6ohzeKIOGjctBgEVkFXWKWQ/1TL8cNYvmmT+pm9oNId5Kl3/YSgD2XCkObg8rTkdlCwKbabAIdyMc0ZEfwu/cwht3g+cxH7ISAVHVbhcNo6eBogRtp7TFiCE2U15m0KAPsALvj56BkwnTIsfEBSa5kICZwOF5V4ErDsyX2YpG2u3s4S4v/gJ+fDgeWbTz1F4b76n/1MTtDGMgUhti96QhuzdmfndZaOzntCsnR2ATD5mu1NA62YiGFx/ueYodFN4OhEw9AgRzlxOxpiERPAe45lvHlAli/zxCYPb6tgqMSGmkoAakCb4C7yN9Ornf8H5THLPTUL opensync_user@revssh-opensync.io" \ tunnel_remote_bind_port:=30337

Node is trying to connect to RevSSH server:

root@opensync:~# ovsh s RevSSH tunnel_status ---------------------------- tunnel_status | connecting | ---------------------------- root@opensync:~# root@opensync:~# ovsh s RevSSH node_pubkey -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- node_pubkey | ssh-rsa AABBB3NzaC1yc2EAAAADAQABAAABAQDq4mtRBkZjs+fZffx8wOklXRNn6uoi3ySncz9GGmaVAxy9gIZWSqMzPsedu1K3PELQ4og2u/BVGknhZno9R2fIXzp33gNECyK8E/XogTg1XROMUKv9lLAySuKQz9YpP1iWShFyIDJf8sd4E/XA78M/Attm9Mj1mRVZ2s4HkF9CewSs+99uxvml66XYDV7oVQY4/Zhmcc1NzT8CQ21/7g+yRMdMblvxeZN40iUPcnglMto6MIqvjmAkLTWkidaWijha6GxT8wpo1ga3x4wEsAHZXtj15/lazvycaj3+1QyN8DdClWXsltdVOjVWZ6PwwFq5ea/glgh5/abcdkyVzPkf root@opensync | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Now, on RevSSH server add node's public key to authorized_keys list:

Once the node's public key is added to RevSSH server's authorized_keys list, the node successfully connects and establishes the revSSH tunnel:

Now, on the RevSSH server, user SSH-s to 127.0.0.1, port 30337 to SSH back to the OpenSync node through the established reverse SSH tunnel:

Tunnel status transitions to "active":

 

A more comprehensive OVSDB config example using Plume's RevSSH server, setting all the tunnel parameters, custom tunnel_local_port to 40337, custom session_max_time to 120 minutes, idle_timeout to 10 minutes and using temporary generation of node's keypair of type Ed25519 size 256 bits (Ed25519 has fixed size, so no need to specify size):