OpenSSH/Cookbook/Multiplexing

< OpenSSH < Cookbook

Multiplexing is the ability to send more than one signal over a single line or connection. With multiplexing, OpenSSH can re-use an existing TCP connection for multiple concurrent SSH sessions rather than creating a new one each time.

Advantages of Multiplexing

An advantage with SSH multiplexing is that the overhead of creating new TCP connections is eliminated. The overall number of connections that a machine may accept is a finite resource and the limit is more noticeable on some machines than on others, and varies greatly depending on both load and usage. There is also significant delay when opening a new connection. Activities that repeatedly open new connections can be significantly sped up using multiplexing.

The difference multiplexing and standalone sessions can be seen by comparing the tables below. Both are selected output from netstat -nt slightly edited for clarity. We see in Table 1, "SSH Connections, Separate", that without multiplexing each new login creates a new TCP connection, one per login session. Following that we see in the other table, Table 2, "SSH Connections, Multiplexed", that when multiplexing is used, new logins are channeled over the already established TCP connection.

#              Local Address       Foreign Address         State

# one connection
tcp    0   0   192.168.x.y:45050   192.168.x.z:22       ESTABLISHED

# two separate connections
tcp    0   0   192.168.x.y:45050   192.168.x.z:22       ESTABLISHED
tcp    0   0   192.168.x.y:45051   192.168.x.z:22       ESTABLISHED

# three separate connections
tcp    0   0   192.168.x.y:45050   192.168.x.z:22       ESTABLISHED
tcp    0   0   192.168.x.y:45051   192.168.x.z:22       ESTABLISHED
tcp    0   0   192.168.x.y:45052   192.168.x.z:22       ESTABLISHED

Table 1: SSH Connections, Separate

Both tables show TCP/IP connections associated with SSH sessions. The table above shows a new TCP/IP connection for each new SSH connection. The table below shows a single TCP/IP connection despite multiple active SSH sessions.

#              Local Address       Foreign Address         State

# one connection
tcp    0   0   192.168.x.y:58913   192.168.x.z:22       ESTABLISHED

# two multiplexed connections
tcp    0   0   192.168.x.y:58913   192.168.x.z:22       ESTABLISHED

# three multiplexed connections
tcp    0   0   192.168.x.y:58913   192.168.x.z:22       ESTABLISHED

Table 2: SSH Connections, Multiplexed

As we can see, with multiplexing only a single TCP connection is set up and used regardless of whether or not there are multiple SSH sessions carried over it.

Or we can compare the time it takes to run true(1) on a slow remote server, using time(1). The two commands would be something like time ssh server.example.org true versus time ssh -S ./path/to/somesocket server.example.org true and use keys with an agent for each of them. First without multiplexing, we see the normal connection time:

real	0m0.658s
user	0m0.016s
sys	0m0.008s

Then we do the same, but use a multiplexed connection to see a faster result:

real	0m0.029s
user	0m0.004s
sys	0m0.004s

The difference is quite large and will definitely add up for any activity where connections are made repeatedly. The speed gain for multiplexed connections doesn't come with the master connection, that is normal speed, but with the second and subsequent multiplexed connections. The second and later connections will reuse the established TCP connection over and over and not need to create a new TCP connection for each new SSH connection. So the overhead for a new SSH connection remains, but the overhead of a new TCP connection is avoided.

Setting Up Multiplexing

OpenSSH does multiplexing using the ControlMaster, ControlPath and ControlPersist configuration directives and get defined in ssh_config(5) (which usually defaults to ~/.ssh/config). All three are described in the manual page for ssh_config(5). See the "TOKENS" section there for the list of tokens available for use in the ControlPath. Any valid tokens used are expanded at run time.

ControlMaster determines whether ssh(1) will listen for control connections and what to do about them. ControlPath is the location for the control socket used by the multiplexed sessions. These can be set globally or locally in ssh_config(5) or specified at run time. Control sockets are removed automatically when the master connection is ended. ControlPersist can be used in conjunction with ControlMaster. If set to 'yes', it will leave the master connection open in the background indefinitely to accept new connections until either killed explicitly or closed with -O. Here is a sample excerpt from ssh_config(5) applicable for starting a multiplexed session to machine1.example.org via the shortcut machine1.

Host machine1
        HostName machine1.example.org
        ControlPath ~/.ssh/controlmasters/%r@%h:%p
        ControlMaster auto
        ControlPersist 10m

With that configuration, the first connection to machine1 will create a control socket in the directory ~/.ssh/controlmasters/ Then any subsequent connections, up to 10 by default as set by MaxSessions, will re-use that control path automatically as multiplexed sessions. Confirmation of each new connection can be required if ControlMaster set to 'autoask' instead of 'auto'.

The option -O can be used to manage the connection using the same shortcut configuration.

$ ssh -O check machine1
$ ssh -O stop machine1

First, the status of the connection is checked. Then the master connection is told not to accept further multiplexing requests. To cancel all existing connections, including the master connection, use 'exit' instead of 'stop'.

Manually Establishing Multiplexed Connections

Multiplexed sessions need a control master to connect to. The run time parameters -M and -S also correspond to ControlMaster and ControlPath, respectively. So first an initial master connection is established using -M when accompanied by the path to the control socket using -S.

$ ssh -M -S /home/fred/.ssh/controlmasters/fred@server.example.org:22 server.example.org

Then subsequent multiplexed connections are made in other terminals. They use ControlPath or -S to point to the control socket.

$ ssh -S /home/fred/.ssh/controlmasters/fred@server.example.org:22 server.example.org

Note that the control socket is named "fred@server.example.org:22", or %r@%h:%p, to try to make the name unique. The sockets should be given unique names. The combination %r, %h and %p stand for the remote user name, the remote host and the remote host's port.

Multiple -M options place ssh(1) into master mode with confirmation required before slave connections are accepted. This is the same as ControlMaster=ask. Both require X to ask for the confirmation.

Here is the master connection for the host server.example.org as set to ask for confirmation of new multiplexed sessions:

$ ssh -MM -S ~/.ssh/controlmasters/fred@server.example.org:22 server.example.org

And here is a subsequent multiplexed connection:

$ ssh -S ~/.ssh/controlmasters/fred@server.example.org:22  server.example.org

The status of the control master connection can be queried using -O check, which will tell if the it is running or not.

$ ssh -O check -S ~/.ssh/controlmasters/fred@server.example.org:22  server.example.org

If the control session has been stopped, the query will return an error about "No such file or directory" even if there are still multiplexed sessions running because the socket is gone.

Alternately, configuration options can be spelled out in full using -o. The way below sets configuration options as run time parameters. First, a control master is started.

$ ssh -o "ControlMaster=yes" -o "ControlPath=/home/fred/.ssh/controlmasters/%r@%h:%p" server.example.org

Then subsequent sessions are connected to the control master via socket at the end of the control path.

$ ssh -o "ControlPath=/home/fred/.ssh/controlmasters/%r@%h:%p"  server.example.org

And of course all that can be put into ssh_config. Starting with 6.7, the combination of %r@%h:%p and variations on it can be replaced with %C which by itself generates a hash from the concatenation of %l%h%p%r. The advantage there is that the hash can be shorter than its combined elements while still uniquely identifying the connection and it obfuscates the connection information which would have been otherwise displayed in the socket's name.

Ending Multiplexed Connections

One way to end multiplexed sessions is to exit all related ssh sessions, including the control master. If the control master has been placed in the background using ControlPersist, then it will be necessary to stop it with -O and either stop or exit. That also requires knowing the full path to and filename of the control socket as used in the creation of the master session, if it has not been defined in a shortcut in ssh_config.

$ ssh -O stop server1
$ ssh -O stop -S ~/.ssh/controlmasters/fred@server1.example.org:22 server1.example.org

The multiplex command -O stop will gracefully shutdown multiplexing. After issuing the command, the control socket is removed and no new multiplexed sessions are accepted for that master. Existing connections are allowed to continue and the master connection will persist until the last multiplexed connection closes.

The multiplex command -O exit removes the control socket and immediately terminates all existing connections.

The directive ControlPersist can also be set to timeout after a set period of disuse. The time interval there is written in a time format listed in sshd_config(5) or else defaults to seconds if no units are given. It will cause the master connection to automatically close if it has no client connections for the specified time.

Host server1
        HostName server1.example.org
        ControlPath ~/.ssh/controlmasters/%r@%h:%p
        ControlMaster yes
        ControlPersist 2h

The above example lets the control master timeout after 2 hours of inactivity. Care should be used. A user that can read and write to a control socket can establish new connections without further authentication.

Multiplexing Options

The values for configuring session multiplexing can be set either in the user-specific ssh_config(5) or the global /etc/ssh/ssh_config or using parameters when running from the shell or a script. ControlMaster can be overridden in the run time arguments to re-use an existing master when ControlMaster is set to 'yes', by setting it explicitly to 'no':

$ ssh -o "ControlMaster=no" server.example.org

ControlMaster accepts five different values: 'no', 'yes', 'ask', 'auto', and 'autoask'.

'no' is the default. New sessions will not try to connect to an established master session, but additional sessions can still multiplex by connecting explicitly to an existing socket.

'yes' creates a new master session each time, unless explicitly overridden. The new master session will listen for connections.

'ask' creates a master each time, unless overridden, and listens for connections. If overridden, ssh-askpass(1) will popup an message in X to ask the master session owner to approve or deny the request. If the request is denied, then the session being created falls back to being a regular, standalone session.

'auto' creates a master session automatically but if there is a master session already available, subsequent sessions are automatically multiplexed.

'autoask' automatically assumes that if a master session exists, that subsequent sessions should be multiplexed, but ask first before adding a session.

Refused connections are logged to the master session.

Host * 
        ControlMaster ask

ControlPath can be a fixed string or include any of several pre-defined variables. %L is for the first component of the local host name and %l for the full local hos t name. %h is the target host name and %n is the original target host name and %p is for the destination port on the remote server. %r is for the remote user name and %u for the user running ssh(1)

Host * 
        ControlMaster ask 
        ControlPath ~/.ssh/controlmasters/%r@%h:%p

ControlPersist accepts 'yes', 'no' or a time interval. If a time interval is given, the default is in seconds. Units can extend the time to minutes, hours, days, weeks or a combination. If 'yes' the master connection stays in the background indefinitely.

Host * 
        ControlMaster ask 
        ControlPath ~/.ssh/controlmasters/%r@%h:%p
        ControlPersist 10m

Port Forwarding After the Fact

It is possible to request port forwarding without having to establish new connections. Here we forward port 8080 on the local host to port 80 on the remote host using -L:

$ ssh -O forward -L 8080:localhost:80 -S ~/.ssh/controlmasters/fred@server.example.org:22  server.example.org

The same can be done for remote forwarding, using -R. The escape sequence ~C is not available for multiplexed sessions, so -O forward is the only way to add port forwarding on the fly.

Port forwarding can be canceled without having to close any sessions using -O cancel.

$ ssh -O cancel -L 8080:localhost:80 -S ~/.ssh/controlmasters/fred@server.example.org:22  server.example.org

The exact same syntax used for forwarding is used for cancelling. However, there is no way currently to look up which ports are being forwarded and how they are forwarded.

Additional Notes About Multiplexing

Never use any publicly accessible directory for the control path sockets. Place those sockets in a directory somewhere else, one to which only your account has access. For example ~/.ssh/socket/ would be a much safer choice and /tmp/ would be a bad choice.

In cases where it is useful to have a standing connection, it is always possible to combine multiplexing with other options, such as -f or -N to have the control master drop to the background upon connection and not load a shell.

In sshd_config(5), the directive MaxSessions specifies the maximum number of open sessions permitted per network connection. This is used when multiplexing ssh sessions over a single TCP connection. Setting MaxSessions to 1 disables multiplexing and setting it to 0 disables login/shell/subsystem sessions completely. The default is 10. The MaxSessions directive can also be set under a Match conditional block.

Errors Preventing Multiplexing

The directory used must be on a file system that allows the creation of sockets. AFS is one example that doesn't, as are some implementations of HFS+. If an attempt is made at creating a socket on a file system that does not allow it, an error something like the following will occur:

$ ssh -M -S /home/fred/.ssh/mux 192.0.2.57
muxserver_listen: link mux listener /home/fred/.ssh/mux.vjfeIFFzHnhgHoOV => /home/fred/.ssh/mux: Operation not permitted

If the file system cannot be reconfigured to allow sockets, then the only other option is to place the control path socket somewhere else on a file system that does support creation of sockets.

Observing Multiplexing

It is possible to make some rough measurements to show the differences between multiplexed connections and one-off connections, as shown in the tables and figures above.

Measuring the Number of TCP Connections

Tables 1 and 2 above use output from netstat(8) and awk(1) to show the number of TCP connections corresponding to the number of SSH connections.

#!/bin/sh
netstat -nt | awk 'NR == 2'

ssh -f server.example.org sleep 60
echo # one connection
netstat -nt | awk '$5 ~ /:22$/'

ssh -f server.example.org sleep 60
echo # two connections
netstat -nt | awk '$5 ~ /:22$/'

ssh -f server.example.org sleep 60
echo # three connections
netstat -nt | awk '$5 ~ /:22$/'

echo Table 1

And

#!/bin/sh
netstat -nt | awk 'NR == 2'

ssh -f -M -S ~/.ssh/demo server.example.org sleep 60
echo # one connection
netstat -nt | awk '$5 ~ /:22$/'

ssh -f -S ~/.ssh/demo server.example.org sleep 60 &
echo # two connections
netstat -nt | awk '$5 ~ /:22$/'

ssh -f -S ~/.ssh/demo server.example.org sleep 60 &
echo # three connections
netstat -nt | awk '$5 ~ /:22$/'

echo Table 2

The sleep(1) period can be increased if more delay is needed. While the connections are active, it is also possible to look on the server using ps(1), such as with ps uwx, to see the processes as a means of verifying that multiple connections are indeed being made.

Measuring Response Time

Measuring the response time requires setting up keys with an agent first so that the agent handles the authentication and eliminates authentication as a source of delay. See the section on keys, if needed. All of the response time tests below will depend on using keys for authentication.

For a one-off connection, just add time(1) to check how long access takes.

time ssh -i ~/.ssh/rsakey server.example.org true

For a multiplexed connection, a master control session must be established. Then subsequent connections will show an increase in speed.

ssh -f -M -S ~/.ssh/demo -i ~/.ssh/rsakey server.example.org 
time ssh  -S ~/.ssh/demo -i ~/.ssh/rsakey server.example.org true
time ssh  -S ~/.ssh/demo -i ~/.ssh/rsakey server.example.org true

These response times are approximate but the difference is nonetheless large enough to be seen between one-off connections and multiplexed connections.

Multiplexing HTTPS and SSH Using sslh

A different kind of multiplexing is when more than one protocol is carried over the same port. sslh does just that with SSL and SSH. It figures out what kind of connection is coming in and forwards it to the appropriate service. Thus it allows a server to receive both HTTPS and SSH over the same port. Note that this does not actually hide SSH. A scan for SSH on the listening port, say with scanssh(1), will show that it is there. What it does is make it possible to have one less port open in the firewall. Here's how sslh(8) can be installed in four steps:

First install your web server and configure it to accept HTTPS. Be sure that it listens for HTTPS on the localhost only. It can also help if it is on a non-standard port, say 2443, instead of 443.

Next, set your SSH server to accept connections on the localhost for port 22. It really could be any port, but port 22 is the standard for SSH.

Next, create an unprivileged user to run sslh(8). The example below has the user 'sslh' for the unprivileged account.

Lastly, install and launch sslh(8) so that it listens on port 443 and forwards HTTPS and SSH to the appropriate ports on the local host. Substitute the external IP number for your machine. The actual name of the executable and path may vary from system to system:

$ /usr/local/sbin/sslh-fork -u sslh -p xx.yy.zz.aa:443 --tls 127.0.0.1:2443 --ssh 127.0.0.1:22

Another option is to use a configuration file with sslh(8) and not pass any parameters at runtime. There should be at least one sample configuration file, basic.cfg or example.cfg, included in the package when it is installed. The finished configuration file should look something like this:

user: "sslh";
listen: ( { host: "xx.yy.zz.aa"; port: "443" } );
on-timeout: "ssl";
protocols:
(
   { name: "ssh"; host: "localhost"; port: "22"; probe: "builtin"; },
   { name: "ssl"; host: "localhost"; port: "2443"; probe: "builtin"; }
);

Mind the quotes, commas, and semicolons.

If an old version of SSH is used in conjunction with now-deprecated TCP Wrappers, as described in hosts_access(5), then the option service: works to provide the name of the service that they need.

   { name: "ssh"; service: "ssh"; host: "localhost"; port: "22"; probe: "builtin"; },

If TCP Wrappers are not used, which is most likely the case, then service: is not needed.

Runtime parameters override any configuration file settings that may be in place. sslh(8) supports the protocols HTTP, SSL, SSH, OpenVPN, tinc, and XMPP out of the box. But actually any protocol that can be identified by regular expression pattern matching can be used. There are two variants of sslh, a forked version (sslh or sslh-fork) and a single-threaded version (sslh-select). See the project web site at http://www.rutschle.net/tech/sslh.shtml for more details.

OpenSSH

Overview • Why Encryption • Protocols • Implementations • Clients • Client Configuration • Server • Patterns • Utilities • Third Party • Logging • Development
Cookbook: Remote Processes • Tunnels • Automated Backup • File Transfer with SFTP • Public Key Authentication • Host-based Authentication • Load Balancing • Multiplexing • Proxies and Jump Hosts

This article is issued from Wikibooks. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.