connector: Fix sftp aio read/write with ProxyJump

Addresses issue #319

The commit description explains:
1. Fix for sftp aio + read
2. Fix for sftp aio + write

1. Fix for sftp aio + read
-------------------------
The reproducer provided in the issue description had a model
as follows (with one jump host):
fd_1---(socket_pair)---fd_2---(connector)----channel(fd_3)-----server

Via debugging, it was noticed that the channel connected directly to
the server stored a lot of unbuffered data (received from the server)
that wasn't being written to fd_2 via the connector API.

(Here on, channel refers to the channel(fd_3) in the diagram connected
directly to the server)

Consider the situation, where after a bit of progress in the transfer,
the server has sent all the requested data (requested via outstanding
requests) and all of that data is stored in channel->stdout_buffer. Say
this data is 10,000 bytes.

At this point, all the client (fd_1) is doing is waiting for all
outstanding requests. (and processing thei responses)

- POLLOUT event callback gets generated indicating that fd_2 is
  available for writing.

- ssh_connector_fd_out_cb() gets called to handle the POLLOUT.

- Assuming connector->in_available was true, 4096 (CHUNKSIZE) bytes
  get read from the channel. (really channel->stdout_buffer) leaving
  10,000 - 4096 = 5904 bytes unread in the channel.

- The read bytes are sent via fd_2 (so that fd_1 can recv them)

- After this, the callback sets connector->in_available to 0 and
  connector->out_wontblock to 0.

- Since out_wontblock has been set to 0 ssh_connector_reset_pollevents()
  (called after the callback returns) will consider POLLOUT events on the
  connector output.

- (Based on assumption before) Since the client (fd_1) is eagerly
  awaiting responses and processing them, the received data gets
  processed quickly and fd_2 is available for sending/writing.

- POLLOUT event gets generated for fd_2 indicating that its available
  for writing/sending to fd_1

- ssh_connector_fd_out_cb() gets called to handle the POLLOUT

- Since connector->in_available is 0 (and
  ssh_connector_channel_data_cb() has not been trigerred in between
  as we have assumed before that all the data has already been received on the
  channel and is stored in the channel->stdout_buffer), ssh_connector_fd_out_cb()
  does nothing besides setting connector->out_wontblock to 1.

- Since out_wontblock has been set to 1 ssh_connector_reset_pollevents()
  (called after the callback returns) will IGNORE POLLOUT events on the
  connector output.

- So, at this point, the channel->buffer contains 5706 bytes and the
  fd_2 is available for writing/sending (out_wontblock is 1), but
  nothing happens and the transfer gets stalled/hanged.

In my opinion, this hanging occurs because connector->in_available was
incorrectly set to 0 despite the channel buffer having 5706 bytes in it.

This commit changes that code to consider the data available to read
on the channel (includes buffered data as well as polled data on
channel's internal fd) and taking that into consideration to set
in_available appropriately. (Instead of unconditionally setting it to 0 as the
current code does) so that the next time POLLOUT gets received on fd_2
the ssh_connector_fd_out_cb() does read from the channel and write to
fd_2 (as the connector->in_available flag would be set).

2. Fix for sftp aio + write
-------------------------------------
On writing tests for sftp aio + proxyjump, it was encountered
that file uploads were also hanging. Though I was not able to
pin point the exact cause for this hanging, the nature of hanging
was observed to be as follows:

- sftp aio write + proxyjump blocks/hangs occasionally (not always)

- It hangs at different points in the test

- hang point 1: Sometimes it hangs after sending the first write request
  (i.e. the second write request call hangs and never returns, at this point
  we are not even waiting for response, just sending data). A lot of pending
  data to write to socket/fd was noticed at this hang point.

- hang point 2: Sometimes it hangs while waiting for the second write request
  response.

- It hangs at ssh_handle_packets_termination (i.e. this is the
  call that never returns), in context to hang point 1, this occurs due to
  trying to flush the channel during sftp_packet_write, and in context to
  hang point 2, this occurs due to trying to read an sftp response packet.

- Not sure why, but more the verbose logging/printing I do, the lesser
  occasionally test hangs (e.g. 1 test in 6-7 test runs), maybe this could
  be a hint for a race condition / thread interaction related bug, but am
  not sure.

Fix: On modifying the connector code to mark out_wontblock
to 0 in case of output channel only when the channel's
remote window is 0, the hanging no longer occured.

Though, as mentioned before, I don't know the exact problem
(i.e. case causing hanging) the fix addresses, but the fix
is logical (if remote window is +ve data can still be written
to channel and hence out_wontblock should not be reset to 0, it should
be set to 1) and fixes the issue hence is added to this commit.

Signed-off-by: Eshan Kelkar <eshankelkar@galorithm.com>
Reviewed-by: Jakub Jelen <jjelen@redhat.com>
Reviewed-by: Andreas Schneider <asn@cryptomilk.org>

include/libssh/libssh.h

@@ -432,6 +432,7 @@ enum ssh_options_e {
     SSH_OPTIONS_ADDRESS_FAMILY,
     SSH_OPTIONS_GSSAPI_KEY_EXCHANGE,
     SSH_OPTIONS_GSSAPI_KEY_EXCHANGE_ALGS,
+    SSH_OPTIONS_NEXT_IDENTITY,
 };
 
 enum {

include/libssh/misc.h

@@ -43,8 +43,9 @@ extern "C" {
 
 /* in misc.c */
 /* gets the user home dir. */
-char *ssh_get_user_home_dir(void);
+char *ssh_get_user_home_dir(ssh_session session);
 char *ssh_get_local_username(void);
+char *ssh_get_local_hostname(void);
 int ssh_file_readaccess_ok(const char *file);
 int ssh_dir_writeable(const char *path);
 

include/libssh/session.h

@@ -246,13 +246,16 @@ struct ssh_session_struct {
     struct {
         struct ssh_list *identity;
         struct ssh_list *identity_non_exp;
+        struct ssh_iterator *identity_it;
         struct ssh_list *certificate;
         struct ssh_list *certificate_non_exp;
         struct ssh_list *proxy_jumps;
         struct ssh_list *proxy_jumps_user_cb;
+        char *proxy_jumps_str;
         char *username;
         char *host;
         char *bindaddr; /* bind the client to an ip addr */
+        char *homedir;
         char *sshdir;
         char *knownhosts;
         char *global_knownhosts;

src/config.c

@@ -493,6 +493,10 @@ ssh_config_parse_proxy_jump(ssh_session session, const char *s, bool do_parsing)
     bool parse_entry = do_parsing;
     bool libssh_proxy_jump = ssh_libssh_proxy_jumps();
 
+    if (do_parsing) {
+        SAFE_FREE(session->opts.proxy_jumps_str);
+        ssh_proxyjumps_free(session->opts.proxy_jumps);
+    }
     /* Special value none disables the proxy */
     cmp = strcasecmp(s, "none");
     if (cmp == 0) {
@@ -509,6 +513,17 @@ ssh_config_parse_proxy_jump(ssh_session session, const char *s, bool do_parsing)
         return SSH_ERROR;
     }
 
+    if (do_parsing) {
+        /* Store the whole string in sesion */
+        SAFE_FREE(session->opts.proxy_jumps_str);
+        session->opts.proxy_jumps_str = strdup(s);
+        if (session->opts.proxy_jumps_str == NULL) {
+            free(c);
+            ssh_set_error_oom(session);
+            return SSH_ERROR;
+        }
+    }
+
     cp = c;
     do {
         endp = strchr(cp, ',');

src/connector.c

@@ -305,6 +305,87 @@ static void ssh_connector_reset_pollevents(ssh_connector connector)
     }
 }
 
+/**
+ * @internal
+ *
+ * @brief Update the connector's flags after a read-write io
+ *        operation
+ *
+ * This should be called after some data is successfully read from
+ * connector's input and written to connector's output.
+ *
+ * @param[in, out] connector Connector for which the io operation occured.
+ *
+ * @warning This does not consider the case when the io indicated failure
+ *
+ * @warning This does not consider the case when the input indicated that
+ *          EOF was encountered.
+ */
+static void ssh_connector_update_flags_after_io(ssh_connector connector)
+{
+    /*
+     * With fds we can afford to mark:
+     * - in_available as 0 after an fd read (even if more pending data can be
+     *   immediately read from the fd)
+     *
+     * - out_wontblock as 0 after an fd write (even if more data can
+     *   be written to the fd without blocking)
+     *
+     * since poll events set on the fd will get raised to indicate
+     * possibility of read/write in case existing situation is apt
+     * (i.e can read/write occur right now) or if situation becomes
+     * apt in future (read data becomes available, write becomes
+     * possible)
+     */
+
+    /*
+     * On the other hand, with channels we need to be more careful
+     * before claiming read/write not possible because channel callbacks
+     * are called in limited scenarios.
+     *
+     * (e.g. connector callback to indicate read data available on input
+     * channel is called only when new data is received on channel. It is
+     * not called when we have some pending data in channel's buffers but
+     * don't receive any new data on the channel)
+     *
+     * Hence, in case of channels, blindly setting flag associated with
+     * read/write input/output to 0 after a read/write may not be a good
+     * idea as the callback that sets it back to 1 again may not be ever
+     * called again.
+     */
+
+    uint32_t window_size;
+
+    /* update in_available based on input source (fd or channel) */
+    if (connector->in_fd != SSH_INVALID_SOCKET) {
+        connector->in_available = 0;
+    } else if (connector->in_channel != NULL) {
+        if (ssh_channel_poll_timeout(connector->in_channel, 0, 0) > 0) {
+            connector->in_available = 1;
+        } else {
+            connector->in_available = 0;
+        }
+    } else {
+        /* connector input is invalid ! */
+        return;
+    }
+
+    /* update out_wontblock based on output source (fd or channel) */
+    if (connector->out_fd != SSH_INVALID_SOCKET) {
+        connector->out_wontblock = 0;
+    } else if (connector->out_channel != NULL) {
+        window_size = ssh_channel_window_size(connector->out_channel);
+        if (window_size > 0) {
+            connector->out_wontblock = 1;
+        } else {
+            connector->out_wontblock = 0;
+        }
+    } else {
+        /* connector output is invalid ! */
+        return;
+    }
+}
+
 /**
  * @internal
  *
@@ -390,8 +471,8 @@ static void ssh_connector_fd_in_cb(ssh_connector connector)
             ssh_set_error(connector->session, SSH_FATAL, "output socket or channel closed");
             return;
         }
-        connector->out_wontblock = 0;
-        connector->in_available = 0;
+
+        ssh_connector_update_flags_after_io(connector);
     } else {
         connector->in_available = 1;
     }
@@ -444,8 +525,8 @@ ssh_connector_fd_out_cb(ssh_connector connector)
                           "Output socket or channel closed");
             return;
         }
-        connector->in_available = 0;
-        connector->out_wontblock = 0;
+
+        ssh_connector_update_flags_after_io(connector);
     } else {
         connector->out_wontblock = 1;
     }
@@ -566,11 +647,7 @@ static int ssh_connector_channel_data_cb(ssh_session session,
             return SSH_ERROR;
         }
 
-        connector->out_wontblock = 0;
-        connector->in_available = 0;
-        if ((unsigned int)w < len) {
-            connector->in_available = 1;
-        }
+        ssh_connector_update_flags_after_io(connector);
         ssh_connector_reset_pollevents(connector);
 
         return w;
@@ -642,8 +719,8 @@ ssh_connector_channel_write_wontblock_cb(ssh_session session,
 
             return 0;
         }
-        connector->in_available = 0;
-        connector->out_wontblock = 0;
+
+        ssh_connector_update_flags_after_io(connector);
     } else {
         connector->out_wontblock = 1;
     }

src/gssapi.c

@@ -198,7 +198,7 @@ int
 ssh_gssapi_handle_userauth(ssh_session session, const char *user,
                            uint32_t n_oid, ssh_string *oids)
 {
-    char hostname[NI_MAXHOST] = {0};
+    char *hostname = NULL;
     OM_uint32 maj_stat, min_stat;
     size_t i;
     gss_OID_set supported; /* oids supported by server */
@@ -210,14 +210,6 @@ ssh_gssapi_handle_userauth(ssh_session session, const char *user,
     int rc;
     char err_msg[SSH_ERRNO_MSG_MAX] = {0};
 
-    rc = gethostname(hostname, 64);
-    if (rc != 0) {
-        SSH_LOG(SSH_LOG_TRACE,
-                "Error getting hostname: %s",
-                ssh_strerror(errno, err_msg, SSH_ERRNO_MSG_MAX));
-        return SSH_ERROR;
-    }
-
     /* Destroy earlier GSSAPI context if any */
     ssh_gssapi_free(session);
     rc = ssh_gssapi_init(session);
@@ -284,7 +276,16 @@ ssh_gssapi_handle_userauth(ssh_session session, const char *user,
         return SSH_OK;
     }
 
+    hostname = ssh_get_local_hostname();
+    if (hostname == NULL) {
+        SSH_LOG(SSH_LOG_TRACE,
+                "Error getting hostname: %s",
+                ssh_strerror(errno, err_msg, SSH_ERRNO_MSG_MAX));
+        return SSH_ERROR;
+    }
+
     rc = ssh_gssapi_import_name(session->gssapi, hostname);
+    SAFE_FREE(hostname);
     if (rc != SSH_OK) {
         ssh_auth_reply_default(session, 0);
         gss_release_oid_set(&min_stat, &both_supported);

src/kex-gss.c

@@ -421,7 +421,7 @@ int ssh_server_gss_kex_process_init(ssh_session session, ssh_buffer packet)
     gss_name_t client_name = GSS_C_NO_NAME;
     OM_uint32 ret_flags = 0;
     gss_buffer_desc mic = GSS_C_EMPTY_BUFFER, msg = GSS_C_EMPTY_BUFFER;
-    char hostname[NI_MAXHOST] = {0};
+    char *hostname = NULL;
     char err_msg[SSH_ERRNO_MSG_MAX] = {0};
 
     rc = ssh_buffer_unpack(packet, "S", &otoken);
@@ -538,8 +538,8 @@ int ssh_server_gss_kex_process_init(ssh_session session, ssh_buffer packet)
         goto error;
     }
 
-    rc = gethostname(hostname, 64);
-    if (rc != 0) {
+    hostname = ssh_get_local_hostname();
+    if (hostname == NULL) {
         SSH_LOG(SSH_LOG_TRACE,
                 "Error getting hostname: %s",
                 ssh_strerror(errno, err_msg, SSH_ERRNO_MSG_MAX));
@@ -547,6 +547,7 @@ int ssh_server_gss_kex_process_init(ssh_session session, ssh_buffer packet)
     }
 
     rc = ssh_gssapi_import_name(session->gssapi, hostname);
+    SAFE_FREE(hostname);
     if (rc != SSH_OK) {
         goto error;
     }

src/legacy.c

@@ -615,10 +615,10 @@ int ssh_publickey_to_file(ssh_session session,
     FILE *fp = NULL;
     char *user = NULL;
     char buffer[1024];
-    char host[256];
+    char *host = NULL;
     unsigned char *pubkey_64 = NULL;
     size_t len;
-    int rc;
+
     if(session==NULL)
         return SSH_ERROR;
     if(file==NULL || pubkey==NULL){
@@ -636,8 +636,8 @@ int ssh_publickey_to_file(ssh_session session,
         return SSH_ERROR;
     }
 
-    rc = gethostname(host, sizeof(host));
-    if (rc < 0) {
+    host = ssh_get_local_hostname();
+    if (host == NULL) {
         SAFE_FREE(user);
         SAFE_FREE(pubkey_64);
         return SSH_ERROR;
@@ -651,6 +651,7 @@ int ssh_publickey_to_file(ssh_session session,
 
     SAFE_FREE(pubkey_64);
     SAFE_FREE(user);
+    SAFE_FREE(host);
 
     SSH_LOG(SSH_LOG_RARE, "Trying to write public key file: %s", file);
     SSH_LOG(SSH_LOG_PACKET, "public key file content: %s", buffer);

src/misc.c

@@ -108,7 +108,7 @@
  */
 
 #ifdef _WIN32
-char *ssh_get_user_home_dir(void)
+static char *ssh_get_user_home_dir_internal(void)
 {
     char tmp[PATH_MAX] = {0};
     char *szPath = NULL;
@@ -298,7 +298,7 @@ int ssh_is_ipaddr(const char *str)
 #define NSS_BUFLEN_PASSWD 4096
 #endif /* NSS_BUFLEN_PASSWD */
 
-char *ssh_get_user_home_dir(void)
+static char *ssh_get_user_home_dir_internal(void)
 {
     char *szPath = NULL;
     struct passwd pwd;
@@ -313,7 +313,6 @@ char *ssh_get_user_home_dir(void)
             return NULL;
         }
         snprintf(buf, sizeof(buf), "%s", szPath);
-
         return strdup(buf);
     }
 
@@ -428,6 +427,29 @@ int ssh_is_ipaddr(const char *str)
 
 #endif /* _WIN32 */
 
+char *ssh_get_user_home_dir(ssh_session session)
+{
+    char *szPath = NULL;
+
+    /* If used previously, reuse cached value */
+    if (session != NULL && session->opts.homedir != NULL) {
+        return strdup(session->opts.homedir);
+    }
+
+    szPath = ssh_get_user_home_dir_internal();
+    if (szPath == NULL) {
+        return NULL;
+    }
+
+    if (session != NULL) {
+        /* cache it:
+         * failure is not fatal -- at worst we will just not cache it */
+        session->opts.homedir = strdup(szPath);
+    }
+
+    return szPath;
+}
+
 char *ssh_lowercase(const char* str)
 {
     char *new = NULL, *p = NULL;
@@ -468,6 +490,38 @@ char *ssh_hostport(const char *host, int port)
     return dest;
 }
 
+static char *
+ssh_get_hexa_internal(const unsigned char *what, size_t len, bool colons)
+{
+    const char h[] = "0123456789abcdef";
+    char *hexa = NULL;
+    size_t i;
+    size_t bytes_per_byte = 2 + (colons ? 1 : 0);
+    size_t hlen = len * bytes_per_byte;
+
+    if (len > (UINT_MAX - 1) / bytes_per_byte) {
+        return NULL;
+    }
+
+    hexa = calloc(hlen + 1, sizeof(char));
+    if (hexa == NULL) {
+        return NULL;
+    }
+
+    for (i = 0; i < len; i++) {
+        hexa[i * bytes_per_byte] = h[(what[i] >> 4) & 0xF];
+        hexa[i * bytes_per_byte + 1] = h[what[i] & 0xF];
+        if (colons) {
+            hexa[i * bytes_per_byte + 2] = ':';
+        }
+    }
+    if (colons) {
+        hexa[hlen - 1] = '\0';
+    }
+
+    return hexa;
+}
+
 /**
  * @brief Convert a buffer into a colon separated hex string.
  * The caller has to free the memory.
@@ -483,28 +537,7 @@ char *ssh_hostport(const char *host, int port)
  */
 char *ssh_get_hexa(const unsigned char *what, size_t len)
 {
-    const char h[] = "0123456789abcdef";
-    char *hexa = NULL;
-    size_t i;
-    size_t hlen = len * 3;
-
-    if (len > (UINT_MAX - 1) / 3) {
-        return NULL;
-    }
-
-    hexa = malloc(hlen + 1);
-    if (hexa == NULL) {
-        return NULL;
-    }
-
-    for (i = 0; i < len; i++) {
-        hexa[i * 3] = h[(what[i] >> 4) & 0xF];
-        hexa[i * 3 + 1] = h[what[i] & 0xF];
-        hexa[i * 3 + 2] = ':';
-    }
-    hexa[hlen - 1] = '\0';
-
-    return hexa;
+    return ssh_get_hexa_internal(what, len, true);
 }
 
 /**
@@ -1189,7 +1222,7 @@ char *ssh_path_expand_tilde(const char *d)
     } else {
         ld = strlen(d);
         p = (char *) d;
-        h = ssh_get_user_home_dir();
+        h = ssh_get_user_home_dir(NULL);
     }
     if (h == NULL) {
         return NULL;
@@ -1211,15 +1244,105 @@ char *ssh_path_expand_tilde(const char *d)
     return r;
 }
 
+char *ssh_get_local_hostname(void)
+{
+    char host[NI_MAXHOST] = {0};
+    int rc;
+
+    rc = gethostname(host, sizeof(host));
+    if (rc != 0) {
+        return NULL;
+    }
+    return strdup(host);
+}
+
+static char *get_connection_hash(ssh_session session)
+{
+    unsigned char conn_hash[SHA_DIGEST_LENGTH];
+    char *local_hostname = NULL;
+    SHACTX ctx = sha1_init();
+    char strport[10] = {0};
+    unsigned int port;
+    int rc;
+
+    if (session == NULL) {
+        return NULL;
+    }
+
+    if (ctx == NULL) {
+        goto err;
+    }
+
+    /* Local hostname %l */
+    local_hostname = ssh_get_local_hostname();
+    if (local_hostname == NULL) {
+        goto err;
+    }
+    rc = sha1_update(ctx, local_hostname, strlen(local_hostname));
+    if (rc != SSH_OK) {
+        goto err;
+    }
+    SAFE_FREE(local_hostname);
+
+    /* Remote hostname %h */
+    rc = sha1_update(ctx, session->opts.host, strlen(session->opts.host));
+    if (rc != SSH_OK) {
+        goto err;
+    }
+
+    /* Remote port %p */
+    ssh_options_get_port(session, &port);
+    snprintf(strport, sizeof(strport), "%d", port);
+    rc = sha1_update(ctx, strport, strlen(strport));
+    if (rc != SSH_OK) {
+        goto err;
+    }
+
+    /* The remote username %r */
+    rc = sha1_update(ctx,
+                     session->opts.username,
+                     strlen(session->opts.username));
+    if (rc != SSH_OK) {
+        goto err;
+    }
+
+    /* ProxyJump */
+    if (session->opts.proxy_jumps_str != NULL) {
+        rc = sha1_update(ctx,
+                         session->opts.proxy_jumps_str,
+                         strlen(session->opts.proxy_jumps_str));
+    }
+    if (rc != SSH_OK) {
+        goto err;
+    }
+
+    /* Frees context */
+    rc = sha1_final(conn_hash, ctx);
+    if (rc != SSH_OK) {
+        goto err;
+    }
+
+    return ssh_get_hexa_internal(conn_hash, SHA_DIGEST_LENGTH, false);
+
+err:
+    free(local_hostname);
+    sha1_ctx_free(ctx);
+    return NULL;
+}
+
 /** @internal
  * @brief expands a string in function of session options
+ *
  * @param[in] s Format string to expand. Known parameters:
- *              %d SSH configuration directory (~/.ssh)
- *              %h target host name
- *              %u local username
- *              %l local hostname
- *              %r remote username
- *              %p remote port
+ *               - %d user home directory (~)
+ *               - %h target host name
+ *               - %u local username
+ *               - %l local hostname
+ *               - %r remote username
+ *               - %p remote port
+ *               - %j proxyjump string
+ *               - %C Hash of %l%h%p%r%j
+ *
  * @returns Expanded string. The caller needs to free the memory using
  *          ssh_string_free_char().
  *
@@ -1227,7 +1350,6 @@ char *ssh_path_expand_tilde(const char *d)
  */
 char *ssh_path_expand_escape(ssh_session session, const char *s)
 {
-    char host[NI_MAXHOST] = {0};
     char *buf = NULL;
     char *r = NULL;
     char *x = NULL;
@@ -1279,11 +1401,9 @@ char *ssh_path_expand_escape(ssh_session session, const char *s)
         case '%':
             goto escape;
         case 'd':
-                if (session->opts.sshdir) {
-                    x = strdup(session->opts.sshdir);
-                } else {
-                    ssh_set_error(session, SSH_FATAL,
-                            "Cannot expand sshdir");
+            x = ssh_get_user_home_dir(session);
+            if (x == NULL) {
+                ssh_set_error(session, SSH_FATAL, "Cannot expand homedir");
                 free(buf);
                 free(r);
                 return NULL;
@@ -1293,16 +1413,13 @@ char *ssh_path_expand_escape(ssh_session session, const char *s)
             x = ssh_get_local_username();
             break;
         case 'l':
-                if (gethostname(host, sizeof(host) == 0)) {
-                    x = strdup(host);
-                }
+            x = ssh_get_local_hostname();
             break;
         case 'h':
             if (session->opts.host) {
                 x = strdup(session->opts.host);
             } else {
-                    ssh_set_error(session, SSH_FATAL,
-                            "Cannot expand host");
+                ssh_set_error(session, SSH_FATAL, "Cannot expand host");
                 free(buf);
                 free(r);
                 return NULL;
@@ -1312,26 +1429,33 @@ char *ssh_path_expand_escape(ssh_session session, const char *s)
             if (session->opts.username) {
                 x = strdup(session->opts.username);
             } else {
-                    ssh_set_error(session, SSH_FATAL,
-                            "Cannot expand username");
+                ssh_set_error(session, SSH_FATAL, "Cannot expand username");
                 free(buf);
                 free(r);
                 return NULL;
             }
             break;
-            case 'p':
-                {
+        case 'p': {
             char tmp[6];
+            unsigned int port;
 
-                  snprintf(tmp, sizeof(tmp), "%hu",
-                           (uint16_t)(session->opts.port > 0 ? session->opts.port
-                                                             : 22));
+            ssh_options_get_port(session, &port);
+            snprintf(tmp, sizeof(tmp), "%u", port);
             x = strdup(tmp);
+            break;
+        }
+        case 'j':
+            if (session->opts.proxy_jumps_str != NULL) {
+                x = strdup(session->opts.proxy_jumps_str);
+            } else {
+                x = strdup("");
             }
             break;
+        case 'C':
+            x = get_connection_hash(session);
+            break;
         default:
-                ssh_set_error(session, SSH_FATAL,
-                        "Wrong escape sequence detected");
+            ssh_set_error(session, SSH_FATAL, "Wrong escape sequence detected");
             free(buf);
             free(r);
             return NULL;
@@ -1346,8 +1470,7 @@ char *ssh_path_expand_escape(ssh_session session, const char *s)
 
         i += strlen(x);
         if (i >= MAX_BUF_SIZE) {
-            ssh_set_error(session, SSH_FATAL,
-                    "String too long");
+            ssh_set_error(session, SSH_FATAL, "String too long");
             free(buf);
             free(x);
             free(r);

src/options.c

@@ -1197,7 +1197,6 @@ int ssh_options_set(ssh_session session, enum ssh_options_e type,
                 ssh_set_error_invalid(session);
                 return -1;
             } else {
-                ssh_proxyjumps_free(session->opts.proxy_jumps);
                 rc = ssh_config_parse_proxy_jump(session, v, true);
                 if (rc != SSH_OK) {
                     return SSH_ERROR;
@@ -1562,7 +1561,16 @@ int ssh_options_get_port(ssh_session session, unsigned int* port_target) {
  *              - SSH_OPTIONS_IDENTITY:
  *                Get the first identity file name (const char *).\n
  *                \n
- *                By default id_rsa, id_ecdsa and id_ed25519 files are used.
+ *                By default `id_rsa`, `id_ecdsa`, `id_ed25519`, `id_ecdsa_sk`
+ *                and `id_ed25519_sk` (when SK support is built in) files are
+ *                used.
+ *
+ *              - SSH_OPTIONS_NEXT_IDENTITY:
+ *                Get the next identity file name (const char *).\n
+ *                \n
+ *                Repeat calls to get all key paths. SSH_EOF is returned when
+ *                the end of list is reached. Another call will start another
+ *                iteration over the same list.
  *
  *              - SSH_OPTIONS_PROXYCOMMAND:
  *                Get the proxycommand necessary to log into the
@@ -1658,6 +1666,30 @@ int ssh_options_get(ssh_session session, enum ssh_options_e type, char** value)
             break;
         }
 
+        case SSH_OPTIONS_NEXT_IDENTITY: {
+            if (session->opts.identity_it != NULL) {
+                /* Move to the next item */
+                session->opts.identity_it = session->opts.identity_it->next;
+                if (session->opts.identity_it == NULL) {
+                    *value = NULL;
+                    return SSH_EOF;
+                }
+            } else {
+                /* Get iterator from opts */
+                struct ssh_iterator *it = NULL;
+                it = ssh_list_get_iterator(session->opts.identity);
+                if (it == NULL) {
+                    it = ssh_list_get_iterator(session->opts.identity_non_exp);
+                }
+                if (it == NULL) {
+                    return SSH_ERROR;
+                }
+                session->opts.identity_it = it;
+            }
+            src = ssh_iterator_value(char *, session->opts.identity_it);
+            break;
+        }
+
         case SSH_OPTIONS_PROXYCOMMAND:
             src = session->opts.ProxyCommand;
             break;
@@ -1963,7 +1995,7 @@ int ssh_options_parse_config(ssh_session session, const char *filename)
 
     /* set default filename */
     if (filename == NULL) {
-        expanded_filename = ssh_path_expand_escape(session, "%d/config");
+        expanded_filename = ssh_path_expand_escape(session, "%d/.ssh/config");
     } else {
         expanded_filename = ssh_path_expand_escape(session, filename);
     }
@@ -2021,7 +2053,7 @@ int ssh_options_apply(ssh_session session)
 
     if ((session->opts.exp_flags & SSH_OPT_EXP_FLAG_KNOWNHOSTS) == 0) {
         if (session->opts.knownhosts == NULL) {
-            tmp = ssh_path_expand_escape(session, "%d/known_hosts");
+            tmp = ssh_path_expand_escape(session, "%d/.ssh/known_hosts");
         } else {
             tmp = ssh_path_expand_escape(session, session->opts.knownhosts);
         }

src/pki.c

@@ -2684,7 +2684,7 @@ int ssh_pki_export_pubkey_file(const ssh_key key,
                                const char *filename)
 {
     char key_buf[MAX_LINE_SIZE];
-    char host[256];
+    char *host = NULL;
     char *b64_key = NULL;
     char *user = NULL;
     FILE *fp = NULL;
@@ -2699,8 +2699,8 @@ int ssh_pki_export_pubkey_file(const ssh_key key,
         return SSH_ERROR;
     }
 
-    rc = gethostname(host, sizeof(host));
-    if (rc < 0) {
+    host = ssh_get_local_hostname();
+    if (host == NULL) {
         free(user);
         return SSH_ERROR;
     }
@@ -2708,6 +2708,7 @@ int ssh_pki_export_pubkey_file(const ssh_key key,
     rc = ssh_pki_export_pubkey_base64(key, &b64_key);
     if (rc < 0) {
         free(user);
+        free(host);
         return SSH_ERROR;
     }
 
@@ -2718,6 +2719,7 @@ int ssh_pki_export_pubkey_file(const ssh_key key,
                   user,
                   host);
     free(user);
+    free(host);
     free(b64_key);
     if (rc < 0) {
         return SSH_ERROR;

src/session.c

@@ -168,7 +168,7 @@ ssh_session ssh_new(void)
     }
 #endif /* WITH_GSSAPI */
 
-    id = strdup("%d/id_ed25519");
+    id = strdup("%d/.ssh/id_ed25519");
     if (id == NULL) {
         goto err;
     }
@@ -179,7 +179,7 @@ ssh_session ssh_new(void)
     }
 
 #ifdef HAVE_ECC
-    id = strdup("%d/id_ecdsa");
+    id = strdup("%d/.ssh/id_ecdsa");
     if (id == NULL) {
         goto err;
     }
@@ -189,7 +189,7 @@ ssh_session ssh_new(void)
     }
 #endif
 
-    id = strdup("%d/id_rsa");
+    id = strdup("%d/.ssh/id_rsa");
     if (id == NULL) {
         goto err;
     }
@@ -200,7 +200,7 @@ ssh_session ssh_new(void)
 
 #ifdef WITH_FIDO2
     /* Add security key identities */
-    id = strdup("%d/id_ed25519_sk");
+    id = strdup("%d/.ssh/id_ed25519_sk");
     if (id == NULL) {
         goto err;
     }
@@ -210,7 +210,7 @@ ssh_session ssh_new(void)
     }
 
 #ifdef HAVE_ECC
-    id = strdup("%d/id_ecdsa_sk");
+    id = strdup("%d/.ssh/id_ecdsa_sk");
     if (id == NULL) {
         goto err;
     }
@@ -384,6 +384,7 @@ void ssh_free(ssh_session session)
     ssh_proxyjumps_free(session->opts.proxy_jumps);
     SSH_LIST_FREE(session->opts.proxy_jumps);
     SSH_LIST_FREE(session->opts.proxy_jumps_user_cb);
+    SAFE_FREE(session->opts.proxy_jumps_str);
 
     while ((b = ssh_list_pop_head(struct ssh_buffer_struct *,
                                   session->out_queue)) != NULL) {
@@ -405,6 +406,7 @@ void ssh_free(ssh_session session)
   SAFE_FREE(session->opts.bindaddr);
   SAFE_FREE(session->opts.username);
   SAFE_FREE(session->opts.host);
+  SAFE_FREE(session->opts.homedir);
   SAFE_FREE(session->opts.sshdir);
   SAFE_FREE(session->opts.knownhosts);
   SAFE_FREE(session->opts.global_knownhosts);

tests/client/torture_auth.c

@@ -376,7 +376,7 @@ torture_auth_autopubkey_protected_auth_function (const char *prompt, char *buf,
     assert_int_equal(echo, 0);
     assert_int_equal(verify, 0);
 
-    expected_id = ssh_path_expand_escape(data->session, "%d/id_rsa_protected");
+    expected_id = ssh_path_expand_escape(data->session, "%d/.ssh/id_rsa_protected");
     assert_true(expected_id != NULL);
 
     rc = ssh_userauth_publickey_auto_get_current_identity(data->session, &id);
@@ -429,7 +429,7 @@ static void torture_auth_autopubkey_protected(void **state) {
 
     /* Try id_rsa_protected first.
      */
-    rc = ssh_options_set(session, SSH_OPTIONS_IDENTITY, "%d/id_rsa_protected");
+    rc = ssh_options_set(session, SSH_OPTIONS_IDENTITY, "%d/.ssh/id_rsa_protected");
     assert_int_equal(rc, SSH_OK);
 
     rc = ssh_connect(session);

tests/client/torture_auth_cert.c

@@ -100,13 +100,10 @@ static int session_setup(void **state)
 static int session_setup_ssh_dir(void **state)
 {
     struct torture_state *s = *state;
-    const char *no_home = "~/.no_ssh";
-    int rc;
 
     session_setup(state);
 
-    rc = ssh_options_set(s->ssh.session, SSH_OPTIONS_SSH_DIR, no_home);
-    assert_ssh_return_code(s->ssh.session, rc);
+    s->ssh.session->opts.homedir = strdup("~/.no_ssh");
 
     return 0;
 }

tests/client/torture_client_config.c

@@ -6,6 +6,7 @@
 #include <errno.h>
 #include "torture.h"
 #include "libssh/session.h"
+#include "libssh/options.h"
 #include "libssh/misc.h"
 
 #define LIBSSH_SSH_CONFIG "libssh_config"
@@ -59,6 +60,23 @@ static int setup_config_files(void **state)
     return 0;
 }
 
+static int setup_session(void **state)
+{
+    struct torture_state *s = *state;
+    int verbosity;
+
+    s->ssh.session = ssh_new();
+    assert_non_null(s->ssh.session);
+
+    verbosity = torture_libssh_verbosity();
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_LOG_VERBOSITY, &verbosity);
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_HOST, TORTURE_SSH_SERVER);
+
+    setenv("NSS_WRAPPER_HOSTNAME", "client.libssh.site", 1);
+
+    return 0;
+}
+
 static int teardown(void **state)
 {
     struct torture_state *s = *state;
@@ -80,6 +98,16 @@ static int teardown(void **state)
     return 0;
 }
 
+static int teardown_session(void **state)
+{
+    struct torture_state *s = *state;
+
+    ssh_disconnect(s->ssh.session);
+    ssh_free(s->ssh.session);
+
+    return 0;
+}
+
 /* This tests makes sure that parsing both system-wide and per-user
  * configuration files retains OpenSSH semantics (the per-user overrides
  * the system-wide values).
@@ -210,10 +238,207 @@ static void torture_client_config_suppress(void **state)
     assert_string_equal(s->ssh.session->opts.username, "bob");
 }
 
+static void torture_client_config_expand(void **state)
+{
+    struct torture_state *s = *state;
+    int ret = 0;
+
+    /* TEST: user home directory */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%d");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    assert_string_equal(s->ssh.session->opts.knownhosts,
+                        BINARYDIR "/tests/home");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: target host name */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%h");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    assert_string_equal(s->ssh.session->opts.knownhosts, TORTURE_SSH_SERVER);
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: local username */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%u");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    assert_string_equal(s->ssh.session->opts.knownhosts, "root");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: local hostname */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%l");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    assert_string_equal(s->ssh.session->opts.knownhosts, "client.libssh.site");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: remote username */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_USER, "alice");
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%r");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    assert_string_equal(s->ssh.session->opts.knownhosts, "alice");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: remote port */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_PORT_STR, "2222");
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%p");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    assert_string_equal(s->ssh.session->opts.knownhosts, "2222");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: empty proxyjump */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%j");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    /* No proxyjump string should not explode */
+    assert_string_equal(s->ssh.session->opts.knownhosts, "");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: proxyjump string present */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%j");
+    ssh_options_set(s->ssh.session,
+                    SSH_OPTIONS_PROXYJUMP,
+                    "user@" TORTURE_SSH_SERVER ":22");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    assert_string_equal(s->ssh.session->opts.knownhosts,
+                        "user@" TORTURE_SSH_SERVER ":22");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: separate list %l-%h-%p-%r-%j with empty ProxyJump */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%l-%h-%p-%r-%j");
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_PROXYJUMP, "none");
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_PORT_STR, "22");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    // Tested by
+    // ret = system(SSH_EXECUTABLE
+    //              " -p 22 -o UserKnownHostsFile=/dev/null"
+    //              " -o KnownHostsCommand='/bin/touch \"/tmp/%l-%h-%p-%r-%j\"'"
+    //             " alice@" TORTURE_SSH_SERVER);
+    // assert_return_code(ret, errno);
+    assert_string_equal(s->ssh.session->opts.knownhosts,
+                        "client.libssh.site-127.0.0.10-22-alice-");
+
+
+    /* TEST: hash of %l%h%p%r%j with empty ProxyJump */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%C");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    // Tested by
+    // ret = system(SSH_EXECUTABLE
+    //              " -p 22 -o UserKnownHostsFile=/dev/null"
+    //              " -o KnownHostsCommand='/bin/touch \"/tmp/%C\"'"
+    //              " alice@" TORTURE_SSH_SERVER);
+    // assert_return_code(ret, errno);
+    assert_string_equal(s->ssh.session->opts.knownhosts,
+                        "133e3957ff9d01fdcf1f6c7f83325a8ce49bf850");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: separate list %l-%h-%p-%r-%j */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%l-%h-%p-%r-%j");
+    ssh_options_set(s->ssh.session,
+                    SSH_OPTIONS_PROXYJUMP,
+                    "user@" TORTURE_SSH_SERVER ":22");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    // Tested by
+    // ret = system(SSH_EXECUTABLE
+    //              " -p 22 -oProxyJump=user@" TORTURE_SSH_SERVER ":22"
+    //              " -o UserKnownHostsFile=/dev/null"
+    //              " -o KnownHostsCommand='/bin/touch \"/tmp/%l-%h-%p-%r-%j\"'"
+    //             " alice@" TORTURE_SSH_SERVER);
+    // assert_return_code(ret, errno);
+    assert_string_equal(s->ssh.session->opts.knownhosts,
+                        "client.libssh.site-127.0.0.10-22-alice-user@"
+                        TORTURE_SSH_SERVER ":22");
+
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+
+    /* TEST: hash of %l%h%p%r%j */
+    ssh_options_set(s->ssh.session, SSH_OPTIONS_KNOWNHOSTS, "%C");
+
+    ret = ssh_options_apply(s->ssh.session);
+    assert_ssh_return_code(s->ssh.session, ret);
+
+    // Tested by
+    // ret = system(SSH_EXECUTABLE
+    //              " -p 22 -oProxyJump=user@" TORTURE_SSH_SERVER ":22"
+    //              " -o UserKnownHostsFile=/dev/null"
+    //              " -o KnownHostsCommand='/bin/touch \"/tmp/%C\"'"
+    //              " alice@" TORTURE_SSH_SERVER);
+    // assert_return_code(ret, errno);
+    assert_string_equal(s->ssh.session->opts.knownhosts,
+                        "adf0b7c4e71a0fee85fd97506507ba8591f3663b");
+
+    /* Reset the flag so we can repeat the test */
+    s->ssh.session->opts.exp_flags &= ~SSH_OPT_EXP_FLAG_KNOWNHOSTS;
+
+}
+
 
 int torture_run_tests(void) {
     int rc;
     struct CMUnitTest tests[] = {
+        /* Keep this first -- following setup is changing user to bob, which we
+         * do not want */
+        cmocka_unit_test_setup_teardown(torture_client_config_expand,
+                                        setup_session,
+                                        teardown_session),
         cmocka_unit_test_setup_teardown(torture_client_config_system,
                                         setup_config_files,
                                         teardown),

tests/client/torture_sftp_aio.c

@@ -11,19 +11,37 @@
 
 #define MAX_XFER_BUF_SIZE 16384
 
+#define DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(TEST_NAME) \
+    {                            \
+        #TEST_NAME,              \
+        TEST_NAME,               \
+        session_setup,           \
+        session_teardown,        \
+        NULL                     \
+    },                           \
+    {                            \
+        #TEST_NAME"_proxyjump",  \
+        TEST_NAME,               \
+        session_proxyjump_setup, \
+        session_teardown,        \
+        NULL                     \
+    }
+
 static int sshd_setup(void **state)
 {
     torture_setup_sshd_server(state, false);
+    torture_setup_sshd_servers(state, false);
     return 0;
 }
 
 static int sshd_teardown(void **state)
 {
+    /* this will take care of the server1 teardown too */
     torture_teardown_sshd_server(state);
     return 0;
 }
 
-static int session_setup(void **state)
+static int session_setup_helper(void **state, bool with_proxyjump)
 {
     struct torture_state *s = *state;
     struct passwd *pwd = NULL;
@@ -35,11 +53,15 @@ static int session_setup(void **state)
     rc = setuid(pwd->pw_uid);
     assert_return_code(rc, errno);
 
+    if (with_proxyjump) {
+        s->ssh.session = torture_ssh_session_proxyjump();
+    } else {
         s->ssh.session = torture_ssh_session(s,
                                              TORTURE_SSH_SERVER,
                                              NULL,
                                              TORTURE_SSH_USER_ALICE,
                                              NULL);
+    }
     assert_non_null(s->ssh.session);
 
     s->ssh.tsftp = torture_sftp_session(s->ssh.session);
@@ -48,6 +70,16 @@ static int session_setup(void **state)
     return 0;
 }
 
+static int session_setup(void **state)
+{
+    return session_setup_helper(state, false);
+}
+
+static int session_proxyjump_setup(void **state)
+{
+    return session_setup_helper(state, true);
+}
+
 static int session_teardown(void **state)
 {
     struct torture_state *s = *state;
@@ -722,37 +754,18 @@ int torture_run_tests(void)
 {
     int rc;
     struct CMUnitTest tests[] = {
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_read_file,
-                                        session_setup,
-                                        session_teardown),
-
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_read_more_than_cap,
-                                        session_setup,
-                                        session_teardown),
-
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_write_file,
-                                        session_setup,
-                                        session_teardown),
-
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_write_more_than_cap,
-                                        session_setup,
-                                        session_teardown),
-
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_read_negative,
-                                        session_setup,
-                                        session_teardown),
-
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_write_negative,
-                                        session_setup,
-                                        session_teardown),
-
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_read_unordered_wait,
-                                        session_setup,
-                                        session_teardown),
-
-        cmocka_unit_test_setup_teardown(torture_sftp_aio_write_unordered_wait,
-                                        session_setup,
-                                        session_teardown),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(torture_sftp_aio_read_file),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(
+            torture_sftp_aio_read_more_than_cap),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(torture_sftp_aio_write_file),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(
+            torture_sftp_aio_write_more_than_cap),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(torture_sftp_aio_read_negative),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(torture_sftp_aio_write_negative),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(
+            torture_sftp_aio_read_unordered_wait),
+        DIRECT_AND_PROXYJUMP_SETUP_TEARDOWN(
+            torture_sftp_aio_write_unordered_wait),
     };
 
     ssh_init();

tests/etc/group.in

@@ -2,4 +2,4 @@ users:x:9000:
 sshd:x:65531:
 nobody:x:65533:
 nogroup:x:65534:nobody
-root:x:65532:
+root:x:0:

tests/etc/passwd.in

@@ -5,5 +5,5 @@ doe:x:5003:9000:doe gecos:@HOMEDIR@/doe:/bin/sh
 frank:x:5003:9000:doe gecos:@HOMEDIR@/frank:/bin/sh
 sshd:x:65530:65531:sshd:@HOMEDIR@:/sbin/nologin
 nobody:x:65533:65534:nobody gecos:@HOMEDIR@:/bin/false
-root:x:65534:65532:root gecos:@HOMEDIR@:/bin/false
+root:x:0:0:root gecos:@HOMEDIR@:/bin/false
 @LOCAL_USER@:x:@LOCAL_UID@:9000:local user:@HOMEDIR@:/bin/false

tests/torture.c

@@ -389,6 +389,143 @@ failed:
     return NULL;
 }
 
+/* always return verification successful */
+static int verify_knownhost_trust_all(UNUSED_PARAM(ssh_session jump_session),
+                                      UNUSED_PARAM(void *user))
+{
+    return SSH_OK;
+}
+
+/**
+ * @brief Create a session connected to server via proxyjump
+ *
+ * @param[in] state A pointer to a pointer to an initialized torture_state
+ *                  structure
+ *
+ * @warning  It is expected that both sshd servers are setup before calling
+ *           this, see torture_setup_sshd_server() and
+ *           torture_setup_sshd_servers()
+ *
+ * TODO: If needed, in future, we can extend this function to:
+ *   - allow caller to pass server host port, user, password similar to
+ *     torture_ssh_session() or club this with that function
+ *
+ *   - allow caller to customize jump hosts and callbacks for each of them
+ */
+ssh_session torture_ssh_session_proxyjump(void)
+{
+    /*
+     * We'll setup the connection chain:
+     * - client
+     * - jump host 1: doe (sshd server, IPV4)
+     * - jump host 2: alice (sshd server1, IPV6)
+     * - server: alice (sshd server, IPV4)
+     */
+    char jump_host_list[1024] = {0};
+    int jump_host_count = 2;
+    const char *jump_host_1_address = torture_server_address(AF_INET);
+    const char *jump_host_2_address = torture_server1_address(AF_INET6);
+    struct ssh_jump_callbacks_struct jump_host_callbacks = {
+        .before_connection = NULL,
+        .verify_knownhost = verify_knownhost_trust_all,
+        .authenticate = NULL,
+    };
+
+    ssh_session session = NULL;
+    bool process_config = false;
+    int rc, i;
+
+    session = ssh_new();
+    if (session == NULL) {
+        fprintf(stderr, "Failed to create new ssh session\n");
+        goto failed;
+    }
+
+    rc = ssh_options_set(session, SSH_OPTIONS_HOST, TORTURE_SSH_SERVER);
+    if (rc < 0) {
+        fprintf(stderr,
+                "Failed to set session host: %s\n",
+                ssh_get_error(session));
+        goto failed;
+    }
+
+    rc = ssh_options_set(session, SSH_OPTIONS_USER, TORTURE_SSH_USER_ALICE);
+    if (rc < 0) {
+        fprintf(stderr,
+                "Failed to set session user: %s\n",
+                ssh_get_error(session));
+        goto failed;
+    }
+
+    rc = ssh_options_set(session, SSH_OPTIONS_PROCESS_CONFIG, &process_config);
+    if (rc < 0) {
+        fprintf(stderr,
+                "Failed to set process config option: %s\n",
+                ssh_get_error(session));
+        goto failed;
+    }
+
+    rc = snprintf(jump_host_list,
+                  sizeof(jump_host_list),
+                  "doe@%s:22,alice@%s:22",
+                  jump_host_1_address,
+                  jump_host_2_address);
+    if (rc < 0) {
+        fprintf(stderr, "snprintf failed: %s\n", strerror(errno));
+        goto failed;
+    }
+
+    if (rc >= (int)sizeof(jump_host_list)) {
+        fprintf(stderr, "Insufficient jump host list buffer size\n");
+        goto failed;
+    }
+
+    rc = ssh_options_set(session, SSH_OPTIONS_PROXYJUMP, jump_host_list);
+    if (rc < 0) {
+        fprintf(stderr,
+                "Failed to set jump hosts for the session: %s\n",
+                ssh_get_error(session));
+        goto failed;
+    }
+
+    for (i = 0; i < jump_host_count; ++i) {
+        rc = ssh_options_set(session,
+                             SSH_OPTIONS_PROXYJUMP_CB_LIST_APPEND,
+                             &jump_host_callbacks);
+        if (rc < 0) {
+            fprintf(stderr,
+                    "Failed to set jump callbacks for jump host %d: %s\n",
+                    i + 1,
+                    ssh_get_error(session));
+            goto failed;
+        }
+    }
+
+    rc = ssh_connect(session);
+    if (rc != SSH_OK) {
+        fprintf(stderr,
+                "Failed to connect to ssh server: %s\n",
+                ssh_get_error(session));
+        goto failed;
+    }
+
+    rc = ssh_userauth_publickey_auto(session, NULL, NULL);
+    if (rc != SSH_AUTH_SUCCESS) {
+        fprintf(stderr, "Public key authentication did not succeed\n");
+        goto failed;
+    }
+
+    return session;
+
+failed:
+    if (ssh_is_connected(session)) {
+        ssh_disconnect(session);
+    }
+    ssh_free(session);
+
+    return NULL;
+}
+
 #ifdef WITH_SERVER
 
 ssh_bind torture_ssh_bind(const char *addr,

tests/torture.h

@@ -115,6 +115,8 @@ ssh_session torture_ssh_session(struct torture_state *s,
                                 const char *user,
                                 const char *password);
 
+ssh_session torture_ssh_session_proxyjump(void);
+
 ssh_bind torture_ssh_bind(const char *addr,
                           const unsigned int port,
                           enum ssh_keytypes_e key_type,

tests/unittests/torture_config.c

@@ -25,7 +25,7 @@ extern LIBSSH_THREAD int ssh_log_level;
 #define HOSTKEYALGORITHMS "ssh-ed25519,ecdsa-sha2-nistp521,ssh-rsa"
 #define PUBKEYACCEPTEDTYPES "rsa-sha2-512,ssh-rsa,ecdsa-sha2-nistp521"
 #define MACS "hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com"
-#define USER_KNOWN_HOSTS "%d/my_known_hosts"
+#define USER_KNOWN_HOSTS "%d/.ssh/my_known_hosts"
 #define GLOBAL_KNOWN_HOSTS "/etc/ssh/my_ssh_known_hosts"
 #define BIND_ADDRESS "::1"
 

tests/unittests/torture_misc.c

@@ -52,7 +52,7 @@ static void torture_get_user_home_dir(void **state) {
 
     (void) state;
 
-    user = ssh_get_user_home_dir();
+    user = ssh_get_user_home_dir(NULL);
     assert_non_null(user);
 #ifndef _WIN32
     assert_string_equal(user, pwd->pw_dir);
@@ -288,7 +288,8 @@ static void torture_path_expand_escape(void **state) {
     const char *s = "%d/%h/%p/by/%r";
     char *e;
 
-    session->opts.sshdir = strdup("guru");
+    /* Set the homedir here to prevent querying the NSS DB */
+    session->opts.homedir = strdup("guru");
     session->opts.host = strdup("meditation");
     session->opts.port = 0;
     session->opts.username = strdup("root");
@@ -310,9 +311,10 @@ static void torture_path_expand_known_hosts(void **state) {
     ssh_session session = *state;
     char *tmp;
 
-    session->opts.sshdir = strdup("/home/guru/.ssh");
+    /* Set the homedir here to prevent querying the NSS DB */
+    session->opts.homedir = strdup("/home/guru");
 
-    tmp = ssh_path_expand_escape(session, "%d/known_hosts");
+    tmp = ssh_path_expand_escape(session, "%d/.ssh/known_hosts");
     assert_non_null(tmp);
     assert_string_equal(tmp, "/home/guru/.ssh/known_hosts");
     free(tmp);
@@ -322,9 +324,10 @@ static void torture_path_expand_percent(void **state) {
     ssh_session session = *state;
     char *tmp;
 
-    session->opts.sshdir = strdup("/home/guru/.ssh");
+    /* Set the homedir here to prevent querying the NSS DB */
+    session->opts.homedir = strdup("/home/guru");
 
-    tmp = ssh_path_expand_escape(session, "%d/config%%1");
+    tmp = ssh_path_expand_escape(session, "%d/.ssh/config%%1");
     assert_non_null(tmp);
     assert_string_equal(tmp, "/home/guru/.ssh/config%1");
     free(tmp);

tests/unittests/torture_options.c

@@ -839,6 +839,7 @@ static void torture_options_get_identity(void **state)
     char *identity = NULL;
     int rc;
 
+    /* This adds an identity to the head of the list and returns */
     rc = ssh_options_set(session, SSH_OPTIONS_ADD_IDENTITY, "identity1");
     assert_true(rc == 0);
     rc = ssh_options_get(session, SSH_OPTIONS_IDENTITY, &identity);
@@ -856,6 +857,48 @@ static void torture_options_get_identity(void **state)
     assert_non_null(identity);
     assert_string_equal(identity, "identity2");
     ssh_string_free_char(identity);
+
+    /* Iterate over all of the identities */
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_OK);
+    assert_string_equal(identity, "identity2");
+    ssh_string_free_char(identity);
+
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_OK);
+    assert_string_equal(identity, "identity1");
+    SAFE_FREE(identity);
+
+    /* here are the default identities */
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_OK);
+    assert_string_equal(identity, "%d/.ssh/id_ed25519");
+    ssh_string_free_char(identity);
+
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_OK);
+    assert_string_equal(identity, "%d/.ssh/id_ecdsa");
+    ssh_string_free_char(identity);
+
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_OK);
+    assert_string_equal(identity, "%d/.ssh/id_rsa");
+    ssh_string_free_char(identity);
+
+#ifdef WITH_FIDO2
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_OK);
+    assert_string_equal(identity, "%d/.ssh/id_ed25519_sk");
+    ssh_string_free_char(identity);
+
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_OK);
+    assert_string_equal(identity, "%d/.ssh/id_ecdsa_sk");
+    ssh_string_free_char(identity);
+#endif /* WITH_FIDO2 */
+
+    rc = ssh_options_get(session, SSH_OPTIONS_NEXT_IDENTITY, &identity);
+    assert_int_equal(rc, SSH_EOF);
 }
 
 static void torture_options_set_global_knownhosts(void **state)
@@ -2067,25 +2110,25 @@ static void torture_options_apply (void **state)
     rc = ssh_list_append(awaited_list, id);
     assert_int_equal(rc, SSH_OK);
     /* append the defaults; this list is copied from ssh_new@src/session.c */
-    id = ssh_path_expand_escape(session, "%d/id_ed25519");
+    id = ssh_path_expand_escape(session, "%d/.ssh/id_ed25519");
     rc = ssh_list_append(awaited_list, id);
     assert_int_equal(rc, SSH_OK);
 #ifdef HAVE_ECC
-    id = ssh_path_expand_escape(session, "%d/id_ecdsa");
+    id = ssh_path_expand_escape(session, "%d/.ssh/id_ecdsa");
     rc = ssh_list_append(awaited_list, id);
     assert_int_equal(rc, SSH_OK);
 #endif
-    id = ssh_path_expand_escape(session, "%d/id_rsa");
+    id = ssh_path_expand_escape(session, "%d/.ssh/id_rsa");
     rc = ssh_list_append(awaited_list, id);
     assert_int_equal(rc, SSH_OK);
 #ifdef WITH_FIDO2
     /* Add security key identities */
-    id = ssh_path_expand_escape(session, "%d/id_ed25519_sk");
+    id = ssh_path_expand_escape(session, "%d/.ssh/id_ed25519_sk");
     rc = ssh_list_append(awaited_list, id);
     assert_int_equal(rc, SSH_OK);
 
 #ifdef HAVE_ECC
-    id = ssh_path_expand_escape(session, "%d/id_ecdsa_sk");
+    id = ssh_path_expand_escape(session, "%d/.ssh/id_ecdsa_sk");
     rc = ssh_list_append(awaited_list, id);
     assert_int_equal(rc, SSH_OK);
 #endif /* HAVE_ECC */