[dpdk-dev] [PATCH 0/7] NIC port restoration

[Wei Dai]

Sometimes configuration and run time environment in PMD layer of a
running port is changed and the port has to stop all bi-directional
traffic and initialize the port device and restore its configurations
and traffic again. Such this procedure can be regarded as restoration.
Some customers found that in some VFN scenario a running port with
i40e VF DPDK PMD has to be restored when the host PF with kernel driver
need to reset the whole PF due to some reasons. For example, i40e HW
may need a reset after reconfiguring a few key registers. Then PF host
needs to Inform all VFs generated by that PF with the event of 'PF
reset'. After VF driver see this event, VF driver needs to restore
the VF port.

In order to make restoration as a common functionality of all PMD,
based on current implementation of rte_ethdev, most of restoration work
can be done in rte_ethdev layer with the settings stored in data
structures in rte_ethdev layer. If some settings is not stored in
rte_ethdev after they are configured before, they should be stored for
restoration by adding data structure in rte_ethdev layer. Ethdev should
also add a API like dev_restore( ) for PMD possible specific work in
restoration process.
The outline of restoration procedure is as follows.
1. rte_eth_dev_stop(port_id);
2. (*dev->dev_ops->dev_uninit)(dev);
3. (*dev->dev_ops->dev_init)(dev);
4. rte_eth_dev_configure(...);
5. rte_eth_dev_rx_queue_config(...) invoked for all Rx queues.
6. rte_eth_dev_tx_queue_config(...) invoked for all Tx queues;
7. rte_eth_start(port)
7.1 rte_eth_dev_config_restore(port); // need to be enhanced
8. (*dev->dev_ops->dev_restore)(dev); // doing PMD specific restoration


Wei Dai (7):
ethdev: add support of NIC restoration
ethdev: add support of restoration of queue state
ethdev: add support of restoration of port status
ethdev: add support of MTU restoration
ethdev: add support of restoration of multicast addr
net/ixgbe: add support of restoration
net/i40e: add support of restoration

drivers/net/i40e/i40e_ethdev.c | 2 +
drivers/net/i40e/i40e_ethdev_vf.c | 5 +
drivers/net/ixgbe/ixgbe_ethdev.c | 4 +
lib/librte_ether/rte_ethdev.c | 280 +++++++++++++++++++++++++++++++--
lib/librte_ether/rte_ethdev.h | 45 +++++-
lib/librte_ether/rte_ether_version.map | 6 +
6 files changed, 326 insertions(+), 16 deletions(-)

[Wei Dai]

The steps of NIC restoration process include following items in order:
dev_stop, dev_uninit, dev_init, dev_configure with stored configuration,
setup each Rx and Tx queue with previous configurations and dev_start.

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 102 ++++++++++++++++++++++++++++++++-
lib/librte_ether/rte_ethdev.h | 36 ++++++++++++
lib/librte_ether/rte_ether_version.map | 6 ++
3 files changed, 142 insertions(+), 2 deletions(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 83898a8..a5a9519 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -795,17 +795,39 @@ rte_eth_dev_configure(uint8_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
/*
* Setup new number of RX/TX queues and reconfigure device.
*/
+ if (dev->data->rxq_conf == NULL) {
+ dev->data->rxq_conf = rte_zmalloc("ethdev->rxq_conf",
+ sizeof(struct rte_eth_rx_queue_conf) * nb_rx_q,
+ RTE_CACHE_LINE_SIZE);
+ if (dev->data->rxq_conf == NULL)
+ return -ENOMEM;
+ }
+
diag = rte_eth_dev_rx_queue_config(dev, nb_rx_q);
if (diag != 0) {
RTE_PMD_DEBUG_TRACE("port%d rte_eth_dev_rx_queue_config = %d\n",
port_id, diag);
+ rte_free(dev->data->rxq_conf);
return diag;
}

+ if (dev->data->txq_conf == NULL) {
+ dev->data->txq_conf = rte_zmalloc("ethdev->txq_conf",
+ sizeof(struct rte_eth_tx_queue_conf) * nb_tx_q,
+ RTE_CACHE_LINE_SIZE);
+ if (dev->data->txq_conf == NULL) {
+ rte_free(dev->data->rxq_conf);
+ rte_eth_dev_rx_queue_config(dev, 0);
+ return -ENOMEM;
+ }
+ }
+
diag = rte_eth_dev_tx_queue_config(dev, nb_tx_q);
if (diag != 0) {
RTE_PMD_DEBUG_TRACE("port%d rte_eth_dev_tx_queue_config = %d\n",
port_id, diag);
+ rte_free(dev->data->rxq_conf);
+ rte_free(dev->data->txq_conf);
rte_eth_dev_rx_queue_config(dev, 0);
return diag;
}
@@ -814,6 +836,8 @@ rte_eth_dev_configure(uint8_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
if (diag != 0) {
RTE_PMD_DEBUG_TRACE("port%d dev_configure = %d\n",
port_id, diag);
+ rte_free(dev->data->rxq_conf);
+ rte_free(dev->data->txq_conf);
rte_eth_dev_rx_queue_config(dev, 0);
rte_eth_dev_tx_queue_config(dev, 0);
return diag;
@@ -1005,6 +1029,7 @@ rte_eth_rx_queue_setup(uint8_t port_id, uint16_t rx_queue_id,
struct rte_eth_dev *dev;
struct rte_eth_dev_info dev_info;
void **rxq;
+ struct rte_eth_rx_queue_conf *rxq_conf;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

@@ -1080,6 +1105,12 @@ rte_eth_rx_queue_setup(uint8_t port_id, uint16_t rx_queue_id,
if (!dev->data->min_rx_buf_size ||
dev->data->min_rx_buf_size > mbp_buf_size)
dev->data->min_rx_buf_size = mbp_buf_size;
+
+ rxq_conf = &dev->data->rxq_conf[rx_queue_id];
+ rxq_conf->nb_rx_desc = nb_rx_desc;
+ rxq_conf->socket_id = socket_id;
+ rxq_conf->rx_conf = *rx_conf;
+ rxq_conf->mp = mp;
}

return ret;
@@ -1093,6 +1124,8 @@ rte_eth_tx_queue_setup(uint8_t port_id, uint16_t tx_queue_id,
struct rte_eth_dev *dev;
struct rte_eth_dev_info dev_info;
void **txq;
+ int ret;
+ struct rte_eth_tx_queue_conf *txq_conf;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

@@ -1136,8 +1169,16 @@ rte_eth_tx_queue_setup(uint8_t port_id, uint16_t tx_queue_id,
if (tx_conf == NULL)
tx_conf = &dev_info.default_txconf;

- return (*dev->dev_ops->tx_queue_setup)(dev, tx_queue_id, nb_tx_desc,
- socket_id, tx_conf);
+ ret = (*dev->dev_ops->tx_queue_setup)(dev, tx_queue_id, nb_tx_desc,
+ socket_id, tx_conf);
+ if (!ret) {
+ txq_conf = &dev->data->txq_conf[tx_queue_id];
+ txq_conf->nb_tx_desc = nb_tx_desc;
+ txq_conf->socket_id = socket_id;
+ txq_conf->tx_conf = *tx_conf;
+ }
+
+ return ret;
}

void
@@ -3472,3 +3513,60 @@ rte_eth_dev_l2_tunnel_offload_set(uint8_t port_id,
-ENOTSUP);
return (*dev->dev_ops->l2_tunnel_offload_set)(dev, l2_tunnel, mask, en);
}
+
+int
+rte_eth_dev_restore(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+ uint16_t q;
+ struct rte_eth_rx_queue_conf *rxq_conf;
+ struct rte_eth_tx_queue_conf *txq_conf;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_uninit, -ENOTSUP);
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_init, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+
+ ret = dev->dev_ops->dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = dev->dev_ops->dev_init(dev);
+ if (ret)
+ return ret;
+
+ ret = rte_eth_dev_configure(port_id, dev->data->nb_rx_queues,
+ dev->data->nb_tx_queues, &dev->data->dev_conf);
+ if (ret)
+ return ret;
+
+ for (q = 0; q < dev->data->nb_rx_queues; q++) {
+ rxq_conf = &dev->data->rxq_conf[q];
+ ret = rte_eth_rx_queue_setup(port_id, q, rxq_conf->nb_rx_desc,
+ rxq_conf->socket_id,
+ &rxq_conf->rx_conf,
+ rxq_conf->mp);
+ if (!ret)
+ return ret;
+ }
+
+ for (q = 0; q < dev->data->nb_tx_queues; q++) {
+ txq_conf = &dev->data->txq_conf[q];
+ ret = rte_eth_tx_queue_setup(port_id, q, txq_conf->nb_tx_desc,
+ txq_conf->socket_id,
+ &txq_conf->tx_conf);
+ if (!ret)
+ return ret;
+ }
+
+ ret = rte_eth_dev_start(port_id);
+
+ if (dev->dev_ops->dev_restore)
+ ret = dev->dev_ops->dev_restore(dev);
+
+ return ret;
+}
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index 0f38b45..0298a1f 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1085,6 +1085,15 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_init_t)(struct rte_eth_dev *dev);
+/** < @internal Function used to initialize a configured Ethernet device. */
+
+typedef int (*eth_dev_uninit_t)(struct rte_eth_dev *dev);
+/** < @internal Function used to uninit a configured Ethernet device. */
+
+typedef int (*eth_dev_restore_t)(struct rte_eth_dev *dev);
+/**< @internal Function used to restore a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1455,6 +1464,9 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_init_t dev_init; /**< Initialize device */
+ eth_dev_uninit_t dev_uninit; /**< Uninit device */
+ eth_dev_restore_t dev_restore; /**< Restore device */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -1689,6 +1701,19 @@ struct rte_eth_dev_sriov {

#define RTE_ETH_NAME_MAX_LEN (32)

+struct rte_eth_rx_queue_conf {
+ uint16_t nb_rx_desc;
+ unsigned int socket_id;
+ struct rte_eth_rxconf rx_conf;
+ struct rte_mempool *mp;
+};
+
+struct rte_eth_tx_queue_conf {
+ uint16_t nb_tx_desc;
+ unsigned int socket_id;
+ struct rte_eth_txconf tx_conf;
+};
+
/**
* @internal
* The data part, with no function pointers, associated with each ethernet device.
@@ -1738,6 +1763,9 @@ struct rte_eth_dev_data {
enum rte_kernel_driver kdrv; /**< Kernel driver passthrough */
int numa_node; /**< NUMA node connection */
const char *drv_name; /**< Driver name */
+
+ struct rte_eth_rx_queue_conf *rxq_conf;
+ struct rte_eth_tx_queue_conf *txq_conf;
};

/** Device supports hotplug detach */
@@ -2168,6 +2196,14 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Resotre a Ethernet device.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_restore(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index d6726bb..b867c4c 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -156,3 +156,9 @@ DPDK_17.05 {
rte_eth_xstats_get_names_by_id;

} DPDK_17.02;
+
+DPDK_17.08 {
+ global:
+
+ rte_eth_dev_restore;
+} DPDK_17.05;

[Wei Dai]

As dev->dev_ops->dev_start may change dev->data->rx_queue_state[]
and dev->data->tx_queue_state[], this patch adds rxq_restore_state[]
and txq_restore_state[ ] for restoration.
In the restoration process, PMD should start or stop each Rx or Tx
queue according to dev->data->rx_restore_state[] or
dev->data->tx_restore_state[].

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 87 +++++++++++++++++++++++++++++++++++++++----
lib/librte_ether/rte_ethdev.h | 5 ++-
2 files changed, 83 insertions(+), 9 deletions(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index a5a9519..97c0044 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -504,6 +504,7 @@ int
rte_eth_dev_rx_queue_start(uint8_t port_id, uint16_t rx_queue_id)
{
struct rte_eth_dev *dev;
+ int ret;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

@@ -522,14 +523,18 @@ rte_eth_dev_rx_queue_start(uint8_t port_id, uint16_t rx_queue_id)
return 0;
}

- return dev->dev_ops->rx_queue_start(dev, rx_queue_id);
-
+ ret = dev->dev_ops->rx_queue_start(dev, rx_queue_id);
+ if (!ret)
+ dev->data->rxq_restore_state[rx_queue_id] =
+ RTE_ETH_QUEUE_STATE_STARTED;
+ return ret;
}

int
rte_eth_dev_rx_queue_stop(uint8_t port_id, uint16_t rx_queue_id)
{
struct rte_eth_dev *dev;
+ int ret;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

@@ -548,14 +553,18 @@ rte_eth_dev_rx_queue_stop(uint8_t port_id, uint16_t rx_queue_id)
return 0;
}

- return dev->dev_ops->rx_queue_stop(dev, rx_queue_id);
-
+ ret = dev->dev_ops->rx_queue_stop(dev, rx_queue_id);
+ if (!ret)
+ dev->data->rxq_restore_state[rx_queue_id] =
+ RTE_ETH_QUEUE_STATE_STOPPED;
+ return ret;
}

int
rte_eth_dev_tx_queue_start(uint8_t port_id, uint16_t tx_queue_id)
{
struct rte_eth_dev *dev;
+ int ret;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

@@ -574,14 +583,18 @@ rte_eth_dev_tx_queue_start(uint8_t port_id, uint16_t tx_queue_id)
return 0;
}

- return dev->dev_ops->tx_queue_start(dev, tx_queue_id);
-
+ ret = dev->dev_ops->tx_queue_start(dev, tx_queue_id);
+ if (!ret)
+ dev->data->txq_restore_state[tx_queue_id] =
+ RTE_ETH_QUEUE_STATE_STARTED;
+ return ret;
}

int
rte_eth_dev_tx_queue_stop(uint8_t port_id, uint16_t tx_queue_id)
{
struct rte_eth_dev *dev;
+ int ret;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

@@ -600,8 +613,11 @@ rte_eth_dev_tx_queue_stop(uint8_t port_id, uint16_t tx_queue_id)
return 0;
}

- return dev->dev_ops->tx_queue_stop(dev, tx_queue_id);
-
+ ret = dev->dev_ops->tx_queue_stop(dev, tx_queue_id);
+ if (!ret)
+ dev->data->txq_restore_state[tx_queue_id] =
+ RTE_ETH_QUEUE_STATE_STOPPED;
+ return ret;
}

static int
@@ -863,6 +879,50 @@ _rte_eth_dev_reset(struct rte_eth_dev *dev)
}

static void
+rte_eth_dev_rx_queue_restore(uint8_t port_id, uint16_t queue_id)
+{
+ struct rte_eth_dev *dev;
+ uint16_t q = queue_id;
+
+ dev = &rte_eth_devices[port_id];
+
+ if (dev->data->in_restoration == 0) {
+ dev->data->rxq_restore_state[q] = dev->data->rx_queue_state[q];
+ return;
+ }
+
+ if (dev->data->rxq_restore_state[q] != dev->data->rx_queue_state[q]) {
+ if (dev->data->rxq_restore_state[q]
+ == RTE_ETH_QUEUE_STATE_STARTED)
+ rte_eth_dev_rx_queue_start(port_id, q);
+ else
+ rte_eth_dev_rx_queue_stop(port_id, q);
+ }
+}
+
+static void
+rte_eth_dev_tx_queue_restore(uint8_t port_id, uint16_t queue_id)
+{
+ struct rte_eth_dev *dev;
+ uint16_t q = queue_id;
+
+ dev = &rte_eth_devices[port_id];
+
+ if (dev->data->in_restoration == 0) {
+ dev->data->txq_restore_state[q] = dev->data->tx_queue_state[q];
+ return;
+ }
+
+ if (dev->data->txq_restore_state[q] != dev->data->tx_queue_state[q]) {
+ if (dev->data->txq_restore_state[q]
+ == RTE_ETH_QUEUE_STATE_STARTED)
+ rte_eth_dev_tx_queue_start(port_id, q);
+ else
+ rte_eth_dev_tx_queue_stop(port_id, q);
+ }
+}
+
+static void
rte_eth_dev_config_restore(uint8_t port_id)
{
struct rte_eth_dev *dev;
@@ -871,6 +931,7 @@ rte_eth_dev_config_restore(uint8_t port_id)
uint16_t i;
uint32_t pool = 0;
uint64_t pool_mask;
+ uint16_t q;

dev = &rte_eth_devices[port_id];

@@ -915,6 +976,12 @@ rte_eth_dev_config_restore(uint8_t port_id)
rte_eth_allmulticast_enable(port_id);
else if (rte_eth_allmulticast_get(port_id) == 0)
rte_eth_allmulticast_disable(port_id);
+
+ for (q = 0; q < dev->data->nb_rx_queues; q++)
+ rte_eth_dev_rx_queue_restore(port_id, q);
+ for (q = 0; q < dev->data->nb_tx_queues; q++)
+ rte_eth_dev_tx_queue_restore(port_id, q);
+
}

int
@@ -3531,6 +3598,8 @@ rte_eth_dev_restore(uint8_t port_id)

rte_eth_dev_stop(port_id);

+ dev->data->in_restoration = 1;
+
ret = dev->dev_ops->dev_uninit(dev);
if (ret)
return ret;
@@ -3568,5 +3637,7 @@ rte_eth_dev_restore(uint8_t port_id)
if (dev->dev_ops->dev_restore)
ret = dev->dev_ops->dev_restore(dev);

+ dev->data->in_restoration = 0;
+
return ret;
}
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index 0298a1f..7a2ce07 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1754,10 +1754,13 @@ struct rte_eth_dev_data {
scattered_rx : 1, /**< RX of scattered packets is ON(1) / OFF(0) */
all_multicast : 1, /**< RX all multicast mode ON(1) / OFF(0). */
dev_started : 1, /**< Device state: STARTED(1) / STOPPED(0). */
- lro : 1; /**< RX LRO is ON(1) / OFF(0) */
+ lro : 1, /**< RX LRO is ON(1) / OFF(0) */
+ in_restoration : 1; /**< In Restoration Yes(1) / NO(0) */
uint8_t rx_queue_state[RTE_MAX_QUEUES_PER_PORT];
+ uint8_t rxq_restore_state[RTE_MAX_QUEUES_PER_PORT];
/** Queues state: STARTED(1) / STOPPED(0) */
uint8_t tx_queue_state[RTE_MAX_QUEUES_PER_PORT];
+ uint8_t txq_restore_state[RTE_MAX_QUEUES_PER_PORT];
/** Queues state: STARTED(1) / STOPPED(0) */
uint32_t dev_flags; /**< Capabilities */
enum rte_kernel_driver kdrv; /**< Kernel driver passthrough */

[Wei Dai]

As dev->data->dev_link.link_status may change when the port is
initialized again, this patch adds dev->data->restore_link for
restoration.
In the restoration process, ethdev layer can restore link status
as up or down by comparing dev->data->restore_link.link_status
and dev->data->dev_link.link_status.

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 37 +++++++++++++++++++++++++++++++++++--
lib/librte_ether/rte_ethdev.h | 1 +
2 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 97c0044..af8ccf6 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -923,6 +923,28 @@ rte_eth_dev_tx_queue_restore(uint8_t port_id, uint16_t queue_id)
}

static void
+rte_eth_dev_link_status_restore(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+
+ dev = &rte_eth_devices[port_id];
+
+ if (dev->data->in_restoration == 0) {
+ dev->data->restore_link.link_status =
+ dev->data->dev_link.link_status;
+ return;
+ }
+
+ if (dev->data->restore_link.link_status
+ != dev->data->dev_link.link_status) {
+ if (dev->data->restore_link.link_status != 0)
+ rte_eth_dev_set_link_up(port_id);
+ else
+ rte_eth_dev_set_link_down(port_id);
+ }
+}
+
+static void
rte_eth_dev_config_restore(uint8_t port_id)
{
struct rte_eth_dev *dev;
@@ -982,6 +1004,7 @@ rte_eth_dev_config_restore(uint8_t port_id)
for (q = 0; q < dev->data->nb_tx_queues; q++)
rte_eth_dev_tx_queue_restore(port_id, q);

+ rte_eth_dev_link_status_restore(port_id);
}

int
@@ -1014,6 +1037,8 @@ rte_eth_dev_start(uint8_t port_id)
if (dev->data->dev_conf.intr_conf.lsc == 0) {
RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->link_update, -ENOTSUP);
(*dev->dev_ops->link_update)(dev, 0);
+ dev->data->restore_link.link_status =
+ dev->data->dev_link.link_status;
}
return 0;
}
@@ -1043,26 +1068,34 @@ int
rte_eth_dev_set_link_up(uint8_t port_id)
{
struct rte_eth_dev *dev;
+ int ret;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

dev = &rte_eth_devices[port_id];

RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_set_link_up, -ENOTSUP);
- return (*dev->dev_ops->dev_set_link_up)(dev);
+ ret = (*dev->dev_ops->dev_set_link_up)(dev);
+ if (!ret)
+ dev->data->restore_link.link_status = 1;
+ return ret;
}

int
rte_eth_dev_set_link_down(uint8_t port_id)
{
struct rte_eth_dev *dev;
+ int ret;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

dev = &rte_eth_devices[port_id];

RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_set_link_down, -ENOTSUP);
- return (*dev->dev_ops->dev_set_link_down)(dev);
+ ret = (*dev->dev_ops->dev_set_link_down)(dev);
+ if (!ret)
+ dev->data->restore_link.link_status = 0;
+ return ret;
}

void
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index 7a2ce07..9428f57 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1734,6 +1734,7 @@ struct rte_eth_dev_data {
void *dev_private; /**< PMD-specific private data */

struct rte_eth_link dev_link;
+ struct rte_eth_link restore_link;
/**< Link-level information & status */

struct rte_eth_conf dev_conf; /**< Configuration applied to device. */

[Wei Dai]

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 23 +++++++++++++++++++++--
lib/librte_ether/rte_ethdev.h | 1 +
2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index af8ccf6..0d9544c 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -945,6 +945,22 @@ rte_eth_dev_link_status_restore(uint8_t port_id)
}

static void
+rte_eth_dev_mtu_restore(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+
+ dev = &rte_eth_devices[port_id];
+
+ if (dev->data->in_restoration == 0) {
+ dev->data->restore_mtu = dev->data->mtu;
+ return;
+ }
+
+ if (dev->data->restore_mtu != dev->data->mtu)
+ rte_eth_dev_set_mtu(port_id, dev->data->restore_mtu);
+}
+
+static void
rte_eth_dev_config_restore(uint8_t port_id)
{
struct rte_eth_dev *dev;
@@ -1005,6 +1021,8 @@ rte_eth_dev_config_restore(uint8_t port_id)
rte_eth_dev_tx_queue_restore(port_id, q);

rte_eth_dev_link_status_restore(port_id);
+
+ rte_eth_dev_mtu_restore(port_id);
}

int
@@ -2106,9 +2124,10 @@ rte_eth_dev_set_mtu(uint8_t port_id, uint16_t mtu)
RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->mtu_set, -ENOTSUP);

ret = (*dev->dev_ops->mtu_set)(dev, mtu);
- if (!ret)
+ if (!ret) {
dev->data->mtu = mtu;
-
+ dev->data->restore_mtu = mtu;
+ }
return ret;
}

diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index 9428f57..aca8510 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1739,6 +1739,7 @@ struct rte_eth_dev_data {

struct rte_eth_conf dev_conf; /**< Configuration applied to device. */
uint16_t mtu; /**< Maximum Transmission Unit. */
+ uint16_t restore_mtu;

uint32_t min_rx_buf_size;
/**< Common rx buffer size handled by all queues */

[Wei Dai]

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 31 ++++++++++++++++++++++++++++++-
lib/librte_ether/rte_ethdev.h | 2 ++
2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 0d9544c..78609f5 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -961,6 +961,20 @@ rte_eth_dev_mtu_restore(uint8_t port_id)
}

static void
+rte_eth_dev_mc_addr_list_restore(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+
+ dev = &rte_eth_devices[port_id];
+
+ if (dev->data->mc_addr_count == 0)
+ return;
+
+ rte_eth_dev_set_mc_addr_list(port_id, dev->data->mc_addr_list,
+ dev->data->mc_addr_count);
+}
+
+static void
rte_eth_dev_config_restore(uint8_t port_id)
{
struct rte_eth_dev *dev;
@@ -1023,6 +1037,8 @@ rte_eth_dev_config_restore(uint8_t port_id)
rte_eth_dev_link_status_restore(port_id);

rte_eth_dev_mtu_restore(port_id);
+
+ rte_eth_dev_mc_addr_list_restore(port_id);
}

int
@@ -3423,12 +3439,25 @@ rte_eth_dev_set_mc_addr_list(uint8_t port_id,
uint32_t nb_mc_addr)
{
struct rte_eth_dev *dev;
+ int ret;

RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);

dev = &rte_eth_devices[port_id];
RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->set_mc_addr_list, -ENOTSUP);
- return dev->dev_ops->set_mc_addr_list(dev, mc_addr_set, nb_mc_addr);
+ ret = dev->dev_ops->set_mc_addr_list(dev, mc_addr_set, nb_mc_addr);
+ if (!ret) {
+ if (dev->data->mc_addr_list != NULL)
+ rte_free(dev->data->mc_addr_list);
+ dev->data->mc_addr_list = rte_zmalloc("dev->mc_addr_list",
+ sizeof(struct ether_addr) * nb_mc_addr, 0);
+ if (dev->data->mc_addr_list != NULL) {
+ rte_memcpy(dev->data->mc_addr_list, mc_addr_set,
+ sizeof(struct ether_addr) * nb_mc_addr);
+ dev->data->mc_addr_count = nb_mc_addr;
+ }
+ }
+ return ret;
}

int
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index aca8510..8f7e772 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1771,6 +1771,8 @@ struct rte_eth_dev_data {

struct rte_eth_rx_queue_conf *rxq_conf;
struct rte_eth_tx_queue_conf *txq_conf;
+ uint32_t mc_addr_count;
+ struct ether_addr *mc_addr_list;
};

/** Device supports hotplug detach */

[Wei Dai]

export dev_uninit and dev_init for restoration to
reset port but keep same port id.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 2083cde..64b8a78 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -525,6 +525,8 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_init = eth_ixgbe_dev_init,
+ .dev_uninit = eth_ixgbe_dev_uninit,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -625,6 +627,8 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_init = eth_ixgbevf_dev_init,
+ .dev_uninit = eth_ixgbevf_dev_uninit,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,

[Wei Dai]

export dev_uninit and dev_init for restoration to
reset port but keep same port id.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 2 ++
drivers/net/i40e/i40e_ethdev_vf.c | 5 +++++
2 files changed, 7 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 4c49673..9512ca4 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -449,6 +449,8 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_init = eth_i40e_dev_init,
+ .dev_uninit = eth_i40e_dev_uninit,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 859b5e8..9405888 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -163,6 +163,9 @@ static void i40evf_handle_pf_event(__rte_unused struct rte_eth_dev *dev,
uint8_t *msg,
uint16_t msglen);

+static int i40evf_dev_init(struct rte_eth_dev *eth_dev);
+static int i40evf_dev_uninit(struct rte_eth_dev *eth_dev);
+
/* Default hash key buffer for RSS */
static uint32_t rss_key_default[I40E_VFQF_HKEY_MAX_INDEX + 1];

@@ -194,6 +197,8 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.dev_configure = i40evf_dev_configure,
.dev_start = i40evf_dev_start,
.dev_stop = i40evf_dev_stop,
+ .dev_init = i40evf_dev_init,
+ .dev_uninit = i40evf_dev_uninit,
.promiscuous_enable = i40evf_dev_promiscuous_enable,
.promiscuous_disable = i40evf_dev_promiscuous_disable,
.allmulticast_enable = i40evf_dev_allmulticast_enable,

[Wu, Jingjing]

-----Original Message-----
From: Dai, Wei
Sent: Saturday, May 27, 2017 4:22 PM
Subject: [PATCH 7/7] net/i40e: add support of restoration
export dev_uninit and dev_init for restoration to
reset port but keep same port id.
---
drivers/net/i40e/i40e_ethdev.c | 2 ++
drivers/net/i40e/i40e_ethdev_vf.c | 5 +++++
2 files changed, 7 insertions(+)
diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 4c49673..9512ca4 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -449,6 +449,8 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_init = eth_i40e_dev_init,
+ .dev_uninit = eth_i40e_dev_uninit,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,

you can just get int and unint func point by eth_dev->driver.eth_dev_init and eth_dev->driver.eth_dev_init

No need to define ops here.

[Dai, Wei]

-----Original Message-----
From: Wu, Jingjing
Sent: Wednesday, June 7, 2017 3:50 PM
Subject: RE: [PATCH 7/7] net/i40e: add support of restoration

-----Original Message-----
From: Dai, Wei
Sent: Saturday, May 27, 2017 4:22 PM
Subject: [PATCH 7/7] net/i40e: add support of restoration
export dev_uninit and dev_init for restoration to reset port but keep
same port id.
---
drivers/net/i40e/i40e_ethdev.c | 2 ++
drivers/net/i40e/i40e_ethdev_vf.c | 5 +++++
2 files changed, 7 insertions(+)
diff --git a/drivers/net/i40e/i40e_ethdev.c
b/drivers/net/i40e/i40e_ethdev.c index 4c49673..9512ca4 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -449,6 +449,8 @@ static const struct eth_dev_ops i40e_eth_dev_ops =

{

.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_init = eth_i40e_dev_init,
+ .dev_uninit = eth_i40e_dev_uninit,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,

you can just get int and unint func point by eth_dev->driver.eth_dev_init and
eth_dev->driver.eth_dev_init
No need to define ops here.

I'm sorry that struct rte_eth_dev doesn't have member driver.
The struct rte_eth_dev_data, struct rte_pci_device and struct rte_pci_device don't have
Any member to refer dev_init() and dev_uninit.

[Wei Dai]

Sometimes a port have to be reset. After reset, if the port goes
through PCI remove() and then PCI probe() for restoration, its
port id may be changed and this is not expected by some customer
DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer
and the other is calling PMD dev_init(). PCI remove( ) release all
resource allocated from rte_ethdev layer in PCI probe( ) and calls
PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ).

New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ),
rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and
rte_eth_dev_start( ) again to restore its previous settings or
to reconfigure itself with different settings.

To test this new feature, 2 testpmd commands are introduced.
The first command "reset_port port_id" calls rte_eth_dev_reset( ).
The second command "reconfig_port port_id" is used to test if the
port can be reconfigured successfully. This command configure the port
with the simplest settings include only 1 Rx queue, only 1 Tx queue,
Rx mode = None and Tx mode = None. It check port by receving at least
100 packets and then forward these packets from itself.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to its VF.
1. run testpmd with ixgbe PF
2. testpmd > set verbose 1 //to observe VF working
3. ifconfig name-of-PF down
4. A message is shown in testmd to indicate PF reset
5. ifconfig name-of-PF up
6. testpmd > stop // stop forwarding to avoid crash during reset
7. testpmd > reset_port port_id_of_VF
8. testpmd > reconfig_port port_id_of_VF


Wei Dai (5):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: add command to test NIC reset
app/testpmd: add command to test NIC restoration

app/test-pmd/cmdline.c | 62 ++++++++++++++++++
app/test-pmd/config.c | 111 +++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 2 +
drivers/net/i40e/i40e_ethdev.c | 16 +++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 +++++
drivers/net/ixgbe/ixgbe_ethdev.c | 38 +++++++++++
lib/librte_ether/rte_ethdev.c | 17 +++++
lib/librte_ether/rte_ethdev.h | 12 ++++
lib/librte_ether/rte_ether_version.map | 2 +-
9 files changed, 275 insertions(+), 1 deletion(-)

[Wei Dai]

A DPDK application can reset a NIC and keep its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset
in similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API dev_reset().

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 17 +++++++++++++++++
lib/librte_ether/rte_ethdev.h | 12 ++++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 81a45c0..5e240f5 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -3349,3 +3349,20 @@ rte_eth_dev_l2_tunnel_offload_set(uint8_t port_id,
-ENOTSUP);
return (*dev->dev_ops->l2_tunnel_offload_set)(dev, l2_tunnel, mask, en);
}
+
+int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+ return ret;
+}
+
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index f6e6c74..3b82dd6 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1087,6 +1087,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1404,6 +1407,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2104,6 +2108,14 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
++ * Reset a Ethernet device.
++ *
++ * @param port_id
++ * The port identifier of the Ethernet device.
++ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index 805e6de..8c0b2c9 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -152,5 +152,5 @@ DPDK_17.08 {
global:

rte_flow_isolate;
-
+ rte_eth_dev_reset;
} DPDK_17.05;

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index ebc5467..5ff8fbe 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -170,6 +170,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -264,6 +265,7 @@ static int ixgbevf_dev_configure(struct rte_eth_dev *dev);
static int ixgbevf_dev_start(struct rte_eth_dev *dev);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -525,6 +527,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -614,6 +617,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2838,6 +2842,23 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -4902,6 +4923,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 16 ++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 ++++++++++++++++
2 files changed, 32 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 4ee1113..7694957 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2135,6 +2137,20 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 2d5a9b5..bf287a0 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -123,6 +123,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -204,6 +205,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2347,6 +2349,20 @@ i40evf_dev_close(struct rte_eth_dev *dev)
}

static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
+static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{
struct i40e_vf *vf = I40E_VSI_TO_VF(vsi);

[Wei Dai]

When a NIC is reset, a message will show it.
And then user can run the command "reset_port port_id"
to process it.

Signed-off-by: Wei Dai <***@intel.com>
---
app/test-pmd/cmdline.c | 31 +++++++++++++++++++++++++++++++
app/test-pmd/config.c | 13 +++++++++++++
app/test-pmd/testpmd.h | 1 +
3 files changed, 45 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index ff8ffd2..90f6bde 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -2588,6 +2588,36 @@ cmdline_parse_inst_t cmd_stop = {
},
};

+/* *** reset a port *** */
+struct cmd_reset_port_result {
+ cmdline_fixed_string_t command;
+ uint8_t port_id;
+};
+
+static void cmd_reset_port_parsed(__attribute__((unused)) void *parsed_result,
+ __attribute__((unused)) struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ struct cmd_reset_port_result *res = parsed_result;
+ reset_port(res->port_id);
+}
+
+cmdline_parse_token_string_t cmd_reset_port_cmd =
+ TOKEN_STRING_INITIALIZER(struct cmd_reset_port_result, command, "reset_port");
+cmdline_parse_token_num_t cmd_reset_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_reset_port_result, port_id, UINT8);
+
+cmdline_parse_inst_t cmd_reset_port = {
+ .f = cmd_reset_port_parsed,
+ .data = NULL,
+ .help_str = "reset_port <port_id>",
+ .tokens = {
+ (void *)&cmd_reset_port_cmd,
+ (void *)&cmd_reset_port_id,
+ NULL,
+ },
+};
+
/* *** SET CORELIST and PORTLIST CONFIGURATION *** */

unsigned int
@@ -13751,6 +13781,7 @@ cmdline_parse_ctx_t main_ctx[] = {
(cmdline_parse_inst_t *)&cmd_write_reg_bit,
(cmdline_parse_inst_t *)&cmd_read_rxd_txd,
(cmdline_parse_inst_t *)&cmd_stop,
+ (cmdline_parse_inst_t *)&cmd_reset_port,
(cmdline_parse_inst_t *)&cmd_mac_addr,
(cmdline_parse_inst_t *)&cmd_set_qmap,
(cmdline_parse_inst_t *)&cmd_operate_port,
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index b0b340e..da3b525 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -3318,3 +3318,16 @@ close_ddp_package_file(uint8_t *buf)

return -1;
}
+
+void
+reset_port(portid_t port_id)
+{
+ int diag;
+
+ if (port_id_is_invalid(port_id, ENABLED_WARN))
+ return;
+ diag = rte_eth_dev_reset(port_id);
+ if (diag == 0)
+ return;
+ printf("Reset port %d failed. diag=%d\n", port_id, diag);
+}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 364502d..956eec5 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -598,6 +598,7 @@ void stop_port(portid_t pid);
void close_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
+void reset_port(portid_t port_id);
int all_ports_stopped(void);
int port_is_started(portid_t port_id);
void pmd_test_exit(void);

[Wu, Jingjing]

-----Original Message-----
From: Dai, Wei
Sent: Tuesday, June 27, 2017 10:07 PM
Subject: [PATCH v2 4/5] app/testpmd: add command to test NIC reset
When a NIC is reset, a message will show it.
And then user can run the command "reset_port port_id"
to process it.
---
app/test-pmd/cmdline.c | 31 +++++++++++++++++++++++++++++++ app/test-
pmd/config.c | 13 +++++++++++++ app/test-pmd/testpmd.h | 1 +
3 files changed, 45 insertions(+)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
ff8ffd2..90f6bde 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -2588,6 +2588,36 @@ cmdline_parse_inst_t cmd_stop = {
},
};
+/* *** reset a port *** */
+struct cmd_reset_port_result {
+ cmdline_fixed_string_t command;
+ uint8_t port_id;
+};
+
+static void cmd_reset_port_parsed(__attribute__((unused)) void *parsed_result,
+ __attribute__((unused)) struct cmdline *cl,
+ __attribute__((unused)) void *data) {
+ struct cmd_reset_port_result *res = parsed_result;
+ reset_port(res->port_id);
+}
+
+cmdline_parse_token_string_t cmd_reset_port_cmd =
+ TOKEN_STRING_INITIALIZER(struct cmd_reset_port_result, command,
+"reset_port"); cmdline_parse_token_num_t cmd_reset_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_reset_port_result, port_id,
UINT8);
+
+cmdline_parse_inst_t cmd_reset_port = {
+ .f = cmd_reset_port_parsed,
+ .data = NULL,
+ .help_str = "reset_port <port_id>",

You can just add one more option in the commands
"port start|stop|close <port_id>" and " port start|stop|close all"
To "port start|stop|close|reset <port_id>" and " port start|stop|close|reset all" instead of defining a new one.

[Dai, Wei]

-----Original Message-----
From: Wu, Jingjing
Sent: Wednesday, June 28, 2017 5:11 PM
Subject: RE: [PATCH v2 4/5] app/testpmd: add command to test NIC reset

-----Original Message-----
From: Dai, Wei
Sent: Tuesday, June 27, 2017 10:07 PM
Subject: [PATCH v2 4/5] app/testpmd: add command to test NIC reset
When a NIC is reset, a message will show it.
And then user can run the command "reset_port port_id"
to process it.
---
app/test-pmd/cmdline.c | 31 +++++++++++++++++++++++++++++++
app/test- pmd/config.c | 13 +++++++++++++ app/test-pmd/testpmd.h |
1 +
3 files changed, 45 insertions(+)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
ff8ffd2..90f6bde 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -2588,6 +2588,36 @@ cmdline_parse_inst_t cmd_stop = {
},
};
+/* *** reset a port *** */
+struct cmd_reset_port_result {
+ cmdline_fixed_string_t command;
+ uint8_t port_id;
+};
+
+static void cmd_reset_port_parsed(__attribute__((unused)) void *parsed_result,
+ __attribute__((unused)) struct cmdline *cl,
+ __attribute__((unused)) void *data) {
+ struct cmd_reset_port_result *res = parsed_result;
+ reset_port(res->port_id);
+}
+
+cmdline_parse_token_string_t cmd_reset_port_cmd =
+ TOKEN_STRING_INITIALIZER(struct cmd_reset_port_result, command,
+"reset_port"); cmdline_parse_token_num_t cmd_reset_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_reset_port_result, port_id, UINT8);
+
+cmdline_parse_inst_t cmd_reset_port = {
+ .f = cmd_reset_port_parsed,
+ .data = NULL,
+ .help_str = "reset_port <port_id>",

You can just add one more option in the commands "port start|stop|close
<port_id>" and " port start|stop|close all"
To "port start|stop|close|reset <port_id>" and " port start|stop|close|reset
all" instead of defining a new one.

Thanks for your feedback.
V3 patch set will follow your guide.

[Wei Dai]

When a NIC is reset, a message will show it.
And then user can run the command "reset_port port_id"
to reset the port and to keep same port id without any
without any configuration.
This patch adds a testpmd command "reconfig_port port_id"
to test whether the port can be reconfigured to have
success Rx and Tx function.
The new command will configure the port with the simplest
setting which includes only 1 Rx queue, only 1 Tx queue,
Rx mode = None and Tx mode = None.
It check if the port can receive and forward some packets.
For example 100 packets in this new command.
Before testing with "reset_port port_id" and then
"reconfig_port port_id", current forwarding should be stopped
first to avoid crash.

Signed-off-by: Wei Dai <***@intel.com>
---
app/test-pmd/cmdline.c | 31 ++++++++++++++++
app/test-pmd/config.c | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
3 files changed, 130 insertions(+)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 90f6bde..1038cee 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -2618,6 +2618,36 @@ cmdline_parse_inst_t cmd_reset_port = {
},
};

+/* *** reconfig a port with simplest settings only for test *** */
+struct cmd_reconfig_port_result {
+ cmdline_fixed_string_t command;
+ uint8_t port_id;
+};
+
+static void cmd_reconfig_port_parsed(__attribute__((unused)) void *parsed_result,
+ __attribute__((unused)) struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ struct cmd_reconfig_port_result *res = parsed_result;
+ reconfig_port(res->port_id);
+}
+
+cmdline_parse_token_string_t cmd_reconfig_port_cmd =
+ TOKEN_STRING_INITIALIZER(struct cmd_reconfig_port_result, command, "reconfig_port");
+cmdline_parse_token_num_t cmd_reconfig_port_id =
+ TOKEN_NUM_INITIALIZER(struct cmd_reconfig_port_result, port_id, UINT8);
+
+cmdline_parse_inst_t cmd_reconfig_port = {
+ .f = cmd_reconfig_port_parsed,
+ .data = NULL,
+ .help_str = "reconfig_port <port_id>",
+ .tokens = {
+ (void *)&cmd_reconfig_port_cmd,
+ (void *)&cmd_reconfig_port_id,
+ NULL,
+ },
+};
+
/* *** SET CORELIST and PORTLIST CONFIGURATION *** */

unsigned int
@@ -13782,6 +13812,7 @@ cmdline_parse_ctx_t main_ctx[] = {
(cmdline_parse_inst_t *)&cmd_read_rxd_txd,
(cmdline_parse_inst_t *)&cmd_stop,
(cmdline_parse_inst_t *)&cmd_reset_port,
+ (cmdline_parse_inst_t *)&cmd_reconfig_port,
(cmdline_parse_inst_t *)&cmd_mac_addr,
(cmdline_parse_inst_t *)&cmd_set_qmap,
(cmdline_parse_inst_t *)&cmd_operate_port,
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index da3b525..1fd6a54 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -3331,3 +3331,101 @@ reset_port(portid_t port_id)
return;
printf("Reset port %d failed. diag=%d\n", port_id, diag);
}
+
+static void
+test_simplest_rxtx(portid_t port)
+{
+#define BURST_SIZE 32
+#define NUM_PACKETS 100
+
+ struct rte_mbuf *bufs[BURST_SIZE];
+ uint16_t nb_rx, nb_tx, total;
+
+ printf("Begin to forward at least %d packets to test reconfiguration\n", NUM_PACKETS);
+ total = 0;
+ while (1) {
+ nb_rx = rte_eth_rx_burst(port, 0, bufs, BURST_SIZE);
+ if (nb_rx == 0)
+ continue;
+ nb_tx = rte_eth_tx_burst(port, 0, bufs, nb_rx);
+ total += nb_tx;
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ if (total >= NUM_PACKETS)
+ break;
+ }
+ printf("Finish forwarding %u packets to test reconfiguration\n", total);
+ return;
+}
+
+int
+reconfig_port(portid_t port)
+{
+#define RX_RING_SIZE 128
+#define TX_RING_SIZE 512
+
+#define NUM_MBUFS 8191
+#define MBUF_CACHE_SIZE 250
+
+ struct rte_mempool *mbuf_pool;
+ struct rte_eth_conf dev_conf = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+ };
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ int retval;
+ uint16_t q;
+
+ if (port_id_is_invalid(port, ENABLED_WARN))
+ return -1;
+
+ /* Creates a new mempool in memory to hold the mbufs. */
+ mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS,
+ MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_eth_dev_socket_id(port));
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &dev_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Display the port MAC address. */
+ struct ether_addr addr;
+ rte_eth_macaddr_get(port, &addr);
+ printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8
+ " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n",
+ (unsigned)port,
+ addr.addr_bytes[0], addr.addr_bytes[1],
+ addr.addr_bytes[2], addr.addr_bytes[3],
+ addr.addr_bytes[4], addr.addr_bytes[5]);
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ test_simplest_rxtx(port);
+
+ return 0;
+}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 956eec5..c4c2e59 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -599,6 +599,7 @@ void close_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
void reset_port(portid_t port_id);
+int reconfig_port(portid_t port_id);
int all_ports_stopped(void);
int port_is_started(portid_t port_id);
void pmd_test_exit(void);

[Wu, Jingjing]

-----Original Message-----
From: Dai, Wei
Sent: Tuesday, June 27, 2017 10:07 PM
Subject: [PATCH v2 5/5] app/testpmd: add command to test NIC restoration
When a NIC is reset, a message will show it.
And then user can run the command "reset_port port_id"
to reset the port and to keep same port id without any without any
configuration.
This patch adds a testpmd command "reconfig_port port_id"
to test whether the port can be reconfigured to have success Rx and Tx function.
The new command will configure the port with the simplest setting which
includes only 1 Rx queue, only 1 Tx queue, Rx mode = None and Tx mode =
None.
It check if the port can receive and forward some packets.
For example 100 packets in this new command.
Before testing with "reset_port port_id" and then "reconfig_port port_id",
current forwarding should be stopped first to avoid crash.

Testpmd has mechanism to support reconfigure, you can check the need_reconfig
Field in port struct.
No need to define a new command.


Thanks
Jingjing

[Dai, Wei]

Thanks for your feedback.
V3 patch set will follow your guide.

-----Original Message-----
From: Wu, Jingjing
Sent: Wednesday, June 28, 2017 5:05 PM
Subject: RE: [PATCH v2 5/5] app/testpmd: add command to test NIC restoration

-----Original Message-----
From: Dai, Wei
Sent: Tuesday, June 27, 2017 10:07 PM
Subject: [PATCH v2 5/5] app/testpmd: add command to test NIC
restoration
When a NIC is reset, a message will show it.
And then user can run the command "reset_port port_id"
to reset the port and to keep same port id without any without any
configuration.
This patch adds a testpmd command "reconfig_port port_id"
to test whether the port can be reconfigured to have success Rx and Tx

function.

The new command will configure the port with the simplest setting
which includes only 1 Rx queue, only 1 Tx queue, Rx mode = None and Tx
mode = None.
It check if the port can receive and forward some packets.
For example 100 packets in this new command.
Before testing with "reset_port port_id" and then "reconfig_port
port_id", current forwarding should be stopped first to avoid crash.

Testpmd has mechanism to support reconfigure, you can check the
need_reconfig Field in port struct.
No need to define a new command.
Thanks
Jingjing

[Peng, Yuan]

Tested-by: Peng, Yuan <***@intel.com>

- Tested commit c1923afc0999b5b6f109190bc5b69b6c3d334635+the 5 patches.
- OS: 4.5.5-300.fc24.x86_64
- GCC: gcc version 6.1.1 20160510 (Red Hat 6.1.1-2) (GCC)
- CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
- NIC: Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
- Default x86_64-native-linuxapp-gcc configuration
- Prerequisites:
- Total 1cases, 1 passed, 0 failed

Steps:
DUT:
1. run testpmd with ixgbe VF
echo 1 >/sys/bus/pci/devices/0000:07:00.0/sriov_numvfs
ip link set ens786f0 vf 0 mac 00:11:22:33:44:11
./usertools/dpdk-devbind.py -b vfio-pci 07:10.0
./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 -- -i

2. testpmd > set verbose 1
testpmd> set fwd rxonly
testpmd> start
tester:
scapy

pkt1 = Ether(dst="00:11:22:33:44:11")/IP()/UDP()/Raw('x' * 20)
sendp(pkt1, iface="ens786f1", count=1)

Vf can receive the packet correctly.

3. ifconfig ens786f0 down
A message is shown in testmd to indicate PF reset:
Port 0: Interrupt reset event

4. ifconfig ens786f0 up

5. testpmd > stop

6. testpmd > reset_port 0

7. testpmd> reconfig_port 0
PMD: ixgbevf_dev_configure(): VF can't disable HW CRC Strip
Port 0 MAC: 00 11 22 33 44 11
Begin to forward at least 100 packets to test reconfiguration

sendp(pkt1, iface="ens786f1", count=100)

9. Finish forwarding 100 packets to test reconfiguration
testpmd> show port stats all

######################## NIC statistics for port 0 ########################
RX-packets: 100 RX-missed: 0 RX-bytes: 6200
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0

Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
############################################################################

The vf can be reconfigured successfully after pf reset.

-----Original Message-----
From: dev [mailto:dev-***@dpdk.org] On Behalf Of Wei Dai
Sent: Tuesday, June 27, 2017 10:07 PM
To: ***@monjalon.net; Lu, Wenzhuo <***@intel.com>; Ananyev, Konstantin <***@intel.com>; Zhang, Helin <***@intel.com>; Wu, Jingjing <***@intel.com>
Cc: ***@dpdk.org; Dai, Wei <***@intel.com>
Subject: [dpdk-dev] [PATCH v2 0/5] Support NIC reset and keep same port id

Sometimes a port have to be reset. After reset, if the port goes through PCI remove() and then PCI probe() for restoration, its port id may be changed and this is not expected by some customer DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer and the other is calling PMD dev_init(). PCI remove( ) release all resource allocated from rte_ethdev layer in PCI probe( ) and calls PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and dev_init( ) in PMD can be called and keep all resources allocated from rte_ethdev layer poart in PCI probe( ).

New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( ) and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore its previous settings or to reconfigure itself with different settings.

To test this new feature, 2 testpmd commands are introduced.
The first command "reset_port port_id" calls rte_eth_dev_reset( ).
The second command "reconfig_port port_id" is used to test if the port can be reconfigured successfully. This command configure the port with the simplest settings include only 1 Rx queue, only 1 Tx queue, Rx mode = None and Tx mode = None. It check port by receving at least
100 packets and then forward these packets from itself.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to its VF.
1. run testpmd with ixgbe vF
2. testpmd > set verbose 1 //to observe VF working 3. ifconfig name-of-PF down 4. A message is shown in testmd to indicate PF reset 5. ifconfig name-of-PF up 6. testpmd > stop // stop forwarding to avoid crash during reset 7. testpmd > reset_port port_id_of_VF 8. testpmd > reconfig_port port_id_of_VF


Wei Dai (5):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: add command to test NIC reset
app/testpmd: add command to test NIC restoration

app/test-pmd/cmdline.c | 62 ++++++++++++++++++
app/test-pmd/config.c | 111 +++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 2 +
drivers/net/i40e/i40e_ethdev.c | 16 +++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 +++++
drivers/net/ixgbe/ixgbe_ethdev.c | 38 +++++++++++
lib/librte_ether/rte_ethdev.c | 17 +++++
lib/librte_ether/rte_ethdev.h | 12 ++++
lib/librte_ether/rte_ether_version.map | 2 +-
9 files changed, 275 insertions(+), 1 deletion(-)

--
2.7.4

[Dai, Wei]

Many thanks to Yuan for your test support.
Your test show the code changes for rte_ether and driver/net can work as expected.
I'll update a v3 patch set according to feedback from Wu Jingjing and your test.
In v3 patch, only code in testpmd will be changed but with similar test method.

-----Original Message-----
From: Peng, Yuan
Sent: Thursday, June 29, 2017 11:52 AM
Subject: RE: [dpdk-dev] [PATCH v2 0/5] Support NIC reset and keep same port id
- Tested commit c1923afc0999b5b6f109190bc5b69b6c3d334635+the 5
patches.
- OS: 4.5.5-300.fc24.x86_64
- GCC: gcc version 6.1.1 20160510 (Red Hat 6.1.1-2) (GCC)
- NIC: Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Network Connection [8086:10fb] (rev 01)
- Default x86_64-native-linuxapp-gcc configuration
- Total 1cases, 1 passed, 0 failed
1. run testpmd with ixgbe VF
echo 1 >/sys/bus/pci/devices/0000:07:00.0/sriov_numvfs
ip link set ens786f0 vf 0 mac 00:11:22:33:44:11 ./usertools/dpdk-devbind.py -b
vfio-pci 07:10.0 ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 -- -i
2. testpmd > set verbose 1
testpmd> set fwd rxonly
testpmd> start
scapy

pkt1 = Ether(dst="00:11:22:33:44:11")/IP()/UDP()/Raw('x' * 20)
sendp(pkt1, iface="ens786f1", count=1)

Vf can receive the packet correctly.
3. ifconfig ens786f0 down
Port 0: Interrupt reset event
4. ifconfig ens786f0 up
5. testpmd > stop
6. testpmd > reset_port 0
7. testpmd> reconfig_port 0
PMD: ixgbevf_dev_configure(): VF can't disable HW CRC Strip Port 0 MAC: 00
11 22 33 44 11 Begin to forward at least 100 packets to test reconfiguration

sendp(pkt1, iface="ens786f1", count=100)

9. Finish forwarding 100 packets to test reconfiguration
testpmd> show port stats all
######################## NIC statistics for port 0
########################
RX-packets: 100 RX-missed: 0 RX-bytes: 6200
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
################################################################
############
The vf can be reconfigured successfully after pf reset.
-----Original Message-----
Sent: Tuesday, June 27, 2017 10:07 PM
Subject: [dpdk-dev] [PATCH v2 0/5] Support NIC reset and keep same port id
Sometimes a port have to be reset. After reset, if the port goes through PCI
remove() and then PCI probe() for restoration, its port id may be changed and
this is not expected by some customer DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer and the
other is calling PMD dev_init(). PCI remove( ) release all resource allocated
from rte_ethdev layer in PCI probe( ) and calls PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and dev_init( ) in
PMD can be called and keep all resources allocated from rte_ethdev layer poart
in PCI probe( ).
New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( ) and
then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ),
rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and
rte_eth_dev_start( ) again to restore its previous settings or to reconfigure
itself with different settings.
To test this new feature, 2 testpmd commands are introduced.
The first command "reset_port port_id" calls rte_eth_dev_reset( ).
The second command "reconfig_port port_id" is used to test if the port can be
reconfigured successfully. This command configure the port with the simplest
settings include only 1 Rx queue, only 1 Tx queue, Rx mode = None and Tx mode
= None. It check port by receving at least
100 packets and then forward these packets from itself.
For example, run "ifconfig PF-name down" will trigger a reset to its VF.
1. run testpmd with ixgbe vF
2. testpmd > set verbose 1 //to observe VF working 3. ifconfig name-of-PF down
4. A message is shown in testmd to indicate PF reset 5. ifconfig name-of-PF up 6.
testpmd > stop // stop forwarding to avoid crash during reset 7. testpmd >
reset_port port_id_of_VF 8. testpmd > reconfig_port port_id_of_VF
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: add command to test NIC reset
app/testpmd: add command to test NIC restoration
app/test-pmd/cmdline.c | 62 ++++++++++++++++++
app/test-pmd/config.c | 111
+++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 2 +
drivers/net/i40e/i40e_ethdev.c | 16 +++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 +++++
drivers/net/ixgbe/ixgbe_ethdev.c | 38 +++++++++++
lib/librte_ether/rte_ethdev.c | 17 +++++
lib/librte_ether/rte_ethdev.h | 12 ++++
lib/librte_ether/rte_ether_version.map | 2 +-
9 files changed, 275 insertions(+), 1 deletion(-)
--
2.7.4

[Wei Dai]

Sometimes a port have to be reset. After reset, if the port goes
through PCI remove() and then PCI probe() for restoration, its
port id may be changed and this is not expected by some customer
DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer
and the other is calling PMD dev_init(). PCI remove( ) release all
resource allocated from rte_ethdev layer in PCI probe( ) and calls
PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ).

New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ),
rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and
rte_eth_dev_start( ) again to restore its previous settings or
to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to its VF.
1. run testpmd with ixgbe VF
2. testpmd > set verbose 1 //to observe VF working
3. ifconfig name-of-PF down
4. A message is shown in testmd to indicate PF reset
5. ifconfig name-of-PF up
6. testpmd > stop // stop forwarding to avoid crash during reset
7. testpmd > port reset port_id_of_VF
8. testpmd > start // restore forwarding

chagnes:
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings

Wei Dai (4):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset

app/test-pmd/cmdline.c | 10 ++++++---
app/test-pmd/testpmd.c | 39 ++++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
drivers/net/i40e/i40e_ethdev.c | 16 ++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 ++++++++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 38 +++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 16 ++++++++++++++
lib/librte_ether/rte_ethdev.h | 12 +++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
9 files changed, 146 insertions(+), 4 deletions(-)

[Wei Dai]

A DPDK application can reset a NIC and keep its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset
in similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 16 ++++++++++++++++
lib/librte_ether/rte_ethdev.h | 12 ++++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
3 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 81a45c0..06b8839 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -3349,3 +3349,19 @@ rte_eth_dev_l2_tunnel_offload_set(uint8_t port_id,
-ENOTSUP);
return (*dev->dev_ops->l2_tunnel_offload_set)(dev, l2_tunnel, mask, en);
}
+
+int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+ return ret;
+}
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index f6e6c74..4922b36 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1087,6 +1087,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1404,6 +1407,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2104,6 +2108,14 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Reset a Ethernet device.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index e0881f0..d569bcb 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -153,5 +153,5 @@ DPDK_17.08 {
global:

rte_flow_isolate;
-
+ rte_eth_dev_reset;
} DPDK_17.05;

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index ebc5467..5ff8fbe 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -170,6 +170,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -264,6 +265,7 @@ static int ixgbevf_dev_configure(struct rte_eth_dev *dev);
static int ixgbevf_dev_start(struct rte_eth_dev *dev);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -525,6 +527,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -614,6 +617,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2838,6 +2842,23 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -4902,6 +4923,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 16 ++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 ++++++++++++++++
2 files changed, 32 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 4ee1113..7694957 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2135,6 +2137,20 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 2d5a9b5..bf287a0 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -123,6 +123,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -204,6 +205,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2347,6 +2349,20 @@ i40evf_dev_close(struct rte_eth_dev *dev)
}

static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
+static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{
struct i40e_vf *vf = I40E_VSI_TO_VF(vsi);

[Wei Dai]

When PF is reset, a message will show it.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without any
without any configuration.
And then user can run "start port_id" to reconfigure
its forwarding mode and parmaters as previous.
To avoid crash, current forwarding should be stopped.

Signed-off-by: Wei Dai <***@intel.com>
---
app/test-pmd/cmdline.c | 10 +++++++---
app/test-pmd/testpmd.c | 39 +++++++++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
3 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index ff8ffd2..58ba6e4 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -950,6 +950,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -959,7 +961,7 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

@@ -994,6 +996,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1003,7 +1007,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1011,7 +1015,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close|reset <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index b29328a..77a517f 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1665,6 +1665,45 @@ close_port(portid_t pid)
}

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Reseting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n", pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ }
+ else
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 364502d..e4c704a 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -596,6 +596,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode,
int start_port(portid_t pid);
void stop_port(portid_t pid);
void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);

[Wei Dai]

Sometimes a port have to be reset. For example PF is reset, all its
VF should also be reset. After reset, if the port goes
through PCI remove() and then PCI probe() for restoration, its
port id may be changed and this is not expected by some customer
DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer
and the other is calling PMD dev_init(). PCI remove( ) release all
resource allocated from rte_ethdev layer in PCI probe( ) and calls
PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ).

New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ),
rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and
rte_eth_dev_start( ) again to restore its previous settings or
to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to
confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF
2. testpmd > set verbose 1 //to observe VF working
3. testpmd > show port info all //show port number, PCI addr and MAC addr
4. testpmd > start
5. let all ports forwarding work for a while
6. testpmd > show port stats all
7. ifconfig name-of-PF down
8. A message is shown in testmd to indicate PF reset
9. ifconfig name-of-PF up
10. testpmd > stop // stop forwarding to avoid crash during reset
11. testpmd > port reset all
12. testpmd > port stop all
13. testpmd > port start all //recofnig all ports
14. testpmd > show port info all
//confirm same mapping of port id and PCI addr
15. testpmd > start // restore forwarding
14. let all ports forwarding work for a while
15. testpmd > show port stats all //confirm all port can work again
16. repeat above step 7 - 15

chagnes:
v4:
add PCI address to confirm its port number keep same
correct test method in cover letter
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings


Wei Dai (5):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: display PCI address in port info
app/testpmd: enhance command to test NIC reset

app/test-pmd/cmdline.c | 10 ++++--
app/test-pmd/config.c | 5 +++
app/test-pmd/testpmd.c | 61 ++++++++++++++++++++++++++++++++--
app/test-pmd/testpmd.h | 1 +
drivers/net/i40e/i40e_ethdev.c | 16 +++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 +++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 38 +++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 16 +++++++++
lib/librte_ether/rte_ethdev.h | 12 +++++++
lib/librte_ether/rte_ether_version.map | 2 +-
10 files changed, 170 insertions(+), 7 deletions(-)

[Wei Dai]

A DPDK application can reset a NIC and keep its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset
in similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 16 ++++++++++++++++
lib/librte_ether/rte_ethdev.h | 12 ++++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
3 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 81a45c0..06b8839 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -3349,3 +3349,19 @@ rte_eth_dev_l2_tunnel_offload_set(uint8_t port_id,
-ENOTSUP);
return (*dev->dev_ops->l2_tunnel_offload_set)(dev, l2_tunnel, mask, en);
}
+
+int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+ return ret;
+}
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index f6e6c74..4922b36 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1087,6 +1087,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1404,6 +1407,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2104,6 +2108,14 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Reset a Ethernet device.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index e0881f0..d569bcb 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -153,5 +153,5 @@ DPDK_17.08 {
global:

rte_flow_isolate;
-
+ rte_eth_dev_reset;
} DPDK_17.05;

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index ebc5467..5ff8fbe 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -170,6 +170,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -264,6 +265,7 @@ static int ixgbevf_dev_configure(struct rte_eth_dev *dev);
static int ixgbevf_dev_start(struct rte_eth_dev *dev);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -525,6 +527,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -614,6 +617,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2838,6 +2842,23 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -4902,6 +4923,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 16 ++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 ++++++++++++++++
2 files changed, 32 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 4ee1113..7694957 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2135,6 +2137,20 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 2d5a9b5..bf287a0 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -123,6 +123,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -204,6 +205,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2347,6 +2349,20 @@ i40evf_dev_close(struct rte_eth_dev *dev)
}

static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
+static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{
struct i40e_vf *vf = I40E_VSI_TO_VF(vsi);

[Wu, Jingjing]

Post by Wei Dai
}
+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;

Have you thought about if DPDK is working as host driver, and there
Are VFs rely on it? If it doesn't support, at least a check should be added.
Or you can add those check in Rte level.

Thanks
Jingjing

[Dai, Wei]

-----Original Message-----
From: Wu, Jingjing
Sent: Friday, June 30, 2017 4:54 PM
Subject: RE: [PATCH v4 3/5] net/i40e: add support of reset

Post by Wei Dai
}
+static int
+i40e_dev_reset(struct rte_eth_dev *dev) {
+ int ret;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;

Have you thought about if DPDK is working as host driver, and there Are VFs
rely on it? If it doesn't support, at least a check should be added.
Or you can add those check in Rte level.

If there is any VF, PF reset need sync with VF, but sync method may be much different with different NIC type.
So, we'd like first reject PF reset if its VF exist.
I'll update it in v5 patch set.

Thanks
Jingjing

[Wei Dai]

Add the PCI address when running "show port info port_id".

Signed-off-by: Wei Dai <***@intel.com>
---
app/test-pmd/config.c | 5 +++++
1 file changed, 5 insertions(+)

diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index b0b340e..c2d5c03 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -438,6 +438,11 @@ port_infos_display(portid_t port_id)
rte_eth_dev_info_get(port_id, &dev_info);
printf("\n%s Infos for port %-2d %s\n",
info_border, port_id, info_border);
+ printf("PCI Address: " PCI_PRI_FMT "\n",
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);
rte_eth_macaddr_get(port_id, &mac_addr);
print_ethaddr("MAC address: ", &mac_addr);
printf("\nDriver name: %s", dev_info.driver_name);

[Wu, Jingjing]

-----Original Message-----
From: Dai, Wei
Sent: Thursday, June 29, 2017 10:58 PM
Subject: [PATCH v4 4/5] app/testpmd: display PCI address in port info
Add the PCI address when running "show port info port_id".
---
app/test-pmd/config.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index b0b340e..c2d5c03 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -438,6 +438,11 @@ port_infos_display(portid_t port_id)
rte_eth_dev_info_get(port_id, &dev_info);
printf("\n%s Infos for port %-2d %s\n",
info_border, port_id, info_border);
+ printf("PCI Address: " PCI_PRI_FMT "\n",
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);
rte_eth_macaddr_get(port_id, &mac_addr);
print_ethaddr("MAC address: ", &mac_addr);
printf("\nDriver name: %s", dev_info.driver_name);
--
2.7.4

The field pci_dev will probably be removed.
Yulong already submit a patch like this and rejected.
http://www.dpdk.org/dev/patchwork/patch/23899/

So, you can add more info when the Bus refine work is done.
You can drop this change from your patch set.

Thanks
Jingjing

[Wei Dai]

When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".

Signed-off-by: Wei Dai <***@intel.com>
---
app/test-pmd/cmdline.c | 10 ++++++---
app/test-pmd/testpmd.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++---
app/test-pmd/testpmd.h | 1 +
3 files changed, 66 insertions(+), 6 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index ff8ffd2..58ba6e4 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -950,6 +950,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -959,7 +961,7 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

@@ -994,6 +996,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1003,7 +1007,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1011,7 +1015,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close|reset <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index b29328a..7773879 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1403,6 +1403,7 @@ start_port(portid_t pid)
queueid_t qi;
struct rte_port *port;
struct ether_addr mac_addr;
+ struct rte_eth_dev_info dev_info;
enum rte_eth_event_type event_type;

if (port_id_is_invalid(pid, ENABLED_WARN))
@@ -1424,9 +1425,14 @@ start_port(portid_t pid)

if (port->need_reconfig > 0) {
port->need_reconfig = 0;
-
- printf("Configuring Port %d (socket %u)\n", pi,
- port->socket_id);
+ rte_eth_dev_info_get(pi, &dev_info);
+ printf("Configuring Port %d (socket %u) with "
+ "PCI Address: " PCI_PRI_FMT "\n",
+ pi, port->socket_id,
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);
/* configure port */
diag = rte_eth_dev_configure(pi, nb_rxq, nb_txq,
&(port->dev_conf));
@@ -1665,6 +1671,55 @@ close_port(portid_t pid)
}

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+ struct rte_eth_dev_info dev_info;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Resetting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding "
+ "configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n",
+ pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ rte_eth_dev_info_get(pi, &dev_info);
+ printf("Finish resetting Port %d with PCI Address: "
+ PCI_PRI_FMT "\n", pi,
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);
+ } else {
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 364502d..e4c704a 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -596,6 +596,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode,
int start_port(portid_t pid);
void stop_port(portid_t pid);
void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);

[Peng, Yuan]

Tested-by: Peng, Yuan <***@intel.com>

- Tested commit bbe569daa7e99b36d44b12bb3d23ddfbc26d383c+the 5 patches.
- OS: 4.8.6-300.fc25.x86_64
- GCC: gcc version 6.2.1 20160916 (Red Hat 6.2.1-2) (GCC)
- CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
- NIC: Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)
- Default x86_64-native-linuxapp-gcc configuration
- Prerequisites:
- Total 1cases, 1 passed, 0 failed

Steps:
DUT:
1. echo 2 >/sys/bus/pci/devices/0000:83:00.0/sriov_numvfs
./usertools/dpdk-devbind.py -b vfio-pci 83:10.0 83:10.2 ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 -- -i

2.testpmd> set verbose 1
testpmd> set fwd mac
testpmd> start
tester:
scapy
sendp([Ether(dst="02:09:C0:63:DA:4B")/IP()/UDP()], iface="ens6f0", count=1)
dut:
testpmd>show port stats all
port0 can fwd the packet normally
testpmd>show port info all
show port number, PCI addr and MAC addr

3. ifconfig ens801f0 down
Port 0: Interrupt reset event
Port 1: Interrupt reset event
ifconfig ens801f0 up

4.testpmd> stop

5. testpmd> port reset all
Resetting ports...
Finish resetting Port 0 with PCI Address: 0000:83:10.0 Finish resetting Port 1 with PCI Address: 0000:83:10.2 Done

6. testpmd> port stop all
Stopping ports...
Checking link statuses...
Done
testpmd> port start all
Configuring Port 0 (socket 1) with PCI Address: 0000:83:10.0 Port 0: 02:09:C0:63:DA:4B Configuring Port 1 (socket 1) with PCI Address: 0000:83:10.2 Port 1: 02:09:C0:37:93:6F Checking link statuses...
Done

7. testpmd> show port info all
confirm same mapping of port id and PCI address.

8.testpmd> start
Tester:
scapy
sendp([Ether(dst="02:09:C0:63:DA:4B")/IP()/UDP()], iface="ens6f0", count=1)
dut:
testpmd>show port stats all
port0 can fwd the packet normally

9.repeat step3 to step8, the same result.

-----Original Message-----
From: dev [mailto:dev-***@dpdk.org] On Behalf Of Wei Dai
Sent: Thursday, June 29, 2017 10:58 PM
To: ***@monjalon.net; Lu, Wenzhuo <***@intel.com>; Ananyev, Konstantin <***@intel.com>; Zhang, Helin <***@intel.com>; Wu, Jingjing <***@intel.com>; ***@dpdk.org
Cc: ***@dpdk.org; Dai, Wei <***@intel.com>
Subject: [dpdk-dev] [PATCH v4 5/5] app/testpmd: enhance command to test NIC reset

When PF is reset, a message will show it and all its VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped before running "port reset port_id".

Signed-off-by: Wei Dai <***@intel.com>
---
app/test-pmd/cmdline.c | 10 ++++++---
app/test-pmd/testpmd.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++---
app/test-pmd/testpmd.h | 1 +
3 files changed, 66 insertions(+), 6 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index ff8ffd2..58ba6e4 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -950,6 +950,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -959,7 +961,7 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

@@ -994,6 +996,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1003,7 +1007,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1011,7 +1015,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id = cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close|reset <port_id>:
+Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index b29328a..7773879 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1403,6 +1403,7 @@ start_port(portid_t pid)
queueid_t qi;
struct rte_port *port;
struct ether_addr mac_addr;
+ struct rte_eth_dev_info dev_info;
enum rte_eth_event_type event_type;

if (port_id_is_invalid(pid, ENABLED_WARN)) @@ -1424,9 +1425,14 @@ start_port(portid_t pid)

if (port->need_reconfig > 0) {
port->need_reconfig = 0;
-
- printf("Configuring Port %d (socket %u)\n", pi,
- port->socket_id);
+ rte_eth_dev_info_get(pi, &dev_info);
+ printf("Configuring Port %d (socket %u) with "
+ "PCI Address: " PCI_PRI_FMT "\n",
+ pi, port->socket_id,
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);
/* configure port */
diag = rte_eth_dev_configure(pi, nb_rxq, nb_txq,
&(port->dev_conf));
@@ -1665,6 +1671,55 @@ close_port(portid_t pid) }

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+ struct rte_eth_dev_info dev_info;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Resetting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding "
+ "configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n",
+ pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ rte_eth_dev_info_get(pi, &dev_info);
+ printf("Finish resetting Port %d with PCI Address: "
+ PCI_PRI_FMT "\n", pi,
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);
+ } else {
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index 364502d..e4c704a 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -596,6 +596,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode, int start_port(portid_t pid); void stop_port(portid_t pid); void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);
--
2.7.4

[Wu, Jingjing]

-----Original Message-----
From: Dai, Wei
Sent: Thursday, June 29, 2017 10:58 PM
Subject: [PATCH v4 5/5] app/testpmd: enhance command to test NIC reset
When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".

Acked-by: Jingjing Wu <***@intel.com>

[Wu, Jingjing]

-----Original Message-----
From: Dai, Wei
Sent: Thursday, June 29, 2017 10:58 PM
Subject: [PATCH v4 5/5] app/testpmd: enhance command to test NIC reset
When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".
---
app/test-pmd/cmdline.c | 10 ++++++---
app/test-pmd/testpmd.c | 61
+++++++++++++++++++++++++++++++++++++++++++++++---
app/test-pmd/testpmd.h | 1 +
3 files changed, 66 insertions(+), 6 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index ff8ffd2..58ba6e4 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -950,6 +950,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -959,7 +961,7 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");
@@ -994,6 +996,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1003,7 +1007,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1011,7 +1015,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close|reset <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index b29328a..7773879 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1403,6 +1403,7 @@ start_port(portid_t pid)
queueid_t qi;
struct rte_port *port;
struct ether_addr mac_addr;
+ struct rte_eth_dev_info dev_info;
enum rte_eth_event_type event_type;
if (port_id_is_invalid(pid, ENABLED_WARN))
@@ -1424,9 +1425,14 @@ start_port(portid_t pid)
if (port->need_reconfig > 0) {
port->need_reconfig = 0;
-
- printf("Configuring Port %d (socket %u)\n", pi,
- port->socket_id);
+ rte_eth_dev_info_get(pi, &dev_info);
+ printf("Configuring Port %d (socket %u) with "
+ "PCI Address: " PCI_PRI_FMT "\n",
+ pi, port->socket_id,
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);

I'm OK with the command change, but could you remove the
PCI print from here. The same reason as your patch 4/5.

Thanks
Jingjing

[Dai, Wei]

Adding PCI address info here is to let user confirm mapping between PCI address and port id is kept after reset.
As this is only for test purpose, are you OK for other part except these PCI address printf ?

-----Original Message-----
From: Wu, Jingjing
Sent: Friday, June 30, 2017 5:10 PM
Subject: RE: [PATCH v4 5/5] app/testpmd: enhance command to test NIC reset

-----Original Message-----
From: Dai, Wei
Sent: Thursday, June 29, 2017 10:58 PM
yuan.pntel.com
Subject: [PATCH v4 5/5] app/testpmd: enhance command to test NIC reset
When PF is reset, a message will show it and all its VF need to be
reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without any
configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding mode and
parmaters as previous ones.
To avoid crash, current forwarding should be stopped before running
"port reset port_id".
---
app/test-pmd/cmdline.c | 10 ++++++--- app/test-pmd/testpmd.c | 61
+++++++++++++++++++++++++++++++++++++++++++++++---
app/test-pmd/testpmd.h | 1 +
3 files changed, 66 insertions(+), 6 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
ff8ffd2..58ba6e4 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -950,6 +950,8 @@ static void cmd_operate_port_parsed(void

*parsed_result,

stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -959,7 +961,7 @@ cmdline_parse_token_string_t

cmd_operate_port_all_cmd =

"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");
@@ -994,6 +996,8 @@ static void cmd_operate_specific_port_parsed(void

*parsed_result,

stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1003,7 +1007,7 @@ cmdline_parse_token_string_t

cmd_operate_specific_port_cmd =

keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1011,7 +1015,7 @@ cmdline_parse_token_num_t
cmd_operate_specific_port_id = cmdline_parse_inst_t

cmd_operate_specific_port = {

.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index
b29328a..7773879 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1403,6 +1403,7 @@ start_port(portid_t pid)
queueid_t qi;
struct rte_port *port;
struct ether_addr mac_addr;
+ struct rte_eth_dev_info dev_info;
enum rte_eth_event_type event_type;
start_port(portid_t pid)
if (port->need_reconfig > 0) {
port->need_reconfig = 0;
-
- printf("Configuring Port %d (socket %u)\n", pi,
- port->socket_id);
+ rte_eth_dev_info_get(pi, &dev_info);
+ printf("Configuring Port %d (socket %u) with "
+ "PCI Address: " PCI_PRI_FMT "\n",
+ pi, port->socket_id,
+ dev_info.pci_dev->addr.domain,
+ dev_info.pci_dev->addr.bus,
+ dev_info.pci_dev->addr.devid,
+ dev_info.pci_dev->addr.function);

I'm OK with the command change, but could you remove the PCI print from
here. The same reason as your patch 4/5.
Thanks
Jingjing

[Peng, Yuan]

Tested-by: Peng, Yuan <***@intel.com>

- Tested commit bbe569daa7e99b36d44b12bb3d23ddfbc26d383c+the 5 patches.
- OS: 4.8.6-300.fc25.x86_64
- GCC: gcc version 6.2.1 20160916 (Red Hat 6.2.1-2) (GCC)
- CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
- NIC: Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)
- Default x86_64-native-linuxapp-gcc configuration
- Prerequisites:
- Total 1cases, 1 passed, 0 failed

Steps:
DUT:
1. echo 2 >/sys/bus/pci/devices/0000:83:00.0/sriov_numvfs
./usertools/dpdk-devbind.py -b vfio-pci 83:10.0 83:10.2
./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 -- -i

2.testpmd> set verbose 1
testpmd> set fwd mac
testpmd> start
tester:
scapy
sendp([Ether(dst="02:09:C0:63:DA:4B")/IP()/UDP()], iface="ens6f0", count=1)
dut:
testpmd>show port stats all
port0 can fwd the packet normally
testpmd>show port info all
show port number, PCI addr and MAC addr

3. ifconfig ens801f0 down
Port 0: Interrupt reset event
Port 1: Interrupt reset event
ifconfig ens801f0 up

4.testpmd> stop

5. testpmd> port reset all
Resetting ports...
Finish resetting Port 0 with PCI Address: 0000:83:10.0
Finish resetting Port 1 with PCI Address: 0000:83:10.2
Done

6. testpmd> port stop all
Stopping ports...
Checking link statuses...
Done
testpmd> port start all
Configuring Port 0 (socket 1) with PCI Address: 0000:83:10.0
Port 0: 02:09:C0:63:DA:4B
Configuring Port 1 (socket 1) with PCI Address: 0000:83:10.2
Port 1: 02:09:C0:37:93:6F
Checking link statuses...
Done

7. testpmd> show port info all
confirm same mapping of port id and PCI address.

8.testpmd> start
Tester:
scapy
sendp([Ether(dst="02:09:C0:63:DA:4B")/IP()/UDP()], iface="ens6f0", count=1)
dut:
testpmd>show port stats all
port0 can fwd the packet normally

9.repeat step3 to step8, the same result.



-----Original Message-----
From: dev [mailto:dev-***@dpdk.org] On Behalf Of Wei Dai
Sent: Thursday, June 29, 2017 10:58 PM
To: ***@monjalon.net; Lu, Wenzhuo <***@intel.com>; Ananyev, Konstantin <***@intel.com>; Zhang, Helin <***@intel.com>; Wu, Jingjing <***@intel.com>; ***@dpdk.org
Cc: ***@dpdk.org; Dai, Wei <***@intel.com>
Subject: [dpdk-dev] [PATCH v4 0/5] Support NIC reset and keep same port id

Sometimes a port have to be reset. For example PF is reset, all its VF should also be reset. After reset, if the port goes through PCI remove() and then PCI probe() for restoration, its port id may be changed and this is not expected by some customer DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer and the other is calling PMD dev_init(). PCI remove( ) release all resource allocated from rte_ethdev layer in PCI probe( ) and calls PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and dev_init( ) in PMD can be called and keep all resources allocated from rte_ethdev layer poart in PCI probe( ).

New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( ) and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore its previous settings or to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to confirm its port number is kept.
And following test case can also be used to confirm the port can work again after reset.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF 2. testpmd > set verbose 1 //to observe VF working 3. testpmd > show port info all //show port number, PCI addr and MAC addr 4. testpmd > start 5. let all ports forwarding work for a while 6. testpmd > show port stats all 7. ifconfig name-of-PF down 8. A message is shown in testmd to indicate PF reset 9. ifconfig name-of-PF up 10. testpmd > stop // stop forwarding to avoid crash during reset 11. testpmd > port reset all 12. testpmd > port stop all 13. testpmd > port start all //recofnig all ports 14. testpmd > show port info all
//confirm same mapping of port id and PCI addr 15. testpmd > start // restore forwarding 14. let all ports forwarding work for a while 15. testpmd > show port stats all //confirm all port can work again 16. repeat above step 7 - 15

chagnes:
v4:
add PCI address to confirm its port number keep same
correct test method in cover letter
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings


Wei Dai (5):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: display PCI address in port info
app/testpmd: enhance command to test NIC reset

app/test-pmd/cmdline.c | 10 ++++--
app/test-pmd/config.c | 5 +++
app/test-pmd/testpmd.c | 61 ++++++++++++++++++++++++++++++++--
app/test-pmd/testpmd.h | 1 +
drivers/net/i40e/i40e_ethdev.c | 16 +++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 +++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 38 +++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 16 +++++++++
lib/librte_ether/rte_ethdev.h | 12 +++++++
lib/librte_ether/rte_ether_version.map | 2 +-
10 files changed, 170 insertions(+), 7 deletions(-)

--
2.7.4

[Wei Dai]

Sometimes a port have to be reset. For example PF is reset, all its
VF should also be reset. After reset, if the port goes
through PCI remove() and then PCI probe() for restoration, its
port id may be changed and this is not expected by some customer
DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer
and the other is calling PMD dev_init(). PCI remove( ) release all
resource allocated from rte_ethdev layer in PCI probe( ) and calls
PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ).

New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ),
rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and
rte_eth_dev_start( ) again to restore its previous settings or
to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to
confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF
2. testpmd > set verbose 1 //to observe VF working
3. testpmd > show port info all //show port number and MAC addr
4. testpmd > start
5. let all ports forwarding work for a while
6. testpmd > show port stats all
7. ifconfig name-of-PF down
8. A message is shown in testmd to indicate PF reset
9. ifconfig name-of-PF up
10. testpmd > stop // stop forwarding to avoid crash during reset
11. testpmd > port reset all
12. testpmd > port stop all
13. testpmd > port start all //recofnig all ports
14. testpmd > show port info all
//get mapping of port id and MAC addr for forwarding
15. testpmd > start // restore forwarding
14. let all ports forwarding work for a while
15. testpmd > show port stats all //confirm all port can work again
16. repeat above step 7 - 15

chagnes:
v5:
remove PCI address output to align with other modification which
will output it in other way
disable PF reset when its VF is ative to avoid unexpected VF behavior
v4:
add PCI address to confirm its port number keep same
correct test method in cover letter
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>

Wei Dai (4):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset

app/test-pmd/cmdline.c | 10 +++++---
app/test-pmd/testpmd.c | 44 +++++++++++++++++++++++++++++++---
app/test-pmd/testpmd.h | 1 +
drivers/net/i40e/i40e_ethdev.c | 20 ++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 +++++++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 42 ++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 16 +++++++++++++
lib/librte_ether/rte_ethdev.h | 12 ++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
9 files changed, 156 insertions(+), 7 deletions(-)

[Wei Dai]

A DPDK application can reset a NIC and keep its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset
in similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 16 ++++++++++++++++
lib/librte_ether/rte_ethdev.h | 12 ++++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
3 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 81a45c0..06b8839 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -3349,3 +3349,19 @@ rte_eth_dev_l2_tunnel_offload_set(uint8_t port_id,
-ENOTSUP);
return (*dev->dev_ops->l2_tunnel_offload_set)(dev, l2_tunnel, mask, en);
}
+
+int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+ return ret;
+}
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index f6e6c74..4922b36 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1087,6 +1087,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1404,6 +1407,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2104,6 +2108,14 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Reset a Ethernet device.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index e0881f0..d569bcb 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -153,5 +153,5 @@ DPDK_17.08 {
global:

rte_flow_isolate;
-
+ rte_eth_dev_reset;
} DPDK_17.05;

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 42 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index ebc5467..e2f1da0 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -170,6 +170,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -264,6 +265,7 @@ static int ixgbevf_dev_configure(struct rte_eth_dev *dev);
static int ixgbevf_dev_start(struct rte_eth_dev *dev);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -525,6 +527,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -614,6 +617,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2838,6 +2842,27 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* To avoid unexpected behavior in VF, disable PF reset */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -4902,6 +4927,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Thomas Monjalon]

Hi,

Post by Wei Dai
+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* To avoid unexpected behavior in VF, disable PF reset */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}

rte_eth_dev_reset() just do
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);

and dev_reset() just do
+ ret = eth_ixgbe_dev_uninit(dev);
+ ret = eth_ixgbe_dev_init(dev);

It is doing one more thing, the check of SR-IOV.
Unfortunately, this restriction is not documented.

This is the documentation of the new API:

/**
+ * Reset a Ethernet device.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);

It is really really too short.
From the beginning of this proposal we are asking you to better explain
why this API is needed. It still does not appear in the doc.
Are you adding it to offer a new service to DPDK application developpers?
Or is it just a secret sauce that you will explain only to your customers?

This is what is expected to be documented:
- why/when this API must be used
- what the API will do
- what is needed to do after

[Thomas Monjalon]

Post by Thomas Monjalon
Hi,

Post by Wei Dai
+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* To avoid unexpected behavior in VF, disable PF reset */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}

rte_eth_dev_reset() just do
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
and dev_reset() just do
+ ret = eth_ixgbe_dev_uninit(dev);
+ ret = eth_ixgbe_dev_init(dev);
It is doing one more thing, the check of SR-IOV.
Unfortunately, this restriction is not documented.
/**
+ * Reset a Ethernet device.
+ *
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
It is really really too short.
From the beginning of this proposal we are asking you to better explain
why this API is needed. It still does not appear in the doc.
Are you adding it to offer a new service to DPDK application developpers?
Or is it just a secret sauce that you will explain only to your customers?
- why/when this API must be used
- what the API will do
- what is needed to do after

I would like to add that the description of the API must also help
other PMD maintainers to implement it.
Adding a new op means more work for PMD maintainers, that's why
they should understand the benefit and acknowledge it.
Ferruh, as the maintainer of next-net, please could you ask for feedbacks
from other PMD maintainers?

[Dai, Wei]

Thanks, Thomas
I have just sent out my v6 patch set which includes more details to explain why/what/when.

-----Original Message-----
From: Thomas Monjalon [mailto:***@monjalon.net]
Sent: Friday, July 7, 2017 4:36 PM
To: Dai, Wei <***@intel.com>; Yigit, Ferruh <***@intel.com>
Cc: ***@dpdk.org; Lu, Wenzhuo <***@intel.com>; Ananyev, Konstantin <***@intel.com>; Wu, Jingjing <***@intel.com>; Zhang, Helin <***@intel.com>; Peng, Yuan <***@intel.com>
Subject: Re: [dpdk-dev] [PATCH v5 2/4] net/ixgbe: add support of reset

Post by Thomas Monjalon
Hi,

Post by Wei Dai
+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev) {
+ int ret;
+
+ /* To avoid unexpected behavior in VF, disable PF reset */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}

rte_eth_dev_reset() just do
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
and dev_reset() just do
+ ret = eth_ixgbe_dev_uninit(dev);
+ ret = eth_ixgbe_dev_init(dev);
It is doing one more thing, the check of SR-IOV.
Unfortunately, this restriction is not documented.
/**
+ * Reset a Ethernet device.
+ *
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
It is really really too short.
From the beginning of this proposal we are asking you to better
explain why this API is needed. It still does not appear in the doc.
Are you adding it to offer a new service to DPDK application developpers?
Or is it just a secret sauce that you will explain only to your customers?
- why/when this API must be used
- what the API will do
- what is needed to do after

I would like to add that the description of the API must also help other PMD maintainers to implement it.
Adding a new op means more work for PMD maintainers, that's why they should understand the benefit and acknowledge it.
Ferruh, as the maintainer of next-net, please could you ask for feedbacks from other PMD maintainers?

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 20 ++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 16 ++++++++++++++++
2 files changed, 36 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 4ee1113..261a5d0 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2135,6 +2137,24 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* To avoid unexpected behavior in VF, disable PF reset */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 2d5a9b5..bf287a0 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -123,6 +123,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -204,6 +205,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2347,6 +2349,20 @@ i40evf_dev_close(struct rte_eth_dev *dev)
}

static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
+static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{
struct i40e_vf *vf = I40E_VSI_TO_VF(vsi);

[Wei Dai]

When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>
---
app/test-pmd/cmdline.c | 10 +++++++---
app/test-pmd/testpmd.c | 44 +++++++++++++++++++++++++++++++++++++++++---
app/test-pmd/testpmd.h | 1 +
3 files changed, 49 insertions(+), 6 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index ff8ffd2..58ba6e4 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -950,6 +950,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -959,7 +961,7 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

@@ -994,6 +996,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1003,7 +1007,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1011,7 +1015,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close|reset <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index b29328a..17eaa31 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1424,9 +1424,6 @@ start_port(portid_t pid)

if (port->need_reconfig > 0) {
port->need_reconfig = 0;
-
- printf("Configuring Port %d (socket %u)\n", pi,
- port->socket_id);
/* configure port */
diag = rte_eth_dev_configure(pi, nb_rxq, nb_txq,
&(port->dev_conf));
@@ -1665,6 +1662,47 @@ close_port(portid_t pid)
}

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Resetting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding "
+ "configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n",
+ pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ } else {
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 364502d..e4c704a 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -596,6 +596,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode,
int start_port(portid_t pid);
void stop_port(portid_t pid);
void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);

[Wu, Jingjing]

-----Original Message-----
From: Dai, Wei
Sent: Friday, June 30, 2017 6:13 PM
Subject: [PATCH v5 0/4] Support NIC reset and keep same port id
Sometimes a port have to be reset. For example PF is reset, all its
VF should also be reset. After reset, if the port goes
through PCI remove() and then PCI probe() for restoration, its
port id may be changed and this is not expected by some customer
DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer
and the other is calling PMD dev_init(). PCI remove( ) release all
resource allocated from rte_ethdev layer in PCI probe( ) and calls
PMD dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ).
New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ),
rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and
rte_eth_dev_start( ) again to restore its previous settings or
to reconfigure itself with different settings.
To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to
confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF
2. testpmd > set verbose 1 //to observe VF working
3. testpmd > show port info all //show port number and MAC addr
4. testpmd > start
5. let all ports forwarding work for a while
6. testpmd > show port stats all
7. ifconfig name-of-PF down
8. A message is shown in testmd to indicate PF reset
9. ifconfig name-of-PF up
10. testpmd > stop // stop forwarding to avoid crash during reset
11. testpmd > port reset all
12. testpmd > port stop all
13. testpmd > port start all //recofnig all ports
14. testpmd > show port info all
//get mapping of port id and MAC addr for forwarding
15. testpmd > start // restore forwarding
14. let all ports forwarding work for a while
15. testpmd > show port stats all //confirm all port can work again
16. repeat above step 7 - 15
remove PCI address output to align with other modification which
will output it in other way
disable PF reset when its VF is ative to avoid unexpected VF behavior
add PCI address to confirm its port number keep same
correct test method in cover letter
update testpmd command
only reset PMD layer resource and keep same port id, but
not restore settings

Acked-by: Jingjing Wu <***@intel.com>

[Dai, Wei]

Hi, Thomas

Would you please have a look at this patch set ?
If any question, please let me know .

Thanks
-Wei


-----Original Message-----
From: Wu, Jingjing
Sent: Friday, June 30, 2017 6:50 PM
To: Dai, Wei <***@intel.com>; ***@monjalon.net; Lu, Wenzhuo <***@intel.com>; Ananyev, Konstantin <***@intel.com>; Zhang, Helin <***@intel.com>; Peng, Yuan <***@intel.com>
Cc: ***@dpdk.org
Subject: RE: [PATCH v5 0/4] Support NIC reset and keep same port id

-----Original Message-----
From: Dai, Wei
Sent: Friday, June 30, 2017 6:13 PM
Subject: [PATCH v5 0/4] Support NIC reset and keep same port id
Sometimes a port have to be reset. For example PF is reset, all its VF
should also be reset. After reset, if the port goes through PCI
remove() and then PCI probe() for restoration, its port id may be
changed and this is not expected by some customer DPDK application.
Normally, PCI probe() includes two parts: one is in rte_ethdev layer
and the other is calling PMD dev_init(). PCI remove( ) release all
resource allocated from rte_ethdev layer in PCI probe( ) and calls PMD
dev_unit( ).
To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated from
rte_ethdev layer poart in PCI probe( ).
New rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.
And then application can go through rte_eth_dev_configure( ),
rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ) and
rte_eth_dev_start( ) again to restore its previous settings or to
reconfigure itself with different settings.
To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored
to confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF 2. testpmd

Post by Peng, Yuan
set verbose 1 //to observe VF working 3. testpmd > show port info

all //show port number and MAC addr 4. testpmd > start 5. let all
ports forwarding work for a while 6. testpmd > show port stats all 7.
ifconfig name-of-PF down 8. A message is shown in testmd to indicate
PF reset 9. ifconfig name-of-PF up 10. testpmd > stop // stop
forwarding to avoid crash during reset 11. testpmd > port reset all
12. testpmd > port stop all 13. testpmd > port start all //recofnig
all ports 14. testpmd > show port info all
//get mapping of port id and MAC addr for forwarding 15. testpmd >
start // restore forwarding 14. let all ports forwarding work for a
while 15. testpmd > show port stats all //confirm all port can work
again 16. repeat above step 7 - 15
remove PCI address output to align with other modification which
will output it in other way
disable PF reset when its VF is ative to avoid unexpected VF
behavior
add PCI address to confirm its port number keep same
correct test method in cover letter
update testpmd command
only reset PMD layer resource and keep same port id, but
not restore settings

Acked-by: Jingjing Wu <***@intel.com>

[Wei Dai]

This patch set adds a function rte_eth_dev_reset( ) in rte_ethdev layer.
Sometimes a port have to be reset passively. For example a PF is reset,
all its VFs should also be reset by application itself to align with
the PF.
A DPDK application also can call this function to trigger an initative
port reset.

When processing reset, if the port goes through PCI remove() and then
PCI probe() for restoration, its port id may be changed and this is not
expected by some DPDK application.

Normally, PCI probe() includes two parts: one is in rte_ethdev layer
to allocate resource in rte_ethdev layer and the other is calling PMD
specific dev_init() to allocate and initialize resource in PMD layer.
PCI remove( ) releases all resource allocated from rte_ethdev layer
in PCI probe( ) and calls PMD specific dev_uninit( ) to releaase
resource allocated by dev_init( ) in PMD layer.

To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ). All these are what
rte_eth_dev_reset() does.

The rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.

After calling rte_eth_dev_reset( ), the application can go through
rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
its previous settings or to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to
confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF
2. testpmd > set verbose 1 //to observe VF working
3. testpmd > show port info all //show port number and MAC addr
4. testpmd > start
5. let all ports forwarding work for a while
6. testpmd > show port stats all
7. ifconfig name-of-PF down
8. A message is shown in testmd to indicate PF reset
9. ifconfig name-of-PF up
10. testpmd > stop // stop forwarding to avoid crash during reset
11. testpmd > port reset all
12. testpmd > port stop all
13. testpmd > port start all //recofnig all ports
14. testpmd > show port info all
//get mapping of port id and MAC addr for forwarding
15. testpmd > start // restore forwarding
14. let all ports forwarding work for a while
15. testpmd > show port stats all //confirm all port can work again
16. repeat above step 7 - 15

chagnes:
v6:
add more comments to explain why the rte_eth_dev_reset( ) is needeed,
when it is called and what the application should do after calling it.
add more comments to explain why reset of PF with SR-IOV is not
supported currently.

v5:
remove PCI address output to align with other modification which
will output it in other way
disable PF reset when its VF is ative to avoid unexpected VF behavior
v4:
add PCI address to confirm its port number keep same
correct test method in cover letter
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>

Wei Dai (4):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset

app/test-pmd/cmdline.c | 10 +++++---
app/test-pmd/testpmd.c | 41 +++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
drivers/net/i40e/i40e_ethdev.c | 28 ++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 ++++++++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 47 ++++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 16 ++++++++++++
lib/librte_ether/rte_ethdev.h | 45 ++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
9 files changed, 205 insertions(+), 4 deletions(-)

[Wei Dai]

This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep
its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what this function does and what an application
should do after calling this function.

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 16 ++++++++++++
lib/librte_ether/rte_ethdev.h | 45 ++++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 2 +-
3 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 76179fd..21ea5bb 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -3414,3 +3414,19 @@ rte_eth_dev_adjust_nb_rx_tx_desc(uint8_t port_id,

return 0;
}
+
+int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+ return ret;
+}
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index fd6baf3..f5fd047 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1113,6 +1113,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1430,6 +1433,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2132,6 +2136,47 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * A DPDK application calls this function to do an initiative or passive
+ * reset of a port.
+ *
+ * Sometimes a port have to be reset passively. For example a PF is reset,
+ * all its VFs should also be reset by application itself to align with the
+ * PF.
+ * A DPDK application also can call this function to trigger an initative
+ * port reset.
+ *
+ * When processing reset, if the port goes through PCI remove() and then
+ * PCI probe() for restoration, its port id may be changed and this is not
+ * expected by some DPDK application.
+ *
+ * Normally, PCI probe() includes two parts: one is in rte_ethdev layer
+ * to allocate resource in rte_ethdev layer and the other is calling PMD
+ * specific dev_init() to allocate and initialize resource in PMD layer.
+ * PCI remove( ) releases all resource allocated from rte_ethdev layer
+ * in PCI probe( ) and calls PMD specific dev_uninit( ) to releaase
+ * resource allocated by dev_init( ) in PMD layer.
+ *
+ * To keep same port id and reset the port, only dev_uninit() and
+ * dev_init( ) in PMD can be called and keep all resources allocated
+ * from rte_ethdev layer poart in PCI probe( ). All these are what
+ * rte_eth_dev_reset() does.
+ *
+ * The rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
+ * and then PMD dev_init( ) to reset a port and keep same port id.
+ *
+ * After calling rte_eth_dev_reset( ), the application can go through
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
+ * its previous settings or to reconfigure itself with different settings.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index 08deb1f..d89a101 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -155,5 +155,5 @@ DPDK_17.08 {
rte_eth_dev_adjust_nb_rx_tx_desc;
rte_flow_copy;
rte_flow_isolate;
-
+ rte_eth_dev_reset;
} DPDK_17.05;

[Jerin Jacob]

-----Original Message-----

Date: Mon, 10 Jul 2017 18:05:41 +0800
Subject: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
X-Mailer: git-send-email 2.7.5
This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep
its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what this function does and what an application
should do after calling this function.
---
/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * A DPDK application calls this function to do an initiative or passive
+ * reset of a port.
+ *
+ * Sometimes a port have to be reset passively. For example a PF is reset,
+ * all its VFs should also be reset by application itself to align with the
+ * PF.

May be I didn't understood this use case properly,But, PF can always send mailbox
message to VF on PF reset. Right?
On such message from PF, VF can transparently reset without application
knowledge(i.e rx and/or tx burst return zero)

+ * A DPDK application also can call this function to trigger an initative
+ * port reset.

When apart from the above use case? Even if it is above case, we should have
some event to notify application to initiate the reset.Right? Without the
event, When or on what basics application needs to initiate reset?

+ *

[Dai, Wei]

-----Original Message-----
From: Jerin Jacob [mailto:***@caviumnetworks.com]
Sent: Monday, July 10, 2017 7:35 PM
To: Dai, Wei <***@intel.com>
Cc: ***@monjalon.net; Lu, Wenzhuo <***@intel.com>; Ananyev, Konstantin <***@intel.com>; Wu, Jingjing <***@intel.com>; Xing, Beilei <***@intel.com>; ***@dpdk.org
Subject: Re: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset

-----Original Message-----

Date: Mon, 10 Jul 2017 18:05:41 +0800
Subject: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
X-Mailer: git-send-email 2.7.5
This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep its
port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what this function does and what an application
should do after calling this function.
---
/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * A DPDK application calls this function to do an initiative or
+ passive
+ * reset of a port.
+ *
+ * Sometimes a port have to be reset passively. For example a PF is
+ reset,
+ * all its VFs should also be reset by application itself to align
+ with the
+ * PF.

May be I didn't understood this use case properly,But, PF can always send mailbox message to VF on PF reset. Right?
[Wei: As to ixgbe, PF kernel driver always send mailbox message to VF when PF is reset. For other NICs, there are maybe
other mechanism to notify VF of PF reset.]
On such message from PF, VF can transparently reset without application knowledge(i.e rx and/or tx burst return zero)
[Wei: VF reset is handling many HW registers which may have effect on working Rx/Tx path and may cause some unexpected
behavior like crash. So application should make some operations like stopping Rx/Tx path before it begin VF reset. This is why
application should handle VF reset itself.]

+ * A DPDK application also can call this function to trigger an
+ initative
+ * port reset.

When apart from the above use case? Even if it is above case, we should have some event to notify application to initiate the reset.Right? Without the event, When or on what basics application needs to initiate reset?
[Wei: Indeed, until now we didn't see any use case which DPDK application initiative port reset itself.
The most usual case is that PF is working with kernel driver and VFs are working with DPDK PMD.
Some operations on kernel PF lead to a PF reset which causes its VF reset.
Anyway this new function provides a possibility for application to trigger an initiative port reset.]

+ *

[Jerin Jacob]

-----Original Message-----

Date: Tue, 11 Jul 2017 01:57:15 +0000
Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset

Post by Wei Dai
+ * A DPDK application also can call this function to trigger an
+ initative
+ * port reset.

When apart from the above use case? Even if it is above case, we should have some event to notify application to initiate the reset.Right? Without the event, When or on what basics application needs to initiate reset?
[Wei: Indeed, until now we didn't see any use case which DPDK application initiative port reset itself.
The most usual case is that PF is working with kernel driver and VFs are working with DPDK PMD.
Some operations on kernel PF lead to a PF reset which causes its VF reset.
Anyway this new function provides a possibility for application to trigger an initiative port reset.]

That's fine. The only concern part is when to call reset API from
application. Can we say on RTE_ETH_EVENT_INTR_RESET event, application
needs to call the reset API? I think, it is important to specify when
application need to call this API, otherwise this api behavior will be
tightly coupled with underneath PMD. Side effect is, a new, non portable PMD specific API.
If a PMD wishes to fixup some overflow case(as an example), by invoking the reset API from
the application BUT same case may not valid for another PMD hence it
will create unexpected behavior from application based on the underneath PMD.

if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the
reset API then create a new event or if it needs to be called in
RTE_ETH_EVENT_INTR_RESET then update the API documentation.

Post by Wei Dai
+ *

[Dai, Wei]

-----Original Message-----
Sent: Tuesday, July 11, 2017 1:17 PM
Subject: Re: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
-----Original Message-----

Date: Tue, 11 Jul 2017 01:57:15 +0000
Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset

Post by Wei Dai
+ * A DPDK application also can call this function to trigger an
+ initative
+ * port reset.

When apart from the above use case? Even if it is above case, we should

have some event to notify application to initiate the reset.Right? Without
the event, When or on what basics application needs to initiate reset?

[Wei: Indeed, until now we didn't see any use case which DPDK application

initiative port reset itself.

The most usual case is that PF is working with kernel driver and VFs are

working with DPDK PMD.

Some operations on kernel PF lead to a PF reset which causes its VF reset.
Anyway this new function provides a possibility for application to
trigger an initiative port reset.]

That's fine. The only concern part is when to call reset API from application.
Can we say on RTE_ETH_EVENT_INTR_RESET event, application needs to
call the reset API? I think, it is important to specify when application need to
call this API, otherwise this api behavior will be tightly coupled with
underneath PMD. Side effect is, a new, non portable PMD specific API.
If a PMD wishes to fixup some overflow case(as an example), by invoking the
reset API from the application BUT same case may not valid for another
PMD hence it will create unexpected behavior from application based on the
underneath PMD.

It is duty of PMD to trigger RTE_ETH_EVENT_INTR_RESET event and application
should also register some callback function to handle this event.
When PMD wants to trigger a reset, it can trigger RTE_ETH_EVENT_INTR_RESET.
On the received event of RTE_ETH_EVENT_INTR_RESET, application can begin to
handle it: stop working queues, make rx and tx function not be called
and then call rte_eth_dev_reset( ).
For thread safety, all these controlling operations had better be made in same thread.
For example, when ixgbe PF is reset, PF send a message to notify VF this event and
also trigger an interrupt to VF. And then in the interrupt service routine DPDK VF
detect this notification message and calls
_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL, NULL).
So it means that PF reset trigger RTE_ETH_EVENT_INTR_RESET event in VF.
The function _rte_eth_dev_callback_process( ) will call the registered callback function.
The callback function can trigger application to handle all operations of VF reset including
something like stopping working Rx/Tx queues and call this rte_eth_dev_reset().
The rte_eth_dev_reset( ) itself is generic function which only does some HW reset operations
through calling dev_unint() and dev_init(). And itself doesn't handle above synchronization which
is handled by application.
PMD itself should not call rte_eth_dev_reset( ). PMD can trigger application to handle reset event.
It is duty of application to handle all synchronization befort it calls rte_eth_dev_reset( ).

if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the reset
API then create a new event or if it needs to be called in
RTE_ETH_EVENT_INTR_RESET then update the API documentation.

Of course, when PMD wants to trigger a reset event, it can trigger other event other than
RTE_ETH_EVENT_INTR_RESET. So the application should know which the alternate event is.
This make application more complex. So it is suggested that only RTE_ETH_EVENT_INTR_RESET
can be used to trigger a port reset.

Post by Wei Dai
+ *

[Jerin Jacob]

-----Original Message-----

Date: Tue, 11 Jul 2017 14:36:57 +0000
Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset

-----Original Message-----
Sent: Tuesday, July 11, 2017 1:17 PM
Subject: Re: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset
-----Original Message-----

Date: Tue, 11 Jul 2017 01:57:15 +0000
Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset

Post by Wei Dai
+ * A DPDK application also can call this function to trigger an
+ initative
+ * port reset.

When apart from the above use case? Even if it is above case, we should

have some event to notify application to initiate the reset.Right? Without
the event, When or on what basics application needs to initiate reset?

[Wei: Indeed, until now we didn't see any use case which DPDK application

initiative port reset itself.

The most usual case is that PF is working with kernel driver and VFs are

working with DPDK PMD.

Some operations on kernel PF lead to a PF reset which causes its VF reset.
Anyway this new function provides a possibility for application to
trigger an initiative port reset.]

That's fine. The only concern part is when to call reset API from application.
Can we say on RTE_ETH_EVENT_INTR_RESET event, application needs to
call the reset API? I think, it is important to specify when application need to
call this API, otherwise this api behavior will be tightly coupled with
underneath PMD. Side effect is, a new, non portable PMD specific API.
If a PMD wishes to fixup some overflow case(as an example), by invoking the
reset API from the application BUT same case may not valid for another
PMD hence it will create unexpected behavior from application based on the
underneath PMD.

It is duty of PMD to trigger RTE_ETH_EVENT_INTR_RESET event and application
should also register some callback function to handle this event.
When PMD wants to trigger a reset, it can trigger RTE_ETH_EVENT_INTR_RESET.
On the received event of RTE_ETH_EVENT_INTR_RESET, application can begin to
handle it: stop working queues, make rx and tx function not be called
and then call rte_eth_dev_reset( ).
For thread safety, all these controlling operations had better be made in same thread.
For example, when ixgbe PF is reset, PF send a message to notify VF this event and
also trigger an interrupt to VF. And then in the interrupt service routine DPDK VF
detect this notification message and calls
_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL, NULL).
So it means that PF reset trigger RTE_ETH_EVENT_INTR_RESET event in VF.
The function _rte_eth_dev_callback_process( ) will call the registered callback function.
The callback function can trigger application to handle all operations of VF reset including
something like stopping working Rx/Tx queues and call this rte_eth_dev_reset().
The rte_eth_dev_reset( ) itself is generic function which only does some HW reset operations
through calling dev_unint() and dev_init(). And itself doesn't handle above synchronization which
is handled by application.
PMD itself should not call rte_eth_dev_reset( ). PMD can trigger application to handle reset event.
It is duty of application to handle all synchronization befort it calls rte_eth_dev_reset( ).

No disagreement on the expected behavior.

if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the reset
API then create a new event or if it needs to be called in
RTE_ETH_EVENT_INTR_RESET then update the API documentation.

Of course, when PMD wants to trigger a reset event, it can trigger other event other than
RTE_ETH_EVENT_INTR_RESET. So the application should know which the alternate event is.
This make application more complex. So it is suggested that only RTE_ETH_EVENT_INTR_RESET
can be used to trigger a port reset.

Yes. I suggest to add this info on documentation. ie "application invokes the
reset API on RTE_ETH_EVENT_INTR_RESET event". That will answer "when"
application need to invoke this API.

Post by Wei Dai
+ *

[Dai, Wei]

Post by Dai, Wei

Subject: RE: [dpdk-dev] [PATCH v6 1/4] ethdev: add support of NIC reset

Post by Wei Dai
+ * A DPDK application also can call this function to trigger an
+ initative
+ * port reset.

When apart from the above use case? Even if it is above case, we should

have some event to notify application to initiate the reset.Right? Without
the event, When or on what basics application needs to initiate reset?

[Wei: Indeed, until now we didn't see any use case which DPDK application

initiative port reset itself.

The most usual case is that PF is working with kernel driver and VFs are

working with DPDK PMD.

Some operations on kernel PF lead to a PF reset which causes its VF reset.
Anyway this new function provides a possibility for application to
trigger an initiative port reset.]

That's fine. The only concern part is when to call reset API from application.
Can we say on RTE_ETH_EVENT_INTR_RESET event, application needs to
call the reset API? I think, it is important to specify when application need to
call this API, otherwise this api behavior will be tightly coupled with
underneath PMD. Side effect is, a new, non portable PMD specific API.
If a PMD wishes to fixup some overflow case(as an example), by invoking the
reset API from the application BUT same case may not valid for another
PMD hence it will create unexpected behavior from application based on the
underneath PMD.
if RTE_ETH_EVENT_INTR_RESET event is not expected event to call the reset
API then create a new event or if it needs to be called in
RTE_ETH_EVENT_INTR_RESET then update the API documentation.

Thanks for your feedback.
I'll add description of this function in API doc in my v7 patch set.

Post by Dai, Wei

Post by Wei Dai
+ *

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 47 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index a0ae089..280f4bb 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -170,6 +170,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -266,6 +267,7 @@ static int ixgbevf_dev_link_update(struct rte_eth_dev *dev,
int wait_to_complete);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -527,6 +529,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -616,6 +619,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2840,6 +2844,32 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to ixgbe PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -5036,6 +5066,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 28 ++++++++++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 +++++++++++++++++++
2 files changed, 47 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 71cb7d3..1b00717 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2136,6 +2138,32 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+/*
+ * Reest PF device only to re-initialize resources in PMD layer
+ */
+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to i40e PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index bab09f8..858d2a5 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -123,6 +123,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -204,6 +205,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2356,6 +2358,23 @@ i40evf_dev_close(struct rte_eth_dev *dev)
i40evf_disable_irq0(hw);
}

+/*
+ * Reest VF device only to re-initialize resources in PMD layer
+ */
+static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{

[Wei Dai]

When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>
---
app/test-pmd/cmdline.c | 10 +++++++---
app/test-pmd/testpmd.c | 41 +++++++++++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
3 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index d4ff608..d096adc 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -970,6 +970,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -979,7 +981,7 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

@@ -1014,6 +1016,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1023,7 +1027,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1031,7 +1035,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close|reset <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index e09b803..8eb4292 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1669,6 +1669,47 @@ close_port(portid_t pid)
}

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Resetting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding "
+ "configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n",
+ pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ } else {
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 73985c3..fb44fdd 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -609,6 +609,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode,
int start_port(portid_t pid);
void stop_port(portid_t pid);
void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);

[Wei Dai]

This patch set adds a function rte_eth_dev_reset( ) in rte_ethdev layer.
Sometimes a port have to be reset passively. For example a PF is reset,
all its VFs should also be reset by application itself to align with
the PF.
A DPDK application also can call this function to trigger an initative
port reset.

When processing reset, if the port goes through PCI remove() and then
PCI probe() for restoration, its port id may be changed and this is not
expected by some DPDK application.

Normally, PCI probe() includes two parts: one is in rte_ethdev layer
to allocate resource in rte_ethdev layer and the other is calling PMD
specific dev_init() to allocate and initialize resource in PMD layer.
PCI remove( ) releases all resource allocated from rte_ethdev layer
in PCI probe( ) and calls PMD specific dev_uninit( ) to releaase
resource allocated by dev_init( ) in PMD layer.

To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ). All these are what
rte_eth_dev_reset() does.

The rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.

After calling rte_eth_dev_reset( ), the application can go through
rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
its previous settings or to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to
confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF
2. testpmd > set verbose 1 //to observe VF working
3. testpmd > show port info all //show port number and MAC addr
4. testpmd > start
5. let all ports forwarding work for a while
6. testpmd > show port stats all
7. ifconfig name-of-PF down
8. A message is shown in testmd to indicate PF reset
9. ifconfig name-of-PF up
10. testpmd > stop // stop forwarding to avoid crash during reset
11. testpmd > port reset all
12. testpmd > port stop all
13. testpmd > port start all //recofnig all ports
14. testpmd > show port info all
//get mapping of port id and MAC addr for forwarding
15. testpmd > start // restore forwarding
14. let all ports forwarding work for a while
15. testpmd > show port stats all //confirm all port can work again
16. repeat above step 7 - 15

chagnes:
v7:
add description of NIC reset in doc/poll_mode_drv.rst

v6:
add more comments to explain why the rte_eth_dev_reset( ) is needeed,
when it is called and what the application should do after calling it.
add more comments to explain why reset of PF with SR-IOV is not
supported currently.

v5:
remove PCI address output to align with other modification which
will output it in other way
disable PF reset when its VF is ative to avoid unexpected VF behavior
v4:
add PCI address to confirm its port number keep same
correct test method in cover letter
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>

Wei Dai (5):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset
doc: add description of the NIC reset API

app/test-pmd/cmdline.c | 12 ++++--
app/test-pmd/testpmd.c | 41 ++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
doc/guides/prog_guide/poll_mode_drv.rst | 40 ++++++++++++++++++
drivers/net/i40e/i40e_ethdev.c | 28 +++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 +++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 47 +++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 17 ++++++++
lib/librte_ether/rte_ethdev.h | 73 +++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 1 +
10 files changed, 275 insertions(+), 4 deletions(-)

[Wei Dai]

This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep
its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 17 ++++++++
lib/librte_ether/rte_ethdev.h | 73 ++++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 1 +
3 files changed, 91 insertions(+)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index a1b7447..4172d82 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -3415,3 +3415,20 @@ rte_eth_dev_adjust_nb_rx_tx_desc(uint8_t port_id,

return 0;
}
+
+int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+
+ return ret;
+}
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index f683727..c458c52 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1113,6 +1113,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1433,6 +1436,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2138,6 +2142,75 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * A DPDK application calls this function to do an initiative or passive
+ * reset of a port.
+ *
+ * Sometimes a port have to be reset passively. For example a PF is reset,
+ * all its VFs should also be reset by application itself to align with the
+ * PF. A DPDK application also can call this function to trigger an initative
+ * port reset.
+ *
+ * When processing reset, if the port goes through PCI remove() and then
+ * PCI probe() for restoration, its port id may be changed and this is not
+ * expected by some DPDK application.
+ *
+ * Normally, PCI probe() includes two parts: one is in rte_ethdev layer
+ * to allocate resource in rte_ethdev layer and the other is calling PMD
+ * specific dev_init() to allocate and initialize resource in PMD layer.
+ * PCI remove( ) releases all resource allocated from rte_ethdev layer
+ * in PCI probe( ) and calls PMD specific dev_uninit( ) to releaase
+ * resource allocated by dev_init( ) in PMD layer.
+ *
+ * To keep same port id and reset the port, only dev_uninit() and
+ * dev_init( ) in PMD can be called and keep all resources allocated
+ * from rte_ethdev layer poart in PCI probe( ). All these are what
+ * rte_eth_dev_reset() does.
+ *
+ * The rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
+ * and then PMD dev_init( ) to reset a port and keep same port id.
+ *
+ * Normally, a DPDK application can invoke this function when
+ * RTE_ETH_EVENT_INTR_RESET event is detected.
+
+ * It is duty of PMD to trigger RTE_ETH_EVENT_INTR_RESET event and application
+ * should also register some callback function to handle this event.
+ * When PMD needs to trigger a reset, it can trigger RTE_ETH_EVENT_INTR_RESET.
+ * On the received event of RTE_ETH_EVENT_INTR_RESET, application can begin to
+ * handle it: stop working queues, make rx and tx function not be called and
+ * then call rte_eth_dev_reset( ).For thread safety, all these controlling
+ * operations had better be made in same thread.
+ *
+ * For example, when PF is reset, PF send a message to notify VF this event and
+ * also trigger an interrupt to VF. And then in the interrupt service routine
+ * DPDK VF detect this notification message and calls
+ * _rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL, NULL).
+ * So it means that PF reset trigger RTE_ETH_EVENT_INTR_RESET event in VF.
+ * The function _rte_eth_dev_callback_process( ) will call the registered
+ * callback function. The callback function can trigger application to handle
+ * all operations of VF reset including something like stopping working Rx/Tx
+ * queues and call this rte_eth_dev_reset().
+ *
+ * The rte_eth_dev_reset( ) itself is generic function which only does some HW
+ * reset operations through calling dev_unint() and dev_init(). And itself
+ * doesn't handle above synchronization which is handled by application.
+ *
+ * PMD itself should not call rte_eth_dev_reset( ). PMD can trigger application
+ * to handle reset event. It is duty of application to handle all
+ * synchronizations befort it calls rte_eth_dev_reset( ).
+ *
+ * After calling rte_eth_dev_reset( ), the application can go through
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
+ * its previous settings or to reconfigure itself with different settings.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index 4283728..e86d51e 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -155,6 +155,7 @@ DPDK_17.08 {
rte_eth_dev_adjust_nb_rx_tx_desc;
rte_flow_copy;
rte_flow_isolate;
+ rte_eth_dev_reset;
rte_tm_capabilities_get;
rte_tm_get_leaf_nodes;
rte_tm_hierarchy_commit;

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 47 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index c54d325..b5df073 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -170,6 +170,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -266,6 +267,7 @@ static int ixgbevf_dev_link_update(struct rte_eth_dev *dev,
int wait_to_complete);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -524,6 +526,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -614,6 +617,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2856,6 +2860,32 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to ixgbe PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -5052,6 +5082,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 28 ++++++++++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 +++++++++++++++++++
2 files changed, 47 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 97a73e1..4ed4cf0 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2151,6 +2153,32 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+/*
+ * Reest PF device only to re-initialize resources in PMD layer
+ */
+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to i40e PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index bab09f8..858d2a5 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -123,6 +123,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -204,6 +205,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2356,6 +2358,23 @@ i40evf_dev_close(struct rte_eth_dev *dev)
i40evf_disable_irq0(hw);
}

+/*
+ * Reest VF device only to re-initialize resources in PMD layer
+ */
+static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{

[Wei Dai]

When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>
---
app/test-pmd/cmdline.c | 12 ++++++++----
app/test-pmd/testpmd.c | 41 +++++++++++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
3 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index d4ff608..8784aff 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -970,6 +970,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -979,14 +981,14 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

cmdline_parse_inst_t cmd_operate_port = {
.f = cmd_operate_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close all: Start/Stop/Close all ports",
+ .help_str = "port start|stop|close all: Start/Stop/Close/Reset all ports",
.tokens = {
(void *)&cmd_operate_port_all_cmd,
(void *)&cmd_operate_port_all_port,
@@ -1014,6 +1016,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1023,7 +1027,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1031,7 +1035,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index e754d12..59fc441 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1683,6 +1683,47 @@ close_port(portid_t pid)
}

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Resetting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding "
+ "configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n",
+ pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ } else {
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index c9d7739..1d1ee75 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -610,6 +610,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode,
int start_port(portid_t pid);
void stop_port(portid_t pid);
void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);

[Wei Dai]

Signed-off-by: Wei Dai <***@intel.com>
---
doc/guides/prog_guide/poll_mode_drv.rst | 40 +++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)

diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst
index 4987f70..60518fc 100644
--- a/doc/guides/prog_guide/poll_mode_drv.rst
+++ b/doc/guides/prog_guide/poll_mode_drv.rst
@@ -525,3 +525,43 @@ call. As an end result, the application is able to achieve its goal of
monitoring a single statistic ("rx_errors" in this case), and if that shows
packets being dropped, it can easily retrieve a "set" of statistics using the
IDs array parameter to ``rte_eth_xstats_get_by_id`` function.
+
+NIC Reset API
+~~~~~~~~~~~~~
+
+.. code-block:: c
+
+ int rte_eth_dev_reset(uint8_t port_id);
+
+Sometimes a port have to be reset passively. For example a PF is reset, all its
+VFs should also be reset by application itself to align with the PF. A DPDK
+application also can call this function to trigger an initative port reset.
+
+Normally, a DPDK application can invake this function when
+RTE_ETH_EVENT_INTR_RESET event is detected.
+
+It is duty of PMD to trigger RTE_ETH_EVENT_INTR_RESET event and application
+should also register some callback function to handle this event.
+When PMD needs to trigger a reset, it can trigger RTE_ETH_EVENT_INTR_RESET.
+On the received event of RTE_ETH_EVENT_INTR_RESET, application can begin to
+handle it: stop working queues, make rx and tx function not be called and
+then call rte_eth_dev_reset( ).For thread safety, all these controlling
+operations had better be made in same thread.
+
+For example, when PF is reset, PF send a message to notify VF this event and
+also trigger an interrupt to VF. And then in the interrupt service routine
+DPDK VF detect this notification message and calls
+_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL, NULL).
+So it means that PF reset trigger RTE_ETH_EVENT_INTR_RESET event in VF.
+The function _rte_eth_dev_callback_process( ) will call the registered
+callback function. The callback function can trigger application to handle
+all operations of VF reset including something like stopping working Rx/Tx
+queues and call this rte_eth_dev_reset().
+
+The rte_eth_dev_reset( ) itself is generic function which only does some HW
+reset operations through calling dev_unint() and dev_init(). And itself doesn't
+handle above synchronization which is handled by application.
+
+PMD itself should not call rte_eth_dev_reset( ). PMD can trigger application to
+handle reset event. It is duty of application to handle all synchronization
+befort it calls rte_eth_dev_reset( ).

[Thomas Monjalon]

Post by Wei Dai
add description of NIC reset in doc/poll_mode_drv.rst

Thanks for bringing more explanations.
The texts have to be reviewed by a native english speaker.

About the organization of the doc, it is good to explain the VF use case
in the prog guide as you do. But you should not add it to the doxygen.
The doxygen must be concise enough and explains only the API:
- when it can be called
- what a driver is doing when calling reset
- what is the expected state of the device after reset

[Dai, Wei]

Hi, Thomas
Thanks for your expert guidance.
I'll rework out v8 patch set to follow your guide and rebase to master brach.

-----Original Message-----
Sent: Monday, July 17, 2017 3:26 AM
Subject: Re: [dpdk-dev] [PATCH v7 0/5] Support NIC reset and keep same
port id

Post by Wei Dai
add description of NIC reset in doc/poll_mode_drv.rst

Thanks for bringing more explanations.
The texts have to be reviewed by a native english speaker.
About the organization of the doc, it is good to explain the VF use case in the
prog guide as you do. But you should not add it to the doxygen.
- when it can be called
- what a driver is doing when calling reset
- what is the expected state of the device after reset

[Dai, Wei]

Remy is a Native English speaker.
He will help me review my v8 patch set.
Thank you, Remy.

-----Original Message-----
Sent: Monday, July 17, 2017 3:26 AM
Subject: Re: [dpdk-dev] [PATCH v7 0/5] Support NIC reset and keep same
port id

Post by Wei Dai
add description of NIC reset in doc/poll_mode_drv.rst

Thanks for bringing more explanations.
The texts have to be reviewed by a native english speaker.
About the organization of the doc, it is good to explain the VF use case in the
prog guide as you do. But you should not add it to the doxygen.
- when it can be called
- what a driver is doing when calling reset
- what is the expected state of the device after reset

[Wei Dai]

s patch set adds a function rte_eth_dev_reset( ) in rte_ethdev layer.
Sometimes a port have to be reset passively. For example a PF is reset,
all its VFs should also be reset by application itself to align with
the PF.
A DPDK application also can call this function to trigger an initative
port reset.

When processing reset, if the port goes through PCI remove() and then
PCI probe() for restoration, its port id may be changed and this is not
expected by some DPDK application.

Normally, PCI probe() includes two parts: one is in rte_ethdev layer
to allocate resource in rte_ethdev layer and the other is calling PMD
specific dev_init() to allocate and initialize resource in PMD layer.
PCI remove( ) releases all resource allocated from rte_ethdev layer
in PCI probe( ) and calls PMD specific dev_uninit( ) to releaase
resource allocated by dev_init( ) in PMD layer.

To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ). All these are what
rte_eth_dev_reset() does.

The rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.

After calling rte_eth_dev_reset( ), the application can go through
rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
its previous settings or to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to
confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF
2. testpmd > set verbose 1 //to observe VF working
3. testpmd > show port info all //show port number and MAC addr
4. testpmd > start
5. let all ports forwarding work for a while
6. testpmd > show port stats all
7. ifconfig name-of-PF down
8. A message is shown in testmd to indicate PF reset
9. ifconfig name-of-PF up
10. testpmd > stop // stop forwarding to avoid crash during reset
11. testpmd > port reset all
12. testpmd > port stop all
13. testpmd > port start all //recofnig all ports
14. testpmd > show port info all
//get mapping of port id and MAC addr for forwarding
15. testpmd > start // restore forwarding
14. let all ports forwarding work for a while
15. testpmd > show port stats all //confirm all port can work again
16. repeat above step 7 - 15

chagnes:
v8:
modify the comments before the declaration of rte_eth_dev_reset( ) to
make doc generated from doxygen be concise.

v7:
add description of NIC reset in doc/poll_mode_drv.rst

v6:
add more comments to explain why the rte_eth_dev_reset( ) is needeed,
when it is called and what the application should do after calling it.
add more comments to explain why reset of PF with SR-IOV is not
supported currently.

v5:
remove PCI address output to align with other modification which
will output it in other way
disable PF reset when its VF is ative to avoid unexpected VF behavior
v4:
add PCI address to confirm its port number keep same
correct test method in cover letter
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>

Wei Dai (5):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset
doc: add description of the NIC reset API

app/test-pmd/cmdline.c | 12 ++++++---
app/test-pmd/testpmd.c | 41 ++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
doc/guides/prog_guide/poll_mode_drv.rst | 40 ++++++++++++++++++++++++++++
drivers/net/i40e/i40e_ethdev.c | 28 ++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 +++++++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 47 +++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 17 ++++++++++++
lib/librte_ether/rte_ethdev.h | 33 +++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 1 +
10 files changed, 235 insertions(+), 4 deletions(-)

[Wei Dai]

This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep
its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.

Signed-off-by: Wei Dai <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 17 +++++++++++++++++
lib/librte_ether/rte_ethdev.h | 33 +++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 1 +
3 files changed, 51 insertions(+)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index a1b7447..2448bde 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -1001,6 +1001,23 @@ rte_eth_dev_close(uint8_t port_id)
}

int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+
+ return ret;
+}
+
+int
rte_eth_rx_queue_setup(uint8_t port_id, uint16_t rx_queue_id,
uint16_t nb_rx_desc, unsigned int socket_id,
const struct rte_eth_rxconf *rx_conf,
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index 224f485..bd74d15 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1111,6 +1111,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1431,6 +1434,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2136,6 +2140,35 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * When a port has to be reset passively, the DPDK application can invoke this
+ * function. For example a PF is reset, all its VFs should also be reset.
+ * Normally, a DPDK application can invoke this function when
+ * RTE_ETH_EVENT_INTR_RESET event is detected. A DPDK application can also call
+ * this function to start an initiative port reset.
+ *
+ * When this function is called, it first stops the port and then call PMD
+ * specific dev_uninit( ) and dev_init( ) to makes the port return to initial
+ * status in which no any Tx queue and no Rx queue are setup and the port has
+ * just be reset and not started. And the port keeps its port id after calling
+ * this function.
+ *
+ * After calling rte_eth_dev_reset( ), the application should go through
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
+ * its previous settings or to reconfigure itself with different settings.
+ *
+ * Note: to avoid unexpected behaviour, the application should stop calling Rx
+ * and Rx function before calling rte_eth_dev_reset( ).For thread safety,
+ * all these controlling operations had better be made in same thread.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index 4283728..e86d51e 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -155,6 +155,7 @@ DPDK_17.08 {
rte_eth_dev_adjust_nb_rx_tx_desc;
rte_flow_copy;
rte_flow_isolate;
+ rte_eth_dev_reset;
rte_tm_capabilities_get;
rte_tm_get_leaf_nodes;
rte_tm_hierarchy_commit;

[Remy Horton]

See suggested wording inline

Post by Wei Dai
This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep
its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.

This patch adds a new eth_dev layer API function rte_eth_dev_reset(),
which a DPDK application can call to reset a NIC and keep its port id
afterwards. It means that all software resources allocated in the ethdev
layer are kept, and software & hardware resources of the NIC within the
NIC's PMD are reset to a state simular to that obtained by calling the
PCI dev_uninit() and then dev_init(). This effective sequence of
dev_uninit() and dev_init() is packed into a single API function
rte_eth_dev_reset().

Please see the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.

Post by Wei Dai
---
lib/librte_ether/rte_ethdev.c | 17 +++++++++++++++++
lib/librte_ether/rte_ethdev.h | 33 +++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 1 +
3 files changed, 51 insertions(+)
/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * When a port has to be reset passively, the DPDK application can invoke this
+ * function. For example a PF is reset, all its VFs should also be reset.
+ * Normally, a DPDK application can invoke this function when
+ * RTE_ETH_EVENT_INTR_RESET event is detected. A DPDK application can also call
+ * this function to start an initiative port reset.

When a port has to be reset passively, the DPDK application can invoke
this function. For example when a PF is reset, all its VFs should also
be reset. Normally a DPDK application can invoke this function when
RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to start
a port reset in other circumstances.

Post by Wei Dai
+ * When this function is called, it first stops the port and then call PMD
+ * specific dev_uninit( ) and dev_init( ) to makes the port return to initial
+ * status in which no any Tx queue and no Rx queue are setup and the port has
+ * just be reset and not started. And the port keeps its port id after calling
+ * this function.

When this function is called, it first stops the port and then calls the
PMD specific dev_uninit( ) and dev_init( ) to return the port to initial
state, in which no Tx and Rx queues are setup, as if the port has been
reset and not started. The port keeps the port id it had before the
function call.

Post by Wei Dai
+ * After calling rte_eth_dev_reset( ), the application should go through
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
+ * its previous settings or to reconfigure itself with different settings.

After calling rte_eth_dev_reset( ), the application should use
rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
rte_eth_tx_queue_setup( ), and rte_eth_dev_start( )
to reconfigure the device as appropriate.

Post by Wei Dai
+ * Note: to avoid unexpected behaviour, the application should stop calling Rx
+ * and Rx function before calling rte_eth_dev_reset( ).For thread safety,
+ * all these controlling operations had better be made in same thread.

Note: To avoid unexpected behavior, the application should stop calling
Tx and Rx functions before calling rte_eth_dev_reset( ). For thread
safety, all these controlling functions should be called from the same
thread.

[Dai, Wei]

Many thanks, Remy.
My v9 patch set has followed your guidance.

-----Original Message-----
From: Horton, Remy
Sent: Thursday, July 20, 2017 9:22 PM
Subject: Re: [dpdk-dev] [PATCH v8 1/5] ethdev: add support of NIC reset
See suggested wording inline

Post by Wei Dai
This patch adds a new eth_dev layer API function rte_eth_dev_reset().
A DPDK application can call this function to reset a NIC and keep
its port id afterwards.
It means that all SW resources allocated in ethdev layer should be
kept and SW and HW resources of the NIC in PMD need to be reset in
similar way that it runs PCI dev_uninit() and then dev_init().
The sequence of dev_uninit() and dev_init() can be packed into a
single interface API rte_eth_dev_reset().
Please See the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.

This patch adds a new eth_dev layer API function rte_eth_dev_reset(),
which a DPDK application can call to reset a NIC and keep its port id
afterwards. It means that all software resources allocated in the ethdev
layer are kept, and software & hardware resources of the NIC within the
NIC's PMD are reset to a state simular to that obtained by calling the
PCI dev_uninit() and then dev_init(). This effective sequence of
dev_uninit() and dev_init() is packed into a single API function
rte_eth_dev_reset().
Please see the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.

Post by Wei Dai
---
lib/librte_ether/rte_ethdev.c | 17 +++++++++++++++++
lib/librte_ether/rte_ethdev.h | 33

+++++++++++++++++++++++++++++++++

Post by Wei Dai
lib/librte_ether/rte_ether_version.map | 1 +
3 files changed, 51 insertions(+)
/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * When a port has to be reset passively, the DPDK application can invoke

this

Post by Wei Dai
+ * function. For example a PF is reset, all its VFs should also be reset.
+ * Normally, a DPDK application can invoke this function when
+ * RTE_ETH_EVENT_INTR_RESET event is detected. A DPDK application

can also call

Post by Wei Dai
+ * this function to start an initiative port reset.

When a port has to be reset passively, the DPDK application can invoke
this function. For example when a PF is reset, all its VFs should also
be reset. Normally a DPDK application can invoke this function when
RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to start
a port reset in other circumstances.

Post by Wei Dai
+ * When this function is called, it first stops the port and then call PMD
+ * specific dev_uninit( ) and dev_init( ) to makes the port return to initial
+ * status in which no any Tx queue and no Rx queue are setup and the port

has

Post by Wei Dai
+ * just be reset and not started. And the port keeps its port id after calling
+ * this function.

When this function is called, it first stops the port and then calls the
PMD specific dev_uninit( ) and dev_init( ) to return the port to initial
state, in which no Tx and Rx queues are setup, as if the port has been
reset and not started. The port keeps the port id it had before the
function call.

Post by Wei Dai
+ * After calling rte_eth_dev_reset( ), the application should go through
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
+ * its previous settings or to reconfigure itself with different settings.

After calling rte_eth_dev_reset( ), the application should use
rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
rte_eth_tx_queue_setup( ), and rte_eth_dev_start( )
to reconfigure the device as appropriate.

Post by Wei Dai
+ * Note: to avoid unexpected behaviour, the application should stop

calling Rx

Post by Wei Dai
+ * and Rx function before calling rte_eth_dev_reset( ).For thread

safety,

Post by Wei Dai
+ * all these controlling operations had better be made in same

thread.
Note: To avoid unexpected behavior, the application should stop calling
Tx and Rx functions before calling rte_eth_dev_reset( ). For thread
safety, all these controlling functions should be called from the same
thread.

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 47 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 9b06ac1..f27c34e 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -169,6 +169,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -265,6 +266,7 @@ static int ixgbevf_dev_link_update(struct rte_eth_dev *dev,
int wait_to_complete);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -523,6 +525,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -613,6 +616,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2855,6 +2859,32 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to ixgbe PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -5051,6 +5081,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 28 ++++++++++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 +++++++++++++++++++
2 files changed, 47 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 97a73e1..4ed4cf0 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2151,6 +2153,32 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+/*
+ * Reest PF device only to re-initialize resources in PMD layer
+ */
+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to i40e PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index dc6c794..e0031d1 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -122,6 +122,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -203,6 +204,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2355,6 +2357,23 @@ i40evf_dev_close(struct rte_eth_dev *dev)
i40evf_disable_irq0(hw);
}

+/*
+ * Reest VF device only to re-initialize resources in PMD layer
+ */
+static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{

[Wei Dai]

When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>
---
app/test-pmd/cmdline.c | 12 ++++++++----
app/test-pmd/testpmd.c | 41 +++++++++++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
3 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 7decb96..84a7180 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -969,6 +969,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -978,14 +980,14 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

cmdline_parse_inst_t cmd_operate_port = {
.f = cmd_operate_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close all: Start/Stop/Close all ports",
+ .help_str = "port start|stop|close all: Start/Stop/Close/Reset all ports",
.tokens = {
(void *)&cmd_operate_port_all_cmd,
(void *)&cmd_operate_port_all_port,
@@ -1013,6 +1015,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1022,7 +1026,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1030,7 +1034,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index e754d12..59fc441 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1683,6 +1683,47 @@ close_port(portid_t pid)
}

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Resetting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding "
+ "configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n",
+ pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ } else {
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index c9d7739..1d1ee75 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -610,6 +610,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode,
int start_port(portid_t pid);
void stop_port(portid_t pid);
void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);

[Wei Dai]

Signed-off-by: Wei Dai <***@intel.com>
---
doc/guides/prog_guide/poll_mode_drv.rst | 40 +++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)

diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst
index 4987f70..60518fc 100644
--- a/doc/guides/prog_guide/poll_mode_drv.rst
+++ b/doc/guides/prog_guide/poll_mode_drv.rst
@@ -525,3 +525,43 @@ call. As an end result, the application is able to achieve its goal of
monitoring a single statistic ("rx_errors" in this case), and if that shows
packets being dropped, it can easily retrieve a "set" of statistics using the
IDs array parameter to ``rte_eth_xstats_get_by_id`` function.
+
+NIC Reset API
+~~~~~~~~~~~~~
+
+.. code-block:: c
+
+ int rte_eth_dev_reset(uint8_t port_id);
+
+Sometimes a port have to be reset passively. For example a PF is reset, all its
+VFs should also be reset by application itself to align with the PF. A DPDK
+application also can call this function to trigger an initative port reset.
+
+Normally, a DPDK application can invake this function when
+RTE_ETH_EVENT_INTR_RESET event is detected.
+
+It is duty of PMD to trigger RTE_ETH_EVENT_INTR_RESET event and application
+should also register some callback function to handle this event.
+When PMD needs to trigger a reset, it can trigger RTE_ETH_EVENT_INTR_RESET.
+On the received event of RTE_ETH_EVENT_INTR_RESET, application can begin to
+handle it: stop working queues, make rx and tx function not be called and
+then call rte_eth_dev_reset( ).For thread safety, all these controlling
+operations had better be made in same thread.
+
+For example, when PF is reset, PF send a message to notify VF this event and
+also trigger an interrupt to VF. And then in the interrupt service routine
+DPDK VF detect this notification message and calls
+_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL, NULL).
+So it means that PF reset trigger RTE_ETH_EVENT_INTR_RESET event in VF.
+The function _rte_eth_dev_callback_process( ) will call the registered
+callback function. The callback function can trigger application to handle
+all operations of VF reset including something like stopping working Rx/Tx
+queues and call this rte_eth_dev_reset().
+
+The rte_eth_dev_reset( ) itself is generic function which only does some HW
+reset operations through calling dev_unint() and dev_init(). And itself doesn't
+handle above synchronization which is handled by application.
+
+PMD itself should not call rte_eth_dev_reset( ). PMD can trigger application to
+handle reset event. It is duty of application to handle all synchronization
+befort it calls rte_eth_dev_reset( ).

[Remy Horton]

Unless the plan is for this patch to be squashed, ought to have a brief
commit message..

Post by Wei Dai
---
doc/guides/prog_guide/poll_mode_drv.rst | 40 +++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
+.. code-block:: c
+
+ int rte_eth_dev_reset(uint8_t port_id);
+

Suggest following text:

Sometimes a port has to be reset passively. For example when a PF is
reset, all its VFs should also be reset by the application to make them
consistent with the PF. A DPDK application also can call this function
to trigger a port reset. Normally, a DPDK application would invokes this
function when an RTE_ETH_EVENT_INTR_RESET event is detected.

It is the duty of the PMD to trigger RTE_ETH_EVENT_INTR_RESET events and
the application should register a callback function to handle these
events. When a PMD needs to trigger a reset, it can trigger an
RTE_ETH_EVENT_INTR_RESET event. On receiving an RTE_ETH_EVENT_INTR_RESET
event, applications can handle it as follows: Stop working queues, stop
calling Rx and Tx functions, and then call rte_eth_dev_reset( ). For
thread safety all these operations should be called from the same thread.

For example when PF is reset, the PF sends a message to notify VFs of
this event and also trigger an interrupt to VFs. Then in the interrupt
service routine the VFs detects this notification message and calls
_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL,
NULL). This means that a PF reset triggers an RTE_ETH_EVENT_INTR_RESET
event within VFs. The function _rte_eth_dev_callback_process( ) will
call the registered callback function. The callback function can trigger
the application to handle all operations the VF reset requires including
stopping Rx/Tx queues and calling rte_eth_dev_reset().

The rte_eth_dev_reset( ) itself is a generic function which only does
some hardware reset operations through calling dev_unint() and
dev_init(), and itself does not handle synchronization, which is handled
by application.

The PMD itself should not call rte_eth_dev_reset( ). The PMD can trigger
the application to handle reset event. It is duty of application to
handle all synchronization before it calls rte_eth_dev_reset( ).


..Remy

[Wei Dai]

This patch set adds a function rte_eth_dev_reset( ) in rte_ethdev layer.
Sometimes a port have to be reset passively. For example a PF is reset,
all its VFs should also be reset by application itself to align with
the PF.
A DPDK application also can call this function to trigger an initative
port reset.

When processing reset, if the port goes through PCI remove() and then
PCI probe() for restoration, its port id may be changed and this is not
expected by some DPDK application.

Normally, PCI probe() includes two parts: one is in rte_ethdev layer
to allocate resource in rte_ethdev layer and the other is calling PMD
specific dev_init() to allocate and initialize resource in PMD layer.
PCI remove( ) releases all resource allocated from rte_ethdev layer
in PCI probe( ) and calls PMD specific dev_uninit( ) to releaase
resource allocated by dev_init( ) in PMD layer.

To keep same port id and reset the port, only dev_uninit() and
dev_init( ) in PMD can be called and keep all resources allocated
from rte_ethdev layer poart in PCI probe( ). All these are what
rte_eth_dev_reset() does.

The rte_eth_dev_reset( ) calls rte_eth_dev_stop( ), PMD dev_uninit( )
and then PMD dev_init( ) to reset a port and keep same port id.

After calling rte_eth_dev_reset( ), the application can go through
rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
rte_eth_tx_queue_setup( ) and rte_eth_dev_start( ) again to restore
its previous settings or to reconfigure itself with different settings.

To test this new feature, a testpmd command "port reset port_id" is added.
The mapping between port number and its PCI address can be monitored to
confirm its port number is kept.
And following test case can also be used to confirm the port can work
again after reset.

A typical test steps are listed as follows:
For example, run "ifconfig PF-name down" will trigger a reset to VF.
1. run testpmd with 2 ixgbe VF ports belonging to same PF
2. testpmd > set verbose 1 //to observe VF working
3. testpmd > show port info all //show port number and MAC addr
4. testpmd > start
5. let all ports forwarding work for a while
6. testpmd > show port stats all
7. ifconfig name-of-PF down
8. A message is shown in testmd to indicate PF reset
9. ifconfig name-of-PF up
10. testpmd > stop // stop forwarding to avoid crash during reset
11. testpmd > port reset all
12. testpmd > port stop all
13. testpmd > port start all //recofnig all ports
14. testpmd > show port info all
//get mapping of port id and MAC addr for forwarding
15. testpmd > start // restore forwarding
14. let all ports forwarding work for a while
15. testpmd > show port stats all //confirm all port can work again
16. repeat above step 7 - 15

chagnes:
v9:
rebase to master branch
modify some text after it is reviewed by an English native speaker

v8:
modify the comments before the declaration of rte_eth_dev_reset( ) to
make doc generated from doxygen be concise.

v7:
add description of NIC reset in doc/poll_mode_drv.rst

v6:
add more comments to explain why the rte_eth_dev_reset( ) is needeed,
when it is called and what the application should do after calling it.
add more comments to explain why reset of PF with SR-IOV is not
supported currently.

v5:
remove PCI address output to align with other modification which
will output it in other way
disable PF reset when its VF is ative to avoid unexpected VF behavior
v4:
add PCI address to confirm its port number keep same
correct test method in cover letter
v3:
update testpmd command
v2:
only reset PMD layer resource and keep same port id, but
not restore settings

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>
Reviewed-by: Remy Horton <***@intel.com>

Wei Dai (5):
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset
doc: add description of the NIC reset API

app/test-pmd/cmdline.c | 12 ++++++---
app/test-pmd/testpmd.c | 41 ++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
doc/guides/prog_guide/poll_mode_drv.rst | 41 ++++++++++++++++++++++++++++
drivers/net/i40e/i40e_ethdev.c | 28 ++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 +++++++++++++
drivers/net/ixgbe/ixgbe_ethdev.c | 47 +++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ethdev.c | 17 ++++++++++++
lib/librte_ether/rte_ethdev.h | 34 ++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 1 +
10 files changed, 237 insertions(+), 4 deletions(-)

[Wei Dai]

This patch adds a new eth_dev layer API function rte_eth_dev_reset(),
which a DPDK application can call to reset a NIC and keep its port id
afterwards. It means that all software resources allocated in the ethdev
layer are kept, and software & hardware resources of the NIC within the
NIC's PMD are reset to a state simular to that obtained by calling the
PCI dev_uninit() and then dev_init(). This effective sequence of
dev_uninit() and dev_init() is packed into a single API function
rte_eth_dev_reset().

Please see the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.

Signed-off-by: Wei Dai <***@intel.com>
Reviewed-by: Remy Horton <***@intel.com>
---
lib/librte_ether/rte_ethdev.c | 17 +++++++++++++++++
lib/librte_ether/rte_ethdev.h | 34 ++++++++++++++++++++++++++++++++++
lib/librte_ether/rte_ether_version.map | 1 +
3 files changed, 52 insertions(+)

diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index d4ebb1b..68ba64d 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -1004,6 +1004,23 @@ rte_eth_dev_close(uint8_t port_id)
}

int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+
+ return ret;
+}
+
+int
rte_eth_rx_queue_setup(uint8_t port_id, uint16_t rx_queue_id,
uint16_t nb_rx_desc, unsigned int socket_id,
const struct rte_eth_rxconf *rx_conf,
diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index 0e99090..fde92a1 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1115,6 +1115,9 @@ typedef int (*eth_dev_set_link_down_t)(struct rte_eth_dev *dev);
typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
/**< @internal Function used to close a configured Ethernet device. */

+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+/** <@internal Function used to reset a configured Ethernet device. */
+
typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
/**< @internal Function used to enable the RX promiscuous mode of an Ethernet device. */

@@ -1435,6 +1438,7 @@ struct eth_dev_ops {
eth_dev_set_link_up_t dev_set_link_up; /**< Device link up. */
eth_dev_set_link_down_t dev_set_link_down; /**< Device link down. */
eth_dev_close_t dev_close; /**< Close device. */
+ eth_dev_reset_t dev_reset; /**< Reset device. */
eth_link_update_t link_update; /**< Get device link state. */

eth_promiscuous_enable_t promiscuous_enable; /**< Promiscuous ON. */
@@ -2140,6 +2144,36 @@ int rte_eth_dev_set_link_down(uint8_t port_id);
void rte_eth_dev_close(uint8_t port_id);

/**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * When a port has to be reset passively, the DPDK application can invoke
+ * this function. For example when a PF is reset, all its VFs should also
+ * be reset. Normally a DPDK application can invoke this function when
+ * RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to start
+ * a port reset in other circumstances.
+ *
+ * When this function is called, it first stops the port and then calls the
+ * PMD specific dev_uninit( ) and dev_init( ) to return the port to initial
+ * state, in which no Tx and Rx queues are setup, as if the port has been
+ * reset and not started. The port keeps the port id it had before the
+ * function call.
+ *
+ * After calling rte_eth_dev_reset( ), the application should use
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ), and rte_eth_dev_start( )
+ * to reconfigure the device as appropriate.
+ *
+ * Note: To avoid unexpected behavior, the application should stop calling
+ * Tx and Rx functions before calling rte_eth_dev_reset( ). For thread
+ * safety, all these controlling functions should be called from the same
+ * thread.
+ *
+ * @param port_id
+ * The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
* Enable receipt in promiscuous mode for an Ethernet device.
*
* @param port_id
diff --git a/lib/librte_ether/rte_ether_version.map b/lib/librte_ether/rte_ether_version.map
index 4283728..e86d51e 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -155,6 +155,7 @@ DPDK_17.08 {
rte_eth_dev_adjust_nb_rx_tx_desc;
rte_flow_copy;
rte_flow_isolate;
+ rte_eth_dev_reset;
rte_tm_capabilities_get;
rte_tm_get_leaf_nodes;
rte_tm_hierarchy_commit;

[Thomas Monjalon]

Post by Wei Dai
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map

This file is now lib/librte_ether/rte_ethdev_version.map

Post by Wei Dai
@@ -155,6 +155,7 @@ DPDK_17.08 {
rte_eth_dev_adjust_nb_rx_tx_desc;
rte_flow_copy;
rte_flow_isolate;
+ rte_eth_dev_reset;
rte_tm_capabilities_get;
rte_tm_get_leaf_nodes;
rte_tm_hierarchy_commit;

The function must be added in DPDK_17.11 now.

I will do the changes.

[Luca Boccassi]

Post by Remy Horton
This patch adds a new eth_dev layer API function rte_eth_dev_reset(),
which a DPDK application can call to reset a NIC and keep its port id
afterwards. It means that all software resources allocated in the ethdev
layer are kept, and software & hardware resources of the NIC within the
NIC's PMD are reset to a state simular to that obtained by calling the
PCI dev_uninit() and then dev_init(). This effective sequence of
dev_uninit() and dev_init() is packed into a single API function
rte_eth_dev_reset().
Please see the comments before the declaration of rte_eht_dev_reset()
in lib/librte_ether/rte_ethdev.h to get more details on why this
function is needed, what it does, when it should be called
and what an application should do after calling this function.
---
 lib/librte_ether/rte_ethdev.c          | 17 +++++++++++++++++
 lib/librte_ether/rte_ethdev.h          | 34
++++++++++++++++++++++++++++++++++
 lib/librte_ether/rte_ether_version.map |  1 +
 3 files changed, 52 insertions(+)
diff --git a/lib/librte_ether/rte_ethdev.c
b/lib/librte_ether/rte_ethdev.c
index d4ebb1b..68ba64d 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -1004,6 +1004,23 @@ rte_eth_dev_close(uint8_t port_id)
 }
 
 int
+rte_eth_dev_reset(uint8_t port_id)
+{
+ struct rte_eth_dev *dev;
+ int ret;
+
+ RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+ dev = &rte_eth_devices[port_id];
+
+ RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP);
+
+ rte_eth_dev_stop(port_id);
+ ret = dev->dev_ops->dev_reset(dev);
+
+ return ret;
+}
+
+int
 rte_eth_rx_queue_setup(uint8_t port_id, uint16_t rx_queue_id,
         uint16_t nb_rx_desc, unsigned int socket_id,
         const struct rte_eth_rxconf *rx_conf,
diff --git a/lib/librte_ether/rte_ethdev.h
b/lib/librte_ether/rte_ethdev.h
index 0e99090..fde92a1 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -1115,6 +1115,9 @@ typedef int  (*eth_dev_set_link_down_t)(struct
rte_eth_dev *dev);
 typedef void (*eth_dev_close_t)(struct rte_eth_dev *dev);
*/
 
+typedef int (*eth_dev_reset_t)(struct rte_eth_dev *dev);
+
 typedef void (*eth_promiscuous_enable_t)(struct rte_eth_dev *dev);
Ethernet device. */
 
@@ -1435,6 +1438,7 @@ struct eth_dev_ops {
  eth_dev_set_link_up_t      dev_set_link_up;   /**< Device
link up. */
  eth_dev_set_link_down_t    dev_set_link_down; /**< Device
link down. */
  eth_dev_close_t            dev_close;     /**< Close device.
*/
+ eth_dev_reset_t    dev_reset;   /**<
Reset device. */
  eth_link_update_t          link_update;   /**< Get device
link state. */
 
  eth_promiscuous_enable_t   promiscuous_enable; /**<
Promiscuous ON. */
@@ -2140,6 +2144,36 @@ int rte_eth_dev_set_link_down(uint8_t
port_id);
 void rte_eth_dev_close(uint8_t port_id);
 
 /**
+ * Reset a Ethernet device and keep its port id.
+ *
+ * When a port has to be reset passively, the DPDK application can invoke
+ * this function. For example when a PF is reset, all its VFs should also
+ * be reset. Normally a DPDK application can invoke this function when
+ * RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to start
+ * a port reset in other circumstances.
+ *
+ * When this function is called, it first stops the port and then calls the
+ * PMD specific dev_uninit( ) and dev_init( ) to return the port to initial
+ * state, in which no Tx and Rx queues are setup, as if the port has been
+ * reset and not started. The port keeps the port id it had before the
+ * function call.
+ *
+ * After calling rte_eth_dev_reset( ), the application should use
+ * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
+ * rte_eth_tx_queue_setup( ), and rte_eth_dev_start( )
+ * to reconfigure the device as appropriate.
+ *
+ * Note: To avoid unexpected behavior, the application should stop calling
+ * Tx and Rx functions before calling rte_eth_dev_reset( ). For thread
+ * safety, all these controlling functions should be called from the same
+ * thread.
+ *
+ *   The port identifier of the Ethernet device.
+ */
+int rte_eth_dev_reset(uint8_t port_id);
+
+/**
  * Enable receipt in promiscuous mode for an Ethernet device.
  *
diff --git a/lib/librte_ether/rte_ether_version.map
b/lib/librte_ether/rte_ether_version.map
index 4283728..e86d51e 100644
--- a/lib/librte_ether/rte_ether_version.map
+++ b/lib/librte_ether/rte_ether_version.map
@@ -155,6 +155,7 @@ DPDK_17.08 {
  rte_eth_dev_adjust_nb_rx_tx_desc;
  rte_flow_copy;
  rte_flow_isolate;
+ rte_eth_dev_reset;
  rte_tm_capabilities_get;
  rte_tm_get_leaf_nodes;
  rte_tm_hierarchy_commit;

Hello Wei,

First of all thanks for your work, we have a use case where it's
extremely important to be able to reset the VFs when the PF (Linux)
changes state.

Question: can the rte_eth_dev_reset call block in the current
implementation that was merged?

Last year a patchset was sent by, I believe, a colleague of yours,
Wenzhuo Lu, and I've been using it in production.

One of the things that I immediately noticed is that the _reset call
was blocking - that was a show stopper for our application, so I sent a
couple of patches to take a "blocking" boolean, and return EAGAIN
instead of blocking.

You can find the original patches this link:

http://dpdk.org/ml/archives/dev/2016-June/042018.html

The description of the problem we had with the blocking call:

http://dpdk.org/ml/archives/dev/2016-July/043446.html

The "blocking" and EAGAIN changes:

http://dpdk.org/ml/archives/dev/2016-July/043535.html
http://dpdk.org/ml/archives/dev/2016-July/043818.html

Functionality-wise, from the point of view of an application, are you
aware of big differences between that patchset and what was merged
today?

Thanks!

[Wei Dai]

Reset a NIC by calling dev_uninit and then dev_init.
Go through same way in NIC PCI remove without release of
ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/ixgbe/ixgbe_ethdev.c | 47 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 194058f..3f176fe 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -169,6 +169,7 @@ static void ixgbe_dev_stop(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_up(struct rte_eth_dev *dev);
static int ixgbe_dev_set_link_down(struct rte_eth_dev *dev);
static void ixgbe_dev_close(struct rte_eth_dev *dev);
+static int ixgbe_dev_reset(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void ixgbe_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void ixgbe_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -265,6 +266,7 @@ static int ixgbevf_dev_link_update(struct rte_eth_dev *dev,
int wait_to_complete);
static void ixgbevf_dev_stop(struct rte_eth_dev *dev);
static void ixgbevf_dev_close(struct rte_eth_dev *dev);
+static int ixgbevf_dev_reset(struct rte_eth_dev *dev);
static void ixgbevf_intr_disable(struct ixgbe_hw *hw);
static void ixgbevf_intr_enable(struct ixgbe_hw *hw);
static void ixgbevf_dev_stats_get(struct rte_eth_dev *dev,
@@ -523,6 +525,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
.dev_set_link_up = ixgbe_dev_set_link_up,
.dev_set_link_down = ixgbe_dev_set_link_down,
.dev_close = ixgbe_dev_close,
+ .dev_reset = ixgbe_dev_reset,
.promiscuous_enable = ixgbe_dev_promiscuous_enable,
.promiscuous_disable = ixgbe_dev_promiscuous_disable,
.allmulticast_enable = ixgbe_dev_allmulticast_enable,
@@ -613,6 +616,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
.xstats_reset = ixgbevf_dev_stats_reset,
.xstats_get_names = ixgbevf_dev_xstats_get_names,
.dev_close = ixgbevf_dev_close,
+ .dev_reset = ixgbevf_dev_reset,
.allmulticast_enable = ixgbevf_dev_allmulticast_enable,
.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
.dev_infos_get = ixgbevf_dev_info_get,
@@ -2857,6 +2861,32 @@ ixgbe_dev_close(struct rte_eth_dev *dev)
ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV);
}

+/*
+ * Reest PF device.
+ */
+static int
+ixgbe_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to ixgbe PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_ixgbe_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbe_dev_init(dev);
+
+ return ret;
+}
+
static void
ixgbe_read_stats_registers(struct ixgbe_hw *hw,
struct ixgbe_hw_stats *hw_stats,
@@ -5058,6 +5088,23 @@ ixgbevf_dev_close(struct rte_eth_dev *dev)
ixgbevf_remove_mac_addr(dev, 0);
}

+/*
+ * Reset VF device
+ */
+static int
+ixgbevf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = eth_ixgbevf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_ixgbevf_dev_init(dev);
+
+ return ret;
+}
+
static void ixgbevf_set_vfta_all(struct rte_eth_dev *dev, bool on)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);

[Thomas Monjalon]

Post by Wei Dai
+/*
+ * Reest PF device.
+ */

Small typo here. I will fix it.

[Wei Dai]

Reset a NIC by calling dev_uninit() and then dev_init().
Go through the same way in NIC PCI remove without release
of ethdev resource and then NIC PCI probe function without
ethdev resource allocation.

Signed-off-by: Wei Dai <***@intel.com>
---
drivers/net/i40e/i40e_ethdev.c | 28 ++++++++++++++++++++++++++++
drivers/net/i40e/i40e_ethdev_vf.c | 19 +++++++++++++++++++
2 files changed, 47 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 9fcccda..1641e00 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -250,6 +250,7 @@ static int i40e_dev_configure(struct rte_eth_dev *dev);
static int i40e_dev_start(struct rte_eth_dev *dev);
static void i40e_dev_stop(struct rte_eth_dev *dev);
static void i40e_dev_close(struct rte_eth_dev *dev);
+static int i40e_dev_reset(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40e_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40e_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -449,6 +450,7 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
.dev_start = i40e_dev_start,
.dev_stop = i40e_dev_stop,
.dev_close = i40e_dev_close,
+ .dev_reset = i40e_dev_reset,
.promiscuous_enable = i40e_dev_promiscuous_enable,
.promiscuous_disable = i40e_dev_promiscuous_disable,
.allmulticast_enable = i40e_dev_allmulticast_enable,
@@ -2155,6 +2157,32 @@ i40e_dev_close(struct rte_eth_dev *dev)
I40E_WRITE_FLUSH(hw);
}

+/*
+ * Reest PF device only to re-initialize resources in PMD layer
+ */
+static int
+i40e_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ /* When a DPDK PMD PF begin to reset PF port, it should notify all
+ * its VF to make them align with it. The detailed notification
+ * mechanism is PMD specific. As to i40e PF, it is rather complex.
+ * To avoid unexpected behavior in VF, currently reset of PF with
+ * SR-IOV activation is not supported. It might be supported later.
+ */
+ if (dev->data->sriov.active)
+ return -ENOTSUP;
+
+ ret = eth_i40e_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = eth_i40e_dev_init(dev);
+
+ return ret;
+}
+
static void
i40e_dev_promiscuous_enable(struct rte_eth_dev *dev)
{
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index f6d8293..f951d4e 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -122,6 +122,7 @@ static void i40evf_vlan_offload_set(struct rte_eth_dev *dev, int mask);
static int i40evf_vlan_pvid_set(struct rte_eth_dev *dev, uint16_t pvid,
int on);
static void i40evf_dev_close(struct rte_eth_dev *dev);
+static int i40evf_dev_reset(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_enable(struct rte_eth_dev *dev);
static void i40evf_dev_promiscuous_disable(struct rte_eth_dev *dev);
static void i40evf_dev_allmulticast_enable(struct rte_eth_dev *dev);
@@ -203,6 +204,7 @@ static const struct eth_dev_ops i40evf_eth_dev_ops = {
.xstats_get_names = i40evf_dev_xstats_get_names,
.xstats_reset = i40evf_dev_xstats_reset,
.dev_close = i40evf_dev_close,
+ .dev_reset = i40evf_dev_reset,
.dev_infos_get = i40evf_dev_info_get,
.dev_supported_ptypes_get = i40e_dev_supported_ptypes_get,
.vlan_filter_set = i40evf_vlan_filter_set,
@@ -2373,6 +2375,23 @@ i40evf_dev_close(struct rte_eth_dev *dev)
i40evf_disable_irq0(hw);
}

+/*
+ * Reest VF device only to re-initialize resources in PMD layer
+ */
+static int
+i40evf_dev_reset(struct rte_eth_dev *dev)
+{
+ int ret;
+
+ ret = i40evf_dev_uninit(dev);
+ if (ret)
+ return ret;
+
+ ret = i40evf_dev_init(dev);
+
+ return ret;
+}
+
static int
i40evf_get_rss_lut(struct i40e_vsi *vsi, uint8_t *lut, uint16_t lut_size)
{

[Wei Dai]

When PF is reset, a message will show it and all its
VF need to be reset.
User can run the command "port reset port_id"
to reset the VF port and to keep same port id without
any configuration. Then user can run "port stop port_id"
and "port start port_id" to reconfigure its forwarding
mode and parmaters as previous ones.
To avoid crash, current forwarding should be stopped
before running "port reset port_id".

Signed-off-by: Wei Dai <***@intel.com>
Tested-by: Yuan Peng <***@intel.com>
Acked-by: Jingjing Wu <***@intel.com>
---
app/test-pmd/cmdline.c | 12 ++++++++----
app/test-pmd/testpmd.c | 41 +++++++++++++++++++++++++++++++++++++++++
app/test-pmd/testpmd.h | 1 +
3 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index b1b36c1..e37d6d7 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -972,6 +972,8 @@ static void cmd_operate_port_parsed(void *parsed_result,
stop_port(RTE_PORT_ALL);
else if (!strcmp(res->name, "close"))
close_port(RTE_PORT_ALL);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(RTE_PORT_ALL);
else
printf("Unknown parameter\n");
}
@@ -981,14 +983,14 @@ cmdline_parse_token_string_t cmd_operate_port_all_cmd =
"port");
cmdline_parse_token_string_t cmd_operate_port_all_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, name,
- "start#stop#close");
+ "start#stop#close#reset");
cmdline_parse_token_string_t cmd_operate_port_all_all =
TOKEN_STRING_INITIALIZER(struct cmd_operate_port_result, value, "all");

cmdline_parse_inst_t cmd_operate_port = {
.f = cmd_operate_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close all: Start/Stop/Close all ports",
+ .help_str = "port start|stop|close all: Start/Stop/Close/Reset all ports",
.tokens = {
(void *)&cmd_operate_port_all_cmd,
(void *)&cmd_operate_port_all_port,
@@ -1016,6 +1018,8 @@ static void cmd_operate_specific_port_parsed(void *parsed_result,
stop_port(res->value);
else if (!strcmp(res->name, "close"))
close_port(res->value);
+ else if (!strcmp(res->name, "reset"))
+ reset_port(res->value);
else
printf("Unknown parameter\n");
}
@@ -1025,7 +1029,7 @@ cmdline_parse_token_string_t cmd_operate_specific_port_cmd =
keyword, "port");
cmdline_parse_token_string_t cmd_operate_specific_port_port =
TOKEN_STRING_INITIALIZER(struct cmd_operate_specific_port_result,
- name, "start#stop#close");
+ name, "start#stop#close#reset");
cmdline_parse_token_num_t cmd_operate_specific_port_id =
TOKEN_NUM_INITIALIZER(struct cmd_operate_specific_port_result,
value, UINT8);
@@ -1033,7 +1037,7 @@ cmdline_parse_token_num_t cmd_operate_specific_port_id =
cmdline_parse_inst_t cmd_operate_specific_port = {
.f = cmd_operate_specific_port_parsed,
.data = NULL,
- .help_str = "port start|stop|close <port_id>: Start/Stop/Close port_id",
+ .help_str = "port start|stop|close <port_id>: Start/Stop/Close/Reset port_id",
.tokens = {
(void *)&cmd_operate_specific_port_cmd,
(void *)&cmd_operate_specific_port_port,
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index e754d12..59fc441 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -1683,6 +1683,47 @@ close_port(portid_t pid)
}

void
+reset_port(portid_t pid)
+{
+ int diag;
+ portid_t pi;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(pid, ENABLED_WARN))
+ return;
+
+ printf("Resetting ports...\n");
+
+ RTE_ETH_FOREACH_DEV(pi) {
+ if (pid != pi && pid != (portid_t)RTE_PORT_ALL)
+ continue;
+
+ if (port_is_forwarding(pi) != 0 && test_done == 0) {
+ printf("Please remove port %d from forwarding "
+ "configuration.\n", pi);
+ continue;
+ }
+
+ if (port_is_bonding_slave(pi)) {
+ printf("Please remove port %d from bonded device.\n",
+ pi);
+ continue;
+ }
+
+ diag = rte_eth_dev_reset(pi);
+ if (diag == 0) {
+ port = &ports[pi];
+ port->need_reconfig = 1;
+ port->need_reconfig_queues = 1;
+ } else {
+ printf("Failed to reset port %d. diag=%d\n", pi, diag);
+ }
+ }
+
+ printf("Done\n");
+}
+
+void
attach_port(char *identifier)
{
portid_t pi = 0;
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index c9d7739..1d1ee75 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -610,6 +610,7 @@ int init_port_dcb_config(portid_t pid, enum dcb_mode_enable dcb_mode,
int start_port(portid_t pid);
void stop_port(portid_t pid);
void close_port(portid_t pid);
+void reset_port(portid_t pid);
void attach_port(char *identifier);
void detach_port(uint8_t port_id);
int all_ports_stopped(void);

[Wei Dai]

This patch add the description of NIC reset API in
doc/guides/prog_guide/poll_mode_drv.rst .
It explains why this API is needed, when it should
be called and some noticeable information.

Signed-off-by: Wei Dai <***@intel.com>
Reviewed-by: Remy Horton <***@intel.com>
---
doc/guides/prog_guide/poll_mode_drv.rst | 41 +++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)

diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst
index 1ac8f7e..7c95b3e 100644
--- a/doc/guides/prog_guide/poll_mode_drv.rst
+++ b/doc/guides/prog_guide/poll_mode_drv.rst
@@ -536,3 +536,44 @@ call. As an end result, the application is able to achieve its goal of
monitoring a single statistic ("rx_errors" in this case), and if that shows
packets being dropped, it can easily retrieve a "set" of statistics using the
IDs array parameter to ``rte_eth_xstats_get_by_id`` function.
+
+NIC Reset API
+~~~~~~~~~~~~~
+
+.. code-block:: c
+
+ int rte_eth_dev_reset(uint8_t port_id);
+
+Sometimes a port has to be reset passively. For example when a PF is
+reset, all its VFs should also be reset by the application to make them
+consistent with the PF. A DPDK application also can call this function
+to trigger a port reset. Normally, a DPDK application would invokes this
+function when an RTE_ETH_EVENT_INTR_RESET event is detected.
+
+It is the duty of the PMD to trigger RTE_ETH_EVENT_INTR_RESET events and
+the application should register a callback function to handle these
+events. When a PMD needs to trigger a reset, it can trigger an
+RTE_ETH_EVENT_INTR_RESET event. On receiving an RTE_ETH_EVENT_INTR_RESET
+event, applications can handle it as follows: Stop working queues, stop
+calling Rx and Tx functions, and then call rte_eth_dev_reset( ). For
+thread safety all these operations should be called from the same thread.
+
+For example when PF is reset, the PF sends a message to notify VFs of
+this event and also trigger an interrupt to VFs. Then in the interrupt
+service routine the VFs detects this notification message and calls
+_rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL,
+NULL). This means that a PF reset triggers an RTE_ETH_EVENT_INTR_RESET
+event within VFs. The function _rte_eth_dev_callback_process( ) will
+call the registered callback function. The callback function can trigger
+the application to handle all operations the VF reset requires including
+stopping Rx/Tx queues and calling rte_eth_dev_reset().
+
+The rte_eth_dev_reset( ) itself is a generic function which only does
+some hardware reset operations through calling dev_unint() and
+dev_init(), and itself does not handle synchronization, which is handled
+by application.
+
+The PMD itself should not call rte_eth_dev_reset( ). The PMD can trigger
+the application to handle reset event. It is duty of application to
+handle all synchronization before it calls rte_eth_dev_reset( ).
+

[Thomas Monjalon]

Post by Wei Dai
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset
doc: add description of the NIC reset API

Sorry for being so late to get back to it.

Applied, thanks for the improvements done in latest versions.

[Dai, Wei]

Hi, Thomas

Thanks for your great support !

-----Original Message-----
Sent: Thursday, September 14, 2017 5:02 AM
Subject: Re: [dpdk-dev] [PATCH v9 0/5] Support of NIC reset and keep same
port id

Post by Wei Dai
ethdev: add support of NIC reset
net/ixgbe: add support of reset
net/i40e: add support of reset
app/testpmd: enhance command to test NIC reset
doc: add description of the NIC reset API

Sorry for being so late to get back to it.
Applied, thanks for the improvements done in latest versions.