Discussion:
[ofa-general] performance to call ibv_poll_cq() vs. call select() on completion channel
Tang, Changqing
2009-09-02 19:53:13 UTC
Permalink
Roland or Mellanox Engineers:

We setup completion channel for a completion queue. We want to check if there is any
event available, and suppose there is NO event on both completion channel and completion queue.

What we can do is:

1. call select() on completion channel with zero timeout and return 0.
2. call ibv_poll_cq() directly and return 0.

Question:

Which way has lower overhead ? We know select() has to switch to kernel mode, does ibv_poll_cq()
switch to kernel mode and may cause a context switch ?


Thanks.
--CQ Tang
Tziporet Koren
2009-09-02 20:10:09 UTC
Permalink
Post by Tang, Changqing
We setup completion channel for a completion queue. We want to check if there is any
event available, and suppose there is NO event on both completion channel and completion queue.
1. call select() on completion channel with zero timeout and return 0.
2. call ibv_poll_cq() directly and return 0.
Which way has lower overhead ?
ibv_poll_cq
Post by Tang, Changqing
We know select() has to switch to kernel mode, does ibv_poll_cq()
switch to kernel mode and may cause a context switch ?
No - it's pure user space function

Tziporet
Tang, Changqing
2009-09-02 22:17:37 UTC
Permalink
Post by Tang, Changqing
Post by Tang, Changqing
1. call select() on completion channel with zero
timeout and return 0.
Post by Tang, Changqing
2. call ibv_poll_cq() directly and return 0.
Which way has lower overhead ?
ibv_poll_cq
Post by Tang, Changqing
We know select() has to switch to kernel mode, does ibv_poll_cq()
switch to kernel mode and may cause a context switch ?
No - it's pure user space function
But I just check the source code, ibv_poll_cq() is actually ibv_cmd_poll_cq(),
and ibv_cmd_poll_cq() calls write() system call on the IB device.

Doesn't this write() system call switch to kernel mode and possiblely casuse
a context switch ?


--CQ
Post by Tang, Changqing
Tziporet
Sean Hefty
2009-09-02 22:25:08 UTC
Permalink
Post by Tang, Changqing
But I just check the source code, ibv_poll_cq() is actually ibv_cmd_poll_cq(),
and ibv_cmd_poll_cq() calls write() system call on the IB device.
Doesn't this write() system call switch to kernel mode and possiblely casuse
a context switch ?
See verbs.h:

static inline int ibv_poll_cq(struct ibv_cq *cq, int num_entries, struct ibv_wc
*wc)
{
return cq->context->ops.poll_cq(cq, num_entries, wc);
}

The userspace provider library sets poll_cq to an internal call.

- Sean
Tang, Changqing
2009-09-02 23:00:11 UTC
Permalink
Sean:
I understand that ops.poll_cq is actually ibv_cmd_poll_cq(), right ?

Do you mean during ibv_poll_cq() call, there is no system call involved ?

--CQ
-----Original Message-----
Sent: Wednesday, September 02, 2009 5:25 PM
Subject: RE: [ofa-general] performance to call ibv_poll_cq()
vs. call select() on completion channel
Post by Tang, Changqing
But I just check the source code, ibv_poll_cq() is actually
ibv_cmd_poll_cq(), and ibv_cmd_poll_cq() calls write()
system call on the IB device.
Post by Tang, Changqing
Doesn't this write() system call switch to kernel mode and
possiblely
Post by Tang, Changqing
casuse a context switch ?
static inline int ibv_poll_cq(struct ibv_cq *cq, int
num_entries, struct ibv_wc
*wc)
{
return cq->context->ops.poll_cq(cq, num_entries, wc); }
The userspace provider library sets poll_cq to an internal call.
- Sean
Roland Dreier
2009-09-02 23:08:39 UTC
Permalink
Post by Tang, Changqing
I understand that ops.poll_cq is actually ibv_cmd_poll_cq(), right ?
No, not for most devices. Look at libmthca, etc to see what the poll_cq
method is set to.
Post by Tang, Changqing
Do you mean during ibv_poll_cq() call, there is no system call involved ?
Right, for most devices poll_cq can be done just by looking at memory in
userspace without involving the kernel at all.
Tang, Changqing
2009-09-02 23:13:20 UTC
Permalink
I did not understand the relation between ops.poll_cq() and ibv_cmd_poll_cq() correctly.

It is clear now.

Thank you.

--CQ
-----Original Message-----
Sent: Wednesday, September 02, 2009 6:09 PM
To: Tang, Changqing
Subject: Re: [ofa-general] performance to call ibv_poll_cq()
vs. call select() on completion channel
Post by Tang, Changqing
I understand that ops.poll_cq is actually
ibv_cmd_poll_cq(), right ?
No, not for most devices. Look at libmthca, etc to see what
the poll_cq method is set to.
Post by Tang, Changqing
Do you mean during ibv_poll_cq() call, there is no
system call involved ?
Right, for most devices poll_cq can be done just by looking
at memory in userspace without involving the kernel at all.
Loading...