zfs + NFS + FreeBSD with performance prob

[Albert Shih]

Hi all,

I'm not sure if the problem is with FreeBSD or ZFS or both so I cross-post
(I known it's bad).

Well I've server running FreeBSD 9.0 with (don't count / on differents
disks) zfs pool with 36 disk.

The performance is very very good on the server.

I've one NFS client running FreeBSD 8.3 and the performance over NFS is
very good :

For example : Read from the client and write over NFS to ZFS:

[root@ .tmp]# time tar xf /tmp/linux-3.7.5.tar

real 1m7.244s
user 0m0.921s
sys 0m8.990s

this client is on 1Gbits/s network cable and same network switch as the
server.

I've a second NFS client running FreeBSD 9.1-Stable, and on this second
client the performance is catastrophic. After 1 hour the tar isn't finish.
OK this second client is connect with 100Mbit/s and not on the same switch.
But well from 2 min --> ~ 90 min ...:-(

I've try for this second client to change on the ZFS-NFS server the

zfs set sync=disabled

and that change nothing.

On a third NFS client linux (recent Ubuntu) I got the almost same catastrophic
performance. With or without sync=disabled.

Those three NFS client use TCP.

If I do a classic scp I got normal speed ~9-10 Mbytes/s so the network is
not the problem.

I try to something like (find with google):

net.inet.tcp.sendbuf_max: 2097152 -> 16777216
net.inet.tcp.recvbuf_max: 2097152 -> 16777216
net.inet.tcp.sendspace: 32768 -> 262144
net.inet.tcp.recvspace: 65536 -> 262144
net.inet.tcp.mssdflt: 536 -> 1452
net.inet.udp.recvspace: 42080 -> 65535
net.inet.udp.maxdgram: 9216 -> 65535
net.local.stream.recvspace: 8192 -> 65535
net.local.stream.sendspace: 8192 -> 65535


and that change nothing either.

Anyone have any idea ?

Regards.

JAS
--
Albert SHIH
DIO bâtiment 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
Téléphone : 01 45 07 76 26/06 86 69 95 71
xmpp: ***@obspm.fr
Heure local/Local time:
jeu 31 jan 2013 23:04:47 CET

[Paul Kraus]

Post by Albert Shih
Well I've server running FreeBSD 9.0 with (don't count / on differents
disks) zfs pool with 36 disk.
The performance is very very good on the server.
I've one NFS client running FreeBSD 8.3 and the performance over NFS is
real 1m7.244s
user 0m0.921s
sys 0m8.990s
this client is on 1Gbits/s network cable and same network switch as the
server.
I've a second NFS client running FreeBSD 9.1-Stable, and on this second
client the performance is catastrophic. After 1 hour the tar isn't finish.
OK this second client is connect with 100Mbit/s and not on the same switch.
But well from 2 min --> ~ 90 min ...:-(
I've try for this second client to change on the ZFS-NFS server the
zfs set sync=disabled
and that change nothing.

I have been using FreeBSD 9 with ZFS and NFS to a couple Mac OS X (10.6.8 Snow Leopard) boxes and I get between 40 and 50 MB/sec throughput on a Gigabit ethernet link. Since you have already ruled out the known sync issue with ZFS and no SSD-based write cache, then perhaps you are running into an NFS 3 vs. NFS 4 issue. I am not sure if Mac OS X is using NFS 3 or NFS 4.

--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company

[Albert Shih]

Le 04/02/2013 ? 11:21:12-0500, Paul Kraus a écrit

Post by Paul Kraus

Post by Albert Shih
Well I've server running FreeBSD 9.0 with (don't count / on
differents disks) zfs pool with 36 disk.
The performance is very very good on the server.
I've one NFS client running FreeBSD 8.3 and the performance over NFS
real 1m7.244s user 0m0.921s sys 0m8.990s
this client is on 1Gbits/s network cable and same network switch as
the server.
I've a second NFS client running FreeBSD 9.1-Stable, and on this
second client the performance is catastrophic. After 1 hour the tar
isn't finish. OK this second client is connect with 100Mbit/s and
not on the same switch. But well from 2 min --> ~ 90 min ...:-(
I've try for this second client to change on the ZFS-NFS server the
zfs set sync=disabled
and that change nothing.

I have been using FreeBSD 9 with ZFS and NFS to a couple Mac OS X
(10.6.8 Snow Leopard) boxes and I get between 40 and 50 MB/sec

Thanks for your answer.

Can you give me the average ping time between you'r client and NFS server ?

Regards.

JAS
--
Albert SHIH
DIO bâtiment 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
Téléphone : 01 45 07 76 26/06 86 69 95 71
xmpp: ***@obspm.fr
Heure local/Local time:
mar 5 fév 2013 16:15:11 CET

[Sašo Kiselkov]

Post by Albert Shih
Hi all,
I'm not sure if the problem is with FreeBSD or ZFS or both so I cross-post
(I known it's bad).
Well I've server running FreeBSD 9.0 with (don't count / on differents
disks) zfs pool with 36 disk.
The performance is very very good on the server.
I've one NFS client running FreeBSD 8.3 and the performance over NFS is
real 1m7.244s
user 0m0.921s
sys 0m8.990s
this client is on 1Gbits/s network cable and same network switch as the
server.
I've a second NFS client running FreeBSD 9.1-Stable, and on this second
client the performance is catastrophic. After 1 hour the tar isn't finish.
OK this second client is connect with 100Mbit/s and not on the same switch.
But well from 2 min --> ~ 90 min ...:-(
I've try for this second client to change on the ZFS-NFS server the
zfs set sync=disabled
and that change nothing.
On a third NFS client linux (recent Ubuntu) I got the almost same catastrophic
performance. With or without sync=disabled.
Those three NFS client use TCP.
If I do a classic scp I got normal speed ~9-10 Mbytes/s so the network is
not the problem.
net.inet.tcp.sendbuf_max: 2097152 -> 16777216
net.inet.tcp.recvbuf_max: 2097152 -> 16777216
net.inet.tcp.sendspace: 32768 -> 262144
net.inet.tcp.recvspace: 65536 -> 262144
net.inet.tcp.mssdflt: 536 -> 1452
net.inet.udp.recvspace: 42080 -> 65535
net.inet.udp.maxdgram: 9216 -> 65535
net.local.stream.recvspace: 8192 -> 65535
net.local.stream.sendspace: 8192 -> 65535
and that change nothing either.
Anyone have any idea ?

What you describe sounds like a bad networking issue. Check your network
via the usual tools like ping, mtr, netperf, etc. Verify cabling and
interface counters on your machines too, for stuff like CRC errors or
jabbers - a few of those and the throughput of a TCP link goes down the
drain.

Cheers,
--
Saso

[Sašo Kiselkov]

Post by Sašo Kiselkov

Post by Albert Shih
Hi all,
I'm not sure if the problem is with FreeBSD or ZFS or both so I cross-post
(I known it's bad).
Well I've server running FreeBSD 9.0 with (don't count / on differents
disks) zfs pool with 36 disk.
The performance is very very good on the server.
I've one NFS client running FreeBSD 8.3 and the performance over NFS is
real 1m7.244s
user 0m0.921s
sys 0m8.990s
this client is on 1Gbits/s network cable and same network switch as the
server.
I've a second NFS client running FreeBSD 9.1-Stable, and on this second
client the performance is catastrophic. After 1 hour the tar isn't finish.
OK this second client is connect with 100Mbit/s and not on the same switch.
But well from 2 min --> ~ 90 min ...:-(
I've try for this second client to change on the ZFS-NFS server the
zfs set sync=disabled
and that change nothing.
On a third NFS client linux (recent Ubuntu) I got the almost same catastrophic
performance. With or without sync=disabled.
Those three NFS client use TCP.
If I do a classic scp I got normal speed ~9-10 Mbytes/s so the network is
not the problem.
net.inet.tcp.sendbuf_max: 2097152 -> 16777216
net.inet.tcp.recvbuf_max: 2097152 -> 16777216
net.inet.tcp.sendspace: 32768 -> 262144
net.inet.tcp.recvspace: 65536 -> 262144
net.inet.tcp.mssdflt: 536 -> 1452
net.inet.udp.recvspace: 42080 -> 65535
net.inet.udp.maxdgram: 9216 -> 65535
net.local.stream.recvspace: 8192 -> 65535
net.local.stream.sendspace: 8192 -> 65535
and that change nothing either.
Anyone have any idea ?

What you describe sounds like a bad networking issue. Check your network
via the usual tools like ping, mtr, netperf, etc. Verify cabling and
interface counters on your machines too, for stuff like CRC errors or
jabbers - a few of those and the throughput of a TCP link goes down the
drain.

Just one more thing: simply doing SCP need not show a problem. SCP is
very uni-directional, and you may be hitting an issue in the opposite
direction (I've seen TP cables where one pair was fine and the other was
giving bad data).

Also check for dropped packets on your source and target machines via
tools like DTrace.

Cheers,
--
Saso