[Gluster-users] What about ZFS ?

[Germain Maurice]

Hi everybody,

I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.

What are the latest news about this ?

Thank you in advance.

[Dipeit]

we have this running with zfsonlinux and glusterfs 3.2.5 and are using a 60TB volume across 3 storage server. In the last 6 months we had one unexplained reboot of one of these. Cause unknown. Other than that it was fast and stable

Dipe

On Feb 22, 2012, at 5:26 AM, Germain Maurice <***@linkfluence.net> wrote:

> Hi everybody,
>
> I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.
>
> What are the latest news about this ?
>
> Thank you in advance.
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Lars Erik Dangvard Jensen]

Den 22/02/2012 kl. 16.46 skrev Dipeit:

> we have this running with zfsonlinux and glusterfs 3.2.5 and are using a 60TB volume across 3 storage server. In the last 6 months we had one unexplained reboot of one of these. Cause unknown. Other than that it was fast and stable
>
> Dipe

How did you manage make symlinks work, have you patched ZFS? Or maybe you don't use symlinks :)

/Lars

[Di Pe]

On Wed, Feb 22, 2012 at 7:56 AM, Lars Erik Dangvard Jensen
<***@dcmediahosting.com> wrote:
> Den 22/02/2012 kl. 16.46 skrev Dipeit:
>
>> we have this running with zfsonlinux and glusterfs 3.2.5 and are using a 60TB volume across 3 storage server. In the last 6 months we had one unexplained reboot of one of these. Cause unknown. Other than that it was fast and stable
>>
>> Dipe
>
> How did you manage make symlinks work, have you patched ZFS? Or maybe you don't use symlinks :)

true symlinks would be very nice .... but we don't use this as general
purpose storage right now, more like a bitbucket for large files and
to experiment with low cost consumer grade sata drives

>
> /Lars
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Shain Miley]

Are you able to talk about the performance you have been seeing using
this setup?

I have been testing zfs + gluster using some kvm vm's, and I have found
it to be stable but kind of slow.

However I believe that the cause of my slowness has more to do with my
setup, not necessarily anything to do with zfs or gluster (outside of
the fact they both use fuse and thus introduce some additional overhead).

Thanks,

Shain

On 02/22/2012 10:46 AM, Dipeit wrote:
> we have this running with zfsonlinux and glusterfs 3.2.5 and are using a 60TB volume across 3 storage server. In the last 6 months we had one unexplained reboot of one of these. Cause unknown. Other than that it was fast and stable
>
> Dipe
>
> On Feb 22, 2012, at 5:26 AM, Germain Maurice<***@linkfluence.net> wrote:
>
>> Hi everybody,
>>
>> I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.
>>
>> What are the latest news about this ?
>>
>> Thank you in advance.
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-***@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>

[Anh Quach]

Shain,

I was just reading your blog here: http://www.shainmiley.com/wordpress/2011/01/17/gluster-on-opensolaris-so-far-part-1/

in which you say " if you choose to use a Solaris variant for your storage nodes, you will be required to use a head node with an OS that does support it on the client side."

Would you mind clarifying that? Does that mean that one of the Gluster nodes has to be Linux in order to export NFS to clients?

Thanks in advance, and thanks for sharing great stuff on your blog.

Anh Quach
Technical Director

DIGITALKITCHEN | 8695 Washington Blvd Suite 206, Culver City, CA 90232 | V 310.499.9255 F 310.559.6647 | THISISDK.COM



On Feb 22, 2012, at 12:07 PM, Shain Miley wrote:

> Are you able to talk about the performance you have been seeing using this setup?
>
> I have been testing zfs + gluster using some kvm vm's, and I have found it to be stable but kind of slow.
>
> However I believe that the cause of my slowness has more to do with my setup, not necessarily anything to do with zfs or gluster (outside of the fact they both use fuse and thus introduce some additional overhead).
>
> Thanks,
>
> Shain
>
> On 02/22/2012 10:46 AM, Dipeit wrote:
>> we have this running with zfsonlinux and glusterfs 3.2.5 and are using a 60TB volume across 3 storage server. In the last 6 months we had one unexplained reboot of one of these. Cause unknown. Other than that it was fast and stable
>>
>> Dipe
>>
>> On Feb 22, 2012, at 5:26 AM, Germain Maurice<***@linkfluence.net> wrote:
>>
>>> Hi everybody,
>>>
>>> I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.
>>>
>>> What are the latest news about this ?
>>>
>>> Thank you in advance.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-***@gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-***@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Shain Miley]

Anh,
What I mean by that is that the gluster client does not work on Solaris variants. So if you want to use Solaris or Opensolaris for example, you can...for your storage bricks...however when you want to mount and access the files...you need to have a linux headnode (we are using VM's) in order to run the gluster client.

Shain



________________________________________
From: Anh Quach [***@blackandcode.com]
Sent: Wednesday, February 22, 2012 3:11 PM
To: Shain Miley
Cc: Dipeit; gluster-***@gluster.org
Subject: Re: [Gluster-users] What about ZFS ?

Shain,

I was just reading your blog here: http://www.shainmiley.com/wordpress/2011/01/17/gluster-on-opensolaris-so-far-part-1/

in which you say " if you choose to use a Solaris variant for your storage nodes, you will be required to use a head node with an OS that does support it on the client side."

Would you mind clarifying that? Does that mean that one of the Gluster nodes has to be Linux in order to export NFS to clients?

Thanks in advance, and thanks for sharing great stuff on your blog.

Anh Quach
Technical Director

DIGITALKITCHEN | 8695 Washington Blvd Suite 206, Culver City, CA 90232 | V 310.499.9255 F 310.559.6647 | THISISDK.COM<http://www.d-kitchen.com/>



On Feb 22, 2012, at 12:07 PM, Shain Miley wrote:

Are you able to talk about the performance you have been seeing using this setup?

I have been testing zfs + gluster using some kvm vm's, and I have found it to be stable but kind of slow.

However I believe that the cause of my slowness has more to do with my setup, not necessarily anything to do with zfs or gluster (outside of the fact they both use fuse and thus introduce some additional overhead).

Thanks,

Shain

On 02/22/2012 10:46 AM, Dipeit wrote:
we have this running with zfsonlinux and glusterfs 3.2.5 and are using a 60TB volume across 3 storage server. In the last 6 months we had one unexplained reboot of one of these. Cause unknown. Other than that it was fast and stable

Dipe

On Feb 22, 2012, at 5:26 AM, Germain Maurice<***@linkfluence.net<mailto:***@linkfluence.net>> wrote:

Hi everybody,

I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.

What are the latest news about this ?

Thank you in advance.
_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Anh Quach]

Sorry for the newb questions… I'm pretty new to Gluster. If I'm understanding correctly, you can /create/ a Gluster volume with Solaris/Opensolaris ZFS bricks, but you can't actually /mount/ the filesystem unless you've got a node that can run the native GlusterFS client - hence no access to the volume for any read/write operations w/out the Linux head. That head node would then export NFS should you choose to… ?

Have you had any luck w/ zfsonlinux (or anyone else on this list..)?

Thanks!


Anh Quach
Technical Director

DIGITALKITCHEN | 8695 Washington Blvd Suite 206, Culver City, CA 90232 | V 310.499.9255 F 310.559.6647 | THISISDK.COM

On Feb 22, 2012, at 12:21 PM, Shain Miley wrote:

> Anh,
> What I mean by that is that the gluster client does not work on Solaris variants. So if you want to use Solaris or Opensolaris for example, you can...for your storage bricks...however when you want to mount and access the files...you need to have a linux headnode (we are using VM's) in order to run the gluster client.
>
> Shain
>
>
>
> ________________________________________
> From: Anh Quach [***@blackandcode.com]
> Sent: Wednesday, February 22, 2012 3:11 PM
> To: Shain Miley
> Cc: Dipeit; gluster-***@gluster.org
> Subject: Re: [Gluster-users] What about ZFS ?
>
> Shain,
>
> I was just reading your blog here: http://www.shainmiley.com/wordpress/2011/01/17/gluster-on-opensolaris-so-far-part-1/
>
> in which you say " if you choose to use a Solaris variant for your storage nodes, you will be required to use a head node with an OS that does support it on the client side."
>
> Would you mind clarifying that? Does that mean that one of the Gluster nodes has to be Linux in order to export NFS to clients?
>
> Thanks in advance, and thanks for sharing great stuff on your blog.
>
> Anh Quach
> Technical Director
>
> DIGITALKITCHEN | 8695 Washington Blvd Suite 206, Culver City, CA 90232 | V 310.499.9255 F 310.559.6647 | THISISDK.COM<http://www.d-kitchen.com/>
>
>
>
> On Feb 22, 2012, at 12:07 PM, Shain Miley wrote:
>
> Are you able to talk about the performance you have been seeing using this setup?
>
> I have been testing zfs + gluster using some kvm vm's, and I have found it to be stable but kind of slow.
>
> However I believe that the cause of my slowness has more to do with my setup, not necessarily anything to do with zfs or gluster (outside of the fact they both use fuse and thus introduce some additional overhead).
>
> Thanks,
>
> Shain
>
> On 02/22/2012 10:46 AM, Dipeit wrote:
> we have this running with zfsonlinux and glusterfs 3.2.5 and are using a 60TB volume across 3 storage server. In the last 6 months we had one unexplained reboot of one of these. Cause unknown. Other than that it was fast and stable
>
> Dipe
>
> On Feb 22, 2012, at 5:26 AM, Germain Maurice<***@linkfluence.net<mailto:***@linkfluence.net>> wrote:
>
> Hi everybody,
>
> I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.
>
> What are the latest news about this ?
>
> Thank you in advance.
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>

[Germain]

My question was a bit on this :)


Le 22/02/12 21:55, Anh Quach a écrit :
> Have you had any luck w/ zfsonlinux (or anyone else on this list..)?

[Shain Miley]

I guess I should have been more clear in my answer below:



I have been testing zfsonlinux + gluster using some kvm vm's, and I have found
it to be stable but kind of slow.

However I believe that the cause of my slowness has more to do with my
setup, not necessarily anything to do with zfsonlinux or gluster (outside of
the fact they both use fuse and thus introduce some additional overhead).

Thanks,

Shain

________________________________________
From: gluster-users-***@gluster.org [gluster-users-***@gluster.org] On Behalf Of Germain [***@linkfluence.net]
Sent: Wednesday, February 22, 2012 4:05 PM
To: gluster-***@gluster.org
Subject: Re: [Gluster-users] What about ZFS ?

My question was a bit on this :)


Le 22/02/12 21:55, Anh Quach a écrit :
> Have you had any luck w/ zfsonlinux (or anyone else on this list..)?

[Di Pe]

On Wed, Feb 22, 2012 at 12:07 PM, Shain Miley <***@npr.org> wrote:
> Are you able to talk about the performance you have been seeing using this
> setup?
>
> I have been testing zfs + gluster using some kvm vm's, and I have found it
> to be stable but kind of slow.
>
> However I believe that the cause of my slowness has more to do with my
> setup, not necessarily anything to do with zfs or gluster (outside of the
> fact they both use fuse and thus introduce some additional overhead).

I posted some of my early experience back in October

http://permalink.gmane.org/gmane.comp.file-systems.gluster.user/7445

gluster performance is quite reasonable and given that we use lots of
5400 rpm drives zfs performance is really great. I If you are able to
use PCI SSD as L2ARC you can get a very snappy file system.

I wanted to see if we could build a supportable production storage
system for under $200/TiB usable and it appears that this is actually
possible.

dipe



>
> Thanks,
>
> Shain
>
>
> On 02/22/2012 10:46 AM, Dipeit wrote:
>>
>> we have this running with zfsonlinux and glusterfs 3.2.5 and are using a
>> 60TB volume across 3 storage server. In the last 6 months we had one
>> unexplained reboot of one of these. Cause unknown. Other than that it was
>> fast and stable
>>
>> Dipe
>>
>> On Feb 22, 2012, at 5:26 AM, Germain Maurice<***@linkfluence.net>
>>  wrote:
>>
>>> Hi everybody,
>>>
>>> I'm looking for information about using GlusterFS with ZFS. I got
>>> information that talked about a sort of incompatibility between the both
>>> technologies because of unsupported xattr feature in ZFS.
>>>
>>> What are the latest news about this ?
>>>
>>> Thank you in advance.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-***@gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-***@gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>
>

[Di Pe]

I should add that if you really want great performance (random,
metadata, throughput) you may want to use this http://www.fhgfs.com/
We installed it next to the gluster folder on our zfs volumes. fhgfs
has better performance than gluster but backup and DR is much more
difficult. However they run excellent side by side.

dipe

[Anh Quach]

I'm very interested in this topic as well.

Is it possible to configure a Gluster cluster using Solaris-based ZFS pools as the underlying storage bricks? If anyone has any experience doing this and would like to share, it would be greatly appreciated. Googling around doesn't turn up much. I think that the Gluster native client isn't stable on Solaris-based systems yet, but does that mean Solaris systems can't be used as storage nodes? If it is possible, can you then export the Gluster volume using NFS?

Thanks in advance,

Anh Quach
Technical Director

DIGITALKITCHEN | 8695 Washington Blvd Suite 206, Culver City, CA 90232 | V 310.499.9255 F 310.559.6647 | THISISDK.COM


On Feb 22, 2012, at 5:26 AM, Germain Maurice wrote:

> Hi everybody,
>
> I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.
>
> What are the latest news about this ?
>
> Thank you in advance.
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Shain Miley]

As far as I know the last version of gluster to compile and run without too much problem is version 3.0.5.

This is the version I have been using in production for over a year.

The setup is stable, however 3.0.5 lacks a lot of the new features that appeared in 3.1 - 3.3, etc....so depending on what your needs are it is possible to do.

I am currently looking at moving away from this platform, mainly because Opensolaris is not actively being developed anymore (outside of illumos, openindiana.org, etc) and I would like take advantage of all of the bugfixes
and development work that has taken place since 3.0.5.

The best case scenario would be for a mature zfs of btrfs option to emerge at some point soon.

Shain


________________________________________
From: gluster-users-***@gluster.org [gluster-users-***@gluster.org] On Behalf Of Anh Quach [***@blackandcode.com]
Sent: Wednesday, February 22, 2012 3:05 PM
To: Germain Maurice
Cc: gluster-***@gluster.org
Subject: Re: [Gluster-users] What about ZFS ?

I'm very interested in this topic as well.

Is it possible to configure a Gluster cluster using Solaris-based ZFS pools as the underlying storage bricks? If anyone has any experience doing this and would like to share, it would be greatly appreciated. Googling around doesn't turn up much. I think that the Gluster native client isn't stable on Solaris-based systems yet, but does that mean Solaris systems can't be used as storage nodes? If it is possible, can you then export the Gluster volume using NFS?

Thanks in advance,

Anh Quach
Technical Director

DIGITALKITCHEN | 8695 Washington Blvd Suite 206, Culver City, CA 90232 | V 310.499.9255 F 310.559.6647 | THISISDK.COM<http://www.d-kitchen.com/>


On Feb 22, 2012, at 5:26 AM, Germain Maurice wrote:

Hi everybody,

I'm looking for information about using GlusterFS with ZFS. I got information that talked about a sort of incompatibility between the both technologies because of unsupported xattr feature in ZFS.

What are the latest news about this ?

Thank you in advance.