[Gluster-users] 3.7.16 with sharding corrupts VMDK files when adding and removing bricks

[ML Wong]

Have anyone encounter this behavior?

Running 3.7.16 from centos-gluster37, on CentOS 7.2 with NFS-Ganesha 2.3.0.
VMs are running fine without problems and with Sharding on. However, when i
either do a "add-brick" or "remove-brick start force". VM files will then
be corrupted, and the VM will not be able to boot anymore.

So far, as i access files through regular NFS, all regular files, or
directories seems to be accessible fine. I am not sure if this somehow
relate to bug1318136, but any help will be appreciated. Or, m i missing any
settings? Below is the vol info of gluster volume.

Volume Name: nfsvol1
Type: Distributed-Replicate
Volume ID: 06786467-4c8a-48ad-8b1f-346aa8342283
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: stor4:/data/brick1/nfsvol1
Brick2: stor5:/data/brick1/nfsvol1
Brick3: stor1:/data/brick1/nfsvol1
Brick4: stor2:/data/brick1/nfsvol1
Options Reconfigured:
features.shard-block-size: 64MB
features.shard: on
ganesha.enable: on
features.cache-invalidation: off
nfs.disable: on
performance.readdir-ahead: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable

thanks,
Melvin

[Krutika Dhananjay]

Hi,

Yes, this has been reported before by Lindsay Mathieson and Kevin Lemonnier
on this list.
We just found one issue with replace-brick that we recently fixed.

In your case, are you doing add-brick and changing the replica count (say
from 2 -> 3) or are you adding
"replica-count" number of bricks every time?

-Krutika

Post by ML Wong
Have anyone encounter this behavior?
Running 3.7.16 from centos-gluster37, on CentOS 7.2 with NFS-Ganesha
2.3.0. VMs are running fine without problems and with Sharding on. However,
when i either do a "add-brick" or "remove-brick start force". VM files will
then be corrupted, and the VM will not be able to boot anymore.
So far, as i access files through regular NFS, all regular files, or
directories seems to be accessible fine. I am not sure if this somehow
relate to bug1318136, but any help will be appreciated. Or, m i missing any
settings? Below is the vol info of gluster volume.
Volume Name: nfsvol1
Type: Distributed-Replicate
Volume ID: 06786467-4c8a-48ad-8b1f-346aa8342283
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Brick1: stor4:/data/brick1/nfsvol1
Brick2: stor5:/data/brick1/nfsvol1
Brick3: stor1:/data/brick1/nfsvol1
Brick4: stor2:/data/brick1/nfsvol1
features.shard-block-size: 64MB
features.shard: on
ganesha.enable: on
features.cache-invalidation: off
nfs.disable: on
performance.readdir-ahead: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable
thanks,
Melvin
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Gandalf Corvotempesta]

Post by Krutika Dhananjay
Hi,
Yes, this has been reported before by Lindsay Mathieson and Kevin

Lemonnier on this list.

Post by Krutika Dhananjay
We just found one issue with replace-brick that we recently fixed.

Don't get me wrong but I'm seeing too many "critical" issues like file
corruptions, crashes or similiar recently
Is gluster ready for production?
I'm scared about placing our production VMs (more or less 80) on gluster,
in case of corruption I'll loose everything

[Kevin Lemonnier]

Post by Gandalf Corvotempesta
Don't get me wrong but I'm seeing too many "critical" issues like file
corruptions, crashes or similiar recently
Is gluster ready for production?
I'm scared about placing our production VMs (more or less 80) on gluster,
in case of corruption I'll loose everything

We've had a lot of problems in the past, but at least for us 3.7.12 (and 3.7.15)
seems to be working pretty well as long as you don't add bricks. We started doing
multiple little clusters and abandonned the idea of one big cluster, had no
issues since :)

[Gandalf Corvotempesta]

Post by Kevin Lemonnier
We've had a lot of problems in the past, but at least for us 3.7.12 (and 3.7.15)
seems to be working pretty well as long as you don't add bricks. We started doing
multiple little clusters and abandonned the idea of one big cluster, had no
issues since :)

Well, adding bricks could be usefull... :)

Having to create multiple cluster is not a solution and is much more
expansive.
And if you corrupt data from a single cluster you still have issues

I think would be better to add less features and focus more to stability.
In a software defined storage, stability and consistency are the most
important things

I'm also subscribed to moosefs and lizardfs mailing list and I don't recall
any single data corruption/data loss event

In gluster, after some days of testing I've found a huge data corruption
issue that is still unfixed on bugzilla.
If you change the shard size on a populated cluster, you break all
existing data.
Try to do this on a cluster with working VMs and see what happens....
a single cli command break everything and is still unfixed.

[Kevin Lemonnier]

Post by Gandalf Corvotempesta
Having to create multiple cluster is not a solution and is much more
expansive.
And if you corrupt data from a single cluster you still have issues

Sure, but thinking about it later we realised that it might be for the better.
I believe when sharding is enabled the shards will be dispersed across all the
replica sets, making it that losing a replica set will kill all your VMs.

Imagine a 16x3 volume for example, losing 2 bricks could bring the whole thing
down if they happen to be in the same replica set. (I might be wrong about the
way gluster disperse shards, it's my understanding only, never had the chance
to test it).
With multiple small clusters, we have the same disk space in the end but not
that problem, it's a bit more annoying to manage but for now that's allright.

Post by Gandalf Corvotempesta
I'm also subscribed to moosefs and lizardfs mailing list and I don't
recall any single data corruption/data loss event

Never used those, might be just because there are less users ? Really have no idea,
maybe you are right.

Post by Gandalf Corvotempesta
If you change the shard size on a populated cluster,A you break all
existing data.

Not really shocked there. Guess the cli should warn you when you try re-setting
the option though, that would be nice.

[Gandalf Corvotempesta]

Post by Kevin Lemonnier
Sure, but thinking about it later we realised that it might be for the better.
I believe when sharding is enabled the shards will be dispersed across all the
replica sets, making it that losing a replica set will kill all your VMs.
Imagine a 16x3 volume for example, losing 2 bricks could bring the whole thing
down if they happen to be in the same replica set. (I might be wrong about the
way gluster disperse shards, it's my understanding only, never had the chance
to test it).
With multiple small clusters, we have the same disk space in the end but not
that problem, it's a bit more annoying to manage but for now that's allright.

I don't use EC because i really love the "gluster feature" to have plain
files stored and not encoded in any way.

Post by Kevin Lemonnier
Not really shocked there. Guess the cli should warn you when you try re-setting
the option though, that would be nice.

Exactly. I've proposed a warning in the cli when changing the shard size
but this is still unfixed and this is scaring me
it's a critical bug, IMHO, and should be addressed asap or any user could
destroy the whole cluster with a simple command and no warning at all.

[Lindsay Mathieson]

Post by Gandalf Corvotempesta
Exactly. I've proposed a warning in the cli when changing the shard
size but this is still unfixed and this is scaring me
it's a critical bug, IMHO, and should be addressed asap or any user
could destroy the whole cluster with a simple command and no warning
at all.

gluster volume reset <volname> *finger twitch*


And boom! volume gone.

Feature request: Ability to *lock* volume settings

[Gandalf Corvotempesta]

Post by Lindsay Mathieson
gluster volume reset <volname> *finger twitch*
And boom! volume gone.

There are too many destructive operations in gluster :)

More security on stored data please!

[David Gossage]

On Sat, Nov 12, 2016 at 7:42 AM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Lindsay Mathieson
gluster volume reset <volname> *finger twitch*
And boom! volume gone.

There are too many destructive operations in gluster :)
More security on stored data please!

I like Lindsay's idea of a lock setting or a flag that adds a confirmation
response requirement after every gluster command run.

also maybe a code monkey to sit at my keyboard and screech at me whenever I
type sudo so I pay attention to what I am about to do.

Post by Gandalf Corvotempesta
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Gandalf Corvotempesta]

Post by David Gossage
also maybe a code monkey to sit at my keyboard and screech at me whenever

I type sudo so I pay attention to what I am about to do.
Obviously yes, but for destructive operation a confirm should always asked
almost every software does it.

In example, with mdadm you can't remove a disk from the array without
failing it first. This is a sort of confirm.

Many other operation require a double confirm by the user

[Alex Crow]

Post by Kevin Lemonnier
Sure, but thinking about it later we realised that it might be for the better.
I believe when sharding is enabled the shards will be dispersed across all the
replica sets, making it that losing a replica set will kill all your VMs.
Imagine a 16x3 volume for example, losing 2 bricks could bring the whole thing
down if they happen to be in the same replica set. (I might be wrong about the
way gluster disperse shards, it's my understanding only, never had the chance
to test it).
With multiple small clusters, we have the same disk space in the end but not
that problem, it's a bit more annoying to manage but for now that's allright.

Post by Gandalf Corvotempesta
I'm also subscribed to moosefs and lizardfs mailing list and I don't
recall any single data corruption/data loss event

Never used those, might be just because there are less users ? Really have no idea,
maybe you are right.

I can add to this. I've been using MooseFS for general file storage with
Samba for over a year now for >25 million files shared to 350+ users.

I've *never* lost even a single file. We had some issues with
permissions but that needed a couple of lines added to our smb.conf
(CTDB cluster).

On the other hand at home, I tried to use GlusterFS for VM images in a
simple replica 2 setup with Pacemaker for HA. VMs were constantly
failing en masse even without making any changes. Very often the images
got corrupted and had to be restored from backups. This was over a year
ago but motivated me to try the VMs on MooseFS. Since then I've not had
a single problem with unexpected downtime or corruption.

It's not the fastest FS in the world but it's well balanced and has a
focus on consistency and reliability, Documentation clearly explains
where all the chunks of your files will be so you can clearly define
your resilience and recovery strategies. IMHO GlusterFS would be a great
product if it tried to:

a) Add less features per release, and/or slowing down the release cycle.
Maybe have a "Feature" branch like RozoFS, with a separate Stable and
Testing/Current. Stable is safe, Testing is risky, and "Feature" is for
those that need to try new, well, features.
b) Concentrate on issues like split-brain, healing, and scaling online
without data loss. Seems to be a common theme on the list where healing
doesn't work without tinkering. It should really "just work".
c) Have a peek at BeeGFS. It's a very well-performing FS that has its
focus on HPC. You can't stand to lose many thousands of CPU-hours of
work if your FS goes down, and it has to be fast.

The biggest question for me is what is the target market for GlusterFS?
Is it:

HPC (performance, reliability on the large scale, ie loss of one file is
OK, all not, no funky features)
VM storage (much the same as HPC but large file performance required,no
loss or corruption of blocks within a file)
General File (medium performance OK, small file and random access
paramount, resilience and consistency need to be 99.999%, features such
as ACLs and XATTRs, snapshots required)

i think if the documentation/wiki addressed these questions it would
make it easier for newcomers to evaluate the product.

Post by Kevin Lemonnier

Post by Gandalf Corvotempesta
If you change the shard size on a populated cluster,A you break all
existing data.

This needs to be a warning or clearly documented. If you lose a couple
of PB of data in a professional role, I'd not fancy your employment
prospects. I've always had the feeling that GlusterFS is a bit of a
playground for new features and the only way to really have a stable
storage system is to stump up the cash to RedHat (and we've purchased a
lot of RHEL/RHEV licences), but having so many problems in the community
version really even puts me off buying the full package!

Cheers

Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: [phone] Fax: [phone].
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[Kevin Lemonnier]

Post by Alex Crow
On the other hand at home, I tried to use GlusterFS for VM images in a
simple replica 2 setup with Pacemaker for HA. VMs were constantly
failing en masse even without making any changes. Very often the images
got corrupted and had to be restored from backups. This was over a year
ago but motivated me to try the VMs on MooseFS. Since then I've not had

Yeah, there has been a lot of bad versions for VM, anything < 3.7.12 has
either huge heal problems or random corruption at runtime. That's why
I keep 3.7.12 everywhere, I know it works well at least with my config,
and since I have no use for the new feature why take the risk to update ?

Interesting comments on MooseFS, I've seen it but never tried it yet
because of the single server managing the cluster, seems like a huge
risk. Guess there must be ways to have that role failover or something.

[David Gossage]

Post by Kevin Lemonnier

Post by Alex Crow
On the other hand at home, I tried to use GlusterFS for VM images in a
simple replica 2 setup with Pacemaker for HA. VMs were constantly
failing en masse even without making any changes. Very often the images
got corrupted and had to be restored from backups. This was over a year
ago but motivated me to try the VMs on MooseFS. Since then I've not had

Yeah, there has been a lot of bad versions for VM, anything < 3.7.12 has
either huge heal problems or random corruption at runtime. That's why
I keep 3.7.12 everywhere, I know it works well at least with my config,
and since I have no use for the new feature why take the risk to update ?
Interesting comments on MooseFS, I've seen it but never tried it yet
because of the single server managing the cluster, seems like a huge
risk. Guess there must be ways to have that role failover or something.

I've locked myself more or less into the version of 3.8 I am running on as
well right now as I just don't feel inclined to have to worry if anything
in an update will give me issues.

--

Post by Kevin Lemonnier
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Gandalf Corvotempesta]

Il 12 nov 2016 9:04 PM, "Alex Crow" <***@integrafin.co.uk> ha scritto:
IMHO GlusterFS would be a great

Post by Alex Crow
a) Add less features per release, and/or slowing down the release cycle.
Maybe have a "Feature"
those that need to try new, well, features.
b) Concentrate on issues like split-brain, healing, and scaling online
without data loss. Seems to be a common theme on the list where healing
doesn't work without tinkering. It should really "just work".
c) Have a peek at BeeGFS. It's a very well-performing FS that has its
focus on HPC. You can't stand to lose many thousands of CPU-hours of
work if your FS goes down, and it has to be fast.
The biggest question for me is what is the target market for GlusterFS?
HPC (performance, reliability on the large scale, ie loss of one file is
OK, all not, no funky features)
VM storage (much the same as HPC but large file performance required,no
loss or corruption of blocks within a file)
General File (medium performance OK, small file and random access
paramount, resilience and consistency need to be 99.999%, features such
as ACLs and XATTRs, snapshots required)
i think if the documentation/wiki addressed these questions it would
make it easier for newcomers to evaluate the product.

Totally agree

Post by Alex Crow
This needs to be a warning or clearly documented. If you lose a couple
of PB of data in a professional role, I'd not fancy your employment
prospects. I've always had the feeling that GlusterFS is a bit of a
playground for new features and the only way to really have a stable
storage system is to stump up the cash to RedHat (and we've purchased a
lot of RHEL/RHEV licences), but having so many problems in the community
version really even puts me off buying the full package!

Again, totally agree

[Pranith Kumar Karampuri]

On Sat, Nov 12, 2016 at 4:28 PM, Gandalf Corvotempesta <

Post by Kevin Lemonnier

Post by Kevin Lemonnier
We've had a lot of problems in the past, but at least for us 3.7.12 (and

3.7.15)

Post by Kevin Lemonnier
seems to be working pretty well as long as you don't add bricks. We

started doing

Post by Kevin Lemonnier
multiple little clusters and abandonned the idea of one big cluster, had

no

Post by Kevin Lemonnier
issues since :)

Well, adding bricks could be usefull... :)
Having to create multiple cluster is not a solution and is much more
expansive.
And if you corrupt data from a single cluster you still have issues
I think would be better to add less features and focus more to stability.

First of all, thanks to all the folks who contributed to this thread. We
value your feedback.

In gluster-users and ovirt-community we saw people trying gluster and
complain about heal times and split-brains. So we had to fix bugs in quorum
in 3-way replication; then we started working on features like sharding for
better heal times and arbiter volumes for cost benefits.

To make gluster stable for VM images we had to add all these new features
and then fix all the bugs Lindsay/Kevin reported. We just fixed a
corruption issue that can happen with replace-brick which will be available
in 3.9.0 and 3.8.6. The only 2 other known issues that can lead to
corruptions are add-brick and the bug you filed Gandalf. Krutika just 5
minutes back saw something that could possibly lead to the corruption for
the add-brick bug. Is that really the Root cause? We are not sure yet, we
need more time. Without Lindsay/Kevin/David Gossage's support this workload
would have been in much worse condition. These bugs are not easy to
re-create thus not easy to fix. At least that has been Krutika's experience.

Take away from this mail thread for me is: I think it is important
to educate users about why we are adding new features. People are coming to
the conclusion that only bug fixing corresponds to stabilization and not
features. It is a wrong perception. Without the work that went into adding
all those new features above in gluster, most probably you guys wouldn't
have given gluster another chance because it used to be unusable before
these features for VM workloads. One more take away is to get the
documentation right. Lack of documentation led Alex to try the worst
possible combo for storing VMs on gluster. So we as community failed in
some way there as well.

Krutika will be sending out VM usecase related documentation after
28th of this month. If you have any other feedback, do let us know.

In a software defined storage, stability and consistency are the most

Post by Kevin Lemonnier
important things
I'm also subscribed to moosefs and lizardfs mailing list and I don't
recall any single data corruption/data loss event
In gluster, after some days of testing I've found a huge data corruption
issue that is still unfixed on bugzilla.
If you change the shard size on a populated cluster, you break all
existing data.
Try to do this on a cluster with working VMs and see what happens....
a single cli command break everything and is still unfixed.
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Gandalf Corvotempesta]

Post by Pranith Kumar Karampuri
To make gluster stable for VM images we had to add all these new features
and then fix all the bugs Lindsay/Kevin reported. We just fixed a corruption
issue that can happen with replace-brick which will be available in 3.9.0
and 3.8.6. The only 2 other known issues that can lead to corruptions are
add-brick and the bug you filed Gandalf. Krutika just 5 minutes back saw
something that could possibly lead to the corruption for the add-brick bug.
Is that really the Root cause? We are not sure yet, we need more time.
Without Lindsay/Kevin/David Gossage's support this workload would have been
in much worse condition. These bugs are not easy to re-create thus not easy
to fix. At least that has been Krutika's experience.

Ok, but this changes should be placed in a "test" version and not
marked as stable.
I don't see any development release, only stable releases here.
Do you want all features ? Try the "beta/rc/unstable/alpha/dev" version.
Do you want the stable version without known bugs but slow on VMs
workload? Use the "-stable" version.

If you relase as stable, users tend to upgrade their cluster and use
the newer feature (that you are marking as stable).
What If I upgrade a production cluster to a stable version and try to
add-brick that lead to data corruption ?
I have to restore terabytes worth of data? Gluster is made for
scale-out, what I my cluster was made with 500TB of VMs ?
Try to restore 500TB from a backup....................

This is unacceptable. add-brick/replace-brick should be common "daily"
operations. You should heavy check these for regression or bug.

Post by Pranith Kumar Karampuri
One more take away is to get the
documentation right. Lack of documentation led Alex to try the worst
possible combo for storing VMs on gluster. So we as community failed in some
way there as well.
Krutika will be sending out VM usecase related documentation after
28th of this month. If you have any other feedback, do let us know.

Yes, lack of updated docs or a reference architecture is a big issue.

[Pranith Kumar Karampuri]

On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Pranith Kumar Karampuri
To make gluster stable for VM images we had to add all these new features
and then fix all the bugs Lindsay/Kevin reported. We just fixed a

corruption

Post by Pranith Kumar Karampuri
issue that can happen with replace-brick which will be available in 3.9.0
and 3.8.6. The only 2 other known issues that can lead to corruptions are
add-brick and the bug you filed Gandalf. Krutika just 5 minutes back saw
something that could possibly lead to the corruption for the add-brick

bug.

Post by Pranith Kumar Karampuri
Is that really the Root cause? We are not sure yet, we need more time.
Without Lindsay/Kevin/David Gossage's support this workload would have

been

Post by Pranith Kumar Karampuri
in much worse condition. These bugs are not easy to re-create thus not

easy

Post by Pranith Kumar Karampuri
to fix. At least that has been Krutika's experience.

Ok, but this changes should be placed in a "test" version and not
marked as stable.
I don't see any development release, only stable releases here.
Do you want all features ? Try the "beta/rc/unstable/alpha/dev" version.
Do you want the stable version without known bugs but slow on VMs
workload? Use the "-stable" version.
If you relase as stable, users tend to upgrade their cluster and use
the newer feature (that you are marking as stable).
What If I upgrade a production cluster to a stable version and try to
add-brick that lead to data corruption ?
I have to restore terabytes worth of data? Gluster is made for
scale-out, what I my cluster was made with 500TB of VMs ?
Try to restore 500TB from a backup....................
This is unacceptable. add-brick/replace-brick should be common "daily"
operations. You should heavy check these for regression or bug.

This is a very good point. Adding other maintainers.

Post by Gandalf Corvotempesta

Post by Pranith Kumar Karampuri
One more take away is to get the
documentation right. Lack of documentation led Alex to try the worst
possible combo for storing VMs on gluster. So we as community failed in

some

Post by Pranith Kumar Karampuri
way there as well.
Krutika will be sending out VM usecase related documentation after
28th of this month. If you have any other feedback, do let us know.

Yes, lack of updated docs or a reference architecture is a big issue.

[Niels de Vos]

Post by Pranith Kumar Karampuri
On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Pranith Kumar Karampuri
To make gluster stable for VM images we had to add all these new features
and then fix all the bugs Lindsay/Kevin reported. We just fixed a

corruption

Post by Pranith Kumar Karampuri
issue that can happen with replace-brick which will be available in 3.9.0
and 3.8.6. The only 2 other known issues that can lead to corruptions are
add-brick and the bug you filed Gandalf. Krutika just 5 minutes back saw
something that could possibly lead to the corruption for the add-brick

bug.

Post by Pranith Kumar Karampuri
Is that really the Root cause? We are not sure yet, we need more time.
Without Lindsay/Kevin/David Gossage's support this workload would have

been

Post by Pranith Kumar Karampuri
in much worse condition. These bugs are not easy to re-create thus not

easy

Post by Pranith Kumar Karampuri
to fix. At least that has been Krutika's experience.

Ok, but this changes should be placed in a "test" version and not
marked as stable.
I don't see any development release, only stable releases here.
Do you want all features ? Try the "beta/rc/unstable/alpha/dev" version.
Do you want the stable version without known bugs but slow on VMs
workload? Use the "-stable" version.
If you relase as stable, users tend to upgrade their cluster and use
the newer feature (that you are marking as stable).
What If I upgrade a production cluster to a stable version and try to
add-brick that lead to data corruption ?
I have to restore terabytes worth of data? Gluster is made for
scale-out, what I my cluster was made with 500TB of VMs ?
Try to restore 500TB from a backup....................
This is unacceptable. add-brick/replace-brick should be common "daily"
operations. You should heavy check these for regression or bug.

This is a very good point. Adding other maintainers.

Obviously this is unacceptible for versions that have sharding as a
functional (not experimental) feature. All supported features are
expected to function without major problems (like corruption) for all
standard Gluster operations. Add-brick/replace-brick are surely such
Gluster operations.

Of course it is possible that this does not always happen, and our tests
did not catch the problem. In that case, we really need to have a bug
report with all the details, and preferably a script that can be used to
reproduce and detect the failure.

FWIW sharding has several open bugs (like any other component), but it
is not immediately clear to me if the problem reported in this email is
in Bugzilla yet. These are the bugs that are expected to get fixed in
upcoming minor releases:
https://bugzilla.redhat.com/buglist.cgi?component=sharding&f1=bug_status&f2=version&o1=notequals&o2=notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline

HTH,
Niels

[Gandalf Corvotempesta]

Post by Niels de Vos
Obviously this is unacceptible for versions that have sharding as a
functional (not experimental) feature. All supported features are
expected to function without major problems (like corruption) for all
standard Gluster operations. Add-brick/replace-brick are surely such
Gluster operations.

Is sharding an experimental feature even in 3.8 ?
Because in 3.8 announcement, it's declared stable:
http://blog.gluster.org/2016/06/glusterfs-3-8-released/
"Sharding is now stable for VM image storage. "

Post by Niels de Vos
FWIW sharding has several open bugs (like any other component), but it
is not immediately clear to me if the problem reported in this email is
in Bugzilla yet. These are the bugs that are expected to get fixed in
https://bugzilla.redhat.com/buglist.cgi?component=sharding&f1=bug_status&f2=version&o1=notequals&o2=notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline

My issue with sharding was reported in bugzilla on 2016-07-12
4 months for a IMHO, critical bug.

If you disable sharding on a sharded volume with existing shared data,
you corrupt every existing file.

[Krutika Dhananjay]

Yes. I apologise for the delay.

Disabling sharding would knock the translator itself off the client stack,
and
being that sharding is the actual (and the only) translator that has the
knowledge of how to interpret sharded files, and how to aggregate them,
removing the translator from the stack will make all shards start to appear
like
isolated files with no way to interpret the correlation between the
individual pieces.

The only way to fix it is to have sharding be part of the graph *even* if
disabled,
except that in this case, its job should be confined to aggregating the
already
sharded files during reads but NOT shard new files that are created, since
it is
supposed to "act" disabled. This is a slightly bigger change and this is
why I had
suggested the workaround at
https://bugzilla.redhat.com/show_bug.cgi?id=1355846#c1
back then.

FWIW, the documentation [1] does explain how to disable sharding the right
way and has been in existence ever since sharding was first released in
3.7.0.

[1] - http://staged-gluster-docs.readthedocs.io/en/release3.7.
0beta1/Features/shard/

-Krutika



On Mon, Nov 14, 2016 at 9:08 PM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Niels de Vos
Obviously this is unacceptible for versions that have sharding as a
functional (not experimental) feature. All supported features are
expected to function without major problems (like corruption) for all
standard Gluster operations. Add-brick/replace-brick are surely such
Gluster operations.

Is sharding an experimental feature even in 3.8 ?
http://blog.gluster.org/2016/06/glusterfs-3-8-released/
"Sharding is now stable for VM image storage. "

Post by Niels de Vos
FWIW sharding has several open bugs (like any other component), but it
is not immediately clear to me if the problem reported in this email is
in Bugzilla yet. These are the bugs that are expected to get fixed in
https://bugzilla.redhat.com/buglist.cgi?component=

sharding&f1=bug_status&f2=version&o1=notequals&o2=
notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline
My issue with sharding was reported in bugzilla on 2016-07-12
4 months for a IMHO, critical bug.
If you disable sharding on a sharded volume with existing shared data,
you corrupt every existing file.
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Gandalf Corvotempesta]

Post by Krutika Dhananjay
The only way to fix it is to have sharding be part of the graph *even* if
disabled,
except that in this case, its job should be confined to aggregating the
already
sharded files during reads but NOT shard new files that are created, since
it is
supposed to "act" disabled. This is a slightly bigger change and this is why
I had
suggested the workaround at
https://bugzilla.redhat.com/show_bug.cgi?id=1355846#c1
back then.

Why not keeping the shard xlator always on but set on a very high value so that
shard is never happening? Something at 100GB (just as proof of concept)

Post by Krutika Dhananjay
FWIW, the documentation [1] does explain how to disable sharding the right
way and has been in existence ever since sharding was first released in
3.7.0.
[1] -
http://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/

Ok but:
1) that's for 3.7 *beta1*. I'm using 3.8
2) "advisable" doesn't mean "you have to". It's an advice, not the
only way to disable a feature
3) i'm talking about a confirm to add in the cli, nothing strange. all
software ask for a confirm when bad things could happens.

[Vijay Bellur]

On Mon, Nov 14, 2016 at 10:38 AM, Gandalf Corvotempesta

Post by Gandalf Corvotempesta

Post by Niels de Vos
Obviously this is unacceptible for versions that have sharding as a
functional (not experimental) feature. All supported features are
expected to function without major problems (like corruption) for all
standard Gluster operations. Add-brick/replace-brick are surely such
Gluster operations.

Is sharding an experimental feature even in 3.8 ?
http://blog.gluster.org/2016/06/glusterfs-3-8-released/
"Sharding is now stable for VM image storage. "

sharding was an experimental feature in 3.7. Based on the feedback
that we received in testing, we called it out as stable in 3.8. The
add-brick related issue is something that none of us encountered in
testing and we will determine how we can avoid missing such problems
in the future.

Post by Gandalf Corvotempesta

Post by Niels de Vos
FWIW sharding has several open bugs (like any other component), but it
is not immediately clear to me if the problem reported in this email is
in Bugzilla yet. These are the bugs that are expected to get fixed in
https://bugzilla.redhat.com/buglist.cgi?component=sharding&f1=bug_status&f2=version&o1=notequals&o2=notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline

My issue with sharding was reported in bugzilla on 2016-07-12
4 months for a IMHO, critical bug.
If you disable sharding on a sharded volume with existing shared data,
you corrupt every existing file.

Accessing sharded data after disabling sharding is something that we
did not visualize as a valid use case at any point in time. Also, you
could access the contents by enabling sharding again. Given these
factors I think this particular problem has not been prioritized by
us.

As with many other projects, we are in a stage today where the number
of users and testers far outweigh the number of developers
contributing code. With this state it becomes hard to prioritize
problems from a long todo list for developers. If valuable community
members like you feel strongly about a bug or feature that need
attention of developers, please call such issues out on the mailing
list. We will be more than happy to help.

Having explained the developer perspective, I do apologize for any
inconvenience you might have encountered from this particular bug.

Thanks!
Vijay

[Gandalf Corvotempesta]

Post by Vijay Bellur
Accessing sharded data after disabling sharding is something that we
did not visualize as a valid use case at any point in time. Also, you
could access the contents by enabling sharding again. Given these
factors I think this particular problem has not been prioritized by
us.

That's not true.
If you have VMs running on a sharded volume and you disable sharding,
with the VM still running, everything crash and could lead to data loss, as VM
will be unable to find their filesystem and so on, qemu currupts the
image and so on.....

If I write to a file that was shareded, (in example a log file), now
when you disable the shard,
the application would write the existing file (the one that was the
first shard).
If you reenable sharding, you lost some data

Example:

128MB file. shard set to 64MB. You have 2 chunks: shard1+shard2

Now you are writing to the file:

AAAA
BBBB
CCCC
DDDD

AAAA+BBBB are placed on shard1, CCCC+DDDD are placed on shard2

If you disable the shard and write some extra data, EEEE, then EEEE
would be placed after BBBB in shard1 (growing more than 64MB)
and not on shard3

If you re-enable shard, EEEE is lost, as gluster would expect it as
shard3. and I think gluster will read only the first 64MB from shard1.
If gluster read the whole file, you'll get something like this:

AAAA
BBBB
EEEE
CCCC
DDDD

in a text file this is bad, in a VM image, this mean data
loss/corruption almost impossible to fix.

Post by Vijay Bellur
As with many other projects, we are in a stage today where the number
of users and testers far outweigh the number of developers
contributing code. With this state it becomes hard to prioritize
problems from a long todo list for developers. If valuable community
members like you feel strongly about a bug or feature that need
attention of developers, please call such issues out on the mailing
list. We will be more than happy to help.

That's why i've asked for less feature and more stability.
If you have to prioritize, please choose all bugs that could lead to
data corruption or similiar.

[Joe Julian]

Features and stability are not mutually exclusive.

Sometimes instability is cured by adding a feature.

Fixing a bug is not something that's solved better by having more developers work on it.

Sometimes fixing one bug exposed a problem elsewhere.

Using free open source community projects with your own hardware and system design weights the responsibility to test more heavily on yourself. If that's not a risk you can afford, you might consider contracting with a 3rd party which has "certified" installation parameters. IMHO.

Post by Gandalf Corvotempesta

Post by Vijay Bellur
Accessing sharded data after disabling sharding is something that we
did not visualize as a valid use case at any point in time. Also, you
could access the contents by enabling sharding again. Given these
factors I think this particular problem has not been prioritized by
us.

That's not true.
If you have VMs running on a sharded volume and you disable sharding,
with the VM still running, everything crash and could lead to data loss, as VM
will be unable to find their filesystem and so on, qemu currupts the
image and so on.....
If I write to a file that was shareded, (in example a log file), now
when you disable the shard,
the application would write the existing file (the one that was the
first shard).
If you reenable sharding, you lost some data
128MB file. shard set to 64MB. You have 2 chunks: shard1+shard2
AAAA
BBBB
CCCC
DDDD
AAAA+BBBB are placed on shard1, CCCC+DDDD are placed on shard2
If you disable the shard and write some extra data, EEEE, then EEEE
would be placed after BBBB in shard1 (growing more than 64MB)
and not on shard3
If you re-enable shard, EEEE is lost, as gluster would expect it as
shard3. and I think gluster will read only the first 64MB from shard1.
AAAA
BBBB
EEEE
CCCC
DDDD
in a text file this is bad, in a VM image, this mean data
loss/corruption almost impossible to fix.

Post by Vijay Bellur
As with many other projects, we are in a stage today where the number
of users and testers far outweigh the number of developers
contributing code. With this state it becomes hard to prioritize
problems from a long todo list for developers. If valuable community
members like you feel strongly about a bug or feature that need
attention of developers, please call such issues out on the mailing
list. We will be more than happy to help.

That's why i've asked for less feature and more stability.
If you have to prioritize, please choose all bugs that could lead to
data corruption or similiar.
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[ML Wong]

Though remove-brick is not an usual act we would do for Gluster volume,
this has consistently failed ending in corrupted gluster volume after
Sharding has been turned on. For bug1387878, it's very similar to what i
had encountered in ESXi world. Add-brick, would run successful, but
virtual-machine files would crash after rebalance in one of my
environments. That did not happen in my another environment under same
version (3.7.16). Difference between 2 was one is changing from Replicate
to Distributed-Replicate, but they are still configured with only
2-replicas. i will have to test 3.8.* with Ganesha to see how it goes.

On Mon, Nov 14, 2016 at 8:29 AM, Gandalf Corvotempesta <

Post by Gandalf Corvotempesta

Post by Vijay Bellur
Accessing sharded data after disabling sharding is something that we
did not visualize as a valid use case at any point in time. Also, you
could access the contents by enabling sharding again. Given these
factors I think this particular problem has not been prioritized by
us.

That's not true.
If you have VMs running on a sharded volume and you disable sharding,
with the VM still running, everything crash and could lead to data loss, as VM
will be unable to find their filesystem and so on, qemu currupts the
image and so on.....
If I write to a file that was shareded, (in example a log file), now
when you disable the shard,
the application would write the existing file (the one that was the
first shard).
If you reenable sharding, you lost some data
128MB file. shard set to 64MB. You have 2 chunks: shard1+shard2
AAAA
BBBB
CCCC
DDDD
AAAA+BBBB are placed on shard1, CCCC+DDDD are placed on shard2
If you disable the shard and write some extra data, EEEE, then EEEE
would be placed after BBBB in shard1 (growing more than 64MB)
and not on shard3
If you re-enable shard, EEEE is lost, as gluster would expect it as
shard3. and I think gluster will read only the first 64MB from shard1.
AAAA
BBBB
EEEE
CCCC
DDDD
in a text file this is bad, in a VM image, this mean data
loss/corruption almost impossible to fix.

Post by Vijay Bellur
As with many other projects, we are in a stage today where the number
of users and testers far outweigh the number of developers
contributing code. With this state it becomes hard to prioritize
problems from a long todo list for developers. If valuable community
members like you feel strongly about a bug or feature that need
attention of developers, please call such issues out on the mailing
list. We will be more than happy to help.

That's why i've asked for less feature and more stability.
If you have to prioritize, please choose all bugs that could lead to
data corruption or similiar.
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Krutika Dhananjay]

Post by ML Wong

Post by Pranith Kumar Karampuri
On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta <

2016-11-14 11:50 GMT+01:00 Pranith Kumar Karampuri <

Post by Pranith Kumar Karampuri
To make gluster stable for VM images we had to add all these new

features

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
and then fix all the bugs Lindsay/Kevin reported. We just fixed a

corruption

Post by Pranith Kumar Karampuri
issue that can happen with replace-brick which will be available in

3.9.0

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
and 3.8.6. The only 2 other known issues that can lead to

corruptions are

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
add-brick and the bug you filed Gandalf. Krutika just 5 minutes back

saw

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
something that could possibly lead to the corruption for the

add-brick

Post by Pranith Kumar Karampuri

bug.

Post by Pranith Kumar Karampuri
Is that really the Root cause? We are not sure yet, we need more

time.

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
Without Lindsay/Kevin/David Gossage's support this workload would

have

Post by Pranith Kumar Karampuri

been

Post by Pranith Kumar Karampuri
in much worse condition. These bugs are not easy to re-create thus

not

Post by Pranith Kumar Karampuri

easy

Post by Pranith Kumar Karampuri
to fix. At least that has been Krutika's experience.

Ok, but this changes should be placed in a "test" version and not
marked as stable.
I don't see any development release, only stable releases here.
Do you want all features ? Try the "beta/rc/unstable/alpha/dev"

version.

Post by Pranith Kumar Karampuri

Do you want the stable version without known bugs but slow on VMs
workload? Use the "-stable" version.
If you relase as stable, users tend to upgrade their cluster and use
the newer feature (that you are marking as stable).
What If I upgrade a production cluster to a stable version and try to
add-brick that lead to data corruption ?
I have to restore terabytes worth of data? Gluster is made for
scale-out, what I my cluster was made with 500TB of VMs ?
Try to restore 500TB from a backup....................
This is unacceptable. add-brick/replace-brick should be common "daily"
operations. You should heavy check these for regression or bug.

This is a very good point. Adding other maintainers.

I think Pranith's intention here was to bring to other maintainers'
attention the point about
development releases vs stable releases although his inline comment may
have been a
bit out-of-place (I was part of the discussion that took place before this
reply of his, in office
today, hence taking the liberty to clarify).

-Krutika

Post by ML Wong
Obviously this is unacceptible for versions that have sharding as a
functional (not experimental) feature. All supported features are
expected to function without major problems (like corruption) for all
standard Gluster operations. Add-brick/replace-brick are surely such
Gluster operations.
Of course it is possible that this does not always happen, and our tests
did not catch the problem. In that case, we really need to have a bug
report with all the details, and preferably a script that can be used to
reproduce and detect the failure.
FWIW sharding has several open bugs (like any other component), but it
is not immediately clear to me if the problem reported in this email is
in Bugzilla yet. These are the bugs that are expected to get fixed in
https://bugzilla.redhat.com/buglist.cgi?component=
sharding&f1=bug_status&f2=version&o1=notequals&o2=
notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline
HTH,
Niels
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[David Gossage]

Post by ML Wong

Post by Pranith Kumar Karampuri
On Mon, Nov 14, 2016 at 4:38 PM, Gandalf Corvotempesta <

2016-11-14 11:50 GMT+01:00 Pranith Kumar Karampuri <

Post by Pranith Kumar Karampuri
To make gluster stable for VM images we had to add all these new

features

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
and then fix all the bugs Lindsay/Kevin reported. We just fixed a

corruption

Post by Pranith Kumar Karampuri
issue that can happen with replace-brick which will be available in

3.9.0

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
and 3.8.6. The only 2 other known issues that can lead to

corruptions are

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
add-brick and the bug you filed Gandalf. Krutika just 5 minutes back

saw

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
something that could possibly lead to the corruption for the

add-brick

Post by Pranith Kumar Karampuri

bug.

Post by Pranith Kumar Karampuri
Is that really the Root cause? We are not sure yet, we need more

time.

Post by Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
Without Lindsay/Kevin/David Gossage's support this workload would

have

Post by Pranith Kumar Karampuri

been

Post by Pranith Kumar Karampuri
in much worse condition. These bugs are not easy to re-create thus

not

Post by Pranith Kumar Karampuri

easy

Post by Pranith Kumar Karampuri
to fix. At least that has been Krutika's experience.

Ok, but this changes should be placed in a "test" version and not
marked as stable.
I don't see any development release, only stable releases here.
Do you want all features ? Try the "beta/rc/unstable/alpha/dev"

version.

Post by Pranith Kumar Karampuri

Do you want the stable version without known bugs but slow on VMs
workload? Use the "-stable" version.
If you relase as stable, users tend to upgrade their cluster and use
the newer feature (that you are marking as stable).
What If I upgrade a production cluster to a stable version and try to
add-brick that lead to data corruption ?
I have to restore terabytes worth of data? Gluster is made for
scale-out, what I my cluster was made with 500TB of VMs ?
Try to restore 500TB from a backup....................
This is unacceptable. add-brick/replace-brick should be common "daily"
operations. You should heavy check these for regression or bug.

This is a very good point. Adding other maintainers.

Obviously this is unacceptible for versions that have sharding as a
functional (not experimental) feature. All supported features are
expected to function without major problems (like corruption) for all
standard Gluster operations. Add-brick/replace-brick are surely such
Gluster operations.
Of course it is possible that this does not always happen, and our tests
did not catch the problem. In that case, we really need to have a bug
report with all the details, and preferably a script that can be used to
reproduce and detect the failure.

I believe this bug relates to this particular issue raised in this email
chain.

https://bugzilla.redhat.com/show_bug.cgi?id=1387878

Kevin found bug, and Lindsay filed report after she was able to recreate it.

Post by ML Wong
FWIW sharding has several open bugs (like any other component), but it
is not immediately clear to me if the problem reported in this email is
in Bugzilla yet. These are the bugs that are expected to get fixed in
https://bugzilla.redhat.com/buglist.cgi?component=
sharding&f1=bug_status&f2=version&o1=notequals&o2=
notequals&product=GlusterFS&query_format=advanced&v1=CLOSED&v2=mainline
HTH,
Niels
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Krutika Dhananjay]

Which data corruption issue is this? Could you point me to the bug report
on bugzilla?

-Krutika

On Sat, Nov 12, 2016 at 4:28 PM, Gandalf Corvotempesta <

Post by Kevin Lemonnier

Post by Kevin Lemonnier
We've had a lot of problems in the past, but at least for us 3.7.12 (and

3.7.15)

Post by Kevin Lemonnier
seems to be working pretty well as long as you don't add bricks. We

started doing

Post by Kevin Lemonnier
multiple little clusters and abandonned the idea of one big cluster, had

no

Post by Kevin Lemonnier
issues since :)

Well, adding bricks could be usefull... :)
Having to create multiple cluster is not a solution and is much more
expansive.
And if you corrupt data from a single cluster you still have issues
I think would be better to add less features and focus more to stability.
In a software defined storage, stability and consistency are the most
important things
I'm also subscribed to moosefs and lizardfs mailing list and I don't
recall any single data corruption/data loss event
In gluster, after some days of testing I've found a huge data corruption
issue that is still unfixed on bugzilla.
If you change the shard size on a populated cluster, you break all
existing data.
Try to do this on a cluster with working VMs and see what happens....
a single cli command break everything and is still unfixed.
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Gandalf Corvotempesta]

Post by Kevin Lemonnier
We've had a lot of problems in the past, but at least for us 3.7.12 (and 3.7.15)
seems to be working pretty well as long as you don't add bricks. We started doing
multiple little clusters and abandonned the idea of one big cluster, had no
issues since :)

I was thinking about this.
If you meant creating multiple volumes, ok, but having to create
multiple clusters is a bad idea.
Gluster performs better with multiple nodes, if you have to split the
infrastructure (and nodes)
in multiple cluster, you'll affect the performance.

Is something like to create multiple enterprise SAN to avoid data corruption.
Data corruption must be address with firmware/software updates, not by
creating multiple storages.

if you create 2 gluster storages, you'll get the same issues in
multiple storages. It's a bad workaround, not a solution.

[Gandalf Corvotempesta]

I don't understand the issue. Let's say I can fit 30 VMs on a 3 node

cluster,

whenever I need to create the VM 31 I just order 3 nodes and replicate the
exact same cluster. I get the exact same performances as on the first

cluster,

since it's the same hardware. For a while it'll even be a bit better since
there is only one VM on it :)

doing so means 3 times cost, 3 times disks to add and manage and so on
is not "commodity"

If you have to create multiple cluster and not using the scale-out feature
why don't you use drbd or similiar?

[Kevin Lemonnier]

Post by Gandalf Corvotempesta
doing so means 3 times cost, 3 times disks to add and manage and so on
is not "commodity"

It's the exact same cost. All the clusters have replica 3, there is absolutly
no difference in cost if they are separate or not.
It's not as easy to manage sure, but it's not like I'm tweaking the conf
everyday.

Post by Gandalf Corvotempesta
If you have to create multiple cluster andA not using the scale-out
feature why don't you use drbd or similiar?

DRBD 8 doesn't support 3 replicas, and DRBD 9 last time I tried it
was basically unusable (well, not surprising for a beta).
And on top of that, gluster heals after a network outage is transparent
and automatic, DRBD's heal are a huge pain. Since we are using OVH servers,
the network really can't be trusted unfortunatly, we do have a lot of heals
happening.

We use DRBD a lot when we need normal master / slave replication, it's great
for that, but to run VMs I really don't like it. One of our client has a
proxmox 3 cluster with DRBD 8, everytime there is a little problem with
the network it's horrible to fix, compared to gluster.