[Gluster-users] GlusterFS as virtual machine storage

Post by Gionatan Danti
Hi all,
I would like to ask if, and with how much success, you are using
GlusterFS for virtual machine storage.

Hi, we have similar clusters.

Post by Gionatan Danti
My plan: I want to setup a 2-node cluster, where VM runs on the nodes
themselves and can be live-migrated on demand.

Use 3 nodes for gluster, or at the very least 2 + 1 arbiter. You really
really don't want a 2 node setup, you'll get split brains.

Post by Gionatan Danti
- do you use GlusterFS for similar setup?

Yes, but always 3 nodes.

Post by Gionatan Danti
- if so, how do you feel about it?

Works pretty well now, we've had problems but 3.7.15 works great. We are
currently testing 3.8, seems to work fine too. I imagine the newer ones
do aswell.

Post by Gionatan Danti
- if a node crashes/reboots, how the system re-syncs? Will the VM files
be fully resynchronized, or the live node keeps some sort of write
bitmap to resynchronize changed/written chunks only? (note: I know about
sharding, but I would like to avoid it);

You really should use sharding, not sure exactly what happens without it
but I know it made the VM unusable during heal (they froze), basically.
Sharding solved that, with it it works well even during heal. It only
retransfert each shard so the amount to re-sync is pretty low.

Post by Gionatan Danti
- finally, how much stable is the system?

Haven't had any problems with gluster itself since we updated to >
3.7.11. It just works, even on a pretty bad network.
Performances aren't amazing, but I imagine you know that, replication
has a cost.

Post by Gionatan Danti
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-08-23 16:14:17 UTC

Hi, after many VM crashes during upgrades of Gluster, losing network
connectivity on one node etc. I would advise running replica 2 with
arbiter.

I once even managed to break this setup (with arbiter) due to network
partitioning - one data node never healed and I had to restore from
backups (it was easier and kind of non-production). Be extremely
careful and plan for failure.

-ps

Post by Gionatan Danti
Hi all,
I would like to ask if, and with how much success, you are using GlusterFS
for virtual machine storage.
My plan: I want to setup a 2-node cluster, where VM runs on the nodes
themselves and can be live-migrated on demand.
- do you use GlusterFS for similar setup?
- if so, how do you feel about it?
- if a node crashes/reboots, how the system re-syncs? Will the VM files be
fully resynchronized, or the live node keeps some sort of write bitmap to
resynchronize changed/written chunks only? (note: I know about sharding, but
I would like to avoid it);
- finally, how much stable is the system?
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Gionatan Danti

2017-08-23 16:51:55 UTC

Post by Pavel Szalbot
Hi, after many VM crashes during upgrades of Gluster, losing network
connectivity on one node etc. I would advise running replica 2 with
arbiter.

Hi Pavel, this is bad news :(
So, in your case at least, Gluster was not stable? Something as simple
as an update would let it crash?

Post by Pavel Szalbot
I once even managed to break this setup (with arbiter) due to network
partitioning - one data node never healed and I had to restore from
backups (it was easier and kind of non-production). Be extremely
careful and plan for failure.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

l***@ulrar.net

2017-08-23 16:59:25 UTC

What he is saying is that, on a two node volume, upgrading a node will
cause the volume to go down. That's nothing weird, you really should use
3 nodes.

Post by Pavel Szalbot
Hi, after many VM crashes during upgrades of Gluster, losing network
connectivity on one node etc. I would advise running replica 2 with
arbiter.

Hi Pavel, this is bad news :(
So, in your case at least, Gluster was not stable? Something as simple
as an update would let it crash?

I would use VM locking via sanlock or virtlock, so a split brain should
not cause simultaneous changes on both replicas. I am more concerned
about volume heal time: what will happen if the standby node
crashes/reboots? Will *all* data be re-synced from the master, or only
changed bit will be re-synced? As stated above, I would like to avoid
using sharding...
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-08-23 19:13:28 UTC

Hi, I believe it is not that simple. Even replica 2 + arbiter volume
with default network.ping-timeout will cause the underlying VM to
remount filesystem as read-only (device error will occur) unless you
tune mount options in VM's fstab.
-ps

Post by l***@ulrar.net
What he is saying is that, on a two node volume, upgrading a node will
cause the volume to go down. That's nothing weird, you really should use
3 nodes.

Post by Pavel Szalbot
Hi, after many VM crashes during upgrades of Gluster, losing network
connectivity on one node etc. I would advise running replica 2 with
arbiter.

Hi Pavel, this is bad news :(
So, in your case at least, Gluster was not stable? Something as simple
as an update would let it crash?

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

l***@ulrar.net

2017-08-23 19:26:45 UTC

Really ? I can't see why. But I've never used arbiter so you probably
know more about this than I do.

In any case, with replica 3, never had a problem.

Post by Pavel Szalbot
Hi, I believe it is not that simple. Even replica 2 + arbiter volume
with default network.ping-timeout will cause the underlying VM to
remount filesystem as read-only (device error will occur) unless you
tune mount options in VM's fstab.
-ps

Post by l***@ulrar.net
What he is saying is that, on a two node volume, upgrading a node will
cause the volume to go down. That's nothing weird, you really should use
3 nodes.

Post by Pavel Szalbot
Hi, after many VM crashes during upgrades of Gluster, losing network
connectivity on one node etc. I would advise running replica 2 with
arbiter.

Hi Pavel, this is bad news :(
So, in your case at least, Gluster was not stable? Something as simple
as an update would let it crash?

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-08-23 19:34:03 UTC

I remember seeing errors like "Transport endpoint not connected" in
the client logs after ping timeout even with arbiter. Arbiter does not
prevent this.

And if you end up in situation when arbiter blames the only running
brick for given file, you are doomed.
-ps

Post by l***@ulrar.net
Really ? I can't see why. But I've never used arbiter so you probably
know more about this than I do.
In any case, with replica 3, never had a problem.

Post by l***@ulrar.net
What he is saying is that, on a two node volume, upgrading a node will
cause the volume to go down. That's nothing weird, you really should use
3 nodes.

Post by Pavel Szalbot
Hi, after many VM crashes during upgrades of Gluster, losing network
connectivity on one node etc. I would advise running replica 2 with
arbiter.

Hi Pavel, this is bad news :(
So, in your case at least, Gluster was not stable? Something as simple
as an update would let it crash?

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

2017-08-24 00:13:46 UTC

That really isnt an arbiter issue or for that matter a Gluster issue. We
have seen that with vanilla NAS servers that had some issue or another.

Arbiter simply makes it less likely to be an issue than replica 2 but in
turn arbiter is less 'safe' than replica 3.

However, in regards to Gluster and RO behaviour

The default timeout for most OS versions is 30 seconds and the Gluster
timeout is 42, so yes you can trigger an RO event.

# cat /sys/block/sda/device/timeout
30

Though it is easy enough to raise as Pavel mentioned

# echo 90 > /sys/block/sda/device/timeout

As a purely observational note, we have noticed that EXT3/4 filesystems
on VMs will go read-only much easier than XFS systems (even with the
default timeout and irregardless of storage type). We have always
wondered about that, though part of that observation is biased because
we tend to use XFS on newer VMs which mean newer, better kernels.

Likewise virtio "disks" don't even have a timeout value that I am aware
of and I don't recall them being extremely sensitive to disk issues on
either Gluster, NFS or DAS.

All our newer VMs use virtio instead of sata/ide emulation AND XFS so we
rarely see a RO situation and if we do, it was a good thing the VMs did
go RO to protect themselves while the storage system freaked out.

Post by l***@ulrar.net
Really ? I can't see why. But I've never used arbiter so you probably
know more about this than I do.
In any case, with replica 3, never had a problem.

Post by l***@ulrar.net
What he is saying is that, on a two node volume, upgrading a node will
cause the volume to go down. That's nothing weird, you really should use
3 nodes.

Post by Pavel Szalbot
Hi, after many VM crashes during upgrades of Gluster, losing network
connectivity on one node etc. I would advise running replica 2 with
arbiter.

Hi Pavel, this is bad news :(
So, in your case at least, Gluster was not stable? Something as simple
as an update would let it crash?

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-08-24 05:44:22 UTC

Hi,

Post by WK
The default timeout for most OS versions is 30 seconds and the Gluster
timeout is 42, so yes you can trigger an RO event.

I get read-only mount within approximately 2 seconds after failed IO.

Post by WK
Though it is easy enough to raise as Pavel mentioned
# echo 90 > /sys/block/sda/device/timeout

AFAIK this is applicable only for directly attached block devices
(non-virtualized).

Post by WK
Likewise virtio "disks" don't even have a timeout value that I am aware of
and I don't recall them being extremely sensitive to disk issues on either
Gluster, NFS or DAS.

We use only virtio and these problems are persistent - temporarily
suspending a node (e.g. HW or Gluster upgrade, reboot) is very scary,
because we often end up with read-only filesystems on all VMs.

However we use ext4, so I cannot comment on XFS.

This discussion will probably end before I migrate VMs from Gluster to
local storage on our Openstack nodes, but I might run some tests
afterwards and keep you posted.

-ps

2017-08-24 20:20:16 UTC

Post by Pavel Szalbot
Hi,

Post by WK
The default timeout for most OS versions is 30 seconds and the Gluster
timeout is 42, so yes you can trigger an RO event.

I get read-only mount within approximately 2 seconds after failed IO.

Hmm, we don't see that, even on busy VMs.
We ARE using QCOW2 disk images though.

Also, though we no longer use Ovirt, I am still on the list. They are
heavy Gluster users and they would be howling if they all had your
experience.

Post by WK
Though it is easy enough to raise as Pavel mentioned
# echo 90 > /sys/block/sda/device/timeout

AFAIK this is applicable only for directly attached block devices
(non-virtualized).

No, if you use SATA/IDE emulation (NOT virtio) it is there WITHIN the VM.
We have a lot of legacy VMs from older projects/workloads that have that
and we haven't bothered changing them because "they are working fine now"
It is NOT there on virtio.

Post by WK
Likewise virtio "disks" don't even have a timeout value that I am aware of
and I don't recall them being extremely sensitive to disk issues on either
Gluster, NFS or DAS.

We use the fuse mount, because we are lazy and haven't upgraded to
libgfapi. I hope to start a new cluster with to libfgapi shortly
because of the better performance.
Also we use a localhost mount for the gluster driveset on each compute
node (i.e. so called hyperconverged). So the only 'gluster' only kit is
the lightweight arb box.
So those VMs in the gluster 'pool' have a local write and then only 1
off-server write (to the other gluster enabled compute host), which
means pretty good performance.

We use the gluster included 'virt' tuning set of:

performance.quick-read=off
performance.read-ahead=off
performance.io-cache=off
performance.stat-prefetch=off
performance.low-prio-threads=32
network.remote-dio=enable
cluster.eager-lock=enable
cluster.quorum-type=auto
cluster.server-quorum-type=server
cluster.data-self-heal-algorithm=full
cluster.locking-scheme=granular
cluster.shd-max-threads=8
cluster.shd-wait-qlength=10000
features.shard=on
user.cifs=off

We do play with shard size and have settled down on 64M, though I've
seen recommendations of 128M and 512M for VMs.
We didn't really notice much of a difference with any of those as long
as they were at least 64M

Post by Pavel Szalbot
This discussion will probably end before I migrate VMs from Gluster to
local storage on our Openstack nodes, but I might run some tests
afterwards and keep you posted.

I would be interested in your results. You may also look into Ceph. It
is more complicated than Gluster, (well, more complicated than our
simple little Gluster arrangement) but the OpenStack people swear by it.
It wasn't suited to our needs, but it tested well, when we looked into
it last year.

Pavel Szalbot

2017-08-24 20:49:02 UTC

Post by Pavel Szalbot
Hi,

Post by WK
The default timeout for most OS versions is 30 seconds and the Gluster
timeout is 42, so yes you can trigger an RO event.

I get read-only mount within approximately 2 seconds after failed IO.

Hmm, we don't see that, even on busy VMs.
We ARE using QCOW2 disk images though.

I am using them as well:

<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/mnt/9b7e67e7ed8dc899919ce25f6b14d094/volume-5299bad3-56f5-4ee7-967e-8881a108406e'>
<seclabel model='selinux' labelskip='yes'/>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>

Post by WK
Though it is easy enough to raise as Pavel mentioned
# echo 90 > /sys/block/sda/device/timeout

AFAIK this is applicable only for directly attached block devices
(non-virtualized).

No, if you use SATA/IDE emulation (NOT virtio) it is there WITHIN the VM.
We have a lot of legacy VMs from older projects/workloads that have that and
we haven't bothered changing them because "they are working fine now"
It is NOT there on virtio.

I kind of expected this. We use virtio everywhere except one Win 2012
VM which suffers from SATA/IDE emulation a lot (performance wise).

Post by WK
Likewise virtio "disks" don't even have a timeout value that I am aware of
and I don't recall them being extremely sensitive to disk issues on either
Gluster, NFS or DAS.

Same here. Had some problems with libgfapi on CentOS back in Gluster
3.7 days with dated libvirt available in the system. If you care about
HA you should have appropriate staging environment - unfortunately
learnt this the hard way. And you definitely should watch issue
tracker and mailing lists...

Post by Pavel Szalbot
This discussion will probably end before I migrate VMs from Gluster to
local storage on our Openstack nodes, but I might run some tests
afterwards and keep you posted.

I would be interested in your results. You may also look into Ceph. It is
more complicated than Gluster, (well, more complicated than our simple
little Gluster arrangement) but the OpenStack people swear by it.
It wasn't suited to our needs, but it tested well, when we looked into it
last year.

At first I was looking into Ceph, but the setup procedure was fairly
complicated (almost 3 years ago, do not remember the
version/codename), I saw the ScaleIO vs. Ceph video and colleague of
mine recommended Gluster because he used it in a hospital for VMs.
Setup was easy-peasy, but tuning was pretty hard and documentation did
not help - mailing list archives did better job.

However this colleague was shocked when I showed him randomly
truncated files during heal process and he also did 72+ hours heal on
some VMs (Gluster 3.4 IIRC). I guess he was more lucky and never lost
the data - I had to restore from backups several times.

Btw our Gluster network is 10Gbps fiber VLAN with stacked switches
(Juniper EX4550s) & LACP on servers to ensure high-availability. I am
still not comfortable running critical production VMs there because of
issues we experienced.

-ps

Gionatan Danti

2017-08-25 07:56:07 UTC

Post by Gionatan Danti
Hi all,
any other advice from who use (or do not use) Gluster as a replicated
VM backend?
Thanks.

Sorry, I was not seeing messages because I was not subscribed on the
list; I read it from the web.

So it seems that Pavel and WK have vastly different experience with
Gluster. Any plausible cause for that difference?

Post by Gionatan Danti
2 node plus Arbiter. You NEED the arbiter or a third node. Do NOT try 2
node with a VM

This is true even if I manage locking at application level (via virlock
or sanlock)?
Also, on a two-node setup it is *guaranteed* for updates to one node to
put offline the whole volume?
On the other hand, a 3-way setup (or 2+arbiter) if free from all these
problems?

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

l***@ulrar.net

2017-08-25 08:50:52 UTC

Post by Gionatan Danti
This is true even if I manage locking at application level (via virlock
or sanlock)?

Yes. Gluster has it's own quorum, you can disable it but that's just a
recipe for a disaster.

Post by Gionatan Danti
Also, on a two-node setup it is *guaranteed* for updates to one node to
put offline the whole volume?

I think so, but I never took the chance so who knows.

Post by Gionatan Danti
On the other hand, a 3-way setup (or 2+arbiter) if free from all these
problems?

Free from a lot of problems, but apparently not as good as a replica 3
volume. I can't comment on arbiter, I only have replica 3 clusters. I
can tell you that my colleagues setting up 2 nodes clusters have _a lot_
of problems.

Gionatan Danti

2017-08-25 09:23:03 UTC

Post by l***@ulrar.net
Yes. Gluster has it's own quorum, you can disable it but that's just a
recipe for a disaster.
Free from a lot of problems, but apparently not as good as a replica 3
volume. I can't comment on arbiter, I only have replica 3 clusters. I
can tell you that my colleagues setting up 2 nodes clusters have _a lot_
of problems.

Thanks, these are very valuable informations.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

Lindsay Mathieson

2017-08-25 12:21:29 UTC

Post by l***@ulrar.net
Free from a lot of problems, but apparently not as good as a replica 3
volume. I can't comment on arbiter, I only have replica 3 clusters. I
can tell you that my colleagues setting up 2 nodes clusters have_a lot_
of problems.

I run Replica 3 VM hosting (gfapi) via a 3 node proxmox cluster. Have
done a lot of rolling node updates, power failures etc, never had a
problem. Performance is better than any other DFS I've tried (Ceph,
lizard/moose). Never did get DRDB working.

nb: ZFS Bricks, with each brick RAID10 - so a little paranoid on the
redundancy :)

For me, glusters biggest problem is its lack of flexibility in adding
bricks and nodes.

--
Lindsay Mathieson

Lindsay Mathieson

2017-08-25 12:22:53 UTC

--
Lindsay Mathieson

Gionatan Danti

2017-08-25 19:34:50 UTC

Post by Lindsay Mathieson
I run Replica 3 VM hosting (gfapi) via a 3 node proxmox cluster. Have
done a lot of rolling node updates, power failures etc, never had a
problem. Performance is better than any other DFS I've tried (Ceph,
lizard/moose).

Hi, very interesting! Are you using client or server quorum?

Post by Lindsay Mathieson
Never did get DRDB working.

This surprise me: I found DRBD quite simple to use, albeit I mostly use
active/passive setup in production (with manual failover)

Post by Lindsay Mathieson
nb: ZFS Bricks, with each brick RAID10 - so a little paranoid on the
redundancy :)

Yeah, I remember you on the zfs-discuss mailing list ;)

Post by Lindsay Mathieson
For me, glusters biggest problem is its lack of flexibility in adding
bricks and nodes.And replacing them is an exercise in nail biting.
Hoping V4 improves on this though maybe that will lead to performance
trade offs.

Can you elaborate? What are the biggest problems/inconvenience?

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

l***@ulrar.net

2017-08-25 19:43:47 UTC

Post by Gionatan Danti
This surprise me: I found DRBD quite simple to use, albeit I mostly use
active/passive setup in production (with manual failover)

I think you are talking about DRBD 8, which is indeed very easy. DRBD 9
on the other hand, which is the one that compares to gluster (more or
less), is a whole other story. Never managed to make it work correctly
either

2017-08-25 20:11:09 UTC

Post by l***@ulrar.net
I think you are talking about DRBD 8, which is indeed very easy. DRBD 9
on the other hand, which is the one that compares to gluster (more or
less), is a whole other story. Never managed to make it work correctly
either

Yes, and I noticed that Digimer's Anvil project still uses DRBD8 as well.

https://www.alteeve.com/w/Build_an_m2_Anvil

If she is still using DRBD8 on her stuff, then you know that DRBD9 isn't
fully baked yet.

That being said we use DRBD on projects such as NFS servers and found
it to be reliable, easy to use and useful as long as we stayed with
active/passive.

Our experiments with active/active were very unsatisfactory. OCFS2 in
particular was very unhappy with lots of disk i/o, even just stats. We
tried GFS2 as well but bailed due to stability (lockup) issues (probably
our fault, but still not worth the effort).

Gionatan Danti

2017-08-25 21:00:13 UTC

Oh yes, absolutely DRBD version 8.4.x
In my opinion, DRBD 9.x needs to mature.

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

2017-08-25 19:48:31 UTC

Post by WK
2 node plus Arbiter. You NEED the arbiter or a third node. Do NOT try 2
node with a VM

This is true even if I manage locking at application level (via
virlock or sanlock)?

We ran Rep2 for years on 3.4. It does work if you are really,really
careful, But in a crash on one side, you might have lost some bits that
were on the fly. The VM would then try to heal.
Without sharding, big VMs take a while because the WHOLE VM file has to
be copied over. Then you might get Split-brain and have to stop the VM,
pick the good one, make sure that is healed on both sides and then
restart the VM.

Arbiter/Replica 3 prevents that. Sharding helps a lot as well by making
the heals really quick, though in a Replica 2 with sharding you no
longer have a nice big .img file sitting on each brick in plain view
and picking a split-brain winner is now WAY more complicated. You would
have to re-assemble things.

We were quite good and fixing broken Gluster 3.4 nodes, but we are
*much* happier with the Arbiter node and sharding. It is a huge difference.
We could go to Rep3 but we like the extra speed and we are comfortable
with the Arb limitations (we also have excellent off cluster backups
<grin>).

Post by Gionatan Danti
Also, on a two-node setup it is *guaranteed* for updates to one node
to put offline the whole volume?

If you still have quorum turned on, then yes. One side goes and you are
down.

Post by Gionatan Danti
On the other hand, a 3-way setup (or 2+arbiter) if free from all these
problems?

Yes, you can lose one of the three nodes and after the pause, everything
just continues. If you have a second failure before you can recover,
then you have lost quorum.

If that second failure is the other actual replica, then you could get
into a situation where the arbiter isn't happy with either copy when you
come back up and of course the arbiter doesn't have a good copy itself.
Pavel alluded to something like that when describing his problem.

That is where replica 3 helps. In theory, with replica 3, you could lose
2 nodes and still have a reasonable copy of your VM, though you've lost
quorum and are still down. At that point, *I* would kill the two bad
nodes (STOMITH) to prevent them from coming back AND turn off quorum.
You could then run on the single node until you can save/copy those VM
images, preferably by migrating off that volume completely. Create a
remote pool using SSHFS if you have nothing else available. THEN I would
go back and fix the gluster cluster and migrate back into it.

Replica2/Replica3 does not matter if you lose your Gluster network
switch, but again the Arb or Rep3 setup makes it easier to recover. I
suppose the only advantage of Replica2 is that you can use a cross over
cable and not worry about losing the switch, but bonding/teaming works
well and there are bonding modes that don't require the same switch for
the bond slaves. So you can build in some redundancy there as well.

Gionatan Danti

2017-08-25 21:08:14 UTC

Post by WK
We ran Rep2 for years on 3.4. It does work if you are really,really
careful, But in a crash on one side, you might have lost some bits
that were on the fly. The VM would then try to heal.
Without sharding, big VMs take a while because the WHOLE VM file has
to be copied over. Then you might get Split-brain and have to stop the
VM, pick the good one, make sure that is healed on both sides and then
restart the VM.

Ok, so sharding needs to be enabled for VM disk storage, otherwise heal
time skyrockets.

Post by WK
Arbiter/Replica 3 prevents that. Sharding helps a lot as well by
making the heals really quick, though in a Replica 2 with sharding you
no longer have a nice big .img file sitting on each brick in plain
view and picking a split-brain winner is now WAY more complicated. You
would have to re-assemble things.

This concern me, and it is the reason I would like to avoid sharding.
How can I recover from such a situation? How can I "decide" which
(reconstructed) file is the one to keep rather than to delete?

Post by WK
We were quite good and fixing broken Gluster 3.4 nodes, but we are
*much* happier with the Arbiter node and sharding. It is a huge difference.
We could go to Rep3 but we like the extra speed and we are comfortable
with the Arb limitations (we also have excellent off cluster backups
<grin>).

Post by Gionatan Danti
Also, on a two-node setup it is *guaranteed* for updates to one node
to put offline the whole volume?

If you still have quorum turned on, then yes. One side goes and you are
down.

Post by Gionatan Danti
On the other hand, a 3-way setup (or 2+arbiter) if free from all these
problems?

Yes, you can lose one of the three nodes and after the pause,
everything just continues. If you have a second failure before you can
recover, then you have lost quorum.
If that second failure is the other actual replica, then you could get
into a situation where the arbiter isn't happy with either copy when
you come back up and of course the arbiter doesn't have a good copy
itself. Pavel alluded to something like that when describing his
problem.
That is where replica 3 helps. In theory, with replica 3, you could
lose 2 nodes and still have a reasonable copy of your VM, though
you've lost quorum and are still down. At that point, *I* would kill
the two bad nodes (STOMITH) to prevent them from coming back AND turn
off quorum. You could then run on the single node until you can
save/copy those VM images, preferably by migrating off that volume
completely. Create a remote pool using SSHFS if you have nothing else
available. THEN I would go back and fix the gluster cluster and
migrate back into it.
Replica2/Replica3 does not matter if you lose your Gluster network
switch, but again the Arb or Rep3 setup makes it easier to recover. I
suppose the only advantage of Replica2 is that you can use a cross
over cable and not worry about losing the switch, but bonding/teaming
works well and there are bonding modes that don't require the same
switch for the bond slaves. So you can build in some redundancy there
as well.

Thank you for the very valuable informations.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

l***@ulrar.net

2017-08-25 21:21:03 UTC

Post by Gionatan Danti
This concern me, and it is the reason I would like to avoid sharding.
How can I recover from such a situation? How can I "decide" which
(reconstructed) file is the one to keep rather than to delete?

No need, on a replica 3 that just doesn't happen. That's the main
advantage of it, that and the fact that you can perform operations on
your servers without having the volume go down.

For a replica 2 though, it will happen. With or without sharding the
operation is the same, it involves fiddling with gfids and is a bit
annoying, but not that hard for one file. But with sharding enabled
you'll need to pick each split brained shard out, which is I imagine a
huge pain .. Again, just don't do 2 nodes, it's a _bad_ idea. Add at the
very least an arbiter.

2017-08-25 23:13:45 UTC

Post by l***@ulrar.net

No need, on a replica 3 that just doesn't happen. That's the main
advantage of it, that and the fact that you can perform operations on
your servers without having the volume go down.
For a replica 2 though, it will happen. With or without sharding the
operation is the same, it involves fiddling with gfids and is a bit
annoying, but not that hard for one file. But with sharding enabled
you'll need to pick each split brained shard out, which is I imagine a
huge pain .. Again, just don't do 2 nodes, it's a _bad_ idea. Add at the
very least an arbiter.

Big +1 on what was Kevin just said. Just avoiding the problem is the
best strategy.

However, for the record, and if you really, really want to get deep
into the weeds on the subject, then the Gluster people have docs on
Split-Brain recovery.

https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/

and if you Google the topic, there are a lot of other blog posts,
emails, etc that discuss it.

I'd recommend reviewing those as well to wrap your head around what is
going on.

Gionatan Danti

2017-08-26 05:38:41 UTC

Post by WK
Big +1 on what was Kevin just said. Just avoiding the problem is the
best strategy.

Ok, never run Gluster with anything less than a replica2 + arbiter ;)

Post by WK
However, for the record, and if you really, really want to get deep
into the weeds on the subject, then the Gluster people have docs on
Split-Brain recovery.
https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/
and if you Google the topic, there are a lot of other blog posts,
emails, etc that discuss it.
I'd recommend reviewing those as well to wrap your head around what is
going on.

I'll surely give a look at the documentation. I have the "bad" habit of
not putting into production anything I know how to repair/cope with.

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

Gionatan Danti

2017-08-26 05:40:57 UTC

Post by Gionatan Danti
I'll surely give a look at the documentation. I have the "bad" habit
of not putting into production anything I know how to repair/cope
with.
Thanks.

Mmmm, this should read as:

"I have the "bad" habit of not putting into production anything I do NOT
know how to repair/cope with"

Really :D

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

Everton Brogliatto

2017-08-30 01:57:23 UTC

Ciao Gionatan,

I run Gluster 3.10.x (Replica 3 arbiter or 2 + 1 arbiter) to provide
storage for oVirt 4.x and I have had no major issues so far.
I have done online upgrades a couple of times, power losses, maintenance,
etc with no issues. Overall, it is very resilient.

Important thing to keep in mind is your network, I run the Gluster nodes on
a redundant network using bonding mode 1 and I have performed maintenance
on my switches, bringing one of them off-line at a time without causing
problems in my Gluster setup or in my running VMs.
Gluster recommendation is to enable jumbo frames across the
subnet/servers/switches you use for Gluster operations. Your switches must
support MTU 9000 + 208 at least.

There were two occasions where I purposely caused a split brain situation
and I was able to heal the files manually.

Volume performance tuning can make a significant difference in Gluster. As
others have mentioned previously, sharding is recommended when running VMs
as it will split big files in smaller pieces, making it easier for the
healing to occur.
When you enable sharding, the default sharding block size is 4MB which will
significantly reduce your writing speeds. oVirt recommends the shard block
size to be 512MB.
The volume options you are looking here are:
features.shard on
features.shard-block-size 512MB

I had an experimental setup in replica 2 using an older version of Gluster
few years ago and it was unstable, corrupt data and crashed many times. Do
not use replica 2. As others have already said, minimum is replica 2+1
arbiter.

If you have any questions that I perhaps can help with, drop me an email.

Regards,
Everton Brogliatto

Post by Gionatan Danti
I'll surely give a look at the documentation. I have the "bad" habit
of not putting into production anything I know how to repair/cope
with.
Thanks.

"I have the "bad" habit of not putting into production anything I do NOT
know how to repair/cope with"
Really :D
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Ivan Rossi

2017-08-30 15:07:44 UTC

There has ben a bug associated to sharding that led to VM corruption that
has been around for a long time (difficult to reproduce I understood). I
have not seen reports on that for some time after the last fix, so
hopefully now VM hosting is stable.

Post by Everton Brogliatto
Ciao Gionatan,
I run Gluster 3.10.x (Replica 3 arbiter or 2 + 1 arbiter) to provide
storage for oVirt 4.x and I have had no major issues so far.
I have done online upgrades a couple of times, power losses, maintenance,
etc with no issues. Overall, it is very resilient.
Important thing to keep in mind is your network, I run the Gluster nodes
on a redundant network using bonding mode 1 and I have performed
maintenance on my switches, bringing one of them off-line at a time without
causing problems in my Gluster setup or in my running VMs.
Gluster recommendation is to enable jumbo frames across the
subnet/servers/switches you use for Gluster operations. Your switches must
support MTU 9000 + 208 at least.
There were two occasions where I purposely caused a split brain situation
and I was able to heal the files manually.
Volume performance tuning can make a significant difference in Gluster. As
others have mentioned previously, sharding is recommended when running VMs
as it will split big files in smaller pieces, making it easier for the
healing to occur.
When you enable sharding, the default sharding block size is 4MB which
will significantly reduce your writing speeds. oVirt recommends the shard
block size to be 512MB.
features.shard on
features.shard-block-size 512MB
I had an experimental setup in replica 2 using an older version of Gluster
few years ago and it was unstable, corrupt data and crashed many times. Do
not use replica 2. As others have already said, minimum is replica 2+1
arbiter.
If you have any questions that I perhaps can help with, drop me an email.
Regards,
Everton Brogliatto

Post by Gionatan Danti
I'll surely give a look at the documentation. I have the "bad" habit
of not putting into production anything I know how to repair/cope
with.
Thanks.

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

l***@ulrar.net

2017-08-30 23:17:27 UTC

Solved as to 3.7.12. The only bug left is when adding new bricks to
create a new replica set, now sure where we are now on that bug but
that's not a common operation (well, at least for me).

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM corruption that
has been around for a long time (difficult to reproduce I understood). I
have not seen reports on that for some time after the last fix, so
hopefully now VM hosting is stable.

Post by Gionatan Danti
I'll surely give a look at the documentation. I have the "bad" habit
of not putting into production anything I know how to repair/cope
with.
Thanks.

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Gionatan Danti

2017-09-03 20:23:01 UTC

Post by l***@ulrar.net
Solved as to 3.7.12. The only bug left is when adding new bricks to
create a new replica set, now sure where we are now on that bug but
that's not a common operation (well, at least for me).

Hi, same question here: is any specific information on the bug
available?
Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

Ivan Rossi

2017-09-04 17:27:04 UTC

The latter one is the one I have been referring to. And it is pretty
dangerous Imho

maintenance,

Post by Everton Brogliatto
etc with no issues. Overall, it is very resilient.
Important thing to keep in mind is your network, I run the Gluster

nodes

Post by Everton Brogliatto
on a redundant network using bonding mode 1 and I have performed
maintenance on my switches, bringing one of them off-line at a time

without

Post by Everton Brogliatto
causing problems in my Gluster setup or in my running VMs.
Gluster recommendation is to enable jumbo frames across the
subnet/servers/switches you use for Gluster operations. Your switches

must

Post by Everton Brogliatto
support MTU 9000 + 208 at least.
There were two occasions where I purposely caused a split brain

situation

Post by Everton Brogliatto
and I was able to heal the files manually.
Volume performance tuning can make a significant difference in

Gluster. As

Post by Everton Brogliatto
others have mentioned previously, sharding is recommended when running

VMs

Post by Everton Brogliatto
as it will split big files in smaller pieces, making it easier for the
healing to occur.
When you enable sharding, the default sharding block size is 4MB which
will significantly reduce your writing speeds. oVirt recommends the

shard

Post by Everton Brogliatto
block size to be 512MB.
features.shard on
features.shard-block-size 512MB
I had an experimental setup in replica 2 using an older version of

Gluster

Post by Everton Brogliatto
few years ago and it was unstable, corrupt data and crashed many

times. Do

Post by Everton Brogliatto
not use replica 2. As others have already said, minimum is replica 2+1
arbiter.
If you have any questions that I perhaps can help with, drop me an

email.

Post by Everton Brogliatto
Regards,
Everton Brogliatto

Post by Gionatan Danti
I'll surely give a look at the documentation. I have the "bad" habit
of not putting into production anything I know how to repair/cope
with.
Thanks.

"I have the "bad" habit of not putting into production anything I do

NOT

Post by Everton Brogliatto

Post by Gionatan Danti
know how to repair/cope with"
Really :D
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Gionatan Danti

2017-09-05 17:22:46 UTC

Post by Ivan Rossi
The latter one is the one I have been referring to. And it is pretty
dangerous Imho

Sorry, I can not find the bug report/entry by myself.
Can you link to some more information, or explain which bug are your
referring and how to trigger it?

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

Gionatan Danti

2017-09-03 20:21:33 UTC

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM corruption
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after the
last fix, so hopefully now VM hosting is stable.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

l***@ulrar.net

2017-09-03 22:25:22 UTC

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM corruption
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after the
last fix, so hopefully now VM hosting is stable.

I don't have a solution, instead of growing my volumes I just create new
ones. Couldn't tell you if it's solved in recent release, never had the
courage to try it out :)
It's a bit hard to trigger too so having it work once might not be
enough.

Pavel Szalbot

2017-09-06 14:12:21 UTC

Hi all,

I have promised to do some testing and I finally find some time and
infrastructure.

So I have 3 servers with Gluster 3.10.5 on CentOS 7. I created
replicated volume with arbiter (2+1) and VM on KVM (via Openstack)
with disk accessible through gfapi. Volume group is set to virt
(gluster volume set gv_openstack_1 virt). VM runs current (all
packages updated) Ubuntu Xenial.

I set up following fio job:

[job1]
ioengine=libaio
size=1g
loops=16
bs=512k
direct=1
filename=/tmp/fio.data2

When I run fio fio.job and reboot one of the data nodes, IO statistics
reported by fio drop to 0KB/0KB and 0 IOPS. After a while, root
filesystem gets remounted as read-only.

If you care about infrastructure, setup details etc., do not hesitate to ask.

Gluster info on volume:

Volume Name: gv_openstack_1
Type: Replicate
Volume ID: 2425ae63-3765-4b5e-915b-e132e0d3fff1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gfs-2.san:/export/gfs/gv_1
Brick2: gfs-3.san:/export/gfs/gv_1
Brick3: docker3.san:/export/gfs/gv_1 (arbiter)
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off

Partial KVM XML dump:

<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source protocol='gluster'
name='gv_openstack_1/volume-77ebfd13-6a92-4f38-b036-e9e55d752e1e'>
<host name='10.0.1.201' port='24007'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>77ebfd13-6a92-4f38-b036-e9e55d752e1e</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>

Networking is LACP on data nodes, stack of Juniper EX4550's (10Gbps
SFP+), separate VLAN for Gluster traffic, SSD only on Gluster all
nodes (including arbiter).

I would really love to know what am I doing wrong, because this is my
experience with Gluster for a long time a and a reason I would not
recommend it as VM storage backend in production environment where you
cannot start/stop VMs on your own (e.g. providing private clouds for
customers).
-ps

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM corruption
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after the
last fix, so hopefully now VM hosting is stable.

Mmmm... this is precisely the kind of bug that scares me... data corruption
:|
Any more information on what causes it and how to resolve? Even if in newer
Gluster releases it is a solved bug, knowledge on how to treat it would be
valuable.
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Alastair Neil

2017-09-06 19:59:14 UTC

you need to set

cluster.server-quorum-ratio 51%

Post by Gionatan Danti
Hi all,
I have promised to do some testing and I finally find some time and
infrastructure.
So I have 3 servers with Gluster 3.10.5 on CentOS 7. I created
replicated volume with arbiter (2+1) and VM on KVM (via Openstack)
with disk accessible through gfapi. Volume group is set to virt
(gluster volume set gv_openstack_1 virt). VM runs current (all
packages updated) Ubuntu Xenial.
[job1]
ioengine=libaio
size=1g
loops=16
bs=512k
direct=1
filename=/tmp/fio.data2
When I run fio fio.job and reboot one of the data nodes, IO statistics
reported by fio drop to 0KB/0KB and 0 IOPS. After a while, root
filesystem gets remounted as read-only.
If you care about infrastructure, setup details etc., do not hesitate to ask.
Volume Name: gv_openstack_1
Type: Replicate
Volume ID: 2425ae63-3765-4b5e-915b-e132e0d3fff1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Brick1: gfs-2.san:/export/gfs/gv_1
Brick2: gfs-3.san:/export/gfs/gv_1
Brick3: docker3.san:/export/gfs/gv_1 (arbiter)
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source protocol='gluster'
name='gv_openstack_1/volume-77ebfd13-6a92-4f38-b036-e9e55d752e1e'>
<host name='10.0.1.201' port='24007'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>77ebfd13-6a92-4f38-b036-e9e55d752e1e</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
Networking is LACP on data nodes, stack of Juniper EX4550's (10Gbps
SFP+), separate VLAN for Gluster traffic, SSD only on Gluster all
nodes (including arbiter).
I would really love to know what am I doing wrong, because this is my
experience with Gluster for a long time a and a reason I would not
recommend it as VM storage backend in production environment where you
cannot start/stop VMs on your own (e.g. providing private clouds for
customers).
-ps

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM corruption
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after the
last fix, so hopefully now VM hosting is stable.

Mmmm... this is precisely the kind of bug that scares me... data

corruption

Post by Gionatan Danti
:|
Any more information on what causes it and how to resolve? Even if in

newer

Post by Gionatan Danti
Gluster releases it is a solved bug, knowledge on how to treat it would

Post by Gionatan Danti
valuable.
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

l***@ulrar.net

2017-09-06 20:06:36 UTC

Mh, I never had to do that and I never had that problem. Is that an
arbiter specific thing ? With replica 3 it just works.

Post by Alastair Neil
you need to set
cluster.server-quorum-ratio 51%

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM corruption
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after the
last fix, so hopefully now VM hosting is stable.

Mmmm... this is precisely the kind of bug that scares me... data

corruption

Post by Gionatan Danti
:|
Any more information on what causes it and how to resolve? Even if in

newer

Post by Gionatan Danti
Gluster releases it is a solved bug, knowledge on how to treat it would

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Alastair Neil

2017-09-07 15:52:07 UTC

*shrug* I don't use arbiter for vm work loads just straight replica 3.
There are some gotchas with using an arbiter for VM workloads. If
quorum-type is auto and a brick that is not the arbiter drop out then if
the up brick is dirty as far as the arbiter is concerned i.e. the only good
copy is on the down brick you will get ENOTCONN and your VMs will halt on
IO.

Post by l***@ulrar.net
Mh, I never had to do that and I never had that problem. Is that an
arbiter specific thing ? With replica 3 it just works.

Post by Alastair Neil
you need to set
cluster.server-quorum-ratio 51%

Post by Gionatan Danti
ask.
Volume Name: gv_openstack_1
Type: Replicate
Volume ID: 2425ae63-3765-4b5e-915b-e132e0d3fff1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Brick1: gfs-2.san:/export/gfs/gv_1
Brick2: gfs-3.san:/export/gfs/gv_1
Brick3: docker3.san:/export/gfs/gv_1 (arbiter)
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source protocol='gluster'
name='gv_openstack_1/volume-77ebfd13-6a92-4f38-b036-e9e55d752e1e'>
<host name='10.0.1.201' port='24007'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>77ebfd13-6a92-4f38-b036-e9e55d752e1e</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
Networking is LACP on data nodes, stack of Juniper EX4550's (10Gbps
SFP+), separate VLAN for Gluster traffic, SSD only on Gluster all
nodes (including arbiter).
I would really love to know what am I doing wrong, because this is my
experience with Gluster for a long time a and a reason I would not
recommend it as VM storage backend in production environment where you
cannot start/stop VMs on your own (e.g. providing private clouds for
customers).
-ps

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM corruption
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after the
last fix, so hopefully now VM hosting is stable.

Mmmm... this is precisely the kind of bug that scares me... data

corruption

Post by Gionatan Danti
:|
Any more information on what causes it and how to resolve? Even if in

newer

Post by Gionatan Danti
Gluster releases it is a solved bug, knowledge on how to treat it

would

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-09-07 18:06:19 UTC

Hi Neil, docs mention two live nodes of replica 3 blaming each other and
refusing to do IO.

https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/#1-replica-3-volume

Post by Alastair Neil
*shrug* I don't use arbiter for vm work loads just straight replica 3.
There are some gotchas with using an arbiter for VM workloads. If
quorum-type is auto and a brick that is not the arbiter drop out then if
the up brick is dirty as far as the arbiter is concerned i.e. the only good
copy is on the down brick you will get ENOTCONN and your VMs will halt on
IO.

Post by l***@ulrar.net
Mh, I never had to do that and I never had that problem. Is that an
arbiter specific thing ? With replica 3 it just works.

Post by Alastair Neil
you need to set
cluster.server-quorum-ratio 51%

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM

corruption

Post by Ivan Rossi
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after

the

Post by Ivan Rossi
last fix, so hopefully now VM hosting is stable.

Mmmm... this is precisely the kind of bug that scares me... data

corruption

Post by Gionatan Danti
:|
Any more information on what causes it and how to resolve? Even if

Post by Gionatan Danti
newer

Post by Gionatan Danti
Gluster releases it is a solved bug, knowledge on how to treat it

would

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Alastair Neil

2017-09-07 19:25:08 UTC

True but to work your way into that problem with replica 3 is a lot harder
to achieve than with just replica 2 + arbiter.

Post by Pavel Szalbot
Hi Neil, docs mention two live nodes of replica 3 blaming each other and
refusing to do IO.
https://gluster.readthedocs.io/en/latest/Administrator%
20Guide/Split%20brain%20and%20ways%20to%20deal%20with%
20it/#1-replica-3-volume

Post by l***@ulrar.net
Mh, I never had to do that and I never had that problem. Is that an
arbiter specific thing ? With replica 3 it just works.

Post by Alastair Neil
you need to set
cluster.server-quorum-ratio 51%

statistics

Post by Gionatan Danti
reported by fio drop to 0KB/0KB and 0 IOPS. After a while, root
filesystem gets remounted as read-only.
If you care about infrastructure, setup details etc., do not

hesitate to

you

Post by Gionatan Danti
cannot start/stop VMs on your own (e.g. providing private clouds for
customers).
-ps

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM

corruption

Post by Ivan Rossi
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after

the

Post by Ivan Rossi
last fix, so hopefully now VM hosting is stable.

Mmmm... this is precisely the kind of bug that scares me... data

corruption

Post by Gionatan Danti
:|
Any more information on what causes it and how to resolve? Even if

Post by Gionatan Danti
newer

Post by Gionatan Danti
Gluster releases it is a solved bug, knowledge on how to treat it

would

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-09-08 05:04:53 UTC

Seems to be so, but if we look back at the described setup and procedure -
what is the reason for iops to stop/fail? Rebooting a node is somewhat
similar to updating gluster, replacing cabling etc. IMO this should not
always end up with arbiter blaming the other node and even though I did not
investigate this issue deeply, I do not believe the blame is the reason for
iops to drop.

Post by Alastair Neil
True but to work your way into that problem with replica 3 is a lot harder
to achieve than with just replica 2 + arbiter.

Post by Pavel Szalbot
Hi Neil, docs mention two live nodes of replica 3 blaming each other and
refusing to do IO.
https://gluster.readthedocs.io/en/latest/Administrator%20Gui
de/Split%20brain%20and%20ways%20to%20deal%20with%20it/#1-replica-3-volume

Post by l***@ulrar.net
Mh, I never had to do that and I never had that problem. Is that an
arbiter specific thing ? With replica 3 it just works.

Post by Alastair Neil
you need to set
cluster.server-quorum-ratio 51%

statistics

Post by Gionatan Danti
reported by fio drop to 0KB/0KB and 0 IOPS. After a while, root
filesystem gets remounted as read-only.
If you care about infrastructure, setup details etc., do not

hesitate to

Post by Gionatan Danti
experience with Gluster for a long time a and a reason I would not
recommend it as VM storage backend in production environment where

you

Post by Gionatan Danti
cannot start/stop VMs on your own (e.g. providing private clouds for
customers).
-ps

Post by Ivan Rossi
There has ben a bug associated to sharding that led to VM

corruption

Post by Ivan Rossi
that has been around for a long time (difficult to reproduce I
understood). I have not seen reports on that for some time after

the

Post by Ivan Rossi
last fix, so hopefully now VM hosting is stable.

Mmmm... this is precisely the kind of bug that scares me... data

corruption

Post by Gionatan Danti
:|
Any more information on what causes it and how to resolve? Even

if in

Post by Gionatan Danti
newer

Post by Gionatan Danti
Gluster releases it is a solved bug, knowledge on how to treat it

would

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-09-08 08:50:36 UTC

FYI I set up replica 3 (no arbiter this time), did the same thing -
rebooted one node during lots of file IO on VM and IO stopped.

As I mentioned either here or in another thread, this behavior is
caused by high default of network.ping-timeout. My main problem used
to be that setting it to low values like 3s or even 2s did not prevent
the FS to be mounted as read-only in the past (at least with arbiter)
and docs describe reconnect as very costly. If I set ping-timeout to
1s disaster of read-only mount is now prevented.

However I find it very strange because in the past I actually did end
up with read-only filesystem despite of the low ping-timeout.

With replica 3 after node reboot iftop shows data flowing only to the
one of remaining two nodes and there is no entry in heal info for the
volume. Explanation would be very much appreciated ;-)

Few minutes later I reverted back to replica 3 with arbiter (group
virt, ping-timeout 1). All nodes are up. During first fio run, VM
disconnected my ssh session, so I reconnected and saw ext4 problems in
dmesg. I deleted the VM and started a new one. Glustershd.log fills
with metadata heal shortly after fio job starts, but this time system
is stable.
Rebooting one of the nodes does not cause any problem (watching heal
log, i/o on vm).

So I decided to put more stress on VMs disk - I added second job with
direct=1 and started it (now both are running) while one gluster node
is still booting. What happened? One fio job reports "Bus error" and
VM segfaults when trying to run dmesg...

Is this gfapi related? Is this bug in arbiter?

Pavel Szalbot

2017-09-08 09:41:13 UTC

Back to replica 3 w/o arbiter. Two fio jobs running (direct=1 and
direct=0), rebooting one node... and VM dmesg looks like:

[ 483.862664] blk_update_request: I/O error, dev vda, sector 23125016
[ 483.898034] blk_update_request: I/O error, dev vda, sector 2161832
[ 483.901103] blk_update_request: I/O error, dev vda, sector 2161832
[ 483.904045] Aborting journal on device vda1-8.
[ 483.906959] blk_update_request: I/O error, dev vda, sector 2099200
[ 483.908306] blk_update_request: I/O error, dev vda, sector 2099200
[ 483.909585] Buffer I/O error on dev vda1, logical block 262144,
lost sync page write
[ 483.911121] blk_update_request: I/O error, dev vda, sector 2048
[ 483.912192] blk_update_request: I/O error, dev vda, sector 2048
[ 483.913221] Buffer I/O error on dev vda1, logical block 0, lost
sync page write
[ 483.914546] EXT4-fs error (device vda1):
ext4_journal_check_start:56: Detected aborted journal
[ 483.916230] EXT4-fs (vda1): Remounting filesystem read-only
[ 483.917231] EXT4-fs (vda1): previous I/O error to superblock detected
[ 483.917353] JBD2: Error -5 detected when updating journal
superblock for vda1-8.
[ 483.921106] blk_update_request: I/O error, dev vda, sector 2048
[ 483.922147] blk_update_request: I/O error, dev vda, sector 2048
[ 483.923107] Buffer I/O error on dev vda1, logical block 0, lost
sync page write

Root fs is read-only even with 1s ping-timeout...

I really hope I have been idiot for almost a year now and someone
shows what am I doing completely wrong because I dream about joining
the hordes of fellow colleagues who store multiple VMs in gluster and
never had a problem with it. I also suspect the CentOS libvirt version
to be the cause.

-ps

Post by Pavel Szalbot
FYI I set up replica 3 (no arbiter this time), did the same thing -
rebooted one node during lots of file IO on VM and IO stopped.
As I mentioned either here or in another thread, this behavior is
caused by high default of network.ping-timeout. My main problem used
to be that setting it to low values like 3s or even 2s did not prevent
the FS to be mounted as read-only in the past (at least with arbiter)
and docs describe reconnect as very costly. If I set ping-timeout to
1s disaster of read-only mount is now prevented.
However I find it very strange because in the past I actually did end
up with read-only filesystem despite of the low ping-timeout.
With replica 3 after node reboot iftop shows data flowing only to the
one of remaining two nodes and there is no entry in heal info for the
volume. Explanation would be very much appreciated ;-)
Few minutes later I reverted back to replica 3 with arbiter (group
virt, ping-timeout 1). All nodes are up. During first fio run, VM
disconnected my ssh session, so I reconnected and saw ext4 problems in
dmesg. I deleted the VM and started a new one. Glustershd.log fills
with metadata heal shortly after fio job starts, but this time system
is stable.
Rebooting one of the nodes does not cause any problem (watching heal
log, i/o on vm).
So I decided to put more stress on VMs disk - I added second job with
direct=1 and started it (now both are running) while one gluster node
is still booting. What happened? One fio job reports "Bus error" and
VM segfaults when trying to run dmesg...
Is this gfapi related? Is this bug in arbiter?

l***@ulrar.net

2017-09-08 09:42:50 UTC

Oh, you really don't want to go below 30s, I was told.
I'm using 30 seconds for the timeout, and indeed when a node goes down
the VM freez for 30 seconds, but I've never seen them go read only for
that.

I _only_ use virtio though, maybe it's that. What are you using ?

Post by Pavel Szalbot
Back to replica 3 w/o arbiter. Two fio jobs running (direct=1 and
[ 483.862664] blk_update_request: I/O error, dev vda, sector 23125016
[ 483.898034] blk_update_request: I/O error, dev vda, sector 2161832
[ 483.901103] blk_update_request: I/O error, dev vda, sector 2161832
[ 483.904045] Aborting journal on device vda1-8.
[ 483.906959] blk_update_request: I/O error, dev vda, sector 2099200
[ 483.908306] blk_update_request: I/O error, dev vda, sector 2099200
[ 483.909585] Buffer I/O error on dev vda1, logical block 262144,
lost sync page write
[ 483.911121] blk_update_request: I/O error, dev vda, sector 2048
[ 483.912192] blk_update_request: I/O error, dev vda, sector 2048
[ 483.913221] Buffer I/O error on dev vda1, logical block 0, lost
sync page write
ext4_journal_check_start:56: Detected aborted journal
[ 483.916230] EXT4-fs (vda1): Remounting filesystem read-only
[ 483.917231] EXT4-fs (vda1): previous I/O error to superblock detected
[ 483.917353] JBD2: Error -5 detected when updating journal
superblock for vda1-8.
[ 483.921106] blk_update_request: I/O error, dev vda, sector 2048
[ 483.922147] blk_update_request: I/O error, dev vda, sector 2048
[ 483.923107] Buffer I/O error on dev vda1, logical block 0, lost
sync page write
Root fs is read-only even with 1s ping-timeout...
I really hope I have been idiot for almost a year now and someone
shows what am I doing completely wrong because I dream about joining
the hordes of fellow colleagues who store multiple VMs in gluster and
never had a problem with it. I also suspect the CentOS libvirt version
to be the cause.
-ps

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-09-08 09:55:28 UTC

Post by l***@ulrar.net
Oh, you really don't want to go below 30s, I was told.
I'm using 30 seconds for the timeout, and indeed when a node goes down
the VM freez for 30 seconds, but I've never seen them go read only for
that.
I _only_ use virtio though, maybe it's that. What are you using ?

As previously mentioned (long thread:-)) - virtio.

Pavel Szalbot

2017-09-08 09:56:29 UTC

This is the qemu log of instance:

[2017-09-08 09:31:48.381077] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5)
[2017-09-08 09:31:48.382451] E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:31:48.382835] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.627341 (xid=0x27ab6)
[2017-09-08 09:31:48.383293] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.627403 (xid=0x27ab7)
[2017-09-08 09:31:48.383639] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.627444 (xid=0x27ab8)
[2017-09-08 09:31:48.384015] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9)
[2017-09-08 09:31:48.384056] E [MSGID: 114031]
[client-rpc-fops.c:1646:client3_3_entrylk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:31:48.384077] E [MSGID: 108007]
[afr-lk-common.c:825:afr_unlock_entrylk_cbk]
0-gv_openstack_1-replicate-0:
/.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on
gv_openstack_1-client
-1 [Transport endpoint is not connected]
[2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(WRITE(13)) called at 2017-09-08 09:31:45.626871 (xid=0x27ab4)
[2017-09-08 09:31:48.388095] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(READ(12)) called at 2017-09-08 09:31:45.628760 (xid=0x27aba)
[2017-09-08 09:31:48.388450] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(READ(12)) called at 2017-09-08 09:31:45.629421 (xid=0x27abb)
[2017-09-08 09:31:48.388757] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.647966 (xid=0x27abc)
[2017-09-08 09:31:48.389158] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.648128 (xid=0x27abd)
[2017-09-08 09:31:48.389502] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)

[2017-09-08 09:31:48.389838] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
[2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
[2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
[2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
[2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
[2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
[2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
[2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
[2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
[2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
[2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
[2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
[2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
[2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371815 (xid=0x27acc)
[2017-09-08 09:31:48.397273] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371913 (xid=0x27acd)
[2017-09-08 09:31:48.399192] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372010 (xid=0x27ace)
[2017-09-08 09:31:48.402229] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372147 (xid=0x27acf)
[2017-09-08 09:31:48.402716] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372247 (xid=0x27ad0)
[2017-09-08 09:31:48.403171] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372347 (xid=0x27ad1)
[2017-09-08 09:31:48.403786] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372438 (xid=0x27ad2)
[2017-09-08 09:31:48.404281] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372535 (xid=0x27ad3)
[2017-09-08 09:31:48.406718] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372632 (xid=0x27ad4)
[2017-09-08 09:31:48.407210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372752 (xid=0x27ad5)
[2017-09-08 09:31:48.407771] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372840 (xid=0x27ad6)
[2017-09-08 09:31:48.408253] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372943 (xid=0x27ad7)
[2017-09-08 09:31:48.408710] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373054 (xid=0x27ad8)
[2017-09-08 09:31:48.409190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373175 (xid=0x27ad9)
[2017-09-08 09:31:48.409671] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373276 (xid=0x27ada)
[2017-09-08 09:31:48.410135] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373375 (xid=0x27adb)
[2017-09-08 09:31:48.410638] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373466 (xid=0x27adc)
[2017-09-08 09:31:48.411136] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373552 (xid=0x27add)
[2017-09-08 09:31:48.412471] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373645 (xid=0x27ade)
[2017-09-08 09:31:48.413783] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373754 (xid=0x27adf)
[2017-09-08 09:31:48.414972] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373852 (xid=0x27ae0)
[2017-09-08 09:31:48.415517] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373936 (xid=0x27ae1)
[2017-09-08 09:31:48.416428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374046 (xid=0x27ae2)
[2017-09-08 09:31:48.417311] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374148 (xid=0x27ae3)
[2017-09-08 09:31:48.418227] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374235 (xid=0x27ae4)
[2017-09-08 09:31:48.419323] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374326 (xid=0x27ae5)
[2017-09-08 09:31:48.419910] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374409 (xid=0x27ae6)
[2017-09-08 09:31:48.420479] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374525 (xid=0x27ae7)
[2017-09-08 09:31:48.422134] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374607 (xid=0x27ae8)
[2017-09-08 09:31:48.424896] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374704 (xid=0x27ae9)
[2017-09-08 09:31:48.427236] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374817 (xid=0x27aea)
[2017-09-08 09:31:48.427802] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374923 (xid=0x27aeb)
[2017-09-08 09:31:48.428202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375012 (xid=0x27aec)
[2017-09-08 09:31:48.428472] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375130 (xid=0x27aed)
[2017-09-08 09:31:48.428728] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375245 (xid=0x27aee)
[2017-09-08 09:31:48.429042] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375333 (xid=0x27aef)
[2017-09-08 09:31:48.429308] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375427 (xid=0x27af0)
[2017-09-08 09:31:48.429557] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375531 (xid=0x27af1)
[2017-09-08 09:31:48.429800] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375651 (xid=0x27af2)
[2017-09-08 09:31:48.430054] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375759 (xid=0x27af3)
[2017-09-08 09:31:48.430351] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375857 (xid=0x27af4)
[2017-09-08 09:31:48.430722] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375956 (xid=0x27af5)
[2017-09-08 09:31:48.431245] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376060 (xid=0x27af6)
[2017-09-08 09:31:48.431796] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376157 (xid=0x27af7)
[2017-09-08 09:31:48.432181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376251 (xid=0x27af8)
[2017-09-08 09:31:48.432487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376346 (xid=0x27af9)
[2017-09-08 09:31:48.432785] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376457 (xid=0x27afa)
[2017-09-08 09:31:48.433089] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376540 (xid=0x27afb)
[2017-09-08 09:31:48.433383] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376630 (xid=0x27afc)
[2017-09-08 09:31:48.433678] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376734 (xid=0x27afd)
[2017-09-08 09:31:48.433967] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376827 (xid=0x27afe)
[2017-09-08 09:31:48.434274] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376950 (xid=0x27aff)
[2017-09-08 09:31:48.434496] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377054 (xid=0x27b00)
[2017-09-08 09:31:48.434778] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377153 (xid=0x27b01)
[2017-09-08 09:31:48.435069] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377249 (xid=0x27b02)
[2017-09-08 09:31:48.435356] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377352 (xid=0x27b03)
[2017-09-08 09:31:48.435637] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377448 (xid=0x27b04)
[2017-09-08 09:31:48.435917] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377554 (xid=0x27b05)
[2017-09-08 09:31:48.436210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377637 (xid=0x27b06)
[2017-09-08 09:31:48.436487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377739 (xid=0x27b07)
[2017-09-08 09:31:48.436765] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377836 (xid=0x27b08)
[2017-09-08 09:31:48.437050] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377938 (xid=0x27b09)
[2017-09-08 09:31:48.437332] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378052 (xid=0x27b0a)
[2017-09-08 09:31:48.437610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378149 (xid=0x27b0b)
[2017-09-08 09:31:48.437888] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378411 (xid=0x27b0c)
[2017-09-08 09:31:48.438176] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378515 (xid=0x27b0d)
[2017-09-08 09:31:48.438451] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378645 (xid=0x27b0e)
[2017-09-08 09:31:48.438727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:31:47.378696 (xid=0x27b0f)
[2017-09-08 09:31:48.438894] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378881 (xid=0x27b10)
[2017-09-08 09:31:48.439196] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379090 (xid=0x27b11)
[2017-09-08 09:31:48.439466] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379200 (xid=0x27b12)
[2017-09-08 09:31:48.439740] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379303 (xid=0x27b13)
[2017-09-08 09:31:48.440016] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379403 (xid=0x27b14)
[2017-09-08 09:31:48.440302] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379537 (xid=0x27b15)
[2017-09-08 09:31:48.440578] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379679 (xid=0x27b16)
[2017-09-08 09:31:48.440853] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379831 (xid=0x27b17)
[2017-09-08 09:31:48.441137] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379921 (xid=0x27b18)
[2017-09-08 09:31:48.441416] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380023 (xid=0x27b19)
[2017-09-08 09:31:48.441688] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380156 (xid=0x27b1a)
[2017-09-08 09:31:48.441960] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380270 (xid=0x27b1b)
[2017-09-08 09:31:48.442244] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380399 (xid=0x27b1c)
[2017-09-08 09:31:48.442520] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380523 (xid=0x27b1d)
[2017-09-08 09:31:48.442792] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380635 (xid=0x27b1e)
[2017-09-08 09:31:48.443079] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380741 (xid=0x27b1f)
[2017-09-08 09:31:48.443355] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380859 (xid=0x27b20)
[2017-09-08 09:31:48.443627] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380969 (xid=0x27b21)
The message "E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]" repeated 109 times between [2017-09-08
09:31:48.382451] and [2017-09-08 09:31:48.626176]
[2017-09-08 09:32:17.489261] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-2: server 10.0.1.200:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:32:17.490362] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055593 (xid=0x3071f)
[2017-09-08 09:32:17.490401] E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-2: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:32:17.490736] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055623 (xid=0x30720)
[2017-09-08 09:32:17.491133] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055654 (xid=0x30721)
[2017-09-08 09:32:17.491480] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055684 (xid=0x30722)
[2017-09-08 09:32:17.491791] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055712 (xid=0x30723)
[2017-09-08 09:32:17.492181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055741 (xid=0x30724)
[2017-09-08 09:32:17.492506] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055770 (xid=0x30725)
[2017-09-08 09:32:17.492811] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055799 (xid=0x30726)
[2017-09-08 09:32:17.493131] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSYNC(16)) called at 2017-09-08 09:32:13.675475 (xid=0x2f186)
[2017-09-08 09:32:17.493461] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.036379 (xid=0x3071d)
[2017-09-08 09:32:17.494173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.055558 (xid=0x3071e)
[2017-09-08 09:32:17.495483] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.077348 (xid=0x30727)
[2017-09-08 09:32:17.495787] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.078270 (xid=0x30728)
[2017-09-08 09:32:17.496077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484276 (xid=0x30729)
[2017-09-08 09:32:17.496378] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484349 (xid=0x3072a)
[2017-09-08 09:32:17.496655] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484435 (xid=0x3072b)
[2017-09-08 09:32:17.496923] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484493 (xid=0x3072c)
[2017-09-08 09:32:17.497202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484553 (xid=0x3072d)
[2017-09-08 09:32:17.497509] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484606 (xid=0x3072e)
[2017-09-08 09:32:17.497780] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484669 (xid=0x3072f)
[2017-09-08 09:32:17.498072] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484722 (xid=0x30730)
[2017-09-08 09:32:17.498397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484776 (xid=0x30731)
[2017-09-08 09:32:17.498666] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484830 (xid=0x30732)
[2017-09-08 09:32:17.499121] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484873 (xid=0x30733)
[2017-09-08 09:32:17.499711] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484956 (xid=0x30734)
[2017-09-08 09:32:17.500893] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485011 (xid=0x30735)
[2017-09-08 09:32:17.501880] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485128 (xid=0x30736)
[2017-09-08 09:32:17.502892] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485243 (xid=0x30737)
[2017-09-08 09:32:17.504237] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485307 (xid=0x30738)
[2017-09-08 09:32:17.505447] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485370 (xid=0x30739)
[2017-09-08 09:32:17.506503] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485462 (xid=0x3073a)
[2017-09-08 09:32:17.507727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485521 (xid=0x3073b)
[2017-09-08 09:32:17.508173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485578 (xid=0x3073c)
[2017-09-08 09:32:17.508925] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485630 (xid=0x3073d)
[2017-09-08 09:32:17.509262] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485688 (xid=0x3073e)
[2017-09-08 09:32:17.509641] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485764 (xid=0x3073f)
[2017-09-08 09:32:17.509957] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:32:16.485786 (xid=0x30740)
[2017-09-08 09:32:17.510224] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486075 (xid=0x30741)
[2017-09-08 09:32:17.511645] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486217 (xid=0x30742)
[2017-09-08 09:32:17.511983] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486340 (xid=0x30743)
[2017-09-08 09:32:17.512326] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486399 (xid=0x30744)
[2017-09-08 09:32:17.512860] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486543 (xid=0x30745)
[2017-09-08 09:32:17.514077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486637 (xid=0x30746)
[2017-09-08 09:32:17.514838] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486734 (xid=0x30747)
[2017-09-08 09:32:17.515413] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486842 (xid=0x30748)
[2017-09-08 09:32:17.516271] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486964 (xid=0x30749)
[2017-09-08 09:32:17.516775] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487082 (xid=0x3074a)
[2017-09-08 09:32:17.517195] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487190 (xid=0x3074b)
[2017-09-08 09:32:17.517635] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487289 (xid=0x3074c)
[2017-09-08 09:32:17.519469] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487400 (xid=0x3074d)
[2017-09-08 09:32:17.520008] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487485 (xid=0x3074e)
[2017-09-08 09:32:17.520411] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487617 (xid=0x3074f)
[2017-09-08 09:32:17.520891] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487699 (xid=0x30750)
[2017-09-08 09:32:17.521190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487836 (xid=0x30751)
[2017-09-08 09:32:17.521489] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487914 (xid=0x30752)
[2017-09-08 09:32:17.521932] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488019 (xid=0x30753)
[2017-09-08 09:32:17.522218] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488125 (xid=0x30754)
[2017-09-08 09:32:17.522548] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488201 (xid=0x30755)
[2017-09-08 09:32:17.524259] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488302 (xid=0x30756)
[2017-09-08 09:32:17.524610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488375 (xid=0x30757)
[2017-09-08 09:32:17.524892] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488463 (xid=0x30758)
[2017-09-08 09:32:17.525911] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488555 (xid=0x30759)
[2017-09-08 09:32:17.526272] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488635 (xid=0x3075a)
[2017-09-08 09:32:17.526555] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488713 (xid=0x3075b)
[2017-09-08 09:32:17.528152] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488793 (xid=0x3075c)
[2017-09-08 09:32:17.528524] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488870 (xid=0x3075d)
[2017-09-08 09:32:17.529167] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488948 (xid=0x3075e)
[2017-09-08 09:32:17.529571] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.489064 (xid=0x3075f)
[2017-09-08 09:32:17.529863] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.489164 (xid=0x30760)
[2017-09-08 09:32:17.530152] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.489247 (xid=0x30761)
[2017-09-08 09:32:33.490188] E [socket.c:2316:socket_connect_finish]
0-gv_openstack_1-client-1: connection to 10.0.1.202:24007 failed (No
route to host); disconnecting socket
The message "E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-2: remote operation failed [Transport endpoint
is not connected]" repeated 72 times between [2017-09-08
09:32:17.490401] and [2017-09-08 09:32:17.530223]
2017-09-08T09:41:36.512856Z qemu-kvm: terminating on signal 15 from pid 1528
[2017-09-08 09:41:36.616581] E [MSGID: 108006]
[afr-common.c:4799:afr_notify] 0-gv_openstack_1-replicate-0: All
subvolumes are down. Going offline until atleast one of them comes
back up.
2017-09-08 09:41:37.714+0000: shutting down
-ps

Oh, you really don't want to go below 30s, I was told.
I'm using 30 seconds for the timeout, and indeed when a node goes down
the VM freez for 30 seconds, but I've never seen them go read only for
that.
I _only_ use virtio though, maybe it's that. What are you using ?

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Diego Remolina

2017-09-08 10:19:43 UTC

The issue of I/O stopping may also be with glusterfsd not being properly
killed before rebooting the server.

For example in RHEL 7.4 with official Gluster 3.8.4, the glusterd service
does *not* stop glusterfsd when you run systemctl stop glusterd

So give this a try on the nose you wish to reboot:

1. Stop glusterd
2. Check if glusterfsd processes are still running. If they are, use:
killall glusterfsd
Double check no more glusterfsd processes are running.
3. Now reboot the server, there should be no stopping of I/O on the VMs.

I was able to reproduce the stopping I/O behaviour yesterday on server
reboots without stopping glusterfsd processes. JoeJulian recommends against
changing network ping timeout. We discussed this via IRC recently.

Diego

On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <***@gmail.com> wrote:

This is the qemu log of instance:

[2017-09-08 09:31:48.381077] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->

/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)
[2017-09-08 09:31:48.389838] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
[2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
[2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
[2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
[2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
[2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
[2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
[2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
[2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
[2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
[2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
[2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
[2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
[2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371815 (xid=0x27acc)
[2017-09-08 09:31:48.397273] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371913 (xid=0x27acd)
[2017-09-08 09:31:48.399192] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372010 (xid=0x27ace)
[2017-09-08 09:31:48.402229] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372147 (xid=0x27acf)
[2017-09-08 09:31:48.402716] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372247 (xid=0x27ad0)
[2017-09-08 09:31:48.403171] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372347 (xid=0x27ad1)
[2017-09-08 09:31:48.403786] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372438 (xid=0x27ad2)
[2017-09-08 09:31:48.404281] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372535 (xid=0x27ad3)
[2017-09-08 09:31:48.406718] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372632 (xid=0x27ad4)
[2017-09-08 09:31:48.407210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372752 (xid=0x27ad5)
[2017-09-08 09:31:48.407771] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372840 (xid=0x27ad6)
[2017-09-08 09:31:48.408253] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372943 (xid=0x27ad7)
[2017-09-08 09:31:48.408710] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373054 (xid=0x27ad8)
[2017-09-08 09:31:48.409190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373175 (xid=0x27ad9)
[2017-09-08 09:31:48.409671] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373276 (xid=0x27ada)
[2017-09-08 09:31:48.410135] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373375 (xid=0x27adb)
[2017-09-08 09:31:48.410638] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373466 (xid=0x27adc)
[2017-09-08 09:31:48.411136] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373552 (xid=0x27add)
[2017-09-08 09:31:48.412471] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373645 (xid=0x27ade)
[2017-09-08 09:31:48.413783] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373754 (xid=0x27adf)
[2017-09-08 09:31:48.414972] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373852 (xid=0x27ae0)
[2017-09-08 09:31:48.415517] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373936 (xid=0x27ae1)
[2017-09-08 09:31:48.416428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374046 (xid=0x27ae2)
[2017-09-08 09:31:48.417311] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374148 (xid=0x27ae3)
[2017-09-08 09:31:48.418227] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374235 (xid=0x27ae4)
[2017-09-08 09:31:48.419323] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374326 (xid=0x27ae5)
[2017-09-08 09:31:48.419910] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374409 (xid=0x27ae6)
[2017-09-08 09:31:48.420479] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374525 (xid=0x27ae7)
[2017-09-08 09:31:48.422134] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374607 (xid=0x27ae8)
[2017-09-08 09:31:48.424896] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374704 (xid=0x27ae9)
[2017-09-08 09:31:48.427236] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374817 (xid=0x27aea)
[2017-09-08 09:31:48.427802] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374923 (xid=0x27aeb)
[2017-09-08 09:31:48.428202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375012 (xid=0x27aec)
[2017-09-08 09:31:48.428472] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375130 (xid=0x27aed)
[2017-09-08 09:31:48.428728] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375245 (xid=0x27aee)
[2017-09-08 09:31:48.429042] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375333 (xid=0x27aef)
[2017-09-08 09:31:48.429308] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375427 (xid=0x27af0)
[2017-09-08 09:31:48.429557] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375531 (xid=0x27af1)
[2017-09-08 09:31:48.429800] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375651 (xid=0x27af2)
[2017-09-08 09:31:48.430054] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375759 (xid=0x27af3)
[2017-09-08 09:31:48.430351] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375857 (xid=0x27af4)
[2017-09-08 09:31:48.430722] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375956 (xid=0x27af5)
[2017-09-08 09:31:48.431245] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376060 (xid=0x27af6)
[2017-09-08 09:31:48.431796] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376157 (xid=0x27af7)
[2017-09-08 09:31:48.432181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376251 (xid=0x27af8)
[2017-09-08 09:31:48.432487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376346 (xid=0x27af9)
[2017-09-08 09:31:48.432785] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376457 (xid=0x27afa)
[2017-09-08 09:31:48.433089] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376540 (xid=0x27afb)
[2017-09-08 09:31:48.433383] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376630 (xid=0x27afc)
[2017-09-08 09:31:48.433678] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376734 (xid=0x27afd)
[2017-09-08 09:31:48.433967] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376827 (xid=0x27afe)
[2017-09-08 09:31:48.434274] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376950 (xid=0x27aff)
[2017-09-08 09:31:48.434496] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377054 (xid=0x27b00)
[2017-09-08 09:31:48.434778] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377153 (xid=0x27b01)
[2017-09-08 09:31:48.435069] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377249 (xid=0x27b02)
[2017-09-08 09:31:48.435356] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377352 (xid=0x27b03)
[2017-09-08 09:31:48.435637] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377448 (xid=0x27b04)
[2017-09-08 09:31:48.435917] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377554 (xid=0x27b05)
[2017-09-08 09:31:48.436210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377637 (xid=0x27b06)
[2017-09-08 09:31:48.436487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377739 (xid=0x27b07)
[2017-09-08 09:31:48.436765] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377836 (xid=0x27b08)
[2017-09-08 09:31:48.437050] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377938 (xid=0x27b09)
[2017-09-08 09:31:48.437332] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378052 (xid=0x27b0a)
[2017-09-08 09:31:48.437610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378149 (xid=0x27b0b)
[2017-09-08 09:31:48.437888] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378411 (xid=0x27b0c)
[2017-09-08 09:31:48.438176] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378515 (xid=0x27b0d)
[2017-09-08 09:31:48.438451] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378645 (xid=0x27b0e)
[2017-09-08 09:31:48.438727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:31:47.378696 (xid=0x27b0f)
[2017-09-08 09:31:48.438894] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378881 (xid=0x27b10)
[2017-09-08 09:31:48.439196] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379090 (xid=0x27b11)
[2017-09-08 09:31:48.439466] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379200 (xid=0x27b12)
[2017-09-08 09:31:48.439740] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379303 (xid=0x27b13)
[2017-09-08 09:31:48.440016] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379403 (xid=0x27b14)
[2017-09-08 09:31:48.440302] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379537 (xid=0x27b15)
[2017-09-08 09:31:48.440578] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379679 (xid=0x27b16)
[2017-09-08 09:31:48.440853] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379831 (xid=0x27b17)
[2017-09-08 09:31:48.441137] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379921 (xid=0x27b18)
[2017-09-08 09:31:48.441416] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380023 (xid=0x27b19)
[2017-09-08 09:31:48.441688] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380156 (xid=0x27b1a)
[2017-09-08 09:31:48.441960] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380270 (xid=0x27b1b)
[2017-09-08 09:31:48.442244] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380399 (xid=0x27b1c)
[2017-09-08 09:31:48.442520] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380523 (xid=0x27b1d)
[2017-09-08 09:31:48.442792] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380635 (xid=0x27b1e)
[2017-09-08 09:31:48.443079] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380741 (xid=0x27b1f)
[2017-09-08 09:31:48.443355] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380859 (xid=0x27b20)
[2017-09-08 09:31:48.443627] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380969 (xid=0x27b21)
The message "E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]" repeated 109 times between [2017-09-08
09:31:48.382451] and [2017-09-08 09:31:48.626176]
[2017-09-08 09:32:17.489261] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-2: server 10.0.1.200:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:32:17.490362] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055593 (xid=0x3071f)
[2017-09-08 09:32:17.490401] E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-2: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:32:17.490736] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055623 (xid=0x30720)
[2017-09-08 09:32:17.491133] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055654 (xid=0x30721)
[2017-09-08 09:32:17.491480] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055684 (xid=0x30722)
[2017-09-08 09:32:17.491791] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055712 (xid=0x30723)
[2017-09-08 09:32:17.492181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055741 (xid=0x30724)
[2017-09-08 09:32:17.492506] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055770 (xid=0x30725)
[2017-09-08 09:32:17.492811] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055799 (xid=0x30726)
[2017-09-08 09:32:17.493131] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSYNC(16)) called at 2017-09-08 09:32:13.675475 (xid=0x2f186)
[2017-09-08 09:32:17.493461] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.036379 (xid=0x3071d)
[2017-09-08 09:32:17.494173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.055558 (xid=0x3071e)
[2017-09-08 09:32:17.495483] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.077348 (xid=0x30727)
[2017-09-08 09:32:17.495787] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.078270 (xid=0x30728)
[2017-09-08 09:32:17.496077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484276 (xid=0x30729)
[2017-09-08 09:32:17.496378] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484349 (xid=0x3072a)
[2017-09-08 09:32:17.496655] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484435 (xid=0x3072b)
[2017-09-08 09:32:17.496923] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484493 (xid=0x3072c)
[2017-09-08 09:32:17.497202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484553 (xid=0x3072d)
[2017-09-08 09:32:17.497509] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484606 (xid=0x3072e)
[2017-09-08 09:32:17.497780] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484669 (xid=0x3072f)
[2017-09-08 09:32:17.498072] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484722 (xid=0x30730)
[2017-09-08 09:32:17.498397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484776 (xid=0x30731)
[2017-09-08 09:32:17.498666] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484830 (xid=0x30732)
[2017-09-08 09:32:17.499121] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484873 (xid=0x30733)
[2017-09-08 09:32:17.499711] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484956 (xid=0x30734)
[2017-09-08 09:32:17.500893] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485011 (xid=0x30735)
[2017-09-08 09:32:17.501880] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485128 (xid=0x30736)
[2017-09-08 09:32:17.502892] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485243 (xid=0x30737)
[2017-09-08 09:32:17.504237] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485307 (xid=0x30738)
[2017-09-08 09:32:17.505447] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485370 (xid=0x30739)
[2017-09-08 09:32:17.506503] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485462 (xid=0x3073a)
[2017-09-08 09:32:17.507727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485521 (xid=0x3073b)
[2017-09-08 09:32:17.508173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485578 (xid=0x3073c)
[2017-09-08 09:32:17.508925] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485630 (xid=0x3073d)
[2017-09-08 09:32:17.509262] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485688 (xid=0x3073e)
[2017-09-08 09:32:17.509641] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485764 (xid=0x3073f)
[2017-09-08 09:32:17.509957] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:32:16.485786 (xid=0x30740)
[2017-09-08 09:32:17.510224] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486075 (xid=0x30741)
[2017-09-08 09:32:17.511645] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486217 (xid=0x30742)
[2017-09-08 09:32:17.511983] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486340 (xid=0x30743)
[2017-09-08 09:32:17.512326] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486399 (xid=0x30744)
[2017-09-08 09:32:17.512860] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486543 (xid=0x30745)
[2017-09-08 09:32:17.514077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486637 (xid=0x30746)
[2017-09-08 09:32:17.514838] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486734 (xid=0x30747)
[2017-09-08 09:32:17.515413] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486842 (xid=0x30748)
[2017-09-08 09:32:17.516271] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486964 (xid=0x30749)
[2017-09-08 09:32:17.516775] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487082 (xid=0x3074a)
[2017-09-08 09:32:17.517195] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487190 (xid=0x3074b)
[2017-09-08 09:32:17.517635] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487289 (xid=0x3074c)
[2017-09-08 09:32:17.519469] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487400 (xid=0x3074d)
[2017-09-08 09:32:17.520008] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487485 (xid=0x3074e)
[2017-09-08 09:32:17.520411] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487617 (xid=0x3074f)
[2017-09-08 09:32:17.520891] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487699 (xid=0x30750)
[2017-09-08 09:32:17.521190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487836 (xid=0x30751)
[2017-09-08 09:32:17.521489] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487914 (xid=0x30752)
[2017-09-08 09:32:17.521932] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488019 (xid=0x30753)
[2017-09-08 09:32:17.522218] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488125 (xid=0x30754)
[2017-09-08 09:32:17.522548] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488201 (xid=0x30755)
[2017-09-08 09:32:17.524259] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488302 (xid=0x30756)
[2017-09-08 09:32:17.524610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488375 (xid=0x30757)
[2017-09-08 09:32:17.524892] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488463 (xid=0x30758)
[2017-09-08 09:32:17.525911] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488555 (xid=0x30759)
[2017-09-08 09:32:17.526272] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488635 (xid=0x3075a)
[2017-09-08 09:32:17.526555] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488713 (xid=0x3075b)
[2017-09-08 09:32:17.528152] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488793 (xid=0x3075c)
[2017-09-08 09:32:17.528524] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488870 (xid=0x3075d)
[2017-09-08 09:32:17.529167] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488948 (xid=0x3075e)
[2017-09-08 09:32:17.529571] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc....

Pavel Szalbot

2017-09-08 10:36:33 UTC

Hi Diego,

indeed glusterfsd processes are runnin and it is the reason I do
server reboot instead of systemctl glusterd stop. Is killall different
from reboot in a way glusterfsd processes are terminated in CentOS
(init 1?)?

However I will try this and let you know.

-ps

Post by Diego Remolina
The issue of I/O stopping may also be with glusterfsd not being properly
killed before rebooting the server.
For example in RHEL 7.4 with official Gluster 3.8.4, the glusterd service
does *not* stop glusterfsd when you run systemctl stop glusterd
1. Stop glusterd
killall glusterfsd
Double check no more glusterfsd processes are running.
3. Now reboot the server, there should be no stopping of I/O on the VMs.
I was able to reproduce the stopping I/O behaviour yesterday on server
reboots without stopping glusterfsd processes. JoeJulian recommends against
changing network ping timeout. We discussed this via IRC recently.
Diego
[2017-09-08 09:31:48.381077] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9)
[2017-09-08 09:31:48.384056] E [MSGID: 114031]
[client-rpc-fops.c:1646:client3_3_entrylk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:31:48.384077] E [MSGID: 108007]
[afr-lk-common.c:825:afr_unlock_entrylk_cbk]
/.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on
gv_openstack_1-client
-1 [Transport endpoint is not connected]
[2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)

Diego Remolina

2017-09-08 10:43:30 UTC

This is exactly the problem,

Systemctl stop glusterd does *not* kill the brick processes.

On CentOS with gluster 3.10.x there is also a service, meant to only stop
glusterfsd (brick processes). I think the reboot process may not be
properly stopping glusterfsd or network or firewall may be stopped before
glusterfsd and so the nodes go into the long timeout.

Once again , in my case a simple reboot hangs my VMs, whereas stopping
glusterd *and* glusterfsd processes, then reboot works perfectly.

Diego

On Sep 8, 2017 6:36 AM, "Pavel Szalbot" <***@gmail.com> wrote:

Hi Diego,

indeed glusterfsd processes are runnin and it is the reason I do
server reboot instead of systemctl glusterd stop. Is killall different
from reboot in a way glusterfsd processes are terminated in CentOS
(init 1?)?

However I will try this and let you know.

-ps

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9)
[2017-09-08 09:31:48.384056] E [MSGID: 114031]
[client-rpc-fops.c:1646:client3_3_entrylk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:31:48.384077] E [MSGID: 108007]
[afr-lk-common.c:825:afr_unlock_entrylk_cbk]
/.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on
gv_openstack_1-client
-1 [Transport endpoint is not connected]
[2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
[2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
[2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
[2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
[2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
[2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
[2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
[2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
[2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
[2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
[2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
[2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
[2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
[2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371815 (xid=0x27acc)
[2017-09-08 09:31:48.397273] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371913 (xid=0x27acd)
[2017-09-08 09:31:48.399192] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372010 (xid=0x27ace)
[2017-09-08 09:31:48.402229] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372147 (xid=0x27acf)
[2017-09-08 09:31:48.402716] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372247 (xid=0x27ad0)
[2017-09-08 09:31:48.403171] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372347 (xid=0x27ad1)
[2017-09-08 09:31:48.403786] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372438 (xid=0x27ad2)
[2017-09-08 09:31:48.404281] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372535 (xid=0x27ad3)
[2017-09-08 09:31:48.406718] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372632 (xid=0x27ad4)
[2017-09-08 09:31:48.407210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372752 (xid=0x27ad5)
[2017-09-08 09:31:48.407771] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372840 (xid=0x27ad6)
[2017-09-08 09:31:48.408253] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372943 (xid=0x27ad7)
[2017-09-08 09:31:48.408710] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373054 (xid=0x27ad8)
[2017-09-08 09:31:48.409190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373175 (xid=0x27ad9)
[2017-09-08 09:31:48.409671] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373276 (xid=0x27ada)
[2017-09-08 09:31:48.410135] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373375 (xid=0x27adb)
[2017-09-08 09:31:48.410638] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373466 (xid=0x27adc)
[2017-09-08 09:31:48.411136] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373552 (xid=0x27add)
[2017-09-08 09:31:48.412471] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373645 (xid=0x27ade)
[2017-09-08 09:31:48.413783] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373754 (xid=0x27adf)
[2017-09-08 09:31:48.414972] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373852 (xid=0x27ae0)
[2017-09-08 09:31:48.415517] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373936 (xid=0x27ae1)
[2017-09-08 09:31:48.416428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374046 (xid=0x27ae2)
[2017-09-08 09:31:48.417311] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374148 (xid=0x27ae3)
[2017-09-08 09:31:48.418227] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374235 (xid=0x27ae4)
[2017-09-08 09:31:48.419323] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374326 (xid=0x27ae5)
[2017-09-08 09:31:48.419910] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374409 (xid=0x27ae6)
[2017-09-08 09:31:48.420479] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374525 (xid=0x27ae7)
[2017-09-08 09:31:48.422134] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374607 (xid=0x27ae8)
[2017-09-08 09:31:48.424896] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374704 (xid=0x27ae9)
[2017-09-08 09:31:48.427236] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374817 (xid=0x27aea)
[2017-09-08 09:31:48.427802] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374923 (xid=0x27aeb)
[2017-09-08 09:31:48.428202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375012 (xid=0x27aec)
[2017-09-08 09:31:48.428472] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375130 (xid=0x27aed)
[2017-09-08 09:31:48.428728] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375245 (xid=0x27aee)
[2017-09-08 09:31:48.429042] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375333 (xid=0x27aef)
[2017-09-08 09:31:48.429308] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375427 (xid=0x27af0)
[2017-09-08 09:31:48.429557] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375531 (xid=0x27af1)
[2017-09-08 09:31:48.429800] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375651 (xid=0x27af2)
[2017-09-08 09:31:48.430054] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375759 (xid=0x27af3)
[2017-09-08 09:31:48.430351] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375857 (xid=0x27af4)
[2017-09-08 09:31:48.430722] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375956 (xid=0x27af5)
[2017-09-08 09:31:48.431245] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376060 (xid=0x27af6)
[2017-09-08 09:31:48.431796] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376157 (xid=0x27af7)
[2017-09-08 09:31:48.432181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376251 (xid=0x27af8)
[2017-09-08 09:31:48.432487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376346 (xid=0x27af9)
[2017-09-08 09:31:48.432785] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376457 (xid=0x27afa)
[2017-09-08 09:31:48.433089] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376540 (xid=0x27afb)
[2017-09-08 09:31:48.433383] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376630 (xid=0x27afc)
[2017-09-08 09:31:48.433678] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376734 (xid=0x27afd)
[2017-09-08 09:31:48.433967] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376827 (xid=0x27afe)
[2017-09-08 09:31:48.434274] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376950 (xid=0x27aff)
[2017-09-08 09:31:48.434496] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377054 (xid=0x27b00)
[2017-09-08 09:31:48.434778] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377153 (xid=0x27b01)
[2017-09-08 09:31:48.435069] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377249 (xid=0x27b02)
[2017-09-08 09:31:48.435356] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377352 (xid=0x27b03)
[2017-09-08 09:31:48.435637] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377448 (xid=0x27b04)
[2017-09-08 09:31:48.435917] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377554 (xid=0x27b05)
[2017-09-08 09:31:48.436210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377637 (xid=0x27b06)
[2017-09-08 09:31:48.436487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377739 (xid=0x27b07)
[2017-09-08 09:31:48.436765] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377836 (xid=0x27b08)
[2017-09-08 09:31:48.437050] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377938 (xid=0x27b09)
[2017-09-08 09:31:48.437332] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378052 (xid=0x27b0a)
[2017-09-08 09:31:48.437610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378149 (xid=0x27b0b)
[2017-09-08 09:31:48.437888] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378411 (xid=0x27b0c)
[2017-09-08 09:31:48.438176] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378515 (xid=0x27b0d)
[2017-09-08 09:31:48.438451] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378645 (xid=0x27b0e)
[2017-09-08 09:31:48.438727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:31:47.378696 (xid=0x27b0f)
[2017-09-08 09:31:48.438894] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378881 (xid=0x27b10)
[2017-09-08 09:31:48.439196] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379090 (xid=0x27b11)
[2017-09-08 09:31:48.439466] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379200 (xid=0x27b12)
[2017-09-08 09:31:48.439740] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379303 (xid=0x27b13)
[2017-09-08 09:31:48.440016] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379403 (xid=0x27b14)
[2017-09-08 09:31:48.440302] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379537 (xid=0x27b15)
[2017-09-08 09:31:48.440578] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379679 (xid=0x27b16)
[2017-09-08 09:31:48.440853] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379831 (xid=0x27b17)
[2017-09-08 09:31:48.441137] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379921 (xid=0x27b18)
[2017-09-08 09:31:48.441416] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380023 (xid=0x27b19)
[2017-09-08 09:31:48.441688] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380156 (xid=0x27b1a)
[2017-09-08 09:31:48.441960] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380270 (xid=0x27b1b)
[2017-09-08 09:31:48.442244] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380399 (xid=0x27b1c)
[2017-09-08 09:31:48.442520] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380523 (xid=0x27b1d)
[2017-09-08 09:31:48.442792] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380635 (xid=0x27b1e)
[2017-09-08 09:31:48.443079] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380741 (xid=0x27b1f)
[2017-09-08 09:31:48.443355] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380859 (xid=0x27b20)
[2017-09-08 09:31:48.443627] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380969 (xid=0x27b21)
The message "E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]" repeated 109 times between [2017-09-08
09:31:48.382451] and [2017-09-08 09:31:48.626176]
[2017-09-08 09:32:17.489261] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-2: server 10.0.1.200:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:32:17.490362] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055593 (xid=0x3071f)
[2017-09-08 09:32:17.490401] E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-2: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:32:17.490736] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055623 (xid=0x30720)
[2017-09-08 09:32:17.491133] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055654 (xid=0x30721)
[2017-09-08 09:32:17.491480] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055684 (xid=0x30722)
[2017-09-08 09:32:17.491791] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055712 (xid=0x30723)
[2017-09-08 09:32:17.492181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055741 (xid=0x30724)
[2017-09-08 09:32:17.492506] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055770 (xid=0x30725)
[2017-09-08 09:32:17.492811] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055799 (xid=0x30726)
[2017-09-08 09:32:17.493131] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSYNC(16)) called at 2017-09-08 09:32:13.675475 (xid=0x2f186)
[2017-09-08 09:32:17.493461] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.036379 (xid=0x3071d)
[2017-09-08 09:32:17.494173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.055558 (xid=0x3071e)
[2017-09-08 09:32:17.495483] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.077348 (xid=0x30727)
[2017-09-08 09:32:17.495787] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.078270 (xid=0x30728)
[2017-09-08 09:32:17.496077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484276 (xid=0x30729)
[2017-09-08 09:32:17.496378] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484349 (xid=0x3072a)
[2017-09-08 09:32:17.496655] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484435 (xid=0x3072b)
[2017-09-08 09:32:17.496923] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484493 (xid=0x3072c)
[2017-09-08 09:32:17.497202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484553 (xid=0x3072d)
[2017-09-08 09:32:17.497509] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484606 (xid=0x3072e)
[2017-09-08 09:32:17.497780] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484669 (xid=0x3072f)
[2017-09-08 09:32:17.498072] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484722 (xid=0x30730)
[2017-09-08 09:32:17.498397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484776 (xid=0x30731)
[2017-09-08 09:32:17.498666] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484830 (xid=0x30732)
[2017-09-08 09:32:17.499121] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484873 (xid=0x30733)
[2017-09-08 09:32:17.499711] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484956 (xid=0x30734)
[2017-09-08 09:32:17.500893] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485011 (xid=0x30735)
[2017-09-08 09:32:17.501880] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485128 (xid=0x30736)
[2017-09-08 09:32:17.502892] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485243 (xid=0x30737)
[2017-09-08 09:32:17.504237] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485307 (xid=0x30738)
[2017-09-08 09:32:17.505447] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485370 (xid=0x30739)
[2017-09-08 09:32:17.506503] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485462 (xid=0x3073a)
[2017-09-08 09:32:17.507727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485521 (xid=0x3073b)
[2017-09-08 09:32:17.508173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485578 (xid=0x3073c)
[2017-09-08 09:32:17.508925] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485630 (xid=0x3073d)
[2017-09-08 09:32:17.509262] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485688 (xid=0x3073e)
[2017-09-08 09:32:17.509641] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485764 (xid=0x3073f)
[2017-09-08 09:32:17.509957] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:32:16.485786 (xid=0x30740)
[2017-09-08 09:32:17.510224] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486075 (xid=0x30741)
[2017-09-08 09:32:17.511645] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486217 (xid=0x30742)
[2017-09-08 09:32:17.511983] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486340 (xid=0x30743)
[2017-09-08 09:32:17.512326] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486399 (xid=0x30744)
[2017-09-08 09:32:17.512860] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486543 (xid=0x30745)
[2017-09-08 09:32:17.514077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486637 (xid=0x30746)
[2017-09-08 09:32:17.514838] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486734 (xid=0x30747)
[2017-09-08 09:32:17.515413] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486842 (xid=0x30748)
[2017-09-08 09:32:17.516271] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486964 (xid=0x30749)
[2017-09-08 09:32:17.516775] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487082 (xid=0x3074a)
[2017-09-08 09:32:17.517195] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487190 (xid=0x3074b)
[2017-09-08 09:32:17.517635] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487289 (xid=0x3074c)
[2017-09-08 09:32:17.519469] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487400 (xid=0x3074d)
[2017-09-08 09:32:17.520008] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487485 (xid=0x3074e)
[2017-09-08 09:32:17.520411] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487617 (xid=0x3074f)
[2017-09-08 09:32:17.520891] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487699 (xid=0x30750)
[2017-09-08 09:32:17.521190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487836 (xid=0x30751)
[2017-09-08 09:32:17.521489] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.487914 (xid=0x30752)
[2017-09-08 09:32:17.521932] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488019 (xid=0x30753)
[2017-09-08 09:32:17.522218] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488125 (xid=0x30754)
[2017-09-08 09:32:17.522548] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488201 (xid=0x30755)
[2017-09-08 09:32:17.524259] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.488302 (xid=0x30756)
[2017-09-08 09:32:17.524610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[

0x7fbcad8d2170]

Pavel Szalbot

2017-09-08 10:46:13 UTC

Post by Diego Remolina
This is exactly the problem,
Systemctl stop glusterd does *not* kill the brick processes.

Yes, I now.

Post by Diego Remolina
On CentOS with gluster 3.10.x there is also a service, meant to only stop
glusterfsd (brick processes). I think the reboot process may not be properly
stopping glusterfsd or network or firewall may be stopped before glusterfsd
and so the nodes go into the long timeout.

I think reboot should switch init to level 1 and processes are
gracefully terminated - same as killall...

Post by Diego Remolina
Once again , in my case a simple reboot hangs my VMs, whereas stopping
glusterd *and* glusterfsd processes, then reboot works perfectly.

Testing it now...

Gandalf Corvotempesta

2017-09-08 10:48:59 UTC

I think this should be considered a bug
If you have a server crash, glusterfsd process obviously doesn't exit
properly and thus this could least to IO stop ?

And server crashes are the main reason to use a redundant filesystem like
gluster

Il 8 set 2017 12:43 PM, "Diego Remolina" <***@gmail.com> ha scritto:

This is exactly the problem,

Systemctl stop glusterd does *not* kill the brick processes.

On CentOS with gluster 3.10.x there is also a service, meant to only stop
glusterfsd (brick processes). I think the reboot process may not be
properly stopping glusterfsd or network or firewall may be stopped before
glusterfsd and so the nodes go into the long timeout.

Once again , in my case a simple reboot hangs my VMs, whereas stopping
glusterd *and* glusterfsd processes, then reboot works perfectly.

Diego

On Sep 8, 2017 6:36 AM, "Pavel Szalbot" <***@gmail.com> wrote:

Hi Diego,

indeed glusterfsd processes are runnin and it is the reason I do
server reboot instead of systemctl glusterd stop. Is killall different
from reboot in a way glusterfsd processes are terminated in CentOS
(init 1?)?

However I will try this and let you know.

-ps

0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9)
[2017-09-08 09:31:48.384056] E [MSGID: 114031]
[client-rpc-fops.c:1646:client3_3_entrylk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:31:48.384077] E [MSGID: 108007]
[afr-lk-common.c:825:afr_unlock_entrylk_cbk]
/.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on
gv_openstack_1-client
-1 [Transport endpoint is not connected]
[2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--

/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
[2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
[2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
[2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
[2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
[2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
[2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
[2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
[2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
[2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
[2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
[2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
[2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
[2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371815 (xid=0x27acc)
[2017-09-08 09:31:48.397273] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371913 (xid=0x27acd)
[2017-09-08 09:31:48.399192] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372010 (xid=0x27ace)
[2017-09-08 09:31:48.402229] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372147 (xid=0x27acf)
[2017-09-08 09:31:48.402716] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372247 (xid=0x27ad0)
[2017-09-08 09:31:48.403171] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372347 (xid=0x27ad1)
[2017-09-08 09:31:48.403786] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372438 (xid=0x27ad2)
[2017-09-08 09:31:48.404281] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372535 (xid=0x27ad3)
[2017-09-08 09:31:48.406718] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372632 (xid=0x27ad4)
[2017-09-08 09:31:48.407210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372752 (xid=0x27ad5)
[2017-09-08 09:31:48.407771] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372840 (xid=0x27ad6)
[2017-09-08 09:31:48.408253] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372943 (xid=0x27ad7)
[2017-09-08 09:31:48.408710] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373054 (xid=0x27ad8)
[2017-09-08 09:31:48.409190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373175 (xid=0x27ad9)
[2017-09-08 09:31:48.409671] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373276 (xid=0x27ada)
[2017-09-08 09:31:48.410135] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373375 (xid=0x27adb)
[2017-09-08 09:31:48.410638] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373466 (xid=0x27adc)
[2017-09-08 09:31:48.411136] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373552 (xid=0x27add)
[2017-09-08 09:31:48.412471] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373645 (xid=0x27ade)
[2017-09-08 09:31:48.413783] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373754 (xid=0x27adf)
[2017-09-08 09:31:48.414972] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373852 (xid=0x27ae0)
[2017-09-08 09:31:48.415517] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373936 (xid=0x27ae1)
[2017-09-08 09:31:48.416428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374046 (xid=0x27ae2)
[2017-09-08 09:31:48.417311] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374148 (xid=0x27ae3)
[2017-09-08 09:31:48.418227] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374235 (xid=0x27ae4)
[2017-09-08 09:31:48.419323] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374326 (xid=0x27ae5)
[2017-09-08 09:31:48.419910] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374409 (xid=0x27ae6)
[2017-09-08 09:31:48.420479] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374525 (xid=0x27ae7)
[2017-09-08 09:31:48.422134] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374607 (xid=0x27ae8)
[2017-09-08 09:31:48.424896] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374704 (xid=0x27ae9)
[2017-09-08 09:31:48.427236] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374817 (xid=0x27aea)
[2017-09-08 09:31:48.427802] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374923 (xid=0x27aeb)
[2017-09-08 09:31:48.428202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375012 (xid=0x27aec)
[2017-09-08 09:31:48.428472] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375130 (xid=0x27aed)
[2017-09-08 09:31:48.428728] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375245 (xid=0x27aee)
[2017-09-08 09:31:48.429042] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375333 (xid=0x27aef)
[2017-09-08 09:31:48.429308] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375427 (xid=0x27af0)
[2017-09-08 09:31:48.429557] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375531 (xid=0x27af1)
[2017-09-08 09:31:48.429800] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375651 (xid=0x27af2)
[2017-09-08 09:31:48.430054] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375759 (xid=0x27af3)
[2017-09-08 09:31:48.430351] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375857 (xid=0x27af4)
[2017-09-08 09:31:48.430722] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375956 (xid=0x27af5)
[2017-09-08 09:31:48.431245] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376060 (xid=0x27af6)
[2017-09-08 09:31:48.431796] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376157 (xid=0x27af7)
[2017-09-08 09:31:48.432181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376251 (xid=0x27af8)
[2017-09-08 09:31:48.432487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376346 (xid=0x27af9)
[2017-09-08 09:31:48.432785] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376457 (xid=0x27afa)
[2017-09-08 09:31:48.433089] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376540 (xid=0x27afb)
[2017-09-08 09:31:48.433383] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376630 (xid=0x27afc)
[2017-09-08 09:31:48.433678] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376734 (xid=0x27afd)
[2017-09-08 09:31:48.433967] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376827 (xid=0x27afe)
[2017-09-08 09:31:48.434274] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376950 (xid=0x27aff)
[2017-09-08 09:31:48.434496] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377054 (xid=0x27b00)
[2017-09-08 09:31:48.434778] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377153 (xid=0x27b01)
[2017-09-08 09:31:48.435069] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377249 (xid=0x27b02)
[2017-09-08 09:31:48.435356] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377352 (xid=0x27b03)
[2017-09-08 09:31:48.435637] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377448 (xid=0x27b04)
[2017-09-08 09:31:48.435917] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377554 (xid=0x27b05)
[2017-09-08 09:31:48.436210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377637 (xid=0x27b06)
[2017-09-08 09:31:48.436487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377739 (xid=0x27b07)
[2017-09-08 09:31:48.436765] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377836 (xid=0x27b08)
[2017-09-08 09:31:48.437050] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377938 (xid=0x27b09)
[2017-09-08 09:31:48.437332] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378052 (xid=0x27b0a)
[2017-09-08 09:31:48.437610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378149 (xid=0x27b0b)
[2017-09-08 09:31:48.437888] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378411 (xid=0x27b0c)
[2017-09-08 09:31:48.438176] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378515 (xid=0x27b0d)
[2017-09-08 09:31:48.438451] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378645 (xid=0x27b0e)
[2017-09-08 09:31:48.438727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378881 (xid=0x27b10)
[2017-09-08 09:31:48.439196] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379090 (xid=0x27b11)
[2017-09-08 09:31:48.439466] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379200 (xid=0x27b12)
[2017-09-08 09:31:48.439740] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379303 (xid=0x27b13)
[2017-09-08 09:31:48.440016] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379403 (xid=0x27b14)
[2017-09-08 09:31:48.440302] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379537 (xid=0x27b15)
[2017-09-08 09:31:48.440578] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379679 (xid=0x27b16)
[2017-09-08 09:31:48.440853] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379831 (xid=0x27b17)
[2017-09-08 09:31:48.441137] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379921 (xid=0x27b18)
[2017-09-08 09:31:48.441416] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380023 (xid=0x27b19)
[2017-09-08 09:31:48.441688] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380156 (xid=0x27b1a)
[2017-09-08 09:31:48.441960] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380270 (xid=0x27b1b)
[2017-09-08 09:31:48.442244] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380399 (xid=0x27b1c)
[2017-09-08 09:31:48.442520] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380523 (xid=0x27b1d)
[2017-09-08 09:31:48.442792] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380635 (xid=0x27b1e)
[2017-09-08 09:31:48.443079] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380741 (xid=0x27b1f)
[2017-09-08 09:31:48.443355] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380859 (xid=0x27b20)
[2017-09-08 09:31:48.443627] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380969 (xid=0x27b21)
The message "E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]" repeated 109 times between [2017-09-08
09:31:48.382451] and [2017-09-08 09:31:48.626176]
[2017-09-08 09:32:17.489261] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-2: server 10.0.1.200:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:32:17.490362] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055623 (xid=0x30720)
[2017-09-08 09:32:17.491133] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055654 (xid=0x30721)
[2017-09-08 09:32:17.491480] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055684 (xid=0x30722)
[2017-09-08 09:32:17.491791] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055712 (xid=0x30723)
[2017-09-08 09:32:17.492181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055741 (xid=0x30724)
[2017-09-08 09:32:17.492506] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055770 (xid=0x30725)
[2017-09-08 09:32:17.492811] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055799 (xid=0x30726)
[2017-09-08 09:32:17.493131] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSYNC(16)) called at 2017-09-08 09:32:13.675475 (xid=0x2f186)
[2017-09-08 09:32:17.493461] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.036379 (xid=0x3071d)
[2017-09-08 09:32:17.494173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.055558 (xid=0x3071e)
[2017-09-08 09:32:17.495483] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.077348 (xid=0x30727)
[2017-09-08 09:32:17.495787] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.078270 (xid=0x30728)
[2017-09-08 09:32:17.496077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484276 (xid=0x30729)
[2017-09-08 09:32:17.496378] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484349 (xid=0x3072a)
[2017-09-08 09:32:17.496655] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484435 (xid=0x3072b)
[2017-09-08 09:32:17.496923] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484493 (xid=0x3072c)
[2017-09-08 09:32:17.497202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484553 (xid=0x3072d)
[2017-09-08 09:32:17.497509] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484606 (xid=0x3072e)
[2017-09-08 09:32:17.497780] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484669 (xid=0x3072f)
[2017-09-08 09:32:17.498072] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484722 (xid=0x30730)
[2017-09-08 09:32:17.498397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484776 (xid=0x30731)
[2017-09-08 09:32:17.498666] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484830 (xid=0x30732)
[2017-09-08 09:32:17.499121] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484873 (xid=0x30733)
[2017-09-08 09:32:17.499711] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484956 (xid=0x30734)
[2017-09-08 09:32:17.500893] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485011 (xid=0x30735)
[2017-09-08 09:32:17.501880] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Post by Diego Remolina
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485128 (xid=0x30736)
[2017-09-08 09:32:17.502892] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb

cad8d2170]

Pavel Szalbot

2017-09-08 10:50:11 UTC

On Fri, Sep 8, 2017 at 12:48 PM, Gandalf Corvotempesta

Post by Gandalf Corvotempesta
I think this should be considered a bug
If you have a server crash, glusterfsd process obviously doesn't exit
properly and thus this could least to IO stop ?

I agree with you completely in this.

Pavel Szalbot

2017-09-08 10:52:37 UTC

Btw now I am experiencing "Transport endpoint disconnects" because of
1s ping-timeout even though nodes are up. This sucks. The network is
not overloaded at all, switches are used only by gluster network and
network consists only of three gluster nodes and one VM hypervisor and
Cinder controller (no traffic here).
-ps

Post by Pavel Szalbot
On Fri, Sep 8, 2017 at 12:48 PM, Gandalf Corvotempesta

Post by Gandalf Corvotempesta
I think this should be considered a bug
If you have a server crash, glusterfsd process obviously doesn't exit
properly and thus this could least to IO stop ?

I agree with you completely in this.

Diego Remolina

2017-09-08 10:57:26 UTC

I would prefer the behavior was different to what it is of I/O stopping.
The argument I heard for the long 42 second time out was that MTBF on a
server was high, and that the client reconnection operation was *costly*.
Those were arguments to *not* change the ping timeout value down from 42
seconds. I think it was mentioned that low ping timeout settings could lead
to high cpu loads with many clients trying to reconnect if a short timeout
was set. This is all hearsay, so the experts should explain it better... ð

Diego

Post by Pavel Szalbot
On Fri, Sep 8, 2017 at 12:48 PM, Gandalf Corvotempesta

Post by Gandalf Corvotempesta
I think this should be considered a bug
If you have a server crash, glusterfsd process obviously doesn't exit
properly and thus this could least to IO stop ?

I agree with you completely in this.

Pavel Szalbot

2017-09-08 11:07:57 UTC

OK, so killall seems to be ok after several attempts i.e. iops do not stop
on VM. Reboot caused I/O errors after maybe 20 seconds since issuing the
command. I will check the servers console during reboot to see if the VM
errors appear just after the power cycle and will try to crash the VM after
killall again...

-ps

Post by Diego Remolina
I would prefer the behavior was different to what it is of I/O stopping.
The argument I heard for the long 42 second time out was that MTBF on a
server was high, and that the client reconnection operation was *costly*.
Those were arguments to *not* change the ping timeout value down from 42
seconds. I think it was mentioned that low ping timeout settings could lead
to high cpu loads with many clients trying to reconnect if a short timeout
was set. This is all hearsay, so the experts should explain it better... ð
Diego

Post by Pavel Szalbot
On Fri, Sep 8, 2017 at 12:48 PM, Gandalf Corvotempesta

Post by Gandalf Corvotempesta
I think this should be considered a bug
If you have a server crash, glusterfsd process obviously doesn't exit
properly and thus this could least to IO stop ?

I agree with you completely in this.

Gandalf Corvotempesta

2017-09-08 11:11:57 UTC

Post by Pavel Szalbot
OK, so killall seems to be ok after several attempts i.e. iops do not stop
on VM. Reboot caused I/O errors after maybe 20 seconds since issuing the
command. I will check the servers console during reboot to see if the VM
errors appear just after the power cycle and will try to crash the VM after
killall again...

Also try to kill the Gluster VM without killing glusterfsd, simulating a
server hard-crash . Or try to remove the network interface.

Pavel Szalbot

2017-09-08 11:21:00 UTC

So even killall situation eventually kills VM (I/O errors).

Gandalf, isn't possible server hard-crash too much? I mean if reboot
reliably kills the VM, there is no doubt network crash or poweroff
will as well.

I am tempted to test this setup on DigitalOcean to eliminate
possibility of my hardware/network. But if Diego is able to reproduce
the "reboot crash", my doubts of hardware/network problems are close
to none.
-ps

On Fri, Sep 8, 2017 at 1:11 PM, Gandalf Corvotempesta

Also try to kill the Gluster VM without killing glusterfsd, simulating a
server hard-crash . Or try to remove the network interface.

Gandalf Corvotempesta

2017-09-08 11:36:07 UTC

Post by Pavel Szalbot
Gandalf, isn't possible server hard-crash too much? I mean if reboot
reliably kills the VM, there is no doubt network crash or poweroff
will as well.

IIUP, the only way to keep I/O running is to gracefully exiting glusterfsd.
killall should send signal 15 (SIGTERM) to the process, maybe a bug in
signal management
on gluster side? Because kernel is already telling glusterfsd to exit,
though signal 15 but glusterfsd
seems to handle this in a bad way.

a server hard-crash doesn't send any signal. I think this could be
also similiar to SIGKILL (9)
that can't be catched/ignored software side.

In other words: is this a bug in gluster's signal management (if
SIGKILL is working and SIGTERM no, i'll almost sure this is a bug in
signal management),
a engineering bug (relying only on a graceful exit [but even SIGTERM
should be threthed as graceful exit] to preserve I/O on clients) or
something else ?

Pavel Szalbot

2017-09-08 11:44:31 UTC

On Sep 8, 2017 13:36, "Gandalf Corvotempesta" <

Post by Pavel Szalbot
Gandalf, isn't possible server hard-crash too much? I mean if reboot
reliably kills the VM, there is no doubt network crash or poweroff
will as well.

IIUP, the only way to keep I/O running is to gracefully exiting glusterfsd.

No, even killall resp. SIGTERMing glusterfsd ends up with I/O errors on the
client.

I did not test SIGKILL because I suppose if graceful exit is bad, SIGKILL
will be as well. This assumption might be wrong. So I will test it. It
would be interesting to see client to work in case of crash (SIGKILL) and
not in case of graceful exit of glusterfsd.

killall should send signal 15 (SIGTERM) to the process, maybe a bug in
signal management
on gluster side? Because kernel is already telling glusterfsd to exit,
though signal 15 but glusterfsd
seems to handle this in a bad way.

a server hard-crash doesn't send any signal. I think this could be
also similiar to SIGKILL (9)
that can't be catched/ignored software side.

In other words: is this a bug in gluster's signal management (if
SIGKILL is working and SIGTERM no, i'll almost sure this is a bug in
signal management),
a engineering bug (relying only on a graceful exit [but even SIGTERM
should be threthed as graceful exit] to preserve I/O on clients) or
something else ?

Gandalf Corvotempesta

2017-09-08 11:53:38 UTC

Post by Pavel Szalbot
I did not test SIGKILL because I suppose if graceful exit is bad, SIGKILL
will be as well. This assumption might be wrong. So I will test it. It would
be interesting to see client to work in case of crash (SIGKILL) and not in
case of graceful exit of glusterfsd.

Exactly. if this happen, probably there is a bug in gluster's signal management.

Pavel Szalbot

2017-09-08 11:56:18 UTC

I added firewall rule to block all traffic from Gluster VLAN on one of
the nodes.

Approximately 3 minutes in and no crash so far. Errors about missing
node in qemu instance log are present, but this is normal.
-ps

On Fri, Sep 8, 2017 at 1:53 PM, Gandalf Corvotempesta

Exactly. if this happen, probably there is a bug in gluster's signal management.

Diego Remolina

2017-09-08 11:56:38 UTC

I currently only have a Windows 2012 R2 server VM in testing on top of
the gluster storage, so I will have to take some time to provision a
couple Linux VMs with both ext4 and XFS to see what happens on those.

The Windows server VM is OK with killall glusterfsd, but when the 42
second timeout goes into effect, it gets paused and I have to go into
RHEVM to un-pause it.

Diego

On Fri, Sep 8, 2017 at 7:53 AM, Gandalf Corvotempesta

Exactly. if this happen, probably there is a bug in gluster's signal management.

Pavel Szalbot

2017-09-08 12:11:06 UTC

Gandalf, SIGKILL (killall -9 glusterfsd) did not stop I/O after few
minutes. SIGTERM on the other hand causes crash, but this time it is
not read-only remount, but around 10 IOPS tops and 2 IOPS on average.
-ps

Post by Diego Remolina
I currently only have a Windows 2012 R2 server VM in testing on top of
the gluster storage, so I will have to take some time to provision a
couple Linux VMs with both ext4 and XFS to see what happens on those.
The Windows server VM is OK with killall glusterfsd, but when the 42
second timeout goes into effect, it gets paused and I have to go into
RHEVM to un-pause it.
Diego
On Fri, Sep 8, 2017 at 7:53 AM, Gandalf Corvotempesta

Exactly. if this happen, probably there is a bug in gluster's signal management.

Gandalf Corvotempesta

2017-09-08 12:13:01 UTC

Post by Pavel Szalbot
Gandalf, SIGKILL (killall -9 glusterfsd) did not stop I/O after few
minutes. SIGTERM on the other hand causes crash, but this time it is
not read-only remount, but around 10 IOPS tops and 2 IOPS on average.
-ps

So, seems to be reliable to server crashes but not to server shutdown :)

Pavel Szalbot

2017-09-08 12:14:49 UTC

Well I really do not like the non-deterministic characteristic of it.
However the server crash did never occur in my production environment
- only upgrades and reboots ;-)
-ps

On Fri, Sep 8, 2017 at 2:13 PM, Gandalf Corvotempesta

So, seems to be reliable to server crashes but not to server shutdown :)

2017-09-09 00:35:06 UTC

Pavel.

Is there a difference between native client (fuse) and libgfapi in
regards to the crashing/read-only behaviour?

We use Rep2 + Arb and can shutdown a node cleanly, without issue on our
VMs. We do it all the time for upgrades and maintenance.

However we are still on native client as we haven't had time to work on
libgfapi yet. Maybe that is more tolerant.

We have linux VMs mostly with XFS filesystems.

During the downtime, the VMs continue to run with normal speed.

In this case we migrated to the VM so date node 2 (c2g.gluster) and
shutdown c1g.gluster to do some upgrades.

# gluster peer status
Number of Peers: 2

Hostname: c1g.gluster
Uuid: 91be2005-30e6-462b-a66e-773913cacab6
State: Peer in Cluster (Disconnected)
Hostname: arb-c2.gluster
Uuid: 20862755-e54e-4b79-96a8-59e78c6a6a2e
State: Peer in Cluster (Connected)

# gluster volume status
Status of volume: brick1
Gluster process                             TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick c2g.gluster:/GLUSTER/brick1       49152     0 Y       5194
Brick arb-c2.gluster:/GLUSTER/brick1        49152     0 Y       3647
Self-heal Daemon on localhost               N/A       N/A Y       5214
Self-heal Daemon on arb-c2.gluster          N/A       N/A Y       3667

Task Status of Volume brick1
------------------------------------------------------------------------------
There are no active volume tasks

When we return the c1g node, we do see a "pause" in the VMs as the
shards heal. By pause meaning a terminal session gets spongy, but that
passes pretty quickly.

Also are your VMs mounted in libvirt with caching? We always use
cache='none' so we can migrate around easily.

Finally, you seem to be using oVirt/RHEV. Is it possible that your platform is triggering a protective response on the VMs (by suspending).

-wk

So, seems to be reliable to server crashes but not to server shutdown :)
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-09-09 06:05:47 UTC

Hi,

Post by WK
Pavel.
Is there a difference between native client (fuse) and libgfapi in regards
to the crashing/read-only behaviour?

I switched to FUSE now and the VM crashed (read-only remount)
immediately after one node started rebooting.

I tried to mount.glusterfs same volume on different server (not VM),
running Ubuntu Xenial and gluster client 3.10.5.

mount -t glusterfs -o backupvolfile-server=10.0.1.202
10.0.1.201:/gv_openstack_1 /mnt/gv_openstack_1/

I ran fio job I described earlier. As soon as I killall glusterfsd,
fio reported:

fio: io_u error on file /mnt/gv_openstack_1/fio.data: Transport
endpoint is not connected: read offset=7022575616, buflen=262144
fio: pid=7205, err=107/file:io_u.c:1582, func=io_u error,
error=Transport endpoint is not connected

And crashed. I still cannot believe I am the only one experiencing
these problems and that tells me, that there must be some problem in
my setup. However I have not experienced any crashes if all nodes were
up. Ever. I suspected disks and network as culprit, but we run SMART
tests frequently (short and long), bricks are on RAID10 (6xSSDs),
switches are Juniper EX4550s (shallow packet buffer, but no drops in
the statistics) pretty much dedicated to Gluster and we ran many VMs
and stored and heavily used other data there. And gluster logs or
system logs do not provide any hint of HW/network failure.

Post by WK
We use Rep2 + Arb and can shutdown a node cleanly, without issue on our VMs.
We do it all the time for upgrades and maintenance.
However we are still on native client as we haven't had time to work on
libgfapi yet. Maybe that is more tolerant.
We have linux VMs mostly with XFS filesystems.

We use whatever official cloud (Openstack) images provide, all tests I
describe are on Ubuntu Xenial VMs, ext4.

Post by WK
During the downtime, the VMs continue to run with normal speed.
In this case we migrated to the VM so date node 2 (c2g.gluster) and shutdown
c1g.gluster to do some upgrades.
# gluster peer status
Number of Peers: 2
Hostname: c1g.gluster
Uuid: 91be2005-30e6-462b-a66e-773913cacab6
State: Peer in Cluster (Disconnected)
Hostname: arb-c2.gluster
Uuid: 20862755-e54e-4b79-96a8-59e78c6a6a2e
State: Peer in Cluster (Connected)
# gluster volume status
Status of volume: brick1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick c2g.gluster:/GLUSTER/brick1 49152 0 Y 5194
Brick arb-c2.gluster:/GLUSTER/brick1 49152 0 Y 3647
Self-heal Daemon on localhost N/A N/A Y 5214
Self-heal Daemon on arb-c2.gluster N/A N/A Y 3667
Task Status of Volume brick1
------------------------------------------------------------------------------
There are no active volume tasks
When we return the c1g node, we do see a "pause" in the VMs as the shards
heal. By pause meaning a terminal session gets spongy, but that passes
pretty quickly.

Hmm, do you see any errors in VM's dmesg? Or any other reasons for "sponginess"?

Post by WK
Also are your VMs mounted in libvirt with caching? We always use
cache='none' so we can migrate around easily.

No cache, virtio:

<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source protocol='gluster'
name='gv_openstack_1/volume-3a7eaf5a-8348-4f01-b59f-f28cd8cea771'>
<host name='10.0.1.201' port='24007'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>3a7eaf5a-8348-4f01-b59f-f28cd8cea771</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>

Post by WK
Finally, you seem to be using oVirt/RHEV. Is it possible that your platform
is triggering a protective response on the VMs (by suspending).

No, this is Openstack environment, I am not aware of any protective mechanisms.

-ps

Pavel Szalbot

2017-09-09 07:09:40 UTC

Sorry, I did not start the glusterfsd on the node I was shutting
yesterday and now killed another one during FUSE test, so it had to
crash immediately (only one of three nodes were actually up). This
definitely happened for the first time (only one node had been killed
yesterday).

Using FUSE seems to be OK with replica 3. So this can be gfapi related
or maybe rather libvirt related.

I tried ioengine=gfapi with fio and job survived reboot.

-ps

Post by Pavel Szalbot
Hi,

Post by WK
Pavel.
Is there a difference between native client (fuse) and libgfapi in regards
to the crashing/read-only behaviour?

I switched to FUSE now and the VM crashed (read-only remount)
immediately after one node started rebooting.
I tried to mount.glusterfs same volume on different server (not VM),
running Ubuntu Xenial and gluster client 3.10.5.
mount -t glusterfs -o backupvolfile-server=10.0.1.202
10.0.1.201:/gv_openstack_1 /mnt/gv_openstack_1/
I ran fio job I described earlier. As soon as I killall glusterfsd,
fio: io_u error on file /mnt/gv_openstack_1/fio.data: Transport
endpoint is not connected: read offset=7022575616, buflen=262144
fio: pid=7205, err=107/file:io_u.c:1582, func=io_u error,
error=Transport endpoint is not connected
And crashed. I still cannot believe I am the only one experiencing
these problems and that tells me, that there must be some problem in
my setup. However I have not experienced any crashes if all nodes were
up. Ever. I suspected disks and network as culprit, but we run SMART
tests frequently (short and long), bricks are on RAID10 (6xSSDs),
switches are Juniper EX4550s (shallow packet buffer, but no drops in
the statistics) pretty much dedicated to Gluster and we ran many VMs
and stored and heavily used other data there. And gluster logs or
system logs do not provide any hint of HW/network failure.

We use whatever official cloud (Openstack) images provide, all tests I
describe are on Ubuntu Xenial VMs, ext4.

Hmm, do you see any errors in VM's dmesg? Or any other reasons for "sponginess"?

Post by WK
Also are your VMs mounted in libvirt with caching? We always use
cache='none' so we can migrate around easily.

Post by WK
Finally, you seem to be using oVirt/RHEV. Is it possible that your platform
is triggering a protective response on the VMs (by suspending).

No, this is Openstack environment, I am not aware of any protective mechanisms.
-ps

Gionatan Danti

2017-09-09 11:32:53 UTC

Post by Pavel Szalbot
Sorry, I did not start the glusterfsd on the node I was shutting
yesterday and now killed another one during FUSE test, so it had to
crash immediately (only one of three nodes were actually up). This
definitely happened for the first time (only one node had been killed
yesterday).
Using FUSE seems to be OK with replica 3. So this can be gfapi related
or maybe rather libvirt related.
I tried ioengine=gfapi with fio and job survived reboot.
-ps

So, to recap:
- with gfapi, your VMs crashes/mount read-only with a single node
failure;
- with gpapi also, fio seems to have no problems;
- with native FUSE client, both VMs and fio have no problems at all.

Is it correct?
Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

Pavel Szalbot

2017-09-09 13:49:26 UTC

Yes, this is my observation so far.

- with gfapi, your VMs crashes/mount read-only with a single node failure;
- with gpapi also, fio seems to have no problems;
- with native FUSE client, both VMs and fio have no problems at all.
Is it correct?
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
GPG public key ID: FF5F32A8

2017-09-09 17:19:24 UTC

Well, that makes me feel better.

I've seen all these stories here and on Ovirt recently about VMs going
read-only, even on fairly simply layouts.

Each time, I've responded that we just don't see those issues.

I guess the fact that we were lazy about switching to gfapi turns out to
be a potential explanation <grin>

-wk

Post by Pavel Szalbot
Yes, this is my observation so far.
- with gfapi, your VMs crashes/mount read-only with a single node failure;
- with gpapi also, fio seems to have no problems;
- with native FUSE client, both VMs and fio have no problems at all.

l***@ulrar.net

2017-09-09 22:08:39 UTC

Mh, not so sure really, using libgfapi and it's been working perfectly
fine. And trust me, there had been A LOT of various crashes, reboots and
kill of nodes.

Maybe it's a version thing ? A new bug in the new gluster releases that
doesn't affect our 3.7.15.

Post by WK
Well, that makes me feel better.
I've seen all these stories here and on Ovirt recently about VMs going
read-only, even on fairly simply layouts.
Each time, I've responded that we just don't see those issues.
I guess the fact that we were lazy about switching to gfapi turns out to
be a potential explanation <grin>
-wk

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

2017-09-10 04:37:37 UTC

I'm on 3.10.5. Its rock solid (at least with the fuse mount <Grin>)

We are also typically on a somewhat slower GlusterFS LAN network (bonded
2x1G, jumbo frames) so that may be a factor.

I'll try to setup a trusted pool to test libgfapi soon.

I'm curious as to how much faster it is, but the fuse mount is fast
enough, dirt simple to use, and just works on all VM ops such as
migration, snaps etc, so there hasn't been a compelling need to squeeze
out a few more I/Os.

Post by l***@ulrar.net
Mh, not so sure really, using libgfapi and it's been working perfectly
fine. And trust me, there had been A LOT of various crashes, reboots and
kill of nodes.
Maybe it's a version thing ? A new bug in the new gluster releases that
doesn't affect our 3.7.15.

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Pavel Szalbot

2017-09-10 09:02:36 UTC

Hey guys,

I got another "reboot crash" with gfapi and this time libvirt-3.2.1
(from cbs.centos.org). Is there anyone who can audit the libgfapi
usage in libvirt? :-)

WK: I use bonded 2x10Gbps and I do get crashes only in heavy I/O
situations (fio). Upgrading system (apt-get dist-upgrade) was ok, so
this might be even related to amount of IOPS.

-ps

Post by WK
I'm on 3.10.5. Its rock solid (at least with the fuse mount <Grin>)
We are also typically on a somewhat slower GlusterFS LAN network (bonded
2x1G, jumbo frames) so that may be a factor.
I'll try to setup a trusted pool to test libgfapi soon.
I'm curious as to how much faster it is, but the fuse mount is fast enough,
dirt simple to use, and just works on all VM ops such as migration, snaps
etc, so there hasn't been a compelling need to squeeze out a few more I/Os.
Mh, not so sure really, using libgfapi and it's been working perfectly
fine. And trust me, there had been A LOT of various crashes, reboots and
kill of nodes.
Maybe it's a version thing ? A new bug in the new gluster releases that
doesn't affect our 3.7.15.
Well, that makes me feel better.
I've seen all these stories here and on Ovirt recently about VMs going
read-only, even on fairly simply layouts.
Each time, I've responded that we just don't see those issues.
I guess the fact that we were lazy about switching to gfapi turns out to
be a potential explanation <grin>
-wk
Yes, this is my observation so far.
- with gfapi, your VMs crashes/mount read-only with a single node failure;
- with gpapi also, fio seems to have no problems;
- with native FUSE client, both VMs and fio have no problems at all.
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

2017-09-10 20:06:55 UTC

Post by Pavel Szalbot
WK: I use bonded 2x10Gbps and I do get crashes only in heavy I/O
situations (fio). Upgrading system (apt-get dist-upgrade) was ok, so
this might be even related to amount of IOPS.
-ps

Well, 20Gbps of writes could overwhelm a lot of DFS clusters. We saw VMs
being overwhelmed by Gigabit speed large file copy ops into the VM with
early versions of Ceph and MooseFS (MFS 1.6.x. was really vulnerable to
that, MFS 3.x is much improved in that regard).

Your success with the fuse client may be because it has a more
complicated path than libgfapi, thus slowing down/buffering data enough
to let the cluster catch up.

Also you are on simple replica. With a more advanced Gluster setup
involving distribute that write load would be spread out more.

2017-09-09 17:26:50 UTC

Post by WK
When we return the c1g node, we do see a "pause" in the VMs as the shards
heal. By pause meaning a terminal session gets spongy, but that passes
pretty quickly.

Hmm, do you see any errors in VM's dmesg? Or any other reasons for "sponginess"?

No, it is a "load" issue. It has to be the shards being collected and
written to the now out of date copy that just came back on line.

Same feeling as if a VM was hit with a dictionary attack or you ran a
huge copy operation on it.

Pavel Szalbot

2017-09-08 12:13:31 UTC

Btw after few more seconds in SIGTERM scenario, VM kind of revived and
seems to be fine... And after few more restarts of fio job, I got I/O
error.
-ps

Exactly. if this happen, probably there is a bug in gluster's signal management.

Diego Remolina

2017-09-08 10:38:46 UTC

If your VMs use ext4 also check this:

https://joejulian.name/blog/keeping-your-vms-from-going-
read-only-when-encountering-a-ping-timeout-in-glusterfs/

I asked him what to do for VMs using XFS and he said he could not find a
fix (setting to change) for those.

HTH,

Diego

/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->

/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)
[2017-09-08 09:31:48.389838] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
[2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
[2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
[2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
[2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
[2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
[2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
[2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
[2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
[2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
[2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
[2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
[2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
[2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371815 (xid=0x27acc)
[2017-09-08 09:31:48.397273] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371913 (xid=0x27acd)
[2017-09-08 09:31:48.399192] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372010 (xid=0x27ace)
[2017-09-08 09:31:48.402229] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372147 (xid=0x27acf)
[2017-09-08 09:31:48.402716] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372247 (xid=0x27ad0)
[2017-09-08 09:31:48.403171] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372347 (xid=0x27ad1)
[2017-09-08 09:31:48.403786] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372438 (xid=0x27ad2)
[2017-09-08 09:31:48.404281] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372535 (xid=0x27ad3)
[2017-09-08 09:31:48.406718] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372632 (xid=0x27ad4)
[2017-09-08 09:31:48.407210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372752 (xid=0x27ad5)
[2017-09-08 09:31:48.407771] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372840 (xid=0x27ad6)
[2017-09-08 09:31:48.408253] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.372943 (xid=0x27ad7)
[2017-09-08 09:31:48.408710] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373054 (xid=0x27ad8)
[2017-09-08 09:31:48.409190] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373175 (xid=0x27ad9)
[2017-09-08 09:31:48.409671] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373276 (xid=0x27ada)
[2017-09-08 09:31:48.410135] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373375 (xid=0x27adb)
[2017-09-08 09:31:48.410638] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373466 (xid=0x27adc)
[2017-09-08 09:31:48.411136] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373552 (xid=0x27add)
[2017-09-08 09:31:48.412471] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373645 (xid=0x27ade)
[2017-09-08 09:31:48.413783] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373754 (xid=0x27adf)
[2017-09-08 09:31:48.414972] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373852 (xid=0x27ae0)
[2017-09-08 09:31:48.415517] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.373936 (xid=0x27ae1)
[2017-09-08 09:31:48.416428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374046 (xid=0x27ae2)
[2017-09-08 09:31:48.417311] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374148 (xid=0x27ae3)
[2017-09-08 09:31:48.418227] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374235 (xid=0x27ae4)
[2017-09-08 09:31:48.419323] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374326 (xid=0x27ae5)
[2017-09-08 09:31:48.419910] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374409 (xid=0x27ae6)
[2017-09-08 09:31:48.420479] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374525 (xid=0x27ae7)
[2017-09-08 09:31:48.422134] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374607 (xid=0x27ae8)
[2017-09-08 09:31:48.424896] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374704 (xid=0x27ae9)
[2017-09-08 09:31:48.427236] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374817 (xid=0x27aea)
[2017-09-08 09:31:48.427802] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.374923 (xid=0x27aeb)
[2017-09-08 09:31:48.428202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375012 (xid=0x27aec)
[2017-09-08 09:31:48.428472] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375130 (xid=0x27aed)
[2017-09-08 09:31:48.428728] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375245 (xid=0x27aee)
[2017-09-08 09:31:48.429042] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375333 (xid=0x27aef)
[2017-09-08 09:31:48.429308] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375427 (xid=0x27af0)
[2017-09-08 09:31:48.429557] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375531 (xid=0x27af1)
[2017-09-08 09:31:48.429800] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375651 (xid=0x27af2)
[2017-09-08 09:31:48.430054] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375759 (xid=0x27af3)
[2017-09-08 09:31:48.430351] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375857 (xid=0x27af4)
[2017-09-08 09:31:48.430722] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.375956 (xid=0x27af5)
[2017-09-08 09:31:48.431245] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376060 (xid=0x27af6)
[2017-09-08 09:31:48.431796] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376157 (xid=0x27af7)
[2017-09-08 09:31:48.432181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376251 (xid=0x27af8)
[2017-09-08 09:31:48.432487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376346 (xid=0x27af9)
[2017-09-08 09:31:48.432785] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376457 (xid=0x27afa)
[2017-09-08 09:31:48.433089] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376540 (xid=0x27afb)
[2017-09-08 09:31:48.433383] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376630 (xid=0x27afc)
[2017-09-08 09:31:48.433678] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376734 (xid=0x27afd)
[2017-09-08 09:31:48.433967] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376827 (xid=0x27afe)
[2017-09-08 09:31:48.434274] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.376950 (xid=0x27aff)
[2017-09-08 09:31:48.434496] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377054 (xid=0x27b00)
[2017-09-08 09:31:48.434778] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377153 (xid=0x27b01)
[2017-09-08 09:31:48.435069] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377249 (xid=0x27b02)
[2017-09-08 09:31:48.435356] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377352 (xid=0x27b03)
[2017-09-08 09:31:48.435637] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377448 (xid=0x27b04)
[2017-09-08 09:31:48.435917] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377554 (xid=0x27b05)
[2017-09-08 09:31:48.436210] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377637 (xid=0x27b06)
[2017-09-08 09:31:48.436487] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377739 (xid=0x27b07)
[2017-09-08 09:31:48.436765] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377836 (xid=0x27b08)
[2017-09-08 09:31:48.437050] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.377938 (xid=0x27b09)
[2017-09-08 09:31:48.437332] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378052 (xid=0x27b0a)
[2017-09-08 09:31:48.437610] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.378149 (xid=0x27b0b)
[2017-09-08 09:31:48.437888] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378411 (xid=0x27b0c)
[2017-09-08 09:31:48.438176] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378515 (xid=0x27b0d)
[2017-09-08 09:31:48.438451] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378645 (xid=0x27b0e)
[2017-09-08 09:31:48.438727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:31:47.378696 (xid=0x27b0f)
[2017-09-08 09:31:48.438894] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.378881 (xid=0x27b10)
[2017-09-08 09:31:48.439196] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379090 (xid=0x27b11)
[2017-09-08 09:31:48.439466] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379200 (xid=0x27b12)
[2017-09-08 09:31:48.439740] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379303 (xid=0x27b13)
[2017-09-08 09:31:48.440016] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379403 (xid=0x27b14)
[2017-09-08 09:31:48.440302] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379537 (xid=0x27b15)
[2017-09-08 09:31:48.440578] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379679 (xid=0x27b16)
[2017-09-08 09:31:48.440853] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379831 (xid=0x27b17)
[2017-09-08 09:31:48.441137] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.379921 (xid=0x27b18)
[2017-09-08 09:31:48.441416] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380023 (xid=0x27b19)
[2017-09-08 09:31:48.441688] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380156 (xid=0x27b1a)
[2017-09-08 09:31:48.441960] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380270 (xid=0x27b1b)
[2017-09-08 09:31:48.442244] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380399 (xid=0x27b1c)
[2017-09-08 09:31:48.442520] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380523 (xid=0x27b1d)
[2017-09-08 09:31:48.442792] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380635 (xid=0x27b1e)
[2017-09-08 09:31:48.443079] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380741 (xid=0x27b1f)
[2017-09-08 09:31:48.443355] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380859 (xid=0x27b20)
[2017-09-08 09:31:48.443627] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:47.380969 (xid=0x27b21)
The message "E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]" repeated 109 times between [2017-09-08
09:31:48.382451] and [2017-09-08 09:31:48.626176]
[2017-09-08 09:32:17.489261] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-2: server 10.0.1.200:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:32:17.490362] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055593 (xid=0x3071f)
[2017-09-08 09:32:17.490401] E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-2: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:32:17.490736] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055623 (xid=0x30720)
[2017-09-08 09:32:17.491133] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055654 (xid=0x30721)
[2017-09-08 09:32:17.491480] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055684 (xid=0x30722)
[2017-09-08 09:32:17.491791] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055712 (xid=0x30723)
[2017-09-08 09:32:17.492181] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055741 (xid=0x30724)
[2017-09-08 09:32:17.492506] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055770 (xid=0x30725)
[2017-09-08 09:32:17.492811] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:32:16.055799 (xid=0x30726)
[2017-09-08 09:32:17.493131] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSYNC(16)) called at 2017-09-08 09:32:13.675475 (xid=0x2f186)
[2017-09-08 09:32:17.493461] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.036379 (xid=0x3071d)
[2017-09-08 09:32:17.494173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.055558 (xid=0x3071e)
[2017-09-08 09:32:17.495483] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.077348 (xid=0x30727)
[2017-09-08 09:32:17.495787] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FSTAT(25)) called at 2017-09-08 09:32:16.078270 (xid=0x30728)
[2017-09-08 09:32:17.496077] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484276 (xid=0x30729)
[2017-09-08 09:32:17.496378] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484349 (xid=0x3072a)
[2017-09-08 09:32:17.496655] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484435 (xid=0x3072b)
[2017-09-08 09:32:17.496923] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484493 (xid=0x3072c)
[2017-09-08 09:32:17.497202] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484553 (xid=0x3072d)
[2017-09-08 09:32:17.497509] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484606 (xid=0x3072e)
[2017-09-08 09:32:17.497780] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484669 (xid=0x3072f)
[2017-09-08 09:32:17.498072] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484722 (xid=0x30730)
[2017-09-08 09:32:17.498397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484776 (xid=0x30731)
[2017-09-08 09:32:17.498666] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484830 (xid=0x30732)
[2017-09-08 09:32:17.499121] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484873 (xid=0x30733)
[2017-09-08 09:32:17.499711] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.484956 (xid=0x30734)
[2017-09-08 09:32:17.500893] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485011 (xid=0x30735)
[2017-09-08 09:32:17.501880] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485128 (xid=0x30736)
[2017-09-08 09:32:17.502892] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485243 (xid=0x30737)
[2017-09-08 09:32:17.504237] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485307 (xid=0x30738)
[2017-09-08 09:32:17.505447] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485370 (xid=0x30739)
[2017-09-08 09:32:17.506503] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485462 (xid=0x3073a)
[2017-09-08 09:32:17.507727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485521 (xid=0x3073b)
[2017-09-08 09:32:17.508173] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485578 (xid=0x3073c)
[2017-09-08 09:32:17.508925] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485630 (xid=0x3073d)
[2017-09-08 09:32:17.509262] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485688 (xid=0x3073e)
[2017-09-08 09:32:17.509641] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:16.485764 (xid=0x3073f)
[2017-09-08 09:32:17.509957] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GF-DUMP)
op(NULL(2)) called at 2017-09-08 09:32:16.485786 (xid=0x30740)
[2017-09-08 09:32:17.510224] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486075 (xid=0x30741)
[2017-09-08 09:32:17.511645] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:32:17.486217 (xid=0x30742)
[2017-09-08 09:32:17.511983] E [rpc-clnt.c:365:sa
...

Pavel Szalbot

2017-09-08 10:43:38 UTC

https://joejulian.name/blog/keeping-your-vms-from-going-read-only-when-encountering-a-ping-timeout-in-glusterfs/

I know about this post, but as I pointed out - ping-timeout does not
seem to prevent read-only remount and errors=continue looks dangerous
and explaining customers/tenants that VMs need to change rootfs mount
options in order to avoid data corruption...

Vijay Bellur

2017-09-10 13:02:48 UTC

Post by Pavel Szalbot
[2017-09-08 09:31:48.381077] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded
in the last 1 seconds, disconnecting.

1 second is not an ideal value for ping timeout. Can you please set it to
30 seconds or so and simulate the problem? I would be interested in
observing your logs with a higher ping timeout value.

Thanks,
Vijay

Pavel Szalbot

2017-09-11 09:16:20 UTC

1 second is not an ideal value for ping timeout. Can you please set it to 30
seconds or so and simulate the problem? I would be interested in observing
your logs with a higher ping timeout value.

I restored the default 42 seconds and do not experience crashes now
even with gfapi except occasional performance drops reported by fio.
This applies to both replica 3 with and without arbiter. I had
previously used timeout 3 or 5 seconds and 1sec was set at the end of
the last week so I tested the 5 seconds timeout now and got crash
quite fast.
I guess the ping to the nodes that are actually up must timeout as
well and that's the reason for IO errors. But 5 seconds seem like
quite a long time. Or could it be something else?

BTW I restart primary mount point (the one specified in
mount.glusterfs) IO stops for several seconds and then fluctuates
between 0 and something more or less expected and this fluctuation is
present until the node comes up. If I restart the other node in the
cluster, this does not happen. Is this to be expected?

Thanks
-ps

2017-09-09 01:27:40 UTC

I've always wondered what the scenario for these situations are (aside
from the doc description of nodes coming up and down).

Aren't Gluster writes atomic for all nodes? I seem to recall Jeff Darcy
stating that years ago.

So a clean shutdown for maintenance shouldn't be a problem at all. If a
node didn't get a write, it is the one likely to fail.

So are we really only talking about a crash with data on the fly.

I suppose a crash during the heal phase after a shutdown could trigger
this issue, especially if you are not using sharding and had huge VM files.

Post by Pavel Szalbot
Hi Neil, docs mention two live nodes of replica 3 blaming each other
and refusing to do IO.
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/#1-replica-3-volume

Gionatan Danti

2017-09-03 20:18:33 UTC

Excellent! So anything with replica 2 + arbiter or better still replica
3 should be safe.

Post by Everton Brogliatto
I had an experimental setup in replica 2 using an older version of
Gluster few years ago and it was unstable, corrupt data and crashed
many times. Do not use replica 2. As others have already said, minimum
is replica 2+1 arbiter.

Yes, I had the same experience about 3 years ago, with frequent split
brains and file corruption. This was a test setup only, but it left a
somewhat "bad taste" in my mouth. I'm glad to ear that stability is much
better now.

Post by Everton Brogliatto
If you have any questions that I perhaps can help with, drop me an email.

If needed, I'll surely do that.
Thank for your kindness.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: ***@assyoma.it - ***@assyoma.it
GPG public key ID: FF5F32A8

2017-08-23 18:50:40 UTC