Discussion:
[Freeipa-users] Fw: ssh problem with migrated FreeIPA client on EL7.1 -->Not Solved
Christopher Lamb
2015-06-01 17:35:11 UTC
Permalink
Hi All

Bad news.

Over the weekend I was able to get the original problem EL7.1 / FreeIPA 4.1
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).

Today I tried a second machine, and had the same problem, ssh connections
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity check
failed"

Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....

Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....

b.t.w, as this machine is a real physical server, I was able to try logging
in direct with my FreeIPA user --> "Authentication Failure"

I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the old
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts to
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the new
FreeIPA server, and successfully authenticates FreeIPA users.

Any ideas?

Chris


----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----

From: Christopher Lamb/Switzerland/***@IBMCH
To: Alexander Bokovoy <***@redhat.com>,
freeipa-***@redhat.com
Date: 30.05.2015 18:52
Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA client on
EL7.1 --> Solved
Sent by: freeipa-users-***@redhat.com



Hi All

It gives me pleasure to report the problem is solved - a minute ago I was
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!

Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.

Regardless of what I did with commands like klist, kvno everything seemed
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.

Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled, configured
- ét voilà I could ssh in!

This leaves the enigma: what caused the problem? I suspect the following:

The host is an EL 7.1, but the first FreeIPA client installed was version
3.3.3 (installed as set of standard packages that we bung on all our
servers).

This worked fine to authenticate against our "old" 3.x FreeIPA server, but
did not work against the "new" 4.1 FreeIPA Server.

When I realised I could not ssh in, one of the first things I did was to
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not help.
The solution was to yum remove the FreeIPA client, then yum install the 4.1
client.

I have some more EL 7.1 servers with the FreeIPA 3.3.3 client installed, so
it will be interesting to see it the problem can be reproduced.

Keep up the good work,

Chris








From: Alexander Bokovoy <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH
Cc: freeipa-***@redhat.com
Date: 29.05.2015 18:04
Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA
client on
EL7.1
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to replace
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully migrated
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started migrating
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server, I
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to work,
and
getent passwd was successful for my FreeIPA user. However when I try and
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though evidently
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and working.
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user. The
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also rejected,
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error in /var/log/messages
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy





--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Jakub Hrozek
2015-06-02 07:21:32 UTC
Permalink
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA 4.1
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh connections
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity check
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try logging
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the old
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts to
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the new
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Date: 30.05.2015 18:52
Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA client on
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I was
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything seemed
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled, configured
- ét voilà I could ssh in!
The host is an EL 7.1, but the first FreeIPA client installed was version
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server, but
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was to
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not help.
The solution was to yum remove the FreeIPA client, then yum install the 4.1
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client installed, so
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Date: 29.05.2015 18:04
Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to replace
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully migrated
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started migrating
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server, I
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to work,
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try and
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though evidently
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and working.
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user. The
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also rejected,
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error in /var/log/messages
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-02 07:43:48 UTC
Permalink
Hi Jakub

The same user / password works with all our FreeIPA hosts - just this one
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.

As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the day.

My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.

Cheers

Chris



From: Jakub Hrozek <***@redhat.com>
To: freeipa-***@redhat.com
Date: 02.06.2015 09:22
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA 4.1
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh connections
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity check
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try logging
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the old
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts to
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the new
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Date: 30.05.2015 18:52
Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I was
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything seemed
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled, configured
- ét voilà I could ssh in!
The host is an EL 7.1, but the first FreeIPA client installed was version
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server, but
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was to
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not help.
The solution was to yum remove the FreeIPA client, then yum install the 4.1
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client installed, so
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Date: 29.05.2015 18:04
Subject: Re: [Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to replace
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully migrated
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started migrating
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server, I
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to work,
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try and
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though evidently
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and working.
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user. The
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also rejected,
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Jakub Hrozek
2015-06-02 07:50:41 UTC
Permalink
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this one
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
Date: 02.06.2015 09:22
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh connections
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the old
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the new
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Date: 30.05.2015 18:52
Subject: Re: [Freeipa-users] ssh problem with migrated FreeIPA
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I was
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything seemed
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
The host is an EL 7.1, but the first FreeIPA client installed was version
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was to
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not help.
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client installed,
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Date: 29.05.2015 18:04
Subject: Re: [Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully migrated
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server, I
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to work,
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try and
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-02 08:39:31 UTC
Permalink
Hi Jakub

Yes root login works, that's how I've been getting into the box.

Surprisingly, kinit with my user seems to work on that box. After entering
my password when prompted, it returns to the commandline without error.

However if I try kinit with another FreeIPA user, then instead of prompting
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.

Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like

"Retrieving host .... with result: .. Matching credential not found"

"Received error from KDC ... Additional pre-authentication required"

"Received error from KDC... Decrypt integrity check failed"

"Received error code 1432158219"

Cheers

Chris





From: Jakub Hrozek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH
Cc: freeipa-***@redhat.com
Date: 02.06.2015 09:50
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this one
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
Date: 02.06.2015 09:22
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh connections
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the old
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the new
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Date: 30.05.2015 18:52
Subject: Re: [Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I was
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything seemed
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
The host is an EL 7.1, but the first FreeIPA client installed was version
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was to
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not help.
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client installed,
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
From: Alexander Bokovoy
To: Christopher
Date: 29.05.2015 18:04
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully migrated
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server, I
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to work,
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try and
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Jakub Hrozek
2015-06-02 18:09:27 UTC
Permalink
Post by Christopher Lamb
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After entering
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of prompting
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Replied more in-depth off-list because the logs came in a private mail
but for anyone having similar symptoms -- the Kerberos tracing info
includes the IP address of the KDC we're trying to talk to. It's worth
checking if it's the server that knows the user principal etc..
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-02 16:15:04 UTC
Permalink
Hi

Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause
of this problem. Let's call them HOST09 and HOST10

Both are mimimum installs of EL7.1, with NTPD installed and configured.

HOST09 had ipa-client 4.1 installed via yum, and was configured to use our
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.

HOST10 had ipa-client 4.1 installed as a dependency of one of our standard
config packages, and was first set to use our old FreeIPA 3.3.3 server. -->
My FreeIPA user authenticates successfully. against this machine.

I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.

This replicates well the behaviour I saw with my production servers, namely
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new 4.1
FreeIPA server authenticate properly.

b) EL 7.1 hosts with ipa-client 4.1 first registered against the old 3.3.3
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly

Chris



----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----

From: Christopher Lamb/Switzerland/***@IBMCH
To: Jakub Hrozek <***@redhat.com>
Cc: freeipa-***@redhat.com
Date: 02.06.2015 10:40
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Sent by: freeipa-users-***@redhat.com



Hi Jakub

Yes root login works, that's how I've been getting into the box.

Surprisingly, kinit with my user seems to work on that box. After entering
my password when prompted, it returns to the commandline without error.

However if I try kinit with another FreeIPA user, then instead of prompting
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.

Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like

"Retrieving host .... with result: .. Matching credential not found"

"Received error from KDC ... Additional pre-authentication required"

"Received error from KDC... Decrypt integrity check failed"

"Received error code 1432158219"

Cheers

Chris





From: Jakub Hrozek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH
Cc: freeipa-***@redhat.com
Date: 02.06.2015 09:50
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this one
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.

Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
Date: 02.06.2015 09:22
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
From: Christopher
To: Alexander Bokovoy
Date: 30.05.2015 18:52
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
Re:
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Martin Kosek
2015-06-03 07:34:28 UTC
Permalink
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use our
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our standard
config packages, and was first set to use our old FreeIPA 3.3.3 server. -->
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers, namely
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new 4.1
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old 3.3.3
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly
Chris
Hello,

This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA client
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that will
probably differ is sssd.conf and krb5.conf as they will have different primary
server set up, based on what your DNS setup is.

It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to help
here, I would suggest including the SSSD logs from the failed login, it may help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
Date: 02.06.2015 10:40
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After entering
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of prompting
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
Date: 02.06.2015 09:50
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this one
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
Date: 02.06.2015 09:22
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
From: Christopher
To: Alexander Bokovoy
Date: 30.05.2015 18:52
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Jakub Hrozek
2015-06-03 07:48:09 UTC
Permalink
Post by Martin Kosek
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use our
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our standard
config packages, and was first set to use our old FreeIPA 3.3.3 server. -->
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers, namely
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new 4.1
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old 3.3.3
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly
Chris
Hello,
This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA client
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that will
probably differ is sssd.conf and krb5.conf as they will have different primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to help
here, I would suggest including the SSSD logs from the failed login, it may help.
In another thread (not sure if public or not, there was many emails from
Christoper recently), we advised to clean the cache after
reinstall/register.
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-03 08:30:22 UTC
Permalink
Hi all

This is a quick(ish) note to bring everybody up to speed on this issue.
Yesterday we had some private mail exchange on this issue as I did not wish
to broadcast the krb5 and ipa install logs to the user list.

The basic situation is that we are in the process of migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As discussed
in a thread some weeks ago we did not do this by replicating (as perhaps we
should have done). Instead we migrated the users across.

We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to
the old KDC. We are now in the process of migrating these hosts to the new
4.1 KDC.

Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these joining
to the new KDC was trouble free, taking a few minutes each. After joining
the new KDC FreeIPA users authenticated properly.

We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were
joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
KDC. These were also trouble free.

The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1 hosts
that were originally joined to the 3.3.3 KDC, and must be moved to join the
4.1 KDC. These machines no longer authenticate valid FreeIPA users. I have
been able to reproduce this behaviour with a freshly setup VM joined first
to the 3.3.3 KDC, then moved to the 4.1 KDC.

While the errors show in the krb5 child logs indicate that the password is
incorrect, the same user / password is happily accepted by all the other
hosts.

It seems that in the process of moving / migrating the EL 7.1 / ipa-client
4.1 from the old KDC to the new KDC, "something" is left behind that causes
problems. We have seen indications in the install logs that the kinit steps
called during ipa-client install are getting responses from the wrong (old)
KDC, and not from the new KDC.

Frustratingly. over the weekend i managed to get one of the problem EL 7.1
boxes to work. However I can't work out exactly what I was that I did that
did the trick. However it seems that some kind of major de-install /
cleanup + reinstall of the ipa-client may be needed.

Rob has suggested that as part of such a cleanup I should do "rm
-f /var/lib/sssd/db/*". I will test this later today and report back.

Thanks to Rob, Jakub, Martin, Alexander et al for their help and
suggestions so far.

Chris




From: Martin Kosek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH,
freeipa-***@redhat.com, Jakub Hrozek <***@redhat.com>
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use our
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our standard
config packages, and was first set to use our old FreeIPA 3.3.3 server. -->
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers, namely
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new 4.1
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old 3.3.3
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly
Chris
Hello,

This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA client
should matter if the deployment is set up properly. The host enrollment
entry
should simply replicate to whole infrastructure. The only thing that will
probably differ is sssd.conf and krb5.conf as they will have different
primary
server set up, based on what your DNS setup is.

It rather seems that the "reregistration" is what causes the issue. It
looks
like something cleanup problem during the process. I will let Jakub to help
here, I would suggest including the SSSD logs from the failed login, it may
help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
Date: 02.06.2015 10:40
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After entering
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of prompting
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
Date: 02.06.2015 09:50
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
Post by Christopher Lamb
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this one
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
From: Jakub Hrozek
Date: 02.06.2015 09:22
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
Post by Christopher Lamb
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Martin Kosek
2015-06-03 08:39:35 UTC
Permalink
Post by Christopher Lamb
Hi all
This is a quick(ish) note to bring everybody up to speed on this issue.
Yesterday we had some private mail exchange on this issue as I did not wish
to broadcast the krb5 and ipa install logs to the user list.
The basic situation is that we are in the process of migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As discussed
in a thread some weeks ago we did not do this by replicating (as perhaps we
should have done). Instead we migrated the users across.
We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to
the old KDC. We are now in the process of migrating these hosts to the new
4.1 KDC.
Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these joining
to the new KDC was trouble free, taking a few minutes each. After joining
the new KDC FreeIPA users authenticated properly.
We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were
joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
KDC. These were also trouble free.
The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1 hosts
that were originally joined to the 3.3.3 KDC, and must be moved to join the
4.1 KDC. These machines no longer authenticate valid FreeIPA users. I have
been able to reproduce this behaviour with a freshly setup VM joined first
to the 3.3.3 KDC, then moved to the 4.1 KDC.
While the errors show in the krb5 child logs indicate that the password is
incorrect, the same user / password is happily accepted by all the other
hosts.
It seems that in the process of moving / migrating the EL 7.1 / ipa-client
4.1 from the old KDC to the new KDC, "something" is left behind that causes
problems. We have seen indications in the install logs that the kinit steps
called during ipa-client install are getting responses from the wrong (old)
KDC, and not from the new KDC.
Frustratingly. over the weekend i managed to get one of the problem EL 7.1
boxes to work. However I can't work out exactly what I was that I did that
did the trick. However it seems that some kind of major de-install /
cleanup + reinstall of the ipa-client may be needed.
Rob has suggested that as part of such a cleanup I should do "rm
-f /var/lib/sssd/db/*". I will test this later today and report back.
Thanks to Rob, Jakub, Martin, Alexander et al for their help and
suggestions so far.
Chris
Thanks for the background. The pain you are getting is exactly the reason why
migration via replication to RHEL-7.1 is a better choice :-) Please let us know
the result, I am curious how this works out.
Post by Christopher Lamb
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use
our
Post by Christopher Lamb
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our
standard
Post by Christopher Lamb
config packages, and was first set to use our old FreeIPA 3.3.3 server.
-->
Post by Christopher Lamb
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers,
namely
Post by Christopher Lamb
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
4.1
Post by Christopher Lamb
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
3.3.3
Post by Christopher Lamb
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly
Chris
Hello,
This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA client
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that will
probably differ is sssd.conf and krb5.conf as they will have different primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to help
here, I would suggest including the SSSD logs from the failed login, it may help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
Date: 02.06.2015 10:40
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After
entering
Post by Christopher Lamb
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of
prompting
Post by Christopher Lamb
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
Date: 02.06.2015 09:50
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
Post by Christopher Lamb
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this
one
Post by Christopher Lamb
Post by Christopher Lamb
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
From: Jakub Hrozek
Date: 02.06.2015 09:22
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
Post by Christopher Lamb
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-04 17:34:05 UTC
Permalink
Hi All

I can now report back success (at least on my throwaway EL7.1 test VM).

To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC to
a new FreeIPA 4.1 KDC 3 steps are required:

1) ipa-client-install --uninstall

2) rm -f /var/lib/sss/db/*

3) ipa-client-install --server ldap.my.example.com --domain my.example.com
-N

Having done this, my free-ipa user successfully authenticates (e.g. ssh
remote login with free-ipa user / password


To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.

Kudos and thanks go to Rob C for suggesting step 2. (Note that the
directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
suggested earlier in this thread.

Cheers

Chris




From: Martin Kosek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH,
freeipa-***@redhat.com
Cc: Jakub Hrozek <***@redhat.com>, Rob Crittenden
<***@redhat.com>
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi all
This is a quick(ish) note to bring everybody up to speed on this issue.
Yesterday we had some private mail exchange on this issue as I did not wish
to broadcast the krb5 and ipa install logs to the user list.
The basic situation is that we are in the process of migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As discussed
in a thread some weeks ago we did not do this by replicating (as perhaps we
should have done). Instead we migrated the users across.
We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to
the old KDC. We are now in the process of migrating these hosts to the new
4.1 KDC.
Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these joining
to the new KDC was trouble free, taking a few minutes each. After joining
the new KDC FreeIPA users authenticated properly.
We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were
joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
KDC. These were also trouble free.
The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1 hosts
that were originally joined to the 3.3.3 KDC, and must be moved to join the
4.1 KDC. These machines no longer authenticate valid FreeIPA users. I have
been able to reproduce this behaviour with a freshly setup VM joined first
to the 3.3.3 KDC, then moved to the 4.1 KDC.
While the errors show in the krb5 child logs indicate that the password is
incorrect, the same user / password is happily accepted by all the other
hosts.
It seems that in the process of moving / migrating the EL 7.1 / ipa-client
4.1 from the old KDC to the new KDC, "something" is left behind that causes
problems. We have seen indications in the install logs that the kinit steps
called during ipa-client install are getting responses from the wrong (old)
KDC, and not from the new KDC.
Frustratingly. over the weekend i managed to get one of the problem EL 7.1
boxes to work. However I can't work out exactly what I was that I did that
did the trick. However it seems that some kind of major de-install /
cleanup + reinstall of the ipa-client may be needed.
Rob has suggested that as part of such a cleanup I should do "rm
-f /var/lib/sssd/db/*". I will test this later today and report back.
Thanks to Rob, Jakub, Martin, Alexander et al for their help and
suggestions so far.
Chris
Thanks for the background. The pain you are getting is exactly the reason
why
migration via replication to RHEL-7.1 is a better choice :-) Please let us
know
the result, I am curious how this works out.
Post by Christopher Lamb
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use
our
Post by Christopher Lamb
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our
standard
Post by Christopher Lamb
config packages, and was first set to use our old FreeIPA 3.3.3 server.
-->
Post by Christopher Lamb
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers,
namely
Post by Christopher Lamb
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
4.1
Post by Christopher Lamb
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
3.3.3
Post by Christopher Lamb
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly
Chris
Hello,
This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA client
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that will
probably differ is sssd.conf and krb5.conf as they will have different primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to help
here, I would suggest including the SSSD logs from the failed login, it may
help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
Date: 02.06.2015 10:40
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
Post by Christopher Lamb
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After
entering
Post by Christopher Lamb
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of
prompting
Post by Christopher Lamb
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
From: Jakub Hrozek
To: Christopher
Date: 02.06.2015 09:50
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
Post by Christopher Lamb
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this
one
Post by Christopher Lamb
Post by Christopher Lamb
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
Jakub Hrozek
02.06.2015 09:22
Post by Christopher Lamb
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
Post by Christopher Lamb
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
Post by Christopher Lamb
Post by Christopher Lamb
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Martin Kosek
2015-06-05 06:06:09 UTC
Permalink
Post by Christopher Lamb
Hi All
I can now report back success (at least on my throwaway EL7.1 test VM).
To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC to
1) ipa-client-install --uninstall
2) rm -f /var/lib/sss/db/*
3) ipa-client-install --server ldap.my.example.com --domain my.example.com
-N
Having done this, my free-ipa user successfully authenticates (e.g. ssh
remote login with free-ipa user / password
To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
Kudos and thanks go to Rob C for suggesting step 2. (Note that the
directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
suggested earlier in this thread.
Cool! Thanks for reaching back. I added this advice to the FreeIPA
Troubleshooting guide too:

http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
Post by Christopher Lamb
Cheers
Chris
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi all
This is a quick(ish) note to bring everybody up to speed on this issue.
Yesterday we had some private mail exchange on this issue as I did not
wish
Post by Christopher Lamb
to broadcast the krb5 and ipa install logs to the user list.
The basic situation is that we are in the process of migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As
discussed
Post by Christopher Lamb
in a thread some weeks ago we did not do this by replicating (as perhaps
we
Post by Christopher Lamb
should have done). Instead we migrated the users across.
We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to
the old KDC. We are now in the process of migrating these hosts to the
new
Post by Christopher Lamb
4.1 KDC.
Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these
joining
Post by Christopher Lamb
to the new KDC was trouble free, taking a few minutes each. After joining
the new KDC FreeIPA users authenticated properly.
We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were
joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
KDC. These were also trouble free.
The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1
hosts
Post by Christopher Lamb
that were originally joined to the 3.3.3 KDC, and must be moved to join
the
Post by Christopher Lamb
4.1 KDC. These machines no longer authenticate valid FreeIPA users. I
have
Post by Christopher Lamb
been able to reproduce this behaviour with a freshly setup VM joined
first
Post by Christopher Lamb
to the 3.3.3 KDC, then moved to the 4.1 KDC.
While the errors show in the krb5 child logs indicate that the password
is
Post by Christopher Lamb
incorrect, the same user / password is happily accepted by all the other
hosts.
It seems that in the process of moving / migrating the EL 7.1 /
ipa-client
Post by Christopher Lamb
4.1 from the old KDC to the new KDC, "something" is left behind that
causes
Post by Christopher Lamb
problems. We have seen indications in the install logs that the kinit
steps
Post by Christopher Lamb
called during ipa-client install are getting responses from the wrong
(old)
Post by Christopher Lamb
KDC, and not from the new KDC.
Frustratingly. over the weekend i managed to get one of the problem EL
7.1
Post by Christopher Lamb
boxes to work. However I can't work out exactly what I was that I did
that
Post by Christopher Lamb
did the trick. However it seems that some kind of major de-install /
cleanup + reinstall of the ipa-client may be needed.
Rob has suggested that as part of such a cleanup I should do "rm
-f /var/lib/sssd/db/*". I will test this later today and report back.
Thanks to Rob, Jakub, Martin, Alexander et al for their help and
suggestions so far.
Chris
Thanks for the background. The pain you are getting is exactly the reason why
migration via replication to RHEL-7.1 is a better choice :-) Please let us know
the result, I am curious how this works out.
Post by Christopher Lamb
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the
cause
Post by Christopher Lamb
Post by Christopher Lamb
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use
our
Post by Christopher Lamb
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our
standard
Post by Christopher Lamb
config packages, and was first set to use our old FreeIPA 3.3.3 server.
-->
Post by Christopher Lamb
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers,
namely
Post by Christopher Lamb
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
4.1
Post by Christopher Lamb
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
3.3.3
Post by Christopher Lamb
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly
Chris
Hello,
This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA
client
Post by Christopher Lamb
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that will
probably differ is sssd.conf and krb5.conf as they will have different primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to
help
Post by Christopher Lamb
here, I would suggest including the SSSD logs from the failed login, it
may
Post by Christopher Lamb
help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
Date: 02.06.2015 10:40
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
Post by Christopher Lamb
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After
entering
Post by Christopher Lamb
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of
prompting
Post by Christopher Lamb
for a password, it gives "Generic preauthentication failure while
getting
Post by Christopher Lamb
Post by Christopher Lamb
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
From: Jakub Hrozek
To: Christopher
Date: 02.06.2015 09:50
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
Post by Christopher Lamb
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this
one
Post by Christopher Lamb
Post by Christopher Lamb
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried
many
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up
2
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of
4.1
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
client.
Cheers
Chris
Jakub Hrozek
02.06.2015 09:22
Post by Christopher Lamb
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 /
FreeIPA
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA
user.
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate,
but
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all
attempts
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015
19:17
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
Post by Christopher Lamb
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server,
while
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install
the
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
Post by Christopher Lamb
Post by Christopher Lamb
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-05 14:25:09 UTC
Permalink
Hi Martin

Thanks for updating the documenation!

The suggested solution works not only my test servers, but also "in the
real world". This morning I migrated the last production server (ipa host)
to the new FreeIPA KDC.

Just out of idle curiosity, why is the rm -f /var/lib/sss/db/* step
required on our EL 7.1 + ipa-client 4.1 boxes, but not on our older EL 6.5
+ ipa-client 3.3.3 machines?

Is the problem down to sssd? (on the EL 6.5 machines we are running sssd
1.9.2, while on EL 7.1 we have sssd 1.12.2

Cheers

Chris



From: Martin Kosek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH, Rob Crittenden
<***@redhat.com>, freeipa-***@redhat.com
Cc: Jakub Hrozek <***@redhat.com>
Date: 05.06.2015 08:06
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Solved
Post by Christopher Lamb
Hi All
I can now report back success (at least on my throwaway EL7.1 test VM).
To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC to
1) ipa-client-install --uninstall
2) rm -f /var/lib/sss/db/*
3) ipa-client-install --server ldap.my.example.com --domain
my.example.com
Post by Christopher Lamb
-N
Having done this, my free-ipa user successfully authenticates (e.g. ssh
remote login with free-ipa user / password
To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
Kudos and thanks go to Rob C for suggesting step 2. (Note that the
directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
suggested earlier in this thread.
Cool! Thanks for reaching back. I added this advice to the FreeIPA
Troubleshooting guide too:

http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
Post by Christopher Lamb
Cheers
Chris
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi all
This is a quick(ish) note to bring everybody up to speed on this issue.
Yesterday we had some private mail exchange on this issue as I did not
wish
Post by Christopher Lamb
to broadcast the krb5 and ipa install logs to the user list.
The basic situation is that we are in the process of migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As
discussed
Post by Christopher Lamb
in a thread some weeks ago we did not do this by replicating (as perhaps
we
Post by Christopher Lamb
should have done). Instead we migrated the users across.
We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined to
the old KDC. We are now in the process of migrating these hosts to the
new
Post by Christopher Lamb
4.1 KDC.
Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these
joining
Post by Christopher Lamb
to the new KDC was trouble free, taking a few minutes each. After joining
the new KDC FreeIPA users authenticated properly.
We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that were
joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
KDC. These were also trouble free.
The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1
hosts
Post by Christopher Lamb
that were originally joined to the 3.3.3 KDC, and must be moved to join
the
Post by Christopher Lamb
4.1 KDC. These machines no longer authenticate valid FreeIPA users. I
have
Post by Christopher Lamb
been able to reproduce this behaviour with a freshly setup VM joined
first
Post by Christopher Lamb
to the 3.3.3 KDC, then moved to the 4.1 KDC.
While the errors show in the krb5 child logs indicate that the password
is
Post by Christopher Lamb
incorrect, the same user / password is happily accepted by all the other
hosts.
It seems that in the process of moving / migrating the EL 7.1 /
ipa-client
Post by Christopher Lamb
4.1 from the old KDC to the new KDC, "something" is left behind that
causes
Post by Christopher Lamb
problems. We have seen indications in the install logs that the kinit
steps
Post by Christopher Lamb
called during ipa-client install are getting responses from the wrong
(old)
Post by Christopher Lamb
KDC, and not from the new KDC.
Frustratingly. over the weekend i managed to get one of the problem EL
7.1
Post by Christopher Lamb
boxes to work. However I can't work out exactly what I was that I did
that
Post by Christopher Lamb
did the trick. However it seems that some kind of major de-install /
cleanup + reinstall of the ipa-client may be needed.
Rob has suggested that as part of such a cleanup I should do "rm
-f /var/lib/sssd/db/*". I will test this later today and report back.
Thanks to Rob, Jakub, Martin, Alexander et al for their help and
suggestions so far.
Chris
Thanks for the background. The pain you are getting is exactly the reason why
migration via replication to RHEL-7.1 is a better choice :-) Please let us
know
the result, I am curious how this works out.
Post by Christopher Lamb
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
Post by Christopher Lamb
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the
cause
Post by Christopher Lamb
Post by Christopher Lamb
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use
our
Post by Christopher Lamb
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our
standard
Post by Christopher Lamb
config packages, and was first set to use our old FreeIPA 3.3.3 server.
-->
Post by Christopher Lamb
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers,
namely
Post by Christopher Lamb
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
4.1
Post by Christopher Lamb
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
3.3.3
Post by Christopher Lamb
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly
Chris
Hello,
This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA
client
Post by Christopher Lamb
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that will
probably differ is sssd.conf and krb5.conf as they will have different primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to
help
Post by Christopher Lamb
here, I would suggest including the SSSD logs from the failed login, it
may
Post by Christopher Lamb
help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
From: Christopher
To: Jakub Hrozek
Date: 02.06.2015 10:40
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
Post by Christopher Lamb
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After
entering
Post by Christopher Lamb
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of
prompting
Post by Christopher Lamb
for a password, it gives "Generic preauthentication failure while
getting
Post by Christopher Lamb
Post by Christopher Lamb
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
Jakub Hrozek
Christopher
02.06.2015 09:50
Post by Christopher Lamb
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
Post by Christopher Lamb
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this
one
Post by Christopher Lamb
Post by Christopher Lamb
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried
many
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up
2
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of
4.1
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
client.
Cheers
Chris
Jakub Hrozek
02.06.2015 09:22
Post by Christopher Lamb
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 /
FreeIPA
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA
user.
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate,
but
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all
attempts
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015
19:17
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server,
while
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install
the
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
Post by Christopher Lamb
Post by Christopher Lamb
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Alexander Bokovoy
2015-06-05 14:30:12 UTC
Permalink
Post by Christopher Lamb
Hi Martin
Thanks for updating the documenation!
The suggested solution works not only my test servers, but also "in the
real world". This morning I migrated the last production server (ipa host)
to the new FreeIPA KDC.
Just out of idle curiosity, why is the rm -f /var/lib/sss/db/* step
required on our EL 7.1 + ipa-client 4.1 boxes, but not on our older EL 6.5
+ ipa-client 3.3.3 machines?
Is the problem down to sssd? (on the EL 6.5 machines we are running sssd
1.9.2, while on EL 7.1 we have sssd 1.12.2
I think there are more object types supported by newer SSSD versions
which aren't invalidated like users or groups.
Post by Christopher Lamb
Cheers
Chris
Date: 05.06.2015 08:06
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Solved
Post by Christopher Lamb
Hi All
I can now report back success (at least on my throwaway EL7.1 test VM).
To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC
to
Post by Christopher Lamb
1) ipa-client-install --uninstall
2) rm -f /var/lib/sss/db/*
3) ipa-client-install --server ldap.my.example.com --domain
my.example.com
Post by Christopher Lamb
-N
Having done this, my free-ipa user successfully authenticates (e.g. ssh
remote login with free-ipa user / password
To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
Kudos and thanks go to Rob C for suggesting step 2. (Note that the
directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
suggested earlier in this thread.
Cool! Thanks for reaching back. I added this advice to the FreeIPA
http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
Post by Christopher Lamb
Cheers
Chris
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi all
This is a quick(ish) note to bring everybody up to speed on this issue.
Yesterday we had some private mail exchange on this issue as I did not
wish
Post by Christopher Lamb
to broadcast the krb5 and ipa install logs to the user list.
The basic situation is that we are in the process of migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As
discussed
Post by Christopher Lamb
in a thread some weeks ago we did not do this by replicating (as perhaps
we
Post by Christopher Lamb
should have done). Instead we migrated the users across.
We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined
to
Post by Christopher Lamb
Post by Christopher Lamb
the old KDC. We are now in the process of migrating these hosts to the
new
Post by Christopher Lamb
4.1 KDC.
Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these
joining
Post by Christopher Lamb
to the new KDC was trouble free, taking a few minutes each. After
joining
Post by Christopher Lamb
Post by Christopher Lamb
the new KDC FreeIPA users authenticated properly.
We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that
were
Post by Christopher Lamb
Post by Christopher Lamb
joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
KDC. These were also trouble free.
The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1
hosts
Post by Christopher Lamb
that were originally joined to the 3.3.3 KDC, and must be moved to join
the
Post by Christopher Lamb
4.1 KDC. These machines no longer authenticate valid FreeIPA users. I
have
Post by Christopher Lamb
been able to reproduce this behaviour with a freshly setup VM joined
first
Post by Christopher Lamb
to the 3.3.3 KDC, then moved to the 4.1 KDC.
While the errors show in the krb5 child logs indicate that the password
is
Post by Christopher Lamb
incorrect, the same user / password is happily accepted by all the other
hosts.
It seems that in the process of moving / migrating the EL 7.1 /
ipa-client
Post by Christopher Lamb
4.1 from the old KDC to the new KDC, "something" is left behind that
causes
Post by Christopher Lamb
problems. We have seen indications in the install logs that the kinit
steps
Post by Christopher Lamb
called during ipa-client install are getting responses from the wrong
(old)
Post by Christopher Lamb
KDC, and not from the new KDC.
Frustratingly. over the weekend i managed to get one of the problem EL
7.1
Post by Christopher Lamb
boxes to work. However I can't work out exactly what I was that I did
that
Post by Christopher Lamb
did the trick. However it seems that some kind of major de-install /
cleanup + reinstall of the ipa-client may be needed.
Rob has suggested that as part of such a cleanup I should do "rm
-f /var/lib/sssd/db/*". I will test this later today and report back.
Thanks to Rob, Jakub, Martin, Alexander et al for their help and
suggestions so far.
Chris
Thanks for the background. The pain you are getting is exactly the reason why
migration via replication to RHEL-7.1 is a better choice :-) Please let
us
Post by Christopher Lamb
know
the result, I am curious how this works out.
Post by Christopher Lamb
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
Post by Christopher Lamb
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the
cause
Post by Christopher Lamb
Post by Christopher Lamb
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use
our
Post by Christopher Lamb
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our
standard
Post by Christopher Lamb
config packages, and was first set to use our old FreeIPA 3.3.3 server.
-->
Post by Christopher Lamb
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers,
namely
Post by Christopher Lamb
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
4.1
Post by Christopher Lamb
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
3.3.3
Post by Christopher Lamb
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do
NOT
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
authenticate properly
Chris
Hello,
This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA
client
Post by Christopher Lamb
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that
will
Post by Christopher Lamb
Post by Christopher Lamb
probably differ is sssd.conf and krb5.conf as they will have different primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to
help
Post by Christopher Lamb
here, I would suggest including the SSSD logs from the failed login, it
may
Post by Christopher Lamb
help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
From: Christopher
To: Jakub Hrozek
Date: 02.06.2015 10:40
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
Post by Christopher Lamb
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After
entering
Post by Christopher Lamb
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of
prompting
Post by Christopher Lamb
for a password, it gives "Generic preauthentication failure while
getting
Post by Christopher Lamb
Post by Christopher Lamb
initial credentials" error.
Having set debug_level=10, when I try and ssh in with my FreeIPA user,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
find errors like
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
Jakub Hrozek
Christopher
02.06.2015 09:50
Post by Christopher Lamb
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
Post by Christopher Lamb
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this
one
Post by Christopher Lamb
Post by Christopher Lamb
box is the problem. So the password should be good. Of course a type
is
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
always possible (especially for strong passwords), but I have tried
many
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
times which should eliminate the odd password typo. The user /
password
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to
the
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
fact the host originally was a FreeIPA 3.3.3 client, updated to
FreeIPA
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
4.1, and switched between 2 FreeIPA servers. I am currently setting up
2
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of
4.1
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
client.
Cheers
Chris
Jakub Hrozek
02.06.2015 09:22
Post by Christopher Lamb
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 /
FreeIPA
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being
ssh
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user
using
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA
user.
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate,
but
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all
attempts
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015
19:17
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server,
while
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail
exchange
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA
server,
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install
the
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
Post by Christopher Lamb
Post by Christopher Lamb
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA
4
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the
server,
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed
was
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA
user.
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA
users.
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
/ Alexander Bokovoy
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Prasun Gera
2015-06-05 17:47:19 UTC
Permalink
I had faced a similar issue a month ago, for which I had created a ticket.
https://fedorahosted.org/freeipa/ticket/4956
Post by Alexander Bokovoy
Post by Christopher Lamb
Hi Martin
Thanks for updating the documenation!
The suggested solution works not only my test servers, but also "in the
real world". This morning I migrated the last production server (ipa host)
to the new FreeIPA KDC.
Just out of idle curiosity, why is the rm -f /var/lib/sss/db/* step
required on our EL 7.1 + ipa-client 4.1 boxes, but not on our older EL 6.5
+ ipa-client 3.3.3 machines?
Is the problem down to sssd? (on the EL 6.5 machines we are running sssd
1.9.2, while on EL 7.1 we have sssd 1.12.2
I think there are more object types supported by newer SSSD versions
which aren't invalidated like users or groups.
Post by Christopher Lamb
Cheers
Chris
Date: 05.06.2015 08:06
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Solved
Post by Christopher Lamb
Hi All
I can now report back success (at least on my throwaway EL7.1 test VM).
To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA 3.3.3 KDC
to
Post by Christopher Lamb
1) ipa-client-install --uninstall
2) rm -f /var/lib/sss/db/*
3) ipa-client-install --server ldap.my.example.com --domain
my.example.com
Post by Christopher Lamb
-N
Having done this, my free-ipa user successfully authenticates (e.g. ssh
remote login with free-ipa user / password
To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
Kudos and thanks go to Rob C for suggesting step 2. (Note that the
directory to be purged is /var/lib/sss/db/, not /var/lib/sssd/db/ as
suggested earlier in this thread.
Cool! Thanks for reaching back. I added this advice to the FreeIPA
http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
Post by Christopher Lamb
Cheers
Chris
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi all
This is a quick(ish) note to bring everybody up to speed on this issue.
Yesterday we had some private mail exchange on this issue as I did not
wish
Post by Christopher Lamb
to broadcast the krb5 and ipa install logs to the user list.
The basic situation is that we are in the process of migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC). As
discussed
Post by Christopher Lamb
in a thread some weeks ago we did not do this by replicating (as perhaps
we
Post by Christopher Lamb
should have done). Instead we migrated the users across.
We have 30+ servers that are IPA clients ("Hosts" in ipa-speak) joined
to
the old KDC. We are now in the process of migrating these hosts to the
new
Post by Christopher Lamb
4.1 KDC.
Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of these
joining
Post by Christopher Lamb
to the new KDC was trouble free, taking a few minutes each. After
joining
the new KDC FreeIPA users authenticated properly.
Post by Christopher Lamb
We also had a small number of new EL 7.1 + ipa-client 4.1 hosts that
were
joined direct to the new 4.1 KDC, never having been joined of the 3.3.3
Post by Christopher Lamb
KDC. These were also trouble free.
The problem occurs with a handful of existing EL 7.1 +ipa-client 4.1
hosts
Post by Christopher Lamb
that were originally joined to the 3.3.3 KDC, and must be moved to join
the
Post by Christopher Lamb
4.1 KDC. These machines no longer authenticate valid FreeIPA users. I
have
Post by Christopher Lamb
been able to reproduce this behaviour with a freshly setup VM joined
first
Post by Christopher Lamb
to the 3.3.3 KDC, then moved to the 4.1 KDC.
While the errors show in the krb5 child logs indicate that the password
is
Post by Christopher Lamb
incorrect, the same user / password is happily accepted by all the other
hosts.
It seems that in the process of moving / migrating the EL 7.1 /
ipa-client
Post by Christopher Lamb
4.1 from the old KDC to the new KDC, "something" is left behind that
causes
Post by Christopher Lamb
problems. We have seen indications in the install logs that the kinit
steps
Post by Christopher Lamb
called during ipa-client install are getting responses from the wrong
(old)
Post by Christopher Lamb
KDC, and not from the new KDC.
Frustratingly. over the weekend i managed to get one of the problem EL
7.1
Post by Christopher Lamb
boxes to work. However I can't work out exactly what I was that I did
that
Post by Christopher Lamb
did the trick. However it seems that some kind of major de-install /
cleanup + reinstall of the ipa-client may be needed.
Rob has suggested that as part of such a cleanup I should do "rm
-f /var/lib/sssd/db/*". I will test this later today and report back.
Thanks to Rob, Jakub, Martin, Alexander et al for their help and
suggestions so far.
Chris
Thanks for the background. The pain you are getting is exactly the reason why
migration via replication to RHEL-7.1 is a better choice :-) Please let
us
Post by Christopher Lamb
know
the result, I am curious how this works out.
Post by Christopher Lamb
,
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users] Fw: ssh
problem with
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the
cause
of this problem. Let's call them HOST09 and HOST10
Post by Christopher Lamb
Both are mimimum installs of EL7.1, with NTPD installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and was configured to use
our
Post by Christopher Lamb
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a dependency of one of our
standard
Post by Christopher Lamb
config packages, and was first set to use our old FreeIPA 3.3.3 server.
-->
Post by Christopher Lamb
My FreeIPA user authenticates successfully. against this machine.
I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my production servers,
namely
Post by Christopher Lamb
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new
4.1
Post by Christopher Lamb
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first registered against the old
3.3.3
Post by Christopher Lamb
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do
NOT
authenticate properly
Post by Christopher Lamb
Post by Christopher Lamb
Chris
Hello,
This is really strange. What I do not fully understand is what is the
"registration against a FreeIPA server". What server you install IPA
client
Post by Christopher Lamb
should matter if the deployment is set up properly. The host enrollment entry
should simply replicate to whole infrastructure. The only thing that
will
probably differ is sssd.conf and krb5.conf as they will have different
Post by Christopher Lamb
primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what causes the issue. It looks
like something cleanup problem during the process. I will let Jakub to
help
Post by Christopher Lamb
here, I would suggest including the SSSD logs from the failed login, it
may
Post by Christopher Lamb
help.
Post by Christopher Lamb
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
Christopher
To: Jakub
Post by Christopher Lamb
Post by Christopher Lamb
Hrozek
02.06.2015 10:40
[Freeipa-users] Fw: ssh problem with
migrated
Post by Christopher Lamb
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting into the box.
Surprisingly, kinit with my user seems to work on that box. After
entering
Post by Christopher Lamb
my password when prompted, it returns to the commandline without error.
However if I try kinit with another FreeIPA user, then instead of
prompting
Post by Christopher Lamb
for a password, it gives "Generic preauthentication failure while
getting
initial credentials" error.
Post by Christopher Lamb
Having set debug_level=10, when I try and ssh in with my FreeIPA user,
I
find errors like
Post by Christopher Lamb
Post by Christopher Lamb
"Retrieving host .... with result: .. Matching credential not found"
"Received error from KDC ... Additional pre-authentication required"
"Received error from KDC... Decrypt integrity check failed"
"Received error code 1432158219"
Cheers
Chris
Jakub Hrozek
Christopher
02.06.2015 09:50
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
Post by Christopher Lamb
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this
one
box is the problem. So the password should be good. Of course a type
is
always possible (especially for strong passwords), but I have tried
Post by Christopher Lamb
Post by Christopher Lamb
many
times which should eliminate the odd password typo. The user /
Post by Christopher Lamb
password
should also be good for both the old and the new FreeIPA Server.
Post by Christopher Lamb
Post by Christopher Lamb
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.
Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to
the
fact the host originally was a FreeIPA 3.3.3 client, updated to
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up
Post by Christopher Lamb
Post by Christopher Lamb
2
throwaway EL 7.1 VMs to better test this. On one I will first install
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3, then upgrade to 4.1. The second will have a direct install of
4.1
client.
Post by Christopher Lamb
Post by Christopher Lamb
Cheers
Chris
Jakub Hrozek
02.06.2015 09:22
Post by Christopher Lamb
[Freeipa-users] Fw: ssh problem with
Post by Christopher Lamb
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 /
FreeIPA
4.1
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being
ssh
remote login with FreeIPA user and password).
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Today I tried a second machine, and had the same problem, ssh
connections
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user
using
the same password?
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA
user.
Argg.....
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
FreeIPA server to the new without a hitch (i.e. they successfully
Post by Christopher Lamb
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate,
but
with problems
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all
attempts
to
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
FreeIPA server, and successfully authenticates FreeIPA users.
Post by Christopher Lamb
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015
19:17
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
[Freeipa-users] ssh problem with
Post by Christopher Lamb
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
able to login via ssh with my FreeIPA user to the problem server,
while
sitting on my terrace with a glass of wine!
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Thanks to Alexander for his helpful advice - we had some mail
exchange
outside the user list as I did not wish to broadcast content of keys,
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
"ok", but I still could not ssh in. Even a ipa-getkeytab did not
help.
Post by Christopher Lamb
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
3.3.3 (installed as set of standard packages that we bung on all our
Post by Christopher Lamb
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA
server,
but
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
The solution was to yum remove the FreeIPA client, then yum install
the
4.1
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander
Bokovoy
Post by Christopher Lamb
Post by Christopher Lamb
Christopher
29.05.2015
18:04
[Freeipa-users] ssh problem with
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
across the users.
Post by Christopher Lamb
Post by Christopher Lamb
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA
4
server by doing an ipa-client-install --uninstall from the old, and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the
server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Post by Christopher Lamb
Post by Christopher Lamb
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
and
Post by Christopher Lamb
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
ssh in, my FreeIPA user / password is not accepted.
Post by Christopher Lamb
Post by Christopher Lamb
Before the migration I could ssh into the problem server (though
evidently
it was using my FreeIPA user from the old FreeIPA server).
Post by Christopher Lamb
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed
was
3.3.3, so I yum updated this to 4.1.0.
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
However I still cannot ssh into the OEL 7.1 box with my FreeIPA
user.
The
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
Post by Christopher Lamb
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
so the problem is not my user, but is probably for all FreeIPA
users.
Post by Christopher Lamb
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Dmitri Pal
2015-06-07 20:30:28 UTC
Permalink
Post by Prasun Gera
I had faced a similar issue a month ago, for which I had created a
ticket. https://fedorahosted.org/freeipa/ticket/4956
Hi Martin
Thanks for updating the documenation!
The suggested solution works not only my test servers, but also "in the
real world". This morning I migrated the last production server (ipa host)
to the new FreeIPA KDC.
Just out of idle curiosity, why is the rm -f
/var/lib/sss/db/* step
required on our EL 7.1 + ipa-client 4.1 boxes, but not on our older EL 6.5
+ ipa-client 3.3.3 machines?
Is the problem down to sssd? (on the EL 6.5 machines we are running sssd
1.9.2, while on EL 7.1 we have sssd 1.12.2
I think there are more object types supported by newer SSSD versions
which aren't invalidated like users or groups.
Cheers
Chris
Date: 05.06.2015 08:06
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Solved
Hi All
I can now report back success (at least on my throwaway
EL7.1 test VM).
To switch an EL 7.1 + ipa-client 4.1 host from an old
FreeIPA 3.3.3 KDC
to
1) ipa-client-install --uninstall
2) rm -f /var/lib/sss/db/*
3) ipa-client-install --server ldap.my.example.com
<http://ldap.my.example.com> --domain
my.example.com <http://my.example.com>
-N
Having done this, my free-ipa user successfully
authenticates (e.g. ssh
remote login with free-ipa user / password
To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not required.
Kudos and thanks go to Rob C for suggesting step 2. (Note that the
directory to be purged is /var/lib/sss/db/, not
/var/lib/sssd/db/ as
suggested earlier in this thread.
Cool! Thanks for reaching back. I added this advice to the FreeIPA
http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client
Cheers
Chris
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh
problem with migrated
FreeIPA
client on EL7.1 -->Not Solved
Hi all
This is a quick(ish) note to bring everybody up to
speed on this issue.
Yesterday we had some private mail exchange on this
issue as I did not
wish
to broadcast the krb5 and ipa install logs to the user
list.
The basic situation is that we are in the process of
migrating from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server
(KDC). As
discussed
in a thread some weeks ago we did not do this by
replicating (as perhaps
we
should have done). Instead we migrated the users across.
We have 30+ servers that are IPA clients ("Hosts" in
ipa-speak) joined
to
the old KDC. We are now in the process of migrating
these hosts to the
new
4.1 KDC.
Most of the hosts run EL 6.5 + ipa-client 3.3.3. For
all of these
joining
to the new KDC was trouble free, taking a few minutes
each. After
joining
the new KDC FreeIPA users authenticated properly.
We also had a small number of new EL 7.1 + ipa-client
4.1 hosts that
were
joined direct to the new 4.1 KDC, never having been
joined of the 3.3.3
KDC. These were also trouble free.
The problem occurs with a handful of existing EL 7.1
+ipa-client 4.1
hosts
that were originally joined to the 3.3.3 KDC, and must
be moved to join
the
4.1 KDC. These machines no longer authenticate valid
FreeIPA users. I
have
been able to reproduce this behaviour with a freshly
setup VM joined
first
to the 3.3.3 KDC, then moved to the 4.1 KDC.
While the errors show in the krb5 child logs indicate
that the password
is
incorrect, the same user / password is happily
accepted by all the other
hosts.
It seems that in the process of moving / migrating the
EL 7.1 /
ipa-client
4.1 from the old KDC to the new KDC, "something" is
left behind that
causes
problems. We have seen indications in the install logs
that the kinit
steps
called during ipa-client install are getting responses
from the wrong
(old)
KDC, and not from the new KDC.
Frustratingly. over the weekend i managed to get one
of the problem EL
7.1
boxes to work. However I can't work out exactly what I
was that I did
that
did the trick. However it seems that some kind of
major de-install /
cleanup + reinstall of the ipa-client may be needed.
Rob has suggested that as part of such a cleanup I
should do "rm
-f /var/lib/sssd/db/*". I will test this later today
and report back.
Thanks to Rob, Jakub, Martin, Alexander et al for
their help and
suggestions so far.
Chris
Thanks for the background. The pain you are getting is
exactly the reason
why
migration via replication to RHEL-7.1 is a better choice
:-) Please let
us
know
the result, I am curious how this works out.
From: Martin Kosek
To: Christopher
Date: 03.06.2015 09:34
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved
Hi
Earlier today I setup 2 throwaway EL7.1 VMs to
help narrow down the
cause
of this problem. Let's call them HOST09 and HOST10
Both are mimimum installs of EL7.1, with NTPD
installed and configured.
HOST09 had ipa-client 4.1 installed via yum, and
was configured to use
our
new FreeIPA 4.1 server, right from the start. -->
My FreeIPA user
authenticates successfully against this machine.
HOST10 had ipa-client 4.1 installed as a
dependency of one of our
standard
config packages, and was first set to use our old
FreeIPA 3.3.3 server.
-->
My FreeIPA user authenticates successfully.
against this machine.
I then de-registered HOST10 from the FreeIPA 3.1
server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA
users does NOT
authenticate successfully.
This replicates well the behaviour I saw with my
production servers,
namely
a) EL 7.1 hosts with ipa-client 4.1 registered
directly against the new
4.1
FreeIPA server authenticate properly.
b) EL 7.1 hosts with ipa-client 4.1 first
registered against the old
3.3.3
FreeIPA server, then reregistered with the new 4.1
FreeIPA server do
NOT
authenticate properly
Chris
Hello,
This is really strange. What I do not fully understand
is what is the
"registration against a FreeIPA server". What server
you install IPA
client
should matter if the deployment is set up properly.
The host enrollment
entry
should simply replicate to whole infrastructure. The
only thing that
will
probably differ is sssd.conf and krb5.conf as they
will have different
primary
server set up, based on what your DNS setup is.
It rather seems that the "reregistration" is what
causes the issue. It
looks
like something cleanup problem during the process. I
will let Jakub to
help
here, I would suggest including the SSSD logs from the
failed login, it
may
help.
----- Forwarded by Christopher
Lamb/Switzerland/IBM on 02.06.2015 16:52
-----
From: Christopher
To: Jakub Hrozek
Date: 02.06.2015 10:40
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved
Hi Jakub
Yes root login works, that's how I've been getting
into the box.
Surprisingly, kinit with my user seems to work on
that box. After
entering
my password when prompted, it returns to the
commandline without error.
However if I try kinit with another FreeIPA user,
then instead of
prompting
for a password, it gives "Generic
preauthentication failure while
getting
initial credentials" error.
Having set debug_level=10, when I try and ssh in
with my FreeIPA user,
I
find errors like
"Retrieving host .... with result: .. Matching
credential not found"
"Received error from KDC ... Additional
pre-authentication required"
"Received error from KDC... Decrypt integrity
check failed"
"Received error code 1432158219"
Cheers
Chris
Jakub Hrozek
Christopher
02.06.2015 09:50
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved
On Tue, Jun 02, 2015 at 09:43:48AM +0200,
Hi Jakub
The same user / password works with all our
FreeIPA hosts - just this
one
box is the problem. So the password should be
good. Of course a type
is
always possible (especially for strong
passwords), but I have tried
many
times which should eliminate the odd password
typo. The user /
password
should also be good for both the old and the
new FreeIPA Server.
Interesting, can you add debug_level=10 to the
domain section of
sssd.conf? Then krb5_child.log should show
Kerberos tracing info
including which exact KDC SSSD was talking to.
As I can neither log in direct, or via ssh to
this box with my FreeIPA
user, I assume Kinit with my user won't work-
i will try later in the
day.
Well, login as a UNIX user (root) should work..
My working assumption is that the problem is
related in some way to
the
fact the host originally was a FreeIPA 3.3.3
client, updated to
FreeIPA
4.1, and switched between 2 FreeIPA servers. I
am currently setting up
2
throwaway EL 7.1 VMs to better test this. On
one I will first install
3.3.3, then upgrade to 4.1. The second will
have a direct install of
4.1
client.
Cheers
Chris
Jakub Hrozek
02.06.2015 09:22
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved
On Mon, Jun 01, 2015 at 07:35:11PM +0200,
Hi All
Bad news.
Over the weekend I was able to get the
original problem EL7.1 /
FreeIPA
4.1
host (FreeIPA client) to authenticate
FreeiPA users (my test being
ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had
the same problem, ssh
connections
with FreeIPA user cause
"[sssd[krb5_child[3445]]]: Decrypt integrity
check
failed"
This really just means wrong password, can you
kinit as that user
using
the same password?
just remove ipa-client and
reinstall via yum, register with the new
FreeIPA server ....
Only with this second machine I still
can't ssh in with a FreeIPA
user.
Argg.....
b.t.w, as this machine is a real physical
server, I was able to try
logging
in direct with my FreeIPA user -->
"Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3
hosts that migrated from the
old
FreeIPA server to the new without a hitch
(i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host
that I was able to migrate,
but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host
that so far defies all
attempts
to
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was
only ever registered with the
new
FreeIPA server, and successfully
authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher
Lamb/Switzerland/IBM on 01.06.2015
19:17
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
[Freeipa-users] ssh problem with
migrated FreeIPA
client on
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem
is solved - a minute ago I
was
able to login via ssh with my FreeIPA user
to the problem server,
while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice
- we had some mail
exchange
outside the user list as I did not wish to
broadcast content of keys,
config files etc.
Regardless of what I did with commands
like klist, kvno everything
seemed
"ok", but I still could not ssh in. Even a
ipa-getkeytab did not
help.
Therefore I decided to opt for brute force
and (partial) ignorance. I
completely uninstalled the FreeIPA client,
and then reinstalled,
configured
- ét voilà I could ssh in!
This leaves the enigma: what caused the
problem? I suspect the
The host is an EL 7.1, but the first
FreeIPA client installed was
version
3.3.3 (installed as set of standard
packages that we bung on all our
servers).
This worked fine to authenticate against
our "old" 3.x FreeIPA
server,
but
did not work against the "new" 4.1 FreeIPA
Server.
When I realised I could not ssh in, one of
the first things I did was
to
yum update the FreeIPA client from 3.3.3
to 4.1 - but that did not
help.
The solution was to yum remove the FreeIPA
client, then yum install
the
4.1
client.
I have some more EL 7.1 servers with the
FreeIPA 3.3.3 client
installed,
so
it will be interesting to see it the
problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
[Freeipa-users] ssh problem with
migrated FreeIPA
client on
EL7.1
Hi All
Some weeks ago I setup a new FreeIPA
4.1.0 on an OEL 7.1 server to
replace
the existing FreeIPA 3.0.0 running on
OEL 6.5, and successfully
migrated
across the users.
We have 50 odd Servers that are
FreeIPA clients. Today I started
migrating
these one-by-one from the old FreeIPA
3.x server to the new FreeIPA
4
server by doing an ipa-client-install
--uninstall from the old, and
ipa-client-install to register with
the new 4.1.0 server.
Most of the FreeIPA clients are
running OEL 6.5, and for these the
migration process above worked
perfectly. After migrating the
server,
I
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The
migration itself seemed to
work,
and
getent passwd was successful for my
FreeIPA user. However when I try
and
ssh in, my FreeIPA user / password is
not accepted.
Before the migration I could ssh into
the problem server (though
evidently
it was using my FreeIPA user from the
old FreeIPA server).
I can ssh in with a local (non ldap)
user, so ssh is running and
working.
From user root I can successfully su
to my FreeIPA user.
Further investigation showed that
version of ipa-client installed
was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the
OEL 7.1 box with my FreeIPA
user.
The
same user continues to work for the
6.5 boxes.
A colleague tried to ssh in with his
FreeIPA user, and was also
rejected,
so the problem is not my user, but is
probably for all FreeIPA
users.
A failed ssh login attempt causes the
following error
in /var/log/messages
[sssd[krb5_child[5393]]]: Decrypt
integrity check failed
It means /etc/krb5.keytab contains keys
from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
Manage your subscription for the
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on
the project
--
Manage your subscription for the
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on
the project
--
Manage your subscription for the Freeipa-users
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the
project
--
Manage your subscription for the Freeipa-users
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
FYI https://fedorahosted.org/freeipa/ticket/5050
--
Thank you,
Dmitri Pal

Director of Engineering for IdM portfolio
Red Hat, Inc.
Christopher Lamb
2015-06-08 07:23:11 UTC
Permalink
Hi Dmitri, Prasun

Thanks for those tickets. I have commented Dimitri's with a reference to
this thread.

Cheers

Chris



From: Dmitri Pal <***@redhat.com>
To: freeipa-***@redhat.com
Date: 07.06.2015 22:33
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Solved
Sent by: freeipa-users-***@redhat.com



On 06/05/2015 01:47 PM, Prasun Gera wrote:
I had faced a similar issue a month ago, for which I had created a
ticket. https://fedorahosted.org/freeipa/ticket/4956

On Fri, Jun 5, 2015 at 7:30 AM, Alexander Bokovoy <
***@redhat.com> wrote:
On Fri, 05 Jun 2015, Christopher Lamb wrote:
Hi Martin

Thanks for updating the documenation!

The suggested solution works not only my test servers, but also
"in the
real world". This morning I migrated the last production server
(ipa host)
to the new FreeIPA KDC.

Just out of idle curiosity, why is the rm -f /var/lib/sss/db/*
step
required on our EL 7.1 + ipa-client 4.1 boxes, but not on our
older EL 6.5
+ ipa-client 3.3.3 machines?

Is the problem down to sssd? (on the EL 6.5 machines we are
running sssd
1.9.2, while on EL 7.1 we have sssd 1.12.2
I think there are more object types supported by newer SSSD
versions
which aren't invalidated like users or groups.



Cheers

Chris



From: Martin Kosek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH, Rob Crittenden
<***@redhat.com>, freeipa-***@redhat.com
Cc: Jakub Hrozek <***@redhat.com>
Date: 05.06.2015 08:06
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
client on EL7.1 -->Solved



On 06/04/2015 07:34 PM, Christopher Lamb wrote:
Hi All

I can now report back success (at least on my throwaway EL7.1
test VM).

To switch an EL 7.1 + ipa-client 4.1 host from an old FreeIPA
3.3.3 KDC
to
a new FreeIPA 4.1 KDC 3 steps are required:

1) ipa-client-install --uninstall

2) rm -f /var/lib/sss/db/*

3) ipa-client-install --server ldap.my.example.com --domain
my.example.com
-N

Having done this, my free-ipa user successfully authenticates
(e.g. ssh
remote login with free-ipa user / password


To switch EL 6.5 + ipa-client 3.3.3 hosts step 2) was not
required.

Kudos and thanks go to Rob C for suggesting step 2. (Note that
the
directory to be purged is /var/lib/sss/db/,
not /var/lib/sssd/db/ as
suggested earlier in this thread.

Cool! Thanks for reaching back. I added this advice to the FreeIPA
Troubleshooting guide too:

http://www.freeipa.org/page/Troubleshooting#Cannot_authenticate_on_client



Cheers

Chris




From: Martin Kosek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH,
freeipa-***@redhat.com
Cc: Jakub Hrozek <***@redhat.com>, Rob
Crittenden
<***@redhat.com>
Date: 03.06.2015 10:39
Subject: Re: [Freeipa-users] Fw: ssh problem
with migrated
FreeIPA
client on EL7.1 -->Not Solved



On 06/03/2015 10:30 AM, Christopher Lamb wrote:
Hi all

This is a quick(ish) note to bring everybody up to speed on
this issue.
Yesterday we had some private mail exchange on this issue as I
did not
wish
to broadcast the krb5 and ipa install logs to the user list.

The basic situation is that we are in the process of migrating
from an
FreeIPA 3.3.3 Server (KDC) to a new FreeIPA 4.1 Server (KDC).
As
discussed
in a thread some weeks ago we did not do this by replicating
(as perhaps
we
should have done). Instead we migrated the users across.

We have 30+ servers that are IPA clients ("Hosts" in ipa-speak)
joined
to
the old KDC. We are now in the process of migrating these hosts
to the
new
4.1 KDC.

Most of the hosts run EL 6.5 + ipa-client 3.3.3. For all of
these
joining
to the new KDC was trouble free, taking a few minutes each.
After
joining
the new KDC FreeIPA users authenticated properly.

We also had a small number of new EL 7.1 + ipa-client 4.1 hosts
that
were
joined direct to the new 4.1 KDC, never having been joined of
the 3.3.3
KDC. These were also trouble free.

The problem occurs with a handful of existing EL 7.1
+ipa-client 4.1
hosts
that were originally joined to the 3.3.3 KDC, and must be moved
to join
the
4.1 KDC. These machines no longer authenticate valid FreeIPA
users. I
have
been able to reproduce this behaviour with a freshly setup VM
joined
first
to the 3.3.3 KDC, then moved to the 4.1 KDC.

While the errors show in the krb5 child logs indicate that the
password
is
incorrect, the same user / password is happily accepted by all
the other
hosts.

It seems that in the process of moving / migrating the EL 7.1 /
ipa-client
4.1 from the old KDC to the new KDC, "something" is left behind
that
causes
problems. We have seen indications in the install logs that the
kinit
steps
called during ipa-client install are getting responses from the
wrong
(old)
KDC, and not from the new KDC.

Frustratingly. over the weekend i managed to get one of the
problem EL
7.1
boxes to work. However I can't work out exactly what I was that
I did
that
did the trick. However it seems that some kind of major
de-install /
cleanup + reinstall of the ipa-client may be needed.

Rob has suggested that as part of such a cleanup I should do
"rm
-f /var/lib/sssd/db/*". I will test this later today and report
back.

Thanks to Rob, Jakub, Martin, Alexander et al for their help
and
suggestions so far.

Chris

Thanks for the background. The pain you are getting is exactly
the reason
why
migration via replication to RHEL-7.1 is a better choice :-)
Please let
us
know
the result, I am curious how this works out.





From: Martin Kosek <
***@redhat.com>
To: Christopher
Lamb/Switzerland/***@IBMCH,
freeipa-***@redhat.com, Jakub Hrozek <
***@redhat.com>
Date: 03.06.2015 09:34
Subject: Re: [Freeipa-users]
Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved



On 06/02/2015 06:15 PM, Christopher Lamb wrote:

Hi

Earlier today I setup 2 throwaway EL7.1 VMs to help narrow
down the
cause
of this problem. Let's call them HOST09 and HOST10

Both are mimimum installs of EL7.1, with NTPD installed and
configured.

HOST09 had ipa-client 4.1 installed via yum, and was
configured to use
our
new FreeIPA 4.1 server, right from the start. --> My FreeIPA
user
authenticates successfully against this machine.

HOST10 had ipa-client 4.1 installed as a dependency of one of
our
standard
config packages, and was first set to use our old FreeIPA
3.3.3 server.
-->
My FreeIPA user authenticates successfully. against this
machine.

I then de-registered HOST10 from the FreeIPA 3.1 server, and
registered
against the new FreeIPA 4.1 server --> My FreeIPA users does
NOT
authenticate successfully.

This replicates well the behaviour I saw with my production
servers,
namely
a) EL 7.1 hosts with ipa-client 4.1 registered directly
against the new
4.1
FreeIPA server authenticate properly.

b) EL 7.1 hosts with ipa-client 4.1 first registered against
the old
3.3.3
FreeIPA server, then reregistered with the new 4.1 FreeIPA
server do
NOT
authenticate properly

Chris

Hello,

This is really strange. What I do not fully understand is what
is the
"registration against a FreeIPA server". What server you
install IPA
client
should matter if the deployment is set up properly. The host
enrollment
entry
should simply replicate to whole infrastructure. The only thing
that
will
probably differ is sssd.conf and krb5.conf as they will have
different
primary
server set up, based on what your DNS setup is.

It rather seems that the "reregistration" is what causes the
issue. It
looks
like something cleanup problem during the process. I will let
Jakub to
help
here, I would suggest including the SSSD logs from the failed
login, it
may
help.




----- Forwarded by Christopher Lamb/Switzerland/IBM on
02.06.2015 16:52
-----

From:
Christopher
Lamb/Switzerland/***@IBMCH
To:
Jakub Hrozek
<***@redhat.com>
Cc:
freeipa-***@redhat.com
Date:
02.06.2015 10:40
Subject:
Re:
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved
Sent by:
freeipa-users-***@redhat.com



Hi Jakub

Yes root login works, that's how I've been getting into the
box.

Surprisingly, kinit with my user seems to work on that box.
After
entering
my password when prompted, it returns to the commandline
without error.

However if I try kinit with another FreeIPA user, then
instead of
prompting
for a password, it gives "Generic preauthentication failure
while
getting
initial credentials" error.

Having set debug_level=10, when I try and ssh in with my
FreeIPA user,
I
find errors like

"Retrieving host .... with result: .. Matching credential not
found"

"Received error from KDC ... Additional pre-authentication
required"

"Received error from KDC... Decrypt integrity check failed"

"Received error code 1432158219"

Cheers

Chris





From:
Jakub Hrozek
<***@redhat.com>
To:
Christopher
Lamb/Switzerland/***@IBMCH
Cc:
freeipa-***@redhat.com
Date:
02.06.2015 09:50
Subject:
Re:
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved



On Tue, Jun 02, 2015 at 09:43:48AM +0200, Christopher Lamb
wrote:
Hi Jakub

The same user / password works with all our FreeIPA hosts -
just this
one
box is the problem. So the password should be good. Of
course a type
is
always possible (especially for strong passwords), but I
have tried
many
times which should eliminate the odd password typo. The
user /
password
should also be good for both the old and the new FreeIPA
Server.

Interesting, can you add debug_level=10 to the domain section
of
sssd.conf? Then krb5_child.log should show Kerberos tracing
info
including which exact KDC SSSD was talking to.


As I can neither log in direct, or via ssh to this box with
my FreeIPA
user, I assume Kinit with my user won't work- i will try
later in the
day.

Well, login as a UNIX user (root) should work..


My working assumption is that the problem is related in
some way to
the
fact the host originally was a FreeIPA 3.3.3 client,
updated to
FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently
setting up
2
throwaway EL 7.1 VMs to better test this. On one I will
first install
3.3.3, then upgrade to 4.1. The second will have a direct
install of
4.1
client.

Cheers

Chris



From:
Jakub Hrozek
<***@redhat.com>
To:
freeipa-***@redhat.com
Date:
02.06.2015 09:22
Subject:
Re:
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved
Sent by:
freeipa-users-***@redhat.com



On Mon, Jun 01, 2015 at 07:35:11PM +0200, Christopher Lamb
wrote:

Hi All

Bad news.

Over the weekend I was able to get the original problem
EL7.1 /
FreeIPA
4.1
host (FreeIPA client) to authenticate FreeiPA users (my
test being
ssh
remote login with FreeIPA user and password).

Today I tried a second machine, and had the same problem,
ssh
connections
with FreeIPA user cause "[sssd[krb5_child[3445]]]:
Decrypt integrity
check
failed"

This really just means wrong password, can you kinit as
that user
using
the same password?


Ahh I thought, I have a solution for that: just remove
ipa-client and
reinstall via yum, register with the new FreeIPA
server ....

Only with this second machine I still can't ssh in with a
FreeIPA
user.
Argg.....

b.t.w, as this machine is a real physical server, I was
able to try
logging
in direct with my FreeIPA user --> "Authentication
Failure"

I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that
migrated from the
old
FreeIPA server to the new without a hitch (i.e. they
successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to
migrate,
but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies
all
attempts
to
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever
registered with the
new
FreeIPA server, and successfully authenticates FreeIPA
users.

Any ideas?

Chris


----- Forwarded by Christopher Lamb/Switzerland/IBM on
01.06.2015
19:17
-----

From:

Christopher
Lamb/Switzerland/***@IBMCH
To:

Alexander Bokovoy
<***@redhat.com>,
freeipa-***@redhat.com
Date:

30.05.2015 18:52
Subject:

Re:
[Freeipa-users] ssh problem with
migrated FreeIPA
client on
EL7.1 --> Solved
Sent by:
freeipa-users-***@redhat.com



Hi All

It gives me pleasure to report the problem is solved - a
minute ago I
was
able to login via ssh with my FreeIPA user to the problem
server,
while
sitting on my terrace with a glass of wine!

Thanks to Alexander for his helpful advice - we had some
mail
exchange
outside the user list as I did not wish to broadcast
content of keys,
config files etc.

Regardless of what I did with commands like klist, kvno
everything
seemed
"ok", but I still could not ssh in. Even a ipa-getkeytab
did not
help.

Therefore I decided to opt for brute force and (partial)
ignorance. I
completely uninstalled the FreeIPA client, and then
reinstalled,
configured
- ét voilà I could ssh in!

This leaves the enigma: what caused the problem? I suspect
the
following:

The host is an EL 7.1, but the first FreeIPA client
installed was
version
3.3.3 (installed as set of standard packages that we bung
on all our
servers).

This worked fine to authenticate against our "old" 3.x
FreeIPA
server,
but
did not work against the "new" 4.1 FreeIPA Server.

When I realised I could not ssh in, one of the first
things I did was
to
yum update the FreeIPA client from 3.3.3 to 4.1 - but that
did not
help.
The solution was to yum remove the FreeIPA client, then
yum install
the
4.1
client.

I have some more EL 7.1 servers with the FreeIPA 3.3.3
client
installed,
so
it will be interesting to see it the problem can be
reproduced.

Keep up the good work,

Chris








From:

Alexander
Bokovoy
<***@redhat.com>
To:


Christopher
Lamb/Switzerland/***@IBMCH
Cc:
freeipa-***@redhat.com
Date:


29.05.2015 18:04
Subject:


Re:
[Freeipa-users] ssh problem with
migrated FreeIPA
client on
EL7.1



On Fri, 29 May 2015, Christopher Lamb wrote:

Hi All

Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1
server to
replace
the existing FreeIPA 3.0.0 running on OEL 6.5, and
successfully
migrated
across the users.

We have 50 odd Servers that are FreeIPA clients. Today I
started
migrating
these one-by-one from the old FreeIPA 3.x server to the
new FreeIPA
4
server by doing an ipa-client-install --uninstall from
the old, and
ipa-client-install to register with the new 4.1.0
server.

Most of the FreeIPA clients are running OEL 6.5, and for
these the
migration process above worked perfectly. After
migrating the
server,
I
could ssh in with my FreeIPA user.

Then I migrated an OEL 7.1 server. The migration itself
seemed to
work,
and
getent passwd was successful for my FreeIPA user.
However when I try
and
ssh in, my FreeIPA user / password is not accepted.

Before the migration I could ssh into the problem server
(though
evidently
it was using my FreeIPA user from the old FreeIPA
server).

I can ssh in with a local (non ldap) user, so ssh is
running and
working.
From user root I can successfully su to my FreeIPA
user.

Further investigation showed that version of ipa-client
installed
was
3.3.3, so I yum updated this to 4.1.0.

However I still cannot ssh into the OEL 7.1 box with my
FreeIPA
user.
The
same user continues to work for the 6.5 boxes.

A colleague tried to ssh in with his FreeIPA user, and
was also
rejected,
so the problem is not my user, but is probably for all
FreeIPA
users.

A failed ssh login attempt causes the following error
in /var/log/messages

[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system
and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy





--
Manage your subscription for the Freeipa-users mailing
list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project




--
Manage your subscription for the Freeipa-users mailing
list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

--
Manage your subscription for the Freeipa-users mailing
list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project









--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

















--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

--
/ Alexander Bokovoy

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project



FYI https://fedorahosted.org/freeipa/ticket/5050

--
Thank you,
Dmitri Pal

Director of Engineering for IdM portfolio
Red Hat, Inc.--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-02 16:44:07 UTC
Permalink
Hi

To narrow down the cause even further, I reverted HOST10 via VM snapshot
back to the state after installing linux and configuring ntpd.

This time I installed ipa-client 4.1 directly (rather then as a dependent
of our standard server packages). So this machine is a basic install of EL
7.1 + ntpd + ipa-client, with nothing else extra.

Again I first registered against the old 3.3.3 FreeIPA Server, then
switched to the new 4.1 Server.

Once again my FreeIPA user does not authenticate.

Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 18:38
-----

From: Christopher Lamb/Switzerland/***@IBMCH
To: freeipa-***@redhat.com, Jakub Hrozek <***@redhat.com>
Date: 02.06.2015 18:28
Subject: [Freeipa-users] Fw: ssh problem with migrated FreeIPA client on
EL7.1 -->Not Solved
Sent by: freeipa-users-***@redhat.com




Hi

Earlier today I setup 2 throwaway EL7.1 VMs to help narrow down the cause
of this problem. Let's call them HOST09 and HOST10

Both are mimimum installs of EL7.1, with NTPD installed and configured.

HOST09 had ipa-client 4.1 installed via yum, and was configured to use our
new FreeIPA 4.1 server, right from the start. --> My FreeIPA user
authenticates successfully against this machine.

HOST10 had ipa-client 4.1 installed as a dependency of one of our standard
config packages, and was first set to use our old FreeIPA 3.3.3 server. -->
My FreeIPA user authenticates successfully. against this machine.

I then de-registered HOST10 from the FreeIPA 3.1 server, and registered
against the new FreeIPA 4.1 server --> My FreeIPA users does NOT
authenticate successfully.

This replicates well the behaviour I saw with my production servers, namely
a) EL 7.1 hosts with ipa-client 4.1 registered directly against the new 4.1
FreeIPA server authenticate properly.

b) EL 7.1 hosts with ipa-client 4.1 first registered against the old 3.3.3
FreeIPA server, then reregistered with the new 4.1 FreeIPA server do NOT
authenticate properly

Chris



----- Forwarded by Christopher Lamb/Switzerland/IBM on 02.06.2015 16:52
-----

From: Christopher Lamb/Switzerland/***@IBMCH
To: Jakub Hrozek <***@redhat.com>
Cc: freeipa-***@redhat.com
Date: 02.06.2015 10:40
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated
FreeIPA
client on EL7.1 -->Not Solved
Sent by: freeipa-users-***@redhat.com



Hi Jakub

Yes root login works, that's how I've been getting into the box.

Surprisingly, kinit with my user seems to work on that box. After entering
my password when prompted, it returns to the commandline without error.

However if I try kinit with another FreeIPA user, then instead of prompting
for a password, it gives "Generic preauthentication failure while getting
initial credentials" error.

Having set debug_level=10, when I try and ssh in with my FreeIPA user, I
find errors like

"Retrieving host .... with result: .. Matching credential not found"

"Received error from KDC ... Additional pre-authentication required"

"Received error from KDC... Decrypt integrity check failed"

"Received error code 1432158219"

Cheers

Chris





From: Jakub Hrozek <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH
Cc: freeipa-***@redhat.com
Date: 02.06.2015 09:50
Subject: Re: [Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi Jakub
The same user / password works with all our FreeIPA hosts - just this one
box is the problem. So the password should be good. Of course a type is
always possible (especially for strong passwords), but I have tried many
times which should eliminate the odd password typo. The user / password
should also be good for both the old and the new FreeIPA Server.
Interesting, can you add debug_level=10 to the domain section of
sssd.conf? Then krb5_child.log should show Kerberos tracing info
including which exact KDC SSSD was talking to.
Post by Christopher Lamb
As I can neither log in direct, or via ssh to this box with my FreeIPA
user, I assume Kinit with my user won't work- i will try later in the
day.

Well, login as a UNIX user (root) should work..
Post by Christopher Lamb
My working assumption is that the problem is related in some way to the
fact the host originally was a FreeIPA 3.3.3 client, updated to FreeIPA
4.1, and switched between 2 FreeIPA servers. I am currently setting up 2
throwaway EL 7.1 VMs to better test this. On one I will first install
3.3.3, then upgrade to 4.1. The second will have a direct install of 4.1
client.
Cheers
Chris
From: Jakub Hrozek
Date: 02.06.2015 09:22
[Freeipa-users] Fw: ssh problem with
migrated
FreeIPA
Post by Christopher Lamb
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi All
Bad news.
Over the weekend I was able to get the original problem EL7.1 / FreeIPA
4.1
Post by Christopher Lamb
host (FreeIPA client) to authenticate FreeiPA users (my test being ssh
remote login with FreeIPA user and password).
Today I tried a second machine, and had the same problem, ssh
connections
Post by Christopher Lamb
Post by Christopher Lamb
with FreeIPA user cause "[sssd[krb5_child[3445]]]: Decrypt integrity
check
Post by Christopher Lamb
failed"
This really just means wrong password, can you kinit as that user using
the same password?
Post by Christopher Lamb
Ahh I thought, I have a solution for that: just remove ipa-client and
reinstall via yum, register with the new FreeIPA server ....
Only with this second machine I still can't ssh in with a FreeIPA user.
Argg.....
b.t.w, as this machine is a real physical server, I was able to try
logging
Post by Christopher Lamb
in direct with my FreeIPA user --> "Authentication Failure"
I now have
* a whole bunch of EL6.5 / FreeIPA 3.3.3 hosts that migrated from the
old
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server to the new without a hitch (i.e. they successfully
authenticate FreeIPA users.)
* one migrated EL7.1 / FreeIPA 4.1 host that I was able to migrate, but
with problems
* one migrated EL7.1 / FreeIPA 4.1 host that so far defies all attempts
to
Post by Christopher Lamb
authenticate with a FreeIPA user
* one EL7.1 / FreeIPA 4.1 host that was only ever registered with the
new
Post by Christopher Lamb
Post by Christopher Lamb
FreeIPA server, and successfully authenticates FreeIPA users.
Any ideas?
Chris
----- Forwarded by Christopher Lamb/Switzerland/IBM on 01.06.2015 19:17
-----
Christopher
Alexander Bokovoy
30.05.2015 18:52
Re:
[Freeipa-users] ssh problem with
migrated FreeIPA
Post by Christopher Lamb
client on
Post by Christopher Lamb
EL7.1 --> Solved
Hi All
It gives me pleasure to report the problem is solved - a minute ago I
was
Post by Christopher Lamb
Post by Christopher Lamb
able to login via ssh with my FreeIPA user to the problem server, while
sitting on my terrace with a glass of wine!
Thanks to Alexander for his helpful advice - we had some mail exchange
outside the user list as I did not wish to broadcast content of keys,
config files etc.
Regardless of what I did with commands like klist, kvno everything
seemed
Post by Christopher Lamb
Post by Christopher Lamb
"ok", but I still could not ssh in. Even a ipa-getkeytab did not help.
Therefore I decided to opt for brute force and (partial) ignorance. I
completely uninstalled the FreeIPA client, and then reinstalled,
configured
Post by Christopher Lamb
- ét voilà I could ssh in!
This leaves the enigma: what caused the problem? I suspect the
The host is an EL 7.1, but the first FreeIPA client installed was
version
Post by Christopher Lamb
Post by Christopher Lamb
3.3.3 (installed as set of standard packages that we bung on all our
servers).
This worked fine to authenticate against our "old" 3.x FreeIPA server,
but
Post by Christopher Lamb
did not work against the "new" 4.1 FreeIPA Server.
When I realised I could not ssh in, one of the first things I did was
to
Post by Christopher Lamb
Post by Christopher Lamb
yum update the FreeIPA client from 3.3.3 to 4.1 - but that did not
help.
Post by Christopher Lamb
Post by Christopher Lamb
The solution was to yum remove the FreeIPA client, then yum install the
4.1
Post by Christopher Lamb
client.
I have some more EL 7.1 servers with the FreeIPA 3.3.3 client
installed,
Post by Christopher Lamb
so
Post by Christopher Lamb
it will be interesting to see it the problem can be reproduced.
Keep up the good work,
Chris
Alexander Bokovoy
Christopher
29.05.2015 18:04
Re:
[Freeipa-users] ssh problem with
Post by Christopher Lamb
migrated FreeIPA
Post by Christopher Lamb
client on
EL7.1
Post by Christopher Lamb
Hi All
Some weeks ago I setup a new FreeIPA 4.1.0 on an OEL 7.1 server to
replace
Post by Christopher Lamb
Post by Christopher Lamb
the existing FreeIPA 3.0.0 running on OEL 6.5, and successfully
migrated
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
across the users.
We have 50 odd Servers that are FreeIPA clients. Today I started
migrating
Post by Christopher Lamb
Post by Christopher Lamb
these one-by-one from the old FreeIPA 3.x server to the new FreeIPA 4
server by doing an ipa-client-install --uninstall from the old, and
ipa-client-install to register with the new 4.1.0 server.
Most of the FreeIPA clients are running OEL 6.5, and for these the
migration process above worked perfectly. After migrating the server,
I
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
could ssh in with my FreeIPA user.
Then I migrated an OEL 7.1 server. The migration itself seemed to
work,
Post by Christopher Lamb
Post by Christopher Lamb
and
Post by Christopher Lamb
getent passwd was successful for my FreeIPA user. However when I try
and
Post by Christopher Lamb
Post by Christopher Lamb
Post by Christopher Lamb
ssh in, my FreeIPA user / password is not accepted.
Before the migration I could ssh into the problem server (though
evidently
Post by Christopher Lamb
Post by Christopher Lamb
it was using my FreeIPA user from the old FreeIPA server).
I can ssh in with a local (non ldap) user, so ssh is running and
working.
Post by Christopher Lamb
Post by Christopher Lamb
From user root I can successfully su to my FreeIPA user.
Further investigation showed that version of ipa-client installed was
3.3.3, so I yum updated this to 4.1.0.
However I still cannot ssh into the OEL 7.1 box with my FreeIPA user.
The
Post by Christopher Lamb
Post by Christopher Lamb
same user continues to work for the 6.5 boxes.
A colleague tried to ssh in with his FreeIPA user, and was also
rejected,
Post by Christopher Lamb
Post by Christopher Lamb
so the problem is not my user, but is probably for all FreeIPA users.
A failed ssh login attempt causes the following error
in /var/log/messages
Post by Christopher Lamb
Post by Christopher Lamb
[sssd[krb5_child[5393]]]: Decrypt integrity check failed
It means /etc/krb5.keytab contains keys from older system and SSSD
picks them up.
Can you show output of 'klist -kKet'?
--
/ Alexander Bokovoy
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project




--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Rob Crittenden
2015-06-02 17:25:03 UTC
Permalink
Post by Christopher Lamb
Hi
To narrow down the cause even further, I reverted HOST10 via VM snapshot
back to the state after installing linux and configuring ntpd.
This time I installed ipa-client 4.1 directly (rather then as a dependent
of our standard server packages). So this machine is a basic install of EL
7.1 + ntpd + ipa-client, with nothing else extra.
Again I first registered against the old 3.3.3 FreeIPA Server, then
switched to the new 4.1 Server.
Once again my FreeIPA user does not authenticate.
I'd start by simlifying things.

Does kinit -kt /etc/krb5.keytab work?

Do basic nss operations work?

getent passwd admin
id admin
groups admin
etc.

Seeing the entire ipaclient-install.log after the 7.1 install may be
helfpul.

Cranking up sssd debuglevel may be helpful.

rob
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Christopher Lamb
2015-06-02 18:04:35 UTC
Permalink
Hi Rob

Thanks

All those commands work, and give expected results.

I will send you the install logs direct.

Cheers

Chris




From: Rob Crittenden <***@redhat.com>
To: Christopher Lamb/Switzerland/***@IBMCH,
freeipa-***@redhat.com, Jakub Hrozek <***@redhat.com>
Date: 02.06.2015 19:25
Subject: Re: [Freeipa-users] Fw: ssh problem with migrated FreeIPA
client on EL7.1 -->Not Solved
Post by Christopher Lamb
Hi
To narrow down the cause even further, I reverted HOST10 via VM snapshot
back to the state after installing linux and configuring ntpd.
This time I installed ipa-client 4.1 directly (rather then as a dependent
of our standard server packages). So this machine is a basic install of EL
7.1 + ntpd + ipa-client, with nothing else extra.
Again I first registered against the old 3.3.3 FreeIPA Server, then
switched to the new 4.1 Server.
Once again my FreeIPA user does not authenticate.
I'd start by simlifying things.

Does kinit -kt /etc/krb5.keytab work?

Do basic nss operations work?

getent passwd admin
id admin
groups admin
etc.

Seeing the entire ipaclient-install.log after the 7.1 install may be
helfpul.

Cranking up sssd debuglevel may be helpful.

rob
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Loading...