[Exist-open] Beginner question on db restore

Discussion:

Robert Han

2011-10-28 19:17:15 UTC

Hi all,

Pardon for the beginner question. I recently did a restore from a back-up of
an eXist from another PC user. After restoring, my existing username and
password won't work, do I need to use the ID/Pasword from that instance of
eXist?

Thanks!

Rob

Jens Østergaard Petersen

2011-11-07 09:02:41 UTC

Permalink

Hi Rob,

This is not really a straight answer, since I have questions about this as well, but here is my take on your question.

When you do a restore, importing a zip file using the java admin client, what is inside the backup will overwrite what is in identical positions on the live system. What is not in identical positions will be left as it is.

If you have a full backup of /db, this backup will include /db/system/users.xml and /db/security/exist/accounts. If you restore such a backup, the information on users and passwords on your live system will be overwritten by the information on users and passwords on the system backed up.

If you are restoring another user's backup, you will then have to authenticate with the other user's credentials when logging in. - I was myself trapped by this, since I unthinkingly imported the backup posted to the list by Alem Areki on the day of your posting - and was locked out of my database as a result (but I had a full backup of my installation, so everything is cool).

On <http://exist-db.org/backup.html#d1864e846>, the restore process is described in (what I consider) a roundabout and unclear way:

"Restoring from a backup (or parts of it) does not mean that the existing data in the current database instance will be deleted entirely. The restore process will upload the collections and documents contained in the backup. Collections and documents which exist in the database but are not part of the backup will not be modified."

What I _believe_ you should do (but don't take my word for it!) is to unzip the backup, remove /db/system/users.xml and /db/security/. You should then remove the information relating to the document "users.xml" and the "security" collection from the /db/system/__contents__.xml file. Finally, you should zip the db directory again and do the restore. You can then log in with your own credentials.

You don't have to zip; if you don't, follow the instructions on <http://exist-db.org/backup.html#d1864e900>, choosing "__content.xml__ files" in the File Format drop-down menu and selecting the uppermost __content.xml__ file.

The last thing I want to do is to give security-related advice, so until someone more knowledgable OKs this, you should not do anything!

Best,

Jens

Post by Robert Han
Hi all,
Pardon for the beginner question. I recently did a restore from a back-up of an eXist from another PC user. After restoring, my existing username and password won't work, do I need to use the ID/Pasword from that instance of eXist?
Thanks!
Rob
------------------------------------------------------------------------------
The demand for IT networking professionals continues to grow, and the
demand for specialized networking skills is growing even more rapidly.
about Cisco certifications, training, and career opportunities.
http://p.sf.net/sfu/cisco-dev2dev_______________________________________________
Exist-open mailing list
https://lists.sourceforge.net/lists/listinfo/exist-open

Jens Østergaard Petersen

2011-11-18 09:48:51 UTC

Permalink

Hi,

I am reposting this from more than a week ago. Could someone tell me if this is an acceptable way to restore a database dump you, for instance, do not know the admin password for?

Cheers,

Jens

Subject: Re: [Exist-open] Beginner question on db restore

Hi Rob,

This is not really a straight answer, since I have questions about this as well, but here is my take on your question.

When you do a restore, importing a zip file using the java admin client, what is inside the backup will overwrite what is in identical positions on the live system. What is not in identical positions will be left as it is.

If you have a full backup of /db, this backup will include /db/system/users.xml and /db/security/exist/accounts. If you restore such a backup, the information on users and passwords on your live system will be overwritten by the information on users and passwords on the system backed up.

If you are restoring another user's backup, you will then have to authenticate with the other user's credentials when logging in. - I was myself trapped by this, since I unthinkingly imported the backup posted to the list by Alem Areki on the day of your posting - and was locked out of my database as a result (but I had a full backup of my installation, so everything is cool).

On <http://exist-db.org/backup.html#d1864e846>, the restore process is described in (what I consider) a roundabout and unclear way:

"Restoring from a backup (or parts of it) does not mean that the existing data in the current database instance will be deleted entirely. The restore process will upload the collections and documents contained in the backup. Collections and documents which exist in the database but are not part of the backup will not be modified."

What I _believe_ you should do (but don't take my word for it!) is to unzip the backup, remove /db/system/users.xml and /db/security/. You should then remove the information relating to the document "users.xml" and the "security" collection from the /db/system/__contents__.xml file. Finally, you should zip the db directory again and do the restore. You can then log in with your own credentials.

You don't have to zip; if you don't, follow the instructions on <http://exist-db.org/backup.html#d1864e900>, choosing "__content.xml__ files" in the File Format drop-down menu and selecting the uppermost __content.xml__ file.

The last thing I want to do is to give security-related advice, so until someone more knowledgable OKs this, you should not do anything!

Best,

Jens

Hungerburg

2011-11-18 10:54:32 UTC

Permalink

[...]

Looks practicable from my /users/ point of view. Maybe you could also
just replace the admin passwd in the backup with a known one or none at
all, so any referenced accounts in the backup can be resolved later.
Depending on how it is crypted this might work. Or even simpler:

Does restoring such a backup prompt you to relogin the webstart admin
client, when the password has changed? If the running admin session
remains valid, you could even just do the backup and set the admin
password as an immediate next step.

--
peter

Jens Østergaard Petersen

2011-11-21 09:23:52 UTC

Permalink

Hi Peter,

Thank you for your thoughts.

Post by Hungerburg

Well, the info looks like this in 1.5.0:

<account xmlns="http://exist-db.org/Configuration" id="1048574"><password>{MD5}WhBei51A4TKXgNYuoiZdig==</password><digestPassword>test1</digestPassword>

It is true, as you say, that if I copy the corresponding string from another admin password, I can change passwords in the database I restore. However, deleting the files strikes me as safer.

What I find a little worrying is that the password appears in the clear in 1.5.0: in 1.4.1, all passwords are encrypted.

In 1.4.1, the same password ("test1") is rendered as

<user name="admin" uid="1" password="{MD5}WhBei51A4TKXgNYuoiZdig==" digest-password="4596428fed747447731bdd7722a08031" home="">

The 1.4.1 behaviour is obviously the correct one - that is, after all, what "digest password" means ….

I guess there is a reason why the password has to be "digested" twice …

Post by Hungerburg
Does restoring such a backup prompt you to relogin the webstart admin
client, when the password has changed? If the running admin session
remains valid, you could even just do the backup and set the admin
password as an immediate next step.

Yes, if I know the password of the installation I am restoring, what you say is correct. I log in with the base installation's admin password and enter the backup's password prior to restore. At this point, if I logged in again, I would have to enter the admin password used for the backup, but if I do not, I can e.g. change the password of the installation to something else. This appears to be OK, security-wise.

However, if I give the wrong password, that is, a password that is not the password of the backup, the process aborts - but not before it has overwritten the security files. This means that if I don't know this password, I am out of luck: my database becomes inaccessible.

If I try to do a restore when logged into the admin client as a user without admin status, the process aborts before anything is overwritten, so here everything is cool.

Jens

Post by Hungerburg
--
peter
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Exist-open mailing list
https://lists.sourceforge.net/lists/listinfo/exist-open

Martin Holmes

2011-11-21 21:46:35 UTC

Permalink

This issue of the password being in clear text bothers me greatly; it
means I have to take great care about where I store my backups, which
wasn't something I'd ever worried about before.

Is there a bug report in for this? If not, should we submit one? It
seems urgent to me.

Cheers,
Martin

Post by Jens Ãstergaard Petersen
Hi Peter,
Thank you for your thoughts.

Post by Hungerburg

<account xmlns="http://exist-db.org/Configuration" id="1048574"><password>{MD5}WhBei51A4TKXgNYuoiZdig==</password><digestPassword>test1</digestPassword>
It is true, as you say, that if I copy the corresponding string from another admin password, I can change passwords in the database I restore. However, deleting the files strikes me as safer.
What I find a little worrying is that the password appears in the clear in 1.5.0: in 1.4.1, all passwords are encrypted.
In 1.4.1, the same password ("test1") is rendered as
<user name="admin" uid="1" password="{MD5}WhBei51A4TKXgNYuoiZdig==" digest-password="4596428fed747447731bdd7722a08031" home="">
The 1.4.1 behaviour is obviously the correct one - that is, after all, what "digest password" means ….
I guess there is a reason why the password has to be "digested" twice …

Yes, if I know the password of the installation I am restoring, what you say is correct. I log in with the base installation's admin password and enter the backup's password prior to restore. At this point, if I logged in again, I would have to enter the admin password used for the backup, but if I do not, I can e.g. change the password of the installation to something else. This appears to be OK, security-wise.
However, if I give the wrong password, that is, a password that is not the password of the backup, the process aborts - but not before it has overwritten the security files. This means that if I don't know this password, I am out of luck: my database becomes inaccessible.
If I try to do a restore when logged into the admin client as a user without admin status, the process aborts before anything is overwritten, so here everything is cool.
Jens

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d

Wolfgang Meier

2011-11-21 22:31:02 UTC

Permalink

Post by Martin Holmes
This issue of the password being in clear text bothers me greatly; it
means I have to take great care about where I store my backups, which
wasn't something I'd ever worried about before.

eXist only keeps and compares the digested password internally. The
clear text password should not be known to eXist, so I'm really
surprised it ends up in the files now. Dmitryi, any idea where this is
coming from?

Wolfgang

Dmitriy Shabanov

2011-11-22 04:38:12 UTC

Permalink

Post by Wolfgang Meier

Most surprising is 'digestPassword', it shouldn't be there at all. What is
the steps to reproduce it?

Note: to break MD5 I need few ms, oops .... back should be in safe/secure
place any way!!!

I do waiting Adam to finish with redesign to submit switch to SHA as
minimum (make it configurable).

--
Dmitriy Shabanov

Wolfgang Meier

2011-11-22 09:48:37 UTC

Permalink

Post by Dmitriy Shabanov
Most surprising is 'digestPassword', it shouldn't be there at all. What is
the steps to reproduce it?

Set a password for admin and create a backup without restarting eXist.
If you restart eXist, digestPassword disappears again.

Post by Dmitriy Shabanov
Note: to break MD5 I need few ms, oops .... back should be in safe/secure
place any way!!!

I originally introduced the MD5 digest to support WebDAV. But if it
doesn't cause any issues, replacing it with SHA on new installs would
be great. But please remember that users still need to be able to
restore a 3 years old backup ;-)

Wolfgang

Dmitriy Shabanov

2011-11-22 12:38:04 UTC

Permalink

Post by Wolfgang Meier

Post by Dmitriy Shabanov
Most surprising is 'digestPassword', it shouldn't be there at all. What

Post by Dmitriy Shabanov
the steps to reproduce it?

Set a password for admin and create a backup without restarting eXist.
If you restart eXist, digestPassword disappears again.

Ok, I'll check.

Post by Wolfgang Meier

Post by Dmitriy Shabanov
Note: to break MD5 I need few ms, oops .... back should be in safe/secure
place any way!!!

Yes, that's why {MD5} prefix there. That information can be used to choose
algorithm.

--
Dmitriy Shabanov

Hungerburg

2011-11-22 13:26:57 UTC

Permalink

Post by Dmitriy Shabanov

Post by Wolfgang Meier
Set a password for admin and create a backup without restarting eXist.
If you restart eXist, digestPassword disappears again.

Ok, I'll check.

For me this would not make the plaintext password vanish from admin.xml
in /etc/... though I did set the password to the same one and not a
different one, so maybe...

--
peter

Martin Holmes

2011-11-21 21:46:35 UTC

Permalink

Post by Jens Ãstergaard Petersen
Hi Peter,
Thank you for your thoughts.

Post by Hungerburg

Yes, if I know the password of the installation I am restoring, what you say is correct. I log in with the base installation's admin password and enter the backup's password prior to restore. At this point, if I logged in again, I would have to enter the admin password used for the backup, but if I do not, I can e.g. change the password of the installation to something else. This appears to be OK, security-wise.
However, if I give the wrong password, that is, a password that is not the password of the backup, the process aborts - but not before it has overwritten the security files. This means that if I don't know this password, I am out of luck: my database becomes inaccessible.
If I try to do a restore when logged into the admin client as a user without admin status, the process aborts before anything is overwritten, so here everything is cool.
Jens

Dmitriy Shabanov

2011-11-22 04:43:57 UTC

Permalink

Post by Jens Ãstergaard Petersen
<account xmlns="http://exist-db.org/Configuration"
id="1048574"><password>{MD5}WhBei51A4TKXgNYuoiZdig==</password><digestPassword>test1</digestPassword>

What is value of 'password-encoding' at conf.xml?

--
Dmitriy Shabanov

Jens Østergaard Petersen

2011-11-22 07:50:03 UTC

Permalink

Post by Jens Ãstergaard Petersen
<account xmlns="http://exist-db.org/Configuration" id="1048574"><password>{MD5}WhBei51A4TKXgNYuoiZdig==</password><digestPassword>test1</digestPassword>
What is value of 'password-encoding' at conf.xml?

EXIST_HOME/conf.xml has nothing on passwords or encoding (<http://exist.svn.sourceforge.net/viewvc/exist/trunk/eXist/conf.xml.tmpl?revision=15470&pathrev=15540>).

In EXIST_HOME/webapp/configuration.xml, line 1170, 'password-encoding' occurs; this information once occurred in EXIST_HOME/conf.xml - you moved it with revision 14136.

Otherwise 'password-encoding' occurs only in EXIST_HOME/src/org/exist/util/Configuration.java. A change were made on April 19 which looks like it could have an effect on password encoding: <http://exist.svn.sourceforge.net/viewvc/exist/trunk/eXist/src/org/exist/util/Configuration.java?r1=13941&r2=14245&pathrev=14245>.

I made a backup of /db/system on April 30 (where I may not have been up-to-date) and here the <digestPassword> is in MD5 digest form. It is identical to <password>.

Jens

Post by Jens Ãstergaard Petersen
--
Dmitriy Shabanov

Adam Retter

2011-11-21 13:45:38 UTC

Permalink

Jens,

You approach seems acceptable to me.

I think there is a larger question here - Should restoring a backup
overwrite the current admin password, or in fact any password of a
user that exists in both the db and the backup to be restored?

My feeling is that the default behaviour should NOT overwrite the
password of any existing users. This could of course be made
configurable.

Post by Jens Ãstergaard Petersen
Hi,
I am reposting this from more than a week ago. Could someone tell me if this
is an acceptable way to restore a database dump you, for instance, do not
know the admin password for?
Cheers,
Jens
Subject: Re: [Exist-open] Beginner question on db restore
Hi Rob,
This is not really a straight answer, since I have questions about this as
well, but here is my take on your question.
When you do a restore, importing a zip file using the java admin client,
what is inside the backup will overwrite what is in identical positions on
the live system. What is not in identical positions will be left as it is.
If you have a full backup of /db, this backup will include
/db/system/users.xml and /db/security/exist/accounts. If you restore such a
backup, the information on users and passwords on your live system will be
overwritten by the information on users and passwords on the system backed
up.
If you are restoring another user's backup, you will then have to
authenticate with the other user's credentials when logging in. - I was
myself trapped by this, since I unthinkingly imported the backup posted to
the list by Alem Areki on the day of your posting - and was locked out of my
database as a result (but I had a full backup of my installation, so
everything is cool).
On <http://exist-db.org/backup.html#d1864e846>, the restore process is
"Restoring from a backup (or parts of it) does not mean that the existing
data in the current database instance will be deleted entirely. The restore
process will upload the collections and documents contained in the backup.
Collections and documents which exist in the database but are not part of
the backup will not be modified."
What I _believe_ you should do (but don't take my word for it!) is to unzip
the backup, remove /db/system/users.xml and /db/security/. You should then
remove the information relating to the document "users.xml" and the
"security" collection from the /db/system/__contents__.xml file. Finally,
you should zip the db directory again and do the restore. You can then log
in with your own credentials.
You don't have to zip; if you don't, follow the instructions on
<http://exist-db.org/backup.html#d1864e900>, choosing "__content.xml__
files" in the File Format drop-down menu and selecting the uppermost
__content.xml__ file.
The last thing I want to do is to give security-related advice, so until
someone more knowledgable OKs this, you should not do anything!
Best,
Jens

Post by Robert Han
Hi all,
Pardon for the beginner question. I recently did a restore from a back-up
of an eXist from another PC user. After restoring, my existing username
and password won't work, do I need to use the ID/Pasword from that
instance of eXist?
Thanks!
Rob
------------------------------------------------------------------------------
The demand for IT networking professionals continues to grow, and the
demand for specialized networking skills is growing even more rapidly.
about Cisco certifications, training, and career opportunities.
http://p.sf.net/sfu/cisco-dev2dev_______________________________________________
Exist-open mailing list
https://lists.sourceforge.net/lists/listinfo/exist-open

--
Adam Retter

eXist Developer
{ United Kingdom }
***@exist-db.org
irc://irc.freenode.net/existdb