[Valgrind-developers] Migrating Valgrind sources from SVN to GIT

[Ivo Raisr]

Dear Valgrind community,

We are pleased to announce an imminent migration of Valgrind sources
from existing Subversion SCM to modern git SCM, as discussed during
our FOSDEM 2017 Valgrind devroom.

What is going on now?
~~~~~~~~~~~~~~~~~
The migration has just started. We are now in beta testing stage.
We still use the official SVN Valgrind repository for our work until
the final migration step. If you have some patches ready now, send
them for review.
You can contribute to the migration process - read below.

What will be migrated:
~~~~~~~~~~~~~~~~~
Valgrind and VEX sources. Precisely sources available today under
svn://svn.valgrind.org/valgrind and svn://svn.valgrind.org/vex, including
all production release branches and tags. Valgrind and VEX repos will
be merged into one, so no more SVN externals.

Where I will find the new repo:
~~~~~~~~~~~~~~~~~~~~~~~
At sourceware.org. Precisely at:
git://sourceware.org/git/valgrind.git/
http://sourceware.org/git/valgrind.git/
Right now a snapshot of SVN sources as of 2017-02-21 is available for
you to test.

How the test migration was performed:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
See recipes at https://github.com/ivosh/valgrind-git-migration

What is the plan for the migration to go forward:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Test migration has been performed and initial tests were successful.
2. The test repo is now available to test and play for others with - see below
for details.
3. Prepare www (website) and nightly build script changes and have
them reviewed.
4. Proceed once 2+3 are successfully done.
5. Announce the final migration.
6. Completely eradicate contents in the GIT repository so the migration
can start from scratch.
7. Switch SVN valgrind+vex repo readonly.
8. Perform the final migration to sourceware.org.
9. Enable email notifications from new git repo.
10. Push www and nightly script changes to the new repo.

What will not be migrated:
~~~~~~~~~~~~~~~~~~~~
- Valgrind www (website) repo. Not now, but later.
- Non production release branches and tags from old SVN Valgrind+VEX repos.
If you need to preserve some other branches or tags, let us know:
https://sourceware.org/git/?p=valgrind.git;a=heads
https://sourceware.org/git/?p=valgrind.git;a=tags

I have a write access to existing SVN repo. What shall I do for the new one?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please contact Julian Seward. He will point you to specific instructions.

What will be my simple workflow in new git SCM?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Not much will be changed from the way we worked in SVN.
We still prepare patches, send them for review, have someone
with write access to push them. A minimalistic workflow would be:

git clone git://sourceware.org/git/valgrind.git/ valgrind
edit/compile
git status/add/show
git pull origin/master
build + test
git commit
[git push - if you have write access]

There are a lot of good tutorials on simple git workflows, so please
have a look. If you are using something more complicated, please
share with us and ideally send us a write up.

I would like to help with the migration.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes, please! Send us your positive and negative feedback.
For example:
- It worked for me!
- This and that did not work for me...
- How do I do such and such thing now?

The test repository is there for you to play with. The contents
will be deleted before the final migration so no reason to worry about
potential mistakes. It is also quite likely that the contents will be
regenerated during the beta testing, to fix any problems found.

We also need a help documenting possible workflows. Especially when
preparing a release - we need to test and document how to work with branches
and releases.

[Austin English]

Post by Ivo Raisr
Dear Valgrind community,
We are pleased to announce an imminent migration of Valgrind sources
from existing Subversion SCM to modern git SCM, as discussed during
our FOSDEM 2017 Valgrind devroom.
What is going on now?
~~~~~~~~~~~~~~~~~
The migration has just started. We are now in beta testing stage.
We still use the official SVN Valgrind repository for our work until
the final migration step. If you have some patches ready now, send
them for review.
You can contribute to the migration process - read below.
~~~~~~~~~~~~~~~~~
Valgrind and VEX sources. Precisely sources available today under
svn://svn.valgrind.org/valgrind and svn://svn.valgrind.org/vex, including
all production release branches and tags. Valgrind and VEX repos will
be merged into one, so no more SVN externals.
~~~~~~~~~~~~~~~~~~~~~~~
git://sourceware.org/git/valgrind.git/
http://sourceware.org/git/valgrind.git/
Right now a snapshot of SVN sources as of 2017-02-21 is available for
you to test.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
See recipes at https://github.com/ivosh/valgrind-git-migration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Test migration has been performed and initial tests were successful.
2. The test repo is now available to test and play for others with - see below
for details.
3. Prepare www (website) and nightly build script changes and have
them reviewed.
4. Proceed once 2+3 are successfully done.
5. Announce the final migration.
6. Completely eradicate contents in the GIT repository so the migration
can start from scratch.
7. Switch SVN valgrind+vex repo readonly.
8. Perform the final migration to sourceware.org.
9. Enable email notifications from new git repo.
10. Push www and nightly script changes to the new repo.
~~~~~~~~~~~~~~~~~~~~
- Valgrind www (website) repo. Not now, but later.
- Non production release branches and tags from old SVN Valgrind+VEX repos.
https://sourceware.org/git/?p=valgrind.git;a=heads
https://sourceware.org/git/?p=valgrind.git;a=tags
I have a write access to existing SVN repo. What shall I do for the new one?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please contact Julian Seward. He will point you to specific instructions.
What will be my simple workflow in new git SCM?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Not much will be changed from the way we worked in SVN.
We still prepare patches, send them for review, have someone
git clone git://sourceware.org/git/valgrind.git/ valgrind
edit/compile
git status/add/show
git pull origin/master
build + test
git commit
[git push - if you have write access]
There are a lot of good tutorials on simple git workflows, so please
have a look. If you are using something more complicated, please
share with us and ideally send us a write up.
I would like to help with the migration.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes, please! Send us your positive and negative feedback.
- It worked for me!
- This and that did not work for me...
- How do I do such and such thing now?
The test repository is there for you to play with. The contents
will be deleted before the final migration so no reason to worry about
potential mistakes. It is also quite likely that the contents will be
regenerated during the beta testing, to fix any problems found.
We also need a help documenting possible workflows. Especially when
preparing a release - we need to test and document how to work with branches
and releases.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Valgrind-developers mailing list
https://lists.sourceforge.net/lists/listinfo/valgrind-developers

Hi Ivo,

I'm very excited for the git move! I tested for
https://bugs.kde.org/show_bug.cgi?id=352395, which I rely upon to keep
track of precisely which version of Valgrind I'm testing with
(especially useful for historical logs).

With git, this is broken:
***@austin2:/tmp/valgrind$ ./vg-in-place -v --version
valgrind-3.13.0.SVN-unknown-vex-unknown

That said, checking out / compiling went great! ;)

[Ivo Raisr]

Post by Austin English
Hi Ivo,
I'm very excited for the git move! I tested for
https://bugs.kde.org/show_bug.cgi?id=352395, which I rely upon to keep
track of precisely which version of Valgrind I'm testing with
(especially useful for historical logs).
valgrind-3.13.0.SVN-unknown-vex-unknown
That said, checking out / compiling went great! ;)

Hi Austin,

Thank you for your feedback!
Indeed, svn related commands must be replaced with git ones.
Can you propose a patch? I assume instead of SVN revisions
we need to get git commit ids.
I will have a look at the other Valgrind parts.

I.

[Austin English]

Post by Austin English
Hi Ivo,
I'm very excited for the git move! I tested for
https://bugs.kde.org/show_bug.cgi?id=352395, which I rely upon to keep
track of precisely which version of Valgrind I'm testing with
(especially useful for historical logs).
valgrind-3.13.0.SVN-unknown-vex-unknown
That said, checking out / compiling went great! ;)

Hi Austin,

Thank you for your feedback!
Indeed, svn related commands must be replaced with git ones.
Can you propose a patch? I assume instead of SVN revisions
we need to get git commit ids.
I will have a look at the other Valgrind parts.

I.


Yeah, I'll look this week probably.

Had I known a git migration was imminent, I would've added native git
support to begin with ;).

[Austin English]

Looking at this in more depth, should I make a patch that simply
converts from SVN to git (simpler), or supports both and then will
require removing the SVN bits later (more complex, and feels like
wasted effort IMO)?

On Sat, Feb 25, 2017 at 11:04 PM, Austin English

Post by Ivo Raisr

Post by Austin English
Hi Ivo,
I'm very excited for the git move! I tested for
https://bugs.kde.org/show_bug.cgi?id=352395, which I rely upon to keep
track of precisely which version of Valgrind I'm testing with
(especially useful for historical logs).
valgrind-3.13.0.SVN-unknown-vex-unknown
That said, checking out / compiling went great! ;)

Hi Austin,
Thank you for your feedback!
Indeed, svn related commands must be replaced with git ones.
Can you propose a patch? I assume instead of SVN revisions
we need to get git commit ids.
I will have a look at the other Valgrind parts.
I.
Yeah, I'll look this week probably.
Had I known a git migration was imminent, I would've added native git
support to begin with ;).

[Austin English]

Post by Austin English
Looking at this in more depth, should I make a patch that simply
converts from SVN to git (simpler), or supports both and then will
require removing the SVN bits later (more complex, and feels like
wasted effort IMO)?
On Sat, Feb 25, 2017 at 11:04 PM, Austin English

Post by Ivo Raisr

Post by Austin English
Hi Ivo,
I'm very excited for the git move! I tested for
https://bugs.kde.org/show_bug.cgi?id=352395, which I rely upon to keep
track of precisely which version of Valgrind I'm testing with
(especially useful for historical logs).
valgrind-3.13.0.SVN-unknown-vex-unknown
That said, checking out / compiling went great! ;)

Hi Austin,
Thank you for your feedback!
Indeed, svn related commands must be replaced with git ones.
Can you propose a patch? I assume instead of SVN revisions
we need to get git commit ids.
I will have a look at the other Valgrind parts.
I.
Yeah, I'll look this week probably.
Had I known a git migration was imminent, I would've added native git
support to begin with ;).

--
-Austin
GPG: 14FB D7EA A041 937B

Sorry for spamming..

Assuming leaving git broken until the migration is okay (which seems
fine, given that it currently is and I'm decently sure I'm the only
person using this ;) ), here's a patch that works for a pure git
repository.

Please do not apply until the git migration is complete.

[Ivo Raisr]

Post by Austin English
Assuming leaving git broken until the migration is okay (which seems
fine, given that it currently is and I'm decently sure I'm the only
person using this ;) ), here's a patch that works for a pure git
repository.
Please do not apply until the git migration is complete.

Thank you for the patch.
I've added it to the migration recipe:
https://github.com/ivosh/valgrind-git-migration/commit/69a007175fa0c2186a498a8d55e6fb0319e35456
and pushed it also to the test repo at sourceware.org.

I.

[Matthias Schwarzott]

Post by Austin English

Post by Austin English
Looking at this in more depth, should I make a patch that simply
converts from SVN to git (simpler), or supports both and then will
require removing the SVN bits later (more complex, and feels like
wasted effort IMO)?
On Sat, Feb 25, 2017 at 11:04 PM, Austin English

Post by Ivo Raisr

Post by Austin English
Hi Ivo,
I'm very excited for the git move! I tested for
https://bugs.kde.org/show_bug.cgi?id=352395, which I rely upon to keep
track of precisely which version of Valgrind I'm testing with
(especially useful for historical logs).
valgrind-3.13.0.SVN-unknown-vex-unknown
That said, checking out / compiling went great! ;)

Hi Austin,
Thank you for your feedback!
Indeed, svn related commands must be replaced with git ones.
Can you propose a patch? I assume instead of SVN revisions
we need to get git commit ids.
I will have a look at the other Valgrind parts.
I.
Yeah, I'll look this week probably.
Had I known a git migration was imminent, I would've added native git
support to begin with ;).

--
-Austin
GPG: 14FB D7EA A041 937B

Sorry for spamming..
Assuming leaving git broken until the migration is okay (which seems
fine, given that it currently is and I'm decently sure I'm the only
person using this ;) ), here's a patch that works for a pure git
repository.
Please do not apply until the git migration is complete.

I wonder if the parameter 10 in --abbrev=10 should be skipped, now that
git has an automatic estimation for the number of digits to be used.

Regards
Matthias

[Ivo Raisr]

Post by Matthias Schwarzott
I wonder if the parameter 10 in --abbrev=10 should be skipped, now that
git has an automatic estimation for the number of digits to be used.

Are you referring to "--short=10" command line option for "git rev-parse"?
The manual [1] states that if no length is specified than 7 is used.
Could you give us a reference where the automatic length estimation is
described?
I.

[1] https://git-scm.com/docs/git-rev-parse/

[Matthias Schwarzott]

Post by Ivo Raisr

Post by Matthias Schwarzott
I wonder if the parameter 10 in --abbrev=10 should be skipped, now that
git has an automatic estimation for the number of digits to be used.

Are you referring to "--short=10" command line option for "git rev-parse"?

Exactly.

Post by Ivo Raisr
The manual [1] states that if no length is specified than 7 is used.
Could you give us a reference where the automatic length estimation is
described?
I.
[1] https://git-scm.com/docs/git-rev-parse/

Documentation can be seen here (only for the config value):
https://git-scm.com/docs/git-config#git-config-coreabbrev

I did not find more documentation.

The new behaviour was added to git with this merge commit:
https://github.com/git/git/commit/d7ae013a3173c621a3556be6834d459ece60e130

A description can be found here:
https://github.com/git/git/commit/e6c587c733b4634030b353f4024794b08bc86892

I think the new behaviour it is contained since at least git v2.11

Regards
Matthias

[Ivo Raisr]

Post by Ivo Raisr
Indeed, svn related commands must be replaced with git ones.
Can you propose a patch? I assume instead of SVN revisions
we need to get git commit ids.
I will have a look at the other Valgrind parts.
Yeah, I'll look this week probably.
Had I known a git migration was imminent, I would've added native git
support to begin with ;).

The SVN->GIT migration was decided at FOSDEM this month:
https://fosdem.org/2017/schedule/track/valgrind/
https://fosdem.org/2017/schedule/event/valgrind_hackaton/ [see
video recording]

I had no idea before that it will happen...
I.

[Christian Borntraeger]

Post by Ivo Raisr
Dear Valgrind community,
We are pleased to announce an imminent migration of Valgrind sources
from existing Subversion SCM to modern git SCM, as discussed during
our FOSDEM 2017 Valgrind devroom.

Very nice, thank you.

Post by Ivo Raisr
What is going on now?
~~~~~~~~~~~~~~~~~
The migration has just started. We are now in beta testing stage.
We still use the official SVN Valgrind repository for our work until
the final migration step. If you have some patches ready now, send
them for review.
You can contribute to the migration process - read below.
~~~~~~~~~~~~~~~~~
Valgrind and VEX sources. Precisely sources available today under
svn://svn.valgrind.org/valgrind and svn://svn.valgrind.org/vex, including
all production release branches and tags. Valgrind and VEX repos will
be merged into one, so no more SVN externals.
~~~~~~~~~~~~~~~~~~~~~~~
git://sourceware.org/git/valgrind.git/
http://sourceware.org/git/valgrind.git/
Right now a snapshot of SVN sources as of 2017-02-21 is available for
you to test.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
See recipes at https://github.com/ivosh/valgrind-git-migration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Test migration has been performed and initial tests were successful.
2. The test repo is now available to test and play for others with - see below
for details.
3. Prepare www (website) and nightly build script changes and have
them reviewed.
4. Proceed once 2+3 are successfully done.
5. Announce the final migration.
6. Completely eradicate contents in the GIT repository so the migration
can start from scratch.
7. Switch SVN valgrind+vex repo readonly.
8. Perform the final migration to sourceware.org.
9. Enable email notifications from new git repo.
10. Push www and nightly script changes to the new repo.
~~~~~~~~~~~~~~~~~~~~
- Valgrind www (website) repo. Not now, but later.
- Non production release branches and tags from old SVN Valgrind+VEX repos.
https://sourceware.org/git/?p=valgrind.git;a=heads
https://sourceware.org/git/?p=valgrind.git;a=tags
I have a write access to existing SVN repo. What shall I do for the new one?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please contact Julian Seward. He will point you to specific instructions.
What will be my simple workflow in new git SCM?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Not much will be changed from the way we worked in SVN.
We still prepare patches, send them for review, have someone
git clone git://sourceware.org/git/valgrind.git/ valgrind
edit/compile
git status/add/show
git pull origin/master

For a pull like approach I would suggest to either
a: do not this step (Linux style git handling)
b: use git pull --rebase (flat history style)

Post by Ivo Raisr
build + test
git commit
[git push - if you have write access]

If we give write accesses, then I suggest to use the --rebase variant.
Not sure what git server is running, but git-o-lite for example allows
to reject non-fast-forward merges, so that we do not fill the
history with single patch merge commits.
Merge commits can be very useful though, if the merge commit contains
a description of a multititude of patches.

Post by Ivo Raisr
There are a lot of good tutorials on simple git workflows, so please
have a look. If you are using something more complicated, please
share with us and ideally send us a write up.
I would like to help with the migration.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes, please! Send us your positive and negative feedback.
- It worked for me!
- This and that did not work for me...
- How do I do such and such thing now?
The test repository is there for you to play with. The contents
will be deleted before the final migration so no reason to worry about
potential mistakes. It is also quite likely that the contents will be
regenerated during the beta testing, to fix any problems found.
We also need a help documenting possible workflows. Especially when
preparing a release - we need to test and document how to work with branches
and releases.

[Ivo Raisr]

Post by Christian Borntraeger
For a pull like approach I would suggest to either
a: do not this step (Linux style git handling)
b: use git pull --rebase (flat history style)

Post by Ivo Raisr
build + test
git commit
[git push - if you have write access]

If we give write accesses, then I suggest to use the --rebase variant.
Not sure what git server is running, but git-o-lite for example allows
to reject non-fast-forward merges, so that we do not fill the
history with single patch merge commits.
Merge commits can be very useful though, if the merge commit contains
a description of a multititude of patches.

Are you referring here to the git merge model?

We had a discussion with Mark Wielaard previously which git model to use:
----------------------

Post by Christian Borntraeger

Post by Ivo Raisr
And given the git model we should also
decide whether we want a merge model or that we ask people to only push
rebased branches at the top of the tree. The merge model is more
flexible and more accurately shows the development history if features
get developed/integrated in parallel. But having just one straight
linear commit history often helps with bisecting and precisely
pin-pointing when bugs were introduced. I know sourceware can setup
commit/push hooks to make sure one or the other model is followed.

I think the second one reflects what we have been using in SVN.
Any reason to change that other than that it's more GIT'ish?

I think that is a fine reason and probably works out nicely when
multiple people push to the same repository. The "more GIT'sh" way is
(IMHO) slightly nicer when there is a single person that pushes to a
repo. Merges often represent work others did and show where they started
off and where they integrated their work. But this can (IMHO) get
slightly messy if multiple people push to the same repo.
---------------------

So we decided to stick with existing (SVN) workflow which translates to
"rebased branches at the top of the tree".
Our valgrind.git config will have (after the final migration happens):

[receive]
denyNonFastforwards = true

So based on this information, could you suggest different git commands
(with some reasoning) than initially outlined by me?

I.

[Christian Borntraeger]

On 02/28/2017 09:56 PM, Ivo Raisr wrote:
[...]

Post by Ivo Raisr
So we decided to stick with existing (SVN) workflow which translates to
"rebased branches at the top of the tree".

Which is a perfectly fine model that makes sense in our project.

Post by Ivo Raisr
[receive]
denyNonFastforwards = true
So based on this information, could you suggest different git commands
(with some reasoning) than initially outlined by me?

The given workflow requires that there is no other push between the pull
and the push. If for some reason somebody else pushes some update after
you have commited, you can recover with a rebase. There is a nice combine
of pull + rebase:
git pull --rebase
and I think we might just want to document that.

[Ivo Raisr]

Post by Christian Borntraeger
The given workflow requires that there is no other push between the pull
and the push. If for some reason somebody else pushes some update after
you have commited, you can recover with a rebase. There is a nice combine
git pull --rebase
and I think we might just want to document that.

Definitely yes. Would you mind suggesting something like
docs/internals/git-HOWTO.txt,
possibly taking into account existing docs/internals/svn-HOWTO.txt.

Thank you,
I.

[Tom Hughes]

Post by Ivo Raisr
So we decided to stick with existing (SVN) workflow which translates to
"rebased branches at the top of the tree".
[receive]
denyNonFastforwards = true

That doesn't actually prevent people pushing merges though, it just
stops history being rewritten - the push to the remote can only move the
remote forward but the pushed commits can include merges.

Tom

[Ivo Raisr]

Post by Ivo Raisr
So we decided to stick with existing (SVN) workflow which translates to
"rebased branches at the top of the tree".
[receive]
denyNonFastforwards = true

That doesn't actually prevent people pushing merges though, it just stops
history being rewritten - the push to the remote can only move the remote
forward but the pushed commits can include merges.

Good point.
So I think it's the time to start gathering config for AdaCore git hooks [1] [2]
which is used at sourceware.org.

----------------------------------
[hooks]
from-domain = sourceware.org
mailinglist = valgrind-***@lists.sourceforge.net

# Allow to include debugging output in commit messages.
max-rh-line-length = 0

# Forces to rebase changes before pushing to master and release branches.
reject-merge-commits = refs/heads/master, refs/heads/VALGRIND_.*

commit-url = "https://sourceware.org/git/?p=valgrind.git;h=%(rev)s"

# No emails from private user branches.
no-emails = /refs/heads/user/.*
----------------------------------

Your thoughts?
I.


[1] https://github.com/adacore/git-hooks
[2] https://sourceware.org/gdb/wiki/GitHooksUsersGuide

[Ivo Raisr]

Post by Ivo Raisr
Dear Valgrind community,
We are pleased to announce an imminent migration of Valgrind sources
from existing Subversion SCM to modern git SCM, as discussed during
our FOSDEM 2017 Valgrind devroom.

So far a number of very useful comments have been received.
Necessary infrastructure changes in Valgrind code repo and Valgrind
www repo were prepared,
together with the migration recipe.

Please review them all here: https://github.com/ivosh/valgrind-git-migration

1. Migration recipe itself:
https://github.com/ivosh/valgrind-git-migration/blob/master/migration-recipe.txt

2. Necessary Valgrind code repo changes:
https://github.com/ivosh/valgrind-git-migration/blob/master/001-Change-SVN-to-GIT-in-various-places.patch
https://github.com/ivosh/valgrind-git-migration/blob/master/002-Update-auxprogs-update-demangler-for-Valgrind-in-GIT.patch
https://github.com/ivosh/valgrind-git-migration/blob/master/003-Fix-nightly-build-script-to-work-with-git.patch
https://github.com/ivosh/valgrind-git-migration/blob/master/004-fix-verbose-version-reporting-for-git.patch
https://github.com/ivosh/valgrind-git-migration/blob/master/005-Change-Subversion-to-GIT-in-various-places.patch
https://github.com/ivosh/valgrind-git-migration/blob/master/006-Convert-release-HOWTO.txt-from-SVN-to-GIT.patch

3. Website changes (residing in valgrind-www SVN repository):
https://github.com/ivosh/valgrind-git-migration/blob/master/valgrind-www-changes.patch

Please provide your comments and suggestions.
We are very close to the final migration!

I.