Working with multiple git repositories

...

Working with multiple git repositoriesWorking with multiple git repositories

Julien Pivotto

T-Dose, EindhovenOctober 26, 2014

..

whoamiwhoamiJulien PivottoJulien Pivotto

• System administrator at inuits.eu

• Git user for 5+ years• DevOps believer• Open-source defender since 2004• roidelapluie on twitter/github

..

Ops <3 DevOps <3 Dev

• Infrastructure as Code▶ SCM all the things▶ Monitoring▶ Configuration▶ Application deployment

• Taking part of software development▶ Understanding▶ Monitoring▶ High Availability▶ …

..

Me and gitMe and git

• I first used subversion (10 years ago)• Private forges and sourceforge• 5 years later a lot of projects moved to git• The Puppet community also uses a lot git• Git/Hg are the de-facto standards in OSprojects

..

Git is a free and open source distributed version control systemdesigned to handle everything from small to very large projects

with speed and efficiency.

Source: http://git-scm.com/

..

git loggit log

• Initiated in 2005 by Linus Torvalds• Replacement of closed-source BitKeeper• Created for the Linux Kernel development• Now used by thousands of projects

..

git is distributedgit is distributed

• Everything can be done in local (exeptpull&push)

• Work with several remotes• Share code with anyone• No unique central repository• A lot of workflows possible

..

Branching and merging is easyBranching and merging is easy

• You can branch locally• Merging is easy and simple• You can squash commits

..

How to access a repoHow to access a repo

• Locally (Local filesystems)• SSH• HTTP/HTTPs• Git protocol

..

git is open-sourcegit is open-source

• Published under GPL-2• Written in C• A lot of frontends/backends• A lot of libraries• A lot of hosting services• And a lot of users

..

Multiple git repos: why?Multiple git repos: why?

• Code isolation: plugins, libraries• Getting faster feedback from testing• Contributing back to upstream projects• Separate apps that are released together• Share code between projects

..

Multiple git repos usecasesMultiple git repos usecases

• Infrastructure as Code• Projects released together• Projects that embed external libraries• …

..

git submodules

..

git submodulesgit submodules

• Built-in with git• Embeds external repositories in asubdirectory or your repo

• Points these external repositories in aparticular commit

..

git submodulesgit submodules

• Submodules act like traditional gitrepositories

• Super repo contains a file with thesubmodules urls

• Super repo contains objects that can beupdated with commit hashes

..

Initiate super-repoInitiate super-repo.

.

$ git init .Initialized empty Git repository in /home/roidelapluie/super-repo/.git/$ > README$ git add README$ git commit -m "Initial commit"[master (root-commit) b526be8] Initial commit1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 README

..

Initiate subrepoInitiate subrepo.

.

$ git init .Initialized empty Git repository in /home/roidelapluie/subrepo/.git/$ > README$ git add README$ git commit -m "Initial commit"[master (root-commit) 41e64b6] Initial commit1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 README

..

Add first module in parent repoAdd first module in parent repo.

.

$ git submodule add ../subrepo/ subrepoCloning into 'subrepo'...done.$ git statusOn branch masterChanges to be committed:

(use "git reset HEAD <file>..." to unstage)

new file: .gitmodulesnew file: subrepo

..

What just happened?What just happened?

• git created a .gitmodules file• It cloned the subrepo in a directory• It created a git object that points to the repo• That object contains the git revision

..

.gitmodules file.gitmodules file

.

.

$ cat .gitmodules[submodule "subrepo"]

path = subrepourl = ../subrepo/

• Unique file for all the submodules• Only contains the path and the url• In this case, we use relative URL

..

Commiting our changeCommiting our change

.

.

$ git commit -m "Add subrepo"[master 3251a63] Add subrepo2 files changed, 4 insertions(+)create mode 100644 .gitmodulescreate mode 160000 subrepo

..

Cloning that super repoCloning that super repo

.

.

$ git clone /home/roidelapluie/super-repo/ .Cloning into '.'...done.$ ls subrepo/$ git submodule initSubmodule 'subrepo' (/home/roidelapluie/subrepo)registered for path 'subrepo'$ git submodule updateCloning into 'subrepo'...done.Submodule path 'subrepo': checked out '41e64b6bc0c970521c296b9c5e6a0b9144778962'

..

Shorcut #1Shorcut #1

.

.

$ git submodule init$ git submodule update

.

.

$ git submodule update --initSubmodule 'subrepo' (/home/roidelapluie/subrepo)registered for path 'subrepo'Cloning into 'subrepo'...done.Submodule path 'subrepo': checked out '41e64b6bc0c970521c296b9c5e6a0b9144778962'

..

Shorcut #2Shorcut #2.

.

$ git clone /home/roidelapluie/super-repo/ .$ git submodule init$ git submodule update

.

.

$ git clone --recurse-submodules /home/roidelapluie/super-repo/ .Cloning into '.'...done.Submodule 'subrepo' (/home/roidelapluie/subrepo)registered for path 'subrepo'Cloning into 'subrepo'...done.Submodule path 'subrepo': checked out '41e64b6bc0c970521c296b9c5e6a0b9144778962'

..

Working in a submoduleWorking in a submodule.

.

$ cd subrepo$ > newfile$ git add newfile$ git commit -m "add newfile"[detached HEAD 6ee0e71] add newfile1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 newfile

$ git pushfatal: You are not currently on a branch.To push the history leading to the current (detached HEAD)state now, use

git push origin HEAD:<name-of-remote-branch>

..

Checking out a branchChecking out a branch.

.

$ git checkout masterWarning: you are leaving 1 commit behind, not connectedto any of your branches:

6ee0e71 add newfile

If you want to keep them by creating a new branch, thismay be a good time to do so with:

git branch new_branch_name 6ee0e71

Switched to branch 'master'Your branch is up-to-date with 'origin/master'.

..

Merging our commitMerging our commit

.

.

$ git merge 6ee0e71Updating 41e64b6..6ee0e71Fast-forwardnewfile | 01 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 newfile

..

Pushing our commitPushing our commit

.

.

$ git pushCounting objects: 2, done.Delta compression using up to 8 threads.Compressing objects: 100% (2/2), done.Writing objects: 100% (2/2), 247 bytes | 0 bytes/s, done.Total 2 (delta 0), reused 0 (delta 0)To /home/roidelapluie/subrepo

41e64b6..6ee0e71 master -> master

..

Pushing our commitPushing our commit

.

.

$ date > newfile$ git commit -am "update newfile"[master 8a56806] update newfile1 file changed, 1 insertion(+)

$ git pushCounting objects: 3, done.Delta compression using up to 8 threads.Compressing objects: 100% (2/2), done.Writing objects: 100% (3/3), 310 bytes | 0 bytes/s, done.Total 3 (delta 0), reused 0 (delta 0)To /home/roidelapluie/subrepo

6ee0e71..8a56806 master -> master

..

Let's rewind..Let's rewind..

• A submodule not attached to a branch▶ Because submodules are referenced by their commit hashonly

▶ It is called the DETACHED HEAD

• How to push then?▶ The easy/safe option is to checkout a branch

• You only need to do that once

..

git submodule foreachgit submodule foreach

• git submodule foreach allow you to run acommand on each repo

▶ git submodule foreach git checkout master▶ git submodule foreach ls▶ git submodule foreach git commit -am "foo"

..

Meanwhile, on the super-project…Meanwhile, on the super-project….

.

$ git statusOn branch masterChanges not staged for commit:

(use "git add <file>..." to update what will becommitted)

(use "git checkout -- <file>..." to discard changes inworking directory)

modified: subrepo (new commits)

no changes added to commit (use "git add" and/or "gitcommit -a")

..

Update a subrepoUpdate a subrepo

.

.

$ git commit -am "update subrepo"[master 731626b] update subrepo1 file changed, 1 insertion(+), 1 deletion(-)

..

LogsLogs

.

.

$ git log -p --submodule=log subrepo/commit 731626bc53d8c248647443b4c0523d96d316cd24Author: Julien Pivotto <roidelapluie@inuits.eu>Date: Sat Oct 25 22:26:56 2014 +0200

update subrepo

Submodule subrepo 41e64b6..01c1b9c:> work on newfile> update newfile> add newfile

..

Change a submodules URLChange a submodules URL

• Edit .gitmodules and commit the file• Run git submodule sync• Now the submodule will use the new URL

..

git submodule --recursivegit submodule --recursive

• Run commands recursively• You can get multiple levels of submodules• But you add complexity

▶ Each level is 1 more commit you need to write▶ Playing with remote urls becomes tricky of you work withremotes

..

Submodules integrationSubmodules integration

• Integrated in git, standard• Integrated in CI tools (Jenkins)• Nothing specific needed for pulling• Need to checkout the submodules afterpulling

..

Submodules in GerritSubmodules in Gerrit

• Gerrit integrates submodules• It can create commits in "super-repo" if acode change is merged

• Not a perfect way of doing if changes in tworepo's are depending of each other

..

Submodules in GerritSubmodules in Gerrit

• Gerrit is integrated in CI tools• Gerrit allows fine tuning ACL• Gerrit code reviews with submoduleschanges don't display changes in thesubmodules

..

Submodules pros and consSubmodules pros and cons

− Each commit in submodule is not insuper-repo

− It is easy to rollback by accident asubmodule

− Getting logs is not easy− You can forget to update super-repo− Submodules are not attached to a branch

..

Submodules pros and consSubmodules pros and cons

+ Built-in+ Can use relative urls+ You can control the version of the

submodules+ Supported by many tools+ Easy rollbacks of submodules+ git bissect works (on the super repo)

..

gitslave a.k.a. gits

..

gitslavegitslave

• First public release in 2010• Wrapper around git• "gits" can be used in place of git• Run each git commands in each subrepos

..

Initiate gitsInitiate gits

.

.

$ git init .Initialized empty Git repository in /home/roidelapluie/superrepo/.git/$ gits prepare[master (root-commit) 594c492] gits creating .gitslave1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 .gitslave

..

Add a git repoAdd a git repo

.

.

$ gits attach /home/roidelapluie/subrepo/ subrepoCloning into 'subrepo'...done.[master 926321f] gits adding "/home/roidelapluie/subrepo/" "subrepo"2 files changed, 2 insertions(+)create mode 100644 .gitignore

$ cat .gitslave"/home/roidelapluie/subrepo/" "subrepo"

..

Cloning a repoCloning a repo

.

.

$ gits clone ../superrepo/ .Cloning into '.'...done.Cloning into 'subrepo'...done.

..

Working in submodulesWorking in submodules

.

.

$ cd subrepo$ date > newfile$ git add newfile$ gits commit -m "Work on newfile"On: subrepo:

[master 1230de3] Work on newfile1 file changed, 1 insertion(+), 1 deletion(-)

On: (super-repo):On branch masterYour branch is up-to-date with 'origin/master'.nothing to commit, working directory clean

..

Pushing changesPushing changes

.

.

$ gits pushOn: subrepo:

To /home/roidelapluie/subrepo/fbcf73c..1230de3 master -> master

On: (super-repo):Everything up-to-date

..

Pulling changesPulling changes

.

.

$ gits pullOn: (super-repo), subrepo:

Already up-to-date.

..

git slave workflowgit slave workflow

• Subrepos are always attached to a branch• You can work within them "normally"• Run almost any git command with "gits"

..

PerformancePerformance

• gits --parallel 8: run 8 parallel git commands• No extra commit in the super repository• Close to traditional git workflow

..

A branching approachA branching approach

• Use gits branch and gits checkout to switchbetween branches

• All the repositories will switch branches (e.g1.0, 1.1)

• Easy merge with gits merge• Avoid upstream traditional branch names

..

gits outputgits output

• gits will simplify the output of gitcommands

• It will group the repos by output• Easier to see the status, the remotes, etc

..

Pros and cons of gitslavePros and cons of gitslave

− Not integrated in tools− Requires an additional binary− Not integrated in git− Hard to rollback on every repositories

..

Pros and cons of gitslavePros and cons of gitslave

+ Close to git workflow+ Less error-prone+ Use traditional git commands+ Every git push arrives in the repo

..

Mitigeate gitslaves' issuesMitigeate gitslaves' issues

• Use git hooks to trigger tests on CI▶ Polling SCM does not scale anyway▶ Use the "Quite period" feature

• Use CI to add the same tag in eachrepository

▶ Rollback to a tag, not a commit▶ Add a tag at each commit

..

Additional tips

..

Use SSH socketsUse SSH sockets

.

.

$ cat .ssh/configHost githost

controlmaster autocontrolpath /tmp/ssh-%r@%h:%pControlPersist 300

• Use only one SSH connection• Do not disconnect SSH

..

Find the tag of a commitFind the tag of a commit

.

.

$ git name-rev --name-only 106cb38dd0bc5ea2d2d0adef0b4006bd61884e42remotes/origin/8.5.2~1

..

git autocorrectgit autocorrect

.

.

$ git config help.autocorrect 10$ git puskWARNING: You called a Git command named 'pusk', which does not exist.Continuing under the assumption that you meant 'push'in 1.0 seconds automatically...

..

Use git hooksUse git hooks

• Multiple repositories = a lot of pull• Jenkins pulling X repos X times every 5minutes is a bad idea

• Git hooks to the resue• Reduce load on Jenkins and the git server• Reduce the time before Jenkins launchesbuilds

..

GPG-Sign your commitsGPG-Sign your commits

• git commit -S• Sign PR, merge commits• Do not sign every commit (then it becomesuseless)

..

Keeping multiple remotes in syncKeeping multiple remotes in sync

• git remote set-url --add• Pull & push will ose both urls

..

HomeworkHomework

• Git subtree (also built-in)• Repo (from Android)• gr• …

..

ConclusionsConclusions

• Git is an awesome, extandable tool• There are several ways to manage multiplegit repositories

• Working with multiple repos is easy• You have to learn tools anyway• Experiment them and pick one :-)

..

Thank youThank you

Any question?

..

ContactContact

Julien Pivottojulien@inuits.eu@roidelapluie

INUITS bvbaBelgium+32 473 441 636https://inuits.eu