Merging Repositories

Question: Can we merge a repository into another without losing history? For merging two existing repositories a lot can be found on the internet. Eventually I merged two found methods into one method. The second part of the first method did not result in the wanted behavior. For that the second method was used.

The found methods:

Tip

Within the following steps repository A will be merged into repository B. It is wise to create a temporary directory in which both repositories will be cloned, so when a certain step does not have the wanted outcome it will be fairly easy to start all over again.

Note

Most of the steps below are shown from a script used to execute the merge between two repositories, see merge_git_repo_a_to repo_b.

Steps

Create temporary directory:

mkdir $HOME/tmp/merge_git_repos
cd $HOME/tmp/merge_git_repos

Clone repository A into a temporary name tmp_repo_a:

git clone <url of repo a> tmp_repo_a

Clone repository B into a temporary name tmp_repo_b:

git clone <url of repo b> tmp_repo_b

Execute script for merging repo a into repo b:

merge_git_repo_a_to_repo_b

Note

The script can also be used with the locations of the two repositories (e.g. merge_git_repo_a_to_repo_b <location repo a> <location repo b>. If the location are omitted the defaults tmp_repo_a and tmp_repo_b are used.

Executed steps from the shell script:

Checkout all the branches you want to copy from repository A:

cd $REPO_A_PATH
git branch -a > $MERGE_PATH/repo_a_branches.txt
for branch in `cat $MERGE_PATH/repo_a_branches.txt`
do
   git checkout $branch
done

Fetch all the tags from repository A:

git fetch --tags

Check if you have all the tags and branches:

git tag
git branch -a

Clear the link to the remote of repository A:

git remote rm origin

Do some filtering for rewriting the history:

git filter-branch -f --tree-filter 'mkdir -p toDelete;git mv -k docs/Makefile toDelete/.;git mv -k docs/_external_templates/conf/* toDelete/.;git mv -k docs/_external_templates/static/* toDelete/.;git mv -k docs/requirements.txt toDelete/.;git mv -k docs/doc/_generic.inc toDelete/.;git mv -k docs/doc/conf.py toDelete/.;git mv -k docs/doc/index.rst toDelete/docs_index.rst;git mv -k README.md toDelete/.' HEAD
git filter-branch -f --tree-filter 'mkdir -p docs/doc/Teams/0.Talk_like_pi;git mv -k docs/doc/teams/2018.1_Talk_like_pi/* docs/doc/Teams/0.Talk_like_pi;git mv -k docs/doc/teams/index.rst toDelete/teams_index.rst' HEAD
git filter-branch -f --tree-filter 'mkdir -p docs/doc/Demonstrators;git mv -k docs/doc/IPSC/* docs/doc/Demonstrators/.' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/MQTT/Bandwidth/Python;mkdir -p docs/doc/Demonstrators/ComparedToOtherProtocols/Bandwidth/MQTT;git mv -k IPSC/MQTT/py/Bandwidth/* src/demonstrators/ComparedToOtherProtocols/MQTT/Bandwidth/Python; git mv -k src/demonstrators/ComparedToOtherProtocols/MQTT/Bandwidth/Python/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/Bandwidth/MQTT/index.rst;git mv -k src/demonstrators/ComparedToOtherProtocols/MQTT/Bandwidth/Python/ipc/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/Bandwidth/MQTT/ipc.rst;git mv -k src/demonstrators/ComparedToOtherProtocols/MQTT/Bandwidth/Python/isc/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/Bandwidth/MQTT/isc.rst' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/MQTT/RoundTrip/Python;mkdir -p docs/doc/Demonstrators/ComparedToOtherProtocols/RoundTrip/MQTT;git mv -k IPSC/MQTT/py/RoundTrip/* src/demonstrators/ComparedToOtherProtocols/MQTT/RoundTrip/Python;git mv -k src/demonstrators/ComparedToOtherProtocols/MQTT/RoundTrip/Python/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/RoundTrip/MQTT/index.rst;git mv -k src/demonstrators/ComparedToOtherProtocols/MQTT/RoundTrip/Python/ipc/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/RoundTrip/MQTT/ipc.rst;git mv -k src/demonstrators/ComparedToOtherProtocols/MQTT/RoundTrip/Python/isc/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/RoundTrip/MQTT/isc.rst' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/MQTT/PingPong/Python;mkdir -p docs/doc/Demonstrators/ComparedToOtherProtocols/PingPong/MQTT;git mv -k IPSC/MQTT/py/* src/demonstrators/ComparedToOtherProtocols/MQTT/PingPong/Python;git mv -k src/demonstrators/ComparedToOtherProtocols/MQTT/PingPong/Python/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/PingPong/MQTT/index.rst' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/ZMQ/Bandwidth/Python;mkdir -p docs/doc/Demonstrators/ComparedToOtherProtocols/Bandwidth/ZMQ;git mv -k IPSC/ZMQ/py/Bandwidth/* src/demonstrators/ComparedToOtherProtocols/ZMQ/Bandwidth/Python;git mv -k src/demonstrators/ComparedToOtherProtocols/ZMQ/Bandwidth/Python/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/Bandwidth/ZMQ/index.rst' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/ZMQ/RoundTrip/Python;git mv -k IPSC/ZMQ/py/RoundTrip/* src/demonstrators/ComparedToOtherProtocols/ZMQ/RoundTrip/Python' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/ZMQ/Push-Pull/Python;git mv -k IPSC/ZMQ/py/Push-Pull/* src/demonstrators/ComparedToOtherProtocols/ZMQ/Push-Pull/Python' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/ZMQ/Request-Reply/PingPong/Python;git mv -k IPSC/ZMQ/py/Request-Reply/PingPong/* src/demonstrators/ComparedToOtherProtocols/ZMQ/Request-Reply/PingPong/Python' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/ZMQ/Request-Reply/RoundTrip/Python;git mv -k IPSC/ZMQ/py/Request-Reply/Roundtrip/* src/demonstrators/ComparedToOtherProtocols/ZMQ/Request-Reply/RoundTrip/Python' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/ComparedToOtherProtocols/ZMQ/PingPong/Python;mkdir -p docs/doc/Demonstrators/ComparedToOtherProtocols/PingPong/ZMQ;git mv -k IPSC/ZMQ/py/* src/demonstrators/ComparedToOtherProtocols/ZMQ/PingPong/Python;git mv -k src/demonstrators/ComparedToOtherProtocols/ZMQ/PingPong/Python/readme.rst docs/doc/Demonstrators/ComparedToOtherProtocols/PingPong/ZMQ/index.rst' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/PingPong/Python;mkdir -p docs/doc/Demonstrators/PingPong;git mv -k IPSC/DDS/CycloneDDS/py/PingPong/* src/demonstrators/PingPong/Python;git mv -k src/demonstrators/PingPong/Python/readme.rst docs/doc/Demonstrators/PingPong/index.rst' HEAD
git filter-branch -f --tree-filter 'mkdir -p src/demonstrators/RoundTrip/Python;mkdir -p docs/doc/Demonstrators/RoundTrip;git mv -k IPSC/DDS/CycloneDDS/py/Loop/* src/demonstrators/RoundTrip/Python;git mv -k src/demonstrators/RoundTrip/Python/readme.rst docs/doc/Demonstrators/RoundTrip/index.rst' HEAD
git filter-branch -f --index-filter 'git rm -r --cached --ignore-unmatch toDelete' HEAD

Tip

The index-filter takes less time in comparison to the tree-filter and can be used for removing files/directories. Also sometimes removing files/directories using the tree-filter does not always rewrite history.

For changing the directory structure the following commands can be used within a tree-filter

> mkdir -p <new directory>;git mv -k <file to move> <new directory>

For removing a file/directory the following command can be used within a index-filter

> git rm -r –cached –ignore-unmatch <file or directory>

Note

The order of the actions within the commands or executed filtering is very important. It is best to first move sources and finally removing files/directories, since the second filtering must use option -f (force) to be executed.

Removing garbage from filtering:

git reset --hard
git gc --aggressive 
git prune
git clean -fd

Perfrom any necessary changes and commit these:

git status
git add .
git commit

Create a new branch in repository B:

cd $REPO_B_PATH
# create a new branch
git checkout -b feature/merge_git_repos

Create a remote connection to repository A as a branch in repository B:

git remote add repo-a $REPO_A_PATH

Note

repo-a can be anything - it’s just a random name

Pull files and history from branches into repository B:

git pull repo-a master --allow-unrelated-histories

Remove the remote connection to repository A:

git remote rm repo-a

Finally, push the changes:

git push origin <new branch name>