With Mercurial sunsetting support, it was time to move our repos. But how do you do this en masse?
When we received the first communication about Bitbucket sunsetting Mercurial support, there was quite a lot of excitement around the office. All of our source code was stored as Mercurial repositories on Bitbucket and – being a product company – a lot of the company’s value is in the source code.
A closer look at the problem presented us with the following challenge:
- 350 Repos across 5 teams to convert
- Retaining the commit history for all projects
- All branches need to be converted
A solution was needed, but many searches and discussions later, nothing suitable was found. So, we decided to wait. There was still plenty of time and we trusted that a solution would surely present itself sooner or later.
However, when the reminder from Bitbucket arrived early in 2020, no solution had yet presented itself and time was starting to run out. A few searches later we found the fast-export converter by frej, a python project on Github that we could use to convert our repos one-by-one. While each migration seems simple enough, our challenge was that we had to do this in bulk.
The challenge to do this in bulk was a great use case for Linx. Linx excels at the rapid development and deployment of back-end automations. Its visual modules accelerate the development process for developers to quickly build unattended automations without having to worry about setting up or scaling their backend infrastructure and the related services.
We created an automated process that would perform the entire conversion against each of our repos without any intervention.
The Process
Within 3 days we developed and tested our Linx solution, which works great. The final solution automatically performed the following steps:
- Calling the Bitbucket API to fetch a list of hg repos and writing the info to a CSV file
- Cloning each hg repo in the file
- Converting each hg repo to git
- Creating the git repo on Bitbucket
- Pushing the git repo to Bitbucket
The only limitation we know of is that closed branches and issues are not migrated. It seems that the closed branches concept does not exist in Git and we don’t use the Bitbucket Issues feature. However, if you need Issues, building the migration of these into the Linx solution should be quick and easy. Bitbucket has an Issue Export API call that you can implement in Linx.
Results
Once ready, each of our teams would need to run this solution themselves as the conversion needed to fit into each team’s release cycle. So, the process each team followed was:
- Everyone checked in all changes at the end of the conversion day
- We ran the Linx process overnight or over a weekend
- In the morning, the teams pulled the new repos and just continued where they had left off
Some conversions took a long time to run, but for most teams have completed their conversions now and development has continued smoothly and without interruptions to date.
Some of our results were pretty impressive;
- The longest conversion process took 12 hours to run
- The oldest project had commit history going back as far as 2006
- The largest project has over 500MB of source code
- Over 300 company and over 80 personal repos converted
Try it! The solution is available for anyone to use. Get the Linx Designer and use the sample provided.