contributed articles

Google's monolithic repository provides a common source of truth for tens of thousands of developers around the world

BY RACHEL POTVIN AND JOSH LEVENBERG

Why Google Stores Billions of Lines of Code in a Single Repository

EARLY GOOGLE EMPLOYEES decided to work with a shared codebase managed through a centralized source control system. This approach has served Google well for more than 16 years, and today the vast majority of Google's software assets continues to be stored in a single, shared repository. Meanwhile, the number of Google software developers has steadily increased, and the size of the Google codebase the technology used to host the codebase has also evolved significantly.

This article outlines the scale of that codebase and details Google's custombuilt monolithic source repository and the reasons the model was chosen. Google uses a homegrown version-control system to host one large codebase ware developers in the company This centralized system is the foundation of many of Google's developer workflows. Here, we provide background on the systems and workflows that make featively with such a large repository: We development" strategy and the support systems that structure workflow and Google's codebase healthy, including software for static analysis, code cleanup; and streamlined code review. keep

Google-Scale

The Google codebase includes approximately one billion files and has a history of approximately 35 million commits spanning Google's entire 18year existence. The repository contains

Google's monolithic software repository, which is used by 95% of its software developers worldwide; meets the definition of an ultra-large-scale' gle-source repository model can be scaled successfully.

• a Total size ofuncompressed content, excluding release branches.

insights key

• Google of source code management can scale to a repository of one billion files; 35 million commits, and tens of thousands of developers. has
• Benefits include unified versioning, extensive code sharing, simplified dependency management, atomic changes, large-scale refactoring, collaboration across teams, flexible code ownership, and code visibility.
• Drawbacks include having to create and scale tools for development and execution and maintain code health; as well as potential for codebase complexity (such as unnecessary dependencies) .

8

two billion lines of code in nine million unique source files. The total number of files also includes source files copied into release branches, files that are deleted at the latest revision; configuration files, documentation; and supporting data files; see the table here for summary of Google's repository statistics from January 2015.

lines of code were changed in approximately 250,000 files in the Google repository on a weekly basis. The Linux kernel is prominent example of a open source software repository containing approximately 15 million large

Google's codebase is shared by more

than 25,000 Google software developers from dozens of offices in countries around the On typical workcommit 16,000 changes to the codebase, and another 24,000 changes are committed by automated systems. Each the repository serves billions ly 800,000 queries per second peak traffic and an average of approximately 500,000 queries per second each workday. Most of this traffic originates from Google's distributed buildworld. day, they day during

both the interactive use case, or huing holidays affecting significant number of employees (such as Christmas and New Year's Ameri can Thanksgiving and American Independence Day) Day Day, Day,

The commits-per-week graph shows the commit rate was dominated by human users until 2012, at which point Google switched to custom-

In October 2012, Google's central repository added support for Windows and Mac users (until then it was Linux Mac repository was merged with the repository merges attributes all historical changes being merged to their original authors, hence the corresponding fect of this merge is also apparent in Figure 1_



source-control implementation for hosting the central as discussed later. Following this transition, automated commits to the rethe commit rate continues primarily due to automation.

Managing this scale of repository and activity on it has been an ongoing challenge for Google. Despite several years of experimentation, Google was not able to find a commercially available or open source versioncontrol system to support such scale in single repository The Google proprietary system that was built to is code-named Piper.

Background

Before reviewing the advantages and disadvantages of working with some background on Google's tooling and work -

Piper and CitC. Piper stores a single large repository and is implemented on structure, originally Bigtable,2 now Spanner:3 Piper is distributed over 10 Google data the to guarantee consistency across licas. This architecture provides high level of redundancy and helps optimize latency for Google software no matter where they work. In addition; caching and asynchronous operations hide much of the network latency from developers This is important because the full benefit of Google's cloudbased toolchain requires developers to be online. top repgaining

Google reliedonone primary Perforce instance, hosted on single machine, coupled with custom caching infrastructure' for more than 10years prior to the launch of Piper. Continued scaling of

the Google repository was the main

Since Google's source code is one of the company's most important assets, security features are consideration in Piper's design. supports file-level access control lists. Most of the repository is visible to users;d however, important configuration files or files including businesscritical algorithms can be more tightly controlled. In addition, read and write access to files in Piper is logged. If sensitive data is accidentally committed to Piper, the file in question can be purged. The read logs allow administrators to determine if anyone accessed the problematic file before it was removed. key Piper

In the Piper workflow (see Figure 4), developers create a local copy of files in the repository before changing them. These files are stored in a workspace owned by the developer: A Piper workspace is comparable to a working copy in Apache Subversion, local clone in Git, or a client in Perforce. Updates from the Piper repository can be pulled A snapshot of the workspace can be shared with other developers for review. Files in a workspace are committed to the central repository only after going through the Google code-review process, as described later.

Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of cloud-based storage backend and a opers see their workspaces as directochanges overlaid on of the full Piper repository. CitC supports code browsing and normal Unix tools with no need to clone or sync state Developers can browse and edit files anywhere across the Piper repository, and only modified files are stored in their workspace. This structure means CitC workspaces typically consume only a small amount of storage (an average workspace has fewer than 10 files) while presenting a seamless view of the entire Piper codebase to the developer. top

All writes to files are stored as snapshots in CitC, making it possible to recover previous stages of work as needed. Snapshots may be explicitly named, restored, or tagged for review

Several workflows take advantage of the availability of uncommitted code in CitC to make software developers working with the large codebase more a change out for code review, developers can enable an auto-commit option; which is particularly useful when code authors and reviewers are in different time zones. When the review is marked as complete, the tests will run; if pass, the code will be committed to the repository without further human intervention. The Google codebrowsing tool CodeSearch supports simple edits using CitC workspaces. browsing the repository, developers can click on a button to enter edit mode and make a simple change (such as typo Or improving ing= they fixing

CitC workspaces are available on any machine that can connect to the cloud-based storage system; making it easy to switch machines and pick up work without interruption. It also makes it possible for developers to view each other's work in CitC workspaces. Storing all in-progress work in the cloud is an important element of the Google workflow process. Workstate is thus available to other tools, including the cloud-based build system; the automated test infrastructure, and the code browsing, editing, and review tools.

a without leaving the code browser, can send their changes out to the appropriate reviewers with auto-commit enabled. they

Piper can also be used without CitC. Developers can instead store Piper workspaces on Piper also has limited interoperability with Git. Over 80% of Piper users today use CitC, with adoption continuing to grow due to the many benefits provided by CitC.

Trunk-based development. Google practices trunk-based development on top of the Piper source repository. The vast users work at the Or most recent, version of a single copy of the code called "trunk" Or Changes are made to the repository in a single, serial orderThe combination of trunk-based development with a central repository defines the monolithic codebase model. Immediately after any commit, the all other developers. The fact that Piper single consistent view of the Google codebase is for prothe advantages described later in this article. ing; key viding

Piper and CitC make working productively with single, monolithic source repository possible at the scale of the Google codebase. and architecture of these systems were both heavily influenced by the trunkbased development paradigm employed at Google as described here

Trunk-based development is benefi cial in part because it avoids the ful merges that often occur when it is time to reconcile long-lived branches. Development on branches is unusual and not well supported at Google, though branches are typically used for releases. Release branches are cut from a specific revision of the repository. Bug fixes and enhancements that must be added to a release are typically developed on mainline then cherrypicked into the release branch (see Figure 6). Due to the need to maintain stability and limit churn on the release branch, a release is typically a snapshot of head, with an optional small number of cherry-picks pulled in from head as needed. Use of long-lived branches with parallel development on the branch and mainline is exceedpain-



CitC Piper and make working productively with a single; monolithic source repository possible at the scale of the Google codebase.

When new features are developed, both new and old code paths commonly exist simultaneously, controlled through the use of conditional flags. for development branch and makes it easy to turn on and through configuration updates rather than full binary releases. While some additional complexity is incurred for developers, the merge problems of development branch are avoided. flips make it much easier and faster to switch users off new implementations that have problems. This specific code, not common library code, and eventually flags are retired uses similar approach routlive traffic through different code paths to perform experiments that can be tuned in real time through configuration changes. Such A/B experiments can measure everything from the performance characteristics of the code to user engagement related to subtle product changes: Flag ing

An important aspect of Google culture that encourages code quality is the expectation that all code is reviewed

Google workflow. Several best practices and supporting systems are required to avoid constant breakage in the trunk-based development model, where thousands of engineers commit thousands of changes to the repository on a daily basis. For instance, Google has an automated testing infrastructure that initiates a rebuild of all fected dependencies on almost every If a change creates widespread build breakage, system is in place to automatically undo the change; To reduce the incidence of bad code committed in customizable Google 'presubmit" infrastructure provides automated testthey are added to the codebase. A set of global presubmit analyses are run for all changes, and code owners can credirectories within the codebase specify A small set of very low-level core libraries uses a mechanism similar to a development branch to enforce additional testing before new versions are exposed to client code. afing they

before being committed to the repository. Most developers can view and propose changes to files anywhere across the entire codebase-_with the exception of a small set of highly confidential code that is more carefully controlled. The risk associated with developers changing code are not deeply familiar with is mitigated through the code-review process and the concept of code ownership. The Google codebase is laid out in a tree structure Each and every directory has a set of owners who control whether change to files in their directory will be accepted. Owners are typically the developers who work on the projchange often receives a detailed code review from one developer, evaluating the quality of the change, and a commit approval from an owner, evaluating the appropriateness of the change to their area of the codebase. they

phases. With this approach, a large backward-compatible change is made first. Once it is complete, a second can tern that is no longer referenced. A Google tool called Rosief supports the first phase of such large-scale cleandevelopers create a large patch, eifind-and-replace operation across the entire repository Or through more complex refactortools. Rosie then takes care of splitting the patch into smaller patches, testing them independently, sending them out for code review, and committing them automati cally once pass tests and a code review Rosie splits patches along project directory lines, relying on the code-ownership hierarchy described priate reviewers. large they

performing large-scale code changes balanced against the cost incurred by teams needing to review the ongoing stream of simple changes Rosie generates. As Rosie's popularity and usage grew, it became clear some control had to be established to limit Rosie's use to high-value changes that would be distributed to many reviewers, rather than to single atomic changes or rejected. In 2013, Google adopted a formal large-scale change-review process thatledtoadecrease inthe number of commits through Rosie from 2013 to 2014. In evaluating a Rosie change, the review committee balances the benefit of the change against the costs of reviewer time and repository churn. We later examine this and similar trade-offs more closely.

Code reviewers comment on asnaming, comment quality, and code style, as language-specific Google style guides: Google has written a code-review tool called Critique that allows the reviewer to view the evolution of the code and comment on any line of the change. It encourages further revisions and a conversation leading to a final "Looks Good To Me" from the reviewer, indicating the review is complete.

A team of Google developers will occasionally undertake a set of widereaching code-cleanup changes to further maintain the health of the codebase. The developers who perform these changes separate

Google's static analysis system (Tricorder ) and presubmit infrastructure also provide data on code quality, test coverage, and test results automaticallyin the Google code-review tool. These computationally intensive checks are triggered periodically, as well as when with one-click code editing for many tant data to increase the effectiveness of code reviews and keep the Google codebase healthy

In sum, Google has developed a number of practices and tools to support its enormous monolithic codebase, including trunk-based development, the distributed source-code repository Piper, the workspace client CitC, and workflow-support-tools Critique, CodeSearch, Tricorder, and Rosie. We discuss the pros and cons of this model here.


This section outlines and expands upon both the advantages of a monolithic codebase and the costs related to maintaining such a model at scale.

Most important; it supports:

Advantages. Supporting the ultralarge-scale of Google's codebase while maintaining performance for tens of thousands of users is a chalmonolithic model due to its compeladvantages . good ling

• Unified versioning; one source of truth;
• Simplified dependency management;
• Extensive code sharing and reuse;
• Atomic changes;
• Collaboration across teams;
• Large-scale refactoring;
• Flexible team boundaries and code ownership; and

A single repository provides unified versioning and a single source of truth. There is no confusion about which repository hosts the authoritative version of a file. If one team wants to depend on another team code, it can depend on it directly. The Google codebase includes a wealth of useful libraries, and the monolithic repository leads to extensive code sharing and reuse.

• Code visibility and clear tree structure providing implicit team namespacing.

The Google build system= 5 makes it easy to include code across directories, simplifying dependency management. changes to the dependencies of a project trigger dependent code. Since all code is ver-

and no concern about independent versioning of dependencies.

Most notably, the model allows Google to avoid the "diamond dependency" problem (see Figure 8) that occurs when A depends on B and C, both version D.1 and C requires version D.2. In most cases it is now impossible to build A. For the base library D, it can become very difficult to release a new version without causing breakage, since all its callers must be updated at the same time. Updating is difficult when the library callers are hosted in

In the open dencies are commonly broken by library updates, and finding library versions that all work together can be a challenge Updating the versions of dependencies can be painful for develbecome very a monolithic source tree it makes sense, and is easier, for the person updating a library to update all affected dependencies at the same time. The technical debt incurred by dependent systems is down immediately as changes are made. Changes to base libraries are instantly propagated through the dependency chain into the final products that rely on the libraries, without requiring separate sync or migration paid step.

Note the diamond-dependency problem can level, as described here as well as between binaries. 12



binary problem is avoided through use

The ability to make atomic changes is also a very powerful feature of the monolithic model. A developer can make a major change touching hundreds or thousands of files across the repository in a single consistent commit and yet not break any builds op-

The availability of all source code in single repository, or at least on a the maintainers of core libraries to perform testing and performance benchmarking for high-impact changes before are committed . This approach is useful for exploring and measuring the value of highly disruptive changes. One concrete example is an experiment to evaluate the feasibility of converting Google data centers to support non-x86 machine architectures. they

With the monolithic structure of the Google repository, developer never has to decide where the repository boundaries lie. Engineers never need to "fork" a shared library or merge across repositories to update versions Team boundaries are fluid. When project ownership changes or plans are made to consolidate systems, all code is already in the same repository. This environment makes it easy to do gradual refactoring and reorganization of the codebase: The change to move a project and update all dependencies can be atomically to the repository; and the development history of the affected code remains intact and available. copied applied

Another attribute of a monolithic repository is the layout of the codenized in a single tree. Each team has directory structure within the main tree that effectively serves project's own namespace. Each source file string_a file that optionally includes a revision number. Browsing the codebase, it is easy to understand how any source file fits into the big picture of the repository. path

The Google codebase is constantly evolving. More complex codebase modernization efforts (such as updat-

rolling out mance optimizations?) are often mancentrally by dedicated codebase half a million variable declarations or function-call sites spread across huncode. Because all projects are centrally stored, teams of specialists can do this work for the entire company rather than require individuals to develop their own tools, techniques , expertise. ing perfor aged many

(often used in conjunction with Rosie) make use of the monolithic view of Google's source to perform high-level transformations of source code. The monolithic codebase captures all dependency information. Old APIs can be removed with confidence, because it can be proven that all callers have been migrated to new APIs . A single common repository vastly simplifies these tools by ensuring atomicity of a single global view of the entire repository at any given time.

at Google employ the most up-to-date toolchains and benefit from the latest improvements in generated code and debuggability The monolithic repository provides the team with full visibility of how various languages are used at Google and allows them to do codebase-wide cleanups to prevent changes from breaking builds or greatly simplifies compiler validation; thus reducing compiler release and making it possible for Google to (typically more than 20 per year for the compilers) . cycles

Storing all source code in a common version-control repository allows codebase maintainers to efficiently ana-

the data generated by performance and regression tests run on entire Google codebase, the Compiler team tunes deFor to this centralized effort; Google's Java developers all saw their garbage collection (GC) CPU consumption decrease by more than 50% and their GC pause time decrease by 109-40% from 2014 to 2015. In addiered, it is often possible for the team to add new warnings to prevent reoccurrence. In conjunction with this scan the entire repository to find and fix other instances of the software issue being addressed, before turning to new compiler errors. Having the compiler-reject patterns that proved problematic in the past is a significant boost to Google's overall code health. Using they

Costs and trade-offs. While important to note monolithic codebase in no way implies monolithic software design; working with this model involves some downsides, as well as trade-offs, that must be considered.

These costs and trade-offs fall into three categories:

• Codebase complexity; including difficulties with code discovery; and
• Tooling investments for both development and execution;
• Effort invested in code health.

In many ways the monolithic reposyields simpler tooling since there is only one system of reference for tools is also necessary that tooling scale to the size of the repository. For instance, the Eclipse integrated development environment (IDE) to make workwith a massive codebase possible from the IDE. Google's code-indexing system supports static analysis, crossreferencing in the code-browsing tool; and rich IDE functionality for Emacs, Vim, and other development environments. These tools require ongoing investment to manage the ever-increasscale of the Google codebase. itory ing

Beyond the investment in building and maintaining scalable tooling; Google must also cover the cost of runthese systems, some of which are very computationally intensive. Much internal suite of developer tools, including the automated test infrastructure and highly scalable build infrastructure, are critical for supporting the size of the monolithic codebase. It is thus necessary to make trade-offs concerning how frequently to run this tooling to balance the cost of execution vs. the benefit of the data provided to developers. ning



An important aspect of Google culture that encourages code quality is the expectation that all code is reviewed before committed being to the repository


The monolithic model makes it easier to understand the structure of the codebase, as there is no crossing of repository boundaries between dependencies. However, as the scale increas es, code discovery can become more difficult, as standard tools like grep down. Developers must be able to explore the codebase, find relevant libraries, and see how to use them often need to see how their APIs are This requires significant investment in code search and browsing tools. However; Google has found this investment highly rewardimproving the productivity of all developers, as described in more detail bog being

Dependency-refactoring and cleanup tools are helpful, but; ideally, code owners should be able to prevent unwanted dependencies from being created in the first place. In 2011, Google started relying on the concept of API the default visibility of new APIs to 'private. This forces developers to explicitly mark APIs as appropriate for use by other teams A lesson learned from Google's experience with large monolithic repository is such mechanisms should be put in place as soon as possible to encourage more hygienic dependency structures.

Access to the whole codebase encourages extensive code sharing and reuse. Some would argue this model, which relies on the extreme scalabilit too easy to add dependencies and reduces the incentive for software developers to produce stable and wellthought-out APIs.

Duetothe easeofcreating dependencies,itis common for teams to not think about their dependency graph, making code cleanup more error-prone. Unnecessary dependencies can increase project exposure to downstream build and create additional work in building and testing: In addition; lost productivity ensues when abandoned projects that remain in the repository continue to be and maintained. updated

Several efforts at Google have sought to rein in unnecessary dependencies. Tooling exists to help identify and remove unused dependencies, or dependencies linked into the uct binary for historical or accidental reasons, that are not needed. Tooling also exists to identify underutilized dependencies, Or dependencies on large libraries that are mostly unneedOne such tool, Clipper, relies on a custom Java compiler to generate an accurate cross-reference index. It then uses the index to construct a reachability graph and determine what classes are never pendency-refactoring efforts by finding targets that are relatively easy to remove or break up. prod-

available to all Google developers has led to a culture where some teams to read their code rather than providing them with separate user documentation. There are pros and cons to this approach. No effort goes toward writing or keeping documentation up to date, but developers sometimes read more than the API code and end up relying on underhavior can create a maintenance burdeprecating features never meant to expose to users exthey

Google invests significant effort in to address plexity and dependency management. automatically detects and removes dead code, splits large refactorings and automatically assigns code reviews (as through Rosie) and marks APIs as deprecated. Human effort is required to run these tools and manage the corresponding large-scale code changes. A cost is also incurred

This model also requires teams to collaborate with one another when usopen source code. An area of the repository is reserved for storing open source code (developed at Google Or prevent dependency conflicts, as outlined earlier, it is important that only one version of open source project be available at any given time. Teams that use open source software are expected to occasionally spend time upgrading their codebase to work with newer versions of open source libraries when library upgrades are performed. ing



A developer can make a major change touching hundreds or thousands of files across the repository in a single consistent operation


by teams that need to review an ongostream of simple refactorings resulting from codebase-wide clean-ups and centralized modernization efforts . ing

repository. This effort is in collaboration with the open source Mercurial community, including contributors from other companies that value the monolithic source model.

Alternatives

As the popularity and use of distributed version control systems (DVCSs) like Git have grown, Google has considered whether to to Git as its primary version-control system. A team at Google is focused on supporting Git, which is used by Google's Android and Chrome teams main Google repository. The use of Git is important for these teams due to external partner and open source collaborations.

The and prefers developers have more and smaller repositories. A Gitcontent to one's local machine, a procedure incompatible with a large repository To move to Git-based source hosting, it would be necessary to split Google's repository into thousands of separate repositories to achieve reasonable performance. Such reorganization As comparison, Google's Git-hosted Android codebase is divided into more than 800 separate repositories. gests

Current investment by the Google source team focuses primarily on the and security of the in-house source systems. The team is also pursuing an experimental effort with Mercurial,g an open source DVCS similar to Git. The is to add scalability features to the Mercurial client so it can efficiently support a codebase the size of Google's. This would provide Google's developers with an alternapopular DVCS-style workflows in conjunction with the central goal using

Given the value gained from the extools Google has built and the many advantages of the monolithic codebase structure, it is clear that movto more and smaller repositories would not make sense for Google's main The alternative of moving to Git or any other DVCS that would not compelling for Google. isting ing

Copyright held by the authors

Conclusion

Google chose the monolithic-sourcemanagement strategy in 1999 when the existing Google codebase was migrated from CVS to Perforce. Early Google engineers maintained that a single repository was strictly better than splitting upthe codebase though at the time did not anticipate the the supporting tooling that would be built to make the scaling feasible. they

The monolithic model of source code management is not for everyone It is best suited to organizations like Google, with an open and collaborative culture. It would not work well of the codebase are private or hidden between groups.

Over the years, as the investment reto continue scaling the centralleader ship occasionally considered whether it would make sense to move from the required, Google repeatedly chose to stick with the central repository due to its advantages. quired

At Google, we have found, with some monolithic model of source management can scale successfully to a codebase with more than one thousands of users around the globe. As the scale andcomplexity ofprojects both inside and outside Google continue to hope the analysis andworkflow described in this article can benefit oth ers weighing decisions on the long-term structure for their codebases .

Acknowledgments

and former members of the Google Developer Infrastructure teams for their dedication in building and maintaining the systems referenced in this article, as well as the many article; in particular: Jon Perkins and Ingo Walther, the current Tech Leads of Piper; Kyle Lippincott and Crutcher Dunnavant, the current and former

Google's large-scale refactoring guru; Morgan Ames, Rob Siemborski, and the Piper and CitC development and support teams for their insightful review comments.

References

• 1 com/rs/perforcelimages/GoogleWhitePaper StillAllonOneServer-PerforceatScale pdf
• Chandra; T., Fikes A. and Gruber; R.E Bigtable: A distributed storage system for structured data. ACM Transactions on Computer
• 3 A Frost; distributed database. ACM Transactions on Computer Systems 31, 3 (Aug:
• K Ultra-large-scale systems. In Companion to the 2lst ACM SIGPLAN Symposium on Object-Oriented Programming Systems; Languages; and Applications 2006, 632-634
• 6 Lamport; L. Paxos made simple ACM Sigact News 32
• 5 Kemper; C. Build in the Cloud: How the Build System http:llgoogle-engtools blogspot.com/2011/08/build in-cloud-how-build-system-works.html
• Bhansali; S. Searching for build debt: Experiences managing technical debt at Google In Proceedings of the Third International Workshop on Managing Technical Debt (Zürich; Switzerland; June 2-9). IEEE
• 8 Hundt; R. Google-wide profiling: A continuous profiling infrastructure for data centers. IEEE Micro 30, 4
• 9. developers search for code: A case study. In of Software Engineering (Bergamo; Italy; 30 Sept. 4). ACM Press; New York, 2015,191-201. Aug
• and Winter; C. Tricorder: Building a program analysis Italy May 16-24). IEEE Press Piscataway; NJ, 2015 598-608.
• 12 php?title-Dependency_hell&oldid-634636715
• Wasserman; L. Scalable; example-based refactorings with Refaster. In Proceedings of the 2013 ACM Workshop on Refactoring Tools (Indianapolis, IN; Oct. 26-31). ACM Press; New York, 2013, 25-28.
• 14, Wikipedia. Linux kernel. Accessed Jan: 20, 2015; http://en wikipedia org/w/index php?title-Linux kernel&oldid-643170399
• Wan; Z Large-scale automated refactoring using Conference on Software Maintenance (Eindhoven; 548-551.

Rachel Potvin (rpotvin@googlecom) is an engineering manager at Google; Mountain View; CA.

Josh Levenberg (joshl@google com) is a software