1. 1. Engineering Culture & Infrastructure Team Programming, Code Review, Robotization, Pipeline and Feedback 2016.12.29 Schubert Zhang
2. 2. Engineering infrastructure (technologies, processes, tools, and culture) that enable engineers to innovate and release software continuously with agility, quality, and productivity. This keynote gives an overview of the overall architecture, workow, and scale.
3. 3. Agenda What? Why? Google Practices Code Review Gerrit, Team Programming Big-Picture of Engineering Pipeline Personal Stories of being a Software Engineer
4. 4. What? Why?
5. 5. Bug
6. 6. Innovation & Creation Innovation Factory We take our jobs to be innovators and we are failing if we are not innovating quickly enough! Eric Schmidt, Google CEO Patrick Copeland, the senior director of Engineering Productivity and the top of the testing food chain at Google
7. 7. Quality Quality Quality Review Instant Feedback Patrick Copeland, upgraded by Schubert Zhang
8. 8. Balance Process-heavy vs. Process-less Burdensome and anti-creative Culture vs. Heroic Culture (contrary to the creative nature of innovation) (unable to repeatedly deliver) There needs to be balanced! We need to focus on staying airborne for the long term. We need to motivate smart minds to solve hard problems and deliver rich features to customers. Uniform Workow Freeform To be agile!
9. 9. process-driven no process wholesale long-term Engineering Culture
10. 10. The entire product team is responsible for quality, and is judged on their ability to enable innovation, anticipate problems, make plans, and implement high quality software. Under a common-sense workow, teams adopt processes that are in their own self interest and that allow them to focus on innovation.
11. 11. development teams write good tests and do more review because they care about the products more time to spend writing / innovating features and less in later phases debugging to x bugs. socially code review and check-in practice.
12. 12. The Relationship btw Culture & Infrastructure Culture Infrastructure Culture Github OpenSource Culture QuoraTwitter Culture Dont make me think Culture
13. 13. The infrastructure and workow with the following principles Speed: All test, review and analysis systems need to return results very fast. If it takes too long, engineers will either ignore or not bother looking for that data. Feedback: The focus of test, review systems must be on high quality feedback. We want engineers to keep code at production quality at all times, not adding time to x code that was broken earlier. Simplicity: Engineers should not have to understand how the underlying build and test systems work. All data and feedback must be easy to understand, integrated into commonly-used productivity tools, and presented in a workow that allows them to take appropriate action. Extensible: The infrastructure pipeline / loop / architecture makes a common sense framework, teams or individuals can add new tools or plugins at any stage. Enjoy: Dont make me think, just enjoy it. By reducing the window of opportunity for bad code to go unnoticed, overall debugging and bug isolation time is radically reduced. The net result is that the engineering teams no longer sink hours into debugging build problems and test failures.
14. 14. We need an approach that provided developers nearly instant feedback on every code check-in.
15. 15. Imagined Basic System for Engineering let it web-based with user experience let it robotization with instant feedback
16. 16. The Mission Accelerating FangDD Productivity at Product and Engineering Creation Google's Innovation Factory: Testing, Culture, And Infrastructure
17. 17. Google Practices
18. 18. Single Large Code Repository Googles monolithic cloud repository provides a common source of truth for tens of thousands of developers around the world. Foundation of many of Googles developer workows. Pros: unied versioning, extensive code sharing, simplied dependency management, atomic changes, large-scale refactoring, collaboration across teams, exible code ownership, code visibility, clear tree structure providing implicit namespacing. Cons: Heavy in-house development of the support infrastructure systems, complexity. Distributed over 10 Data-Center: Paxos to guarantee consistency across replicas (Bigtable -> Spanner) Most trafc originates from Googles distributed build-and-test systems. (Powerful cloud-based build system and test infrastructure, tools.) Google 1999 Perforce Repository Git Git Git
19. 19. Google Code Infrastructure Other stuff: Mercurial GFLAG / GMOCK / GTEST for C++ Bazel Build System: http://www.bazel.io Google CodeStyle Guide Google Code Repository System in Cloud Perforce -> Piper (Workow, ACL) On Google Infrastructure, Bigtable -> Spanner Heuristics CitC local client Heuristics CitC local client Heuristics CitC local client Code Writing Critique Web-based Code Review Instant Feedback Code Search Codestyle Checker Codebase Health (Rosie) Maintainer tools Refaster, ClangMR, Clipper Static Analysis (Tricorder) Code Review Tools (Mandrian->Citique) Automatic Build Infrastructure Automatic Test Infrastructure Code Browsing Search, Editing Tools Unit-Test Verication test coverage world-wide developers Google presubmit infrastructure provides automated testing and analysis of changes before they are added to the codebase.
20. 20. Presubmit Workow Automatic Testing, Analysis, and Code Review Culture: All code is reviewed before being committed to the repository. No code, for any product, for any project, gets checked in until it gets a positive review. Many automatic checks and verication are run presubmit. LGTM: Looks Good To Me and automatic analysis, automatic testing
21. 21. Trunk-based Development (one source tree, no branching) With the exception of a few core systems, all of Google works from a single source tree and from head. changes are made to repos in a single, serial ordering Developers work on a consistent view of codebase Avoids the painful merges that often occurs for long-lived branches Development on branches is unusual, branches are typically used for releases. Release branches are cut from a specic revision of the repository. Bug xes and enhancements that must be added to a release are typically developed on mainline, then cherry-picked into the release branch. Use of long-lived branches with parallel development on the branch and mainline is exceedingly rare. When new features are developed, both new and old code paths commonly exist simultaneously, controlled through the use of conditional ags, through conguration. (eg. GFLAG, refers to RFC1122) A small set of very low-level core libraries uses a mechanism similar to a development branch to enforce additional testing before new versions are exposed to client code. ()
22. 22. Code Review Principles All change lists (CLs) must be reviewed First thing they check about readability. Documentation for classes and functions and module. Code will be linted (automatically) to make it error free. Check whether code written use best practices and use low resources. Reusability Any CL can be reviewed by any engineer at Google. The author may get review comments from the guys who reviewed le. Once review done he will get LGTM. Each directory / projects has a list of owners Its not a process, its a habit, or culture, and they are all proud of it.
23. 23. Open and Collaborative Culture Over 99% of code is visible to all full-time Google engineers. Anyone can use, study, review, comment or contribute any code. Code Ownership (Tree based) Cross team collaboration The fact that most Google code is available to all Google developers has led to a culture where some teams expect other developers to read their code rather than providing them with separate user documentation.
24. 24. How Code Review Works at Google
25. 25. Originally Code Review by email then Mondrian from 2006, a web-based Code Review Workbench
26. 26. Organization Flat & Autonomous Bottom-up Social Teams are aligned along business lines / focus areas / scrums Projects live and die based on free-market Darwinism, projects must produce value to survive.
27. 27. For Your Reading Googles Innovation Factory: Testing, Culture, And Infrastructure Why Google Stores Billions of Lines of Code in a Single Repository Book: How Google Test Software Quora: What is Google's internal code review policy/process? Quora: How does Google manage to solve the compilation, build, code review process? GoogleTechTalks on Youtube: Mondrian, Code Review On The Web GoogleTechTalks on Youtube: Using Gerrit to enhance your Git What we learned from Google: code reviews arent just for catching bugs
28. 28. Code Review
29. 29. OA
30. 30. Catch bugs (the latest value, trivial) Shared code style Build stability Social (with social psychology) Knowledge sharing (mentoring, learning, and avoid SPoF) Early feedback Team engagement Qualitative code selection
31. 31. Pitfalls and Mistakes of Inexperienced Reviewers (bad experiences, cause a lot of trouble) To nd all bugs () Judging code by whether it's what the reviewer would have written. (hard feelings and frustration, ! ) Feels obligated to say something. () Speed: shouldn't rush through, but ASAP () http://goodmath.scientopia.org/2011/07/06/things-everyone-should-do-code-review/
32. 32. Code Review is a misleading Goal is cooperation and engineering social, not bug-nding Co-located teams, Collaboration, Share, Discuss Open, Transparency Social The most positive result of a Code Review cycle is, in addition to higher code quality standards, it is about getting different people to look at the version of the solution and having a constructive discussion around it.
33. 33.
34. 34. Psychological Effects If you're programming and you know that your coworkers are going to look at your code, you program differently. You'll write code that's neater, better documented, and better organized -- because you'll know that people who's opinions you care about will be looking at your code. Without review, you know that people will look at code eventually. But because it's not immediate, it doesn't have the same sense of urgency, and it doesn't have the same feeling of personal judgement.
35. 35. Pre-launch Review, and review anywhere to ensure that products answer common sense questions before release Is the design secure and customer data private? Will the service scale with the anticipated load? Does the UI meet standards? What are the data center utilization estimates? What are the latency estimates?
36. 36. For Your Reading Things Everyone Should Do: Code Review Things Everyone Should Do: Coding Standards What we learned from Google: code reviews arent just for catching bugs Ways to Make Code Reviews More Effective Code Review code review code review Microsoft: Code Reviews Do Not Find Bugs - How the Current Code Review Best Practice Slows Us Down
37. 37. Gerrit, Team Programming
38. 38. Gerrit Practices Google Android, Wikipedia, Spotify, Baidu, Eclipse, LibreOfce, Openstack, SAP, HP, Motorola, SONY Mobile, Intel, Qualcomm Palantir, eBay, , and many Startups Initially founded by Shawn Pearce (Google) as tool for the Android OS Development, forked from Google Mandrian Google Mondrian Google Rietveld and Critique Gerrit Google for Android
39. 39. Covers the Benets of Team Programming Catch bugs (the latest value, trivial) Shared code style Build stability Social (with social psychology) Knowledge sharing (mentoring, learning, and avoid SPoF) Early feedback Team engagement Qualitative code selection refers to previous page
40. 40. Gerrit Jargons Project Change (CL=Change List) Patch Set Label Code Review Submit change Merge change Abandon change
41. 41. Roles in Code Review Contributor Reviewer Committer Maintainer
42. 42. Workow refers to previous page
43. 43. Gerrit magic branch for Collaboration
44. 44. Review Labels
45. 45. Group and Project Hierarchy then powerful rules for tuning
46. 46. Gerrit Style vs. Gitlab(Github) Style Gerrit Github for a one-man bands project Contributor 1. fork the project 2. clone the project 3. create commit 4. push to the forked project and 5. create pull request 1. clone the project 2. create commit, 3. push for review
47. 47. BTW: Github, Gerrit Gerrits permission scheme (Project + Group + Permission) Subject: Single or multiple sets of people identied by Gerrit Action: The ability to allow or deny a specic operation. Resource: Single or multiple sets of Gerrit objects (typically Git reference) that are controlled by the permission Githubs Organization + Team + People
48. 48. Review Etiquette () Review as discussion board Review each le Its all about the code Always answer all comments Use code in comments One change, one thing Use topics commit message jargons for commit message [Minor], [Major], [Crucial] [RFC] (Request For Comments) [WIP] (Work In Progress) [TBR] (To Be Released) [TBL] (To Be Launched)
49. 49. Android World-Wide Collaborative Development
50. 50. robotization Life of a Patch in Android Project
51. 51. FangDD Collaborative Development
52. 52. Review
53. 53. Review
54. 54. For Your Reading Handbook: Learning Gerrit Code Review Book Review: Learning Gerrit Code Review Gerrit Code Review - A Quick Introduction About Gerrit and it's Google Gene (FanQiang) GerritForge: Gerrit Code Review Enterprise Andoid - Submit Patchs (FanQiang) Android - Life of a Patch (FanQiang) GoogleTechTalks on Youtube: Using Gerrit to enhance your Git Quora: What is Google's internal code review policy/process? A successful Git branching model Top 10 tools for code review and 15 Best Code Review Tools for Developers
55. 55. Our Big-Picture of Engineering Pipeline / Loop
56. 56. Code Heuristics SWE SET TE Ops Bots Crowd Code Workow Code
57. 57. Shenzhen Ofce Local Git Repos Shanghai Ofce Local Git Repos sync sync Shenzhen Local CI for Dev & Test Robots Shanghai Local CI for Dev & Test Robots fetch fetch Developers at Shenzhen Developers at ShanghaiDevelopers Anywhere Collaborative work Collaborative work Collaborative work Codebase in the Cloud Git Repos in IDC (Review Board) Online Services IDC
58. 58. + Defect Robot Level Teamcode
59. 59. For Your Reading Static code analysis: do it the right way Analysis vs. Preview vs. Incremental Preview in SonarQube Google Engineering & Technology List of tools for static code analysis
60. 60. Personal Stories of being a Software Engineer
61. 61. Study to Write Bug-free Programs Steve Maguire Rob Pike Brian Kernighan Jerry Weinberg
62. 62. readability code structure, clarity, docs in code
63. 63. Be Proud of Being a Software Engineer @Baidu, 2013 in 37 years old proud
64. 64. http://issues.apache.org/jira/browse/HBASE Try to Contribute Open-Source Projects
65. 65. & Why programmers should write Hire great writes
66. 66. vs.
67. 67. Hands-On -- ->
68. 68. to be a engineer the route to be a rich engineer Thank You!
69. 69. 2017 ( + )