Transcript
Page 1: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Large Drupal Site BuildsSXSW Interactive, 2015

Page 2: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Who the Heck are We?Rob Ristroph

● Technical Architect at Acquia, a digital experience company based in Boston, MA and with remote team members worldwide.

● Based in Austin, TX!● ATX Hacker Space Member● Drupal/Open Source contributor

Page 3: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Who the Heck are We?Jenn Sramek

● Sr. Engagement Manager at Acquia, based in the San Francisco Bay Area

● Manage large-scale Drupal projects for organizations and companies that are often new to Drupal

● Formerly Dir. of Operations at CivicActions

● Board President for the Haight Ashbury Food Program

Page 4: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Who the Heck are We?Alex Urevick-Ackelsberg

● Co-Founder/CEO of Zivtech● Extensive large Drupal project experience● Wears many hats

Nick Lewis● Site Architect at Zivtech● Extensive Drupal site architecture

experience● Sock Puppet Master

Page 5: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Who the Heck are You?Drupal experience?Large-scale web?Open Source?Have an active/current large-scale project?Planning one?Waterfall/Agile?Local/from other states/countries?Anything else interesting?

Page 6: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

LogisticsConnectivity● Power/Wireless

Biology● Restrooms● Food/Drink

Overall Schedule● 9:30 am - 1:30 pm

Shared Notes and Resources https://pad.lullabot.com/p/Large_Drupal_Sites_Workshop_SXSW_2015

Page 7: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Topic Overview● Project Management - what’s different about big projects ?

Technical:● Development workflow ● Custom APIs and the ecosystem of API consumers ● Scaling and performance at the enterprise level - configuring cache, finding bottlenecks ● Large-scale data migration● Monitoring

Social:● Open Source and corporate culture ● Training● Larger development team dynamics - culture of quality● Multiple teams and vendors

Tell us what to focus on

Page 8: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Defining “Large” Projects

SXSW Interactive, 2015

Page 9: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

What is “Large”?Customer has a lot of money ?Site has a lot of traffic or data ?Project build runs for a long time ?

Whatever is pushing past your comfort / experience zone in some way.

Page 10: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

What is “Large”● More than one website

○ Platform and distribution builds○ Another team uses your platform or distribution

● Multiple stakeholders○ Company vs. Divisions or Franchisees○ Technical vs. Marketing departments

● APIs and System interactions○ mail delivery○ SSO authentication, including roles and other metadata○ Web metrics

● Workflows, editorial processes● Significant migration of data

Page 11: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

What is “Large”● Content Staging

○ Workflow on steroids / between servers○ Inflexible legal requirements

● Solutions○ Content staging server and clone○ Site Preview System (SPS)○ Deploy suite of modules

● Challenging, Interesting and Fun○ Inter-disciplinary development○ Complex feature development○ Cutting-edge interaction design

Page 12: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

What is “Large”The most common single feature of a “large” project is the interaction with more team members and more stakeholders.

Page 13: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Working with Large TeamsWorking with large teams is the most likely source of project failure...

● If you are new to big projects, interacting with and managing big teams is probably the most unfamiliar area and the greatest area of risk

● Even large organizations can fail at large project implementation repeatedly

● All the same technical issues that show up in large projects can show up in smaller projects, but it is harder to maintain clarity, quality code, expectations, and scope when dealing with large teams

Page 14: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Working with Large Teams● You start with a client / customer● Add a design / strategy vendor● Add another developer team on the customer side● Add other outside developer team(s), with sub-contractor relationship● Add multiple other vendors, with no sub-contractor relationships● Separate projects with out-of-sync timelines

All still working on the same project with ultimately the same goal for success...

Page 15: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Working with Large Teams● Timeline differences● Scope differences or gaps● Resource gaps● Process differences● Deliverables differences or gaps that may be significant● Coordinating vacations/time off● Internal team mentorship, developer “ladders,” or other process for fully

on-boarding new team members (including client)

Page 16: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Working with Large TeamsMultiple parties make “DevOps” essential

Practices that just provide efficiency at the scale of one team become absolutely necessary:● Pull request workflows● Peer code review● Standardized development environments● Automated testing

Tech demo ?

Page 17: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Working with Large TeamsThe most rewarding parts of big projects● Personal growth and learning● Exposure to other teams is like having several jobs without switching

bosses● Part of a larger community

Page 18: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Getting Large ProjectsPick the projects, don’t let them pick you

● Initiate contact with particular clients and projects that interest you○ Talk with your whole team when going after something large○ Long term projects should be something people find fulfilling

● Winning by the lowest bid is a red flag● Spend much more time vetting/ do formal Discovery● Be picky; build padding into quotes

Page 19: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Picking a Vendor● Is the vendor interested enough to be committed to a long project ?● Unrealistically low bids are a red flag● Budget for discovery, and pay for it● If the vendor has not done projects of your size before:

○ Allocate more to planning○ Smallest MVP possible○ A partnered bid may mitigate some risk○ If you fall behind, cut scope instead of adding resources

Page 20: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Overall Process and Project Management

SXSW Interactive, 2015

Page 21: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Project (Program) Management

Overall...● The basics become essential

○ JIRA or another tool○ Discipline in using that tool

● Communications and Report Generation is key, and may involve “X-facing veation● Reporting is part of the effort of a Project or Program Manager● More project managers on different teams, coordination● Often multiple JIRA instances and trackers in play● On big projects, project managers can really shine and make an obvious difference,

and technical people who hate it are more likely to be able to avoid it (not necessarily a good idea ?)

Page 22: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Discovery for Large Scale Drupal

Curiosity is a Skill...

Page 23: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Discovery for Large Scale DrupalDiscovery is really a very long list of questions to answer before you start development:

- Create shared understanding of the work to be done and its priority, and start technical requirements docs

- Align everyone on the process that will be used for development, quality assurance, signoff/acceptance

- Surface any risks or potential issues that will require mitigation planning

- Document any schedule or resource considerations- “Right-size” the effort of development

Page 24: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Discovery for Large Scale DrupalTechnical Requirements:

- Content types and fields- User roles/permissions- Content and site management

workflows- Integrated service requirements (CDN,

analytics, single-sign-on, Translation Management Services, payment processing, feeds,)

- Migration details (size and complexity of database, content type/field mapping, consistency of data)

Non-Technical Requirements:- Resources including third-parties, dev team(s),

schedules, dependencies, internal technical resources, training needs

- Internal timelines and schedule constraints (vacations, major internal events, conferences, product releases)

- Security/Governance workflow and approval process (incl. clearance for those with data access and secure systems for data transmission and storage)

- Access to associated systems and tools

Page 25: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Planning for Large Scale Drupal- Timeline- Dependencies- Risks and Mitigation Planning- Resources (considering attrition of internal and external teams)- Technical/Architectural planning (including POCs/rapid prototypes)- Code management- Security management- Plan for addressing technical debt- Governance (assignment and acceptance)- Maintenance plan, development cycles, post-launch- Pre-launch/Post-launch planning

Page 26: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Planning for Large Scale Drupal● Involves LOTS of technical-specific planning (by the architect). A site

architect may do as much technical planning as the PM does project planning (code management, maintenance, release planning, etc).

● Documenting risks protects everyone, and the project● Interim maintenance and remediation● Documentation needs may differ for large projects

Page 27: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Now that all the pieces are in place...● Waterfall?

OR

● Agile?

Page 28: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Working with the “MVP”

Page 29: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Tools for Large Scale Development

SXSW Interactive, 2015

Page 30: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Tools for Large Scale DevelopmentPlan for tools to scale - know what happens to them when the project is over● Ticketing system● Document repository● Code Management tools● UX/Design Management tools● Communications Channel(s)● Training tools

Page 31: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Tools for Large Scale Development● JIRA/Rally/others - plan for tools at scale and know what happens to them

when the project is over.● Code management tools - Github, BitBucket (more integrated with JIRA

for one PR per JIRA ticket)○ Process more than product○ Onboarding dev documentation.

● UX/Design management tools - differ by firm, many use BaseCamp● Living document for coding/best practices/standard practices

Page 32: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Development● Managing multi-team interactions● Deployment tools and procedures will take up a lot of time

○ Delays between code and UAT● Deployment and DevOps can have huge (%30 ?) impacts on efficiency● Re-work is the biggest cost

Page 33: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Scalable Code Management

SXSW Interactive, 2015

Page 34: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Iterative Development● Developers cannot JUST develop, have to document how they developed.

There is a much larger team depending on clarity.○ Can effect technology choices

● Iterative can differ at the enterprise level● There can be more than one effective Agile team● Include maintenance rounds in iteration● Plan for longer lags between dev and signoff● Get sprint signoff outside of JIRA or other ticket system

Page 35: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Performance PlanningPlan for addressing performance issues early● Collect realistic requirements with context● Benchmark legacy site for reference● Test early● How do you review contributed modules/code for performance?

● Not intuition - always check● What tools to use to benchmark?

○ Make sure they match your expected traffic● How do you populate the site with enough content to test?

○ If benchmarking can be automated, just repeat as project proceeds

See:http://www.slideshare.net/jrbeeman/drupalcon-austin-planning-for-performance

Page 36: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Scaling and Performance● Scaling often matters more than Performance● Editorial interactions, cache updating are more important● Collect performance, scaling, time-to-refresh requirements just like any

other, and test for them● Optimizing Drupal for performance is a specific skill that requires Drupal

experience○ But if you can automatically test it can just like any other debuggin

Hard to designate the performance issues to one person or team; again interactions between teams are essential

Page 37: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Scaling and Performance● Scaling may mean more small sites not one big site

○ See multisite debates● Enforcing Code quality and standards

○ Peer Reviews○ Views Cache Bully and similar○ Automated testing○ Automated code sniffer

Automated testing is another thing you may not get to work on except on large projects.

Page 38: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

DeploymentFor each deployment of code, it will be important to have a consistent and documented process: Preparing for Deployment: What do you need to do to get ready? Includes reviewing all merged pull requests, updating release notes, and preparing and validating any needed deployment steps.QA Steps: What do you need to do to prepare for QA?Includes tagging the release, cloning UAT database to QA environment, smoke and unit tests.Acceptance: What do you need to do to gain client acceptance?Includes preparing a demo, regression testing, and preparing the client for UAT testing.

Page 39: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Large-scale data migration and content management

SXSW Interactive, 2015

Page 40: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Data Migration● Review and get agreement on the process● Consider data security● Clean the data● Do not “scope” it until you have mapped it● Make sure you know what is all is!● Dump and Map it! (content type by content type, field

by field)

Page 41: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Data Migration● Re-Scoping migration after mapping● Incremental migration● Differential migration● Special considerations for beta sites (where there may

be two versions of a “live” site)● Plan for deprecated systems● Plan for wrapping up with “manual” migration

(copy/paste or rewrite/edit)

Page 42: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Migrate Module● A complex migration can be a large portion of

development effort● Plan for “continuous migration”

○ Run a scheduled migration until “canonical DB switchover”

○ Iterate like any other development○ Make sure client can devote resources to checking

content throughout the process

Page 43: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Content Staging

SXSW Interactive, 2015

Page 44: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Content StagingOften discussed as part of Discovery, because many enterprise companies have structures that utilize a content staging environment.

With Drupal, this is most often not needed or can be fairly complex. Content staging, if needed, may be one of the “interesting problems to solve” that inspires your team to take on the project.

Page 45: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Content Staging● Draft / published● Workflows● Workflows with SPS - complexities multiply● Staging server, simple DB copy - limiting● Staging server, with workflow, and deploy module

If editorial staff get involved late, this is often a late-in-project scope explosion.

Page 46: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Training

SXSW Interactive, 2015

Page 47: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Site Owner / Builder Training● Talk about it during Discovery!!!● Training needs to be built into the schedule● Project Managers can become de-facto trainers● Include teaching clients to do UAT (as requisite for end-

of-project training)● Leverage available resources (Drupalize.me, Lynda.com,

Drupal.org, BuildAModule and others) for training● Plan to train the trainer

Page 48: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Training Model● Start with defined “needs”● Decide on the format of training (in-person, training

materials on separate site or tool, formal training)● Quantify trainees and role-specific tasks required of

them in order to plan and estimate● Plan to train at least some roles incrementally (often

some training will be needed for UAT )● Leverage QA and UAT testing steps for training

materials

Page 49: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Handoff and Transitions

SXSW Interactive, 2015

Page 50: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

End Goals and TransitionBig projects will involve transitioning in some way:

● To other vendors● To internal teams● From active development to maintenance mode● Re-acquiring the project later (in another phase)

Documentation and consistent development practices are essential

Page 51: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

End Goals and Transition● What happens next?● Discuss at the beginning of the project what the end of it

looks like.● Maintainability - avoid risk to support obligations● Technical debt - document it● Who will manage code after the “final” launch?● Process and tools changes?

Page 52: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

SXSW Interactive, 2015

Giving Back to the Open Source Community

Page 53: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Takin’ it to the CommunityContribution makes Drupal go!● Discuss the value of and plans for contribution during

Discovery (even if you do not plan to contribute code right away)

● Making “contributable code” a design principle○ Reduce future risk / support costs

● Celebrate contributions (and give time to devs to do it as part of the project schedule and effort)

Page 54: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Takin’ it to the CommunityHow to Contribute back to the community:● Report Bugs! ● Contribute a patch if you fix the issue● Design for Contribution● Document how the module or code works● Employ best practices● Get a peer review

Page 55: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Takin’ it to the Community

Some examples of sponsored modules:- https://www.drupal.org/project/media- https://www.drupal.org/project/i18n- https://www.drupal.org/project/commerce

Page 56: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Takin’ it to the CommunityNow Available! Attributions for contributions on Drupal.org! - https://www.drupal.org/node/2451283

Page 57: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

SXSW Interactive, 2015

Questions for Us? Or for Each Other?

Page 58: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

SXSW Interactive, 2015

Resources

Page 59: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

ResourcesShared Resources and Notes document

● https://pad.lullabot.com/p/Large_Drupal_Sites_Workshop_SXSW_2015

Page 60: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Contact UsRob Ristroph - @robgr - [email protected]

Jenn Sramek - @ideaseed - [email protected]

Page 61: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

(we would love your) feedback, please...

sxsw.feedogo.com/fdbk.do?sid=IAP42024

Page 62: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015
Page 63: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015
Page 64: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Questions Answered1. What issues should receive special attention early in life time of a large project, that need to be set up in the first

few weeks to establish best practices and culture through the longer life of a big project?2. How do you keep a larger team of developers working in close collaboration, especially as bigger projects are

more likely to have developers spread across time-zones and even different companies through multiple vendors and sub-contractors?

3. How can I assure that project and team can handle people leaving and joining the team on a regular basis, given that large Drupal projects can survive long enough to experience developer turnover, and often require specialists to join the team for a shorter phase (such as a migration expert at the start)?

4. What special efforts might be required to make sure that enhancements to Drupal and other open source projects can get outside of the organization and back to the parent project?

5. In terms of the "ecosystem" of a large enterprise drupal project, what special needs will surprise someone who has only worked on smaller projects -- APIs and Documentation that need long term maintenance, automated testing, large numbers of stakeholders and resulting approval bureaucracy?

6. What things are related to doing large projects that someone who had only done small/medium ones would not know? Partner JIRA vs. Own JIRA - Tools differences? other diffs?

Page 65: Large drupal site builds   a workshop for sxsw interactive - march 17, 2015

Iterative Development● On large projects, devs spend a lot of time figuring out where a ticket is and whether or not it can

be moved forward. Another portion is if you do not document technical debt as it accumulates, you cannot do maintenance sprints very efficiently, nor can you argue for a maintenance sprint effectively. People can be hesitant to keeping a list of technical debt. By documenting it, you protect yourself. Make it part of your process to document when you make technical decisions that create technical debt or find technical debt on code review, and get signoff as part of sprint planning.

● Estimate work only after the details of the story have been approved (so there is not scope creep on each item as scope is added that client expects the same estimate still applies).