CHECK-IN QUARTERLY...Workshop coorganizers at Stanford and EPFL 6 WikiCite organizers + substantial support from Legal, Eng-Admin, Developer Relations Hosted the annual Wiki Research

QUARTERLY CHECK-INTechnology (Services)

TECH GOAL QUADRANTA Foundation level goals C Features that we build to improve our technology

offering

B Features we build for others D Modernization, renewal and tech debt goals

The goals in each team pack are annotated using this scheme illustrate the broad trends in our priorities

Agenda

Photos (left to right)

● CTO Team● Research and Data● Design Research● Performance● Release Engineering● Security● Technical Operations

Technology (Services)

CTOJuly 2017 quarterly check-in

All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise.

https://creativecommons.org/licenses/by-sa/4.0/

CTO Team

Photos (left to right)

● Victoria Coleman - Chief Technology Officer● Joel Aufrecht - Program Manager (Technology)● Lani Goto - Project Assistant● Megan Neisler - Senior Project Coordinator● Sarah Rodlund - Senior Project Coordinator● Kevin Smith - Program Manager (Engineering)

CHECK IN TEAM/DEPT PROGRAM

GOAL: expand and strengthen our technical communities

What is your objective / workflow?

Who are you working with? What impact / deliverables are you expecting?

Program 4: Technical community building

Outcome 5: Organize Wikimedia Developer Summit

Objective 1: Developer Summit web page published four months before the event(B)

LAST QUARTER

(none)

NEXT QUARTER

Technical Collaboration Decide on event location, dates, theme, deadlines, etc. and publicize the information

STATUS: OBJECTIVE IN PROGRESS

CTOJuly 2017

WIKIMEDIA FOUNDATION ANNUAL PLAN4.5 [LINK]

https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2016-2017/revised


https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final



https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology

July, 2017 quarterly check-in

<Confidentiality notice if applicable>All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise.


Research and Data


Senior Research ScientistPrincipal Research Scientist

Director, Head of Research

Data Analyst

Software Engineer Research Fellow Research Fellow

5 f/t staff • 2 fellows • 3 contractors • 16 collaborators

2 research scientist positions we’re actively hiring for

We use research methods to design new technology and produce knowledge to understand and empower our communities

We act as the bridge between the organization, the Wikimedia movement and the academic community


OUTCOME: Annual Workshops and outreach


Who is working on this? What impact / deliverables are you expecting?

Annual workshops and outreach

(B)

LAST QUARTER

● Workshop coorganizers at Stanford and EPFL

● 6 WikiCite organizers + substantial support from Legal, Eng-Admin, Developer Relations

● Hosted the annual Wiki Research Workshop at WWW '17 in Perth, Australia

● Hosted WikiCite 2017 in Vienna, Austria● Attended AI for Good Summit in Geneva,

Switzerland● Attended the Wikimedia Hackathon in Vienna

NEXT QUARTER

● Write up a report from WikiCite

STATUS: OBJECTIVE COMPLETE

ResearchJuly 2017WIKIMEDIA

FOUNDATION ANNUAL PLAN<2> [LINK]

http://wikiworkshop.org/

http://wikiworkshop.org/

https://meta.wikimedia.org/wiki/WikiCite

http://www.itu.int/en/ITU-T/AI/Pages/201706-default.aspx






https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2016-2017/Final#Program_2:_Expand_research_capabilities_to_equip_product_development_to_better_meet_users.E2.80.99_needs

● More than 60 researchers attended our 4th annual Wiki Research Workshop at the WWW ‘17 conference in Perth

● 10 papers were accepted and the authors presented their work as part of the poster presentation

Annual Workshops and outreach (1/2)

http://wikiworkshop.org/2017/#papers

● Significant progress in laying the foundation for citations as structured data to support free knowledge

● Nearly 100 attendees from 22 countries attended a 3-day event, with 16 conference presentations, 17 summit sessions, 38 lightning talks, over 20 hackathon demos.

● Building technical partnerships with Internet Archive, Zotero, Crossref, DBLP, OCLC and relationships with funders

Annual Workshops and outreach (2/2)

https://meta.wikimedia.org/wiki/WikiCite_2017#Attendees

https://meta.wikimedia.org/wiki/WikiCite_2017/Program#May_23.2C_2017:_Conference

https://meta.wikimedia.org/wiki/WikiCite_2017/Program#May_24.2C_2017:_Summit

https://etherpad.wikimedia.org/p/WikiCite17LightningTalks

https://etherpad.wikimedia.org/p/WikiCite17LightningTalks

https://etherpad.wikimedia.org/p/WikiCite17Day3ShowAndTell


OUTCOME: 1: Innovate tools that use AI; make wiki-work more efficient.

What is your objective? Who are you working with? What impact / deliverables are you expecting?

AI as a service

(B,C)

LAST QUARTER

Scoring Platform (Amir)Research contractors (Morten Warncke-Wang, Andrew Hall, Meen Chul Kim)Lots of volunteers from various communities

● Wikidata item quality model in ORES. (Done)● Complete a research study to characterize and model

article importance. (Ongoing)● Conduct research to characterize the value of

statements in Wikidata. (Done)● Design machine-learning methods to extract and

analyze citations and their context. (Ongoing)

NEXT QUARTER

Scoring PlatformCommunity Engagement

● Deploy thresholds selection system (1.1) ● Advanced support for Albanian and Romanian WP,

basic support for Greek & Tamil WP (1.1) ● Design schema and outreach for meta ORES (2.1)


WIKIMEDIA FOUNDATION ANNUAL PLAN<5> [LINK]Research → Scoring PlatformJuly 2017






https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Draft/Programs/Technology#Program_5._Scoring_Platform_.28ORES.29

CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION ANNUAL PLAN

Deliverable 1: Wikidata item quality model in ORES. (DONE)

Completed outreach and labeling campaign. Trained and deployed model. Model shows a high level of fitness.

Deliverable 2: Complete a research study to characterize and model article importance. (ONGOING)

Literature review and modeling work complete. Outreach to WikiProjects (the owners of “importance”) has been substantial. Many lessons learned about the meaning of “importance”. Missing infrastructure for ORES deployment identified and tasked.

Deliverable 3: Conduct research to characterize the value of statements in Wikidata. (DONE)

Analysis of value of entire entities (more coarse than *statements*) complete. Dataset release complete. Statement tracking blocked on Wikidata engineering. Also, the contract was substantially delayed so work started ~1 month late.

Deliverable 4: Design machine-learning methods to extract and analyze citations and their context (ONGOING)

Citation extraction complete and extraction schema standardized. Report in progress for GROUP’18. Contract delayed for almost the entire quarter. Machine learning work is delayed but in progress.

Research → Scoring PlatformJuly 2017 <5> [LINK]






https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Draft/Programs/Technology#Program_5._Scoring_Platform_.28ORES.29

● The Keilana Effect (blog and paper accepted)● New models for:

○ English Wikipedia (Draft quality)○ Estonian Wikipedia○ Finnish Wikipedia○ Hebrew Wikipedia○ Korean Wikipedia○ Wikidata (Item quality)

● Mentorship @ Wikimedia Hackathon brought in new volunteers from:

○ Tamil Wikipedia○ Greek Wikipedia○ German Wikipedia○ Finnish Wikipedia

● Worked with WMF Product to support new RC Filters

○ E.g. a study of overlap between “damaging” and “goodfaith” predictions for newcomers

https://blog.wikimedia.org/2017/03/07/the-keilana-effect/

https://meta.wikimedia.org/wiki/Research:Interpolating_quality_dynamics_in_Wikipedia_and_demonstrating_the_Keilana_Effect

https://meta.wikimedia.org/wiki/Research_talk:Automated_classification_of_edit_quality/Work_log/2017-05-04

● Initiated research to expand the results of Why We Read Wikipedia to 14 languages. We worked with the community to prepare the surveys for their languages and ran them. The result is a collection of 254,000 responses that we are analyzing in Q1 and Q2.

● Continued research on building recommendation systems for helping editathon organizers and newcomers with automatic template generation. The focus has been on deriving an algorithm that can turn the category graph of Wikipedia to a hierarchical graph that can be read by machines. First results are available, but much more improvement needed for the algorithm to be usable.

● Nearly completed productization of the Article Recommendation API, to be completed in Q1.● Hosted an AMA on Reddit on AI and community dynamics at Wikimedia● Rebuttal to “Even Good Bots Fight” submitted to CSCW (positive initial reviews). Blog post in

progress. See inane media coverage (e.g. The Growing Problem of Bots that Fight Online)

Other Q4 accomplishments

https://meta.wikimedia.org/wiki/Research:Characterizing_Wikipedia_Reader_Behaviour/Robustness_across_languages

https://meta.wikimedia.org/wiki/Research:Expanding_Wikipedia_stubs_across_languages

https://www.reddit.com/r/IAmA/comments/6epiid/im_the_principal_research_scientist_at_the/

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0171774

https://www.technologyreview.com/s/602421/the-growing-problem-of-bots-that-fight-online/


Design ResearchJuly 2017 quarterly check-in



1 Manager/ Lead Design Researcher1 Senior Design Researcher


GOAL: Use research-centered approach to drive product development



New Editor Experiences Research

● Lit review of prior research about new editors

● Contextual inquiry: new editor retention in South Korea and Czech Republic

LAST QUARTER

WMF Team (Editing and Design Research (two people)Reboot Team (two peopleFour local researchers (two from South Korea, two from Czech Republic)

● Lit review, analysis of prior research about new editors: done

● Contextual inquiry in South Korea: done● Contextual inquiry in Czech Republic: done

NEXT QUARTER

People on Editing, Communications, Community Engagement and Research teams.

● Synthesis workshop with Reboot and sharing -done ● Complete report from both contextual inquiries● Begin collaboratively applying findings● Begin Open Access release of corpus


Design ResearchJuly 2017WIKIMEDIA

FOUNDATION ANNUAL PLAN1 [LINK]








GOAL: Use research-centered approach to drive product development



New Readers: ● Heuristic evaluation of

Kiwix App (WikiMed)● Provide

recommendations for improvement to Kiwix team

● Contribute Design Research perspectives to Affordability and Awareness tracks as needed

LAST QUARTER

New Readers Reading (Anne, Toby, Nirzar)Kiwix (Emmanuel)Communication (Zack)Partnerships (Jack, Jorge, Ravi)

● Completed Heuristic evaluation of Kiwix -done● Provide recommendations for improvements -done● Clarification of findings with Emmanuel at Kiwix -

done● Collaborated with Awareness team on choosing media

partners in Nigeria and India - done

NEXT QUARTER

TBD



FOUNDATION ANNUAL PLAN1 [LINK]








GOAL 1: Build an open infrastructure



Understand editing culture

LAST QUARTER

Structured Data Program, External Collaborator (Carnegie Mellon)

● Analyze GLAM user workshop data (NOT DONE)● Publish Teahouse study results (NOT DONE)

NEXT QUARTER

Structured Data Program, External Collaborator (Carnegie Mellon)

● Develop interview protocol; interview 6-8 GLAM stakeholders; draft user stories; share initial findings at Structured Data offsite

● Publish Teahouse study results and data in peer-reviewed venue and open repos; share with anti-harassment team

STATUS: OBJECTIVE IN PROGRESS (Delayed)


FOUNDATION ANNUAL PLAN2. Expand Research capabilities








GOAL 3: Perform evaluative design research



Conduct evaluative design research; Expand capacity for standardized iterative user testing

LAST QUARTER

Reading (Apps) ● Complete a user study with crowdworkers to compare ‘top read’ and ‘trending’ lists of article recommendations on mobile app

NEXT QUARTER

Reading (Apps/Services) ● Complete 1-2 additional user studies of ‘trending’ edits model in mobile features

● Reduce per-study overhead by automating data cleaning process

● Document the method











GOAL 4: Improve communication and access of all research projects



Create a research minisite that would act as a landing page for all resources related to research at the organization

LAST QUARTER

External contractor Scope the project, hire a content writer, and draft native content for the Research Landing Page

NEXT QUARTER

External contractor, Reading design, Communication

Finalize all native content, build the Research landing page, write blog posts

STATUS : OBJECTIVE IN PROGRESS










PerformanceJuly 2017 quarterly check-in



Gilles Dubuc Aaron Schulz Peter Hedenskog Timo Tijhof

Performance Team

Photos (left to right)Myleen Hollero / CC BY-SA-3.0Myleen Hollero / CC BY-SA-3.0Myleen Hollero / CC BY-SA-3.0

Sage Ross / CC BY-SA 2.0


GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users



Improve page delivery (D)

LAST QUARTER

We can’t look at First Paint over the course of a year (historical data is tainted). We didn’t improve it by the targeted 15%, however, it has remained stable since the metrics have been fixed.

Save Timing was improved by 25% for the 99th percentile, and by 15% for the median. Annual objective was 25%.

NEXT QUARTER

We won’t be committing to arbitrary performance targets like this anymore, since it’s impossible to know ahead of time what is actually feasible.

Efforts will continue as part of FY17-18 Program 1.

STATUS: OBJECTIVE PARTIALLY COMPLETE

PerformanceJuly 2017

WIKIMEDIA FOUNDATION ANNUAL PLAN4.1.1 [LINK]






https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2016-2017/Final#Program_4:_Improve_site_performance





Improve site availability (B)

LAST QUARTER

Technical Operations Evaluated Dynomite as a potential replacement for Mcrouter for active/active datacenter support of our caching. (T156938)

NEXT QUARTER

Will continue as part of FY17-18 Program 1.



WIKIMEDIA FOUNDATION ANNUAL PLAN4.1.3 [LINK]

https://phabricator.wikimedia.org/T156938











Migrate media thumbnailing from MediaWiki to a new dedicated open-source service (D)

LAST QUARTER

Technical Operations Thumbor now handles all production traffic of thumbnails for all public wikis. (T121388)

NEXT QUARTER
















Set up monitoring of performance metrics with alerts and notifications (C)

LAST QUARTER

Performance alerts extended to real user metrics (Navigation Timing) and article Save Timing.Caught several incidents already.

NEXT QUARTER
















Research isolated browser testing environment and proxy (C)

LAST QUARTER

Found a synthetic software stack that allows us to achieve very small variance between runs.

NEXT QUARTER

Work will continue in Q2 of FY 17-18, once the fork of an open source project we’ve been working with (created by a PHD student) is publicly released. Will continue as part of FY17-18 Program 1.















Communicate about our performance work (B)

LAST QUARTER

Communications Published multiple blog posts, one of which was picked up by Communications for the main blog.

Improved the performance.wikimedia.org portal.

NEXT QUARTER





https://phabricator.wikimedia.org/phame/blog/view/7/








Screenshot from twitter.com

Screenshot from twitter.com

● 4 blog posts● 10 incident investigations● 1 extension review● 2 org-wide perf audits

Other tasks and unplanned work

FY 2017-2018

NASA


OUTCOME: All production sites and services maintain current levels of availability or better



Assist in the architectural design of new services and making them operate at scale (B)

LAST QUARTER

NEXT QUARTER

Technical Operations Follow-up bugfixes and improvements after Thumbor deployment to production

STATUS: OBJECTIVE ON TRACK









https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_1:_Availability.2C_performance.2C_and_maintenance


OUTCOME: All our users consistently experience systems that perform well



Catch and address performance regressions in a timely fashion through automation (B)

LAST QUARTER

NEXT QUARTER

Technical Operations Test user performance from Asia to validate changes when the Asia Cache PoP goes live















Modernize our performance toolset. We will measure performance metrics that are closer to what users experience (C)

LAST QUARTER

NEXT QUARTER

Rework Navigation Timing metrics to make them stackable

Add metrics for master queries on HTTP GET/HEAD
















Infrastructural improvements aiming for performance gains (D)

LAST QUARTER

NEXT QUARTER

Preload base module requests

Upgrade to jQuery 3













OUTCOME: Our audiences enjoy improved MediaWiki and REST API availability



MediaWiki support for having read-only “read” requests (GET/HEAD) be routed to other data centers (B)

LAST QUARTER

NEXT QUARTER

Technical Operations Enable HTTPS for Swift and MariaDB clients

Deploy mcrouter on Beta












https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_8._Multi-datacenter_support


Release EngineeringJuly 2017 quarterly check-in



CHECK IN TEAM/DEPT PROGRAMStreamlined Services DeliveryJuly 2017

WIKIMEDIA FOUNDATION ANNUAL PLAN6

Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment.What is your objective / workflow?


Build a runnable image from tested and confirmed code that is suitable for promotion.(C - features)

LAST QUARTER

Operations, Services ● define 'tested and confirmed' / determine the entry-point for building a container

● generate or consume a docker file for building● build image, upload and tag it in docker registry

Set up a continuous integration and deployment pipeline (C - features)

NEXT QUARTER

Operations, Services ● Define functional tests for Mathoid running on the staging Kubernetes cluster for use in future gating decisions

● Define method for monitoring and to the above functional tests

STATUS: OBJECTIVE IN PROGRESS, ON TRACK







GOAL: Train 2.0 / Unified deployment tooling



Move MW+Extension deploys to scap3 (D - tech debt)

LAST QUARTER

Operations Integrate scap with etcd/pybal to automatically depool and repool servers

Not complete - pending cross-team code review. ETA of early July.

Deprecate use of Trebuchet across production (D - tech debt)

NEXT QUARTER

Operations, Security, Discovery, Cloud Services

Trebuchet is our deprecated deployment tool and is no longer supported, it should be removed everywhere


Release EngineeringJuly 2017WIKIMEDIA

FOUNDATION ANNUAL PLANNone







GOAL: Consistent MediaWiki third-party releasesWhat is your objective / workflow?


Release MediaWiki 1.29(D - Tech debt)

LAST QUARTER

All MW developers A quality MediaWiki 1.29 is successfully released

Late - Unexpected blocker issues arose late in the process. Released July 13th.

Improve release process (D - Tech debt)

NEXT QUARTER

Security, All MW developers Finish implementation of nightly build server



FOUNDATION ANNUAL PLANNone







GOAL: Effective and easy-to-use testing infrastructure toolingWhat is your objective / workflow?


Develop and migrate to a Javascript-based browser testing stack(C - improve our own feature set; D - Tech debt)

LAST QUARTER

NEXT QUARTER

Wikidata, CirrusSearch, All MW developers

Migrate majority of developers to Javascript-based browser test framework (webdriver.io)



FOUNDATION ANNUAL PLAN1






● free some repositories from their unintended chain to mediawiki/core

● Add support for skin and extension dependencies in new skin unit test

● Convert all repos to use npm Jenkins job with jsonlint and eslint

● Raise priority for operations-mw-config-composer-hhvm-jessie from the gate-and-submit pipeline

● Depool precise jenkins instances● Enable embedding of videos from

Commons on Phabricator● scap now allows restarting multiple

services

● Deploy logstash/plugins with scap3● Set up a warm failover in Dallas/codfw for

Gerrit.● Automatically clean up unused wmfXX

versions● setup automatic deletion of old l10nupdate● Release MediaWiki 1.28.1/1.27.2/1.23.16● Building Better Software (Hack-a-thon session)● Improved scap rollback behavior● Migrate NodeJS Nodepool jobs from Trusty to

Jessie● Alert when Zuul/Gearman queue is stalled● WebdriverIO CREDIT showcase

Other Successes


SecurityJuly 2017 quarterly check-in



Darian Anthony Patrick

Brian Wolff Sam Reed


GOAL: Improve wiki account authentication



Improve wiki account authentication (C, D)

LAST QUARTER

Community Tech - Improve security messages to users (LoginNotify extension being tested)

NEXT QUARTER

Editing - Support Community Tech in deployment of LoginNotify- Provide insight into active sessions


SecurityJuly 2017WIKIMEDIA

FOUNDATION ANNUAL PLAN3 & 4 [LINK]






https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2016-2017/Final#Program_3:_Improve_site_reliability


GOAL: Improve MediaWiki security release process



Improve MediaWiki security release process (C)

NEXT QUARTER

Release Engineering, Tech. Ops. - Adherence to a stable, regular security release timeline- Improved ability to make emergency releases with lower impact to Security and RelEng teams- Earlier detection of unreleased patch inapplicability as code development progresses


SecurityJuly 2017WIKIMEDIA

FOUNDATION ANNUAL PLAN2 & 3 [LINK]








Technical OperationsJuly 2017 quarterly check-in



Mark Bergsma Riccardo Coccioli

Alexandros Kosiaris

Jaime Crespo Filippo Giunchendi Ariel Glenn

Giuseppe Lavagetto Faidon Liambotis

Rob Halsell

Manuel Arostegui Moritz Mühlenhoff Daniel Zahn

Chris Johnson Papaul Tshibamba Brandon Black Emanuele Rocca Arzhel Younsi

Keith Herron





Perform a datacenter switchover (A)

LAST QUARTER

● MediaWiki● Performance● Release Engineering● Services (RESTbase)● Discovery (Search)● Parsing (Parsoid)● Analytics (EventBus)● Community Liaisons

● Successfully switch backend traffic (MediaWiki, Swift, ElasticSearch, RESTBase, and Parsoid) to be served from codfw with no downtime and reduced read-only time

● Serve the site from codfw for a longer period of time than the last switchover (at least 2 weeks)


TechOps / Core OpsJuly 2017WIKIMEDIA

FOUNDATION ANNUAL PLAN

8.4.3 Improve site performance & reliability









GOAL: Keep Wikimedia's sites and services running responsibly, reliably, and securely, including protection against and response to outages, data loss, and breaches


Who are you working with?

What impact / deliverables are you expecting?

Reduce Swift technical debt (D)

LAST QUARTER

● Support diversified storage policies such as different levels of replication and low-latency containers

● Upgrade the Swift infrastructure to run using Debian jessie (or later)

● Allow rate-limited container operations for better isolation among Swift users and use cases

● Delete non-used/non-requested thumbnail sizes periodically to reclaim space




8.3.1.1 Operational excellence








CHECK IN

July 2017TEAM/DEPT

Technical Operations

● Thumbnail disk space reduced by ~20 TB

● Replication factor 6● 120 TB raw space saved● 3 storage backend

machines


GOAL: Streamlined service delivery




Prepare to serve applications from Kubernetes (C)

LAST QUARTER

Services, Release Engineering

● Eliminate SPOFs in the existing eqiad infrastructure● Expand the infrastructure to codfw● Prepare and maintain base container images● Define a production process to keep images up-to-date● Design/implement a Kubernetes-based staging environment (s)

NEXT QUARTER

Services, Release Engineering

● Implement a pod networking policy approach● Upgrade to Kubernetes >= 1.5● Standardize on a "default" pod setup● Experiment with ingress solutions (stretch)



FOUNDATION ANNUAL PLAN(none in FY16-17)

6: Streamlined service delivery






https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_6._Streamlined_service_delivery





Deploy Asian caching POP (A)

LAST QUARTER

Legal, Finance, DC Ops ● Finalize and sign contract with the data center vendor

● Finalize system specifications and purchase the necessary hardware

● Prepare and deploy configuration changes needed for the new systems

● Allocate the data center's IPv4/IPv6 address space

A vendor & site recommendation along with selection criteria has now been provided to and approved by the C-team, but no contract has been signed yet.

STATUS: OBJECTIVE AT RISK

TechOps / TrafficJuly 2017WIKIMEDIA


8.4.1.4 Improve site performance












Deploy Asian caching POP (A)

NEXT QUARTER

Legal, Finance, DC Ops, Partnerships

Continue Asia Cache PoP procurement, installation, and configuration tasks

● Finish up trailing purchasing tasks from previous quarter (DC, hardware, network links, etc).

● Procure at least one transit or peering link to help advance address space issue

● Physically install all hardware● Acquire address space & communicate it to

Wikipedia Zero partners (via the Zero team)● Turn up network links (stretch)● Configure network devices and hosts (stretch)

STATUS: OBJECTIVE AT RISK

TechOps / TrafficJuly 2017WIKIMEDIA


8.4.1.4 Improve site performance









GOAL: We will maintain the availability of Wikimedia’s sites and services for our global audiences and ensure they’re running reliably, securely, and with high performance.




Outcome 1: All production sites and services maintain current levels of availability or better.Objective 1: Deploy, update, configure, and maintain production services (Traffic infrastructure, databases & storage, MediaWiki application servers, (micro)services, network, and miscellaneous sites & services) (C, D)

NEXT QUARTER

Release Engineering, WMCS

Remove Salt from our infrastructure● Port debdeploy to Cumin● Migrate the reimage script to Cumin● Remove support for the Trebuchet deployment system● Remove Salt from production & WMCS




1 Availability, performance, and maintenance













Outcome 1: All production sites and services maintain current levels of availability or better.Objective 1: Deploy, update, configure, and maintain production services (Traffic infrastructure, databases & storage, MediaWiki application servers, (micro)services, network, and miscellaneous sites & services) (C, D)

NEXT QUARTER

Release Engineering Prepare for Puppet 4● Support directory environments in our Puppet infrastructure

and add an environment that uses Puppet's future parser● Switch at least 3 node groups to the future parser

environment● Force both current and future parser for every test in the

puppet-compiler● Integrate puppet-compiler with the Continuous Integration

infrastructure (task T166066) (stretch)● Speed up CI for operations/puppet (task T166888) and add

future parser validation



















Outcome 1: All production sites and services maintain current levels of availability or better.Objective 1: Deploy, update, configure, and maintain production services (Traffic infrastructure, databases & storage, MediaWiki application servers, (micro)services, network, and miscellaneous sites & services) (D)

NEXT QUARTER

Improve database backups' coverage, monitoring and data recovery time (part 1)

● Adjust configuration management manifests to support MariaDB multi-instances

● Migrate at least 2 instances on 1 dbstore host to the new multi-instance setup

● Research backup storage options and prepare a design document

● Investigate and experiment with replacements of mysqldump













Technical OperationsJuly 2017 8.3.1.1 Operational excellence

● Migrated application servers to HHVM 3.18

● Removed torrus, replaced with prometheus

● Expanded mail server monitoring● Migrated eqiad row D to new switches● Upgraded switch firmware in data center

rows● mariadb-systemd integration● stretch support for mariadb● MariaDB 10.1 compatibility● ~200000 schema changes adding PKs● All core servers/servers with core data

support TLS connections and use it for replication

● Backup server (dbstore1001) migrated to jessie and upgraded its MariaDB version.

● Refactored mediawiki comment/revision table

● Reduced mediawiki max query to 60s● Rebuilt pc hosts with file-per-table● Migrated tendril to dedicated VMs● Migrated Codfw mediawiki db to jessie● Ganeti installations became multi-row

increasing availability● poolcounter hardware VMized reducing

costs● url-downloader made redundant● Patched an etherpad vulnerability● Upgraded IRC bots to newer versions of

supporting libraries● Migrated multiple puppet roles to the

profile pattern● Started work on the puppet future parser



Technical OperationsJuly 2017

● Enabled structured facts in puppet● Upgraded apertium with Translation team● Migrated various deployed softwares to

scap3 (with services, releng)● Upgraded OTRS to 5.0.19 and then to

5.0.20● Upgraded librenms to latest version● Made icinga redundant in a master/slave

pattern● 2/3 of the jessie servers were migrated to

Linux 4.9 (Stretch kernel)● Reconfigured wikidata xml dumps to run

much faster● Enabled TCP BBR on cache nodes● Implemented rate limiting of HTTP

requests● Prometheus monitoring of cache systems

and load balancers

● Production load balancer (pybal): added unit tests, solidified BGP

● Improved network monitoring● Upgraded Prometheus to bare metal● Improved investigation of HTTP errors

with Logstash

8.3.1.1 Operational excellence


Documents

CHECK-IN QUARTERLY...Workshop coorganizers at Stanford and EPFL 6 WikiCite organizers + substantial support from Legal, Eng-Admin, Developer Relations Hosted the annual Wiki Research