Community source:the good, the bad

and the ugly

Stephen Marquard,

panellists and

participants

July 2008

+

Looking for buried gold in the midst of two warring factions …

Format

• What is Community Source?• Community Poll• Reflections from UCT on our Sakai experiences• Reflections on Sakai and community processes• Opening comments from panel members• Audience discussion with the panel

…Let’s talk about how our community has worked in the past and how we’d like it to work in the future.

Disclaimer

• Idiosyncratic view from our perspective.• Mostly about core Sakai, not much

experience with OSP.

5

The Cathedral

The Bazaar

Models of Software Production(Brad Wheeler, Indiana)

6

A hole….

Need for a Hybrid Model

…Community Source

“The Pub…the Place Between the Cathedral and the Bazaar”

Community poll

• Email request to sakai-dev and pedagogy lists on 24 June 08 (“Your views on community source”)

• Not very many responses (some soundbites, a few essays)

• Low response rate could be a good thing:community temperature is relatively low

• OTOH, perhaps Sakai is simply so opaque that no-one knows what to think!

I love Sakai and community source, especially …

“the ability to turn round fixes quickly”

“when someone else at another University has already spotted and fixed a bug that has just hit us!”

“the fun, funky and creative people that take part in sakai, and who are endlessly willing to help out on things that are really no concern of theirs at all, even at the weekend”

“when I see new practices and technologies for teaching, learning and collaboration being created quickly and disseminated broadly, with institutions that have many reasons to persist standing behind the work”

Sakai and community source drive me mad! Even the cute Sakaiger cannot

make me smile when ...

“everyone takes ages to agree on a decision and the upshot is that nothing gets done”

“the community cannot even agree on a formatting standard, let alone any coding standards”

“poor code that ‘seems to work’ is committed by inexperienced developers with little to no formal review and no warning notices” 

“we end up with 'one size fits none' software”

“I feel like local needs are being pursued exclusively even when community needs would only be a little bit more ambitious”

Imitation the sincerest form of flattery

“We are among the early movers in the space between The Cathedral and the Bazaar, and now we see much of our practice being discovered by the commercial sector.

Industry [is] innovating the classic open source model to look more like what we have done with Community Source.”

Soundbite: theory and practice

The cooperative model makes a lot of sense in theory.

In Sakai's practice, the developer voice is too loud wrt to the user experience, we are technology-led not design-led and we do not communicate / collaborate well yet - so we are not achieving the theoretical potential, but it is still a pretty good community.

Soundbite: hard to collaborate

“I love community source however it sometimes takes a bit longer to get things done. I wish we could collaborate more in the community. Due to staff shortages we are seriously neglecting the community.”

Soundbite: what works

“Community source in Sakai works when there are collaborative representatives of various universities and supporting organizations that understand how to work together for the good of all, as well as the necessity for all to contribute in order to receive the benefit of collaborative work. 

Community source does not work when institutions insist on getting things their own way regardless of the needs of others, and/or when few are able or willing to contribute the needed resources to make things happen. It is as simple and as complex as that.”

Soundbite: directing effort

“The biggest challenge is that the development community remains diffuse like open source, and is difficult to direct aggressively in the necessary directions, as in ‘closed source’ systems. 

Community members will face political opposition inside their own institutions if efforts are made to fund the principle ‘software development staff’ at the foundation level, but that is what's needed to convert Community Source from a ‘feel good’ change to a creative engine.”

Mini-essay: developer-led

“The most difficult part of Community Source for me, a part of the community over the past 4 years, has been determining what type of input I can have when we can't provide programmer support.  The Sakai development has been driven by those institutions who … provide programmers to the effort.  […] That programming often occurs in a vacuum without the input of end users, teachers, instructional designers, etc.  

Fortunately, in the past six months the inclusion of non-programmers has been changing to allow other types of community members to have input […] However, … there is still not an articulated vision or plan for how to integrate these other groups of people into the development mix.”

Mini-essay: deliberate ambiguity[Community Source] means many things to many people … folks define it however it makes them most comfortable - and the lack of a clear definition allows people with conflicting views to fight over the definition of CS - to advance one or the other perspective - as winning the definition fight then changes the "constitution of Sakai".

[…] many important aspects of Sakai governance are left very vague. My gut feel is that on the whole the Sakai community would prefer to leave these things vague for fear that if things were clarified - they would lose something. As long as things are vague, everyone can assume that things are as they hope in terms of governance. As long as things remain vague - we can all assume that the real rules (unwritten as they are) are favorable to each of us. And as a community, as long as we avoid conflict - it is never necessary to really know the rules of governance. We move through each day - building, fixing, testing, and releasing software - functioning each day much like the previous day.

  For me the thing to fear is that without clarity - it is difficult to attract new deeply involved members. Those of us who are in already - instinctively know enough to be satisfied - but new folks have no idea whether we are a cult, company, or open source project. And this makes it so they never look close enough to get past the fog to see that inside we are actually a pretty cool bunch of folks who work pretty effectively.

  It is kind of like an uncomfortable marriage - as long as things are not too bad - it is better to stay together and let folks assume whatever they like instead of digging through the issues and potentially stirring up a bunch of conflict.

Mini-essay: bazaar, commune or temple?

Sakai tends to practice the ‘commune’ model. Really, I think it might be better labeled the "temple" approach or something, because while it appears to be (and is in fact) wide open to the public, it depends for its smooth operation on the work of a few high priests.

There seems to be less risk in this approach, but the downside is that someone picking up the code is confronted with 10 different ways of doing any given thing, with no clear guidance of what the right approach may be. One can't just depend on the code, but must turn to the actual community to ask what the hell is going on.

Given the nature of Sakai I think this is exactly the right approach. It's appropriate to encourage people to join the community in a real way, rather than just download, install, and disappear. It seems like the main risk here is that big but worthwhile changes won't happen, or will be seriously delayed, because you can't get agreement from the priests on how to fix the problem. This is hypothetical, of course: as a newcomer I'm not aware of any such disputes. But they would seem to be inevitable.

Our Sakai journey

• Aimed to replace both home-grown legacy system and WebCT with Sakai.

• Signed up to Sakai consortium (SEPP) in 2004

• Started working with Sakai in June 2005• Production deployment Feb 2006• Retired legacy systems 2006 and 2007• Approx 25K students and staff, approx 10K

distinct users per day.

Some naïve assumptions

• Sakai will be successful because large well-known universities are involved.

• Large well-known universities will naturally employ many developers who will all produce high quality code.

• If larger university X is using tool Y, then of course it will work fine for us.

• When there’s a spec, the tool will surely follow within the suggested timeframes.

• JSF is the way of the future.• We have little to contribute.

Some early frustrations (2005/6)

• Having our bug reports converted to feature requests (the “kiss of death”)

• Apparent community indifference to UI inconsistencies (“it was even worse before…”)

• The amazingly long length of time it took to win even the smallest of UI victories (e.g. Edit/Revise)

• A QA process that was initially completely independent of (=ignored by) the release process (aka ‘release on time no matter what’).

• Amazingly bad and careless UI in many tools.• It wasn’t quite what we were hoping for, and it wasn’t

clear how to get there from here.

Realizations

• The more you know about why other people make the decisions they do, the less annoyed you get about it …

• Anxiety levels go down the more you know about the real community processes.

• Good collaboration is really hard. Mostly the odds are stacked against effective collaboration.

• Managing participation in an open source community is a new type of activity with its own set of risks and rewards.

Thinking about the community

• A nature/nurture debate?– Some characteristics and behaviours are

emergent properties of the community– Some things we just ought to learn how to do

better (growing up)

• Some persistent problems• Some persistent fault lines

In search of a vision

• We don’t agree on very much.• We’ve learnt how to agree on some things to

achieve the things we don’t agree on.

What shapes Community Source?

• Institutions– Much more so than Linux or Apache

• Money– Grant funding

(a short-term distortion of the open source model)– Limited resources

• Managers– Walking a tightrope between local and community

interests

Sakai as a CoP

“Communities of practice are groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly.”

http://www.ewenger.com/theory/

• Characteristics include:– Shared professional practice– Legitimate peripheral participation– Support for inbound trajectories

(from the periphery to the core)

Tool evolution• Imagine a tool with:

– Visionary design, outstanding execution, continuous attention and improvement in response to community feedback.

• Most “mature” tools in Sakai are perhaps just “feature-adequate” (Clay F)

• Why do tools get good but not great? • Are our tools wanted but unloved?• Tools drop off the priority radar

(when they’re ‘good enough’)• Tools are traded or improvements brokered through

bilateral / multi-lateral agreements rather than broader community processes

Functional boundaries

• Building tools is easy.• Adding capabilities is hard.• So most people build tools.• Sooner or later, the tools get stuck because

they need better platform capabilities.• So users get stuck in tool silos.• Projects develop inertia, insurrection is

hard.

Institutional inhibitors

• Institutions reassign developers to other priorities, usually in a somewhat opaque way.

• Local decision making and design usually trumps community design.

Process evolution

• Open, public, searchable mailing lists• More process (to fix things),

e.g. Sakai Community Practices WG• QA: continuous improvement• Contrib/provisional/core tool promotion criteria• “Branch [vote] and merge” contribution style• Performance testing• Board elections• Community branch managers• Tool scorecards• Less process (fewer DGs, WGs, charters),

more lightweight decision-making

Functional overlaps

• Parallel projects in the same functional space:– Tests and Quizzes, Mneme– Discussion, Forums, JForum, …– Assignments, Assignment 2, Mneme

• Are these good or bad for the community?• Can we afford them?

Tests & Quizzes / Mneme• T&Q: Highly complex app developed over a long period of time.

Early production problems, matured over several versions to be successfully used in multiple institutions and community contributions.– Foundational technologies (JSF/Hibernate) have become to

be regarded as problematic.• Mneme: started off as a delivery engine rewrite, morphed into a

complete rewrite.– Narrow, highly focused development effort.

• Explicit decision not to collaborate.• A common community question, “Is Mneme replacing T&Q?”, is

unanswerable.• The lack of congruent communication is a symptom of a

complex community episode (“here be dragons!”) with multiple narratives.

T&Q / Mneme dilemma

• In the context of parallel competing projects with different timelines, where should one invest effort?– Easy for Stanford: T&Q– Easy for Foothill: Mneme– For early adopters of T&Q, total cost actually increases.

Worst-case scenario is an investment in both tools.

• Strategies that may be rational and in the best interests of the primary roleplayers may in fact increase total cost for the community.

Why do we care about community process?

• Our primary and founding (aspirational?) beliefs:– we have (sufficient) shared purpose and values– we can harness the collective energies of the community

such that 2 +2 = 5 rather than 2 + 2 = 3

• Long-term success is not about the software we build today, but about how well our community works.

• Our work is building and engineering community (an argument about ecology and ecosystems)

• Can we find some predictors for long-term success?• If we don’t always know how to improve things, can

we get good feedback about when they’ve improved?

A community scorecard• Community of Practice index

(lurkability, easy journeys from the periphery to the core)

• ‘New capability’ index (adding a tool, service or capability): barriers to entry, speed of iteration, time to acceptance

• Efficiency and effectiveness of collaboration (favourable cost/benefit ratios) in the areas of:

– application design (pedagogy, teaching and learning, research)

– UX design and development

– code design and development

• Morphology index: size and shape of projects, diversity and balance

• ‘Adoptability’ index: effort required to understand and evaluate Sakai

• Product TCO: cost to adopt, customize, deploy and support

• ‘Shareability’ index: cost to share and adopt innovations

• Local:Community resource allocation ratio

• Short:Long term planning ratio