The initial discussion on abandoning SourceCred was met with resistance for a bunch of reasons.
In this actual proposal, I will attempt to synthesize that discussion into a punchier post against using SourceCred and a toned down proposal to pause it & reassess rather than permanently shut it down.
Why stop using SourceCred
A bunch of reasons, actually:
- Firstly, I think rewarding every PR the same regardless of whether its a 1 line of code or a whole repo refactoring is an absolute nonsense
- You can get more XP on your PR if there’s a lot of discussion in it, which incentivizes discussion.
- It took us almost 3 years to realize this. We’ve also tried finding people to optimize the weights configuration but never succeeded - which speaks to a lack of transparency & clunkyness of it.
- No ability for budgeting between different categories of contributions.
- Right now we require people to post in both Coordinape as well as didathing - completely unnecessary.
- It’s been abandoned & no longer being developed. First, it was abandoned by the founder for a loss of faith in the technology & a year later the organization itself fell apart.
- Given there’s a limited number of participants to emoji things + the fact people almost never click “1” or “2” even if the contribution is very small results in SourceCred favoring small contributions over big ones - quantity of didathings beats quality.
- In SourceCred, allocations are made community-wide which makes it much harder because people working across different functions don’t necessarily understand each other’s work & efforts needed. Even if everyone has full context, its a chore. If its already not working well with 20 people, imagine if we had a community of 200 or 2000 active participants. It doesn’t work & it’s just not scalable.
- Finally, it may be personal but I find writing in Coordinape less anxiety-inducing than writing in #didathing & thus easier to do + don’t have to spam & bury others’ work with my report-unloading (which makes it so buried reports are rewarded less - that doesn’t happen in Coordinape)
Overall, the incentive structure in SourceCred is biased towards small contributions vs big strategic ones - it rewards quantity over quality (both in didathing & github).
Over the course of 3 years, this has resulted in a lot of weird distributions.
And this was without MetaGame ever exploding in popularity - in 1Hive, they literally had to stop it because too many people were trying to game the system.
Why Coordinape is superior
Here’s 8 reasons for that too:
- Grouping contributions by epochs gives a much better overview of a player’s total contribution
- Grouping contributions by epochs + allowing people to write epoch recaps makes it much easier to put things into perspective & properly reward people.
- In Coordinape, allocations are limited to teams which means the people who are allocating have much more context into value of contributions. It’s much easier for writers to reward writers than devs.
- With more context & allocation power, it’s easier to notice & fix if somebody inside the team is writing a lot but not actually doing a lot. You can just allocate them less (instead of proposing distribution results change to the whole community, which never happened).
- Having the amount of GIVEs limited & all contributions visible at the same time forces people to properly evaluate contributions with each other (vs. being able to click 3 or 5 on everything & being done with it)
- In Coordinape, you can also leave a little private note explaining your allocation to the person if you’re worried they might take it the wrong way or wonder why you allocated them too little or too much.
- We can batch-report all contributions without feeling bad about spamming or burying other people’s contributions.
- Given Coordinape has an API with all the needed functionality, it would also be much easier to build our own reporting system or analytics inside MetaGame’s dashboard
Recap
A few points to recap & add what was missing…
There are essentially 2 ways things are valued through SourceCred:
- Algorithmic - automatically tracking github PRs, comments & talking. This sucks because it rewards every PR the same regardless of it’s actual size & value. It also sucks because it incentivizes commenting & writing a lot on github, discord & forums which is not necessarily a proxy of value creation.
- Manual - posting things in didathing & props. This sucks because it rewards quantity over quality for a variety of reason (limited no. of emojiers, unlimited emojis, lack of context, unwillingness to click low value emojis even for low value contributions etc.)
- Overall, SourceCred rewards frequency of posting/quantity over quality & it always has been. It has been skewed towards small contributions from the beginning & we have been aware of it the whole time, never able to fix it but having nothing better to replace it with. Now there is - and Coordinape is far from being the only one (eg. Praise, Govrn & Karma are some of the others)
- In SourceCred, people being less active incentivizes more people to be less active because others being less active means they will get paid less (less people reacting), whereas in Coordinape its the opposite.
- I’ve been told the only really significant project using it has stopped (Maker) & now the only significant project using it is Smart Contract Research Forum - for which it makes complete sense to reward writing as its a research forum
- Some of its problems could be solved by building SourceCred UI but it has no team & MetaGame has 4 somewhat active developers that aren’t a single person working full time even when all 4’s time is added up (afaik? am I wrong?).
- I see absolutely no reason to cling so tightly to an abandoned piece of software that almost nobody is using & waste the little resources that we have on trying to improve it instead of adopting better solutions that have teams building them & moving on with building MetaGame into a massive online coordination game that it was meant to be (by improving the platform itself rather than focusing on maintaining a dead rewards system)
What about XP
Firstly, the XP system is already broken because only about half of the people are actually posting in #didathing vs reporting only in Coordinape.
- The simplest thing would be to base it off Seeds earned.
XP = Seeds earned x 135 (or whatever) - We could also freeze the XP count as it is currently, come up with an GIVE to XP ratio & use that going forward (making it based on GIVEs)
Both of these would also make it more stable instead of going up & down.
Proposal
Pause SourceCred for X months, then re-evaluate & propose to continue running it if we see things were better with SourceCred running.
Meanwhile, we could test something like Praise (which could be used for new contributors while they prove themselves, before they are invited to a Coordinape circle & for eg. player of the month or random props), or we could just go with Coordinape only for a month or two & see how that goes without adding more overhead in case its unnecessary.