The permissionless Rating System has problems around incentive design. The purpose of the proposal is to re-vamp the system to fix issues in order to include high-quality reports with no false statements. The proposal suggests employing a partial tokenengineering methodology and iterating on the system as we go along in prod.
Problems with the current permissionless Rating System:
The incentive design is not balanced in the permissionless rating system
The community wants to engage more in the voting
Governors are not sufficiently incentivised
Reviewer incentive structure is skewed towards one-time review
Rewards may not be balanced
The process is not scalable and needs to be designed with scalability in mind
Implementation is subject to practical limitations. Many processes are still limited in operational efficiency and scalability (this means simplicity and prioritisation is key)
Decision-making on outcomes of snapshot votes is opaque (i.e. it is unclear to the rater why the snapshot failed)
Discrepancy between reviewer quality and statements
RXP issuance is not balanced
Too what extent should reviewers act as educators? When is a report ready for governance voting?
How do we create benchmarking standards for governors?
How do we encourage TIPs?
When can a project be re-submitted after failed attempts?
Malicious attempts through double identities (i.e. Governor starts new rating identity, reviews and votes for it) (or #sybil attacks more broadly)
How to not discourage new raters that fail re-peated governor vote?
Slashing of Governor RDR
More?
Proposed Solution:
Employ a Tokenengineering inspired methodology (see and see) to build robust incentive design around the permissionless rating with the community by going through the tokenengineering system requirements & (partial) design phase together.
The ultimate goal is to ensure that only high-quality reports are included in the final score and that no false statement
While I don’t want to encourage a waterfall approach I want to encourage a systems view and thus would like the community to keep this perspective considering all stakeholders and moving through different scenarios that happen through an interaction of stakeholders or across scales
Workplan
Facilitator: xm3van* Duration: 5 Weeks - 8 Weeks max accounting for the planning fallacy Start: 28.06.2022 Meeting Frequency: Tuesday and Thursday from 17.00-18.00 BST Restrictions: Gated to holders of 30RXP (exception apply pls contact xm3van) Documentation & Tools:
Mapping of entity relationships through a casual-loop diagram
Homework: User research if Q’s are open
==> Goal: Understand the Stakeholders in our ecosystem
Session 3: Causal Loop diagram to map the relationship between entities & mechanism design
Date: 12.07.2022 & 14.07.2022 - 17:00 - 18:00
Map the system in a less formal easily accessible way (based on this we can formalise if we want to)
Refinement of stakeholder and their interactions
==> Goal: Understand the Stakeholders in our ecosystem & their interactions
Session 4: Mechanism design
Date: 19.07.2022 & 21.07.2022 - 17:00 - 18:00
Ideation of mechanism (may require more research to speculate on behaviour)
(Imaginary) Scenario analysis (i.e. what if the system has 1000 rater, what if the system has 1 rater?)**
==> Goal: Understand the Stakeholders in our ecosystem & their interactions & solution of how they ought to behave
Session 5: Evaluation of ideated mechanism based on cost, benefit, realism under resource constraint & action plan
Date: 26.07.2022 & 28.07.2022 - 17:00 - 18:00
Evaluation of the system under resource constraints for a fast (most likely not perfect) implementation
Define Roadmap:
Fast implementation
Ideal implementation
Reporting structure to track and communicate transparently the implementation of the system
Implementation Taskforce
==> Goal: Make the workshop actionable
Outputs of this Proposal
Improved system around permissionless rating that better produces reliable, trustworthy, rating reports
An action plan for a short-term solution to immediately address the problems listed above
Roadmap “ideal” system (prioritised by the ease of implementation/ maintenance and importance)
Community alignment with the developed system
=============================
*xm3van: I am not an expert but familiar with the tools and in private working on becoming a tokenengineer.
** Ideally we would develop models for this to validate it - yet maybe this can be done in the second iteration after a fast solution for an ideal solution
Much needed discussion, thanks xm3van. Another incentive design “problem” I wanted to throw in there is that there is no incentive at the moment for the community to post such forum posts, comment on them or participate in governance of any kind.
How can we incentivize community members to show up for the meetings you are proposing here?
One more comment as an after-thought. Putting token engineering forward as the primary solution here may be too early? Would be good to understand what are the problems we want to solve, categorize them and determine which ones we can realistically solve with token eng. vs. other methods?
p.s: in case it was not clear, I would like to be part of the workshop tackling this.
Governance participation is an issue in all communities (not just crypto)… And no good solutions have been found (yet).
If a community member holds D2D, the reasoning would go that this person would have incentives to act in the best intentions of PrimeDAO, therefore, speak up in governance. This is sadly not reality.
What can be done is give incentives for governance interactions, also called Governance Mining, in some other communities. Giving rewards (D2D orRXP) for voting gives the risk of members voting without reading what the vote is about.
Giving members rewards for posting on the forum gives the risk of spam. And if you want to start moderating that, then comes the issue of who decides what is a good post and what isn’t?
Could give tiny amounts of RXP, too little to bother spamming, but enough for members to feel they get rewarded? 1 RXP for showing up in a call?
In the end I think community engagement will always bring more people to start truthfully interacting with governance than these incentive structures will. The above touches upon a lot of raters and reviewers feedback, so it should be shared a lot and I’m sure the right people will then come and join the discussions.
I personally would have anticipated given the sentiment over the last couple of months that there would be some people interested in changing the situation and that this would act as incentive to join. But I am open to discuss other methods (i.e. paying people or handing out poaps etc.)
Tokenengineering is more a methodology for me consisting out of a set of tools from traditional business and consulting. We can take other ways too - for me the goal of the series it is more to think in structured way through the incentive design form a systems perspective. I was hoping the structure of the session (which builds on tooling outside of “tokenengineering”) would illustrate it - but the methodology uses essentially quite conventional consulting tools especially in the system requirements phase (at least from my understanding thus far). Maybe some samples of what kind of templates that are used can be seen in my personal notes here and here.
That was very detailed @xm3van and I think you bring real problems that need to be solved.
I will add one more problem which I think is the trigger and affect almost all problems that you mentioned. It all starts with a submission report on feedback, and I will point out that I personally made those mistakes in ~3 months rating period, which I see are becoming more common:
It’s impossible to perform fundamental analysis and write a decent report in less than a week. If somebody writes a report in 2 days (my last report on metaverse rate-athon) that brings more work to the reviewer.
The reviewer needs to spend more time reviewing that report (in 1 day ) and after the rater implements suggestions - what is a product ready for governance voting? The report that is made up of superficial information that can be found by literally anyone on Google in 1 hour, and if we talk about a “product” or “service” that someone needs to use we have a problem (even worst - paying)
That problem comes to governors who have to reduce MVP criteria to a very low level, which bring them into a trade-off situation and spending more time - which they should not.
The fundamental template for Defi needs a few corrections, but the Metaverse template needs a complete redesign. Raters need meaningful questions in order to give meaningful answers.
If the rater needs to score most or all questions in a section with 0, maybe that protocol shouldn’t be on the rating list. Last time a lot of raters support the “0 option” but from a logical standpoint, 0 isn’t a score. Protocols without a governance system deserve a minimal score - 1. If we are DAO members that should be decided by group decision, not by individual.
I just point out this problem from rater perspective.
agree with work plan and incentive mechanism redesign is a great idea
Agree with you @RatingPepe. I don’t have a solution in mind, my point was that we need to include this governance participation problem into @xm3van’s workshop scope. I’ve learned from experience that incentives & system design wins out over even the best intentions. So, want to encourage us to think in that direction for Prime Rating governance too.
Fair point! I am wondering whether this is however work for the general Governance of Prime Rating beyond the permissionless rating product.
For me it feel part of the goverance of changing processes (i.e. for me Organisational Governance/ DAO Goverance), whereas I see the scope of this workshop as governance of prime rating product (i.e. Product Governance).
And I think that needs open discord channel, where Governors after Snapshot voting explain their decisions and point out where the rater made mistake and where reviewer provided a false review, because without that practice all participants in the rating process will continue to make the same mistake - raters, reviewers, and governors. With better communication and coordination this process can be significantly improved and can save our time, at least.
Agree with you, that is problem of all communities, but again I think governors also need to be incentivized. Although they spend the least time in the process (maybe), they have more responsibility than raters and reviewers. COW protocol has an interesting solution and high voting turnover for now - " There are 10,614 total vCOW holders on both chains (Ethereum - 752, Gnosis - 9,862) which means that, on average, ~ 59% of governors are active in governance voting."
In order to advance this proposal and based on the feedback received I will make concrete suggestions to each of the questions put forwards:
Suggestion: Based on the feedback received - I’d opt for 10 RXP this means that a given rater has completed at least 1 successful submissions. I think this encourages most voices to be heard while avoiding uninformed opinions who have not a complete understanding of the whole process. Exception can be made please reach out to me (Suggesting to leave this to me or Governor for efficiency sake - happy to outsource or backdown form this if community comments in this thread)!
Suggestion: I suggest starting after the completion of the season this means 27.06.2022. Starting with bi-weekly appointments every Tuesday and Thursday from 17.00-18.00 BST going forward from that date (i.e. first appointment 28.06.2022).
In terms of duration I think it depends what we are aiming for and how we want to iterate on version one. Maybe something like this should be periodically repeated in quarter-year cycles. If this is a one off I think a longer period is required, if this should become an organisational practice we can move a bit faster knowing it is not final.
I will amend the the post with any more issues that come in. Pls let the feedback come in.
Wholly in favor of a workshop like this. Thanks for pushing this forward @xm3van! I would like to participate as well.
On the subject of RXP limits for participation, I think the threshold should be higher than 10RXP honestly. Acknowledging that governance apathy is real, this would help make sure participants in the workshop have had the interest and commitment to complete more than just 1 report. I would advocate for a limit that also emphasizes quality if possible. A 30RXP limit could be a good middle ground (3 reports or a prize winner). Definitely open to a lower threshold though if we want to have more of the community involved.
Thank you very much for getting this started @xm3van ! It’s a great idea to involve all stakeholders in the workshop as especially the raters are impacted by these important decisions on how we structure Prime Ratings processes. Would love to be part of it.
We will take a look at the classification of issues and any additional issues added between Tuesday and today
Note: I tried to break the issues down into an issue tree - from this we can decided if we want to further validate whether the issues are true and move after this to a hypothesis tree to validate potential solution. It definitely can do with refinement (ME CE).
2/3) Workshop participants check how to proceed:
Option 1: Plan to validate identified issues
Option 2: Accept issues as true and proceed to the hypothesis tree
Move on to Stakeholder mapping & Token utility canvas
Everyone: pls add additional possible solutions to the miro board until Wednesday 06/07/2022 - noon BST
@xm3van will add create a spreadsheet listing all possible solutions and add evaluation criteria with an initial judgement in order to provide pirotisiation. xm3van will share this in advance in discord chat.
