Re-vamping Incentivise Design around the permissionless Rating Process

tl;dr

The permissionless Rating System has problems around incentive design. The purpose of the proposal is to re-vamp the system to fix issues in order to include high-quality reports with no false statements. The proposal suggests employing a partial tokenengineering methodology and iterating on the system as we go along in prod.

Problems with the current permissionless Rating System:

The incentive design is not balanced in the permissionless rating system

  • The community wants to engage more in the voting
  • Governors are not sufficiently incentivised
  • Reviewer incentive structure is skewed towards one-time review
  • Rewards may not be balanced
  • The process is not scalable and needs to be designed with scalability in mind
  • Implementation is subject to practical limitations. Many processes are still limited in operational efficiency and scalability (this means simplicity and prioritisation is key)
  • Decision-making on outcomes of snapshot votes is opaque (i.e. it is unclear to the rater why the snapshot failed)
  • Discrepancy between reviewer quality and statements
  • RXP issuance is not balanced
  • Too what extent should reviewers act as educators? When is a report ready for governance voting?
  • How do we create benchmarking standards for governors?
  • How do we encourage TIPs?
  • When can a project be re-submitted after failed attempts?
  • Malicious attempts through double identities (i.e. Governor starts new rating identity, reviews and votes for it) (or #sybil attacks more broadly)
  • How to not discourage new raters that fail re-peated governor vote?
  • Slashing of Governor RDR
  • More?

Proposed Solution:

  • Employ a Tokenengineering inspired methodology (see and see) to build robust incentive design around the permissionless rating with the community by going through the tokenengineering system requirements & (partial) design phase together.

  • The ultimate goal is to ensure that only high-quality reports are included in the final score and that no false statement

  • There are multiple ideas also from the core team (#Lavi) suggesting mechanisms to alter one or more functions. (Discussion: Adjusting RAP - Google Docs)

  • While I don’t want to encourage a waterfall approach I want to encourage a systems view and thus would like the community to keep this perspective considering all stakeholders and moving through different scenarios that happen through an interaction of stakeholders or across scales

Workplan

Facilitator: xm3van*
Duration: 5 Weeks - 8 Weeks max accounting for the planning fallacy
Start: 28.06.2022
Meeting Frequency: Tuesday and Thursday from 17.00-18.00 BST
Restrictions: Gated to holders of 30RXP (exception apply pls contact xm3van)
Documentation & Tools:

  • Recorded community working group calls
  • Collaboration in Miroboard (https://miro.com/)
  • Meeting notes
  • Summary after each session in forum thread

Note: You can access the all relevant link through this token-gated link if you have 30 RXP or more: MintGate: NFT Token Gate the Metaverse

Sessions

Session 1: Collection & Structuring of problems & definition of properties “objective function” of permissionless rating

  • Date: 28.06.2022 & 30.06.2022 - 17:00 - 18:00
  • Potential tools:
    • Problem-definition sheet
    • Issue tree
    • Hypothesis tree
  • Else structured discussion to identify issues
    ==> Goal: Have a clear idea of what problems we are trying to fix!

Session 2: Stakeholder mapping & Token utility canvas

  • Date: 05.07.2022 & 07.07.2022 - 17:00 - 18:00
  • Identification & Classification of Stakeholders
  • Mapping of entity relationships through a casual-loop diagram
  • Homework: User research if Q’s are open
    ==> Goal: Understand the Stakeholders in our ecosystem

Session 3: Causal Loop diagram to map the relationship between entities & mechanism design

  • Date: 12.07.2022 & 14.07.2022 - 17:00 - 18:00
  • Map the system in a less formal easily accessible way (based on this we can formalise if we want to)
  • Refinement of stakeholder and their interactions
    ==> Goal: Understand the Stakeholders in our ecosystem & their interactions

Session 4: Mechanism design

  • Date: 19.07.2022 & 21.07.2022 - 17:00 - 18:00
  • Ideation of mechanism (may require more research to speculate on behaviour)
  • (Imaginary) Scenario analysis (i.e. what if the system has 1000 rater, what if the system has 1 rater?)**
    ==> Goal: Understand the Stakeholders in our ecosystem & their interactions & solution of how they ought to behave

Session 5: Evaluation of ideated mechanism based on cost, benefit, realism under resource constraint & action plan

  • Date: 26.07.2022 & 28.07.2022 - 17:00 - 18:00
  • Evaluation of the system under resource constraints for a fast (most likely not perfect) implementation
  • Define Roadmap:
    • Fast implementation
    • Ideal implementation
  • Reporting structure to track and communicate transparently the implementation of the system
  • Implementation Taskforce
    ==> Goal: Make the workshop actionable

Outputs of this Proposal

  • Improved system around permissionless rating that better produces reliable, trustworthy, rating reports
  • An action plan for a short-term solution to immediately address the problems listed above
  • Roadmap “ideal” system (prioritised by the ease of implementation/ maintenance and importance)
  • Community alignment with the developed system

=============================
*xm3van: I am not an expert but familiar with the tools and in private working on becoming a tokenengineer.
** Ideally we would develop models for this to validate it - yet maybe this can be done in the second iteration after a fast solution for an ideal solution

7 Likes

Much needed discussion, thanks xm3van. Another incentive design “problem” I wanted to throw in there is that there is no incentive at the moment for the community to post such forum posts, comment on them or participate in governance of any kind.

How can we incentivize community members to show up for the meetings you are proposing here?

1 Like

One more comment as an after-thought. Putting token engineering forward as the primary solution here may be too early? Would be good to understand what are the problems we want to solve, categorize them and determine which ones we can realistically solve with token eng. vs. other methods?

p.s: in case it was not clear, I would like to be part of the workshop tackling this.

1 Like

Awesome to see this is being kickstarted!

1 Like

Governance participation is an issue in all communities (not just crypto)… And no good solutions have been found (yet).
If a community member holds D2D, the reasoning would go that this person would have incentives to act in the best intentions of PrimeDAO, therefore, speak up in governance. This is sadly not reality.

What can be done is give incentives for governance interactions, also called Governance Mining, in some other communities. Giving rewards (D2D orRXP) for voting gives the risk of members voting without reading what the vote is about.
Giving members rewards for posting on the forum gives the risk of spam. And if you want to start moderating that, then comes the issue of who decides what is a good post and what isn’t?
Could give tiny amounts of RXP, too little to bother spamming, but enough for members to feel they get rewarded? 1 RXP for showing up in a call?

In the end I think community engagement will always bring more people to start truthfully interacting with governance than these incentive structures will. The above touches upon a lot of raters and reviewers feedback, so it should be shared a lot and I’m sure the right people will then come and join the discussions.

4 Likes

I personally would have anticipated given the sentiment over the last couple of months that there would be some people interested in changing the situation and that this would act as incentive to join. But I am open to discuss other methods (i.e. paying people or handing out poaps etc.)

1 Like

Tokenengineering is more a methodology for me consisting out of a set of tools from traditional business and consulting. We can take other ways too - for me the goal of the series it is more to think in structured way through the incentive design form a systems perspective. I was hoping the structure of the session (which builds on tooling outside of “tokenengineering”) would illustrate it - but the methodology uses essentially quite conventional consulting tools especially in the system requirements phase (at least from my understanding thus far). Maybe some samples of what kind of templates that are used can be seen in my personal notes here and here.

1 Like

That was very detailed @xm3van and I think you bring real problems that need to be solved.

I will add one more problem which I think is the trigger and affect almost all problems that you mentioned. It all starts with a submission report on feedback, and I will point out that I personally made those mistakes in ~3 months rating period, which I see are becoming more common:

  1. It’s impossible to perform fundamental analysis and write a decent report in less than a week. If somebody writes a report in 2 days (my last report on metaverse rate-athon) that brings more work to the reviewer.
  2. The reviewer needs to spend more time reviewing that report (in 1 day ) and after the rater implements suggestions - what is a product ready for governance voting? The report that is made up of superficial information that can be found by literally anyone on Google in 1 hour, and if we talk about a “product” or “service” that someone needs to use we have a problem (even worst - paying)
  3. That problem comes to governors who have to reduce MVP criteria to a very low level, which bring them into a trade-off situation and spending more time - which they should not.
  4. The fundamental template for Defi needs a few corrections, but the Metaverse template needs a complete redesign. Raters need meaningful questions in order to give meaningful answers.
    If the rater needs to score most or all questions in a section with 0, maybe that protocol shouldn’t be on the rating list. Last time a lot of raters support the “0 option” but from a logical standpoint, 0 isn’t a score. Protocols without a governance system deserve a minimal score - 1. If we are DAO members that should be decided by group decision, not by individual.
    I just point out this problem from rater perspective.

agree with work plan and incentive mechanism redesign is a great idea

4 Likes

Great point! Thanks for outlining it so detailed!

1 Like

Agree with you @RatingPepe. I don’t have a solution in mind, my point was that we need to include this governance participation problem into @xm3van’s workshop scope. I’ve learned from experience that incentives & system design wins out over even the best intentions. So, want to encourage us to think in that direction for Prime Rating governance too.

3 Likes

Fair point! I am wondering whether this is however work for the general Governance of Prime Rating beyond the permissionless rating product.

For me it feel part of the goverance of changing processes (i.e. for me Organisational Governance/ DAO Goverance), whereas I see the scope of this workshop as governance of prime rating product (i.e. Product Governance).

1 Like

And I think that needs open discord channel, where Governors after Snapshot voting explain their decisions and point out where the rater made mistake and where reviewer provided a false review, because without that practice all participants in the rating process will continue to make the same mistake - raters, reviewers, and governors. With better communication and coordination this process can be significantly improved and can save our time, at least.

Agree with you, that is problem of all communities, but again I think governors also need to be incentivized. Although they spend the least time in the process (maybe), they have more responsibility than raters and reviewers. COW protocol has an interesting solution and high voting turnover for now - " There are 10,614 total vCOW holders on both chains (Ethereum - 752, Gnosis - 9,862) which means that, on average, ~ 59% of governors are active in governance voting."

1 Like

I received feedback in my DMs on the proposal which I perceived as valuable and thus wanted to share it here too:

1 Like

In order to advance this proposal and based on the feedback received I will make concrete suggestions to each of the questions put forwards:

Suggestion: Based on the feedback received - I’d opt for 10 RXP this means that a given rater has completed at least 1 successful submissions. I think this encourages most voices to be heard while avoiding uninformed opinions who have not a complete understanding of the whole process. Exception can be made please reach out to me (Suggesting to leave this to me or Governor for efficiency sake - happy to outsource or backdown form this if community comments in this thread)!

Suggestion: I suggest starting after the completion of the season this means 27.06.2022. Starting with bi-weekly appointments every Tuesday and Thursday from 17.00-18.00 BST going forward from that date (i.e. first appointment 28.06.2022).

In terms of duration I think it depends what we are aiming for and how we want to iterate on version one. Maybe something like this should be periodically repeated in quarter-year cycles. If this is a one off I think a longer period is required, if this should become an organisational practice we can move a bit faster knowing it is not final.

I will amend the the post with any more issues that come in. Pls let the feedback come in.

2 Likes

Wholly in favor of a workshop like this. Thanks for pushing this forward @xm3van! I would like to participate as well.

On the subject of RXP limits for participation, I think the threshold should be higher than 10RXP honestly. Acknowledging that governance apathy is real, this would help make sure participants in the workshop have had the interest and commitment to complete more than just 1 report. I would advocate for a limit that also emphasizes quality if possible. A 30RXP limit could be a good middle ground (3 reports or a prize winner). Definitely open to a lower threshold though if we want to have more of the community involved.

3 Likes

Thank you very much for getting this started @xm3van ! It’s a great idea to involve all stakeholders in the workshop as especially the raters are impacted by these important decisions on how we structure Prime Ratings processes. Would love to be part of it.

1 Like

Updated the initial post!

2 Likes

Summary Session 1a

Attendee’s: OriginalSK, Dabar90, Salome, Xindaneran, xm3van

Planned Agenda:

  • Problem-definition sheet to align on project (Living documents)
  • Identify Problems
  • Categorise Problems

Summary of session:

  • We aligned and talked about PDS.
  • We identified some problems within the incentive design and beyond
  • We went over the current process of permissionless rating
  • We agreed on the next steps for the coming session

Recording:

Task for next Session 1b):

  • xm3van: Provide number for active raters, reviewers and governors
  • xm3van: Refined copy of BPMN model copied as image into Miro board so that we can lay value flows over it
  • all: Identify more problems async
  • all: Categorise Problems into freely and individually invented themes
1 Like

Summary Session 1b

Attendee’s: Xindaneran, Dabar90, Lavi, xm3van

Planned Agenda:

    1. We will take a look at the classification of issues and any additional issues added between Tuesday and today
    • Note: I tried to break the issues down into an issue tree - from this we can decided if we want to further validate whether the issues are true and move after this to a hypothesis tree to validate potential solution. It definitely can do with refinement (ME CE).

  • 2/3) Workshop participants check how to proceed:

    • Option 1: Plan to validate identified issues
    • Option 2: Accept issues as true and proceed to the hypothesis tree
    1. Move on to Stakeholder mapping & Token utility canvas

Summary of the session:

  • Tough session yet productive
  • Clarification of Scope of the Workshop

Recording of the session:

The task for next session 2a) :

  • Map issues to permissionless rating workflow
    • Note: Ask 6x why in order to identify the root cause of the issues.

  • Feel free to drop you design idea’s

**Summary Session 2a) :

Attendee’s:

Planned Agenda:

  1. Shared Review Miro board - finalise Problem Identification [Partially Achieved]

  2. Decide the following for “Root Cause” : [Not Achieved]

  • if we want empirical validation
    • if yes, a concrete action plan (when, where, how, who, why) to provide empirical validation
  • which should be addressed with the re-design of the incentive structure
  1. Optional: Breakdown of potential solution into a Hypothesis tree [Not Achieved]

3/4) Stakeholder mapping and their incentives in relation to selected root causes. Tools that may prove useful: [Not Achieved]

  • Entity Relationship diagram
  • Causal loop
  • Empathy map
  • an adaption of token utility canvas

Summary of the session:

  • Review of miro board
  • tough session - further ideation on solution for possible solution to identified problems

Recording of the session:

Task for session 2b):

  • Everyone: pls add additional possible solutions to the miro board until Wednesday 06/07/2022 - noon BST
  • @xm3van will add create a spreadsheet listing all possible solutions and add evaluation criteria with an initial judgement in order to provide pirotisiation. xm3van will share this in advance in discord chat.
    • Ease of implementation
    • Relevance for incentive restructuring
    • Resources cost
1 Like