First Quest Data & KPIs [DAO Dash] [Nov 30]

[paulapivat]

I want to share some data we have around First Quest role tag(s) that could shed light on our onboarding efforts at Bankless DAO.

The primary audience for this post are the fine folks working on First Quest, @Behold , @angyts and team (pls tag ), but I’m hoping to share this process with the community to spark ideas and conversations in other projects, guilds or domain areas.

Disclaimers: This is a work-in-progress, some metrics/chart are more readily available, while others require more time to extract out from various sources. Things are constantly in flux, in DAOs, Bankless DAO and I’d imagine the First Quest team, so I wanted to get some info out sooner than later. This way if new metrics or KPIs need to be added, we can make those adjustments.

Around early-to-mid October, I had reached out to members of First Quest to inquire about metrics and data that would be of use for their work of building the best onboarding experience in Web3.

Here are a couple metrics the team wanted:

• new member growth
• % new members who obtain ‘First Quest Complete’ role
• % new members with ‘First Quest Complete’ role that are active after 1 day, 1 week, and 1 month
• Time to first human contact ?

New Member Growth

note: x-axis: 4 = April, 5 = May and so on.

This chart shows the number of new members joining Bankless DAO server by month from April - November. We can see a large spike in May (genesis) at ~3.5K and a huge uptick in September (3.42K) & October (3.33K).

With these numbers, we can look at:

% new members who obtain ‘First Quest Complete’ role

This metric tells us how quickly it takes Discord newcomers to complete the onboarding process. The faster the better.

We had to pull data for when roles held by DAO members was activated (thanks to @McKethanor!). The present (n = 167) data shows “First Quest Complete” tags that were activated were activated in November.

To keep the chart readable, I split the join month of Aug, Sept, Oct for another chart:

A couple caveats for interpretation. First, there are people who obtained their First Quest Complete tags before November, but data is (presently) only available for November. Moving forward we’ll be able to track the gap between joining Discord and obtaining First Quest Complete in real time.

For now, we can look backwards to see that we had several members who joined in May, June, July and only got their First Quest Complete tags in November. This would make sense as we were forming as a DAO and coordinating to develop First Quest as an onboarding mechanism in the first place.

It is encouraging, in my view, to see that many more members in who joined in August, September and especially October, obtain their First Quest Complete tag in November, showing that later joiners were made aware of First Quest and went through the onboarding process.

Next we want to look at:

% new members with ‘First Quest Complete’ role that are active after 1 day, 1 week, and 1 month

Since data for an activated “First Quest Complete” tag is only available for November (and onwards), we’ll have to get to the end of December or January to truly see how active people who both joined the Bankless DAO Discord server and activated their First Quest Complete tag in November are in the following weeks and months.

First Quest Funnel

Perhaps the real crypto rabbit hole is First Quest because there are multiple levels beyond “First Quest Complete”. As of this writing, here are the “levels” in First Quest, starting with “first-quests-welcome”:

We can go beyond the attainment of a single tag (“First Quest Complete”) to see the patterns of movement as new discord joiners are encouraged to obtain multiple tags.

Ideally, we want the same number of “first-quests-welcome” as “first-quests-guest-pass” or simply “first-quest” where all new members go through the “First Quest Funnel” with no drop off, but our data suggests otherwise.

Here’s the pattern of First Quest tag(s) attainment:

We see the most engagement with First Quest Welcome and then a drop off to First Quests Membership, when the latter is just a channel below. The drop-off continues as we go down to First Quest Complete.

In fact, we see only 895 members get their Guest Pass or 20% of people who start at First Quest Welcome stick around to getting their Guest Pass. In other words, there was an 80% drop-off from the top of the funnel.

A challenge for the team is preventing drop-off as people make their way down the funnel.

However, we see more tags from the raw data (in DAO Dash) than what’s visible in the discord channel, so some re-organization could be in the works. One member had mentioned the phasing out of certain tags, so we may have to re-visit the data once new tags are in place.

Does the Number of First Quest Tag(s) correlate with engagement?

By examining % new members with ‘First Quest Complete’ role that are active after 1 day, 1 week, and 1 month, we may be overly focused on a single tag, but what if it’s the number of tags that influence activity?

Perhaps new members who put in the effort to obtain multiple tags are more likely to be active members?

To examine this, we can look at both number of messages and number of first quest tag(s):

This view filters for total number of messages (100 - 2.25K) and number of “First Quest” related tag(s) obtained (1-8) and places them side by side. However, the difference in scale makes it hard to see if a relationship is there.

What we might do is to visualize number First Quest tags as 8 distinct colors, we can do this be exporting the data from DAO Dash:

This doesn’t quite help us discern a relationship, so we’ll try another view splitting the number of messages by number of first quest role tags as facets:

We can clearly see that the individuals who sent the most messages (high activity level) only had 1-2 of the 8 First Quest-related tags.

Just to be sure, I ran a correlation between Number of Messages and Number of First Quest Role Tag(s) obtain. (r = 0.07, 95-Confidence Interval: 0.02 - 0.12).

So the relationship is rather weak. We cannot presume that First Quest attainments translates to higher engagement/activity levels.

Nevertheless, I’d say the jury is still out.

The presently available data highlights members who may have started in Bankless DAO in May-June and only obtains 1-2 First Quest tags, but are very active nonetheless.

To get a fair reading of the data, we’ll want to track these metrics moving forward.

Take-Aways

The TLDR:

1. New Discord Joinees are finding their way to the First Quests channels faster now than they were over the summer.

2. First Quest can be visualized as a Funnel and the challenge is engineering ways to prevent drop off as people make their way through the funnel.

3. There is presently a weak relationship between Engagement and Number of First Quest tags attained. However, we should get an even clearer picture with more data.

We are continuing to both build the pipelines for getting more data and making it more easily accessible. If you made it this far, thank you for your attention. Hopefully it sparks some ideas and conversation. We appreciate feedback on what additional metrics, charts and analysis you’d like to see, .

[ThePsychGuy]

This is great work! Thanks @paulapivat. Happy to see first-quest work paying off. We will try to reduce the drop-off rate at membership and curate better content.

PS: Very interested to see a breakdown of people with most messages sent in the DAO. I have a feeling that @ernest_of_gaia will top that one as well lol.

[paulapivat]

Most messages sent goes to @ernest_of_gaia by a healthy margin

[Jengajojo]

Purely speculating here: Perhaps adding some audio/video content to the first quest can improve the conversion rate, especially for the audio-visual learners

[Behold]

Incredible work, as always @paulapivat

[angyts]

OMG, ernest are you a bot? Haha!

Grateful for you hanging out so much in the BanklessDAO, really appreciate ernest’s sharings and teachings!

[angyts]

Thanks paul for these metrics. Really important to us. We will try to take all these metrics into consideration in our upcoming season planning.