On organizing relevant data [Analytics Guild]

Archive
1

First of all, I’m very thrilled with the direction the DAO is heading and it’s rapid development so far.
I’m only in for a week or so and already feel engaged with the community :smiley:

I thought a lot about how we could better organize the collection of analytics data.
@paulapivat did great work of collecting and providing data for twitter analytics and on chain treasury activity. Collecting all this data manually and keeping it up to date is a lot of work and prone to errors. So I think the best we can do is to gradually standardize and automate this process.
For this reason I created this topic to start a discussion about how we can approach this.

I’ve started to work on the base architecture for a typescript GraphQL API we can use as a basis.
@Devs Please take a look at this hello world example and tell me if you like the architecture so far and your ideas of how it can be improved: github link
I think I will flesh it out a bit more and then migrate it to the banklessDAO analytics repo soon.

Here are some things we need to discuss:

  1. Where do we host the database server and api layer ?
    I saw the community website is hosted on netlify. Do you think netlify is also the right choice for
    hosting a db server (maybe a postgre docker container ?) and a graphql api?
    If not, what should we use instead?

  2. We need to somehow manage access rights for write access to the db.
    I think the most straightforward solution would be to integrate the forums account system.
    If we’d have crawler bots to collect twitter data for example maybe they could get write access
    to the db via the auth system we already have for the forum.

  3. What should we focus on first?
    The marketing guild showed much interest in the twitter analytics, so I think this should be
    the first effort we should focus on. Any ideas welcome.

Those are my thoughts so far but I do not have nearly enough experience to realize such a big project on my own. So I’m glad for any help and ideas you come up with. First I want to see if my ideas go into the right direction or if there are some better ways to approach this.

6 Likes
2

Great thought and early effort to help move the DAO forward. Did not understand most of what your ideas and questions are but just want to say how much I appreciate your willingness to contribute. Wish I had some technical skill to help out.

4 Likes
3

Thanks for this @baerlux , jumping in here to add some thoughts:

Where do we host the database server and api layer ?

I believe @garywolf (in the Analytics channel) had proposed this stack back mid-May, i’m quoting him:

"my recommendation for a low-cost but solid stack: 1. PostgreSQL as database; 2. Power BI for reporting - The idea is too avoid reinventing the wheel using python or R plotting libraries and building a front-end, open source BI alternatives as just not good enough for me, PBI is great and cheap; "

From what I’ve seen the DAO has need for several data sources from 1. Twitter Analytics, 2 POAP metrics (perhaps poap.xyz subgraph), 3. SQL queries from Dune Analytics (tho we don’t have a premium subscription that would allow us to download anything) to potentially Discord data and Data from Google Forms.

So there’s a diversity of data sources we’ll likely deal with. That might inform what kind of database server we use.

We need to somehow manage access rights for write access to the db.

We might consult the Dev Guild to see how they manage access. But if this involves integrating with forum account, then I believe @AboveAverageJoe would have some insights here.

What should we focus on first?

I think your suggestion of Twitter Analytics is a good idea as it’s something that it likely recurring and would be of interest for the Marketing Guild. Another recurring could be POAP claims metrics. I’ve been downloading CSV files from poap.gallery, but I’ve also been toying with using Python to query the Subgraph API - would love to get your thoughts on that specific script.

Those two come to mind in terms of data that’s recurring and would be desirable for the DAO.

Happy to brainstorm further :fire:

1 Like
4

Moving to archive. Please reply to reopen.