Data about Yellowdig's clients and users were stored in:
Datomic - Powers Yellowdig's main app, housing user data and events.
Elasticsearch - A database of events used to optimize search in the app and to aggregate and visualize client-level participation metrics in Kibana.
Beamer - I chose and integrated this for user announcements and continuous NPS polling of 1% of users, it contains NPS scores and written feedback on the platform and features tied to specific user profiles.
Thinkific - Yellowdig's LMS used for our automated certification courses, it allowed us to know how much interaction Educators had with our courses and when.
Salesforce - Contained data on interactions different users had with Yellowdig personnel and key information about contracts that was helpful for servicing clients.
Even though I could personally do many of the aggregations and analyses myself, every dataset did not have straightforward keyed variables to merge them easily. The time it took to manage the merging and aggregations made doing many analyses hard to justify without some clear goal; exploring just to try to create helpful insights was effectively off the table.
We had many levels of data within all of Yellowdig's systems and client questions frequently cut across them.
Yellowdig's network system hierarchically houses data by:
Organization (institution, company, etc.)
And Sub-Organization (schools, departments, business units, etc.)
Inside of those network nodes were:
Community - often numbering in the hundreds or thousands (e.g, classes taught by Educators or clubs run by Learners)
In communities, Organization members could be:
The Owner (the community creator and main admin -often Educators)
Facilitators (other admins -often co-instructors and TAs)
Members (often Learners)
Distinctly challenging was that stakeholders sometimes wanted a combination of both aggregated metrics and highly specific raw data to explore questions. For example, someone looking into the impacts of an organizational setting change might want to know when a user made a specific change and then want aggregated data from before and after that, to see if community engagement got better or worse.
The more complex the aggregations and the more places the data were housed, the more complicated the data merges, the more likelihood of errors by our team, and the longer it took to provide the requested data.
I focused first on creating an app for the GTM teams with just a couple of the Yellowdig data sources, which I could build on as needed.
Though it was previously possible for an analyst, data scientist, or software engineer to run analyses and merge data across these sources, doing so was beyond the capabilities of most of Yellowdig's GTM teams. That made many deeper data inquiries from clients require expensive centralized analytics resources or created distractions from product building for the engineering team.
Even if the Yellowdig platform itself could have been engineered to do many of these things, it would have required extensive engineering resources that slowed delivery of other important roadmap features. It would have been difficult to meet the wide variety of needs expressed by so many stakeholders. Also, developing it traditionally would have required a long specification and design exercise; I was already aware of most of what it needed to do from working with clients, training Success managers, and helping the Partnerships and marketing teams.
A challenge for building more dashboards in Yellowdig was that certain desired calculations were mathematically complex. It was historically a challenge for building dashboards to know what calculations might be slow based on the structure of the data, even for an engineer, until they had already spent significant time trying to build it. Using AI on small amounts of data to prototype, I was able to get a better sense of the bounds of what was quick and then could quickly decide what constraints needed more engineering, like pre-calculating, creating parallel processes, or serializing calculations. I could also make informed decisions about what statistics simply might not be worth including.
It was important our GTM teams could always represent data accurately to clients and I expected them to copy visualizations I wanted clients to easily understand, so as I built I communicated most of the math backing the calculations as copy or in help icons. That also meant most looking at the app could understand what it was meant to do and how it was supposed to be doing it. I did create a basic spec doc and saved the AI prompting throughout the process, but from my prototype almost anyone hunting a bug or maintaining the code would be able to understand the intended functionality.
It was difficult to get employees appropriate access to five systems in the first place, let alone keep up with managing their ability to get logged in to all of them. As data security protocols got stricter (e.g., achieving Soc II Type 2 compliance), risks got bigger (i.e., more clients and users) and as the company expanded, this was no small consideration.
Helping employees understand how to use all of the systems was also a major educational challenge and they would often forget after being trained, since they did not use many of them daily. Employees would need to distract others to get help using some of the systems, especially if we did not have good internal documentation. Often, employees in a hurry would just skip using them if they could not get in or did not know how. Though I do not think clients often noticed, the team was slowed down by multiple systems, was losing opportunities to wow clients with quick answers to their questions, and was less able to encourage specific action or change that would have ultimately improved their outcomes and both their revenue and Yellowdig's revenue.
It had one universal filter users could easily get familiar with controlling the scope of data for all dashboards and resports:
It allowed you to analyze data from any combination of organizations or suborganizations.*
You could set a date range for the data, if desired.
You could select suborganizations, communities, or users to focus on.*
You could select any level of user to focus on (admins, network members, owners, facilitators, community members).*
Setting the filter triggered various underlying data operations. Most calculations would appear near instantaneously as you then navigated the app. If you were preparing for a meeting you could configure filters that made sense for the client a few minutes ahead of time and the app would have even the beefiest aggregations and calculations ready for even the biggest clients.
The app then had many pages for different purposes:
Dashboards that presented data aggregating for different levels of the filtered data:
Organization and Suborganization activity totals and trends.
Community-level data with activity by user groups (e.g., community admins versus members) and engagement patterns over time.
Activity by individual filtered Users including community names where they participated and graphs showing their frequency of participation over time.
An AI chat interface that allowed natural language queries of the filtered data.
Customizable report templates that used the AI interface to auto-generate common reports for regular meetings or interesting analyses for other Client Success meetings, including:
Quarterly Success meeting reports.
A report highlighting healthy communities to guide Designers and enable institutions and new users to easily learn from their own most successful community leaders.
Thematic analysis to get a high level understanding of organization or community content and what students were discussing.
Sentiment analysis to understand whether students were interacting helpfully and happily or were otherwise frustrated or interacting unnaturally.
It was possible to copy any visualization or table into a report or office suite application and the dashboards could be exported to PDFs or CSVs for sharing or further analysis. Everything also hyperlinked organization, community, and user names to jump directly to the Yellowdig platform where users could easily take advantage of whatever of Yellowdig's existing functionality they had the permissions to use.
Dashboards answered many typical clients questions or requests.
The AI chat interface enabled responsiveness to rarer questions and client-specific needs.
The Success and Partnerships teams could immediately address client needs during a meeting, for example, rather than creating a new Support ticket or data inquiry that would create a delayed response. Clients liked the responsiveness and it made employees come across as more knowledgable and capable.
With Excavator many support tickets became as easy as enacting a filter and looking at a dashboard or asking the AI chat interface a question, rather than needing to know and use significantly more complex Datomic and Elasticsearch queries or signing in to the other data warehouses.
Analysts could access already-merged and aggregated data that could be easily exported for additional manual analysis.
It was able to produce some fairly sophisticated one-off analyses, though it would sometimes take a long time and eventually fail for larger analyses with large organizations.
It was often good at producing quick visualizations to better understand or illustrate interesting data patterns noticed in other dashboards.
Useful for supporting internal business decisions and and making on-the-fly strategy choices. For example, it extrapolated user growth forward a year based on recent usage patterns, which could guide Success interventions or estimate hiring needs and AWS budgets.
Conservatively our report generation and prep for a quarterly success meeting took upward of 8 person hours per client.
Though the app took months in total elapsed time to build between my other tasks, it likely took no more than 120 hours of my dedicated attention. In 20 meetings that time was recovered.
With 200 clients we were spending almost 6,400 hours a year among three distinct employees (a data scientist, reporting specialist, and the assigned success manager) on quarterly meetings. We were not really keeping up that intended cadence or were shortchanging reports if they were not experiencing success problems, which did not help with communicating value.
The app:
Cut total meeting preparation and made it possible to produce the reports and analysis that could optimize growth with all clients. Reports could be generated in <10 minutes.
Allowed each Success Manager to prep for their own meetings on their own time:
Allowed two employees working nearly full-time on report prep to focus on more value-added and less repetitive work.
Avoided those employees being a bottleneck when unavailable (e.g., overtaxed or on PTO); interruptions happened often at certain times of year.
It eliminated the need for Success Managers to accurately communicate details of report requests (e.g., contract dates) and meetings or Slack conversations to iterate on the results.
It reduced mistakes and the costs of them:
Created a single, easy to use filtering interface for scoping reports and alterable templates that triggered a standard version.
Request parameters were clearly documented within the report, so everyone understood the parameters and could notice an error before the meeting (e.g., partnerships employees who attended).
Quick regeneration nearly eliminated the costs of a mistake in the setup.
Once tuned and checked, the app automated many manual report generating processes that were individually subject to errors that could be hard to detect.
Allowed Success Managers to focus on strategy, not busy work:
They could focus on adding context-dependent advice or reshaping the report to meet strategic partnership goals.
Additional analyses or visualizations could be prompted in plain language to explore interesting patterns in the standard report, to help motivate specific action, or to quickly follow-up on partner questions generated by the meeting and report.
Created more consistent reports and a higher floor for quality:
When manually generated, the Success Manager's experience and attention to detail or the team's overall available time had clear impacts on the quality of reports.
Better reporting tended to drive better feedback on quarterly meetings that was less dependent on the Success Managers prior experience.
As we gained trust in the system, we believed something similar could safely be used with good impact directly by our clients. Doing so could remove us as a go-between for many of their typical data needs.
Sharing some details and walkthroughs of Excavator with clients, it seemed some would also be willing to pay a premium for something similar. Based on those conversations it seemed, at minimum, we could safely cover the costs associated with offering the new product.
It would enable new capabilities for them using better access to student data, like more quickly identifying individual students who are disengaging and allowing advisors or Educators to intervene sooner.
It would also enable institutional research divisions to more rapidly do analyses. I had seen these divisions tended to be resource constrained and, not surprisingly, they were much more likely to do efficacy research that tended to be very good for us when it was easier for them to execute.
Our engineers worked with Synthetiq as they enabled the ability to easily add more sophisticated identity controls and data guardrails. We ensured no possible data leakage across institutions, which was very necessary for a client-facing deployment.
We also connected the app to AWS Bedrock:
That removed the need for individual users to each have more expensive AI accounts.
The Bedrock connection eliminated possibilities for student data to be used for training AI systems, if accounts integrated with the apps were not properly configured.
It helped us centrally monitor the costs of AI as they were deployed across more users.
It enabled us to centrally control the AI models used, helping to control AI costs and exercising better quality control as models were released and sunsetted.
It is now being offered as a new product marketed as AI Insights.
In the first versions of my app I realized some load times were taking just long enough that I would flip to a different browser window or grab my phone. It was just enough long enough to want to use that time for something, but not long enough to get anything meaningful done. I thought about how I could keep a user's attention on the app or bring it back. Initially I just added a sound after the load completed, but if I had shifted to another task and especially an actually useful one, it still took me time to switch back.
I thought about ways I could meaningfully use that loading time instead. Knowing our GTM teams would be using it, I decided to create an adaptive multiple-choice trivia game based on concepts we used at SmartyPal. They could spend the short loading times learning a few things about our product. It would keep a quick glance away during a 1-minute from turning into 10 minutes of reading emails (or watching cat videos!). I quickly scraped our knowledge base and made a bank of over 100 questions to load into the game via an Excel spreadsheet template that could be easily updated to add questions later. I made the game end whenever the load finished, after the user answered their last question. I even created a really quick dashboard in the user profile that gave the learner feedback on what they knew, allowed them to independently choose to play the game, added a leaderboard to celebrate the most knowledgable, and created a dashboard in the admin accounts area to show a manager how much different users played and what they knew.
The game and underlying reporting system was not as sophisticated or production-ready as what we built at SmartyPal. I was ultimately able to tame the load times before anyone dedicated more time to it, so the game was removed from later versions.
Nonetheless, that game and the Excavator app itself were eye-opening for me personally. In only about two hours I was able to create a simple game that played in a browser which very closely mimicked a platform using basic versions of the algorithms we had spent months developing. A decade earlier we had gotten an innovation grant worth almost $1M in total to build a game system I prototyped from nothing inside two hours. The prototype even included a bank of over 100 real learning items that a couple of years earlier would have taken longer to write than I spent building the entire game.
Suffice it to say I no longer needed to be convinced of how much AI was going to change developing digital technology.