Empower Your Team with Generative AI Skills: A Use Case from the Udemy Learning Team
This blog series is the collaborative work of the Udemy Learning team, which includes Melissa Daimler, Chief Learning Officer; Justin Mass, Sr. Director Enterprise Learning; Joshua Ehrenreich, Senior Learning Program Manager; John O’Neill, Senior Learning Partner; Lauren Hauser, Learning Program Manager; and Vicki Lang, Learning Designer.
Generative AI is a game-changer. The impact of this technology on productivity alone could add trillions of dollars to the world’s economies, according to recent research from McKinsey. Enabling our teams to use generative AI for greater efficiency and productivity while freeing them to engage with more strategic projects is essential to keep up with the evolving nature of work.
The final post in our series on how to upskill teams in generative AI focuses on the benefits teams could see as a result of using these tools in the flow of their daily work. Here, we present a use case from our own organization, and specifically on the Learning team. In April 2023, Udemy’s internal Learning team focused on how to learn generative AI tools and techniques. We made the following observations about the process and suggested additional approaches and tips to make the learning meaningful and engaging.
AI solution use case: Generating manager training assessment items
In revamping Udemy’s manager development program, Udemy Manager, the Learning team aimed to incorporate a skills assessment on the Udemy platform as part of the new design. This posed significant challenges in terms of time, cost, and resources. The solution — leveraging the generative AI technology, ChatGPT — led to substantial improvements in speed and efficiency while maintaining quality.
What was the challenge?
Developing a quality bank of assessment items requires considerable expertise, multiple stakeholders, and significant lead time. These assessment items also come with some vendor costs and lengthy development processes. The challenge was to generate a pilot assessment of 30 high-quality items under tight deadlines and within budget.
- Time was short, with only four months between the kick-off exploration and the target launch date, whereas developing an assessment with the typical process takes closer to six months.
- The estimated vendor cost of the typical process, in addition to the cost of internal resources — would be beyond the limited budget allotted to the project.
- Human resources for the process were also limited, with only a few days available for two team members to dedicate to the project, whereas the typical process requires much more time from six or more people.
How would the team generate the 30 items needed without breaking the budget or the timeline?
What were the possible solutions?
Before turning to AI, the team explored various options:
- Outsourcing to a vendor: This is the typical process, but because of the cost and the longer production time frame that would deliver the items past the target launch date, it was not feasible for this project.
- Internal Subject Matter Expert (SME) authoring: An internal learning designer could train the team’s management SME in item-writing and work closely together to develop the items themselves. However, this approach was time-intensive and created significant additional work that was not planned for. Since it was also a new skill for the team, delivery time would also be extended.
- Internal item-writing retreat: By recruiting 10 Udemy managers to attend a one-day, intensive item-writing workshop developed and delivered by internal learning designers, these managers might be able to write three usable items each by the end of the day. However, this approach would be resource- and time-intensive for participating managers, with the additional risk of failing to achieve usable results.
How did the Learning team leverage generative AI to explore a solution?
Each of those three solutions explored came with pros and cons. Given the team was currently learning about and experimenting with generative AI, an AI-driven solution emerged. Leveraging generative AI’s advanced capabilities in text generation, the Learning team developed a prompt that provided the generative AI tool with the learning objectives and style requirements for the assessment items. This enabled the AI tool to speedily generate an initial item set, which the team then reviewed and refined to create the final assessment. Using this approach, the team completed the project in a fraction of the time and cost that other approaches would have required.
What were the outcomes?
- Time efficiency: Generating 30 items took about one day, rather than the four to six weeks a vendor typically requires.
- Streamlined review process: The Learning Designer could edit and finesse the items in real-time during generation, and afterward the Senior Learning Partner could also edit them directly. This approach saved hours usually spent providing detailed feedback to human authors for them to integrate asynchronously.
- Lower cost: By not outsourcing the work to a vendor, vendor fees were saved.
What’s next?
The team was eager to find other applications after the remarkable success of the generative AI implementation in this use case. Ideas included:
- Pilot this skills assessment: Continue working on the assessment and pilot it within the revamped Udemy Manager development program, gathering feedback to guide further improvements.
- Explore broader applications: Share our findings with other internal teams to optimize existing processes and investigate scalability.
Additional Udemy Learning team AI use cases and resulting Outcomes
In addition to the manager training assessment items use case, the Learning team has also found multiple other uses for generative AI in their work processes:
- Utilized ChatGPT to synthesize the top three skills desired from a qualitative application question within a mentoring program application. Leveraged across 100+ applications, this approach enabled deep insights into skills development within the organization, saving hours of time and increasing quality-of-match.
- Utilized ChatGPT to create course descriptions requiring a specific word count. With just a couple minutes spent providing prompts on key points about the course and a suggested word length, ChatGPT was able to provide a draft that was then quickly edited and utilized. This early drafting saved 30 minutes of administrative time per course created.
- Utilized ChatGPT to synthesize wording on early drafts of a newly launched Career Levels framework, prompting it to offer suggestions to make descriptions more concise while supporting consistency across descriptions for progressive levels. ChatGPT was also prompted to create a framework from scratch around identified categories as a quick stress test of the framework in development.
At the conclusion of this process, the Learning team saw many additional future opportunities for generative AI usage in the flow of work. Our recommendation is that all teams work toward the goal of finding efficiencies, reducing costs, and focusing team members on more strategic processes.
Learn more about how Udemy can help support professionals acquire generative AI skills.
