This blog series is the collaborative work of the Udemy Learning team, which includes Melissa Daimler, Chief Learning Officer; Justin Mass, Sr. Director Enterprise Learning; Joshua Ehrenreich, Senior Learning Program Manager; John O’Neill, Senior Learning Partner; Lauren Hauser, Learning Program Manager; and Vicki Lang, Learning Designer.

Generative AI is a game-changer. The impact of this technology on productivity alone could add trillions of dollars to the world’s economies, according to recent research from McKinsey. Enabling our teams to use generative AI for greater efficiency and productivity while freeing them to engage with more strategic projects is essential to keep up with the evolving nature of work. 

The final post in our series on how to upskill teams in generative AI focuses on the benefits teams could see as a result of using these tools in the flow of their daily work. Here, we present a use case from our own organization, and specifically on the Learning team. In April 2023, Udemy’s internal Learning team focused on how to learn generative AI tools and techniques. We made the following observations about the process and suggested additional approaches and tips to make the learning meaningful and engaging.

AI solution use case: Generating manager training assessment items

In revamping Udemy’s manager development program, Udemy Manager, the Learning team aimed to incorporate a skills assessment on the Udemy platform as part of the new design. This posed significant challenges in terms of time, cost, and resources. The solution — leveraging the generative AI technology, ChatGPT — led to substantial improvements in speed and efficiency while maintaining quality.

What was the challenge?

Developing a quality bank of assessment items requires considerable expertise, multiple stakeholders, and significant lead time. These assessment items also come with some vendor costs and lengthy development processes. The challenge was to generate a pilot assessment of 30 high-quality items under tight deadlines and within budget.

How would the team generate the 30 items needed without breaking the budget or the timeline? 

What were the possible solutions?

Before turning to AI, the team explored various options:

  1. Outsourcing to a vendor: This is the typical process, but because of the cost and the longer production time frame that would deliver the items past the target launch date, it was not feasible for this project.
  2. Internal Subject Matter Expert (SME) authoring: An internal learning designer could train the team’s management SME in item-writing and work closely together to develop the items themselves. However, this approach was time-intensive and created significant additional work that was not planned for. Since it was also a new skill for the team, delivery time would also be extended.
  3. Internal item-writing retreat: By recruiting 10 Udemy managers to attend a one-day, intensive item-writing workshop developed and delivered by internal learning designers, these managers might be able to write three usable items each by the end of the day. However, this approach would be resource- and time-intensive for participating managers, with the additional risk of failing to achieve usable results.

How did the Learning team leverage generative AI to explore a solution?

Each of those three solutions explored came with pros and cons. Given the team was currently learning about and experimenting with generative AI, an AI-driven solution emerged. Leveraging generative AI’s advanced capabilities in text generation, the Learning team developed a prompt that provided the generative AI tool with the learning objectives and style requirements for the assessment items. This enabled the AI tool to speedily generate an initial item set, which the team then reviewed and refined to create the final assessment. Using this approach, the team completed the project in a fraction of the time and cost that other approaches would have required. 

What were the outcomes?

  1. Time efficiency: Generating 30 items took about one day, rather than the four to six weeks a vendor typically requires.
  2. Streamlined review process: The Learning Designer could edit and finesse the items in real-time during generation, and afterward the Senior Learning Partner could also edit them directly. This approach saved hours usually spent providing detailed feedback to human authors for them to integrate asynchronously.
  3. Lower cost:  By not outsourcing the work to a vendor, vendor fees were saved.

What’s next?

The team was eager to find other applications after the remarkable success of the generative AI implementation in this use case. Ideas included:

  1. Pilot this skills assessment: Continue working on the assessment and pilot it within the revamped Udemy Manager development program, gathering feedback to guide further improvements.
  2. Explore broader applications: Share our findings with other internal teams to optimize existing processes and investigate scalability.

Additional Udemy Learning team AI use cases and resulting Outcomes 

In addition to the manager training assessment items use case, the Learning team has also found multiple other uses for generative AI in their work processes:

At the conclusion of this process, the Learning team saw many additional future opportunities for generative AI usage in the flow of work. Our recommendation is that all teams work toward the goal of finding efficiencies, reducing costs, and focusing team members on more strategic processes. 

Learn more about how Udemy can help support professionals acquire generative AI skills.

Read all the articles in this series: