Confidentiality Reminder
As a reminder, the information shared during this event is confidential and shouldn't be shared with anyone else.
If you are only able to watch the recording of the event, please be sure to complete the event survey after viewing.
Product Sneak Peek Summary - February 15, 2023
Presenters:
Prasad Raje - Chief Product Officer
Jacob Coulter - Sr. Director of Product Management - Instructors
Kristin Pyun - Sr. Director of Product Management - Organization Success
Ajay Waghray - Director of Product Management - Marketplace Conversion
Grant Lindsay - Sr. Director of Product Management - Learner Success
What is a Product Sneak Peek event?
- Disclosures:
- Instructor Partners are under NDA
- Product Roadmap is not set in stone and subject to change in terms of focus and timeline
- This presentation is not comprehensive of all product investments
Jacob Coulter:
- Help learners get “unstuck”
- Encourage better learner outcomes and higher consumption of your content
- Decrease the effort and potential dependency on teaching assistants
- Over 1 millions questions per year in the current setup, not best setup to help you respond to high volumes
- As part of the beta, 52% of responses generated are being posted without edits which is very promising
- More research and enhancements to come
- Offer learners opportunities to develop and validate skills
- Align practice tests with certification exams with the new “fill in the blank” question type
- Increase creating and updating content with AI
- As part of the beta, 55% of instructors that consumed coding exercise insights went on to make an edit within 5 days
- Sign up for the Beta here
- Provide data and insights on content consumption and performance for continuous content improvement
- Based on results, looking to extend to quizzes, labs, and assignments
- Orient your creation efforts to organizational skilling needs
- Increase usability with localization and data pipelines
Kristin:
- A skills-based organization is a company where skills & capabilities are the basis for talent management & workforce decisions instead of traditional jobs. Instead of asking themselves “How do we get people who can fill these jobs?”, skills-based organizations are asking themselves “How do we get people who have these skills?”.
- Enables admins and L&D leaders to easily find content that builds needed skills
- Improves learner experience by more effectively mapping skills to content
- The solution we’re currently concept testing with UB customers is a three step flow. The first step of this solution is information gathering from the admin on what skill they want to further develop and how that skill will be used in practice. The next step in the flow is for us to generate what we are calling a skill tree. For our beta a skill tree is a 3 level hierarchical breakdown of the high level skill into more specific sub-skills that the learner will need to demonstrate. The final step in the flow, is the skill to content mapping to create a cohesive learning path aligned to the skill tree.
Ajay:
- Re-designed page to show portfolio of projects to attract professional learners
- With higher brand & product awareness, result is more traffic and subscription starts
- New frontend architecture on more modern tech stack which improved Google Core Web Vitals metrics by 30-40%
- Faster performance leads to better SEO ranking & more traffic to your content
- Highlighting the learning activities you build on the course landing page
- Surfacing value for learners for practice tests after instructor investment
- Increases brand reputation and learner outcome verification
- Aims to increase conversion for certification prep courses
Grant:
- Enable users to discover courses through natural language queries and conversations to simplify the content discovery process
- Provide learners with a personalized on-demand chatbot that can help them get unstuck by providing timely support
Q&A:
Question: “Does Udemy train generative AI models using instructor content?” / How do I know how Udemy is using my content in how you use AI?”
Answer: Udemy’s currently planned upcoming gen AI capabilities currently do not involve training any generative AI model directly on instructor content. We achieve the instructor specificity in our GenAI features by a variety of techniques like Retrieval Augmented Generation that do not involve Generative AI model training.
This is a rapidly evolving field and things might change in the future, but we have not yet pursued model training.
To clarify, when we create features like the Learning Assistant, to get the answers from your course content, it’s not necessary to take your content and apply it to a large language model. The way we accomplish the specificity in the answers is through the RAG technique described above. That’s what we mean when we say we don’t do training.
Question: “Why does AI-assisted Q&A only show up for some questions in courses where the feature is available?”
Answer: A response will only appear if:
- In an appropriate category/sub category (only certain categories/subcategories are included in the current beta)
- The original post does NOT have an image attached
- The post is ~350 words or less
- There is also a timing mechanism. You can anticipate a 5-11 hours delay in responses being available since our workflow runs every 6 hours.
Question: “For AI skills mapping - How can I make sure my content is chosen for the skills map? Is there some way to see if my content is getting chosen for skills maps?”
Answer: To drive the mapping of content to skills we are leveraging course topics as well as course transcripts. Beyond continuing to invest in the accuracy of topic tagging and transcripts, there is nothing additional you need to do.
All consumption of content within a learning path counts towards your overall UB consumption in the UB revenue share calculation, so if your content is often getting chosen for skill-based learning paths, you should see your revenue go up. Once the product is released to a wider admin audience, we will explore explicit ways to expose which content is getting chosen for these skill-based learning paths. But also keep in mind that given the org specific nature of these learning paths, that data may be hard to action against. So we may want to explore exposing other types of data first like which skills we are associated or not associated with your content to help inform you for future content development.
Question: “Is there a place where I can see the badges that are being highlighted in certification prep?”
Answer: Udemy is aligned with open badging standards through our partnership with 1EdTech, and we surface courses for certification prep in accordance with those guidelines.
Question: “For AI-Learning Assistant - What happens if a learner asks the AI learning assistant a question that isn't covered in my course?”
Answer: Our goal is to keep users learning in your courses. The assistant will use content from the lectures in the course, your responses in Q&A, and the intrinsic knowledge of the LLM to try and answer the question. If it is not able to answer the question, it will direct the user to post their question to the Q&A forum in the course.
Question: “How are these practice test improvements different from what Udemy launched in 2023?”
Answer: This is the same scope of work from last fall. The updated beta launch will offer improvements to the instructor experience to create and maintain practice tests, a new question/answer type, and the ability to generate answer explanations with AI. Beta set to be released to select instructors by end of month.
Question: “I created a course through the UB Content Opportunities tool, but it hasn’t been added to UB yet…why not?”
Answer: The UBCO tool is intended to be a guide to help you make decisions about what courses to create next. However, courses created through the UBCO are not guaranteed to be curated into UB. All courses published through the UBCO tool are still required to meet our curation standards. We monitor course performance on a weekly basis and courses will be added to the collection if the team determines they are a good fit.
Post-event Q&A summary for Instructor Partner Hub
Thanks again to all who joined the Product Sneak Peek event, and for engaging with questions and comments throughout. While we cannot answer every question submitted, here are some responses to the most commonly asked questions:
AI-assisted Q&A and Practice Tests
What’s the ETA to launch the AI Q&A to all instructors? And will that be available in other languages? Do we have an ETA for multi-language?
- At this point in time, we do not have a timeline for expanding the current beta launch to more instructors or languages. We are evaluating the performance of the current beta tool by conducting data analysis and instructor feedback interviews. Based on those results, we may need to enhance the tool before scaling to a broader audience with additional language capabilities. Once a decision is made, we will communicate it to our instructor community.
Where do we get the beta forms for AI Q&A and Practice Tests?
- There currently is no access beta form for AI-assisted Q&A. As mentioned in the session, beta access is limited to English courses with IT Certifications, Network & Security, and Programming Languages subcategories. We will let you know when access is expanded.
- You can access the Practice Tests beta interest form here
Showcasing Practice Tests - is it also applicable for non-coding exercises? If a practice test is available for a non-coding course, will it be highlighted like this?
- This product feature is in the early stages of development, but will most likely be shown for all courses with practice tests.
How do you think about practice tests for “soft skills”? It doesn’t seem as straightforward an application as, say, coding.
- While we're enhancing our practice tests to better align with third-party certification exam experiences, we're not limiting its use cases. Instructors are free to use the practice tests as they see fit, whether for assessing technical skills like coding or evaluating softer skills.
UB Content Opportunities Tool
Why is the content opportunities tool not updated as new insights emerge?
- The Udemy Business Content Opportunities Tool is updated on a quarterly basis. In the future, we hope to update the list of opportunities in near real time.
With allowing customers to select topics for UB, can you not also allow them to pick courses from the marketplace to include too? This could super charge inclusion into UB.
- Udemy has a process for this already! As you can see in the link here, commercial Udemy Business customers can suggest a Marketplace course for the UB collection.
- (For more context on the Udemy Business Content Opportunities tool, check out the Teaching Center article here).
Skills Tree
Is it possible for us (as instructors) to see what the admins of Udemy Business can see in the skill tree? Is there any way we can optimize to be included in the skills tree?
- Not at this stage of development. We'll first be releasing this feature as a beta, and expect multiple iterations to improve the quality of auto-generated skill trees to meet the needs of Udemy Business customers. However, we recognize the value in exposing skill tree data to instructors to inform content creation, so we plan to explore this in the future.
Badging
How do badges directly map to courses? How can we make sure our courses can be mapped to a badge?
- Badges are mapped to eligible courses based on the topic relevancy and alignment with open badging standards. Eligible topics (such as many certifications) are those supported by Udemy’s partnership with 1Edtech. No action is required by Instructors with eligible courses. Check out the article here for more information.
Learning Assistant
Question regarding the Learning Assistant tool. Can we view chat logs that students have with the tool? This would give more insight on what students are getting confused about.
- Feedback from learner conversations will be provided to instructors that enable insights into where learners are struggling so that the can improve their course. As we mentioned in the presentation, we are in the early stages of development for the AI Learning Assistant. The specific mechanism and timing of how this will work and how it will be surfaced to the instructor has not been determined. The team is considering offering an Instructor beta for this feature down the road--we will keep you posted on the developments here
With the Learning Assistant Chatbot if a learner asks a question that is outside the scope of the course what would happen? Pull from other courses or a wider Udemy LLM?
- Our goal is to keep users learning in your courses. The assistant will use content from the lectures in the course, your responses in Q&A, and the intrinsic knowledge of the LLM to try and answer the question. If it is not able to answer the question, it will direct the user to post their question to the Q&A forum in the course.
Regarding Udemy Learning Assistant, as an instructor, which course metadata contributes to this factor? I.e. what improvements should I make to my courses to make it effective for this assistant?
- The assistant will use content from the course including lectures in the course and the responses in Q&A to try and answer questions.
What is the difference between Learning Assistant and the AI-assisted Q&A chatbot?
- AI Q&A is a tool for instructors that seeks to assist you in answering high volumes of questions from Q&A by leveraging your previous answers, course content, and AI to suggest a response for you to post. Allowing you to edit before formally responding to your learner.
- The learning assistant is a tool to help learners get unstuck when learning new skills by providing them with a personalized on-demand AI chatbot that can provide them with timely support.
AI Strategy questions
How will AI features handle course updates or complete overhauls of courses?
- Ability to leverage generative AI to write test cases in a coding exercise
- [In beta] Ability to leverage generative AI to create answer explanations in a practice test
- [In beta] Ability to leverage generative AI to suggest a response to a learner question via the Q&A forum
- We will continue to closely follow the emergence of generative AI and create product experiences that support our instructors in meeting the skill-based needs of learners and organizations around the world.
These AI features will depend on the quality of the data they are trained or augmented with. Where do we stand on improvements to course transcripts that would presumably fuel these features?
- Currently, there are no roadmap commitments to improve course transcripts, but that is always subject to change depending on business strategy, data, and user feedback.
- Generally speaking, as it relates to assessing the quality of AI-generated content, we leverage a combination of subject matter expert review, AI assessment, and user behavior signals to determine whether or not an ML or AI solution addresses each specific user problem effectively. The details of this evaluation will vary by use case.
There seems to have been very few implementations from suggestions from the Idea Exchange. Are any there on the timeline for implementation?
- Ability for the coding exercises platform to support Kotlin language
- Ability for the coding exercises platform to support React 18
- Updates to the instructor flow to unpublish/publish a coding exercise
- Addition of funnel percentages to the coding exercises insights data visuals
- Usability updates in the curriculum builder when adding curriculum items
- Addition of mobile insights (device breakdown) and minutes taught her active learner to the course engagement section of the performance dashboard
- Better filtering in the performance dashboard to allow visibility into historical performance data
- Addition of Month-Over-Month and Year-Over-Year tooltips to consumption graphs of the performance dashboard
Can you speak about how Udemy is thinking about how AI transforms business learning?
- At Udemy, we are actively considering how AI can revolutionize business learning as part of our commitment to enhancing lives through education. By leveraging artificial intelligence, we aim to optimize the learning experience for our instructors, learners, and organizations.
- Our approach to integrating AI into our platform is driven by a dedication to offering top-quality, pertinent learning opportunities, that stem from the collective expertise of our instructors and our dynamic learning environment. We prioritize the responsible application of AI, engaging in deliberate testing and learning processes that uphold our commitment to delivering a trustworthy platform.
- By creating tools and experiences backed by AI, we aim to empower and unite our community of learners and instructors, reinforcing the competitive edge offered by our marketplace.
- Our AI roadmap entails developing tools to assist organizations in achieving business objectives efficiently and staying ahead in a competitive landscape through tailored, engaging, and dynamic learning experiences.
- By identifying skills gaps and offering personalized learning paths, we aim to enhance employee engagement and skill acquisition across the organization, providing tailored content in various languages to cater to individual learning goals with measurable results.
- Recognizing the time constraints faced by learners, we are dedicated to facilitating faster skill acquisition and credential attainment through improved search functionalities and personalized learning experiences.
If you can create a chatbot for students, why can't a company create a chatbot for its employees without Udemy?
- The chatbot created by Udemy aims to provide assistance to learners, organizations, and instructors within the Udemy Intelligent Skills Platform, ensuring a consistent learning experience for all users and democratizing learning. Companies may opt to develop their own chatbot to support their employees, though this is typically a viable solution for larger companies with the necessary expertise, resources, and financial means for development and maintenance. By incorporating the chatbot within the Udemy platform, all learners, organizations, and instructors can benefit from chatbot support.