3 Comments
User's avatar
Lance Cummings's avatar

I am curious if you are developing any methods for validating AI’s analysis.

While I find this use case interesting and useful, I’m still skeptical of the reliablilty and accuracy when using LLMs on their own. I usually find problems when I seek to validate this on smaller data sets.

But i haven’t worked much with either of these.

Expand full comment
Marc Watkins's avatar

All responses still need human review. Both pulled reflections from the file and accurately associated those reflections with criteria I prompted for the sentiment analysis. Claude 2 was superior in depth and usability. NotebookLM was more prone to hallucination but still included the referenced data so I could manually double check its work.

You do need to know what you’re researching and though I didn’t code the student reflections when I gathered and anonymized them, I did read them and got a clear sense/ overview of their sentiment using the AI tools. There are a number of existing sentiment analysis tools that have used NLP you can use to preform baselines and compare.

Expand full comment
Craig Van Slyke's avatar

Nice article! I've used ChatGPT for a similar use (analyzing student-based content) and the results were decent. Also, I'm currently using Notebook LM to help me teach a doctoral seminar. It's pretty amazing. You can check it out at https://open.substack.com/pub/aigoestocollege/p/googles-hidden-powerhouse-notebook

Expand full comment