I am curious if you are developing any methods for validating AI’s analysis.
While I find this use case interesting and useful, I’m still skeptical of the reliablilty and accuracy when using LLMs on their own. I usually find problems when I seek to validate this on smaller data sets.
All responses still need human review. Both pulled reflections from the file and accurately associated those reflections with criteria I prompted for the sentiment analysis. Claude 2 was superior in depth and usability. NotebookLM was more prone to hallucination but still included the referenced data so I could manually double check its work.
You do need to know what you’re researching and though I didn’t code the student reflections when I gathered and anonymized them, I did read them and got a clear sense/ overview of their sentiment using the AI tools. There are a number of existing sentiment analysis tools that have used NLP you can use to preform baselines and compare.
I am curious if you are developing any methods for validating AI’s analysis.
While I find this use case interesting and useful, I’m still skeptical of the reliablilty and accuracy when using LLMs on their own. I usually find problems when I seek to validate this on smaller data sets.
But i haven’t worked much with either of these.
All responses still need human review. Both pulled reflections from the file and accurately associated those reflections with criteria I prompted for the sentiment analysis. Claude 2 was superior in depth and usability. NotebookLM was more prone to hallucination but still included the referenced data so I could manually double check its work.
You do need to know what you’re researching and though I didn’t code the student reflections when I gathered and anonymized them, I did read them and got a clear sense/ overview of their sentiment using the AI tools. There are a number of existing sentiment analysis tools that have used NLP you can use to preform baselines and compare.