This is super helpful. I routinely teach how to write literature reviews for PhD students - and how to "read" for them efficiently. My sense is that AI can do some of the heavy lifting, but not nearly enough, especially, as you say, when it's a more obscure topic. I will continue exploring the effectiveness of other tools to carry out this task (did you try jenny.ai or claude? just curious). Anyway I'm going to save this and share the next time I teach academic reading. thx!
Thanks for the recommendation Delia, I'll try jenny and let you know! Claude performs not unlike perplexity and scholargpt, but I bet it would do better if I had the pro version and trained it on longer conversations about what I'm looking for. But since I know exactly what I'm looking for in this experiment, I wonder if it's cheating to tell these bots "go find the community archive I know is out there" when in a normal lit review that's something we'd want to discover.
This is an incredibly important post. Thanks for sharing. As someone who teaches research to secondary school students, it's important to be aware of the current shortfalls of truncating the research process, especially with regard to more niche or esoteric topics (which is not all that surprising). The million dollar question is whether some of these companies will negotiate deals with archival digital resources and other hard to access information in order to expand their data set. Given that this seems to be the direction we are moving, my guess would be yes, but I think there will still be a place for manual research to ensure everything is captured. But humans miss things also.
Great question Steve. I hope they do! If I could use these tools and be confident that they'd send me metadata for sources I should go find in person, and/or access hard-to-find community archives, I'd be all in.
Nice write up! Thanks for the analysis. Did you test some of the more thorough tools such as OpenAI Deep Research, SciSpace or the Pro version of Elicit? One could argue that you focus on the tools that are known to be not that good in the first place.
Hi Michael, yes, this is a great and fair question! SciHub performs a bit like Elicit, it finds papers that address the question about integration of language and ecology in an interesting way, but both really struggle to discern which Dene language is which. I wonder if they are both using citation count as a heuristic, which I think leads them to pull examples from other languages over more relevant texts. I tried this once on the Pro version of Elicit when I had access to it a few months ago with similar results (but didn't pay for the pro version to try again now). If anyone has access I'd welcome them to try it and I'd be curious to hear if it has improved, same with OpenAI DR.
Whether or not prioritizing articles with a high citation count is a problem might depend on your field. In fields like anthropology, where scholarly and primary resources from within a community and in grey literature are exactly what you want to find, it might be more of an issue.
Btw, I probably would not have asked for the synthesis in the same prompt. In order to create a good synthesis, the LLM needs to put the entire documents into the prompt and that runs up the token limit pretty quickly. As a result the LLM might limit the number of sources it will use. The way I would approach it is to ask for general literature research first, then run the resulting sources through something like ResearchRabbit or Semantic Scholar to identify anything that might be missing. After that I would review the sources manually and identify the ones most relevant, feed the full text back into the LLM and ask for the synthesis. That manual step is currently necessary to circumvent input token limits. You could also ask the LLM to create abstracts for each source first and then do the synthesis based on these abstract, but I find that this usually gives inferior results.
Ok, here you go. OpenAI DR always comes back with some follow up questions first. This is the entire dialogue after putting in your prompt. Substack would not let me copy the links, so this text ends up being a bit of a mess to read inside a comment reply. But I'm assuming this gives you enough info.
--- DR ---
To ensure a thorough and systematic literature review, could you clarify the following details:
Timeframe: Should the literature review focus on recent publications (last 10 years) or include older foundational works?
Sources: Do you prefer academic journal articles, government reports, indigenous organization reports, or all of the above?
Languages: Should I focus only on English sources, or include sources in Dene kedǝ and other relevant languages?
Scope: Should I include literature on general indigenous language and cultural revitalization, or strictly limit to Dene kedǝ, Dene ts'ı̨lı̨, and Dene ts’ǫ́ dane in the Sahtú Region?
Synthesis Format: Would you like the 300-word synthesis to be structured according to the three key objectives outlined in the grant proposal, or as a general summary of the literature?
Let me know so I can tailor the research accordingly!
--- me ---
Consider all foundational works and all relevant sources. Include sources in other languages. For the other two questions, use your best judgement.
--- DR ---
I will conduct a systematic literature review on the revitalization of Dene kedǝ (North Slavey language) and Dene ts'ı̨lı̨ (Dene ways of life), with a focus on Dene ts’ǫ́ dane (Dene youth). This will include:
1. A list of the most relevant academic and institutional sources, including foundational works and recent studies, across English and other relevant languages.
2. A 300-word synthesis summarizing key insights related to the three core objectives outlined in your grant proposal:
– The status of revitalization programs for Dene kedǝ and Dene ts'ı̨lı̨
– Data trends regarding language and cultural continuity
– Lessons learned in indigenous language and cultural revitalization within governance and policy frameworks.
I will let you know once the research is ready.
List of Relevant Sources
• “From Dene Kedǝ to Dene Ts'ı̨lı̨: Rethinking Resurgence in the Sahtú Region” – A comprehensive knowledge synthesis report (2017) reviewing 50 years of Sahtú Dene/Métis language and way-of-life initiatives (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region) (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). It identifies key domains of resurgence and lessons learned for Dene kedǝ (language) and Dene ts'ı̨lı̨ (culture).
• Dene Kede Curriculum (1993) – A foundational Dene language and culture curriculum for NWT schools, developed with Sahtú elders. Celebrated by the Royal Commission on Aboriginal Peoples for its cultural grounding (), it set expectations for Dene kedǝ as a first or second language in education.
• Language Nest Programs (2000s) – Early childhood immersion “language nest” programs established in 18 NWT communities (including Sahtú). These were one of the few formally evaluated Indigenous language initiatives, and showed successes in engaging infants with fluent Elder caregivers ().
• Tutcho (2016), “The language is like ets’ulah” – A Master’s thesis by a Sahtú Dene researcher exploring Dene love songs as a vehicle for language revitalization in Délı̨nę. It highlights youth perspectives, noting that English dominates among younger generations and urging pride-based strategies to strengthen Dene kedǝ ().
• NWT Bureau of Statistics – Language & Cultural Data – Longitudinal survey data (1984–2014) on North Slavey use and traditional practices. These data show steep generational declines in Sahtú Dene language fluency since the late 20th century (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx), alongside trends in on-the-land activities (hunting, fishing, etc.) (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx).
• Sahtú Governance and Cultural Initiatives – Examples include the Nę K’ǝ Dene Ts'ı̨lı̨ (Living on the Land) Forum, a regional co-management advisory body integrating Dene ts'ı̨lı̨ in resource governance (), and the Dene Ts'ı̨lı̨ School, an on-the-land cross-cultural education program for youth (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). Such initiatives tie language/culture revival to modern governance structures.
• NWT Indigenous Languages Framework and Action Plan – Territorial policy documents (2010–2018) that support community-driven revitalization. They outline funding transfers to regional Indigenous authorities and emphasize integrating Indigenous languages into public services and education (Indigenous Languages and Education | Education, Culture and Employment) ().
• “Dene Heroes of the Sahtu” (2017–2020) – A community literacy project engaging Sahtú youth in publishing books about local heroes. It improved youth literacy and cultural pride, honoring Dene traditions and language, and earned the 2018 Arctic Inspiration Prize (Dene Heroes of the Sahtu | Royal Roads University) (Dene Heroes of the Sahtu | Royal Roads University).
Synthesis of Key Findings (300 Words)
Status of Revitalization Programs: The literature reveals a wide range of revitalization efforts for Dene kedǝ and Dene ts'ı̨lı̨ in the Sahtú. Formal school-based programs include the Dene Kede curriculum (introduced 1993) which integrated Dene language and worldview into NWT classrooms (). Community-driven initiatives have also been vital – from language nests in early childhood (immersive daycare with Elders, launched in Sahtú communities in the 2000s) (), to on-the-land culture camps and new cross-cultural education models like the Dene Ts'ı̨lı̨ School for youth (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). Numerous government and Indigenous programs exist, but past reviews note they were often fragmented; a 2017 synthesis emphasized coordinating these efforts across “seven domains” (from education to policy) for holistic impact (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). Overall, Dene kedǝ remains a living language in the Sahtú region, yet its day-to-day use is now largely limited to older generations.
Data Trends – Variation, Continuity, Change: Quantitative indicators show a clear decline in intergenerational transmission of Dene kedǝ. In 2014, only about 15–18% of Sahtú youth (15–24) could carry on a conversation in North Slavey, versus 78% of elders (). Community surveys since the 1980s trace a steep drop in speakers beginning in the late 20th century (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx). Some communities (e.g. Délı̨nę, Colville Lake) have retained relatively more young speakers than others (), but English has become dominant among youth (). At the same time, many Dene ts'ı̨lı̨ practices persist. A majority of Sahtú households still engage in traditional activities like hunting, fishing, and consuming country foods at higher rates than the territorial average (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx). This underscores a dynamic of cultural continuity amid language shift – the land-based way of life endures stronger than the use of the ancestral language, marking both concern and resilience.
Lessons from Governance and Policy: Across the literature, a key lesson is that language and culture revitalization must be rooted in Indigenous governance and driven by the community – especially youth. Holistic programs linking language learning with being on the land have been shown to strengthen community wellness and local governance capacity (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region) (). In the Sahtú, co-management structures and self-government initiatives offer platforms for resurgence: for instance, the Nę K’ǝ Dene Ts'ı̨lı̨ Forum and community-driven wildlife plans incorporate Dene knowledge and terms, aligning cultural revival with resource governance (). Experiences also highlight that Dene ts’ǫ́ dane (Dene youth) need to be at the forefront of initiatives. Youth-driven projects – such as the Sahtú Youth Network’s efforts, on-the-land youth camps, and the Dene Heroes publishing project – have proven effective in building pride, leadership, and intergenerational knowledge transfer () (Dene Heroes of the Sahtu | Royal Roads University). Finally, the policy environment is gradually adapting: the NWT’s Indigenous Languages Action Plan and federal Indigenous language funding agreements are facilitating more community-led programming (Indigenous Languages and Education | Education, Culture and Employment) (). Still, authors stress the importance of long-term support, coordination, and recognizing that revitalization of Dene kedǝ and Dene ts'ı̨lı̨ is integral to self-determination.
Neat, thanks for this Michael. This is the first version I've seen that I would find acceptable: indeed, the first one that has accurately picked sources about this region/community/language and blended academic and "grey" literature with government stats, etc. Some of the picks and exclusions are still a bit odd, but overall if a human gave me this I would assume they'd done it in good faith. I think I should add - playing with different tools and prompts, interrogating differences between the results to the list of takeaways for student researchers.
Great question, which of course I will defer to Faun to answer (and she does note that she did not use the pro version of Elicit). But I would also note that for a lot of us who are in the classroom, Perplexity Deep Research is one of the tools our students are most likely to have access to due to deals they are making with campuses, and so from that teaching perspective, the takeaways here are quite useful.
Love that you drew the line from using AI in research directly to teaching about research in the age of AI. Wonderful essay.
Excellent, and very clear. I really enjoyed the specificity of the example you walked readers through.
This is super helpful. I routinely teach how to write literature reviews for PhD students - and how to "read" for them efficiently. My sense is that AI can do some of the heavy lifting, but not nearly enough, especially, as you say, when it's a more obscure topic. I will continue exploring the effectiveness of other tools to carry out this task (did you try jenny.ai or claude? just curious). Anyway I'm going to save this and share the next time I teach academic reading. thx!
Thanks for the recommendation Delia, I'll try jenny and let you know! Claude performs not unlike perplexity and scholargpt, but I bet it would do better if I had the pro version and trained it on longer conversations about what I'm looking for. But since I know exactly what I'm looking for in this experiment, I wonder if it's cheating to tell these bots "go find the community archive I know is out there" when in a normal lit review that's something we'd want to discover.
Cool. Let me know how you get on. Fyi I've not tried either yet but my students have
This is an incredibly important post. Thanks for sharing. As someone who teaches research to secondary school students, it's important to be aware of the current shortfalls of truncating the research process, especially with regard to more niche or esoteric topics (which is not all that surprising). The million dollar question is whether some of these companies will negotiate deals with archival digital resources and other hard to access information in order to expand their data set. Given that this seems to be the direction we are moving, my guess would be yes, but I think there will still be a place for manual research to ensure everything is captured. But humans miss things also.
Great question Steve. I hope they do! If I could use these tools and be confident that they'd send me metadata for sources I should go find in person, and/or access hard-to-find community archives, I'd be all in.
Nice write up! Thanks for the analysis. Did you test some of the more thorough tools such as OpenAI Deep Research, SciSpace or the Pro version of Elicit? One could argue that you focus on the tools that are known to be not that good in the first place.
Hi Michael, yes, this is a great and fair question! SciHub performs a bit like Elicit, it finds papers that address the question about integration of language and ecology in an interesting way, but both really struggle to discern which Dene language is which. I wonder if they are both using citation count as a heuristic, which I think leads them to pull examples from other languages over more relevant texts. I tried this once on the Pro version of Elicit when I had access to it a few months ago with similar results (but didn't pay for the pro version to try again now). If anyone has access I'd welcome them to try it and I'd be curious to hear if it has improved, same with OpenAI DR.
Whether or not prioritizing articles with a high citation count is a problem might depend on your field. In fields like anthropology, where scholarly and primary resources from within a community and in grey literature are exactly what you want to find, it might be more of an issue.
If you send me a prompt, I can run it through OpenAI DR.
Thanks Michael! I'd be curious to know if the prompt in the footnote in the piece generates any different findings.
Btw, I probably would not have asked for the synthesis in the same prompt. In order to create a good synthesis, the LLM needs to put the entire documents into the prompt and that runs up the token limit pretty quickly. As a result the LLM might limit the number of sources it will use. The way I would approach it is to ask for general literature research first, then run the resulting sources through something like ResearchRabbit or Semantic Scholar to identify anything that might be missing. After that I would review the sources manually and identify the ones most relevant, feed the full text back into the LLM and ask for the synthesis. That manual step is currently necessary to circumvent input token limits. You could also ask the LLM to create abstracts for each source first and then do the synthesis based on these abstract, but I find that this usually gives inferior results.
Ok, here you go. OpenAI DR always comes back with some follow up questions first. This is the entire dialogue after putting in your prompt. Substack would not let me copy the links, so this text ends up being a bit of a mess to read inside a comment reply. But I'm assuming this gives you enough info.
--- DR ---
To ensure a thorough and systematic literature review, could you clarify the following details:
Timeframe: Should the literature review focus on recent publications (last 10 years) or include older foundational works?
Sources: Do you prefer academic journal articles, government reports, indigenous organization reports, or all of the above?
Languages: Should I focus only on English sources, or include sources in Dene kedǝ and other relevant languages?
Scope: Should I include literature on general indigenous language and cultural revitalization, or strictly limit to Dene kedǝ, Dene ts'ı̨lı̨, and Dene ts’ǫ́ dane in the Sahtú Region?
Synthesis Format: Would you like the 300-word synthesis to be structured according to the three key objectives outlined in the grant proposal, or as a general summary of the literature?
Let me know so I can tailor the research accordingly!
--- me ---
Consider all foundational works and all relevant sources. Include sources in other languages. For the other two questions, use your best judgement.
--- DR ---
I will conduct a systematic literature review on the revitalization of Dene kedǝ (North Slavey language) and Dene ts'ı̨lı̨ (Dene ways of life), with a focus on Dene ts’ǫ́ dane (Dene youth). This will include:
1. A list of the most relevant academic and institutional sources, including foundational works and recent studies, across English and other relevant languages.
2. A 300-word synthesis summarizing key insights related to the three core objectives outlined in your grant proposal:
– The status of revitalization programs for Dene kedǝ and Dene ts'ı̨lı̨
– Data trends regarding language and cultural continuity
– Lessons learned in indigenous language and cultural revitalization within governance and policy frameworks.
I will let you know once the research is ready.
List of Relevant Sources
• “From Dene Kedǝ to Dene Ts'ı̨lı̨: Rethinking Resurgence in the Sahtú Region” – A comprehensive knowledge synthesis report (2017) reviewing 50 years of Sahtú Dene/Métis language and way-of-life initiatives (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region) (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). It identifies key domains of resurgence and lessons learned for Dene kedǝ (language) and Dene ts'ı̨lı̨ (culture).
• Dene Kede Curriculum (1993) – A foundational Dene language and culture curriculum for NWT schools, developed with Sahtú elders. Celebrated by the Royal Commission on Aboriginal Peoples for its cultural grounding (), it set expectations for Dene kedǝ as a first or second language in education.
• Language Nest Programs (2000s) – Early childhood immersion “language nest” programs established in 18 NWT communities (including Sahtú). These were one of the few formally evaluated Indigenous language initiatives, and showed successes in engaging infants with fluent Elder caregivers ().
• Tutcho (2016), “The language is like ets’ulah” – A Master’s thesis by a Sahtú Dene researcher exploring Dene love songs as a vehicle for language revitalization in Délı̨nę. It highlights youth perspectives, noting that English dominates among younger generations and urging pride-based strategies to strengthen Dene kedǝ ().
• NWT Bureau of Statistics – Language & Cultural Data – Longitudinal survey data (1984–2014) on North Slavey use and traditional practices. These data show steep generational declines in Sahtú Dene language fluency since the late 20th century (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx), alongside trends in on-the-land activities (hunting, fishing, etc.) (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx).
• Sahtú Governance and Cultural Initiatives – Examples include the Nę K’ǝ Dene Ts'ı̨lı̨ (Living on the Land) Forum, a regional co-management advisory body integrating Dene ts'ı̨lı̨ in resource governance (), and the Dene Ts'ı̨lı̨ School, an on-the-land cross-cultural education program for youth (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). Such initiatives tie language/culture revival to modern governance structures.
• NWT Indigenous Languages Framework and Action Plan – Territorial policy documents (2010–2018) that support community-driven revitalization. They outline funding transfers to regional Indigenous authorities and emphasize integrating Indigenous languages into public services and education (Indigenous Languages and Education | Education, Culture and Employment) ().
• “Dene Heroes of the Sahtu” (2017–2020) – A community literacy project engaging Sahtú youth in publishing books about local heroes. It improved youth literacy and cultural pride, honoring Dene traditions and language, and earned the 2018 Arctic Inspiration Prize (Dene Heroes of the Sahtu | Royal Roads University) (Dene Heroes of the Sahtu | Royal Roads University).
Synthesis of Key Findings (300 Words)
Status of Revitalization Programs: The literature reveals a wide range of revitalization efforts for Dene kedǝ and Dene ts'ı̨lı̨ in the Sahtú. Formal school-based programs include the Dene Kede curriculum (introduced 1993) which integrated Dene language and worldview into NWT classrooms (). Community-driven initiatives have also been vital – from language nests in early childhood (immersive daycare with Elders, launched in Sahtú communities in the 2000s) (), to on-the-land culture camps and new cross-cultural education models like the Dene Ts'ı̨lı̨ School for youth (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). Numerous government and Indigenous programs exist, but past reviews note they were often fragmented; a 2017 synthesis emphasized coordinating these efforts across “seven domains” (from education to policy) for holistic impact (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region). Overall, Dene kedǝ remains a living language in the Sahtú region, yet its day-to-day use is now largely limited to older generations.
Data Trends – Variation, Continuity, Change: Quantitative indicators show a clear decline in intergenerational transmission of Dene kedǝ. In 2014, only about 15–18% of Sahtú youth (15–24) could carry on a conversation in North Slavey, versus 78% of elders (). Community surveys since the 1980s trace a steep drop in speakers beginning in the late 20th century (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx). Some communities (e.g. Délı̨nę, Colville Lake) have retained relatively more young speakers than others (), but English has become dominant among youth (). At the same time, many Dene ts'ı̨lı̨ practices persist. A majority of Sahtú households still engage in traditional activities like hunting, fishing, and consuming country foods at higher rates than the territorial average (Microsoft Word - Appendix D. Statistics. FR KR. 1-01-17.docx). This underscores a dynamic of cultural continuity amid language shift – the land-based way of life endures stronger than the use of the ancestral language, marking both concern and resilience.
Lessons from Governance and Policy: Across the literature, a key lesson is that language and culture revitalization must be rooted in Indigenous governance and driven by the community – especially youth. Holistic programs linking language learning with being on the land have been shown to strengthen community wellness and local governance capacity (2017-ongoing From Dene Kedǝ to Dene Ts’ı̨lı̨: Rethinking Resurgence in the Sahtú Region) (). In the Sahtú, co-management structures and self-government initiatives offer platforms for resurgence: for instance, the Nę K’ǝ Dene Ts'ı̨lı̨ Forum and community-driven wildlife plans incorporate Dene knowledge and terms, aligning cultural revival with resource governance (). Experiences also highlight that Dene ts’ǫ́ dane (Dene youth) need to be at the forefront of initiatives. Youth-driven projects – such as the Sahtú Youth Network’s efforts, on-the-land youth camps, and the Dene Heroes publishing project – have proven effective in building pride, leadership, and intergenerational knowledge transfer () (Dene Heroes of the Sahtu | Royal Roads University). Finally, the policy environment is gradually adapting: the NWT’s Indigenous Languages Action Plan and federal Indigenous language funding agreements are facilitating more community-led programming (Indigenous Languages and Education | Education, Culture and Employment) (). Still, authors stress the importance of long-term support, coordination, and recognizing that revitalization of Dene kedǝ and Dene ts'ı̨lı̨ is integral to self-determination.
Neat, thanks for this Michael. This is the first version I've seen that I would find acceptable: indeed, the first one that has accurately picked sources about this region/community/language and blended academic and "grey" literature with government stats, etc. Some of the picks and exclusions are still a bit odd, but overall if a human gave me this I would assume they'd done it in good faith. I think I should add - playing with different tools and prompts, interrogating differences between the results to the list of takeaways for student researchers.
Great question, which of course I will defer to Faun to answer (and she does note that she did not use the pro version of Elicit). But I would also note that for a lot of us who are in the classroom, Perplexity Deep Research is one of the tools our students are most likely to have access to due to deals they are making with campuses, and so from that teaching perspective, the takeaways here are quite useful.
I found that perplexity tends to hallucinate research papers that don’t exist. I have not used it in a while, so it might have gotten better though.