# data[].attributes.headline
Technology is enabling more immersive remote course delivery

# data[].attributes.subject
Technology is enabling more immersive remote course delivery

# data[].attributes.body
Evidence: https://www.theguardian.com/technology/2024/jan/21/hologram-lecturers-thrill-students-at-trailblazing-uk-university

Impact: Student expectations for immersive learning experiences is growing. Universities will have to increasingly deliver.

# data[].attributes.body
Generative AI changing the nature of research publications

# data[].attributes.attachment_link.readable_text
Over the past few weekends, I've been playing with large language models (LLMs) and combining them with tools and interfaces to build a simple assistant. Along the way, I noticed some issues with retrieval and thought of a few ideas on how to solve them.

It was a lot of fun, and I came away from this experience optimistic that LLMs have great potential to augment how we work, especially how we research, reflect, and plan.

Here's the code to speed run this. It's messy and relies largely on LangChain abstractions. Could be useful as a reference, but not as learning material.

Tools to summarize, query, and advise

My first project was inspired by Simon's post on how ChatGPT is unable to read content from URLs. Thus, I tried to help it do just that with /summarize and /eli5. The former can /summarize content from URLs into bullet points while the latter reads the content and explains like I'm five (eli5). They help me skim content before deciding if I want to read the details in full ( tweet thread).

Next, I explored building agents with access to tools like SQL and search. /sql takes natural language questions, writes and runs SQL queries, and returns the result. /sql-agent does the same but as a zero-shot agent. Though /sql-agent didn't work as reliably as I hoped (see Appendix), watching it struggle and eventually get it right was endearing and motivating ( tweet thread).

I also built /search, an agent that can use tools to query search provider APIs (e.g., Google Search). This way, the LLM can find recent data that it hasn't been trained on and return an accurate and up-to-date response. (This was before ChatGPT plugins that now have this functionality out of the box. Even so, it was fun building it from scratch.)

Most recently, I built a /board of advisors. It's based on content from informal mentors-and prolific writers-like Paul Graham, Marc Andreessen, Will Larson, Charity Majors, and Naval Ravikant. /board provides advice on topics including technology, leadership, and startups. Its response includes source URLs for further reading, which can be chained with /summarize and /eli5 ( tweet thread).

I also built /ask-ey which is similar to /board but based on my own writing. Because I'm more familiar with my work, it's easier to spot issues such as not using an expected source (i.e., recall issue) or using irrelevant sources (i.e., ranking issue).

Combining LLMs, databases, search APIs, and Discord

To extract content from URLs, I used good ol' and BeautifulSoup. For LLMs, I worked with OpenAI's gpt-3.5-turbo and , primarily the former due to its cost-effectiveness. LangChain made it easy to apply the LLM chains, agents, and tools. For search, I used Google's custom search through the google-api-python-client wrapper. To embed documents and queries, I used OpenAI's text-embedding-ada-002.

The application server was hosted on Railway. To host, serve, and find nearest neighbours on embeddings, I used Pinecone. Lastly, I integrated everything with Discord via the interactions wrapper.

Shortcomings in retrieval and how to solve them

While experimenting with /board and /ask-ey, I noticed that it wasn't retrieving and using the expected sources some of the time.

For example, when I asked the /board "How do I decide between being a manager or an IC", it fails to use (as a source) any of Charity's writing on the manager- engineer pendulum or management. However, tweaking the question to "How do I decide between being a manager or an engineer " resolved this.

Similarly, when I /ask-ey "What bandits are used in recommendation systems", it didn't retrieve my main writing on bandits. But updating the question to " How are bandits used in recommendation systems" fixed this issue.

But when I checked the retrieved sources, it was disappointing to see that only the top hit came from the expected URL, and even that was an irrelevant chunk of the content. (Text from each URL is split into chunks of 1,500 tokens.) I had expected embedding-based retrival to fetch more chunks from the bandit URL. This suggests there's room to improve on how I processed the data before embedding and highlights the importance of data prep.

This issue is partially due to poor recall. Here are a few hypotheses on why this happens:

ANN indices might be tuned sub-optimally

Most (if not all) embedding-based retrieval use approximate nearest neighbours (ANN). If we use exact nearest neighbours, we would get perfect recall of 1.0 but with higher latency (think seconds). In contrast, ANN offers good-enough recall (~0.95) with millisecond latency. I've previously compared several open-source ANNs and most achieved ~0.95 recall at throughput of hundreds to thousands of queries per second.

If the issue lies in a sub-optimally tuned ANN index, we can tune the index parameters to achieve the recall/latency trade-off we need. However, this requires more effort compared to a plug-and-play index as a service. I'm also not sure if cloud vector databases offer the option to tune the ANN. As a result, we could end up with as low as 50% recall.

Off-the-shelf embeddings may transfer poorly across domains

Off-the-shelf embeddings may be too generic and don't transfer well to other domains. From the examples in this OpenAI forum, we see unexpectedly high cosine similarity between seemingly different text. (While the failure examples above seem generic, the point is that we should pay attention when applying embeddings to our domain.)

A possible solution: If we have both positive and negative examples, we can fine-tune an embedding model via triplet loss. This way, we can ensure that the distance between the anchor and positive example is closer than the distance between the anchor and negative example (by a margin). This is especially helpful when embedding private data that contains language that foundational models may not have seen.

Preparing these (anchor, positive, negative) triplets is the bulk of the work. One way is to collect explicit feedback by returning sources in responses and asking people to thumbs up/down on them. Alternatively, implicit feedback is available in settings such as e-commerce, where we can consider results that users ignore as negatives, or search, where we provide sources in results (à la Bing Chat) and observe if users click on them.

Documents may be inadequately chunked

Third, if we're using LangChain, we're probably taking the default approach of using its text splitter and chunking content into documents of 1,000 - 2,000 tokens each. While we can have such large documents because recent embedding models can scale to long input text, problems may arise when the input is overloaded with multiple concepts.

Imagine embedding a 3,000-word document that has five high-level concepts and a dozen lower-level concepts. Embedding the entire document may force the model to place it in the latent space of all concepts, making retrieval based on any single concept difficult. Even if we split it into multiple chunks of 1,500 tokens each, each chunk's embedding could be a muddy blend of multiple concepts.

A more effective approach could be to chunk documents by sections or paragraphs. After all, this is how most content is organized, where each section/chapter discusses a high-level concept while paragraphs contain lower-level concepts. This should enhance the quality of embeddings and improve embedding-based retrieval. Thankfully, most writing is organized by sections or chapters, with paragraphs separated by /n/n.

I suspect there are large gains to be made here though it's more art than science. It also requires more manual work-scraping and preparing data for my document store took as much, if not more, effort as building the tools. Also see this OpenAI example of how they cleaned, chunked, and embedded wikipedia articles.

Embedding-based retrieval alone might be insufficient

Lastly, relying solely on document and query embeddings for retrieval may be insufficient. While embedding-based retrieval is great for semantic retrieval, it can struggle when term matching is crucial. Because embeddings represent documents as dense vectors, they may fail to capture the importance of individual words in the documents, leading to poor recall. And if the search query is precise and short, embedding-based retrieval may not add that much value, or perform worse. Also, simply embedding the entire query might be too crude and could make the results sensitive to how the question is phrased.

One solution is to ensemble semantic search with keyword search. BM25 is a solid baseline when we expect at least one keyword to match. Nonetheless, it doesn't do as well on shorter queries where there's no keyword overlap with the relevant documents-in this case, averaged keyword embeddings may perform better. By combining the best of keyword search and semantic search, we can improve recall for various types of queries.

Query parsing can also help by identifying and expanding (e.g., synonyms) keywords in the query, ensuring that questions are interpreted consistently regardless of minor differences in phrasing. Spelling correction and autocomplete can also guide users toward better results. (A simple hack is to have the LLM parse the query before proceeding with retrieval.)

We can also rank retrieved documents before including them in the LLM's context. In the bandit query example above, the top hit doesn't offer any useful information. One solution is to rank documents via query-dependent and query-independent signals. The former is done via BM25 and semantic search while the latter includes user feedback, popularity, recency, PageRank, and so on. Heuristics such as document length may also help.

Update: The next iteration uses a hybrid of OpenSearch and E5 embeddings.

LLM-augmented research, reflection, and planning

While the tools above were hacked together over a few weekends, they hint at the potential in LLM-augmented workflows. Here are some ideas in the adjacent possible.

Enterprise/Personal Search and Q&A

Picture yourself as part of an organization where internal documents, meeting transcripts, code, and other resources were stored as retrievable documents. For confidentiality and security reasons, you would only be able to access documents that you have permissions for. To navigate this vast knowledge base, you could ask simple queries such as:

What were the common causes of high-severity tickets last month?

What were our biggest wins and most valuable lessons from last quarter?

What are some recent "Think Big" ideas or PRFAQs the team has written?

Then, instead of returning links to documents that we would have to read, why not have an LLM /summarize or /eli5 the information? It could also synthesize via /board and find common patterns, uncovering root causes for seemingly unrelated incidents or finding synergies (or duplication) in upcoming projects. To augment the results, it could /sql or /sql-agent databases or /search for recent data on the internet.

Let's consider another scenario which uses a personal knowledge base. Over the years, I've built up a library of books, papers, and disorganized notes. Unfortunately, my memory degrades over time and I forget most of the details within a week. To address this, I can apply similar techniques to my personal knowledge base and /ask-ey:

For each of those questions, I've invested effort into research, distilling, and publishing the answers as permanent notes. And the process was invaluable for clarifying my thoughts and learning while writing. That said, I think the tools above can get us ~50% there with far less effort. Products like Glean (enterprise) and Rewind (personal) seem to do this.

Research, Planning, and Writing

Back to the scenario of internal docs and meeting transcripts. Let's say you're a leader in your org and need to write an important doc. It could be a six-week plan to tackle tech debt or a more ambitious three-year roadmap. How can we make writing this doc easier?

To write the tech debt document, we'll first need to understand what are the most pressing issues. We can start by asking /board to gather details about the problems we're already aware of. /board can help us retrieve and synthesize the relevant trouble tickets, war room meeting transcripts, and more via /sql and internal /search. Then, we can expand to broader queries to find problems we're unaware of, before diving deeper as needed.

With the top three issues, we can start writing an introduction that outlines the purpose of the document (aka the Why aka the prompt). Then, as we created section headers for each issue, a document retrieval + LLM copilot helps with filling them out, providing data points (/sql), links to relevant sources (/search), and even suggesting solutions (/board). Bing Chat has this in some form. Also, I believe this is the vision for Office 365 Copilot.

As the main author, we'll still need to apply our judgment to check the relevance of the sources and prioritize the issues. We'll also need to assess suggested solutions and decide if tweaking one solution could address multiple issues, thereby reducing effort and duplication. Nonetheless, while we're still responsible for writing the document, our copilot can help gather and prepare the data, significantly reducing our workload.

LLMs: Not a knowledge base but a reasoning engine

"The right way to think of the models that we create is a reasoning engine, not a fact database. They can also act as a fact database, but that's not really what's special about them. What we want them to do is something closer to the ability to reason, not to memorize." - Sam Altman

We've seen that LLMs are adept at using tools, summarizing information, and synthesizing patterns. Being trained on the entire internet somehow gave them reasoning abilities. (Perhaps this is due to learning on Github and StackOverflow data, since code is logic?) Nonetheless, while they can reason, they're often constrained by the lack of in-depth or private knowledge, like the kind found in enterprise or personal knowledge bases.

I think the key challenge, and solution, is getting them the right information at the right time. Having a well-organized document store can help. And by using a hybrid of keyword and semantic search, we can accurately retrieve the context that LLMs need-this explains why traditional search indices are integrating vector search, why vector databases are adding keyword search, and why some apps adopt a hybrid approach ( Vespa.ai, FB search).

It's hard to foresee how effective or widespread LLMs will become. I've previously asked whether LLMs might have the same impact as computers, mobile phones, or the internet. But as I continue experimenting with them, I'm starting to think that their potential could be even greater than all those technologies combined.

And even if I end up being wrong, at least I can still have fun getting LLMs to explain headlines in the style of Dr. Seuss or make up quirky quotes on my Raspberry Pi Pico.

While some use LLMs to disrupt industries and more,
Others build ChatGPT plugins, pushing boundaries galore.

Yet here I am with my Raspberry Pi loose,
Using LLMs to explain headlines via Dr. Seuss. pic.twitter.com/0MJn6HC2cw
- Eugene Yan (@eugeneyan) April 10, 2023

Appendix

Here's an example of how /sql-agent struggled and eventually figured out that it should check the database schema. While it finally executed the right query and got the results, it also ran out of iterations before it could respond 🥲 (Back to top)

If you found this useful, please cite this write-up as:

Yan, Ziyou. (Apr 2023). Experimenting with LLMs to Research, Reflect, and Plan. eugeneyan.com. https://eugeneyan.com/writing/llm-experiments/.

or

@article{yan2023llmapps, title = {Experimenting with LLMs to Research, Reflect, and Plan}, author = {Yan, Ziyou}, journal = {eugeneyan.com}, year = {2023}, month = {Apr}, url = {https://eugeneyan.com/writing/llm-experiments/} }

[
Browse related tags: llm deeplearning learning🛠🔥 ]

Search

Join 6,300+ readers getting updates on machine learning, RecSys, LLMs, and engineering.

# data[].attributes.attachment_link.title
Experimenting with LLMs to Research, Reflect, and Plan

# data[].attributes.attachment_link.description
Also, shortcomings in document retrieval and how to overcome them with search & recsys techniques.

# data[].attributes.headline
Renewable energy is the fastest-growing energy source in the United States

# data[].attributes.subject
Renewable energy is the fastest-growing energy source in the United States

# data[].attributes.body
Evidence: https://www.c2es.org/content/renewable-energy/

Impact: More students will likely need skills related to renewable energy as they enter the workforce.

# data[].attributes.attachment_link.description
Our mission is to secure a safe and stable climate by accelerating the global transition to net-zero greenhouse gas emissions and a thriving, just, and resilient economy.

# data[].attributes.headline
University enrollments are decreasing in the United States.

# data[].attributes.subject
University enrollments are decreasing in the United States.

# data[].attributes.body
Evidence: https://www.bestcolleges.com/research/college-enrollment-decline/

Impact: Universities might increase efforts to recruit "nontraditional students."

# data[].attributes.attachment_link.readable_text
College enrollment has been declining since 2010. Learn more about college enrollment trends and COVID-19's impact on enrollment.

Data Summary

College enrollment has been declining since 2010.[1]

In the past decade, total college enrollment has dropped by about 1.95 million students, or by about 10%.[2], [3]

The undergraduate college enrollment decline has accelerated since the pandemic began, resulting in a loss of nearly 1.4 million students, or almost 9% of total enrollment, between spring 2019 and spring 2023.Note Reference [3]

Graduate student enrollment, however, has increased by about 76,000 students, or by about 3%.Note Reference [3]

Financial concerns stemming from the COVID-19 pandemic caused many would-be students to change their plans.[4]

College enrollment has fallen since the start of the pandemic, but it's actually been trending downward for the past decade.

Different factors impact college enrollment, like falling birth rates, rising college tuition, and the economic impact of the COVID-19 pandemic. This report covers long-term enrollment patterns, the recent decline, and some of the causes behind this dip.

College Enrollment Decline: 1980s-2020s

After increasing for decades, undergraduate college enrollment peaked in 2010 at around 18.1 million students, according to data from the National Center for Education Statistics (NCES). Note Reference [1] From there, it declined steadily. In fall 2021, about 15.4 million students were enrolled in college.

Between 1985-2010, college enrollment increased at an average rate of 2.2% a year.

From 2011-2021, it decreased at an average rate of 1.6% a year.

In 2021, it was 14.6% down from peak enrollment in 2010.

College Enrollment Decline by School Type

When you look at enrollment by school type - two-year and four-year, public and private - you'll find that only certain types of schools were driving the overall college enrollment decline before the pandemic. Note Reference [1]

Two-year public colleges lost 38% of their enrollment between 2010 and 2021.

Two-year private for-profit colleges lost about 59% of their enrollment in the same period.

Four-year private for-profit colleges lost about 54% of their enrollment.

Enrollment grew at four-year public colleges (by 15.1%) and four-year nonprofit colleges (by 2.7%).

Two-year private schools appear in a separate graph due to their lower enrollment numbers.

Pandemic College Enrollment Decline: 2019-2023

The undergraduate college enrollment decline has accelerated since the start of the COVID-19 pandemic. Public institutions - especially two-year colleges - experienced the steepest declines.

International enrollment and transfer enrollment also saw sharp declines during the pandemic.

The National Student Clearinghouse Research Center (NSCRC) found that between spring 2019 and spring 2023: Note Reference [3]

Postsecondary institutions lost about 1.3 million unduplicated students, or about 7% of total enrollment.

Undergraduate student enrollment fell by nearly 1.4 million students, or almost 9% of total enrollment.

Graduate student enrollment grew by about 76,000 students, or by about 3%.

The college enrollment decline slowed substantially between spring 2022 and spring 2023.

Source: NSCRCNote Reference [3]

Enrollment declined the most at two-year public schools; it increased at four-year private for-profit schools. Note Reference [3]

COVID-19's Impact on College Plans

In an experimental study conducted by the U.S. Census Bureau, the majority of adults who had household members enrolled in college for the fall 2021 term said that their school plans changed. Note Reference [5]

32% said their classes would occur in different formats.

16% canceled all plans to attend.

12% took fewer classes.

Those whose plans were canceled cited multiple reasons, including getting COVID-19 and changes in financial aid. But the most popular reason was finances. Almost half (48%) of respondents whose plans were canceled reported an inability to pay for educational expenses because of the pandemic. Note Reference [4]

Income level was correlated to a need to change college plans during the pandemic. Note Reference [5]

Among households making over $150,000 a year, 57% had no change in plans to attend college classes - 13 percentage points above the average.

In households making under $50,000, just 37% planned to carry on their education without making a change - 7 percentage points below average.

Almost a quarter of households making less than $25,000 a year canceled all their college plans -compared to less than 10% of households making $100,000 or more a year.

Additionally, a 2023 ACT report found that students from low-income families were more than twice as likely as other students to question their decision to attend college.

College Enrollment Rate Over Time

In 2021, the percentage of all U.S. residents between 18-24 enrolled in college was 38.1%, the lowest it's been since 2006. Note Reference [6]

Additionally, the percentage of recent high school graduates enrolled in college was the lowest it's been in years. In 2021, 61.8% of high school grads enrolled in college versus 70% in 2009. Note Reference [7]

What's Causing the College Enrollment Decline?

Before the pandemic, people in the largest college-going age group were going to college at about the same rate as they were 10 years ago. So why is college enrollment declining? And what does the future of higher education hold?

To answer that, you'll have to look further back. Following an economic recession in the early 1990s, the U.S. birth rate fell. Note Reference [8] This could explain a college enrollment decline about 18 years later.

The birth rate dropped again during the 2007-2009 recession. For this reason, experts predict another enrollment drop - or "cliff" - after 2025.

Dive Deeper Into College Enrollment Statistics

Frequently Asked Questions About the College Enrollment Decline

Are more students going to college?

Today, fewer people are going to college than in recent years. In 2021, 15.4 million students were enrolled in an undergraduate degree program. That's the lowest fall enrollment since 2006. Note Reference [1]

The NSCRC estimates that even fewer undergraduates were enrolled in spring 2023, just 14.1 million students. Note Reference [3]

However, in general, more people go to college today than in the early 2000s and previous decades. In 2021, more than twice as many people went to college as in 1970. Note Reference [1]

Why is college enrollment declining?

One factor behind falling college enrollment is a shrinking U.S. birth rate. Tuition inflation might be another factor behind the college enrollment decline. In a 2022 BestColleges survey, more than 6 in 10 Americans said that the financial burden of earning a degree made college inaccessible.

Since the pandemic, the college enrollment decline has accelerated. The pandemic's economic impact forced some to reconsider their plans to attend college, especially lower-income students.

Is college becoming less popular?

Outside of the pandemic, it's hard to say that college is becoming less popular. When you look at the college enrollment rate among 18-24-year olds, roughly the same percentage enrolled in college in 2020 as they did 10 years prior. Note Reference [6] Instead, today's enrollment decline could be a result of slower birth rates following economic recessions.

College isn't becoming more popular, however. After a gradual increase over the decades, the college enrollment rate has basically stopped growing. Time will tell how the college enrollment rate will change in the aftermath of the COVID-19 pandemic.

1. Table 303.70. Total undergraduate fall enrollment in degree-granting postsecondary institutions, by attendance status, sex of student, and control and level of institution: Selected years, 1970 through 2031. National Center for Education Statistics (NCES). December 2022. (back to content ⤶)

2. Current Term Enrollment Estimates Spring 2013 (PDF). National Student Clearinghouse Research Center (NSCRC). May 2013. (back to content ⤶)

3. Berg, B., Lee, S., Randolph, B., Ryu, M., & Shapiro, D. Current Term Enrollment Estimates: Spring 2023, Herndon, VA: National Student Clearinghouse Research Center. May 2023. (back to content ⤶)

4. Table 302.85. Among adults 18 years old and over who reported all fall attendance plans for at least one adult in their household had been canceled, percentage reporting on reasons for changes in plans, by selected respondent characteristics: August 18 to 30, 2021. NCES. October 2021. (back to content ⤶)

5. Table 302.80. Percentage of adults 18 years old and over who reported changes to household members' fall postsecondary plans, by selected respondent characteristics: August 18 to 30, 2021. NCES. October 2021. (Note: Interpret data with caution. Response rates do not meet NCES standards.) (back to content ⤶)

6. Table 302.60. Percentage of 18- to 24-year-olds enrolled in college, by level of institution and sex and race/ethnicity of student: 1970 through 2021. NCES. August 2022. (back to content ⤶)

7. Table 302.10 Recent high school completers and their enrollment in college, by sex and level of institution: 1960 through 2021. NCES. October 2022. (back to content ⤶)

8. Chart: Birth Rate, crude per 1,000 people - United States. The World Bank. 2020. (back to content ⤶)

# data[].attributes.attachment_link.title
U.S. College Enrollment Decline: Facts and Figures | BestColleges

# data[].attributes.attachment_link.description
College enrollment has been declining for years. Learn more about college enrollment trends and COVID-19's impact on enrollment.