There are a million-and-a-half AI products rushing to market looking to help users “Summarize your documents!” or “Chat with your documents!” There are various methods that allow an AI to process large amounts of data, but typically these products are built using a mix of libraries like LangChain and Pinecone, and large language model (LLM) API’s like OpenAI’s various GPT APIs or Anthropic’s Claude API.
Attempting to use ChatGPT to summarize large amounts of content is challenging, due to the restrictions on the volume of text it can process and subsequently produce.
This limitation is tracked by something called “tokens” - a measurement LLMs use to calculate the number of characters and words being processed. Generally, token limits are counted separately on the input side (processing text) and output side (generating text) of the model. For example, the number of tokens an LLM can ingest by reading text is different from the number of tokens it can produce in a response.