The most common mistake I see academics making in their interaction with conversational agents is restricting themselves to a single prompt or a small series of prompts. This approach fails not just because it often doesn’t provide sufficient context for an adequate response, or because it precludes the reflexive work involved in refining a prompt over a series of stages. More fundamentally, it fails to recognize how the real intellectual value of conversational agents comes from inducting them into your domain of expertise.
At the time of writing, GPT-4o has a context window of 128k tokens and the latest Claude models have a context window of 200k tokens. Tokens aren’t exactly equivalent to words because single multisyllabic words can be broken up into multiple tokens, multiple words can be incorporated into a single token, and non-word elements count as tokens. However, it can be a useful rule of thumb to treat them as roughly equivalent, such that a 128k or 200k context window means that a conversation can include approximately 128k or 200k words. This could encompass as many as 500 pages depending on text formatting.
This might sound like a lot until you include the model’s responses in the count. If the model’s answers are typically longer than the user prompts (though this varies immensely depending on the kind of conversation), the proportion of this 500 pages available to the user rapidly diminishes. If you upload text to provide background knowledge, this also contributes toward the hard limit of the context window. In practice, the context window is much smaller than might initially be imagined, particularly if you embrace the practice of having long and sustained conversations with models.
It’s crucial to understand these dynamics if you want to realize the intellectual potential of models. While strictly speaking the model never ‘understands’ your work, it can build up a capacity to contribute coherently to it within the context window in ways that the baseline instance of the model lacks. This manifests in often mundane ways, such as recalling a past argument you have made or referencing an earlier question at a later stage of a conversation. The real value comes from linking these instances together into a sustained sense of what you are working on, which the model draws upon in its responses.
At points this can be eerie or even irritating, as the model infers something about what you are doing which is relevant but contrary to your intentions. I have repeatedly had the experience of a model which infers that I’m working on a writing project try to complete my writing, despite my not making any such request. What makes these experiences unnerving is the extent to which they immediately engender temptation, particularly when the writing does reflect your own style. It’s not so much that the model is doing something you don’t want it to do; rather, it’s doing something you want but are disavowing. In these moments, it can feel that it understands where you are intellectually to a degree that can feel ambivalent.
The introduction of Claude’s Project functionality provides a further extension of this capability. It enables documents to be uploaded which provide context across a range of conversations, meaning this domain expertise isn’t limited to a particular thread. It is difficult to convey quite how useful this is unless you’re familiar with the limitations of traditional conversational interfaces.
There is a pleasure to be found in this pseudo-understanding, a sense of the model’s capacity to practically engage with what you’re doing. If you provide this context, you can assume it as the conversation progresses, lessening the explanatory work involved in making a request. It’s a practical matter of the contribution which the model can make, but the pleasure involved goes more deeply than that. There’s a satisfaction in the capacity granted by this contribution, how it makes it easier to unfurl your still nascent thoughts and give them form.
Mark Carrigan 