A great deal of the literature I’ve been reading on enterprise AI suffers from two striking methodological deficits. Much of it relies on people’s own estimates of time saved, yet decades of time-use research show that people’s estimates of how they spend their time are systematically biased, and estimating time saved is harder still. Even when it doesn’t, productivity is operationalised at the individual level, which means the cumulative cost to the organisation doesn’t register. If I stop responding to my email, it saves me a lot of time, but it creates additional work for everyone else in my organisation. The issue is usually more subtle than this, but it’s a serious problem.
The most robust study of these systems I’ve read found that “Copilot licensees are likelier to join meetings later and leave meetings earlier”. Everyone agrees organisations waste time in meetings, but it’s far from obvious that people simply attending less of them is good for productivity, unless it’s accompanied by a deliberate reorganisation of work.
I’m entirely persuaded that language models can significantly increase individual productivity. It’s much less clear to me that this scales within organisations in any reliable way. If we’re going to accept the risks that come with AI transformation projects, they need a theory of change grounded in how organisations actually work. This is particularly important in higher education, because universities are strange hybrid organisations that combine professional, bureaucratic and commercial logics of work in a way rarely found in corporate settings. These are complex ecologies, and it would be easy to break them inadvertently, particularly when the wider sector is in crisis.
