A further point I want to come back to later is how metricisation figures into automation. This remark by Andrew Abbott powerfully captures how metrics are contrary to professional self-regulation; the extent to which outcomes can be compared reduces the force with which professionals are able to define exclusive jurisdiction over their problem area:
On the other hand, results that are too easily measurable lead to easy evaluation from outside the profession and consequent loss of control. They may also make it easier for competitors to demonstrate treatment superiority if they have it.
The System of Professions, pg 54
I’ve argued in the past that two forms of stratification in academic life (collegial ranking and citational measurement) were joined by a third (platform popularity) to produce an overdetermined grid of where stand in relation to each other.
The function of metrics is commensurability: making otherwise dissimilar items similar enough to be compared to each other. What intrigues me about generative AI is that non-quantitative. For example I just asked Claude AI to compare the two automation blog posts I wrote this morning and yesterday in terms of the following criteria: originality, clarity, creativity and impact. I didn’t define these terms but in a more rigorous exercise I would suggest at least a few hundred words of carefully articulated criteria for each. This was the comparison it undertook:
Clarity:
The first post is clearer in laying out key concepts like jurisdiction, direction/delivery split, and linking them explicitly to academic work. The second post brings in additional concepts like routinization without fully explaining their connection. The first post has a more linear structure.
Originality:
The second post is more original in extending Abbott’s framework to theorize how AI automates both direction and delivery. The hypothetical examples of fully automated academic workflows demonstrate original application of Abbott’s ideas. The first post covers more familiar ground.
Creativity:
The first post is more creative in spanning social media’s impact to imaginative professional reinvention enabled by AI. The hypotheticals of new forms of digital literacy and jurisdictional expansion show creative theorizing. The second post has creative elements like the triaging example but stays closer to established concepts.
Impact:
The second post likely has more impact due to provocative examples of AI takeover of academic work, which vividly highlight stakes. The triaging example crystallizes impact on academic roles. The first post has theoretical impact but lacks these visceral illustrations.
In summary, the first post edges out on clarity and creativity while the second has greater originality and impact regarding AI’s threat to academic jurisdiction over knowledge work. Both have strengths but the second post’s risk examples sharpen the issues Abbott’s framework raises for AI’s potential displacement of academic professions.
Now please don’t misunderstand me here. I’m not suggesting this is a good way of evaluating academic work, but it should be seen alongside the contingent flaws of present day research evaluation rather than an imagined ideal evaluator who immerses themselves in a text in the most charitable spirit possible. Increasingly I think we need a sociological cynicism about how we compare emerging modes of knowledge production to the modes they have the potential to undermine or replace; not because of a commitment to ‘disruption’ but simply because to work with an idealised image of present day knowledge production risks propping up a system which isn’t working very well to begin with.
Mark Carrigan 