Benchmarking locally hosted language models for journal editorial work on a compact desktop workstation

Abstract

Journals are beginning to consider language models for manuscript handling, but submitted manuscripts are unpublished, and where policy forbids sending them to an external service the model must run on hardware the journal controls. The capability of locally hosted models on editorial work has not been measured. Here we constructed a benchmark of eight editorial tasks from a journal&#39;s Instructions for Authors, from manuscripts carrying defects we seeded and verified independently, and from published reviews of a preprint, and evaluated twenty open-weight models spanning a twenty-five-fold range of weight size on a compact desktop workstation of the kind a laboratory or small editorial office can adopt. The strongest model detected 36 of 40 seeded guideline violations and occupied 81 GB; a 17 GB model detected 33. Across the best configurations tested, we observed no consistent monotonic association between weight size and score: rank correlations were negligible on every task (Spearman |rho| <= 0.19), and within one model family the larger member scored below its smaller sibling. A deterministic checker of regular expressions and arithmetic, using no model, detected 31 of the same violations in a fraction of a second, and the union of its detections with those of the strongest model covered all 40. On the single peer-review case, the best model recovered 6 of 12 points from three published reviews. Prompt structure substantially altered scores within individual models. This level of performance is therefore within reach of a workstation of this class, once the deterministic checks are written.