JudgeGPT for Pakistani Courts: A $38.50 Return For Every Dollar Spent

This is the biggest ever randomized trial of an AI tool in a government setting. A team of researchers from ETH Zurich, Imperial College London and the New Economic School deployed JudgeGPT, a retrieval-augmented generation (RAG) system based on GPT-4, to 1,559 judges in 118 courts in Pakistan (about half of all first instance judges in the country).

JudgeGPT is trained on a corpus of 129,235 documents, including 128,292 court cases and 943 laws in Pakistan. The results show that in districts where judges were given special training, case closure rates went up by 6.3%, meaning that each district was able to process 1,848 additional cases every year. ROI: $38.50 per dollar spent in the baseline scenario, $10 per dollar spent in the conservative scenario.

Trained judges used the system much more frequently (~60 sessions instead of ~20) and asked more questions (200+ vs. <50), and they used it in a different way, for example for document editing and summarization instead of for finding laws (the place where hallucinations are likely to be most harmful).

There was no deterioration in decision making. Appeal rates didn’t change, and there was a slight improvement in decision quality. There was also no indication of an increase in gender or religious discrimination. Conclusion: AI can boost the productivity of public sector organizations — but only if we train people how to use it properly.