Google trial finds AI improves patent drafts, but junior judgment does not improve on average

Google Research said access to an AI writing tool improved patent lawyers’ drafting quality in a three-month randomized trial. Junior lawyers, however, showed no average improvement on a separate test completed without AI.

The experiment involved 133 practicing patent lawyers at 11 U.S. intellectual-property firms. AI access raised drafting quality by 0.34 standard deviations after 10 days and 0.38 standard deviations after 90 days. On the unassisted test at the end of the trial, the gain was concentrated among lawyers with at least seven years of experience.

Researchers randomly assigned two-thirds of participants within each firm to early access to a then-unreleased Google Labs patent-writing assistant. The other lawyers received some training on AI use but could not use the study tool until the experiment ended. Those with access could use either the Google tool or another AI product for simulated drafting exercises after 10 and 90 days.

Expert patent attorneys who did not know which lawyers had AI access graded the work. Third-party patent professionals assessed enforceability, accuracy, strategic ambiguity, completeness and clarity. Google Research said the treatment group produced fewer poor drafts and more good ones, but AI access did not increase the frequency of excellent scores. Junior lawyers with access also completed the 10-day exercise 18 minutes faster than the control-group average of 124 minutes.

At 90 days, every participant completed a patent-redlining exercise without AI. The study used that task to measure professional judgment after the period of tool exposure, rather than output produced with assistance. Across experience levels, the treatment group scored 0.32 standard deviations above controls.

Results diverged by seniority. The researchers classified lawyers with fewer than seven years of experience as junior and those with at least seven years as senior. Senior lawyers in the treatment group scored 0.45 standard deviations above senior controls on the unassisted exercise. Junior lawyers recorded no average improvement; their results shifted toward more poor and more good scores, with fewer in the middle.

The reported drafting effects had p-values of 0.03 after 10 days and 0.01 after 90 days. The overall unassisted result had a p-value of 0.04, while the senior subgroup result had a p-value of 0.02.

DataPhoenix has separately covered uneven gains from Earth AI embeddings in public-health studies. In the patent study, the authors cautioned against treating their findings as a general verdict on AI and expertise. The sample covered 11 firms with regular, non-exclusive business relationships with Google, and three months is short compared with the years required to develop professional expertise. The opened sources did not report subgroup sample sizes, confidence intervals, statistical power or attrition by trial arm, and they did not establish that the working paper had undergone independent peer review.