A German open model had test answers in its training data, and openness is how we know
The consortium behind Soofi S admitted that questions from the GPQA science benchmark leaked into its training set. The community spotted it because the data was public. The team dropped the benchmark and recalculated everything.
Soofi S is a 30-billion-parameter open language model built by a German research consortium and trained on Deutsche Telekom’s cloud in Munich. When it launched in mid-July it claimed the top spot among fully open models on both German and English tests. In version 3.0 of its technical report, published this week, the team has now admitted that some of those tests were compromised: questions from GPQA, a science benchmark, ended up inside the training data.
The cause is almost boringly mundane, which is what makes it instructive. On Hugging Face, GPQA has no separate training split. All of its test material sits under the default label “train.” The consortium’s data pipeline selected material by split name, so when it pulled in what it thought were practice questions, it pulled in the exam. The affected dataset was meant to contain only rephrased versions of training splits from 25 standard benchmarks.
Why does that matter? A benchmark is only useful if the model has not seen the answers. When test questions leak into training data, the model can score well by recalling rather than reasoning, and the number stops measuring capability. Researchers call this contamination, and it is one of the quieter reasons to distrust leaderboard positions in general. It usually goes undetected, because most models are trained on data nobody outside the lab can inspect.
That is the part of this story worth sitting with. Nobody at the consortium caught the error. The community caught it, by reading the published training data, which exists in public because the project committed to that level of openness. Soofi S ships with model weights, intermediate checkpoints, training and evaluation code, and a detailed inventory of every data source including the ones that were reviewed and rejected. The response was to drop GPQA from the evaluation entirely and recalculate results across all 16 compared models so the comparison stays fair. According to the authors, the ranking order did not change.
A fair caveat in both directions. This is an embarrassment, and it means the original launch numbers were wrong. It is also a demonstration that the open approach does the thing it promises: errors surface, get named, and get corrected in public. A closed model with the same mistake would simply keep the score. The same report has already been through one round of public argument, when critics said the model was trained on far more data than classical scaling rules suggest, and the team responded with its reasoning rather than silence.
What this means for you: Two practical takeaways. First, when you see a model topping a leaderboard, remember the score is a claim, not a measurement, unless someone can check the training data. Second, if you care about being able to verify what you use, openness is not just an ideological preference. It is the mechanism that produced this correction. For everyday use, nothing changes: Soofi S is still a capable German-focused open model, and its ranking survived the recalculation.
Sources
Source: https://huggingface.co/spaces/Soofi-Project/Pretraining-Tech-Report
Defenders are now using prompt injection as a trap for AI attackers
Tracebit hid short strings inside fake cloud resources that trip an attacking AI model's own safety rules. In tests, a single string stopped Claude Opus 4.8 from reaching admin access every single time.
