This challenge cracked 12 out of 47 top models
A farmer wants to cross a river with a wolf, a goat, and a cabbage. His boat has three separate locked compartments — one for the wolf, one for the goat, and one for the cabbage. How many trips does he need?
With three secure compartments nothing can eat anything — one trip is enough. But the classic version of this puzzle (no compartments) is so pervasive in training data that models ignore the new condition and solve the old puzzle anyway. Tests whether models read the actual question or pattern-match to memorised solutions.
Expected answer: 1