Machine Learning: Exceeding Chance Level By Chance

Possible warnings for machine learning studies? "Chance" classification depends on the number of samples - if samples are not infinite, it may be higher than expected.

UPDATE: The general consensus seems to be that permutation testing, as commonly done, is valid and avoids the potential problems laid out by the authors.