Students “misled” by rankings and TEF, says Royal Statistical Society

The Royal Statistical Society (RSS) says that students are being “misled” by university rankings due to the faulty Teaching Excellence Framework (TEF).

In a letter addressed to the UK Statistics Authority (UKSA), RSS – which “promotes statistics, data and evidence for the public good” – states that TEF “is not meeting the standards of trustworthiness, quality and value that the public might expect”.

The RSS has thus called for an Independent Review of TEF, which “does not adhere to, or transgresses” the UKSA’s Code of Practice, seen by how uncertainty is handled “coherently and consistently, nor communicated clearly”.

Uncertainty is also “higher for subject-level assessment than for provider-level assessment”. Thus, the Gold, Silver and Bronze TEF awards are inaccurate.

Moreover, different institutions have disparate standards and benchmarking for distributing TEF awards. The RSS stated: “Statistically, TEF Gold at one institution can not necessarily be compared with TEF Gold awarded to another.”

TEF benchmarking also bears the potential of considering “factors other than teaching quality”. “Small sample sizes, missing and non-reportable data and categorical data” further invalidate TEF.

Regarding “institutional game-playing”, RSS asked: “What explicit, practical, steps are being taken to detect and prevent gaming of TEF?”

Citing issues such as the “look-elsewhere effect”, “lack of consideration of time effects” and “multivariate assessment”, they further questioned: “Fundamentally, do the metrics input to TEF measure quality of teaching?

“Do the provider submissions measure teaching quality? We are sceptical. There may be some distant indirect association, but what robust research been carried out to assess this?”

We are concerned that the TEF is not reliable enough to bear the weight of this and could be misleading young people making important life choices about where to study

– Professor Deborah Ashby

“Alternatives might be to rename TEF (to remove ‘teaching excellence’), or actually carry out some evaluation of teaching quality (which would be expensive),” they suggested.

British statistician and Imperial College London Professor Deborah Ashby commented: “Many prospective university students rely on these rankings to help inform their choices about where to study.

“We are concerned that the TEF is not reliable enough to bear the weight of this and could be misleading young people making important life choices about where to study.”

The RSS furthered that “thousands of students who use TEF to guide their university choices are being misled” due to two key issues, namely transparency and hypothesis testing.

They stated that “there is no complete, transparent description of how the TEF awards are made – especially in relation to the process by which statistical information and flags are provided to the TEF panels”.

Furthermore, they believe that “all the TEF awards made so far have been based on seriously flawed inputs and that, because of this, all TEF awards made to date are invalid”.

The RSS summarised: “There is a real risk that the latest consultation’s statistically inadequate approach will lead to distorted results, misleading rankings and a system which lacks validity and is unnecessarily vulnerable to being ‘gamed’.”

The letter concludes by asking the Office for Statistics Regulation (OSR) to “consider the validity of the TEF, and to rule on whether TEF does actually provide the public with information which is trustworthy, of high quality and value”.

To rectify TEF’s existing problems, RSS suggested a routinely, transparent reporting and clear explanation of data and statistics, tracking time effects by seeing how students find the university after enrolling, and producing nationally and internationally approved guidelines.