Part 3 - The Pension Regulator’s AI plan – imperfective assurance

In my presentation at our LGPS Officers conference in Edinburgh I mentioned a Russian proverb. "Doveryai no proveryai" (or, if you prefer, “Доверяй, но проверяй”). 

This translates roughly as “Trust. But verify.” – a popular nugget of wisdom that more recently has been appropriated by the software testing community. It’s used to emphasise the idea that whilst you can trust a developer’s good intent when writing code, you should always test it to catch hidden or emergent bugs. 

Interestingly, the proverb is written in the “imperfective” case - something we don’t have a direct equivalent for in English (but is common in Slavic languages).  

Imperfectives imply an ongoing activity. A close comparison in English might be “I was writing my book when you came to the door.” In English, this is in the past tense. However, in this case I didn’t mean to suggest my book “was written” in its entirety.  

In English, it’s technically ambiguous as to whether the book was finished or not. But, in Russian, it’s made explicit that this is an ongoing activity. 

Similarly, this ongoing nature of assurance should be explicit in our governance processes. The Pensions Regulator’s (TPR) Artificial Intelligence (AI) plan supports this by asking for more than a “one-off” sign-off. 

The expectation is that any AI should be subject to rigorous testing, assurance and monitoring, not only at the point of delivery or implementation, but on a continuous basis afterwards.  

The “afterwards” is the challenge 

Assurance isn’t a certificate you earn once. AI models are regularly updated, either through the algorithms they use or the body of reference data they’re trained on. Data quality also changes over time, and so AI tools that behaved impeccably last year may quietly stop doing so this year as things change.  

TPR’s guidance recognises this and so - in a similar way to the GDPR - their expectations run downstream through the supply chain. 

Funds are asked to assure themselves that not only officers and committees, but their administrators, providers and advisers have similarly robust arrangements in place. 

It would of course be easy to view this assurance as extra work, which acts as a brake on progress, whereas I think it could mean the opposite. 

Done well, governance shouldn’t be a blocker but instead enable change. A modern, proportionate governance framework can let funds and suppliers adopt AI with confidence, due to a level of transparency and trust (rather than holding back action due to caution). This should enable better standards and downstream trust and collaboration. In turn, this leads to better member outcomes and it’s this more optimistic view that I believe TPR has adopted. 

Ultimately, the funds that generate the most value from AI will be likely be the ones where their own controls, and those of their suppliers, are good enough to ensure that adoption is safe.  

All that said, governance can only assure what the underlying data and information within the system enables or supports. A brilliant governance layer sitting over the top of poor processes and low-quality data is simply a well-documented substandard process.  

In conclusion 

The key takeaway here is that AI should not be treated as exceptional or separate from good governance. It’s that the familiar aspects of governance such as accountability, oversight, regular assessment and a focus on positive outcomes for members still apply. Not only that, but they apply even more than they did in the pre-AI era of only a few years ago. 

Next time, we’ll take a closer look at the foundation that determines whether any impactful use of AI can really work in practice… Data. 

To discuss any of the information in more detail, please get in touch, we'd love to hear from you.

This blog is based upon our understanding of events as at the date of publication. It is a general summary of topical matters and should not be regarded as financial advice. It should not be considered a substitute for professional advice on specific circumstances and objectives. Where this blog refers to legal matters please note that Hymans Robertson LLP is not qualified to provide legal opinion and therefore you may wish to obtain independent legal advice to consider any relevant law and/or regulation. Please read our Terms of Use - Hymans Robertson.