London and South East Region’s ‘Electric Avenue’ event in September considered the future shape and feel of our cities. Among the keynote speakers, Simon Hobbs outlined how AI is already changing competency assessment – and not always for the better. This is an abridged version of his presentation
By Simon Hobbs
AI – or maybe, as Donald Trump now wants it to be called, ‘super intelligence’ – is, we all know, moving at a ridiculous pace. You only have to look at the news to see how quickly things are changing, and the debate about how to control it is changing. So, there is every chance that in 12 months, maybe less, this presentation is going to look hopelessly out of date.
I want to start with a real piece of evidence from a routine supervisor visit to a streetlight. It was to observe an operative changing a lantern; the supervisor recorded a video of the activity.
For the past six months or so, we’ve been using AI to review this sort of video – and it had been remarkably good. It was useful, consistent and, frankly, it was getting increasingly easy to trust.
Completely out of the blue, the AI identified a serious safety incident, failed the operative and criticised the supervisor. But there was one other major problem – the incident never happened; the AI made it up.
Now, we all know AI gets things wrong. The issue is that it got something so badly wrong, after having been so consistently good, that I felt it warranted a deep dive to understand the issues behind it, and the lessons this could potentially teach us.
So, what did the video show, according to the AI? The video, time-stamped to 15.59pm, it argued, ‘showed’ a lantern being dropped from a MEWP basket to the ground. The operative, it therefore also concluded, was not competent. Its report was structured and specific; it looked authoritative. Its conclusions were not just reviewing and commenting on the video, it was using its observation to reach an assessment judgement.
Furthermore, on another run of the video, the AI went further and said the supervisor should have intervened and stopped the job. In one report it even used the word ‘thrown’ rather than ‘dropped’ from the MEWP. It even questioned the evidence, arguing that it was a ‘heavily edited’ video, and raised concerns about ‘whether the observed actions were valid’.
WRONG INTERPRETATION
These are, naturally, pretty serious statements to put in a report. The technology is no longer just commenting on a piece of work, it is questioning somebody’s supervision and the reliability of the evidence supplied.
But, when reviewed by a human, the video clearly showed the AI’s interpretation was totally wrong. It was a windy day and, very simply, the wind blew over a plastic barrier, which then hit the pavement. The lantern, throughout, was still on the column.
So, how did it get from what actually happened to what the AI said happened? Up until that point, the AI had earned a fair amount of trust. We had been using it for months and the outputs had been consistently useful. That is how trust creeps in. Not because anyone decides AI is infallible. But because it gets things right often enough that the checking starts to feel less important.
Assessments, assessment method statement checks, defect photos, site footage, reports; these are exactly the sort of tasks where AI is attractive. And where I am sure we are all probably using it to some extent already.
To understand what happened, we needed to dig down into and understand how AI works. AI ‘learns’ through a large language model (LLM) process. Essentially – and this is of course a simplification – it puts information into a form it can process. Words, images, audio, video – they all have to be represented in a way the system will recognise.
When it writes an ‘answer’, it builds this a piece at a time, using the context and the patterns it has learned. Think of it a bit like auto-correct on your phone or in a document, albeit vastly more capable.
Now, let’s think about the report. The sentence about the dropped lantern made sense, grammatically. It fitted the type of work that we were doing; the safety language was familiar. The problem only became obvious when we compared that sentence to what actually happened in the video.
The system builds from the information available to it and, if it doesn’t have the information, it will then predict it, a bit like if you were to say, ‘I’m going to make a cup of…’ . The AI may go with ‘tea’ as the next most logical answer from what it ‘knows’. Or perhaps ‘coffee’. And then it will continue with that as a fact.
AN ANSWER VERSUS EVIDENCE
But the important point, for us, was that producing an answer isn’t the same as checking it against reality. A sentence can be fluent, it can be technically plausible, without the event it describes even actually happening. And that’s the distinction to hold on to when the output is being used as evidence.
So, let’s think about our video. We had the lantern worker working at height, followed by loud bang. How might that sentence continue for the AI? They were removing the lantern, there was a loud bang and the lantern was…. dropped, removed, lowered, smashed – it will fill in what it thinks is the most plausible, predictive answer. Because the description fits the ‘learned’ context.
The plastic barrier was the crucial information missing from the report’s account. The barrier starts upright. The wind catches it. It begins to tip and lands on the pavement. As the camera comes back up, the lantern is still on the column. The audio track aligns with the barrier hitting the ground. So, there was a real impact; that part wasn’t invented. And the surrounding job involved the lantern being worked on at height.
You can begin to see why a dropped object might be plausible. That is really important, because it is about context. The context for what the AI was doing was that it was having a lantern changed at height. If it had had a different context it would (probably) have come up with a different response. The report failed to account for the barrier.
We haven’t 100% managed to establish exactly why the system missed it, although we think we have a good idea. What we do know is that it did hear a real impact, it had the lantern work going on, and therefore (we think) it simply assigned the sound to the wrong object. Another intriguing element is that we ran the same footage through the system multiple times – across different models in fact – and every time the report said the same thing, that the lantern had been dropped from a height. Consistency tells you something is repeatable. That does not mean it’s true.
ROLE OF AI PROMPTS
One obvious point here is simply to argue, ‘well, clearly, Simon, your prompt was rubbish’. We found that wasn’t the case. The prompt we use has worked really well; it was detailed, it gave the system clear assessment criteria, it asked for direct evidence, it wanted time stamps, it asked for safety concerns, it asked for competency judgements. And up to then it had produced consistently useful results.
Could we improve it? Of course. There will always be wording you can tighten or additional checks you can build in. But prompting has its limits. Rather than asking whether the prompt was good or bad, the more useful question is ‘what did the prompt help with and what can it never guarantee?’. First, one of the safeguards in our prompts was the instruction, ‘only record what is visible or audible and do not infer anything’.
That is a pretty safe thing to put, isn’t it? From a supervisor or assessment point of view that is exactly what we want. But there is an important limitation to that instruction. An AI model has to interpret the information it receives in order to produce an answer. It makes inferences from patterns in the input all the time.
What we really mean by ‘do not infer’ is ‘do not turn something uncertain into an unsupported finding’. But AI is an inference model; that is what AI does. And that is where it failed; it treated the ‘dropped’ lantern as something it had observed. The use of time stamps in this instance also did not help because, if anything, they simply made what was an unsupported finding look even more like ‘evidence’. We had asked the AI to highlight any concerns or non-compliances.
Once the bang had been linked to the lantern, that instruction gave the model somewhere to put the event, as a critical failure; it had somewhere to go. We asked for a competency judgement. This is where the error became really consequential. That one false observation now affected the outcome for the operative, deeming him incompetent.
Finally, because the prompt also required a mandatory supervisor review, the same false event created a second consequence. If a dropped lantern really had happened, the supervisor might reasonably be criticised for failing to intervene. But because the event never happened that criticism was also built on sand. To be clear, the prompts did not create the mistake.
Once the evidence had been misunderstood it gave that mistake structure, precision, authority. A good prompt can reduce the error, but it can also make a mistaken finding more plausible and authoritative than it is. It would be easy on this basis to dismiss AI as useless. But in that same video the AI identified a lot that was in fact useful: safe isolation sequence, that the operative was using a torque wrench correctly, that his waste handling skills were good and so on.
So we don’t ignore incredibly useful observations simply because of the error. Any more than we should ignore the error because some other observations were good. That’s the balance that I would like to strike.
THE ‘COMPETENCY PARADOX’
In conclusion, as we all know, AI can do a lot of useful work for us. Reviewing stuff, organising and presenting evidence – it can make the competent person’s job easier. But I would rather have it help a knowledgeable person work through evidence than pretend that that knowledgeable person is no longer needed.
In an assessment like this, that person’s ability to question a result is part of what makes this whole process useful. The check must be done by someone who understands the subject, you still need a subject expert.
Being good at operating the AI isn’t the same as being a judge of that person; yes, the useful AI output and the competent human check belong in the same workflow. But neither should replace the other. This also brings me to what is known as the ‘competency paradox’.
Consequential AI outputs should be checked by someone who is experienced and competent. But where does that person come from? Engineering, software and assessment, those people don’t arrive as experts. They become experts by doing a lot of fairly ordinary work first. Schedules, grid specifications, inspecting installations, writing routine code and so on. They make mistakes and they have those mistakes corrected.
Yet those tasks are some of the easiest things to give to an AI. Therein is the paradox. The more successfully we automate this junior-level work, the more careful we may need to be that we do not automate away the processes that creates tomorrow’s experts.
That does not mean we shouldn’t automate; we are all doing it. But that does mean training has to become part of an AI strategy. It may sometimes mean we need to give developing engineers and technicians work that an AI can technically do faster.
Because that is how they are going to learn. AI is a very useful tool. But human judgement matters, and (hopefully) always will.
Simon Hobbs is owner and managing director of Electrical Testing Ltd
This is an abridged version of the article that appears in the November-December edition of Lighting Journal. To read the full article, simply click on the page-turner to your right.
Image: Electrical Testing Ltd
