Case test

Cases groupConclusionCases groupAPG takes the next s...

APG takes the next stepin client contact with generative AI

Future Facts Conclusion helps pension administrator to create intelligent chatbot The pension provider APG manages pensions for no less than 4.5 million Dutch people, who have taken out pensions with one of its eight affiliated pension funds. APG has very little contact with most clients during the pension accrual phase. When their retirement date approaches, clients suddenly have a great many of questions, and it often gets quite complex. How can the administrator provide information more efficiently? And what role can ChatGPT play in this? APG approached Future Facts Conclusion with those questions.

A few years ago, the pension provider APG developed a number of technical solutions for clients to get answers to their questions without contacting an employee or specialist directly, using a chatbot. This platform took some of the contact centre’s workload, and was quite successful. But as it so often goes with technology: once it’s up and running, you don’t look at it again. “At the same time, we saw that the technology behind chatbots is developing incredibly quickly,” says Ron Jongen, who, like his colleague Chris Veerkamp, is an Innovator with the Groeifabriek at APG. “It is time for the move towards a true conversational platform. The market and the technology are ready for that now.”

Groeifabriek as idea generator

The Groeifabriek, or “Growth Factory”, is APG’s innovative heart for new products, services, and concepts. Ron: “We focus on development and bringing services and solutions to the market that help people get a handle of their finances. By doing that, we want to contribute to their prosperity and well-being; now and later. We don’t just do this for ourselves, but precisely for our affiliated pension funds and their members.”

As innovators, Ron and Chris’ responsibilities include idea generation within APG. “We draw inspiration from trends in the market, from reports by the CPB and other organisations, from technological developments, and also from sparring sessions, for example with Future Facts Conclusion, one of our partners in the field of innovation. And of course we listen very carefully to our internal clients. Technology must contribute to solving their problems, and is not an end in itself.”

Easier access to knowledge base

One of the problems was that many questions were so complex that customers calling the contact centre were on hold for a long time. Ron: “We have a very extensive knowledge base. For complex questions, contact centre agents have to search that knowledge base for the right answer. That takes a lot of time. With the advent of ChatGPT, we thought: can’t we use this technology to find that answer?”

He asked Future Facts Conclusion to contribute ideas. A good 20 to 25 ideas emerged from this exploration phase, says Chris. The team used the ‘Now, How, Wow’ methodology to plot the ideas according to business opportunities and technical difficulty. “We think it’s important to look at the innovative character, and also the feasibility and workability within APG. That’s why we want to start with ideas that contribute directly to the business, using technology that is already proven in the market.”

Because there are plenty of ideas. Like, for example, the desire to add a button to the MyPension environment with ‘explanation of your pension’. Chris: “When you log in now, you immediately see a rather complicated graph. If you have good financial or numerical literacy, you’ll have no problem with that. But some people are immediately put off by a graph like that. We want to make the information more accessible. How cool is it if you just get a spoken explanation of that graph, in your own language and at your own level?”

"We think it’s important to look at the innovative character, and also the feasibility and workability within APG."

-Chris Veerkamp

Setting up a framework

The exploration phase made it clear that the basic idea - a chatbot for contact centre agents so they can find answers to the sometimes-complex questions from clients more quickly - was worth developing. And that leads you to think about how you can get the ideas accepted in an organisation like APG. We found that there was common ground for various ideas involving generative AI. Given the other ideas, it seemed like a good idea not to develop one solution to one problem, but to set up a framework that could be used to work out multiple ideas for ways to improve customer service using generative AI.

The framework makes it easier to address issues that are in line with each other. “We actually took a step back first,” says Chris. “The primary aim of this first project is to explore what needs to be done to implement generative AI successfully within APG. Once we know that, we’ll have a foundation that we can use for a number of solutions to be developed in this area.” The framework will also help to make better assessments of which technology to use, for example.

Translation and rewriting

In the meantime, APG is also exploring whether using AI can improve services in other pension fund issues. An example is the question of the Schoonmaak (“Cleaning”) pension fund, which has many members who are not comfortable in the Dutch language. Chris: “They asked us to help think about how to reduce barriers in using and processing the communication.”

The first step was to investigate what kind of barriers there are. It turns out that aside from language, the complexity of information was also a barrier. With his background as a communication researcher, Chris immediately knew that it is best to offer information in multiple layers in that kind of situation. He explains: “We took inspiration from the Lees Simpel (“Read Simple”) app. With that app, you take a picture of a complicated letter you have received from an authority. The app then translates the letter into layman’s terms and explains what is expected of you, point by point.

Our ambition is to make pension information accessible in a similar way, point by point and in very clear language. And not just in Dutch, but in the pension plan member’s native language. This is especially important for the Schoonmaak pension fund, because many members didn’t grow up in the Netherlands.”

"AI tends to hallucinate. The technology predicts what should be there and then presents the information as fact."

-Bastiaan Sjardin

Technical complexity

The thought many people will have is: how hard can it be to create a chatbot for complex pension questions and a translation and/or rewriting app based on ChatGPT? After all, ChatGPT can already generate easy-to-read text at the six language levels distinguished by the European reference framework, CEFR. You can also specify exactly which documents can be used as input when formulating an answer. That reduces the chances of the chatbot using incorrect information to zero.

Even so, it is complex, says Bastiaan Sjardin, CTO and AI Strategist at Future Facts Conclusion. “Because AI tends to hallucinate. The technology predicts what should be there and then presents the information as fact. The chances of that going wrong are very low for simple and straightforward questions, but a lot higher when complex information needs to be interpreted. Of course, APG can’t afford to make mistakes. We aren’t talking about ordering a pair of shoes; this involves very important information about people’s financial futures. We have to completely rule out even the smallest mistake.”

"The prompt has to take into account the desired tone of the answer and the level of accuracy required."

-Bastiaan Sjardin

Importance of prompt engineering

That means a lot of energy has been put into prompt engineering. Sjardin: “You tell the chatbot how you want it to answer the questions, and what context should be included. So the prompt has to take into account the desired tone of the answer and the level of accuracy required. To make sure the bot always gives the right answer to a question, you’ll have to create very many prompts. Then you need to validate the output generated with both specialists and end-users. People’s responses will probably vary, and after a validation session like this, you’ll end up with a lot of input to improve the prompts. It is a process you will have to keep repeating. That’s why by far the most energy goes into this.”

"With our framework approach, we automatically take lessons learned in one project into another."

-Ron Jongen

Another complexity is in translating text. “There are quite a few niche languages that are widely spoken in the cleaning industry in particular, like Bulgarian and Albanian,” says Bastiaan. “Those language models are nowhere near as rich as the models of more commonly used languages like English or German. So we started with nine languages, including Polish, Ukrainian, Moroccan and Arabic.” It turned out, for example, that Moroccan and Arabic are easily lumped together in language models, even though they really are two different languages.

Future

At the time of the interview, APG is in the pilot phase of this chatbot solution. However, ten contact centre agents are gaining experience in a pilot programme with a prototype chatbot that consults the knowledge base. Ron: “We have an implementation horizon of one to two years. During that period, we aim to deliver several products based on generative AI. With our framework approach, we automatically take lessons learned in one project into another, so you get more and more acceleration towards the end.

Right now, we’re still in the phase of gaining experience with how this tooling can start adding value in practice. We start small, in the contact centre, and then expect it to spread like an oil slick. Because in theory you can use this technology in all forms of customer contact, but to reap the benefits, you need to approach it carefully. Because if you cut corners and the chatbot comes up with the wrong answer once, people lose trust. We absolutely need to prevent that.”

Emerging Technologies Trend Report 2024 #2

trend report 2024 #2Emerging technologies

The Conclusion companies continuously monitor emerging IT trends and explore new technologies that may be relevant to our clients. In our semi-annual trend report, we highlight several technologies that, in our view, deserve special attention.