Auditing performance is a routine necessity of good business management. Where underwriting and claims functions are concerned, how good are quality and accuracy? Is basic philosophy applied in practice? Are procedures and processes being adhered to? Are outsource suppliers meeting the required standards? And internal staff too?
But that is only half the picture. Performance auditing demonstrates quality to stakeholders in and associated with the business, and can highlight other issues, such as the need for improved processes, new training and adjustment of core philosophy.
Technical audits
Historically the classic ‘technical audit’ involves the review of a sample of say 200 to 300 case files and has been performed by reinsurers. This is usually on an annual basis with the objective of demonstrating that an insurer’s underwriters are correctly following underwriting philosophy, often laid down by the reinsurer in their underwriting manual. Companies receive a report detailing any errors or differences of opinion, often graded from minor to severe. Some of the better audits go further, encompassing areas such as process efficiency and the quality of documentation. Sometimes separate administration audits are carried out.
Reinsurers have also used similar methodology to carry out due diligence audits on a new client to ensure that the underwriting standards assumed in the pricing are being followed.
More recently reinsurers have realized that the traditional case audit is akin to looking for a needle in a hay stack and that companies often have their own internal underwriting audit process in place which reviews more cases than the reinsurer ever could in a couple of days spent on site. So the emphasis of the audit can switch from a case audit to ‘audit the auditors’. where the focus is on reviewing the outcomes and methodology of a company’s audit process.
But recent years have seen the introduction of new methods of risk appraisal, such as tele-interviewing and underwriting engines, and these too need to be included in the audit process. After all an engine can process many times more cases than can a human underwriter with a corresponding increase in the error rate if the engine has not been implemented, tested and monitored correctly.
Tele-interviewing
Tele-interviewing is now firmly established in many markets as an excellent way to obtain risk appraisal information quickly and cost effectively. Many companies audit those cases where tele-interviews have been obtained.
In 2010 SelectX published a report on tele-interviewing in conjunction with Hank George and SCOR Global Life, drawing heavily on a worldwide survey on the subject. We asked companies about their audit program for tele-interviewing and, given the importance of regular auditing to the management of a successful tele-interview operation, it was disappointing to see that a significant number of companies did not audit at all, or only did so on a reactive ‘ad hoc’ basis.
In North America, UK, Ireland and continental Europe, some 80% of companies said they audited their tele-interviews. Of these over half did so on a regular basis and the balance ‘as needed’, that is episodically. Surprisingly, dependent on the market, between 10% and 20% of companies did no auditing at all.
More worryingly a number of those companies who outsource their tele-interviews to third party providers did not audit these cases and even among those who did audit their tele-interview cases up to a quarter did not listen to the interviews themselves.
So what factors should be part of a robust tele-interview audit? Is it just about reviewing outcomes and technical underwriting decisions or is there more to it than that?
- Is the process working smoothly? And can it be improved?
- Do interviewers establish rapport and empathy with the applicant for maximum spontaneous disclosure?
- Are interviewers capturing detail accurately?
- Are interview outcomes good enough for swift and final underwriting decision-making?
- How does the process work for customers and distributors?
Ideally the traditional review of underwriting files in which decisions are based on tele-interview information should be extended beyond whether or not the underwriting decision is correct. The review should look at the decision-making process and whether it draws fully on the power of the interview. It should consider how well underwriting manuals and internal guidelines support tele-interview information by giving direction appropriate to the new type of information and enabling decision-making without recourse to traditional evidence.
An audit program needs to recognize that the quality of a tele-interview is critical, especially as for many companies the tele-interview replaces a significant amount of medical reports.
Interview quality depends primarily on the skills of the interviewer: how they manage the conversational exchange with the applicant and create an effective working relationship.
Auditing rules engines
“To err is human but to really foul things up requires a computer.” Bill Vaughan, American columnist and author
Once a rules engine has been put into use, it should be reviewed in a consistent manner, not only to pick up inappropriate decision outcomes but to also to make certain the rule sets are refined and enhanced in the light of experience. In the global survey of rules engine users we asked companies how often they audit their engine:
Source: Underwriting engines: the new strategic imperative in life and disability business. Report by SelectX and Hank George
In the vast majority of companies, auditing is performed by underwriters. However, in 20% IT staff with no underwriting knowledge was used. Approximately half of companies have auditors who also have other responsibilities, while only one in five carriers has one or more full-time auditors.
It is not sufficient to view a rules engine audit in the same way as one addressing technical underwriting; operational risk management needs to be an integral part.
There is more to the rules engine audit program than sampling a percentage of cases to see if the outcomes are consistent with those of a human underwriter. The risk of repeated and systematic errors caused by even a minor programming mistake needs to be managed by a rigorous system of checks and controls. Regular review of management information (MI) needs to be an integral part of any audit program. And considering what is at risk, an external expert assessment is often desirable as well.
The ability to audit adequately is inextricably linked to the quality of the engine’s testing tools and its MI (management information) capabilities.
The ability to produce detailed MI reports is an essential feature of any rules-based new business system. Many engines are now supplied with a module that enables analysis of underwriting data and operational metrics. The most comprehensive ones come with a standard set of reports plus a user- friendly tool that allows underwriting management to run ad hoc reports and to analyze these data without involving IT colleagues.
Given that implementing an engine involves an ongoing program of reporting, analysis, refinement and improvement, making full use of a good reporting tool is a critical success factor in this process.
A robust audit program is an essential part of running a rules engine.
With the numbers of cases that can be put through an engine, there is considerable operational risk with implications for reinsurance treaties and other stakeholder interests. A rules engine audit program needs to encompass both the principles of operational risk management as well as those of a technical underwriting audit.
So what should a rules engine audit include? Well a combination of approaches to ‘examine the elephant’ from various angles. It could include some or all of:
- A comparison of decisions produced by the engine with those of the human underwriter
- A review of the operational risk management issues surrounding an engine
- Running through a number of scenarios using the engine’s test harness
- A review of a number of decision trees using the rules- authoring software
- A critical look at the MI and the lessons learned to improve STP rates and the customer journey
- A comparison with best practice.
Summary It should be clear that the days of the basic technical underwriting audit that only reviews the decisions on a sample of cases to see if the underwriter ‘got it right’ are numbered, particularly if this is a one-off or annual event in isolation. Today what is needed is a holistic approach to auditing that looks at all of the various components of the risk appraisal process to ensure that they are fit for the modern world.
