Frequently Asked Questions: Evaluation of Compliance Program Effectiveness

They are independent reviews performed by experts that evaluate and validate the outcome of the program objective of reducing the likelihood of unwanted events or acts that could give rise to adverse consequences.  The focus is on whether the compliance program is achieving desired results. It includes reviewing existing compliance documents, not unlike that done in a Gap Analysis, but goes beyond that to evaluate the results and impact from these efforts.  It is a complete 360° review of the Compliance Program that includes debriefings and interviews of key leaders and selected staff for as well as evaluating the underpinnings of each program element. The focus is evidencing progress with implementation of the USSC, DOJ and OIG’s seven compliance program elements and in meeting regulatory requirements, industry best practices, standards, and enforcement trends.  

(1) Checklist Reviews are process reviews, not effectiveness evaluations, that are more like ongoing monitoring of the compliance program processes, but do not evidence the results of the process. 

(2) Gap Analyses are like the Check List review but is performed by outside parties who bring more objectivity to the results; however, these are also process reviews of outputs and do not evidence the effectiveness of the process.    

Reasons for having the Compliance Program evaluated include: (a) It provides the Compliance Officer with independent and credible evidence on the progress of program development, as well as identifying opportunities for improvements and enhancement; (b) It is called for in the OIG Compliance Program Guidance; (c) DOJ prosecutors, when investigating an organization, will ask if there were credible evaluations in determining whether to bring charges or negotiate pleas or other agreements; (d) Often executive leadership and boards request evaluations to assist on if there has been a solid return on investment in supporting the Compliance Program; etc. 

Process results are the most common ways that Compliance Officers use in their reports regarding the progress of the compliance program. The OIG convened a roundtable with compliance officers that resulted in a compilation of 500 different process ideas to help improve compliance programs.  However, results of process have limited value in that they do not directly evidence compliance program effectiveness.  They provide only output data, often expressed numerically, but not the outcome of the process.  One of the few ideas in the OIG Roundtable Report for evidencing effectives was the use of employee surveys. it was cited over 60 times throughout the report for all seven compliance program elements. 

The OIG Compliance Program Guidance documents cited only three methods for evidencing effectiveness.  First and foremost is having a compliance program effectiveness evaluation by independent experts that examines not only the processes put in place but what resulted from it (outcome).  The second method is using questionnaires and surveying of employees on compliance knowledge, awareness, attitudes and commitment. To be credible the survey instrument should be professionally developed, validated, tested, and independently administered.  The third method relates to claims processing quality assurance program that uses quality control reviews of a sample of claims in process to identify and correct errors and make educational contacts of those making errors.  The reduction of errors and improper n claims provides evidence of effectiveness. 

Gap Analysis is a process review, normally by outside parties, with results characterized as outputs, many of which lend themselves to numerical results.  Examples of such include the number of (a) policies implemented, (b) individual receiving compliance training, (c) individuals and entities sanctioned screened, (d) hotline reports received, (e) frequency of compliance meetings with oversight committees, etc.  Theses reviews capture a limited portion of needed information needed to evidence Compliance Program effectiveness which is determined by the outcome of these processes. It is not often that effectiveness can be presented numerical values.  They are more difficult to express, such as whether employees understood the training and applied them in their work.   Reviews of this type can be useful to organizations under development to help guide the continuation of building the Compliance Program. 

Conducting an Internal Check List review is a process similar to a Gap Analysis but performed internally, usually by the Compliance Officer, with results being characterized as outputs, not outcome.  As such, they do not provide direct evidence of compliance program effectiveness which are the outcome from the process.  The results carry less credibility, to outside parties, than Gap Analyses performed by parties independent of the program reviewed.  These reviews should be considered as part of meeting the ongoing monitoring obligations of the Compliance Officer. However, to validate the results of the effort, it is necessary to have an Independent Compliance Program Effectiveness Evaluation. 

Annual evaluations do not make sense, in that it takes time to implement changes and enhancements to the program and have it taken hold in the workplace.   Having such reviews every three years should be adequate. 

If a survey is developed with the right content, and has been validated, tested, and is properly conducted; the results can provide considerable credible evidence relating to program effectiveness.  The OIG guidance specifically cites this as a sound method for gathering such intelligence.  Also, in a roundtable sponsored by the OIG with compliance officers, hundreds of ideas relating to evaluating compliance programs were mentioned, although the vast majority were related to process outputs not outcome.  The big exception in the Roundtable Report was over 60 references to survey results for evidencing effectiveness in all seven compliance program elements.  

Statistic and numbers provide data concerning the result or output of a process, they do not address the outcome of the process by which effectiveness is measured.  Process outputs often lend themselves to numerical results, outcome data seldom does.  Many Compliance Officers make the mistake of flooding leadership and the board with numbers: employees trained on compliance, individual and entities sanction screened, hotline reports, compliance program policies implemented.  These process output results don’t answer the questions as to outcome of the process, such as whether employees understood and applied the compliance training lessons, what was accomplished by sanction screening, how investigation of hotline complaints reduced exposure to liabilities, or what was accomplished as result of policy implementation.