“Quality is never an accident. It is always the result of intelligent effort”

(John Ruskin 1819-1900).

Quality is an abstract concept and a term with which we are all familiar and tend to take for granted.  It is something to which all aspire in life and certainly do so in the conduct of clinical research and trials, where Quality Assurance and Quality Control (QA and QC) functions are considered paramount.  However, what does it really mean and, more to the point, how do you measure it validly in this scenario?

In recent years new clinical trial management techniques have become popular and have been advocated as ‘quality drivers’ – for example ‘Risk-based Monitoring’ (1) and ‘Quality by Design’ (2).  Nevertheless, the impact has not been immense and this fact, a study (3) by two founders of a company specialising in pharmaceutical performance analytics, is because we have been using the wrong measures to assess quality all these years.

They focussed the survey-based statistically validated study on clinical trial conduct (as opposed to study start-up or close-down) and aimed to identify key drivers of performance and relate them to quality.  In this way they identified metrics that have the greatest impact on trial quality.

Firstly, through a series of interviews over 18 months with CRO and sponsor trial managers they identified seven broad ‘performance activity’ areas and drivers of quality considered important: adaptability, adherence, enrolment, functions, monitoring, project management, and site relations.  Interestingly, the consensus was that last two were considered to be the most important, carrying the greatest impact.  Each area could, of course, comprise more than one item – for example ‘adaptability’ could refer to adaptability in terms of protocol amendments, change-order processes, protocol violation (PV) management, query resolution.

After this initial qualitative fact-finding phase, quantitative statistical analysis was then performed on data from a second phase using an online survey tool (82 usable surveys returned from Phase II – IV trial managers) to distil the previously identified areas down to ‘essential indicator’ items within each broad area and produce figures for the magnitude of their relationship (i.e. importance) to quality of trial conduct.

The authors of the study report do acknowledge that terms they use such regression modelling, predictive or business analytics may be off-putting but emphasise relative underlying basic nature of such techniques, which can be simply performed on an excel spreadsheet. Indeed, some readers of the original ACT article reporting the study may find the statistical analytical techniques just a little bamboozling (and, incidentally, that does include this reader and author of this post, despite having decades of involvement in clinical trials). Suffice to say that this was a sophisticated but entirely appropriate analysis with probabilities of significance being derived from calculations of the standardised regression coefficient beta, indicating the strength of a relationship and gamma statistic, providing a measure of association for ordinal variables. This should not detract from the essential findings and messages of the exercise.

Items found to be significant (t statistic, p<0.05) from those originally considered within each main performance area (italicised below) were as follows:

Adaptability: protocol amendments and PVs

Adherence: to study protocol and to medical management/safety plan

Enrolment: adhering to the timeline for ‘last patient entered’ (only – no other enrolment-related metrics were not found to be significantly related to enrolment performance)

Functions: performance on project management, data management, regulatory, centralised diagnostic service, CRF tracking, and external data sources.

Project Manager, site relations and monitoring: each of these three areas measured as a single global indicator in their own right (i.e. no separate indicator items).

Using the model, beta coefficients were computed to describe the strength of the relationship between the performance activity and quality.

Applying the model for these items, the relationship between the performance drivers and quality are shown in Figure below with histogramatic representation of the regression coefficients (b):

Thus the drivers with the most impact on quality were Project Management performance (b=0.35, p=0.001) and managing Site Relations (b=0.31, p=0.002). Adaptability and enrolment (considered in terms of getting the last patient in on time) were also worth focussing on in terms of trial management.

1. http://www.fda.gov/downloads/Drugs/…/Guidances/UCM269919.pdf (accessed 20/11/2015)
2. Lawrence X. Yu, Pharmaceutical Quality by Design: Product and Process Development, Understanding, and Control. Pharmaceutical Research, April 2008, Volume 25, Issue 4, pp 781-791
3. Howley, MJ and Malamis, P. Quality drivers in clinical trial conduct available at: http://read.findpharma.com/i/569396-applied-clinical-trials-ebook-2015 (accessed 25/11/2015)