CHAPTER 3: METHODS

Over the years of my consulting and teaching internationally, I have used focus groups, Likert scales, cross-stakeholder dialogue collaboration, mathematical modeling, and idealized system design as research methods and as participant assessment tools. Sometimes I have used one of the methods alone. Sometimes I have used multiple methods. Millward (1995) assures us that "although surveys and focus groups originate from different research paradigms, there is nothing inherent in the methods themselves that should prevent them being integrated into one research design" (p. 278).

Bickman and Rogg (1998) believe that

(o)ne method of enhancing validity is to develop converging lines of evidence. [A] clear hallmark of applied research is the triangulation of methods and measures to compensate for the fallibility of any single method or measure. The validity of both qualitative and quantitative applied research is bolstered by triangulation in data collection. [They, with Hedrick,] support the use of multiple data collection methods in all types of applied research. (p. xvii)

This is what I set out to do. In the pages to follow, I note my methods one by one: design, as a benefit to be covered by an underlying health plan, of a managed care mental health plan which responds to the losses; study of such a plan's plan premium cost-neutrality; administration of a focus group (which also functioned as a stakeholder expert group and as an idealized system design imaging group); administration of three Likert scales during the focus group; and qualitative focus group transcript content analysis.

Here are my research methods one by one.

1. Design of a Point-of-Service Triple Option Outpatient Managed Care Mental Health Plan

Having been a health plan systems designer, and having studied the literature of these four losses and become disturbed by compelling losses to consumers and clinicians, I decided to try a particular direction of outpatient managed care mental health plan re-design (POSTO), to see if such a design could ameliorate some of the damage. I used a direction in which my colleagues and I had taken the development of an early point of service triple option managed care overall health plan. I did a series of draft plan designs, limiting myself to the area of normal outpatient mental health, settled on one design, studied it further, then brought a design forth.

(I also saw the emerging plan design as a good way to stimulate expert focus group thinking on idealized system design imaging, a group design process explored during the focus group meeting, particularly in the later stages of the meeting.)

Seeing the POSTO plan design as a method of exploring and possibly easing the "four losses," I will show it in my RESULTS section.

2. Feasibility study of POSTO cost-neutrality

As a pilot study, I settled on doing a feasibility study of whether such a new POSTO plan (by having its benefits, eligibility panels, and co-pays formulated and then fine-tuned) could be priced reasonably cost-neutrally in comparison to managed care outpatient mental health plans already in the marketplace. (Even a good new plan design will likely be ignored if it costs more than what is on the marketplace now.) I had a senior California underwriter from a major health plan do an independent confirmation of my conclusions. That feasibility study will be shown in my RESULTS section, and the independent cost-neutrality underwriting confirmation report written by Chris Alesso will be shown in the APPENDIX.

It seemed to me that I must provide evidence that the mental health benefit plan design I was exploring would not be likely to cost health plans more in operation. If the cost were neutral or better, policy makers and users could concentrate much better on the plan design's real potential merits and demerits. If the plan design study were to turn out reasonably cost-neutral, it would lend the design further credence and reinforce the value of further study.

I did the study and developed math and logic which led me to the conclusion that cost-neutral operation of a next generation POSTO plan design was feasible. I then had a senior California underwriter, the Assistant Director of Underwriting for Kaiser Permanente in California, write and calculate an appendix to the study giving her independent opinion. She concluded that reasonable cost-neutrality was feasible, given the design I had developed, the panel assumptions, and the consumer co-pays.

3. Focus group (also functioning as a stakeholder expert group and as an idealized system design imaging group)

I capped my study by holding a half-day group session of stakeholder experts in this problem area. In April 1999, I conducted a focus group for the following purposes: For my own research, for study for the POSTO managed care outpatient mental health subject I am doing with the Association for Humanistic Psychology, and in preparation for possible participation in a conference on mental health plan design, a conference which was then to be held shortly.

For my dissertation, I took the data I gathered and archived during this April 1999 half-day meeting (meeting audiotapes turned into written transcriptions) and analyzed the manuscript, side-by-side with results of three Likert scale administrations taken at that meeting and a set of idealized system design imaging session outcomes, or "recommendations."

The group I held discussed explicitly how they were, in fact, acting in three capacities at the same time: as a focus group, as a dialogue group among multiple stakeholder experts, and as an idealized system design imaging group. To me, a small group could fit more than one classification at a single sitting. I knew that I wanted to study a small group made up of experts from different stakeholder groups. This is a method often recommended in conflict resolution. I knew I wanted them to take a hand in idealized system design imaging, as a counterweight to my own design thinking. I also knew they would be functioning as a regular focus group, being led by a skilled facilitator, in common business terms considering a problem and evaluating a potential product.

I analyzed that April 1999 archived material. I note below some of the valuable attributes that this material brought to my study.

I designed the focus group composition to contain planners from a number of different sides of the controversy (over consumer and clinician loss in managed care outpatient mental health) and from distinctly different stakeholder groups. This was an unusual opportunity. The study assumes that it is advantageous to have experts from major stakeholders act as idealized system planners of needed design reform (Banathy, 1996).

I have designed a draft of a managed care mental health plan which I think might improve on the losses reported. This individual effort served as a stimulus for a stakeholder group, those who must live with the design being developed or used, to consider what they deeply want. After the stimulus that my mental health plan design provided got them thinking before and at the group meeting, participants then tried to deepen the conversation to consider images or foundations of a more ideal system they want to see created. The April 1999 group held such an imaging process during their session together, with the last hour focusing pointedly on their new ideas, six foundational ideas highly agreed upon about how to develop managed care outpatient mental health plans. I presented their six recommendations, which they strongly agreed on, in Chapter 6.

4. Focus group research use of Likert scales

I gathered evidence on this study's questions from the focus group by using Likert scales. I have analyzed the forms returned to me for the RESULTS section.

A Likert scale is a summated rating scale (Kerlinger, 1964). It "is a set of attitude items, all of which are considered approximately equal 'attitude value,' and to each of which subjects respond with degrees of agreement or disagreement (intensity)" (p. 496).

The Likert scales were set to ask each participant for his or her (intensity-weighted) opinions about each of this dissertation's research questions, one by one -- about whether the participant believed any of the losses have taken place, whether those losses were perceived to be serious, whether POSTO might be able to help and what other principles might help.

Likert scale research questions about the "four losses" were asked at the end of the first hour. Research questions about the POSTO's pertinence were asked at the end of the second hour. Agreements on these and other focus group participant ideas which emerged were registered at the end of the third and last hour.

Even though focus group results do not easily lend themselves to generalization, these experts were articulate, and a multi-stakeholder expert group being in strong agreement or disagreement adds to the other evidence gathered.

The Likert questions asked the participants my primary research questions. The categories of these questions also form the outline for the final report. Key selections from the focus group transcript were organized around these categories. I constructed my assessment from the combination of evidence gathered.

5. Qualitative focus group transcript content analysis

As I studied focus group transcript analysis techniques, I found Bickman and Rog (1998) and Millward (1995) describing an analytic process Millward calls "qualitative content analysis" (p. 288) in which the "emphasis is on meaning rather than on quantification" (p. 288). The process involves studying the transcript carefully, sorting transcript content into primary categories of thought and inquiry (to be then organized around the research questions outline) and then studying the participant comments together in each content-specific sorting.

In the perspective of the several methods of gathering evidence, I constructed a major outline made of the research questions. In outline order, I will add explanatory material from each of the "Four losses." I added to that key material from the transcript other sources of evidence directed to the issue of each particular research question.

I pulled the archived transcript of the focus group's audiotaped dialogue and analyzed it by means of qualitative focus group transcript content analysis. I included this analysis in the RESULTS section.

Stewart and Shamdasani (1998) say that "there is no one best or correct approach to the analysis of focus group data" (p. 515) [and] that "(w)here analysis is treated [in the literature], the discussion is often limited to efforts to identify key themes in focus group sessions." A colleague who does focus group consulting as well as teaching (David Van Nuys, personal communication, December 16, 1998) described to me such a procedure: He reads the focus group transcript carefully, marks it up by looking for themes and key phrases, separates comments on current ideas and proposed ideas, labels the useful material, and writes his conclusions.

According to Stewart and Shasmdasani (1998), in the qualitative text analysis to be used, sets of categorized materials "provide the basis for the development of a summary report. Each topic is treated in turn, with a brief introduction. The various pieces of interview transcription are used as supporting materials [in the case of this research, along with Likert scale results, etc.] and incorporated within an interpretative analysis" (p. 516).

Selection of participants for the focus group

By providing space for the variety of stakeholders and thus for the different perspectives notable in different stakeholder groups, by providing background material on potential problems and potential approaches, it was anticipated that dialogue among participants with these different backgrounds could produce useful evidence.

In Millward's (1995) view,

it is not the intention of focus group methodology to yield generalisable data, so random sampling is not necessary. None the less, it is important to employ a systematic strategy when deciding on group composition. The sample should be chosen on theoretical grounds as reflecting those segments of the population who will provide the most meaningful information in terms of the project objectives. (p. 279)

Morse (1994) says that "with intensity sampling, one selects participants who are experiential experts and who are authorities about a particular experience" (p. 229). I wanted to allocate a balance of seats in the focus group for participants with an expert and leadership background in the service of one or another of the main stakeholders concerned with covered outpatient psychotherapy. I define main stakeholders of the outpatient mental health benefit issues to be those plan participants receiving outpatient mental health service, those professional clinicians providing service, the employers and unions working to reach decisions about employee benefit design under California collective bargaining laws for the services and professional panels to be provided, the health plans they contract with, the managed care organizations which the health plans may subcontract with for mental health services, academics studying these phenomena, and public bodies deciding and carrying out public policy in relationship to them for the benefit of the public at large.

Desirable characteristics

I decided that to be eligible for participation in the group persons would need the following characteristics.

1. 20 years of more of background in public jurisdiction work regarding large Northern California public jurisdictions or health care work serving those jurisdictions

2. Leadership within that organization or organizations

3. Significant experience over the years studying health plans and employee benefits for the workers, families, and retirees

4. If possible, familiarity or experience with Point of Service Triple Option health plan design

Table 1

Stakeholder categories: One or more a necessary background for focus group members

(Definition: One or more such backgrounds, with at least 20 years of expert experience)

1. Consumers

2. Outpatient psychotherapy clinicians

3. Employers

4. Unions

5. Health plans

6. Managed care organizations

7. Public bodies

8. Academics

Note: These categories represent such interests as service recipients, professional providers, employers/buyers,employees/consumers/clients/patients and their unions, health systems providing benefits, public policy developers in and out of government, and the academics at all levels and consultants/researchers who serve them, and governments at each level.

Choosing group participants

For the expert group I wanted to bring together in April 1999 in Berkeley, I listed for myself who I would most want to have in a focus group on this subject. Each expert I had under consideration was someone with whom I had worked substantially in the 1980s in our public jurisdiction health care system design work.

I asked the top group. Fortunately, each one accepted.

List of focus group members who accepted the April 1999 Berkeley meeting

Important note: Listing of employer, where applicable, is for identification purposes only. Each focus group participant came to the group as an individual.

Each an expert, with experience as a leader in one or more primary stakeholder groups

Employers: Hal Cronkite, former City Manager, City of Berkeley; Adele Amodeo, then health policy and legislation coordinator, President's Office, Office of the Vice President for Health Affairs, University of California System; currently Policy Director, The Partnership for the Public's Health, Public Health Institute, Oakland.

Employees: (consumers of covered outpatient mental health services): Ernie Ciarrocchi, Deputy Executive Director, California Teachers Association, former legislative advocate (lobbyist) for CTA, former local CTA representative, Richmond Unified School District

Clinicians: (providers of outpatient mental health services): Maureen O'Hara, psychotherapist in private practice; President of Saybrook Graduate School and Research Institute, San Francisco; former President of the Association for Humanistic Psychology*

Health Plans: Chris Alesso, senior underwriter, Kaiser Permanente Health Plan, California*

Managed Care Mental Health Plans: (Chris Alesso and) Charla Parker, teacher and consultant; former Director of the Alta Bates EAP plan, the Mills Peninsula Hospital EAP plan and Alta Bates Hospital's EAP plan

Public bodies: (Hal Cronkite, former City Manager) and Jennae Wallach, health care consultant to employer and employee groups at cities, school districts, and large non-profits

Academics: (Maureen O'Hara, Adele Amodeo, Charla Parker)

Focus group facilitator: Bruce Johnsen, consultant; Chief of Staff for International Projects, Conflict Resolution, Research, and Resource Institute (CRI), Tacoma; partnering facilitator, consultant, Monterey

*Two were called away to last-minute emergencies. Instead of sitting in the focus group, each of the two participated instead in a one and 1/2 hour one-to-one session with me the next week, our dialogue patterned after the agenda and assessment instrument issues of the focus group meeting.

Rationale for using a half-day focus group to study: (a) four losses, (b) a POSTO outpatient mental health plan design, and (c) the participants' own fundamental thinking emerging together

"Within psychology alone," says Millward (1995-6?) "[the focus group] has gained a substantial foothold since 1988 and it is especially popular within applied psychology, particularly health psychology" (p. 275).

Emergence of stakeholder opinion

Millward (1995) says that [f]ocus groups let participants react to problem dimensions or the acceptability and unacceptability of proposed solutions in a non-structured environment" (p. 277).

Bickman and Rog (1998) tell us that "[f]ocus groups are widely used in marketing to explore consumers' reactions to products and packaging" (p. 275).

The case at hand is both centered in health psychology and in exploring consumer and other stakeholder opinions about the conditions of outpatient mental health care under managed care and also about possible use of the POSTO outpatient mental health plan to help reduce the losses being studied. Bickman and Rog (1998) say:

Focus groups also have a place as a confirmatory method that may be used for testing hypotheses. This application may arise when the researcher has strong reasons to believe a hypothesis is correct, and where dis-confirmation by even a small group would tend to result in rejection of the hypothesis. (p. 506)

Distance

I felt that I was so immersed in the data that, to seek evidence further and deeper, I needed the distance that an independent focus group could bring to the problem to give a fair look toward at least a distillation of those consumer and clinician losses and that POSTO outpatient mental health plan design. And I wondered what the expert participants, in mid-focus group, would begin to say.

Rich body of data

Morgan (1988) says the "hallmark for focus groups is the explicit use of the group interaction to produce data and insights that would be less accessible without the interaction found in a group" (p. 12). I thought that if I were fortunate, the focus group might produce "a very rich body of data expressed in the respondents' own words and context" (p. 514). I was hoping for Morgan's "main advantage focus groups offer": "the opportunity to observe a large amount of interaction on a topic in a limited period of time" (p. 17).

Client satisfaction

Because the "four losses" under scrutiny may possibly represent an extreme in client (and stakeholder) dissatis-faction, the fact that focus groups are an increasingly common method for collective consumer product evaluation further suggests a focus group's use.

Group size

Millward (1995) says that "a systematic perusal of recent focus group research in psychology yields an average of nine participants per session as conventional with a range of six to twelve which on the whole is consistent with figures quoted in the focus group literature, although some would advocate between six and eight participants as ideal" (p. 280-81). I got eight confirmations for our focus group date. I knew then that we might sit down with 7 or 6. In the face of two last-minute emergencies, we did sit down with six participants.

Preparing for the focus group meeting

I have constructed, facilitated, and assessed a number of focus groups and focus group-like experiences over the years in my consulting and teaching practices.

I called each participant, first to gain agreement to participate, and second to explore a possible half-day meeting date, which ended up Friday, April 23, 1:30-4:30 pm.

Next I secured an appropriate meeting room. I thought we needed good accoustics (both for the discussion and for the taping quality), good chairs and space for the focus group setting, reasonably aesthetic surroundings, a comfortable place to break, good bathroom facilities, and a convenient and reasonably central location. After two months of checking, I settled on a conference room in the Berkeley City Club. That room met all of the criteria. As a bonus, it is an architectural treasure designed by Julia Morgan.

I sent off a pre-meeting mailing to the participants. The mailing had several parts: a cover letter, "Four Losses (a part of this study's literature review that looks at the reported losses in some detail," a list of fellow participants, a draft agenda outline for the focus group meeting, and directions to the City Club. The package went out four weeks before the agreed upon date of our meeting.

You can find a final version of the "Four Losses" manuscript as Chapter 2 in this dissertation, pages 15-53, especially the sections starting with the section "Four Losses," pages 28-48.

Table 2

The study's sources of evidence to confirm or disconfirm the hypotheses

Main research questions	Study methods
Have the four losses occurred?
Literature review Focus group participant statements Likert scale on whether losses occurred and frequency
If one or more losses have occurred, how seriously?
Literature review Focus group participant statements on whether Likert scale on seriousness of losses
If serious losses have occurred regularly, might POSTO help?
Literature review sections on POSTO Re-design study Design cost-neutrality study Idealized system design draft as stimulus to idealized system imaging process Focus group participant statements Likert scale on POSTO applicability
What agreements or other ideas does the focus group itself add or subtract?
Idealized system design imaging Focus group transcription analysis Likert scale testing of any ideas and emergent agreements

Table 3

Subjects studied by each research method

You are visitor number

Website by: Chakra 5 studios

Top