The Value of Scientific Conferences: a longitudinal survey
James W. Aiken, Ph.D.
Chief Executive
Officer
Keystone Symposia on
Molecular and Cellular Biology
I can’t remember
the last time I attended a conference where I wasn’t given a form to fill in
evaluating the event. Most are questions that help organizers plan future
events; few ask questions that get to the core of the conference’s lasting
value. While writing a grant proposal for the Bill & Melinda Gates
Foundation, I found myself in the position of having to propose a method for
quantitative assessment of the value of our Keystone
Symposia conferences. Had I been proposing to vaccinate children, well I
could count them, but what about a conference? Keystone Symposia has stated objectives: To connect the scientific
community for the benefit of society, for example, or to catalyze scientific
progress and accelerate achieving research goals. Contemplating how to measure
these objectives was the origin of the survey described below. It is a first
step to document whether attending a conference produces factual value. The
Survey was conducted over two seasons of Keystone
Symposia’s meetings; Winter/Spring 2004 and Winter/Spring 2005. The list of
conferences included in the survey is shown in Table 1. [Find at the end of
this document].
Format of Survey
The survey asked respondents whether they
agreed or disagreed with a statement about the conference attended, using a five
point scale: Strongly Agree; Agree;
Neutral; Disagree; Strongly Disagree. Data were collected,
initially, during the final 24 hours of eight selected conferences using a
paper questionnaire, and again, nine months following each conference using an
email questionnaire (10 conferences with nine-month data; the original 8 plus 2
additional ones), and again by email at 18 months after the 2004 conferences (5
conferences with 18 month data). Responses
were anonymous and at each time period the data were independent, i.e. we could
not link a response from any person at one survey time to their response at the
next survey time.
The results presented herein focus on
the nine month data, which has been repeated over two seasons and includes 1013
responders from 10 conferences (30% response rate). Independent scientists were
56% of the responders, postdoctoral fellows 25%, and Students 19%, a similar
distribution to the actual attendance. Data from the 18 month survey, which was
obtained from 268 responders for just the five 2004 conferences, are included
sparingly, mainly for the purpose of showing that responses, when compared with
nine months, were consistent with earlier survey data and/or changed in the
expected direction over time. Nine month results from the two conferences whose
attendees did not see the survey during the conference were not significantly
different from the eight who did.
Collaboration and Sharing of Information
Results from the
questionnaire given out on the last day of meeting were of interest only to
compare what people anticipated versus
what actually happened within 9 months after the meeting. For example, at the
time of the meeting, they were asked to respond to this statement:
I
established contact with someone at this conference that I anticipate will lead
to either a collaboration or future sharing of information, data, or
techniques.
At the time, 70% agreed or
strongly agreed that they anticipated a contact made would lead to either
collaboration or future sharing of data or techniques; 26% neutral, and 4%
disagreed.
In the 9-Month
survey, the statement was reworded to determine what actually had occurred
since the meeting:
Since the conference, I have fostered contact with
someone I met at this conference that has lead to either collaboration or
sharing of information, data, or techniques.
|
|
Strongly
Agree
|
Agree
|
Neutral
|
Disagree
|
Strongly
Disagree
|
|
Number
|
195
|
412
|
293
|
91
|
18
|
|
Percent
|
19%
|
41%
|
29%
|
9 %
|
2%
|
At nine months, 60% of responders
agreed that their contact with someone had actually led to collaboration. It
was reassuring to find that at eighteen months 63% agreed, both confirming the
9-month data and suggesting a little more movement in the direction the
originally anticipated 70% outcome. An often stated objective of Keystone Symposia is “to connect the
scientific community”, so these results were particularly encouraging for us.
At the time of
the meetings, 90% agreed with the statement that they anticipated that they
would share information learned at the meeting with colleagues who did not
attend. Nine months after the meeting, again 90% agreed that they had, in fact,
shared new information with colleagues who did not attend.
Since the
conference, I have shared new information that I learned at the conference with
my colleagues who did not attend.
|
|
Strongly Agree
|
Agree
|
Neutral
|
Disagree
|
Strongly Disagree
|
|
Number
|
360
|
543
|
87
|
14
|
4
|
|
Percent
|
36%
|
54%
|
9%
|
>2%
|
>1%
|
It is
not unexpected that almost everyone agreed with this statement, but even so, it
is heartening to confirm that the information learned at the conference is being
disseminated by such a high percentage of attendees to a broader audience.
During the
meeting, 64% agreed with the statement: “I will leave this conference planning
to accelerate publication of some of my data.” At nine months the statement was
reworded:
Since the conference, I have submitted or published
some of my data related to topics discussed during the conference.
|
|
Strongly Agree
|
Agree
|
Neutral
|
Disagree
|
Strongly Disagree
|
|
Number
|
144
|
248
|
340
|
218
|
59
|
|
Percent
|
14%
|
25%
|
34%
|
22%
|
6%
|
After the conference, 49% agreed
that they had submitted or published data related to topics discussed during
the conference; at eighteen months, 52% agreed with the statement
Impact on Research
At nine months
and eighteen months, respectively, 64% and 66% of the attendees agreed with the
statement that they became aware of a new idea or concept at the conference
that, since then, has altered the direction of their research.
Since the conference, a new idea or concept (either
openly stated or created in your mind), which I became aware of during the
conference, has altered the direction of my research.
|
|
Strongly Agree
|
Agree
|
Neutral
|
Disagree
|
Strongly Disagree
|
|
Number
|
139
|
497
|
295
|
67
|
7
|
|
Percent
|
14%
|
50%
|
29%
|
7%
|
>1%
|
Similarly, 39% and 52% at nine
months and eighteen months, respectively, agreed that since the meeting there
had been a major advance in the field that was based on an idea or concept
revealed or conceived of during the conference.
Since the conference, there has been a major advance
in this area of research, either by me or others in the field, based upon an
idea or concept that was revealed or conceived of during the conference.
|
|
Strongly Agree
|
Agree
|
Neutral
|
Disagree
|
Strongly Disagree
|
|
Number
|
76
|
316
|
513
|
89
|
12
|
|
Percent
|
8%
|
31%
|
51%
|
9%
|
1%
|
This survey did not address how
an investigator’s research direction was altered, or exactly what breakthrough
they thought happened during the following 18 months. Answers to those
questions, however, would be relevant to assessing specific outcomes resulting
from information learned at the conference, a rigorous test of the value of any
conference.
At the time of
the conference a very high percentage, 85%, agreed with this statement:
I
learned something specific at this conference that I anticipate will save me
time, money or accelerate reaching a research goal.
Nine months later, 42% of
attendees agreed that they had actually used something they learned to save
time, money or accelerate reaching a goal. Even though only 6% disagreed with
the statement (48% were neutral), the “agree” results at nine months were only
half of what had been anticipated during the meeting. Eighteen months after the
meeting, agreement with the statement had jumped to 52%, but still below
anticipated. With all five of the previous statements above, there was no
difference in the way students, postdoctoral fellows or independent scientists
responded, but with this one there was. Students or postdoctoral fellows
comprised about 44% of the responders at nine months and independent scientists
the remaining 56%. We found that Independent Scientists (data below) were more
likely to agree with this statement:
Since the conference, I have used something specific that I learned at
the conference that has saved time and/or money, or accelerated reaching a research
objective.
|
|
Strongly
Agree
|
Agree
|
Neutral
|
Disagree
|
Strongly
Disagree
|
|
Number
|
127
|
164
|
251
|
20
|
3
|
|
Percent
|
23%
|
29%
|
44%
|
4%
|
>1%
|
Fifty-two percent of independent
scientists agreed with the statement; 30% of trainees agreed (not shown above).
Furthermore, as expected over time, because as trainees move on, the portion of
responders who were independent scientists went up, from 49% during the
meeting, to 56% at nine months, to 62% at 18 months, which may account in part for
the increased “agree” response at 18 months. These results make one speculate
whether students and postdoctoral fellows might sometimes learn new information
that adds to their workload, an issue not addressed in this survey. If so, hopefully,
the information that added work also improved the final quality of their
investigation.
A follow up
question was included for all of those who agreed with the statement above,
asking them to quantify the time saved and/or the money saved. Some responders
commented that they were not able to do that very precisely and did not attempt
it. Some gave answers that were not useable for quantifying data, such as
“weeks” or “months” or “thousands”. My favorite comment was: “I was awarded an
R01 grant – Priceless”. In fact, several
responders indicated that they saw a direct connection between information
learned at the meeting and a successful grant application.
The actual amount
of time saved by 250 responders, grouped in quintiles of 50, is shown in Table
1, along with the average estimate of money saved (or diverted to more
effective use) from 140 responders; 28 in each quintile. The range of responses
was wide, and skewed by some high values. (One extreme one was excluded.) Hence
the top quintiles were substantially higher than the medians, which were 6 weeks
and $6000 saved.
Table 2: Time and money saved
assessed 9-months following the conference.
|
Quintiles
|
Lowest
|
4th
|
3rd
|
2nd
|
Highest
|
|
Average Time saved
(wks)
|
2
|
4
|
7
|
10
|
33
|
|
Average Money Saved
(US$)
|
$1000
|
$3300
|
$6800
|
$12,000
|
$48,000*
|
* Highest value, $2.5 Million,
not included in this mean.
Potential Impact on Careers
At
nine months, 33% of attendees agreed or strongly agreed with the statement:
Since the
conference, someone with whom I established contact during the conference has
been helpful to my career.
|
|
Strongly Agree
|
Agree
|
Neutral
|
Disagree
|
Strongly Disagree
|
|
Number
|
75
|
259
|
481
|
164
|
30
|
|
Percent
|
7%
|
26%
|
48%
|
16%
|
3%
|
At eighteen months, 39% agreed. Interestingly, the data for
independent scientists, students and postdoctoral fellows were not different.
Discussion
What did we
learn? I think we learned what we hoped to be the case, or perhaps what we previously
had assumed was true: Attending a scientific
conference is not only fun but also has immediate value for your research
program. This survey quantifies, at least for one organization’s conferences,
the extent of the certain benefits for the attendees.
The
two aspects that stand out most are, first, the high number of scientists who
met someone with whom they started collaboration, and second, the high number
of scientists that learned something that changed the direction of their
research. I would like to think that some of the collaborations were catalyzed
by the retreat-like environment conducive to informal interactions that we strive
for at our conferences. The average attendance of the events in this survey was
about 330, and our conferences are held at venues where attendees have lots of
time to mingle and do unstructured activities together. It would be interesting
to compare these data to results from a much larger (or smaller) meeting held
at an urban venue.
The finding that
66% of attendees learned or conceived of something new that changed the
direction of their research is the tip of an important iceberg. What lies below to be discovered is the
impact of the changes. A challenge for meeting organizers is to design
practical studies that obtain information to confirm whether or not the change
in direction led to positive outcomes. Comments added to the survey would
suggest that this is the case, but the evidence is still anecdotal.
It
would make sense that changes in research direction are linked in some way to the
responders who claim to have saved time, money or reached goals faster. In fact,
80% of those who said they saved time also agreed that they learned something
that changed the direction of their research. Saving time might be as simple as
meeting someone at the conference who can give you a needed reagent that may
have taken you two weeks to prepare yourself. Some claim to have saved two
years; it would be interesting to know the details here. The value of time
saved is hard to assess, which may account for some responders not attempting
to answer that question with a dollar amount. At the easy end of the spectrum,
one can simply translate the cost of running a project for a week and multiply
it by the weeks saved, and assume that the funds were diverted to more
effective use or to advancing the project towards its goal ahead of plans. At
the more complex end, how does one assess the value of reaching a goal faster
than anticipated?
The amount of
money that participants “saved” has to be considered a rough estimate. The
value of a meeting may not be easily translated into dollars, and I question following
that path in future surveys. Rather, I would like to pursue avenues that reveal
specific, and perhaps unexpected, outcomes that resulted from ideas generated
because the event took place. One aspect of the money diverted to more
effective use, however, is worth mentioning.
Conservative extrapolation to the entire Keystone Symposia season of meetings would suggest that at least
$20 to $30 Million in research funds are diverted to more effective use each
year use because of information learned at our conferences. Since most of the
responders are Academic and Government scientists (over 80%), the more
effective use of research funds is likely to be money that has come from major
funding agencies (NIH, MRC, etc). It would follow that one consequence of the
conferences is that government (i.e. taxpayer) funded research support is used
more efficiently.
The results are
of potential value to Keystone Symposia. They provide a benchmark from which we can
attempt to raise standards. Continuous improvement in conference quality should
be an expected outcome. We have shared most of this information with other
non-profit meeting organizations, and at least one that I am aware of, the Wellcome Trust at Hinxton, UK,
plans to use it internally as a benchmark. I began this article with the
implication that post-conference surveys were routine, perhaps too routine. We
use a different email survey following every meeting, which provides useful
feedback on the operational success of a meeting, as wells important comments
for helping future organizers. A challenge for us is to measure our progress in
enhancing conference quality without being too bothersome for attendees. All meeting organizations have objectives. Keystone Symposia wants to “connect the
scientific community for the benefit of society”. This survey provides evidence that we are
doing fairly well connecting the scientific community; more data are needed to
determine to what extent that results in benefits for society.
Table 1: Keystone Symposia conferences included
in the survey
2004
NF-kB:
Biology and Pathology
Biology of Hypoxia:
Role of Oxygen Sensing Normal Function and
Disease
siRNAs and miRNAs
Molecular Mechanisms
of HIV Pathogenesis
HIV Vaccine
Development: Progress and Prospects
2005
Diabetes Mellitus: Molecular Mechanisms, Genetics
and New Therapies
Obesity: Molecular
Physiology and Genetics of the Control of Body Weight
Tuberculosis: Integrating Host and Pathogen Biology
Molecular Helminthology: An
Integrated Approach
Drugs against
Protozoan Parasites: Target Selection, Structural Biology, and Medicinal Chemistry
____________________________________________________________________
