Communication of methodology and statistics in research proposals
This draft (Jan 02, 2024) aims to introduce researchers to the key ideas in research methodology that would help plan studies and write research proposals.
Our target audience is primarily the research community at VUB / UZ Brussel, those applying for funding at the WFWG or the ethical committee in particular.
Note that we present our view, suitable for communicating research at VUB / UZ Brussel, not necessarily outside. Therefore, what we present should only be used for guidance, not as an argument or proof of any kind.
We welcome your feedback: wilfried.cools@vub.be
Your research proposal
- convince referees that
- your study addresses interesting questions ~ WHY → peers
- your study will be successful: effective ~ HOW → peers and maybe statisticians
- your study will be successful with minimal (not too high) costs: efficient ~ HOW
- cost defined in terms of money, directly or indirectly, and/or ethically
- at least, findings will outweigh the cost
- ideally, findings obtained with minimal cost
- cost valued dependent on type of application
- funding: show return value of investment
- ethical approval: show necessity of potential risk/harm/stress/…
- cost defined in terms of money, directly or indirectly, and/or ethically
- convince statisticians in particular
- include necessary methodological / statistical arguments
- in a way a statistical referee understands (no clue about your area of expertise)
- ideally separate the WHY and HOW, the latter read by statisticians
Methodology and statistics
- explain your study
- explain WHAT you want to show
- explain HOW you want to show that
- DO NOT
- merely list statistical tests
- copy-paste from other proposals
- write generic texts that always apply
- include irrelevant details
- repeat WHY; stick to WHAT and HOW
Key Ingredients
- aim of the study: WHAT you want (confirmatory, exploratory, preparatory, techn(olog)ical)
- design of the study: HOW you can do it (quantity, quality, generalization)
- aim should match design: often linked by statistics
Research Aim: first key ingredient
- research aim: concisely express what the study intends to achieve
- frame in terms of the essence, avoid unnecessary (technical) details
- be specific, go beyond very general statements
- operationalize, relate to empirical evidence
- focus, highlight research questions of primary interest
- argue what the results should be -at a minimum- for it to be a successful study
- comment on additional gains the study could offer
- example: The aim is to show that the new treatment P is not worse than the common treatment Q, and we consider our study a success when the average score on measurement Y, with values expected between 16 and 64, is maximally 10% less for P. It will further be explored to what extent patient characteristics X could further explain the scores Y in both treatments.
- with detailed statement instead of vague statements
- compare: aim is to test treatment
- linked to the empirical evidence
- focus, not a vague list of measurements
- with detailed statement instead of vague statements
- very very bad example: We will test normality, and it if is not normal we will… and if it is normal we will… and we will use R and SPSS and Excel, and we will summarize with means and …
- focus on WHAT and HOW ?
link your conceptual research question to one that is measurable
clearly define
- the variables of interest
- how they are measured
- their relation
be mindful of wording
- differ, change or evolve ?
- …
Categorizations of Research Aims
the type of research aim determines how to deal with it (properties and requirements)
various categorizations can be considered, for example:
- confirmatory / exploratory / preparatory / technological
- quantitative / qualitative
- inferential / descriptive
note: type labels are informal, to be used for guidance only
typically, studies tend to be valued more (scientifically) if
- they build on an understanding of future results (without being certain)
- they offer a future understanding beyond the study (wider / deeper)
- thus: if you can, frame it as such
Descriptive versus Inferential Research
- inferential, study a population using a sample, implies generalization
- (ideally) uses representative samples, large enough, randomly sampled
- more ambitious, thus difficult → guarantee inference is possible
- relates to statistical testing, estimation, prediction
- descriptive, study the observed data as such, no generalization
- present data -as is- without reference to uncertainty nor p-values
- easy to perform but less easy to sell as important
- requires argument that data in itself is of interest
- typically includes
- summaries (average, median, …, correlation)
- noticeable values
- note: inferential studies typically imply descriptive preliminary analysis
Quantitative versus Qualitative Research
- quantitative research addresses quantifiable empirical aspects
- typically makes use of visualization and statistics to summarize and generalize
- can be descriptive and/or inferential
- typically requires to reduce complexity (operationalization before data analysis)
- main focus is summarizing and generalization
- → determines how to argue for it
- qualitative research addresses understanding
- especially focused on reasons, opinions, motivations, …
- is descriptive and can be hypothesis generating (inductive)
- typically embraces complexity
- main focus is interpretation / understanding ~ meaning
- → determines how to argue for it
- mixed methods combines both
- typically very simple quantitative part to support the qualitative one
- rarely purely one or other
- e.g., operationalization often implies qualitative moment
- e.g., statements about more or less imply quantitative moment
- note: leaving out statistics does not make it a qualitative study
- note: asking respondents questions does not make it qualitative
Types of Objectives
While a confirmatory study is scientifically stronger and more convincing, it is also much more demanding to perform and justify.
Confirmatory Research Aims
- goal:: confirm an expected difference, relation, …
- means:: in advance specify results -at a minimum- to support claim
- aim for
- significant difference (superiority / non-inferiority) → statistical test
- equivalence given a margin of error → statistical test
- absence of evidence is not evidence of absence
- accurate estimate → statistical estimation
- expected observations → falsifying alternative hypotheses
- …
- aim for
- focus::
- explain (statistical) link research design and (especially) primary aim
- for a statistical test or estimation → calculate sample size using standard error
- make assumptions and conditions explicit: effect size, statistics, type I and II errors
- conclude on required sample size: costs & availability
Exploratory Research Aims
- goal:: explore observations, possible differences, relations, …
- without any guarantee on what will be the result
- means:: in advance specify results -at a minimum- to support merit (no reference to significance or accuracy)
- interest in data as such (descriptive)
- no -primary- interest in statistical testing or estimation
- interest in parameter estimates (differences, relations, …)
- aim for statistical testing or estimation but
- no guarantee sufficient power and/or accuracy
- maybe try ‘upgrading’ it to confirmatory research ? → decide on effect of interest
- aim for statistical testing or estimation but
- interest in prediction
- aim for cross-validation (does not include standard errors)
- open question how to argue sample size, refer to similar research / common practice
- possible to justify afterwards using bootstrapping → discuss criteria
- aim for cross-validation (does not include standard errors)
- qualitative research
- interest in data as such (descriptive)
- focus::
- argue why the data or parameter estimates by themselves are of interest
- even if not significant/inaccurate
- or why significance/accuracy is likely
- based on similar existing research
- put more weight on substantive arguments
- sample size -justification- with balance information and cost
- argue that merit outweighs cost
- low cost of collection, or data already available (eg., retrospective)
- still explain (statistical) link research design and potential inferences
- argue why the data or parameter estimates by themselves are of interest
Preparatory Research Aims
- goal:: prepare for a future study… typically a small scale set-up
- means:: in advance specify what information is required for a future study and how it will be obtained
- pilot study
- aim to successfully set up future study
- phase I and II clinical designs
- aim to avoid risk, harm, … future study
- requires decision criteria to proceed or not
- database development, data collection procedures, …
- pilot study
- focus::
- argue based on information required for future (actual) study
- explain how unavailable information is obtained
- argument could be (partially) qualitative, descriptive, …
- example: understand instructions, register observations, …
- results are not by themselves of interest
- no statistical testing is implied, that is for future (actual) studies
- sample size -justification- based on an absolute minimal cost
- for example, with animal experiments typically 3 animals per condition
- allow the estimation of variance
- low cost of collection, or data already available (eg., retrospective)
- for example, with animal experiments typically 3 animals per condition
- argue based on information required for future (actual) study
Techn(olog)ical advancements
- goal:: to design, engineer, create, …
- not to extract information from the outside world
- means:: argue merit of the final product, rarely any statistics involved
- focus::
- argument based on what the advancement offers, in balance with the costs
- no statistical justification, and that is alright !!
Research Design: second key ingredient
- research design: strategy to achieve the research aim
- effective (question can be answered) and efficient (with acceptable/minimal costs)
- determines how (potential) observations provide information on research aim
- a poor design makes a study inefficient at best, completely ineffective at worst
- statistics can not solve design problems
- example: study different treatment effects with fixed order ~ confounding
- three types of design attributes
- quantity of observations (sample size)
- quality of observations, dependent on
- what is observed
- how it is observed (method)
- under which conditions it is observed (~ variables)
- generalizability (from sample to population), dependent on
- selection/allocation of research units
- missing data mechanism
- focus, highlight the good choices when relevant
- show what is done right, that you have it under control
- discuss in relation to relevance, maybe name-drop
- do not dwell on what you (may) fail to deal with
specify the conditions under which observations are made
clearly define
- the conditions of interest
- e.g., treatment - control
- e.g., before - after treatment
- the number of observations
- possibly their correlation
- the conditions of interest
Quantity of Observations
- collect enough relevant observations
- more observations are more informative, ensure that sufficient (effective)
- more observations required if they are less informative by themselves (see quality)
- more observations may be more costly, ensure not too many (efficient)
- remember, costs in terms of time, money, risk, stress, …
- more observations are more informative, ensure that sufficient (effective)
- justify quantity of observations
- typically only required for the primary research questions, and/or costly observations
- for confirmatory research, perform sample size -calculation-
- specify statistical test(s) or estimation(s) in focus
(implied by what -at a minimum- ensures a successful study). - specify effect size aimed for, justify both the effect and the uncertainty
- effect: ideally justified substantively or at least referring to common practice or literature
- uncertainty: variance of measurement, ideally based on earlier data/research or pilot
- use rules of thumb when all reasoning fails, only
- specify operational characteristics (type I error \(\alpha\) and type II error \(\beta\) are related)
- note: an \(\alpha\) of .05 and power of .8 (\(\beta\) = .2) implies type I error 4 times more severe
- specify statistical test(s) or estimation(s) in focus
- for exploratory / preparatory research this includes a sample size -justification-
- feasibility and/or low cost
- minimum requirement for a first impression
- similar / equivalent research
Quality of Observations
- collect informative observations (especially important if relatively few)
- observations made with most informative method
- validity & reliability
- example stress:
- self rating vs. neuro-endocrine vs. …
- memory vs. now vs. imagined situation vs. …
- observations made under the most informative conditions
- link with research question
- include suspected confounders
- example stress:
- short/long term difference - evolution
- lab - naturalistic ?
- observations made with most informative method
- general principle: isolate the effect, avoid / measure unwanted influences
- control confounding variables (~ complete the model)
- maximize systematic variability (~ explained variance)
- minimize non-systematic variability (~ unexplained variance)
- highlight how quality is ensured, lack of quality is avoided
- show you have it under control, you know why you set up your study this way
- be specific, relate to your research design and aim
- do not highlight weaknesses (too strongly)
Quality of Observations — Confounders
- explain control of confounding (unwanted outside influence)
- balance out confounding
- randomization (large enough sample size)
- stratified randomization
- measure confounding
- repeated measures (~ conditioning)
- cross-over designs (~ conditioning)
- blocking, keep variances sources stable and estimate (~mini-experiments)
- avoid confounding
- matching, create similar groups to compare
- (double-) blinding, avoid influence experimenter/experimented
- and more …
- balance out confounding
- often more complex designs are more efficient but also more complex to analyze
- eg., mixed models to deal with repeated measures
Quality of Observations — Non-Systematic Variability
- variability that is not understood ~ uncertainty
- explain minimization of non-systematic variability
- use of proper measurement tools
- ensure reliability / precision to avoid noisy measurements
- combine measurement tools
- repeat observations (replications) to average out noisy measurements
- use tools that are well understood/studied
- use of all information available
- include all relevant predictors
- not only focus on main variables of interest (~ model)
- potentially consider combinations (interaction, polynomial, …)
- avoid the use of categories when continuous registrations are available
- consider (multiple) imputation when missingness is modest
- include all relevant predictors
- use of proper measurement tools
Quality of Observations — Systematic Variability
- variability that is understood ~ information
- explain maximization of systematic variability
- maximally differentiate conditions
- DoE, design of experiments
- example: detect change, effectiveness, …
- largely implies minimization of non-systematic variability
- ~ appropriate and sufficiently rich model for the observations
- maximally differentiate conditions
Quality of Observations — Experimental Control
- safeguard quality of information easier if (some) control on conditions
- experimental study, exerts control by definition
- choose the conditions of observation
- randomize the allocation to conditions
- necessary condition for causal conclusions
- observational study, does not exert control, efficiency loss at best
- increase quantity to compensate loss in quality
- includes naturalistic data collections, surveys, retrospective data collection, …
- experimental study, exerts control by definition
- highlight what you can control, and how you use that control
Generalizability
- collect a sample of observations with the aim to generalize (inference)
- conclude upon more than just the observed sample → avoid biased sampling
- type of sampling
- probabilistic (sampling: random, stratified, multi-stage, ….)
- necessary for conclusive, unbiased, objective inferences
- conclude upon population sampled from
- non-probabilistic (sampling: diversity, expert, ….)
- potentially biasing, more subjective, and best used only exploratory, descriptive
- argue why not biasing in your case
- no issue for qualitative studies
- probabilistic (sampling: random, stratified, multi-stage, ….)
- missing data: safeguard against / remediate
- generalization fails when substantial non-random missing data
- depends on mechanism of missingness
- problem if not at random → biased
- results in loss of precision
- depends on mechanism of missingness
- explain how it is avoided and dealt with
- how many missing values to expect, for what reason
- what to do to further minimize this number
- improve data collection
- obtain information to deal with missingness
- generalization fails when substantial non-random missing data
Statistics
- link between research design and research aim
- highlight how given a design, statistics is able to resolve the aims
- focus on primary research questions
- sketch of secondary research questions → maybe do some name-dropping
- reflect on the type of data and challenges they offer
- continuous/ordinal/nominal
- skewed, outliers, boundary values, …
- but avoid being vague or to give a lecture statistics 101
- highlight how given a design, statistics is able to resolve the aims
- introduce intended statistical analysis
- include
- statistical tests (evaluate whether effects exist; p-values)
- statistical estimation (evaluate confidence interval)
- prediction (evaluate model fit using individual observations)
- specify the expected inferences, be specific
- include
- note: sample size calculations are not part of statistics
- often based on simplified statistical testing/estimation
- typically not discussed separately
specify how observations obtained under specific conditions offer information on the operationalized hypothesis
clearly define
- the statistical techniques
- the statistical outcomes of interest
Small Example
The aim is to show that the proposed treatment is an improvement over the current standard method, as it will show in higher scores on my measurement.
Participants are randomized to either the treatment or control group. The control group is given a dummy treatment of which a post experiment survey addresses whether participants were aware. Each participant is measured twice, once immediately before and once immediately after the (dummy) treatment was administered. Each measurement results in a continuous score on a 0-10 scale. A mixed model compares the change between post and pre treatment measurement while accounting for pre-treatment scores. Also the following possible confounders will be included: …
A sample size was derived for a t-test that focuses on the detection of a post-treatment difference of 2 in favor of the treatment. The minimal clinical difference was decided upon by our expert panel. In literature, the standard method is indicated to have a population standard deviation of about 4. Because no information is available on the new treatment it is assumed that the same population standard deviation applies. This leads to a sample size of 51 patients in each of both groups, required for a one-sided test, type I error of .05 and power of .8. Earlier experiments showed a drop-out of about 10%, so 51 * (100/(100-10)) < 57 patients are included per group. Because the more advanced statistical test and the inclusion of various potential confounders, this number can be considered conservative but realistic. This number will also present no difficulties to collect in our center.
Some Practical Suggestions
- isolate methodological / statistical arguments from substantive reasoning
- use concise and consistent labeling of data and conditions
- visualize and structure wherever possible
data collection process
categories of observations and their relation
a text a statistical referee does not want to read
We will compare the effect of [treatment] on [condition] by randomizing mice over groups with [description of procedure the statistician does not need to know about] and groups with [description of another procedure the statistician does not need to know about ]. Each procedure will be performed with 2 [substances] and each at 3 different doses, namely [dosages].
We expect procedure A with substance 1, respectively substance 2 to achieve on average 13 units, respectively 15 units and a standard deviation of 1, respectively 1.5 at the lowest dosis, with a 10% increase in both mean and standard deviation at the second dosis and 20% increase for the third. For procedure B with substance 1, respectively substance 2 to achieve on average 11 units, respectively 13 units and a standard deviation of 1, respectively 1.3 at the lowest dosis, with a 10% increase in both mean and standard deviation at the second dosis and 20% increase for the third. We will also include a control group which achieves an average of 6 and standard deviation 0.5, where we expect dosis to have no effect.could be turned into
experiments can also be visualized, eg., time-line
design:
tables to list conditions between (rows) and within (columns)
Conclusion
- be clear on your aim and how to reach it with an appropriate design
- disentangle the WHY from the HOW
- highlight what is essential for your type of study
- highlight the good choices that you have made
- focus on what deserves focus
- use a language understood by the relevant referee
- statistical lingo
- visualization / structure
SQUARE: Methodological and statistical support to help make a difference
SQUARE Mission:
At SQUARE we meet to provide complementary support in statistics and methodology (qualitative and quantitative) to our research community, for individual researchers and research groups, in order to get the best out of their research.
SQUARE aims to further enhance the quality of both the research and how it is communicated.
website: https://square.research.vub.be/
includes information on who we serve, and how, and a link to our booking tools