# Supplementary online material (SOM3): Results for the WHYQUALITY question

## Table of Contents

Open question WHYQUALITY (answered by all respondents by typing in)

- InterRater Reliability (IRR)
- Valid answers
- Abbreviations
- Problems detected
- Number of themes
- Number of characters

## InterRater Reliability (IRR)

Cohen’s Kappa and percentage of agreement across Coders 1 and 2 for each variable studied

Metric | Cohen_Kappa | Agreement_Percentage
--- | --- | ---
VALID | 0.68 | 91.44
ABBR | 0.46 | 98.06
PB | 0.19 | 97.94
NTHEMES | 0.62 | 73.61

## Valid answers

### Proportions per group

Proportions of non sense, not in line, DK and valid answers using the codes of Coder 1

Coder 1 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
1 Non sense | 0.4 | 2.1 | 0.8 | 0.8
10 Depends | 5.7 | 7.4 | 9.6 | 6.1
2 Not in line | 4.5 | 3.7 | 3.2 | 6.1
3 DK | 5.7 | 3.7 | 4.0 | 4.9
4 Valid | 83.6 | 83.1 | 82.3 | 82.0
Number of observations | 244 | 243 | 249 | 245

Proportions of non sense, not in line, DK and valid answers using the codes of Coder 2

Coder 2 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
1 Non sense | 0.4 | 1.2 | 0.8 | 0.8
10 Depends | 2.5 | 7.0 | 6.0 | 6.1
2 Not in line | NA | 1.6 | NA | 2.9
3 DK | 6.6 | 2.9 | 4.8 | 5.3
4 Valid | 90.6 | 87.2 | 88.0 | 84.9
9 To be defined | NA | NA | 0.4 | NA
Number of observations | 244 | 243 | 249 | 245

Proportions of non sense, not in line, DK and valid answers using the codes of Coder 3

Coder 3 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
1 Non sense | 0.4 | 2.1 | 0.8 | 0.8
10 Depends | 6.1 | 7.8 | 8.4 | 5.7
2 Not in line | 2.5 | 2.5 | 2.4 | 5.7
3 DK | 5.7 | 3.7 | 3.6 | 4.9
4 Valid | 85.2 | 84.0 | 84.7 | 82.9
Number of observations | 244 | 243 | 249 | 245

### Tests of significance across groups

Tests non sense Coder 1

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 1.000 | ns | 1 | ns
Control | PushRecording | 1.000 | ns | 1 | ns
PushDictation | PushRecording | 1.000 | ns | 1 | ns
Control | Choice | 0.216 | ns | 1 | ns
PushDictation | Choice | 0.427 | ns | 1 | ns
PushRecording | Choice | 0.440 | ns | 1 | ns

Tests non sense Coder 2

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 1.000 | ns | 1 | ns
Control | PushRecording | 1.000 | ns | 1 | ns
PushDictation | PushRecording | 1.000 | ns | 1 | ns
Control | Choice | 0.613 | ns | 1 | ns
PushDictation | Choice | 0.978 | ns | 1 | ns
PushRecording | Choice | 0.993 | ns | 1 | ns

Tests non sense Coder 3

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 1.000 | ns | 1 | ns
Control | PushRecording | 1.000 | ns | 1 | ns
PushDictation | PushRecording | 1.000 | ns | 1 | ns
Control | Choice | 0.216 | ns | 1 | ns
PushDictation | Choice | 0.427 | ns | 1 | ns
PushRecording | Choice | 0.440 | ns | 1 | ns

Tests not in line Coder 1

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.608 | ns | 1 | ns
Control | PushRecording | 0.553 | ns | 1 | ns
PushDictation | PushRecording | 0.186 | ns | 1 | ns
Control | Choice | 0.827 | ns | 1 | ns
PushDictation | Choice | 0.959 | ns | 1 | ns
PushRecording | Choice | 0.305 | ns | 1 | ns

Tests not in line Coder 2

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushRecording | 0.0227 |  | 0.106 | ns
PushDictation | PushRecording | 0.0211 |  | 0.106 | ns
Control | Choice | 0.1310 | ns | 0.378 | ns
PushDictation | Choice | 0.1260 | ns | 0.378 | ns
PushRecording | Choice | 0.5510 | ns | 0.551 | ns

Tests not in line Coder 3

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 1.000 | ns | 1.000 | ns
Control | PushRecording | 0.112 | ns | 0.612 | ns
PushDictation | PushRecording | 0.102 | ns | 0.612 | ns
Control | Choice | 1.000 | ns | 1.000 | ns
PushDictation | Choice | 1.000 | ns | 1.000 | ns
PushRecording | Choice | 0.114 | ns | 0.612 | ns

Tests DK Coder 1

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.497 | ns | 1 | ns
Control | PushRecording | 0.832 | ns | 1 | ns
PushDictation | PushRecording | 0.797 | ns | 1 | ns
Control | Choice | 0.398 | ns | 1 | ns
PushDictation | Choice | 1.000 | ns | 1 | ns
PushRecording | Choice | 0.669 | ns | 1 | ns

Tests DK Coder 2

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.5230 | ns | 1.000 | ns
Control | PushRecording | 0.6930 | ns | 1.000 | ns
PushDictation | PushRecording | 0.9670 | ns | 1.000 | ns
Control | Choice | 0.0893 | ns | 0.536 | ns
PushDictation | Choice | 0.3780 | ns | 1.000 | ns
PushRecording | Choice | 0.2610 | ns | 1.000 | ns

Tests DK Coder 3

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.366 | ns | 1 | ns
Control | PushRecording | 0.832 | ns | 1 | ns
PushDictation | PushRecording | 0.628 | ns | 1 | ns
Control | Choice | 0.398 | ns | 1 | ns
PushDictation | Choice | 1.000 | ns | 1 | ns
PushRecording | Choice | 0.669 | ns | 1 | ns

Tests valid Coder 1

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.797 | ns | 1 | ns
Control | PushRecording | 0.735 | ns | 1 | ns
PushDictation | PushRecording | 1.000 | ns | 1 | ns
Control | Choice | 0.984 | ns | 1 | ns
PushDictation | Choice | 0.909 | ns | 1 | ns
PushRecording | Choice | 0.844 | ns | 1 | ns

Tests valid Coder 2

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.4270 | ns | 1.000 | ns
Control | PushRecording | 0.0759 | ns | 0.455 | ns
PushDictation | PushRecording | 0.3900 | ns | 1.000 | ns
Control | Choice | 0.3050 | ns | 1.000 | ns
PushDictation | Choice | 0.9190 | ns | 1.000 | ns
PushRecording | Choice | 0.5370 | ns | 1.000 | ns

Tests valid Coder 3

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.975 | ns | 1 | ns
Control | PushRecording | 0.550 | ns | 1 | ns
PushDictation | PushRecording | 0.656 | ns | 1 | ns
Control | Choice | 0.787 | ns | 1 | ns
PushDictation | Choice | 0.907 | ns | 1 | ns
PushRecording | Choice | 0.839 | ns | 1 | ns

# For valid answers only from now on

## Abbreviations

### Proportions per group

Abbreviations, using the codes of Coder 1

Coder 1 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
Abbreviation(s) | 3.4 | 1.5 | 1.4 | 2.0
None | 96.6 | 98.5 | 98.6 | 98.0
Number of observations | 208 | 204 | 211 | 203

Abbreviations, using the codes of Coder 2

Coder 2 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
Abbreviation(s) | 1.4 | 0.5 | 1.4 | 3.0
None | 98.6 | 99.5 | 98.6 | 97.0
Number of observations | 208 | 204 | 211 | 203

### Tests of significance across groups

Tests abbreviations Coder 1

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.326 | ns | 1 | ns
Control | PushRecording | 0.568 | ns | 1 | ns
PushDictation | PushRecording | 0.959 | ns | 1 | ns
Control | Choice | 0.353 | ns | 1 | ns
PushDictation | Choice | 1.000 | ns | 1 | ns
PushRecording | Choice | 0.995 | ns | 1 | ns

Tests abbreviations Coder 2

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 1.000 | ns | 1.000 | ns
Control | PushRecording | 0.477 | ns | 1.000 | ns
PushDictation | PushRecording | 0.464 | ns | 1.000 | ns
Control | Choice | 0.629 | ns | 1.000 | ns
PushDictation | Choice | 0.639 | ns | 1.000 | ns
PushRecording | Choice | 0.126 | ns | 0.754 | ns

## Problems detected

### Proportions per group

Problems, using the codes of Coder 1

Coder 1 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
Incomplete sentence(s) | 1.0 | 1.0 | 2.4 | NA
No problem detected | 97.6 | 98.5 | 95.3 | 100.0
Other problem(s) | 1.4 | 0.5 | 2.4 | NA
Number of observations | 208 | 204 | 211 | 203

Problems, using the codes of Coder 2

Coder 2 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
Incomplete sentence(s) | 0.5 | 0.5 | 0.5 | NA
No problem detected | 99.5 | 99.5 | 99.5 | 100.0
Number of observations | 208 | 204 | 211 | 203

### Tests of significance across groups

Tests problems Coder 1: no problem detected

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.30600 | ns | 0.7450 | ns
Control | PushRecording | 0.07630 | ns | 0.3810 | ns
PushDictation | PushRecording | 0.00481 | ** | 0.0288 |
Control | Choice | 0.74200 | ns | 0.7450 | ns
PushDictation | Choice | 0.10300 | ns | 0.4130 | ns
PushRecording | Choice | 0.24800 | ns | 0.7450 | ns

Tests problems Coder 2: no problem detected

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 1 | ns | 1 | ns
Control | PushRecording | 1 | ns | 1 | ns
PushDictation | PushRecording | 1 | ns | 1 | ns
Control | Choice | 1 | ns | 1 | ns
PushDictation | Choice | 1 | ns | 1 | ns
PushRecording | Choice | 1 | ns | 1 | ns

Tests problems Coder 1: Incomplete sentence(s)

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.457 | ns | 1.000 | ns
Control | PushRecording | 0.489 | ns | 1.000 | ns
PushDictation | PushRecording | 0.079 | ns | 0.474 | ns
Control | Choice | 1.000 | ns | 1.000 | ns
PushDictation | Choice | 0.473 | ns | 1.000 | ns
PushRecording | Choice | 0.481 | ns | 1.000 | ns

Tests problems Coder 2: Incomplete sentence(s)

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 1 | ns | 1 | ns
Control | PushRecording | 1 | ns | 1 | ns
PushDictation | PushRecording | 1 | ns | 1 | ns
Control | Choice | 1 | ns | 1 | ns
PushDictation | Choice | 1 | ns | 1 | ns
PushRecording | Choice | 1 | ns | 1 | ns

Tests problems Coder 1: Other problem(s)

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.736 | ns | 1.000 | ns
Control | PushRecording | 0.255 | ns | 1.000 | ns
PushDictation | PushRecording | 0.079 | ns | 0.474 | ns
Control | Choice | 0.629 | ns | 1.000 | ns
PushDictation | Choice | 0.233 | ns | 1.000 | ns
PushRecording | Choice | 1.000 | ns | 1.000 | ns

Tests problems Coder 2: Other problem(s)

We do not have any “Other problem(s)” for this coder.

## Number of themes

### Proportions per group

Number of themes, using the codes of Coder 1: average and median

Coder 1 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
mean | 1.67 | 1.58 | 1.68 | 1.54
median | 1.00 | 1.00 | 1.00 | 1.00

Number of themes, using the codes of Coder 2: average and median

Coder 2 | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
mean | 1.98 | 1.93 | 2.09 | 1.88
median | 2.00 | 2.00 | 2.00 | 2.00

Tests differences average number of themes Coder 1

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.894 | ns | 1 | ns
Control | PushRecording | 0.270 | ns | 1 | ns
PushDictation | PushRecording | 0.217 | ns | 1 | ns
Control | Choice | 0.472 | ns | 1 | ns
PushDictation | Choice | 0.396 | ns | 1 | ns
PushRecording | Choice | 0.706 | ns | 1 | ns

Tests differences average number of themes Coder 2

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.3090 | ns | 1.000 | ns
Control | PushRecording | 0.3820 | ns | 1.000 | ns
PushDictation | PushRecording | 0.0592 | ns | 0.355 | ns
Control | Choice | 0.6310 | ns | 1.000 | ns
PushDictation | Choice | 0.1350 | ns | 0.677 | ns
PushRecording | Choice | 0.6940 | ns | 1.000 | ns

## Number of characters

Number of characters: average and median

No. Char | Control | Choice | PushDictation | PushRecording
--- | --- | --- | --- | ---
mean | 90 | 88 | 93 | 85
median | 70 | 69 | 72 | 64

Tests differences average number of characters

group1 | group2 | p | p.signif | p.adj | p.adj.signif
--- | --- | --- | --- | --- | ---
Control | PushDictation | 0.787 | ns | 1 | ns
Control | PushRecording | 0.486 | ns | 1 | ns
PushDictation | PushRecording | 0.332 | ns | 1 | ns
Control | Choice | 0.728 | ns | 1 | ns
PushDictation | Choice | 0.535 | ns | 1 | ns
PushRecording | Choice | 0.729 | ns | 1 | ns