1FAU Erlangen-Nürnberg, 2HelloBetter Institut für Gesundheitstrainings GmbH/HelloBetter
Aims: This individual patient data (IPD) meta-analysis aims to evaluate the overall effectiveness of the internet-based intervention “HelloBetter Stress” in individuals with elevated stress symptoms.
Methods: We included individual patient data of \(k = 6\) randomized controlled studies examining the effects of “HelloBetter Stress” compared to a control condition (\(N_{total} = 1239\)). Only patients with elevated stress symptoms as measured by the 10 item-version of the perceived stress scale (PSS-10 \(\geq\) 22) and depressive symptoms (CES-D \(\geq\) 16) were included. One-step IPD meta-analysis procedures were used to obtain pooled effect estimates. Primary outcome was the perceived stress (PSS-10) at post-test (6-7 weeks). Secondary outcomes included results on depression, as well as effects on burnout symptoms, work engagement and quality of life.
Results: At post-test, a significant within-group (\(d\) = -1.28; 95 %CI: -1.37 – -1.19) and between-group effect (\(d\) = -0.82; 95 %CI: -0.96 – -0.69) favoring the intervention was found. More than twice as many participants in the intervention groups (\(n\) = 394, 60.06%) achieved reliable response at post-test compared to controls (\(n\) = 135, 23.16 %). Significant effects on depressive symptoms at both post-test and follow-up (\(d_{within}\) = -0.69 (post), -0.83 (follow-up); \(d_{between}\) = -0.50 (post), -0.62 (follow-up)) and burnout symptoms (\(d_{within}\) = -0.81 (post), -0.98 (follow-up); \(d_{between}\) = -0.64 (post), -0.73 (follow-up)) as well as work engagement (\(d_{within}\) = 0.14 (post), 0.19 (follow-up); \(d_{between}\) = 0.24 (post), 0.29 (follow-up)) were obtained. Quality of life increased significantly at follow-up (\(d_{within}\) = 0.29; \(d_{between}\) = 0.25). Subgroup analysis showed that effects on stress symptoms did not vary significantly between different guidance formats used in the primary studies.
No negative intervention effects on stress symptoms could be detected.
Discussion: Results of this IPD meta-analysis indicate that “HelloBetter Stress” can effectively reduce symptoms of stress, burnout and depression, as well as increase work engagement.
Trial Registration:DRKS00005687, DRKS00005112, DRKS00004749, DRKS00005384, DRKS00005990, DRKS00005699.
In this individual patient data (IPD) meta-analysis, we aim to evaluate the effectiveness the internet-based intervention “HelloBetter Stress” in patients with adjustment disorder symptoms. We hypothesized the intervention to be superior to control groups in terms of effects on stress symptoms, as well as the proportion of individuals achieving reliable response with respect to their stress symptomatology.
The trials investigated in this study have been registered in the German clinical trials register ( DRKS00005687, DRKS00005112, DRKS00004749, DRKS00005384, DRKS00005990, DRKS00005699). We present the methods and results of this secondary analysis in accordance with the CONSORT Statement (Moher et al., 2010), and the Guidelines for Executing and Reporting Research on Internet Interventions (Proudfoot et al., 2011). The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Moher et al., 2009) were followed only were applicable since data used in this study is not based on a comprehensive literature search. The material used for the analyses in this study has been made openly available in an Open Science Framework (OSF) repository DOI 10.17605/OSF.IO/J9MU5.
A total of \(k = 6\) primary studies were included in this meta-analysis. The trials will be referenced as stress_3arm_406, stress_fod_263, stress_gui_264, stress_sh_264, stress_ua_263 and stress_up_396 throughout this report. Included studies are previously conducted randomized controlled trials evaluating the intervention “HelloBetter Stress” (research title: “GET.ON Stress”).
The included studies are two- or three-armed randomized controlled trials. The intervention groups (IG; stress_3arm_406: \(n\) = 270; stress_fod_263: \(n\) = 132; stress_gui_264: \(n\) = 132; stress_sh_264: \(n\) = 132; stress_ua_263: \(n\) = 131; stress_up_396: \(n\) = 198; \(n_{total}\) = 995) participated in the internet-based intervention “HelloBetter Stress”. The control groups were waiting list control conditions (CG; stress_3arm_406: \(n\) = 136; stress_fod_263: \(n\) = 131; stress_gui_264: \(n\) = 132; stress_sh_264: \(n\) = 132; stress_ua_263: \(n\) = 132; stress_up_396: \(n\) = 198; \(n_{total}\) = 861).
The primary studies had different guidance formats, that were unguided (stress_up_396, stress_sh_264), minimal guidance (adherence focused feedback on demand)(stress_fod_263) and full guidance (adherence focused guidance after each session)(stress_gui_264). The study stress_3arm_406 compared the unguided (\(n\) = 135) and minimal guided (\(n\) = 135) “HelloBetter Stress”.
All intervention arms had full access to treatment as usual (TAU). Participants in all primary studies were assessed at baseline (T0), post-treatment (7 weeks, T1), and at 6-months follow-up (T2).
Only individuals with elevated symptoms of stress on the 10 item-version of the perceived stress scale (PSS-10 \(\geq\) 22) and depressive symptoms (CES-D \(\geq\) 16) were considered for this analysis. The cut-off on the perceived stress scale was chosen to select participants with a heightened level of subjective stress as identified by one standard deviation (SD = 6.2) above the mean (\(\hat\mu_{PSS-10}\) = 15.3) in a large working population (Lesage, Berjot & Deschamps, 2012). Depressive symptoms were evaluated with the short German version of the Center for Epidemiological Studies’ Depression Scale (CES-D).
The primary study stress_3arm_406 originally included \(N\) = 406 participants recruited via a primary prevention and occupational health management program of a large health insurance company. For inclusion participants had to state elevated symptoms of perceived stress (PSS-10 \(\geq\) 22). Participants received the unguided, minimal guided or full guided internet-based stress management intervention “HelloBetter Stress” or a waiting list control condition. Studies stress_fod_263, stress_gui_264 and stress_sh_264 originally included \(N\) = 263, \(N\) = 264 and \(N\) = 264 participants, respectively. The recruitment also took place in the general working population, recruited primarily via a large German health insurance company. To be eligible for inclusion participants had to meet the same criteria as in the primary study stress_3arm_406. Participants received an internet-based stress management intervention or waiting list control condition.
The studies stress_ua_263 and stress_up_396 originally included \(N\) = 263 and \(N\) = 396 participants, respectively. The recruitment proceeded as previously described for the other trials. Study stress_ua_263 participants included employees that were exposed to unfavorable working conditions in the form of high demands and low gratification in their work activity. To select these participants the Effort-Reward Imbalance (ERI) questionnaire (Sigrist, Li, Montano, 2014; Lehr, Koch, Hillert, 2009)) was used, only those with an ERI > 0.715 were included. Lastly, the study stress_up_396 was open to all employees in routine working conditions according to universal prevention principles. Participants also received an internet-based stress management intervention or waiting list control condition.
Further eligibility criteria applied in all included primary studies were (i) being at least 18 years old, (ii) having Internet access and (iii) provision of informed consent, (iv) employment, (v) no suicidal tendency at baseline (BDI-II item 9 < 1) and (vi) no diagnosed psychosis or dissociative symptoms in the past. There were no restrictions on maximum age or gender.
Randomization in all primary studies was conducted using an automated computer-based random integer generator (Randlist, Datinf GmbH, Tübingen, Germany). Randomization was carried out by a researcher who was not otherwise involved in the study. During the randomization procedure, allocation was concealed from participants, recruitment staff, diagnosticians, and e-coaches.
The six primary studies all evaluated the effectiveness of “HelloBetter Stress”. The iSMI consists of seven sessions and one booster session, that all include problem-solving and emotion regulation techniques. The intervention was developed based on the Lazarus and Folkman transactional model of stress and the distinction of problem-focused and emotion-focused coping. The intervention was developed using evidence-based material on problem-solving and emotion regulation (Heber, Ebert & Lehr 2013).
Participants in the waiting list control condition got full access to complete the training after a waiting period of six month in each of the studies included. Furthermore, these participants had access to any kind of interventions offered by workplace occupational health management programs and routine mental health services (i.e., treatment as usual).
The main outcome was perceived stress as measured by the 10 item-version of the perceived stress scale (PSS-10) at post-test (6-7 weeks after baseline). Cronbach’s alphas for the PSS-10 range from .78 to .91. The scale fits the theoretical basis of these interventions well, as it is based on Lazarus’ transactional model of stress (Cohens & Janicki-Deverts, 2012).
Secondary outcomes included depressive symptoms, as well as the Maslach Burnout Inventory (MBI) and the Utrecht Work Engagement Scale (UWES).
To allow for joint analyses, depressive symptom scores were transformed to common metrics, using the common metrics model developed by Wahl et al. (2014). A common metrics is an Item Response Theory model, e.g. a Graded Response Model (GRM) or Generalized Partial Credit Model (GPCM), that comprises parameters of items from different measures of a common latent variable (in this case depressive symptoms). The item parameters describe the relation between item response and the latent variable. These statistical models allow estimating the latent variable by a subset of items, e.g. if different questionnaires were used or if missing data occurs.
We used common metrics to assess effects on depressive symptom severity and to select participants that met the inclusion criteria (CES-D \(\geq\) 16) since the CES-D was not used in all primary studies.
Secondary outcomes were measured at baseline (T0), post-treatment (7 weeks, T1), and at 6-months follow-up (T2) in all included primary studies.
Quality of life was assessed with the german version of the EuroQol questionnaire (Graf et al., 1998) in all studies, apart from stress_3arm_406, at baseline (T0) and at 6-months follow-up (T2) and was therefor analyzed in a subset of the five remaining studies (\(N\) = 917).
Client satisfaction with the intervention was assessed using the Client Satisfaction Questionnaire (adapted to the online context; CSQ-8; Boß et al., 2016; Nguyen, Attkisson, & Stegner, 1983; 8 items; assessed at post-test in the IGs only)
To evaluate the effectiveness of the intervention in comparison to the CG, analyses based on the intention-to-treat (ITT) principle were conducted. R version 3.6.2 (R Core Team, 2013) was used for data-analyses.
A joint modeling, multilevel-multiple imputation by chained equations (MICE) model was used to impute missing data (Jolani et al., 2015; Schafer & Yucel, 2002). Trial affiliation was used as a level-2 variable in the imputation model to account for the nested data structure (patients-in-trials). All subsequent analyses were conducted in the \(m\) = 50 multiply imputed data sets. Test statistics and parameter estimates were calculated using Rubin’s rule (Barnard & Rubin, 1999).
We tested if the intervention was superior the control condition in terms of (i) effects on participants perceived stress (PSS) and additional outcomes from baseline to post-test (T1), and from baseline to three-month follow-up (T2). We also compared the proportion of participants with (ii) reliable response and (iii) reliable symptom deterioration between the IG and CG at T1 and T2. A significance level of 0.05 (two-sided) was used for all analyses.
Differences in effects between the two study conditions across the included primary studies were assessed using one-step IPD meta-analysis methods. We used linear mixed-effects models which included (1) a random study intercept and random group slope, as well as (2) a fixed-effect term controlling for baseline symptom severity to determine the overall intervention effects.
To determine if the perceived stress symptoms of patients had reliably decreased, participants were coded as responders or non-responders using the Reliable Change Index (RCI; Jacobson & Truax, 1991). The proportions of reliable responders in the IGs and CGs at post-test and follow-up were compared using \(\chi^2\)-tests. Using the RCI, we also determined potential negative effects, defined as cases with a reliable symptom deterioration. Differences in deterioration cases between groups were also compared using \(\chi^2\)-tests.
Subgroup analyses were conducted to compare the effects between different guidance formats on the primary outcome perceived stress (PSS-10). A linear mixed model including the guidance format as a level-2 predictor was compared to a linear mixed model without, using Anova. The proportions of reliable respondents in each guidance format were compared using \(\chi^2\)-tests.
Lastly, we used descriptive statistics to analyze the intervention satisfaction reported by IG patients.
The study flow is depicted in Figure 1 . In the active CG, follow-up data from 20 participants (3%) at post-test, and another 67 (11%) after 12-24 months could not be obtained. In the IG, 77 (12%) and 141 (21%) participants were lost to follow-up at T2 and T3, respectively.
Figure 1. Flowchart
The mean age of patients in the analyzed sample was \(m\) = 42.23 (\(SD\) = 9.78). Descriptive data for all continuous outcomes at the three assessment points is shown in Table 1.
A total of \(n\) = 940 participants was female (76.22%; IG: 77.38%; CG: 74.92%). A total of \(n\) = 1011 (81.60%) participants identified themselves as white/caucasian (IG: 80.34%; CG: 83.02%).
Most participants were in the income category 2 (\(n\) = 325, 26,23%; IG: 27.13%; CG: 25.21%), which equals a yearly income of 10.000 - 30.000.
In line with the recruiting approach, the vast majority of participants were employees (99.19%; IG: 99.09%; CG: 99.31%). The overall education level in the analyzed sample was high, with \(n\) = 530 (42.78%; IG: 43.45%; CG: 42.02%) receiving at least some post-high school education. The minority of participants (\(n\) = 525, 42.37 %; IG: 39.02%; CG: 46.14%) were married or in a committed relationship. A total of \(n\) = 628 (50.69 %; IG: 48.63%; CG: 53.00%) reported that they have children.
The minority of participants (\(n\) = 558; 45.11%) reported that they had previous experiences with psychotherapy (IG: 45.63%; CG: 44.65%) or health trainings (\(n\) = 159; 12.85%, IG: 10.85%; CG: 15.09%).
Table 1. Descriptive data for continuous outcomes at all assessment points, based on multiple imputation.
A total of \(N\) = 1239 patients matched the inclusion criteria (CES-D \(\geq\) 16, PSS-10 \(\geq\) 22). There were \(n\) = 583 patients in the control groups and \(n\) = 656 patients in the intervention groups.
Variable names in this analysis are in accordance to the Trial Warehouse guidelines. The documentation can be accessed here. If you need access rights, ask Mathias.
The distribution of stress symptoms across groups and assessment points is visualized in Figure 2.
Figure 2. Perceived Stress in the intervention and control groups at all analyzed assessment points
Table 2. Effects on stress symptoms.
There was a significant between-group effect at post-test (\(d\) = -0.82; 95%CI: -0.96 – -0.69; \(p\) < 0.001) and follow-up (\(d\) = -0.85; 95%CI: -1.00 – -0.71; \(p\) < 0.001). The heterogeneity of the intervention effects between studies was \({\tau^{2}}_{slope}\) = 0.42 at post-test and \({\tau^{2}}_{slope}\) = 0.50 at follow-up. The within-group effect sizes in the control group were \(d\) = -0.44 (95%CI: -0.53 – -0.36; \(p\) < 0.001; post) and \(d\) = -0.56 (95%CI: -0.65 – -0.47; \(p\) < 0.001; follow-up). The within-group effect sizes in the intervention group were \(d\) = -1.28 (95%CI: -1.37 – -1.19; \(p\); post) and \(d\) = -1.41 (95%CI: -1.51 – -1.32; \(p\) < 0.001; follow-up).
More Participants in the intervention group were coded as reliable responders according to the RCI then in the control group for perceived stress at post-test, as well as follow-up. (IG: \(n\) = 394, 60.06% (post-test), \(n\) = 442, 67.38% (follow-up), CG: \(n\) = 135, 23.16% (post-test), \(n\) = 157, 26.93% (follow-up)). Less participants in the intervention group had a reliable deterioration than in the control group at post-test and follow-up (IG: \(n\) = 4, 0.61% (post-test), \(n\) = 5, 0.76% (follow-up); CG: \(n\) = 15, 2.57% (post-test), \(n\) = 13, 2.23% (follow-up)). There was a significant overall difference in RCI status between the intervention and control group both at post-test and follow-up (\({D_2}\) = 80.91; \(p\) < 0.001 (post-test), \({D_2}\) = 82.26 ; \(p\) < 0.001 (follow-up)).
Table 3. Comparison of the RCI Status in control and intervention group for the primary outcome perceived stress.
Table 4. \(\chi^2\)-test for RCI differences between the treatment groups regarding the primary outcome perceived stress.
Fig. 3 Proportion of reliable responders in the intervention and control groups at post-test and follow-up regarding perceived Stress
The mean depressive symptoms according to the Common Metrics were \(m\) = 65.84 at baseline. Common metrics are standardized to have a general population mean at \(\mu\) = 50, with a standard deviation of \(\sigma\) = 10. This means the participants were more than 1.5 SD above the general population mean at baseline. At post-test and follow-up, depressive symptoms in the intervention group are reduced to \(m\) = 59.13 and 57.66, respectively, which is within the 1 SD range for the general population. This was not the case for the control group (63.80, resp. 63.24). The Maslach Burnout Inventory (MBI) had a mean of \(m\) = 4.85 at baseline. In the intervention group scores were reduced to \(m\) = 4.11 at post-treatment and 3.87 at follow-up. In the control group scores were reduced to \(m\) = 4.71 at post-treatment and 4.60 at follow-up. The Utrecht Work Engagement Scale (UWES) had a mean of \(m\) = 26.72 at baseline. In the intervention group scores increased to \(m\) = 28.15 at post-treatment and 28.68 at follow-up, while they were reduced to \(m\) = 25.56 at post-treatment and 25.71 at follow-up in the control group. Note that concerning the UWES higher scores indicate higher work engagement and therefore a better outcome.
We found a significant between-group effect on symptoms of depression at post-test (\(d\) = -0.50; 95%CI: -0.60 – -0.41; \(p\) < 0.001) and follow-up (\(d\) = -0.62; 95%CI: -0.77 – -0.46; \(p\) < 0.001). Within-group intervention effects on depression were \(d\) = -0.69 (95%CI:-0.76 – -0.61; \(p\) < 0.001; post) and \(d\) = -0.83 (95%CI: -0.91 – -0.75; \(p\) < 0.001; follow-up). Comprehensive results are displayed in Table 5.
Table 5. Effects on depressive symptoms.
More participants in the intervention group were coded as reliable responders according to the RCI than in the control group regarding the depression severity (IG: \(n\) = 278, 42.38% (post-test), \(n\) = 332, 50.61% (follow-up); CG: \(n\) = 107, 18.35% (post-test), \(n\) = 130, 22.30% (follow-up)). Less participants in the intervention group had a reliable deterioration (IG: \(n\) = 11, 1.68% (post-test), \(n\) = 10, 1.52% (follow-up); CG: \(n\) = 37, 6.35% (post-test), \(n\) = 34, 5.83% (follow-up)). The overall difference in the RCI status between the intervention and the control group was significant at post-test and follow-up (\({D_2}\) = 45.21; \(p\) < 0.001 (post-test), \({D_2}\) = 49.53 ; \(p\) < 0.001 (follow-up))
Table 6. Comparison of the RCI Status in control and intervention group for the secondary outcome depression severity.
Table 7. \(\chi^2\)-test for RCI differences between the treatment groups regarding the secondary outcome depression severity.
Results indicated a significant between-group effect on burnout symptoms (MBI) at post-test (\(d\) = -0.64; 95%CI: -0.78 – -0.51; \(p\) < 0.001) and follow-up (\(d\) = -0.73; 95%CI: -0.86 – -0.60; \(p\) < 0.001). Within-group intervention effects on depression were \(d\) = -0.81 (95%CI: -0.91 – -0.71; \(p\) < 0.001; post) and \(d\) = -0.98 (95%CI: -1.08 – -0.88; \(p\) < 0.001; follow-up).
Table 8. Effects on the burnout symptoms.
Results indicated a significant between-group effect on work engagement (UWES) at post-test (\(d\) = 0.24; 95%CI: 0.13 – 0.35; \(p\) < 0.001) and follow-up (\(d\) = 0.29; 95%CI: 0.17 – 0.40; \(p\) < 0.001). Within-group intervention effects on work engagement were \(d\) = 0.14 (95%CI: 0.03 – 0.25; \(p\) = 0.015; post) and \(d\) = 0.19 (95%CI: 0.07 – 0.30; \(p\) = 0.002; follow-up).
Table 9. Effects on work engagement.
Results indicated a significant between-group effect on the quality of life (EQOL) at follow-up (\(d\) = 0.25; 95%CI: 0.04 – 0.46; \(p\) = 0.021). Within-group intervention effects on quality of life were in the intervention group \(d\) = 0.29 (95%CI: 0.14 – 0.44; \(p\) < 0.001; follow-up).
Table 10. Effects on quality of life.
Our analyses did not show any negative side-effects of the intervention. Reliable deterioration (defined via RCI) of stress symptoms in the intervention groups was very rare (Post: \(n\) = 4, 0.61%; Follow-up: \(n\) = 5, 0.76%) did not exceed the deterioration rates in the control group (2.57% (post-test) and 2.23% (follow-up)).
For depression reliable deterioration also rarely occurred in the intervention group (Post: \(n\) = 11, 1.68%; Follow-up: \(n\) = 10, 1.52%) and was therefore below the deterioration rates in the control group (6.35% (post-test) and 5.83% (follow-up)).
Subgroup analyses showed that between group effectsizes for all three guidance formats, did not differ significantly. Neither the feedback-on-demand (\(p\) = 0.132), nor the full-guidance format were superior to the unguided format, we used as a reference in the mixed model (\(p\) = 0.919).
Anova testing showed that the linear mixed model including guidance format as a level-2 predictor was not significantly better then the simpler model (\(p\) = 0.269).
Further subgroup analysis of the reliable change index showed superiority of the intervention over the control condition on perceived stress, regardless of the guidance format at post-test and follow-up (unguided: \({D_2}\) = 40.75; \(p\) < 0.001 (post-test), \({D_2}\) = 36.56; \(p\) < 0.001 (follow-up), feedback-on-demand: \({D_2}\) = 22.29; \(p\) < 0.001 (post-test), \({D_2}\) = 19.45; \(p\) < 0.001 (follow-up), full-guidance: \({D_2}\) = 16.69; \(p\) < 0.001 (post-test), \({D_2}\) = 22.81; \(p\) < 0.001 (follow-up).
In addition, the subgroup analysis revealed that there was no significant difference between the guidance format in the intervention groups in terms of the RCI, neither at post-test nor at follow-up (post-test: \({D_2}\) = 1.363; \(p\) = 0.26, follow-up: \({D_2}\) = 2.76; \(p\) = 0.06.
Intervention satisfaction data was available for \(n\) = 880 patients. Mean ratings were all above 3 (3 = “Somewhat agree”, 4 = “Totally agree”). Agreement (somewhat or totally) to the quality statements ranged between 85.80% (satisfied with extent of help) and 95.23% (high quality of help). Results are summarized in Table 11.
Table 11. Intervention satisfaction.
This IPD meta-analysis investigated the pooled effectiveness of the internet-based intervention “HelloBetter Stress”, on perceived stress and additional outcomes in patients with elevated stress symptoms.
Individual patient data of \(k\) = 6 randomized-controlled trials was combined in this analysis. We found a greater reduction in stress symptoms compared to control groups at post-test and follow-up.
The moderate between-group effect sizes were -0.81 at post test and -0.85 at follow-up.
The majority of participants in the IGs achieved reliable response in stress symptoms both at post-test (60.06%) and follow-up (67.38%), more than twice as many as in the control groups.
Reliable deterioration was rare in stress and depressive symptoms and particularly uncommon in the intervention groups.Therefore, no negative side effects of the intervention could be detected.
Apart from stress, we also found that the intervention was effective on the three additional outcomes, depression, burnout, work engagement and quality of life. Effects on the primary outcome did not vary significantly between different guidance formats used in the primary studies. Overall, the intervention was well accepted.
The large majority (55%) of included patients did have no prior experience with psychotherapy, although participants showed mild to moderate depressive symptoms (CESD \(\geq\) 16) and an elevated stress level (PSS-10 \(\geq\) 22) at baseline.
In sum, results of this IPD meta-analysis indicate that “HelloBetter Stress” can be an effective treatment for patients with elevated stress, and that these effects can be sustained up to 24 weeks.
Barnard, J., & Rubin, D. B. (1999). Miscellanea. Small-sample degrees of freedom with multiple imputation. Biometrika, 86(4), 948-955. https://doi.org/10.1093/biomet/86.4.948
Boß, L., Lehr, D., Reis, D., Vis, C., Riper, H., Berking, M., & Ebert, D. D. (2016). Reliability and validity of assessing user satisfaction with web-based health interventions. Journal of medical Internet research, 18(8), e234. https://doi.org/10.2196/jmir.5952
Cohen, S., & Janicki‐Deverts, D. (2012). Who’s stressed? Distributions of psychological stress in the United States in probability samples from 1983, 2006, and 2009 1. Journal of applied social psychology, 42(6), 1320-1334. https://doi.org/10.1111/j.1559-1816.2012.00900.x
Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of health and social behavior, 385-396. https://doi.org/10.2307/2136404
Graf, J. M., Claes, C., Greiner, W., & Uber, A. (1998). Die deutsche Version des EuroQol-Fragebogens. Zeitschrift für Gesundheitswissenschaften= Journal of public health, 6(1), 3-20.
Heber, E., Ebert, D. D., Lehr, D., Nobis, S., Berking, M., & Riper, H. (2013). Efficacy and cost-effectiveness of a web-based and mobile stress-management intervention for employees: design of a randomized controlled trial. BMC public health, 13(1), 1-12. https://doi.org/10.1186/1471-2458-13-655
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19. https://doi.org/10.1037/0022-006X.59.1.12
Jolani, S., Debray, T. P., Koffijberg, H., van Buuren, S., & Moons, K. G. (2015). Imputation of systematically missing predictors in an individual participant data meta‐analysis: a generalized approach using MICE. Statistics in medicine, 34(11), 1841-1863. https://doi.org/10.1002/sim.6451
Lehr, D., Koch, S., & Hillert, A. (2010). Where is (im) balance? Necessity and construction of evaluated cut‐off points for effort‐reward imbalance and overcommitment. Journal of Occupational and Organizational Psychology, 83(1), 251-261. https://doi.org/10.1348/096317909X406772
Lesage, F.-X., Berjot, S., & Deschamps, F. (2012). Psychometric properties of the French versions of the Perceived Stress Scale. International journal of occupational medicine and environmental health, 25(2), 178-184. https://doi.org/10.2478/S13382-012-0024-8
Wahl, I., Löwe, B., Bjorner, J. B., Fischer, F., Langs, G., Voderholzer, U., . . . Rose, M. (2014). Standardization of depression measurement: a common metric was developed for 11 self-report depression measures. Journal of clinical epidemiology, 67(1), 73-86. https://doi.org/10.1016/j.jclinepi.2013.04.019
Feiler, M., Schuurmans, L., Stephani, V., Heber, E., Ebert, D. D., & Harrer, M. (2020, November 13). “HelloBetter Stress” In Patients with Adjustment Disorder Symptoms Individual Patient Data Meta-Analysis. https://doi.org/10.17605/OSF.IO/J9MU5