Statistical methods used in comparative analysis of pre- and post-data
Posted: Wed Dec 11, 2024 5:45 am
answer
Since we are comparing means, we will use a t-test. Excel has a convenient function called T.TEST , so it is a good idea to use that. The smaller the value that comes out (p-value), the more significant it is . In the above example morocco phone number resource it was below 0.01, so we can conclude with 99% significance that IMP has increased . (See sample data file)
supplement
The concept of the t-test is that the average and standard deviation of groups A and B are the basis of the test. Therefore, the less the fluctuation in the data and the larger the N number, the higher the significance tends to be. In short, it answers the question, "If we assume that these are normally distributed, at what percentage of the extremes do they overlap?" In the above example, it would be "to say that the conclusion is reversed only at the 0.57% extreme."
One thing to note when using this method is that it only compares the average values statistically. Therefore, if there are multiple factors that have influenced the increase in IMP, it is naturally difficult to evaluate using this method alone.
② Proportion test (chi-square test)
What can be verified?
Is there a statistically significant difference between the ratios of group A and group B?
Example Problem
Starting on Friday, December 11th, we implemented a certain measure to increase click-through rates. The click-through rates are higher pre- and post-post, but can we say that this is statistically significant?
Sample Data
Chi-Square Test.xlsx
answer
Since we are comparing ratios, we will use the chi-square test. As with the t-test, Excel has a convenient function called CHISQ.DIST.RT, so it is a good idea to use that. The smaller the value (p-value) that is obtained, the more significant it is . In the above example, it was below 0.01, so we can conclude with 99% significance that the click-through rate is higher .
supplement
The concept of the chi-square test is that the numerator, denominator, and ratio of group A and group B are the basis of the test. Therefore, the larger the N number, the higher the significance tends to be.
In addition, the following statistical processing is performed. (We will refrain from going into detail, but will list this for reference only.)
The test is conducted by comparing the expected frequency (the number of occurrences if it were the overall proportion) with the original data to see how much difference there is.
We are testing whether the two groups are independent when we assume that these follow a chi-square distribution.
The t-test is two-sided, whereas the chi-square test is one-sided (right-sided).
One thing to note is that, like with t-tests, if there are multiple factors that influenced the increase in CTR, it will naturally be difficult to evaluate using this method alone.
Since we are comparing means, we will use a t-test. Excel has a convenient function called T.TEST , so it is a good idea to use that. The smaller the value that comes out (p-value), the more significant it is . In the above example morocco phone number resource it was below 0.01, so we can conclude with 99% significance that IMP has increased . (See sample data file)
supplement
The concept of the t-test is that the average and standard deviation of groups A and B are the basis of the test. Therefore, the less the fluctuation in the data and the larger the N number, the higher the significance tends to be. In short, it answers the question, "If we assume that these are normally distributed, at what percentage of the extremes do they overlap?" In the above example, it would be "to say that the conclusion is reversed only at the 0.57% extreme."
One thing to note when using this method is that it only compares the average values statistically. Therefore, if there are multiple factors that have influenced the increase in IMP, it is naturally difficult to evaluate using this method alone.
② Proportion test (chi-square test)
What can be verified?
Is there a statistically significant difference between the ratios of group A and group B?
Example Problem
Starting on Friday, December 11th, we implemented a certain measure to increase click-through rates. The click-through rates are higher pre- and post-post, but can we say that this is statistically significant?
Sample Data
Chi-Square Test.xlsx
answer
Since we are comparing ratios, we will use the chi-square test. As with the t-test, Excel has a convenient function called CHISQ.DIST.RT, so it is a good idea to use that. The smaller the value (p-value) that is obtained, the more significant it is . In the above example, it was below 0.01, so we can conclude with 99% significance that the click-through rate is higher .
supplement
The concept of the chi-square test is that the numerator, denominator, and ratio of group A and group B are the basis of the test. Therefore, the larger the N number, the higher the significance tends to be.
In addition, the following statistical processing is performed. (We will refrain from going into detail, but will list this for reference only.)
The test is conducted by comparing the expected frequency (the number of occurrences if it were the overall proportion) with the original data to see how much difference there is.
We are testing whether the two groups are independent when we assume that these follow a chi-square distribution.
The t-test is two-sided, whereas the chi-square test is one-sided (right-sided).
One thing to note is that, like with t-tests, if there are multiple factors that influenced the increase in CTR, it will naturally be difficult to evaluate using this method alone.