To determine the appropriate sample size for our A/B test focused on improving conversion rates, we first identified the conversion rate as our primary metric and estimated the minimum effect size we considered significant for our business, guided by historical data. We then set our alpha at 0.05 and our power at 0.80, reflecting a 5% chance of a false positive and an 80% chance of detecting a true effect, respectively. Using these parameters, we utilized a sample size calculator, inputting our baseline conversion rate and the desired effect size to compute the necessary sample size. This methodical approach highlighted the importance of statistical parameters in ensuring the reliability and validity of our findings. Moreover, it demonstrated the vital role of historical data in grounding our expectations and refining our sample size estimates.
We use two approaches for determining sample size for a study. If we are looking to sample from a population and make generalizable results back to that population, we use sampling theory calculations (my online favorite: http://www.raosoft.com/samplesize.html) to determine the recommended sample size needed for a particular population size based on the confidence level we need (i.e., 90%, 95%), the margin of error we can accept (i.e., 4%, 5%), and the response distribution (i.e., 50%, 45%). When you survey a sample of the population, you don't know that you've found the correct answer, but you do know that there is a 95% (confidence level you chose) chance that you're within 4% (the margin of error you chose) of the correct answer. The second approach for determining sample size is power calculations (g*power is a great free tool). A power analysis helps us avoid type I and type II errors. It is the calculation used to estimate the smallest sample size needed for an experiment, given a required significance level, statistical power, the analysis or statistical test used, and effect size. It helps to determine if a result from an experiment or survey is due to chance, or if it is a correct result and significant. This is important because testing, experiments, and surveys are expensive to conduct and we do not want to run an experiment and realize afterwards that the sample size was too small to determine if the outcome was genuine or not.
I draw insights from protected electronic health records. In these observational studies, sample size is determined by eligibility criteria defined by medical researchers. For example, you may want to study adults older than 65 who have an illness. You must draw these subjects from your already finite database. Alas, you may not have big data as in textbooks. This is where you must learn what methods are appropriate and emphasize what interpretations are possible.
As CEO of a tech company, conducting a study on user behavior, I faced the challenge of determining the right sample size. Rather than randomly shooting for a number, we identified our diverse users and segregated them into strata, based on their demographics, use rate, and industry nature. Our sample was a balanced mix from these strata, giving a practical and diverse overview. The process taught me two things: representation matters, and accurate sample sizes are more about logical stratification than just numbers.
In order to determine the appropriate sample size for our study on VPNs, we conducted a thorough analysis of 188,000 data points. This extensive dataset allowed us to gain a comprehensive understanding of the various VPN providers available in the market. However, during the process, we realized that we needed more data to ensure the accuracy and reliability of our findings. We recognized the importance of expanding our sample size to include a wider range of VPNs and gather more information on their features, performance, and user reviews. Of note: By increasing our sample size, we were able to create a more robust and comprehensive list of VPNs, providing a more reliable resource for individuals seeking the best VPN option for their needs.