Why sample variance divided by n 1

And that's denoted, usually denoted, by s with a subscript n. And what is the biased estimator, how we calculate it? Well, we would calculate it very similar to how we calculated the variance right over here. But what we would do it for our sample, not our population. So for every data point in our sample --so we have n of them-- we take that data point. And from it, we subtract our sample mean. We subtract our sample mean, square it, and then divide by the number of data points that we have.

But we already talked about it in the last video. How would we find-- what is our best unbiased estimate of the population variance? This is usually what we're trying to get at. We're trying to find an unbiased estimate of the population variance. Well, in the last video, we talked about that, if we want to have an unbiased estimate --and here, in this video, I want to give you a sense of the intuition why. We would take the sum. So we're going to go through every data point in our sample.

We're going to take that data point, subtract from it the sample mean, square that. But instead of dividing by n, we divide by n minus 1. We're dividing by a smaller number. And when you divide by a smaller number, you're going to get a larger value.

So this is going to be larger. This is going to be smaller. And this one, we refer to the unbiased estimate. And this one, we refer to the biased estimate. If people just write this, they're talking about the sample variance. It's a good idea to clarify which one they're talking about. But if you had to guess and people give you no further information, they're probably talking about the unbiased estimate of the variance.

So you'd probably divide by n minus 1. But let's think about why this estimate would be biased and why we might want to have an estimate like that is larger.

And then maybe in the future, we could have a computer program or something that really makes us feel better, that dividing by n minus 1 gives us a better estimate of the true population variance. So let's imagine all the data in a population. And I'm just going to plot them on number a line. So this is my number line. This is my number line. And let me plot all the data points in my population.

So this is some data. This is some data. Here's some data. And here is some data here. And I can just do as many points as I want. So these are just points on the number line. The response consists of some statistical jargon that confuses me more, rather than less.

Some of the responses were very useful, though, so I recommend checking out the replies to the tweet. Based on some of the responses I received, I will try to describe my favorite way of looking at the issue. If you want to follow along in R, you can copy the code from each code section; beginning with some setup code. The variance is a measure of the dispersion around the mean, and in that sense this formula makes sense.

We then divide this sum by the number of observations as a scaling factor. If we ignore this number, we could get a very high variance simply by observing a lot of data. So, to fix that problem, we divide by the total number of observations. However, this is the formula for the population variance.

The formula for calculating the variance of a sample is:. If you Google this question, you will get a variety of answers. It does not actually help me understand 1 the problem and 2 why the solution is the solution that it is. So, below I am going to try to figure it out in a way that actually makes conceptual and intuitive sense to me.

The problem with using the population variance formula to calculate the variance of a sample is that it is biased. It is biased in that it produces an underestimation of the true variance. We simulate a population of data points from a uniform distribution with a range from 1 to Below I show the histogram that represents our population.

The variance is 8. To start, we can draw a single sample of size 5. Say we do that and get the following values: 7, 6, 3, 5, 5. In the former case, this will result in 1. Below I show the results of draws from our population. I simulated drawing samples of size 2 to 10, each different times. We see that the biased measure of variance is indeed biased. Watch this, it precisely answers your question.

There is one constraint which is that the sum of the deviations is zero. I think it's worth pointing out the connection to Bayesian estimation. You want to draw conclusions about the population. The Bayesian approach would be to evaluate the posterior predictive distribution over the sample, which is a generalized Student's T distribution the origin of the T-test.

The generalized Student's T distribution has three parameters and makes use of all three of your statistics. If you decide to throw out some information, you can further approximate your data using a two-parameter normal distribution as described in your question.

From a Bayesian standpoint, you can imagine that uncertainty in the hyperparameters of the model distributions over the mean and variance cause the variance of the posterior predictive to be greater than the population variance. I'm jumping VERY late into this, but would like to offer an answer that is possibly more intuitive than others, albeit incomplete.

The non-bold numeric cells shows the squared difference. My goodness it's getting complicated! I thought the simple answer was You just don't have enough data outside to ensure you get all the data points you need randomly. The n-1 helps expand toward the "real" standard deviation. Sign up to join this community.

The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Learn more.

Ask Question. Asked 11 years ago. Active 10 months ago. Viewed k times. Improve this question. Tal Galili Tal Galili You ask them "why this?

Watch this, it precisely answers you question. Add a comment. Active Oldest Votes. Improve this answer. Michael Lew Michael Lew In essence, the correction is n-1 rather than n-2 etc because the n-1 correction gives results that are very close to what we need. More exact corrections are shown here: en. What if it overestimates? Show 1 more comment. Dror Atariah 2 2 silver badges 15 15 bronze badges. Why is it that the total variance of the population would be the sum of the variance of the sample from the sample mean and the variance of the sample mean itself?

How come we sum the variances? See here for intuition and proof. Show 4 more comments. I have to teach the students with the n-1 correction, so dividing in n alone is not an option. As written before me, to mention the connection to the second moment is not an option. Although to mention how the mean was already estimated thereby leaving us with less "data" for the sd - that's important. Regarding the bias of the sd - I remembered encountering it - thanks for driving that point home.

In other words, I interpreted "intuitive" in your question to mean intuitive to you. Thank you for the vote of confidence :. The loose of the degree of freedom for the estimation of the expectancy is one that I was thinking of using in class. But combining it with some of the other answers given in this thread will be useful to me, and I hope others in the future.

Show 3 more comments.