The method of Ordinary Least Squares

The method of Ordinary Least Squares explained briefly

This method is attributed to Carl Friedrich Gauss. This method has very attractive statistical properties that have made it the most powerful and popular method of regression analysis. To understand this method, we first explain the OLS principle.

Two variable PRF


PRF is not directly observable. We estimate it from SRF:


Yi=Ŷi+ûi         where Ŷi =β12Xi

Ŷi is the estimated (conditional mean) value of Yi.

How is the SRF itself determined? To see this let us proceed as follows.


ûi=Yi-(β12Xi)          putting the value of Ŷi

The method of Ordinary Least Squares

This shows the ui is simply the difference between the actual value and estimated Y values.

Choose the SRF is such a way that the sum of the residents ∑ûi=∑(Yi-Ŷi) is as small as possible.


∑ûi2=∑(Yi-Ŷi)2           squaring both sides


Minimizing    β1, β2


∂∑ûi2 / ∂β1=2∑(Yi- β12Xi)-1

=-2∑((Yi- β12Xi)

=-2∑ûi                  where ûi=Yi- β12Xi

∂∑ûi2 / ∂β2=2∑(Yi- β12Xi)-Xi

=-2∑((Yi- β12Xi)Xi

=-2[∑((YiXi- β1Xi-β2Xi2)]


= 0

Related Articles:

Classical Linear Regression Model

The method of Ordinary Least Squares

OLS Formulation Proving with equations

The OLS Estimators: Properties and formula


%d bloggers like this: