The method of Ordinary Least Squares explained briefly
This method is attributed to Carl Friedrich Gauss. This method has very attractive statistical properties that have made it the most powerful and popular method of regression analysis. To understand this method, we first explain the OLS principle.
Two variable PRF
PRF is not directly observable. We estimate it from SRF:
Yi=Ŷi+ûi where Ŷi =β1-β2Xi
Ŷi is the estimated (conditional mean) value of Yi.
How is the SRF itself determined? To see this let us proceed as follows.
ûi=Yi-(β1-β2Xi) putting the value of Ŷi
The method of Ordinary Least Squares
This shows the ui is simply the difference between the actual value and estimated Y values.
Choose the SRF is such a way that the sum of the residents ∑ûi=∑(Yi-Ŷi) is as small as possible.
∑ûi2=∑(Yi-Ŷi)2 squaring both sides
Minimizing β1, β2
∂∑ûi2 / ∂β1=2∑(Yi- β1-β2Xi)-1
=-2∑ûi where ûi=Yi- β1-β2Xi
∂∑ûi2 / ∂β2=2∑(Yi- β1-β2Xi)-Xi