Regression: lines of regression, regression…

Core facts

Regression literally means "going back" (stepping back towards the average). The word was first used by Sir Francis Galton (heights of fathers and sons).
Use: to estimate or predict the value of a dependent variable for a given value of an independent variable.
Line of regression of Y on X (estimates Y from X): y − ȳ = byx (x − x̄), with byx = r σy / σx.
Line of regression of X on Y (estimates X from Y): x − x̄ = bxy (y − ȳ), with bxy = r σx / σy.
Both lines are fitted by the method of least squares: the sum of squared errors of the dependent variable is made a minimum.
Properties of regression coefficients:
- r = ±√(byx × bxy): r is the geometric mean of the two coefficients.
- byx, bxy and r all have the same sign.
- byx × bxy = r² ≤ 1: if one coefficient is more than 1, the other must be less than 1.
- (byx + bxy)/2 ≥ r when r > 0: the arithmetic mean of the coefficients is at least r.
- Independent of change of origin, but not of scale.
The two lines intersect at (x̄, ȳ).
r = ±1: the two lines coincide. r = 0: the lines are perpendicular (parallel to the axes).
r² = coefficient of determination = share of the variation in y explained by x.
Standard error of estimate of Y on X: Syx = σy √(1 − r²).

Explained

1. Correlation versus regression.

Point	Correlation	Regression
Measures	degree of linear relationship	average relationship (a line)
Variables	treated alike	one dependent, one independent
Value	r is a pure number, −1 to +1	byx, bxy carry units; one can exceed 1
Origin and scale	r unaffected by both	unaffected by origin only
Symmetry	rxy = ryx	byx ≠ bxy in general
Use	how closely they move	estimate one from the other

Neither measure proves cause and effect.

2. Why two lines? The line of Y on X makes the squared vertical errors (errors in y) as small as possible. The line of X on Y makes the squared horizontal errors (errors in x) as small as possible. These are different problems, so unless r = ±1 they give different lines. Always use Y on X to estimate y, and X on Y to estimate x.

3. Normal equations. For y = a + bx, least squares gives:

Σy = na + bΣx
Σxy = aΣx + bΣx²

So byx = [nΣxy − ΣxΣy] / [nΣx² − (Σx)²] and a = ȳ − byx x̄. Swap x and y for X on Y: bxy = [nΣxy − ΣxΣy] / [nΣy² − (Σy)²].

Example 1 (from means, SDs and r). x̄ = 20, ȳ = 30, σx = 4, σy = 6, r = 0.8. Find both lines and estimate y when x = 25. Solution.

byx = 0.8 × 6/4 = 1.2. Line: y − 30 = 1.2(x − 20), so y = 1.2x + 6.
bxy = 0.8 × 4/6 ≈ 0.533. Line: x − 20 = 0.533(y − 30), so x ≈ 0.533y + 4.
At x = 25: y = 1.2 × 25 + 6 = 36.

Answer: y = 1.2x + 6; x ≈ 0.533y + 4; estimate y = 36

Example 2 (from sums). x: 1, 2, 3, 4, 5; y: 2, 4, 5, 4, 5. Solution.

n = 5, Σx = 15, Σy = 20, Σxy = 66, Σx² = 55, Σy² = 86. Means x̄ = 3, ȳ = 4.
nΣxy − ΣxΣy = 330 − 300 = 30. nΣx² − (Σx)² = 275 − 225 = 50. nΣy² − (Σy)² = 430 − 400 = 30.
byx = 30/50 = 0.6; bxy = 30/30 = 1.0.
Y on X: y − 4 = 0.6(x − 3), so y = 2.2 + 0.6x. X on Y: x − 3 = 1.0(y − 4), so x = y − 1.
r = √(0.6 × 1.0) = √0.6 ≈ 0.775 (positive, as both coefficients are positive).

Answer: y = 2.2 + 0.6x; x = y − 1; r ≈ 0.775

4. Identifying which line is which. When two lines are given without labels:

Solve them together: the meeting point is (x̄, ȳ).
Assume the first is Y on X: write it as y = … and read byx. Write the second as x = … and read bxy.
If byx × bxy ≤ 1, the assumption is right. If the product is more than 1, swap the roles.
r = ±√(product), with the common sign of the coefficients.

Example 3. The regression lines are 3x + 2y = 26 and 6x + y = 31. Find x̄, ȳ and r. Solution.

From the second, y = 31 − 6x. Put in the first: 3x + 62 − 12x = 26, so x = 4 and y = 7.
Assume 3x + 2y = 26 is Y on X: y = 13 − 1.5x, so byx = −1.5. Then 6x + y = 31 is X on Y: x = 31/6 − y/6, so bxy = −1/6.
Product = 1.5 × 1/6 = 0.25 ≤ 1, so the assumption is valid. (The other way gives byx = −6, bxy = −2/3, product 4 > 1: impossible.)
r = −√0.25 (negative, since both coefficients are negative).

Answer: x̄ = 4, ȳ = 7, r = −0.5

5. Reading a coefficient in words. If the regression of expenditure (y, ₹) on income (x, ₹) has byx = 0.7, then on average, expenditure rises by ₹0.70 for every ₹1 rise in income. The coefficient is the slope: the change in the dependent variable per one-unit change in the independent variable. It is not a percentage and not a correlation.

Example 4 (r², standard error). σy = 5 and r = 0.6 for the regression of y on x. Find the explained share of variation and the standard error of estimate. Solution. r² = 0.36, so 36% of the variation in y is explained by x and 64% is not. Syx = 5 × √(1 − 0.36) = 5 × 0.8 = 4.

Answer: 36%; Syx = 4

Deeper

1. Change of origin and scale. Let u = (x − A)/h and v = (y − B)/k. Then:

byx = (k/h) × bvu and bxy = (h/k) × buv.
The origins A and B drop out; the scales h and k do not.

Example 5. u = (x − 10)/2, v = (y − 50)/5 and bvu = 0.4. Find byx. Solution. byx = (5/2) × 0.4. Answer: byx = 1.0

2. Angle between the lines. If θ is the acute angle between them:

tan θ = [(1 − r²)/|r|] × [σx σy / (σx² + σy²)]

r = 0: tan θ is infinite, θ = 90°. The lines are perpendicular: y = ȳ and x = x̄.
r = ±1: tan θ = 0, θ = 0°. The lines coincide.
The higher |r|, the smaller the angle, the closer the two lines.

Example 6. r = 0.5 and σx = σy. Find θ. Solution. tan θ = (0.75/0.5) × (σ²/2σ²) = 1.5 × 0.5 = 0.75. Answer: θ ≈ 36.9°

3. Standard error of estimate.

Syx = σy √(1 − r²) for Y on X; Sxy = σx √(1 − r²) for X on Y.
From the fitted line: Syx = √[Σ(y − ŷ)²/n] = √[(Σy² − aΣy − bΣxy)/n].
Syx = 0 when r = ±1 (perfect fit); Syx = σy when r = 0 (x gives no help).
In Example 2: (86 − 2.2 × 20 − 0.6 × 66)/5 = (86 − 44 − 39.6)/5 = 0.48, so Syx = √0.48 ≈ 0.693. The same comes from σy√(1 − r²): σy = √(86/5 − 16) = √1.2, and √1.2 × √0.4 = √0.48.

4. Coefficient of determination.

r² = explained variation ÷ total variation = Σ(ŷ − ȳ)² ÷ Σ(y − ȳ)².
1 − r² is the coefficient of non-determination (unexplained share).
r² is the same whether we regress y on x or x on y.

5. Regression to the mean. Galton found that tall fathers had sons who were tall, but on average less tall than the fathers; short fathers had sons less short. Values move back towards the mean. Since |r| ≤ 1, a deviation of one SD in x predicts a deviation of only r SDs in y (in standard units the slope is r).

6. Which coefficient can exceed 1? byx = r σy/σx can be more than 1 when σy is much larger than σx. But both cannot exceed 1, since their product is r² ≤ 1. Also, byx = bxy only when σx = σy (or r = 0).

Common traps

✗ "byx = r σx/σy." ✓ byx = r σy/σx (σ of the dependent variable on top).
✗ "Use X on Y to estimate y when it is handy." ✓ Estimate y only from Y on X; x only from X on Y.
✗ "byx = 1.6 and bxy = 0.8 are possible." ✓ Product 1.28 > 1, so r² > 1: impossible.
✗ "byx = 0.5 and bxy = −0.4 are possible." ✓ Both coefficients and r must have the same sign.
✗ "r is the arithmetic mean of byx and bxy." ✓ r is the geometric mean; the arithmetic mean is ≥ r.
✗ "Regression coefficients are unaffected by change of scale." ✓ Unaffected by origin only; scale changes them.
✗ "r = 0 means the lines coincide." ✓ r = 0: lines perpendicular; r = ±1: lines coincide.
✗ "The lines meet at the origin." ✓ They meet at the means (x̄, ȳ).

Quick revision

Y on X: y − ȳ = r(σy/σx)(x − x̄). X on Y: x − x̄ = r(σx/σy)(y − ȳ).
byx = [nΣxy − ΣxΣy]/[nΣx² − (Σx)²].
r = ±√(byx · bxy); same sign for all three.
byx · bxy = r² ≤ 1; one > 1 forces the other < 1.
Independent of origin, not scale: byx = (k/h)bvu.
Lines meet at (x̄, ȳ).
r = 0 perpendicular; r = ±1 coincide.
Unlabelled lines: assume, check product ≤ 1, else swap.
r² explained share; Syx = σy√(1 − r²).
Coefficient = average change in y per unit change in x.

Medha AI writes its own notes and checks them against the sources listed. Medha AI is an independent preparation service. It is not affiliated with, endorsed by or connected to APPSC, TGPSC (formerly TSPSC), MPSC, UPSC or any other commission or government body.

5.1.3 Regression: lines of regression, regression coefficients and their properties; correlation versus regression

Download this PDF

Core facts

Explained

Deeper

Common traps

Quick revision