Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Integrated Business FoundationBUS B200F Lee Shau Kee School of Business and Administration Module 9 Decision Making Skills Simple Linear Regression Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Lecture Outline Regression and Correlation Least Squares Regression Equation Regression in Excel Regression Assumptions Inferences about the Slope and Correlation Coefficients Estimation of Mean Values and Prediction of Individual Values Remarks for Regression 2 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression What is Regression Analysis Regression Analysis is used to Predict the value of a Dependent Variable (Response, Y) based on the value of at least one Independent Variable (Predictor, Xi) Explain the impact of changes in Independent Variable (Predictor, Xi) an on the Dependent Variable (Response, Y) Simple Linear Regression Only one independent variable, X Relationship between X and Y is described by a linear function Changes in the continuous variable Y are assumed to be related to changes in X 3 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Dependent and Independent Variables Dependent variable: the variable we wish to predict or explain Independent variable: the variable (or factor) used to explain the dependent variable Examples: Dependent variable Y Independent variable X Exam score Test score Sales of ice-cream Outdoor temperature Project completion time Number of workers 4 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Check for Linear Relationship To observe if there is any linear relationship between variables X and Y, we need to take n pairs of observations, (x1,y1), …, (xn,yn), then either Plot the data set into a scatter plot (of Y against X) to show the relationship between the two variables Perform a correlation analysis to measure the strength of the linear association between the two variables 2222 YYnXXn YXXYn r 5 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Types of Relationships between Variables 6 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Types of Relationships between Variables 7 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Types of Relationships between Variables 8 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Correlation Coefficient (r ) Measures the strength of linear relationship of two variables Sample correlation coefficient, r where and sX and sY are the sample standard deviation of X and Y respectively YYXX XY SS S r 2 2 22 1 XXX sn n X XXXS 2 2 22 1 YYY sn n Y YYYS n YX XYYYXXSXY 9 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Range of r Values range from -1 to +1, i.e. -1 ≤ r ≤ 1 r > 0 Positively linear relationship between X and Y, i.e. X Y r < 0 Negatively linear relationship between X and Y, i.e. X Y or X Y r = 0 No linear relationship between X and Y r closes to +1 r closes to -1 r closes to 0 10 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Simple Linear Regression Model iii εXββY 10 Linear component Population Y intercept Population Slope Coefficient Random Error term Dependent Variable Independent Variable Random Error component 11 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Random Error and Regression Line 12 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Simple Linear Regression Equation (Prediction Line) The simple linear regression equation provides an estimate of the population regression line ii XbbY 10 Estimate of the regression intercept Estimate of the regression slope Estimated (or predicted) Y value for observation i Value of X for observation i 13 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Least Squares Estimation (LSE) A method to find the regression equation By minimizing the sum of squared errors (or residuals) between the actual y values and predicted y values: n i ii n i ii XbbYYY 1 2 10 1 2 min min 14 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Least Squares Regression Coefficients Least squares estimate for slope coefficient: Interpretation: the estimated change in the mean value of Y when X is increased by one unit Least squares estimate for intercept coefficient: Interpretations: the estimated mean value of Y when X is zero; or the part of Y that is not affected by X 221 XXn YXXYn S S b XX XY XbY n X b n Y b 110 15 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent A real estate agent wishes to examine the relationship between the selling price of a home (in USD1,000) and its size (in square feet) in Arizona, U.S.A. She sampled 10 houses in the state at random and the results are as follows. a) Construct a scatter plot between House Price and Square Feet. b) Find the least square regression line and add it on to the scatter plot. c) Interpret the slope and intercept coefficients. d) Find the correlation coefficient and interpret its meaning. e) Predict the price of a house of size 2000 square feet. Square Feet (X) 1400 1600 1700 1875 1100 1550 2350 2450 1425 1700 House Price (Y) 245 312 279 308 199 219 405 324 319 255 16 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Answer) Answer: a) 0 50 100 150 200 250 300 350 400 450 0 500 1000 1500 2000 2500 3000 H o u s e P ri c e ( $ 1 0 0 0 s ) Square Feet 17 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Answer) Answer: b) From the data set, we have n = 10, x = 17150, x2 = 30983750, y = 2865, y2 = 853423, xy = 5085975, The least square regression line is given as 1715 X 5.286 Y 2483.9817151098.05.28610 XbYb XY 1098.02483.98 1571500 10 17150 30983750 22 2 n X XSXX 172500 10 286517150 5085975 n YX XYSXY 1098.0 1571500 172500 1 XX XY S S b 18 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Answer) Answer: b) 0 50 100 150 200 250 300 350 400 450 0 500 1000 1500 2000 2500 3000 Square Feet H o u s e P ri c e ( $ 1 0 0 0 s ) feet) (square 0.1098 98.2483 price house Slope = 0.10977 Intercept = 98.2483 19 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Answer) Answer: c) There is fixed cost of USD98,248.33 (= b0*USD1,000) which is independent to the size of the square feet. (Note: it is meaningless to interpret as “the average price of house of size 0 square foot is USD98,248.33.) For each extra square foot of the house, it is expected to cost USD109.8 (= b1*USD1,000) more for the house price on average. d) There is a relatively strong positive linear relationship between House Price and Square Feet. e) The predicted price for a house of 2000 square feet is USD317,784. 7621.0 5.326001571500 172500 YYXX XY SS S r 7838.31720001098.02483.98 Y 20 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Linear Regression in Excel Activate the ‘Data Analysis’ add-in (click File/Option / Add-Ins / Go, select ‘Analysis ToolPak’, and then click OK) 21 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Linear Regression in Excel Input the data of Exercise 1.1 into two adjacent columns in an Excel file and then click Data/Data Analysis/ Regression, specify the ‘Input Y Range’ (B1:B8) and the ‘Input X Range’ (A1:A8), and click ‘Labels’) $B$1:$B$11 $A$1:$A$11 22 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Regression Output in Excel Regression Statistics Multiple R 0.76211 R Square 0.58082 Adjusted R Square 0.52842 Standard Error 41.33032 Observations 10 ANOVA df SS MS F Significance F Regression 1 18934.9348 18934.9348 11.0848 0.01039 Residual 8 13665.5652 1708.1957 Total 9 32600.5000 Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Intercept 98.24833 58.03348 1.69296 0.12892 -35.57720 232.07386 Square Feet 0.10977 0.03297 3.32938 0.01039 0.03374 0.18580 The regression equation is: feet) (square 0.10978 98.2483 price house 23 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Regression Assumptions Linearity The relationship between X and Y is linear Independence of Errors Error values are statistically independent Normality of Error Error values are normally distributed for any given value of X Equal Variance (also called homoscedasticity) The probability distribution of the errors has constant variance 24 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Residual Residual (ei ) Observed Error Estimation of the random error Sum of Squared Errors (SSE) Least Squares Estimation Choose the coefficients b0 and b1 such that SSE is minimized i iiii YYe XX XY YYii S S SYYeSSE 2 2 2 25 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Standard Error of Estimate Mean Square Error Point estimate of the constant variance of errors Standard error of the estimate, s iVar 2 2 22 n SSE MSEs 2 n SSE MSEs 26 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Standard Error of the Estimate (cont’d) s (standard error of the estimate) is a measure of the variation of the observed Y values from the regression line The magnitude of s should always be judged relative to the size of the Y values in the sample data 27 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Measures of Variation Total variations is made up of two parts where: = Mean value of the dependent variable Yi = Observed value of the dependent variable = Predicted value of Y for the given Xi valueiY Y 21 YYY snSSST Total Sum of Squares 2)( YYSST i Total Variation SST Error Sum of Squares 2) ( ii YYSSE Unexplained Variation SSE Regression Sum of Squares 2) ( YYSSR i Explained Variation SSR 28 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Measures of Variation (cont’d) 29 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Coefficient of Determination, r2 The coefficient of determination is the portion of the total variation in the dependent variable that is explained by variation in the independent variable The coefficient of determination is also called r-squared and is denoted as r2 Values range from 0 to 1, i.e. 0 ≤ r2 ≤ 1 r2 = 1 Perfect match between the regression line and the data points r2 = 0 No linear relationship between X and Y SST SSE SST SSR r 12 30 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Coefficient of Determination, r2 (cont’d) r2 = 1 Perfect linear relationship Between X and Y; 100% of the variation in Y is explained by variation in X. 0 ≤ r2 ≤ 1 Weaker linear relationships between X and Y; Some but not all of the variation in Y is explained by variation in X. r2 = 0 No linear relationship between X and Y; The value of Y does not depend on X. (None of the vari
ation in Y is explained. 31 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) A real estate agent wishes to examine the relationship between the selling price of a home (in USD1,000) and its size (in square feet) in Arizona, U.S.A. She sampled 10 houses in the state at random and the results are as follows. f) Find and interpret the standard error of the estimate. g) Find and interpret the coefficient of determination. h) Based on the results in parts (f) and (g), comment on whether the prediction made by this regression model. Square Feet (X) 1400 1600 1700 1875 1100 1550 2350 2450 1425 1700 House Price (Y) 245 312 279 308 199 219 405 324 319 255 32 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Answer) Answer: (Revisit) From the data set, we have n = 10, x = 17150, x2 = 30983750, y = 2865, y2 = 853423, xy = 5085975, The least square regression line is given as 1715 X 5.286 Y 2483.9817151098.05.28610 XbYb XY 1098.02483.98 1571500 10 17150 30983750 22 2 n X XSXX 172500 10 286517150 5085975 n YX XYSXY 1098.0 1571500 172500 1 XX XY S S b 33 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) f) The standard error of the estimate is moderately small relative to house prices in the $200K – $400K range. g) 58.08% of the variation in house prices is explained by the variation in square feet. 5652.13665 1571500 172500 5.32600 22 XX XY YY S S SSSE 3303.41 210 5652.13665 2 n SSE MSEs 3303.41 210 5652.13665 2 n SSE MSEs 5808.0 5.32600 5652.13665 1112 YYS SSE SST SSE r 34 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) h) With a small standard error of estimate and a moderately strong positive relationship between the two variables, the regression model is useful in predicting Y. 35 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Regression Output in Computer Regression Statistics Multiple R 0.76211 R Square 0.58082 Adjusted R Square 0.52842 Standard Error 41.33032 Observations 10 ANOVA df SS MS F Significance F Regression 1 18934.9348 18934.9348 11.0848 0.01039 Residual 8 13665.5652 1708.1957 Total 9 32600.5000 Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Intercept 98.24833 58.03348 1.69296 0.12892 -35.57720 232.07386 Square Feet 0.10977 0.03297 3.32938 0.01039 0.03374 0.18580 36 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Inferences about the Slope The standard error of the estimate for the slope coefficient of regression line is: where Confidence Interval for population slope β1 d.f. = n – 2 XX b S s s 1 2 n SSE MSEs XXS s tb 21 37 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Inferences about the Slope t-test for the population slope β1 Hypotheses vs Test Statistic where where is the hypothetical value d.f. = n – 2 For , reject H0 if or if p-value ≤ α XXb obs Ss b s b t 0 1 0 1111 0110 : H 0111 : H 0111 : H 0111 : H 01 ttobs ttobs 22 or tttt obsobs 0111 : H 0111 : H 0111 : H 38 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Inferences about the Slope (cont’d) To test the significance of a linear regression model, one can set up a test of: (no linear relationship) vs (linear relationship exists) This test is included in regression output of Excel and SPSS (and many other statistical software) If the regression assumption of linearity holds, we should reject at the significance level α 0: 10 H 0: 11 H 0: 10 H 39 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) A real estate agent wishes to examine the relationship between the selling price of a home (in USD1,000) and its size (in square feet) in Arizona, U.S.A. She sampled 10 houses in the state at random and the results are as follows. i) Construct a 95% confidence interval for the slope coefficient. j) Based on the interval in i), suggest whether the regression model is significant. k) At α = 0.05, test whether every 20 extra square feet of a house can induce an average increase of the house price by USD800 or more. Square Feet (X) 1400 1600 1700 1875 1100 1550 2350 2450 1425 1700 House Price (Y) 245 312 279 308 199 219 405 324 319 255 40 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Answer) Answer: (Revisit) From the data set, we have n = 10, x = 17150, x2 = 30983750, y = 2865, y2 = 853423, xy = 5085975, The least square regression line is given as 1715 X 5.286 Y 2483.9817151098.05.28610 XbYb XY 1098.02483.98 1571500 10 17150 30983750 22 2 n X XSXX 172500 10 286517150 5085975 n YX XYSXY 1098.0 1571500 172500 1 XX XY S S b 41 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) i) A 95% confidence interval for β1 We are 95% confident that the average impact on house price is between USD33.7 and USD185.8 per square foot of house size. 1858.0 ,0337.0 1571500 3303.41 306.21098.0 21 XXS s tb 1b s1b 1 01 bs b p-value 42 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) j) The confidence interval for β1 remains above zero that gives enough evidence to reject H0: β1 = 0 and to conclude the significance of the regression model. k) It is required to test H0: 20β1 = 0.8 (=USD800/USD1000) vs H1: 20β1 > 0.8 or equivalently H0: β1 = 0.04 vs H1: β1 > 0.04 Test statistic There is sufficient evidence that every 20 extra square feet can induce an average increase of the house price by USD800 or more. 0 8;05.0 11 Reject 860.11161.2 15715003303.41 04.01098.0 0 H t Ss b t XX obs 43 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Inference about Correlation Coefficient Hypotheses to test the population correlation coefficient ρ H0: ρ = 0 (no correlation between X and Y) H1: ρ ≠ 0 (correlation exists) Test statistic where it takes n-2 degrees of freedom 2 1 2 0 n r ρr tobs 0if 0if 1 2 1 2 b rr b rr 44 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) A real estate agent wishes to examine the relationship between the selling price of a home (in USD1,000) and its size (in square feet) in Arizona, U.S.A. She sampled 10 houses in the state at random and the results are as follows. l) At α = 0.05, test ρ for a linear relationship between square feet and house price. m) Construct and interpret a 95% confidence interval for the mean price of houses of size 2000 square feet. n) Construct and interpret a 95% prediction interval for the price of a house of size 2000 square feet. Square Feet (X) 1400 1600 1700 1875 1100 1550 2350 2450 1425 1700 House Price (Y) 245 312 279 308 199 219 405 324 319 255 45 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) l) H0: ρ = 0 vs H1: ρ ≠ 0 Test statistic There is sufficient evidence of a linear association at the 5% level of significance. 0 025.0 22 0 Reject 306.23294.3 210 7621.01 07621.0 2 1 H t n r ρr tobs 46 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression What is predicted Before using the regression model, we need to assess how well the model fits the data If we are satisfied with how well the model fits the data, we can use it to predict… Population mean response E(Y | X=xp) Individual response Y | X=xp 47 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression What is predicted The point on the regression line corresponding to a particular value of xp of the independent variable x is It is unlikely that this value will equal the mean value of y when X equals xp, i.e. E(Y | X=xp) Therefore, we need to place bounds on how far the predicted value might be from the actual value We can do this by calculating a confidence interval mean for the value of Y, E(Y | X=xp), and a prediction interval for an individual value of Y | X=xp 010 xbbYi 48 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Confidence Interval and Prediction Interval Both the confidence interval for the mean value of Y and the prediction interval for an individual value of Y employ a quantity called the distance value The distance value for a particular value xp of X is The distance value is a measure of the distance between the value xp of X and Notice that the farther xp from , the larger the distance valueX XX p S xX n 2 1 49 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Confidence Interval and Prediction Interval Goal: to form intervals around Y to express the uncertainty about the value of Y for a given xp 50 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Confidence Interval for E(Y | X=xp) A (1 – )*100% confidence interval for the population mean response E(Y | X=xp) is given as: A (1 – )*100% prediction interval for an individual response Y | X=xp is given as: XX p S xX n stYstY 2 22 1 valueDistance XX p S xX n stYstY 2 22 1 1 valueDistance1 This extra term adds to the interval width to reflect the added uncertainty for an individual case 51 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) m) A 95% confidence interval for the mean price of houses of size 2000 square feet is We have 95% confident that the mean price of houses of size 2000 square feet falls between USD280,664 to USD354,903. 9032.354 ,6644.280 1571500 20001715 10 1 3303.41306.27838.317 1 valueDistance 2 2 22 XX p S xX n stYstY 7838.31720001098.02483.98 Y 52 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Example: Real Estate Agent (Revisit) n) A 95% prediction interval for the price of a house of size 2000 square feet is We have 95% confident that the price of a house of size 2000 square feet falls between USD215,503 to USD420,065. 20.06494 ,5027.215 1571500 20001715 10 1 13303.41306.27838.317 1 1 valueDistance1 2 2 22 XX p S xX n stYstY 7838.31720001098.02483.98 Y 53 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Remark 1: Causation A statistically significant regression of Y on X need not imply a causal relationship between the two variables A non-significant linear regression need not imply the lack of a causal relationship if the causal relationship is non-linear 54 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Remark 2: Small Samples Significant regressions can be obtained by chance, i.e. even when no (linear) causal relationship exists This is especially true if sample sizes are small 55 Integrated Business FoundationBUS B200F DM Module | Simple Linear Regression Remark 3: Extrapolation and Interpolation Be careful when Predictions lie outside range of sample (Extrapolation) Be careful when predictions are for values where data are sparse (Interpolation) 56