Regression

Introduction

Often time we measure some observable $\phi$ as a function of some independent variable, $x$, and we have some theoretical function that we expect the observations to obey. The function may have parameters in it that we need to determine. The process of determining the best values of the parameters that fit the function to the given data is known as regression.

Example

Consider the Arrhenius equation commonly used to describe the rate constant in chemistry,

$k = A \exp \left(-\frac{E_a}{RT} \right)$

where

• $k$ is the rate constant of the chemical reaction,
• $A$ is the pre-exponential factor
• $E_a$ is the activation energy,
• $R$ is the gas constant
• $T$ is the temperature.

If we measured $k$ as a function of $T$ we could determine $A$ and $E_a$ using regression.

Linear Least Squares Regression

Linear Least Squares Regression can be applied if we can write our function in polynomial form, $y(x)= a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^m$. If we can do this, then we are looking for the coefficients of the polynomial, $a_j$ given $n$ observations $(x_i,y_i)$. We can write this as

$\begin{align} y_1 &= a_0 + a_1 x_1 + a_2 x_1^2 + \cdots + a_m x_1^m \\ y_2 &= a_0 + a_1 x_2 + a_2 x_2^2 + \cdots + a_m x_2^m \\ \vdots &= \vdots \\ y_n &= a_0 + a_1 x_n + a_2 x_n^2 + \cdots + a_m x_n^m \\ \end{align}$

or, in matrix form,

$\underbrace{ \left[ \begin{array}{ccccc} 1 & x_1 & x_1^2 & \cdots & x_1^m \\ 1 & x_2 & x_2^2 & \cdots & x_2^m \\ \vdots & \vdots & \cdots & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^m \end{array} \right] }_{A} \underbrace{ \left( \begin{array}{c} a_0 \\ a_1 \\ a_2 \\ \vdots \\ a_m \end{array}\right) }_{\phi} = \underbrace{ \left( \begin{array}{c} y_1 \\ y_2 \\ \vdots \\ y_n \end{array}\right) }_{b}$

Regression applies when $n>m$. In other words, when we have many more equations than we have unknowns. In this case, the least-squares solution is written as

$A^\mathsf{T}A \phi = A^\mathsf{T} b$

Example

Let's illustrate linear least squares regression with our example above. First, we must rewrite the problem:

$\begin{align} k &= A \exp \left(-\frac{E_a}{RT} \right) \\ \ln k &= \ln A - \frac{E_a}{RT} \end{align}$

Note that this now looks like a polynomial, $y=a_0+a_1 x$, where $y \equiv \ln k$, $a_0 \equiv \ln A$, $a_1 \equiv E_a$, and $x \equiv -1/(RT)$.

If we write these equations for each of our $n$ data points in matrix form we obtain

$\underbrace{ \left[ \begin{array}{cc} 1 & \frac{-1}{RT_1} \\ 1 & \frac{-1}{RT_2} \\ 1 & \frac{-1}{RT_3} \\ \vdots & \vdots \\ 1 & \frac{-1}{RT_n} \end{array} \right] }_{A} \underbrace{ \left(\begin{array}{c} a_0 \\ a_1 \end{array}\right) }_{\phi} \underbrace{ \left(\begin{array}{c} \ln k_1 \\ \ln k_2 \\ \ln k_3 \\ \vdots \\ \ln k_n \end{array}\right) }_{b}$

and the equations to be solved are

$A^\mathsf{T}A\phi=A^\mathsf{T}b$

Note that since $A$ is a 2x$n$ matrix and $b$ is a $n$x1 vector, we have a 2x2 system of equations to solve (see notes on matrix multiplication).

Example 2

Let's consider another example, where we want to determine the value of the gravitational constant, $g$, via regression. To do this, we could drop an object from a set height, $h_0$ at time $t=0$ and measure its height as a function of time. Neglecting air resistance, we have

$h=h_0 + v_0 t + \tfrac{1}{2}gt^2$

where $g=-9.8$ m/s² is the gravitational constant. Assume that $h_0=10$ meters is known precisely, and that since we drop the ball from rest, $v_0=0$ m/s. The data we collect is given in the table below:

Height as a function of time

time (s)	0.08	0.28	0.48	0.53	0.70	1.05	1.31	1.36
h(m)	9.90	9.58	8.86	8.76	7.58	4.71	1.47	1.01

Let's go through this problem by hand:

1. First, we rearrange our equation as follows:

   $h-h_0 = \tfrac{1}{2}gt^2$

   now we are left with an equation of the form $y=ax$.
2. Write down the equations in matrix form, one equation for each of the $n$ data points.

   $\underbrace{ \left[\begin{array}{c} \tfrac{1}{2}t_1^2 \\ \tfrac{1}{2}t_2^2 \\ \vdots \\ \tfrac{1}{2}t_n^2 \end{array} \right] }_{A} \underbrace{ \left( g \right) }_{\phi} = \underbrace{ \left( \begin{array}{c} h_1-h_0 \\ h_2-h_0 \\ \vdots \\ h_n-h_0 \end{array} \right) }_{b}$

3. Forming the normal equations, we have $A^\mathsf{T}A\phi = A^\mathsf{T} b$,

   $\left[\begin{array}{cccc} \tfrac{1}{2}t_1^2 & \tfrac{1}{2}t_2^2 & \cdots & \tfrac{1}{2}t_n^2 \end{array} \right] \left[\begin{array}{c} \tfrac{1}{2}t_1^2 \\ \tfrac{1}{2}t_2^2 \\ \vdots \\ \tfrac{1}{2}t_n^2 \end{array} \right] \left( g \right) = \left[\begin{array}{cccc} \tfrac{1}{2}t_1^2 & \tfrac{1}{2}t_2^2 & \cdots & \tfrac{1}{2}t_n^2 \end{array} \right] \left( \begin{array}{c} h_1-h_0 \\ h_2-h_0 \\ \vdots \\ h_n-h_0 \end{array} \right)$

4. Using the rules of matrix multiplication, we can rewrite this as

   $\sum_{i=1}^n \left(\tfrac{1}{2} t_i^2 \right)^2 \cdot g = \sum_{i=1}^n \left( \tfrac{1}{2}t_i^2 \right) \left(h_i-h_0\right)$

   This may be easily solved for g

   $g = 2\frac{ \sum_{i=1}^n t_i^2 \left( h_i-h_0 \right) }{ \sum_{i=1}^n t_i^4 }$

5. Substituting numbers in from our above table, we obtain g=-9.75 m/s²

Converting a Nonlinear Problem Into a Linear One

This section is a stub and needs to be expanded. If you can provide information or finish this section you're welcome to do so and then remove this message afterwards.

The R-Squared Value

The $R^2$ value is a common way to measure how well a predicted value matches the observed data. It is calculated as

$R^2 = 1 - \frac{\sum_{i=1}^{n} \left( y_{i}^{observed} - y_i^{predicted} \right)^2} {\sum_{i=1}^{n} \left( y_i^{observed} - \bar{y} \right)^2 },$

where $y_{i}^{predicted}$ is a predicted value of $y$, $y_i^{observed}$ is the observed value, and $\bar{y}=\frac{1}{n}\sum_{i=1}^n y_i^{observed}$ is the average value of the observed value of $y$.

The $R^2$ value is always less than unity, $R^2 < 1$.

Nonlinear Least Squares Regression

The discussion and examples above focused on linear least squares regression, where the problem could be recast as a polynomial and we were solving for the coefficients of that polynomial. We showed an example of the Arrhenius equation which was originally nonlinear in the parameters but we rearranged it to be solved using linear regression.

Occasionally, it is not possible to reduce a problem to a linear one in order to perform regression. In such cases, we must use nonlinear regression, which involves solving a system of nonlinear equations for the unknown parameters.

Derivation of the Linear Least Squares Equations

This section is a stub and needs to be expanded. If you can provide information or finish this section you're welcome to do so and then remove this message afterwards.