## Numerical Differentiation |

In this section we will numerically find \(f'(x)\) evaluated at \(x=x_0\). We need numerical techniques for derivatives when
\(f'(x)\) has a complicated expression or \(f(x)\) is not explicitly given. By the limit definition of derivative,
\[f'(x_0)=\lim_{h\to 0}\frac{f(x_0+h)-f(x_0)}{h}.\]
So when \(h>0\) is small, we have
\[f'(x_0)\approx \frac{f(x_0+h)-f(x_0)}{h},\]
which is called the ** two-point forward difference formula** (FDF). Similarly the ** two-point backward difference formula**
(BDF) is
\[f'(x_0)\approx \frac{f(x_0)-f(x_0-h)}{h}.\]
Taking the average of the FDF and BDF, we get the ** two-point centered difference formula** (CDF) is
\[f'(x_0)\approx \frac{f(x_0+h)-f(x_0-h)}{2h}.\]

Note that CDF gives better accuracy than FDF and BDF (explained later). But CDF does not work if \(f(x)\) is not known in one side of \(x_0\). All the difference formulas suffer from round-off errors when \(h\) is too small.

**Example.**
\(f(x)=x^2e^{x}\). Approximate \(f'(1)\) using the FDF, BDF, and CDF with \(h=0.2\).

* Solution.*
\[\begin{align*}
\text{Two-point FDF}:\; f'(1) &\approx \frac{f(1+0.2)-f(1)}{0.2}=\frac{4.78-2.71}{0.2}=10.35 \\
\text{Two-point BDF}:\; f'(1) &\approx \frac{f(1)-f(1-0.2)}{0.2}=\frac{2.71-1.42}{0.2}=6.45\\
\text{Two-point CDF}:\; f'(1) &\approx \frac{f(1+0.2)-f(1-0.2)}{2(0.2)}= \frac{4.78-1.42}{0.4}=8.4
\end{align*}\]
Analytically we know \(f'(1)=3e\). So the absolute errors are \(|10.35-3e|=2.19\), \(|6.45-3e|=1.7\), and \(|8.4-3e|=0.24\)
respectively. So CDF gives the least error.

** Errors in finite difference formulas:**
The maximum error in FDF is
\[\frac{h}{2} \displaystyle\max_{x\in (x_0,x_0+h)}|f''(x)|.\]

By the Taylor's theorem on \(f\) about \(x_0\), we get
\[\begin{align*}
f(x)&= f(x_0)+\frac{f'(x_0)}{1!}(x-x_0)+
\frac{f''(\xi)}{2!}(x-x_0)^2.
\end{align*}\]
Plugging \(x=x_0+h\), we get
\[\begin{align*}
f(x_0+h)&= f(x_0)+\frac{f'(x_0)}{1!}h+
\frac{f''(\xi_1)}{2!}h^2,
\end{align*}\]
for some \(\xi_1\in (x_0,x_0+h)\). Solving for \(f'(x_0)\), we get
\[f'(x_0)= \frac{f(x_0+h)-f(x_0)}{h}-\frac{f''(\xi_1)}{2}h.\]

So the error in FDF is \(O(h)\) (i.e., absolute error \(\leq ch\) for some \(c>0\)). It means small step size \(h\)
results in more accurate derivative. We say FDF is * first-order accurate*. Similarly BDF is also first-order
accurate with the maximum error
\[\frac{h}{2} \displaystyle\max_{x\in (x_0-h,x_0)}|f''(x)|.\]
The maximum error in CDF is
\[\frac{h^2}{6} \displaystyle\max_{x\in (x_0-h,x_0+h)}|f'''(x)|.\]
So the error in CDF is \(O(h^2)\), i.e., CDF is second-order accurate which is better than first-order accurate as
\(h^2<< h\) for small \(h >0\).

For CDF, note that
\[\begin{align*}
f(x_0+h)&= f(x_0)+\frac{f'(x_0)}{1!}h+
\frac{f''(x_0)}{2!}h^2+\frac{f'''(\xi_1)}{3!}h^3,\\
f(x_0-h)&= f(x_0)-\frac{f'(x_0)}{1!}h+
\frac{f''(x_0)}{2!}h^2-\frac{f'''(\xi_2)}{3!}h^3,
\end{align*}\]
for some \(\xi_1\in (x_0,x_0+h)\) and \(\xi_2\in (x_0-h,x_0)\). Subtracting we get,
\[\begin{align*}
f(x_0+h)-f(x_0-h)&= 2f'(x_0)h+\frac{f'''(\xi_1)+f'''(\xi_2)}{6}h^3\\
\frac{f(x_0+h)-f(x_0-h)}{2h}&= f'(x_0)+\frac{f'''(\xi_1)+f'''(\xi_2)}{12}h^2\\
f'(x_0) &= \frac{f(x_0+h)-f(x_0-h)}{2h}-\frac{f'''(\xi_1)+f'''(\xi_2)}{12}h^2.
\end{align*}\]
Assuming continuity of \(f'''\) and using the IVT on \(f'''\), we get
\[f'''(\xi)=\frac{f'''(\xi_1)+f'''(\xi_2)}{2},\]
for some \(\xi\in (\xi_2,\xi_1)\subset (x_0-h,x_0+h)\). Thus
\[f'(x_0)= \frac{f(x_0+h)-f(x_0-h)}{2h}-\frac{f'''(\xi)}{6}h^2.\]

**Example.**
Consider \(f(x)=x^2e^{x}\). Find the maximum error in approximating \(f'(1)\) by the FDF, BDF, and CDF with \(h=0.2\).

* Solution.* \(f''(x)=(x^2+4x+2)e^{x}\) and \(f'''(x)=(x^2+6x+6)e^{x}\) are increasing functions for \(x>0\).
So \(\max_{x\in (1,1.2)}|f''(x)|=|f''(1.2)|=27.3\).
\[\begin{align*}
\text{Maximum error in two-point FDF}:\;&\frac{0.2}{2}\max_{x\in (1,1.2)}|f''(x)|=0.1|f''(1.2)|=2.73 \\
\text{Maximum error in two-point BDF}:\;& \frac{0.2}{2}\max_{x\in (0.8,1)}|f''(x)|=0.1|f''(1)|=1.9\\
\text{Maximum error in two-point CDF}:\;& \frac{(0.2)^2}{6}\max_{x\in (0.8,1.2)}|f'''(x)|=\frac{0.04}{6}|f'''(1.2)|=0.32\\
\end{align*}\]

** Derivative from Lagrange polynomial:** If \(f\) is not explicitly given but we know \((x_i,f(x_i))\) for
\(i=0,1,\ldots,n\), then \(f\) is approximated by the Lagrange polynomial:
\[f(x)=\sum_{i=0}^n f(x_i)L_i(x)+\frac{f^{(n+1)}(\xi(x))}{(n+1)!} \prod_{i=0}^n (x-x_i),\]
where \(\xi\in (x_0,x_n)\) and \(L_i(x)=\displaystyle\prod_{\substack{j=0\\j\neq i}}^n\frac{(x-x_j)}{(x_i-x_j)}\).
Differentiating both sides and evaluating at \(x=x_j\), we get
(steps skipped but note that
\(\frac{d}{dx} \prod_{i=0}^n (x-x_i)\Big]_{x=x_j}=\prod_{\substack{i=0\\i\neq j}}^n (x_j-x_i)\))
\[f'(x_j)=\sum_{i=0}^n f(x_i)L_i'(x_j)+\frac{f^{(n+1)}(\xi)}{(n+1)!} \prod_{\substack{i=0\\i\neq j}}^n (x_j-x_i).\]
If the points \(x_0,x_1,\ldots,x_n\) are equally-spaced, i.e., \(x_j=x_0+jh\), then we get
\[f'(x_j)=\sum_{i=0}^n f(x_i)L_i'(x_j)+\frac{f^{(n+1)}(\xi)}{(n+1)!} O(h^n). \]
It can be verified that two-point FDF and BDF are obtained from the preceding equation using \(n=1\). Similarly \(n=2\) gives the three-point FDF
and BDF and two-point CDF whose errors are \(O(h^2)\):
\[\begin{align*}
\text{Three-point FDF}:\; f'(x_0) &\approx\frac{-3f(x_0)+4f(x_0+h)-f(x_0+2h)}{2h}\\
\text{Three-point BDF}:\; f'(x_0) &\approx\frac{3f(x_0)-4f(x_0-h)+f(x_0-2h)}{2h} \\
\end{align*}\]

**Example.**
From the following table approximate \(f'(1)\) by the three-point FDF and BDF.
\[\begin{array}{|c|c|c|c|c|c|}
\hline
x & 0.6 & 0.8 & 1 & 1.2 & 1.4\\
\hline
f(x) & 0.65 & 1.42 & 2.71 & 4.78 & 7.94\\
\hline
\end{array}\]
* Solution.* Here \(h=0.2\).
\[\begin{align*}
\text{Three-point FDF}:\; f'(1) &\approx\frac{-3f(1)+4f(1+0.2)-f(1+2(0.2))}{2(0.2)}=7.62\\
\text{Three-point BDF}:\; f'(1) &\approx\frac{3f(1)-4f(1-0.2)+f(1-2(0.2))}{2(0.2)} =7.75
\end{align*}\]
Note that the table is given for \(f(x)=x^2e^x\). So \(f'(1)=3e\). Then the absolute errors are \(|7.62-3e|=0.53\)
and \(|7.75-3e|=0.4\) respectively. Notice that three-point FDF and BDF give less error than two-point FDF and BDF
respectively.

Last edited