\(\nabla\) Engineering & Mathematics

Introduction

Differential equations are the back bone of practically all engineering and physics problems therefore it is a good idea to have an overview of the subject. This primer in particular focus on ordinary differential equations. Of course there is an impressive amount of literature available that will probably do the job but I like to think that there is still some room for another point of view ;).

Table of Contents

• What is a differential equation?
• Why do we study differential equations?
• Historical notes and other curiosities
• What is a PDE (Partial Differential Equation)?
• What is an ODE (Ordinary Differential Equation)?
• Examples of first and second order ODE
• First order systems
• Newton's law recast as a first order system
• Existence and uniqueness of solution for IVP (initial value problems)
• References

General considerations

Unless otherwise stated the following holds, all functions and variables are real valued, bold variables represent vectorial quantities (\(x\) is a scalar while \({\bf x}\) is a vector), the variable \(t\) represents time, \(x=x(t)\), \( C^k(U, V)\) is the set of functions \(f:U\rightarrow V\) having continuous derivatives up to order \(k\) and time derivative is represented as \begin{equation} x'=\dot{x}=\frac{dx}{dt} \end{equation}

What is a differential equation? [Top]

A differential equation is an equation involving an unknown function, its derivatives and eventual known quantities.

Why do we study differential equations? [Top]

Differential equations are the result of the methods that we use to describe nature's behavior and to predict it we have to solve them. The more we understand about differential equations the better we become in solving them. This until we find a new description method.

Historical notes and other curiosities [Top]

The development of the theory of infinitesimal calculus has been carried out independently by Gottfried Wilhelm von Leibniz and Sir Isaac Newton. The claim of its invention is a very bitter subject which generated an intellectual war between the two mathematicians.

We now know that Newton first developed it (first notes between 1665-1666, first official publication 1687) but it was actually Leibniz who first officially published his ideas on the subject (first manuscript in 1675, first official publication 1684) [1].

At that time however this was not the case and scientists vigorously defended both contenders over who had been first. The dispute grew at such a level that due to accusation of plagiarism made against Leibniz he decided to appeal to the Royal Society to resolve the matter. The Royal Society which at that time was under the presidency of Newton set up a committee to investigate. In the report, Leibniz was found guilty of concealing his knowledge of the prior, relevant achievement of others. Though this was not a formal accusation of plagiarism in practice it had the same effect. It turned out that the whole investigation and the report was a farce set up by Newton himself [2]. It appears that Newton wasn't the nicest person of the planet [3].

At this point both parties accused the other of plagiarism. The dispute lasted more than a century creating an alienation between England and Europe which stopped almost all interchange of ideas on scientific subjects [1], [4].

But returning to the math, it was Leibniz who introduced the term differential equation ("aequatio differentialis" precisely, from latin), Newton used the term fluxional equation.

The integration symbol \( \int \) which first apeared on the 29th of October 1675 ("...the memorable day on which the notation of the new calculus came to be" [1]) and the derivative form \(\frac{dx}{dy}\) introduced on the 11th of November 1675 [5] are both due to Leibniz as well. He also assigned the term "integral" to the symbol \(\int\) but this upon suggestion of the Bernoulli brothers [6].

Other important notations for derivatives are that of Newton (\(\dot x \)), and Lagrange (\(x' \)).

As a closing note, to those of you who just got initiated to calculus and feel a little bit lost (I did for sure), remember, when Leibniz started to work on it he wasn't able to tell if \( dxdy\) is the same as \( d(xy) \) (they are not) [1]. I hope this takes some pressure off ;).

What is a PDE (Partial Differential Equation)? [Top]

A PDE (Partial Differential Equation) is an equation involving an unknown function of more than one independent variable and at least one partial derivative of that function. The order of a partial differential equation is the order of the highest derivative involved. An example of first order PDE for the function \( u\left(x_1, ..., x_n \right) \) is the following \begin{equation}\label{PDE} f \left(x_1, ..., x_n, u, \frac{\partial u}{\partial x_1}, ..., \frac{\partial u}{\partial x_n} \right)=0 \end{equation} This type of differential equation will not be pursue any further since this topic is quite complex and beyond the purpose of this primer.

What is an ODE (Ordinary Differential Equation)? [Top]

An ODE (Ordinary Differential Equation) is an equation involving an unknown function of one independent variable, its derivatives and eventual known quantities. The term ordinary is due to the fact that the derivative of a single independent variable function is called ordinary derivative. ODE's most general form, called implicit, is the following \begin{equation}\label{ODE_def} F \left(t, x, x', x'',... , x^{(m)} \right)=0 \end{equation} where \(t\) is the independent variable and \(x(t)\), the dependent variable, is the unknown function. \(m\), the highest time derivative of \(x(t)\) in \(F\), defines the order of equation \eqref{ODE_def}. \( x: I \rightarrow \mathbb{R} \) with \(I \subseteq \mathbb{R}\) open and \(F: U \rightarrow \mathbb{R} \) with \(U \subseteq \mathbb{R}^{m+2}\). The explicit dependence of \(F\) on \(t\) is used to describe a system that is either driven by an external input, has a time varying structure or both.

Another form, which is very important, is the explicit one \begin{equation}\label{expicit_ODE_def} x^{(m)} = f\left(t, x, x', x'',... , x^{(m-1)} \right) \end{equation} where \(f: U\rightarrow \mathbb{R} \) with \(U \subseteq \mathbb{R}^{m+1}\) and \(t\), \(x\) and order are defined as in \eqref{ODE_def}. Equation \eqref{expicit_ODE_def} is also called ODE in normal form. Usually the laws of physics can be expressed in this form. Working with \eqref{expicit_ODE_def} is desirable because under specific conditions, which includes known initial conditions (see next sections), it always has a unique solution.

Furthermore, if \(F\) from \eqref{ODE_def} is continuously differentiable (the derivative exists and is itself a continuous function) in \(x^{(m)}\) on an open set \(U_1 \subset U \subset \mathbb{R}^n\), \eqref{ODE_def} and \begin{equation} \label{IFT} \frac{\partial F}{\partial x^{(m)}} \neq 0 \end{equation} holds for a point \( {\bf x_0} \in U_1\) then the Implicit Function Theorem can be invoked and, at least locally, \eqref{ODE_def} can be in principle cast into \eqref{expicit_ODE_def} [9].

These are the reasons why, together with a large amount of theory and books on it, we mainly see around explicit form \eqref{expicit_ODE_def} rather than \eqref{ODE_def}.

ODE are often called 1D (one dimensional) problems because they have only one independent variable.

Unless otherwise stated from now on we will use the term differential equations implying differential equations with ordinary derivatives.

A note on Implicit Function Theorem, previously it has been stated that in principle we can move from \eqref{ODE_def} to \eqref{expicit_ODE_def}, why in principle? This is due to the fact that the theorem confirms the existence of \(f \) in \eqref{expicit_ODE_def}, not its expression. For example, if you consider the following equation \begin{equation} \label{IFT_eq} F(\dot x, x, t)=\textrm{atan}(\dot x ) + \dot x + x - u(t) = 0 \end{equation} with \(u(t)\) being a know function, though both \( \frac{\partial F}{\partial \dot x} \neq 0\) and \( \frac{\partial F}{\partial \dot x}\) continuity holds globally, and therefore \eqref{expicit_ODE_def} exists, it is not possible to find an explicit form for \(f\). However, fear not, there are other ways to get around the problem, for example with Taylor expansion or, given the solution exists and is unique, using a numeric solver but these topics goes beyond the purpose of this primer and we will not touch them. It is also important to observe that the example \eqref{IFT_eq} has been built on purpose and you will probably not find it in real life problems.

Examples of first and second order ODE [Top]

Using the above notation a first order ODE will be written as \begin{equation}\label{1st_O_def_implicit} F \left(t, x, x' \right)=0 \end{equation} or in normal form \begin{equation}\label{1st_O_def_explicit} x' = f \left(t, x \right) \end{equation}

An example of first order ODE (implicit and explicit form) is \begin{equation}\label{1st_O_ex} V_{in}(t) - x(t) - CRx'(t) = 0 \Longleftrightarrow x'(t) = \frac{V_{in}(t)}{CR} - \frac{x(t)}{CR} \end{equation} where \( V_{in}(t) \) is a known function, \( R \) and \( C \) are all constants. Do you recognize it? It's the equation that describes the behaviour af an RC circuit (\(x(t)\) represent the voltage over the capacitor) or, if you prefer, a first order low pass filter.

For a second order differential equation we have \begin{equation}\label{2nd_O_def} F \left(t, x, x', x'' \right)=0 \end{equation} or in normal form \begin{equation} x''=f \left(t, x, x' \right) \end{equation} and as example we can chose the very famous relation (implicit and explicit form) \begin{equation}\label{2nd_O_ex} Mx''(t) - f(t) = 0 \Longleftrightarrow x''(t) = \frac{f(t)}{M} \end{equation} which is Newton's second law of motion where \( f(t) \) is the resultant of all forces acting on a point mass of mass \( M \) which is positioned in \(x(t)\).

First order systems [Top]

When working with algebraic equations we often find ourself solving a problem that is actually composed of several equations and variables like \begin{equation} \begin{cases} x^2_1+x_2 =3 \\ \ x_1 x_2 = 5 \end{cases} \end{equation} In the same way we can have a system of differential equations \begin{equation} \begin{cases} x'_1 = -x_1 + x_2 \\ x'_2 = -x^2_1 x_2+sin(t) \end{cases} \end{equation} More generally, we can define a system of \(n\) first-order differential equations where the unknown is a vector-valued function, \({\bf x}(t): \mathbb{R} \rightarrow \mathbb{R}^n\), that satisfies a vectorial form of equation \eqref{1st_O_def_implicit} \begin{equation} {\bf F}(t,{\bf x},{\bf x}′)=0 \end{equation} where \({\bf F} ∶ \mathbb{R} \times \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}^n \). In components \begin{equation} F_i(t, \;\; \underbrace{ x_1,...,x_n}_{\bf x}, \;\; \underbrace{ x'_1,...,x'_n }_{{\bf x}'} ) =0, \qquad i = 1,...,n \end{equation} Unless otherwise stated we will refer to first order systems of ordinary differential equations as first order systems.

First order systems are very important because they can be used to study any m-th order differential equation. Furthermore, they can be used to study any system of m-th order differential equation which is even more important since this is how we usually describe the interactions between two or more physical entities (for example, the air pressure produced by a loudspeaker when driven by an electric input).

To show this, let us assume we have a system of \(n\) m-th order differential equation (the single equation case can be obtained by setting \(n=1\)) \begin{equation}\label{mth_system} {\bf G}(t, {\bf y}, {\bf y}', ..., {\bf y}^{(m)}) =0 \end{equation} in terms of components \eqref{mth_system} can be written as \begin{equation} G_i(t, \;\; \underbrace{ y_1,...,y_n }_{\bf y}, \;\; \underbrace{ y'_1,...,y'_n }_{{\bf y}'}, \;\; ..., \;\; \underbrace{ y^{(m)}_1,...,y^{(m)}_n }_{{\bf y}^{(m)}} ) =0, \qquad i = 1,...,n \end{equation} If we now introduce the following change of variable \begin{equation} \begin{aligned} {\bf y} &&= {\bf x}_1 \\ {\bf y}' &= {\bf x}'_1 &= {\bf x}_2\\ & \vdots &\vdots \\ {\bf y}^{(m-1)} &= {\bf x}'_{(m-1)} &= {\bf x}_m\\ {\bf y}^{(m)} &={\bf x}'_m\\ \end{aligned} \end{equation} system \eqref{mth_system} becomes \begin{equation} \label{newSyst} \begin{cases} {\bf x}'_1 - {\bf x}_2 = 0\\ \vdots \\ {\bf x}'_{(m-1)} - {\bf x}_m =0\\ {\bf G}(t, \;\; {\bf x}_1, ..., {\bf x}_m, \;\; {\bf x}'_m) =0 \end{cases} \end{equation} And defining \begin{equation} \label{v_change} {\bf x} = \begin{pmatrix} {\bf x}_1 \\ \vdots \\ {\bf x}_m \end{pmatrix} = \begin{pmatrix} {\bf y} \\ {\bf y}' \\ \vdots \\ {\bf y}^{(m-1)} \end{pmatrix} \end{equation} \eqref{newSyst} can be written in a more compact way as \begin{equation}\label{firstOrdSyst} {\bf F} \left(t, {\bf x}, {\bf x}' \right)=0 \end{equation} which is a first order system. But now the number of equations have increased from \(n\) to \(nm\), in fact we have \({\bf x}(t): \mathbb{R} \rightarrow \mathbb{R}^{nm}\) and \({\bf F} ∶ \mathbb{R} \times \mathbb{R}^{nm} \times \mathbb{R}^{nm} \rightarrow \mathbb{R}^{nm} \).

The same transformation can be used with systems in explicit form \begin{equation}\label{mth_system_explicit} {\bf y}^{(m)} = {\bf g}(t, {\bf y}, {\bf y}', ..., {\bf y}^{(m-1)}) \end{equation} where \begin{equation}\label{mth_system_explicit_components} y^{(m)}_i=g_i(t, \;\; \underbrace{ y_1,...,y_n }_{\bf y}, \;\; \underbrace{ y'_1,...,y'_n }_{{\bf y}'}, \;\; ..., \;\; \underbrace{ y^{(m-1)}_1,...,y^{(m-1)}_n }_{{\bf y}^{(m-1)}} ) , \qquad i = 1,...,n \end{equation} From \eqref{v_change} we obtain \begin{equation}\label{1st_system_explicit} {\bf x}' = {\bf f} (t, {\bf x}) \;\; \Longleftrightarrow \;\; \begin{cases} {\bf x}'_1 = {\bf x}_2\\ \vdots \\ {\bf x}'_{(m-1)} = {\bf x}_m\\ {{\bf x}'_m =\bf g}(t, {\bf x}_1, ..., {\bf x}_m) \end{cases} \end{equation}

where \({\bf x}(t): \mathbb{R} \rightarrow \mathbb{R}^{nm}\) and \({\bf f} ∶ \mathbb{R} \times \mathbb{R}^{nm} \rightarrow \mathbb{R}^{nm} \).

This means that any system of ordinary differential equation can be studied as a first order system.

It is important to note that the implicit function theorem can be used on implicit first order systems as well and if its conditions, which for vectorial functions become more complex (the Jacobian matrix \( \frac{\partial {\bf F} }{\partial {\bf x'}}\) is assumed to be nonsingular in a point \({\bf x_0} \in U_1\) with \(U_1 \subset \mathbb{R} \times \mathbb{R}^{nm}\times \mathbb{R}^{nm} \) open), are satisfied then \eqref{firstOrdSyst} can be in principle cast, at least locally, into \eqref{1st_system_explicit} [10].

Newton's law recast as a first order system [Top]

As an example of first order system we can take Newton's second law of motion defined by \eqref{2nd_O_ex} \begin{equation}\label{1st_Syst_ex} Mx''(t) - f(t) = 0 \end{equation} It can be noted that n=1 and m=2 . This means that the associated first order system will have two equations, that is \begin{align} \label{Newton_first} \begin{cases} x'_1 - x_2 =0\\ Mx_2' - f = 0 \end{cases} && \textrm{with} && \begin{array} {l} x =x_1 \\ x' =x'_1 = x_2 \end{array} \end{align} where \(x_1\) and \(x_2\) are respectively position and speed of the point mass.

Existence and uniqueness of solution for IVP (Initial Value Problem) [Top]

One of the most important properties of explicit ODE systems is that if initial conditions are available and specific conditions on \({\bf f}\) and \({\bf x}\) from \eqref{1st_O_def_explicit_2} are met the solution exists and is unique. There will probably be some questions here, like: what is an initial condition? What is an initial value problem and why do we care about existence and uniqueness of solution?

Let us start with initial conditions, given the first order system \begin{equation} \label{1st_O_def_explicit_2} {\bf x}' = {\bf f} \left(t, {\bf x} \right) \end{equation} where \( {\bf f}: U\rightarrow \mathbb{R}^{n} \) with \(U \subseteq \mathbb{R}^{n+1}\) open, \( {\bf x}: I \rightarrow \mathbb{R}^{n}\) with \(I \subseteq \mathbb{R}\) open, an initial condition for \eqref{1st_O_def_explicit_2} is the value of the dependent variable \({\bf x}\) at some point in time designated as the initial time (typically denoted \(t =t_0= 0\)) \begin{equation} \label{ic} {\bf x} \left(t_0 \right) = {\bf x}_0 \end{equation}

From a physical point of view initial conditions represent a picture of the system right before we start to predict its future evolution. As an example, if we think about the motion of a point mass \begin{equation} \begin{cases} x'_1 - x_2 =0\\ Mx_2'(t) - f(t) = 0 \end{cases} \end{equation} where \(x_1\) and \(x_2\) are defined as in \eqref{Newton_first}, assuming the starting time of the evolution of \({\bf x}\) is \(t_0=0\), its initial conditions will be \begin{equation} \begin{cases} x_1(0), \textrm{the initial position}\\ x_2(0), \textrm{the initial speed} \end{cases} \end{equation}

To answer the second question, the ODE system \eqref{1st_O_def_explicit_2} together with the initial conditions \eqref{ic} forms an initial value problem or simply IVP. The IVP is also known as the Cauchy problem, in fact it was Cauchy that first presented a rigorous study on existence and uniqueness of this type of problems [7]. Before that existence was tacitly assumed. It has to be noted that for certain problems this still happens. As an example (this actually refers to a PDE which is topic that is not covered by this primer but since existence problem applies to it as well, it is worth mentioning), the Clay Mathematics Institute has designated a one million dollars prize to who is going to prove the existence (and uniqueness) of Navier–Stokes equation's solution (rumours has it this is probably the worst way to earn it...). The Navier–Stokes equation describes the behaviour of fluids and are widely used in engineering and physics.

Before we move to the last question it is important to define the meaning of solution for an IVP.

A solution of the IVP \eqref{1st_O_def_explicit_2}-\eqref{ic} on an open interval \(J \subseteq I\), with \(t_0 \in J \), is a function \({\bf x}_s(t)\) such that

• \( {\bf x}_s \left(t_0 \right) = {\bf x}_0 \)
• \({\bf x}'_s(t)\) exists \(\forall t \in J \)
• \( \left( t, {\bf x}_s \right) \in U \), \( \forall t \in J \)
• \({\bf x}'_s = f(t, {\bf x}_s), \forall t \in J \)

Now, it has been soon realized that many differential equations can not be solved explicitly. Therefore, among other problems that this condition created, it was of prime importance to verify solution's existence. In practice, searching for a solution that doesn't exist will not help solving a differential equation ;).

Finally, about uniqueness, once existence of solution holds its uniques is required in order to predict the evolution of the solution, this is mandatory. Also, uniqueness allows you to literally guess the solutions. In fact, if you think that a function \({\bf x}_c\) could be a solution of your problem you can check it against the definition of IVP solution (at this stage \({\bf x}_c\) is called the candidate solution or simply candidate) and if it satisfies the requirements that is the solution.

After these premises we can state, with the theorem below, a set of sufficient conditions for guaranteeing the existence and uniqueness of the solution of an IVP.

Theorem: Let \({\bf f}: U \rightarrow \mathbb{R}^{n}\) with \( U \subset \mathbb{R}^{n+1}\) open, \( {\bf f} \) differentiable in \(x_i\) with \(i=1 \ldots n\) and \((t_0,{\bf x}_0) \in U\). If \({\bf f}: U \rightarrow \mathbb{R}^{n}\) and \( \partial_{x_i} {\bf f}\) with \(i=1 \ldots n\) are continuous then there is an open interval \(J\) containing \(t_0\) and a unique solution of \({\bf x}' = {\bf f}(t, {\bf x}) \) defined on \(J\) which satisfy \( {\bf x}(t_0) = {\bf x}_0\).

The proof of the theorem is beyond the purpose of this primer but if you're curious to explore the subject (quite complex) the bibliography is very rich. You can for example have a look at [8].

From a practical point of view, provided initial conditions are satisfied, if \( {\bf f} \left(t, {\bf x} \right)\) from \eqref{1st_O_def_explicit_2} is well behaved (continuously differentiable in \( {\bf x}\) and at least continuous in \(t\)), at least locally, the solution exist and is unique.

It has to be noted that IVP is not the only type of problem that guarantees existence and uniqueness of solutions. Another one is, for example, the boundary value problem where the unknown function and some of its derivatives are fixed at more than one value (referred to as boundary conditions) of the independent variable.

This concludes the primer, the important point that it tries to show is that the implicit function theorem, the fisrt order systems description and the existance and uniqueness theorem give a sistematic way to solve a wide range of problems. If you like what you just read and would like to use it, feel free to do so but don't forget to cite this website ;).

PS: But what happens if the Jacobian of \({\bf F}\) from \eqref{firstOrdSyst} is singular? Well... :)

References [Top]

[1] Cajori, Florian. 1909. A history of mathematics. New York: The Macmillan Company; London, Macmillan & Co., Ltd.

[2] Hall, A. Rupert. 2002. Philosophers at War: The Quarrel between Newton and Leibniz. Cambridge University Press

[3] Hawking, S., Mlodinow, L. 2005. A Briefer History of Time. New York, New York: Bantam Dell.

[4] Ince, E.L., 1956. Ordinary Differential Equations. Mineola, New York: Dover Publications.

[5] Cajori, Florian. 1993. A History of Mathematical Notations. Mineola, New York: Dover Publications.

[6] Boyer, Carl, B. 1959. The History Of The Calculus And Its Conceptual Development. Mineola, New York: Dover Publications.

[7] Agarwal, R.P., O’Regan. 2008. An Introduction to Ordinary Differential Equations. New York: Springer.

[8] Hirsch, M.W., Smale. Differential Equations, Dynamical Systems, and Linear Algebra. Academic Press, New York, 1974.

[9] Teschl, Gerald. Ordinary Differential Equations and Dynamical Systems. American Mathematical Society, 2012.

[10] Ascher, U.M., Petzold. Computer Methods for Ordinary Differential Equations and Differential-Algebraic Equations. Society for Industrial and Applied Mathematics, 1998.