DatumEdgeSource code for my personal website
https://www.datumedge.co.uk/
Thu, 31 Dec 2020 09:52:34 +0000Thu, 31 Dec 2020 09:52:34 +0000Jekyll v3.9.0Reproducible simulations with Singularity<p>
Reproducing the result of a scientific experiment is necessary to establish trust, and reproducibility has long been a key part of the scientific method.
Traditionally, an experiment could be repeated by following the method documented by the original scientists: setting up apparatus, taking measurements, and so on. If the method was sufficiently well documented then it was, perhaps, likely that the original results could be reproduced.
These ‘wet lab’ experiments continue today, but many experiments are now performed entirely on computers.
Such computational experiments involve no physical apparatus, but merely the processing of input data files through some scientific software before writing more data files for later analysis and plotting.
</p>
<p>
Repeating computational experiments is particularly difficult because, before any results can be obtained, there are many pieces of software apparatus that must be assembled: we must install an operating system, choose the correct version of our programming language and all the necessary scientific libraries, and we must use input parameters that are identical to those used in the original experiment.
Assembling any of these pieces incorrectly might lead to subtly incorrect results, obviously incorrect results, or a failure to obtain any results at all.
All this places a burden on the original scientists to document every piece of software, its version number and input parameters, and places a burden on the scientist wishing to reproduce the results.
</p>
<p>
There are a variety of tools that help to relieve this burden by automating the process of conducting computational experiments.
<a href="http://singularity.lbl.gov">Singularity</a> is one such tool, having been purposebuilt for automating computational experiments.
A scientist creates a single configuration file that provides all the information Singularity needs to assemble the pieces of software apparatus and perform the experiment.
This way, instead of writing a ‘method’ section that is only <a href="https://en.wikipedia.org/wiki/Humanreadable_medium">humanreadable</a>, the scientist has written a configuration file that is both humanreadable and <a href="https://en.wikipedia.org/wiki/Machinereadable_data">machinereadable</a>.
Using this configuration, Singularity will create an image file with all the correct versions of scientific software preinstalled.
The scientist can verify their work by reproducing their experiment themselves, and they can run the same experiment just by copying the image file between their personal laptop, office workstation, or their institution's HPC cluster.
And they can send their Singularity configuration file and image files to other scientists, or they can obtain a <a href="https://en.wikipedia.org/wiki/Digital_object_identifier">DOI</a> by uploading the files to <a href="http://help.zenodo.org/features/">Zenodo</a>, making their computational experiments citeable in the same way as their journal publications.
</p>
<p>
I've used Singularity to run my own atmospheric simulations using the <a href="https://openfoam.org/">OpenFOAM computational fluid dynamics</a> software.
While my results have yet to be reproduced by others, I regularly use Singularity to reproduce my own results on my laptop, university desktop and <a href="https://aws.amazon.com/ec2/">AWS cloud compute servers</a>, giving me confidence that my software and my results are robust.
Whenever I've been stuck, the friendly Singularity developers have been quick to help out <a href="https://twitter.com/singularityapp">on twitter</a>.
But overall, I've found Singularity to be easy to use, and anyone that is familiar with git commands should feel right at home using it. Give it a try!
</p>
Thu, 15 Feb 2018 10:00:00 +0000
https://www.datumedge.co.uk/2018/02/15/reproduciblesimulationswithsingularity/
https://www.datumedge.co.uk/2018/02/15/reproduciblesimulationswithsingularity/The advection process: simulating wind on computers<p>
If we know which way the wind is blowing then we can predict a lot about the weather.
We can easily observe the wind moving clouds across the sky, but the wind also moves air pollution and greenhouse gases.
This process is called <em>transport</em> or <em><a href="https://en.wikipedia.org/wiki/Advection">advection</a></em>.
Accurately simulating the advection process is important for forecasting the weather and predicting climate change.
</p>
<p>
I am interested in simulating the advection process on computers by dividing the world into boxes and calculating the same equation in every box. There are many existing advection methods but many rely on these boxes having the correct shape and size, otherwise these existing methods can produce inaccurate simulations.
</p>
<p>
During my PhD, I've been developing a new advection method that produces accurate simulations regardless of cell shape or size.
In this post I'll explain how advection works and how we can simulate advection on computers.
But, before I do, let's talk about how we observe the weather from the ground.
</p>
<p>
In meteorology, we generally have an incomplete picture of the weather. For example, a <a href="http://www.metoffice.gov.uk/learning/science/firststeps/observations/weatherstations">weather station</a> measures the local air temperature, but <a href="http://www.metoffice.gov.uk/media/pdf/k/5/Fact_sheet_No._17.pdf">there are only a few hundred such stations dotted around the UK</a>.
The temperature at another location can be approximated by looking at the temperatures reported by nearby stations. In fact, we can approximate the temperature at <em>any</em> location by <em>reconstructing</em> a continuous temperature field using the weather station measurements.
</p>
<h2>The advection equation</h2>
<p>
So far we have only talked about temperatures varying geographically, but temperatures also vary over time. One reason that temperatures change over time is because the wind is blowing. For example, a wind blowing from the north transports, or <em>advects</em>, cold air from the arctic southwards over the UK. How fast the temperature changes depends on the wind speed, and the size of the temperature contrast between the arctic air and the air further south. We can write this as an equation. Let's call the wind speed \(v\) and assume that the wind speed and direction are always the same everywhere. We'll label the temperature \(T\), label time \(t\), and label the southtonorth direction \(y\), then we can write down the advection equation using <a href="https://en.wikipedia.org/wiki/Partial_derivative">partial derivative</a> notation,
\begin{equation*}
\frac{\partial T}{\partial t} =  \frac{\partial T}{\partial y} \times v
\end{equation*}
This equation tells us that the local temperature will vary over time (\(\frac{\partial T}{\partial t}\)), depending on the northsouth temperature contrast (\( \frac{\partial T}{\partial y}\)) multiplied by the wind speed \(v\).
</p>
<h2>Solving the advection equation</h2>
<p>
One way to solve the advection equation on a computer is to divide the world into boxes, called <em>cells</em>.
The complete arrangement of cells is called a <em><a href="https://en.wikipedia.org/wiki/Types_of_mesh">mesh</a></em>.
At a point at the centre of each cell we store meteorological information such as temperature, water vapour content or pollutant concentration. At the cell faces where two cells touch we store the wind speed and direction. The arrangement looks like this:
<figure>
<img style="width: 20em; maxwidth:80%;" src="/images/britaincgrid.svg"/>
<figcaption>
A mesh of cells with temperatures stored at cell centres and winds stored at cell faces. For illustration, the temperature and winds are only shown in one cell. This arrangement of data is known as an <a href="https://en.wikipedia.org/wiki/Arakawa_grids">Arakawa C‑grid</a>. Figure adapted from <a href="https://commons.wikimedia.org/wiki/File:British_National_Grid.svg">WikiMedia Commons</a>, <a href="https://creativecommons.org/licenses/bysa/3.0/">CC BYSA 3.0</a>.
</figcaption>
</figure>
</p>
<p>
The above example of a mesh over the UK uses cubeshaped cells stacked in columns above the Earth, and arranged along latitude and longitude lines. But more recently, weather forecasting models are using <a href="https://doi.org/10.1002/qj.958">different types of mesh</a>. These models tesselate the globe with squares, hexagons or triangles.
<figure>
<ul class="subfig">
<li><p>Triangularicosahedral mesh</p><img style="maxwidth: 12em;" src="/images/tri3.svg"/></li>
<li><p>Cubedsphere mesh</p><img style="maxwidth: 12em;" src="/images/cubed8.svg"/></li>
<li><p>Hexagonalicosahedral mesh</p><img style="maxwidth: 12em;" src="/images/hex3.svg"/></li>
</ul>
<figcaption>
The surfaces of some different types of global mesh. The cells are <a href="https://en.wikipedia.org/wiki/Prism_(geometry)">prismatic</a> since they are stacked in columns above the surface.
</figcaption>
</figure>
</p>
<p>
Weather models must also rearrange cells in order to represent mountains, valleys, cliffs and other terrain. Once again, different models rearrange cells differently. One method, called the <em><a href="https://en.wikipedia.org/wiki/Sigma_coordinate_system">terrainfollowing method</a></em>, shifts cells up or down to accommodate the terrain. Another method, called the <em>cutcell method</em>, cuts cells where they intersect the terrain. Here's what these methods look like when we use them to represent an idealised, waveshaped mountain:
<figure>
<ul class="subfig">
<li><p>Terrainfollowing mesh</p><img style="maxwidth: 20em;" src="/images/btf.svg"/></li>
<li><p>Cutcell mesh</p><img style="maxwidth: 20em;" src="/images/cutCell.svg"/></li>
</ul>
<figcaption>
Two different methods for representing terrain in weather forecast models. The terrainfollowing method is widely used but suffers from large distortions above steep slopes. The cut cell method alleviates this problem but cells may be very much smaller than most others in a cut cell mesh.
</figcaption>
</figure>
</p>
<p>
Once we've chosen a mesh and stored temperature at cell centres and the wind at cell faces, we can start calculating a solution to the advection equation which enables us to forecast how the temperature will vary over time.
We can solve the advection equation for every cell separately by <a href="https://en.wikipedia.org/wiki/Discretization">discretising</a> the advection equation. Let's consider a cell with a north face and a south face.
We want to know how the temperature stored at the cell centre, \(T_\mathrm{cell}\), will vary over time.
We can calculate this by <em>reconstructing</em> a continuous temperature field and using this to approximate temperature values at the north and south faces of the south, \(T_\mathrm{north}\) and \(T_\mathrm{south}\),
\begin{equation*}
\frac{\partial T_\mathrm{cell}}{\partial t} =  \frac{T_\mathrm{north}  T_\mathrm{south}}{\Delta y} \times v
\end{equation*}
where \(\Delta y\) is the distance between the north and south cell faces.
This is the same reconstruction process that we described earlier, only, instead of approximating temperatures using nearby weather station measurements, we are approximating temperatures using nearby cell centre values.
</p>
<p>
There are many existing numerical methods for solving the advection equation but many do not cope well when meshes are distorted, such as terrainfollowing meshes, or when cells have very different sizes, such as those cells in cutcell meshes. Inaccurate solutions to the advection equation lead to inaccuracies in the weather forecast. In extreme cases, very poor solutions can cause the model software to crash, and this is known as a <em>numerical instability</em>.
</p>
<p>
<figure>
<img width="100%" src="/images/slugslantedCellslinearUpwind.gif"/>
<figcaption>
An idealised simulation of a blob advected over steep mountains. A numerical instability develops because the cells are so distorted over the mountain.
</figcaption>
</figure>
We can see a numerical instability growing in this idealised example. A blob is being advected from left to right over a range of steep, waveshaped mountains. This example is using a simple advection method which cannot cope with the distorted cells in this mesh.
</p>
<p>
We've developed a new method for solving the advection equation with almost any type of mesh using cubes or hexagons, terrainfollowing or cutcell methods.
The advection method works by reconstructing a continuous field from data stored at cell centre points. A separate reconstruction is made for every face of every cell in the mesh using about twelve nearby cell centre values.
Given that weather forecast models have millions of cells, this sounds like an awful lot of calculations. But it turns out that we can make most of these calculations just once, store them, and reuse them for all our simulations.
</p>
<p>
<figure>
<img width="100%" src="/images/slugslantedCellscubicFit.svg"/>
<figcaption>
Our new advection method avoids the numerical instability that occurred using the simple method.
</figcaption>
</figure>
Here's the same idealised simulation using our new advection method. The results are numerically stable and accurate.
</p>
<h2>Further reading</h2>
<p>
A preprint of our journal article documenting the new advection method is <a href="https://arxiv.org/abs/1702.00233">available on ArXiv</a>. I also have <a href="http://www.datumedge.co.uk/2017/01/20/creatingahighorderfinitevolumescheme/">another blog post</a> that talks about how to make the method even more accurate.
Or <a href="https://twitter.com/hertzsprrrung">follow me on Twitter</a> for more animations of the numerical methods I'm developing.
</p>
Thu, 16 Mar 2017 12:00:00 +0000
https://www.datumedge.co.uk/2017/03/16/simulatingwindoncomputers/
https://www.datumedge.co.uk/2017/03/16/simulatingwindoncomputers/Creating a highorder finite volume transport scheme<p>Over the last year of my PhD I have been working to create a finite volume transport scheme, called ‘cubicFit’, that is secondorder convergent on arbitrary meshes.
Recently, I have tried modifying the cubicFit transport scheme to obtain highorder convergence: that is, convergence greater than secondorder.
Here I'll introduce a onedimensional version of the cubicFit scheme and explain how it can be modified to obtain highorder convergence.</p>
<h2>Onedimensional finite volume approximation</h2>
<p>To formulate the onedimensional cubicFit transport scheme, I'll start from the transport equation for a dependent variable \(\phi\) assuming a nonnegative velocity \(u\) that is constant in time and space,
\begin{equation}
\frac{\partial \phi}{\partial t} =  u \frac{\partial \phi}{\partial x} \label{eqn:transport}
\end{equation}
where the term on the righthand side is called the <em>flux divergence</em>. With a <a href="https://en.wikipedia.org/wiki/Arakawa_grids">Cgrid staggering</a>, velocity values are stored at faces and \(\phi\) values are stored at cell centres. Using the finite volume method to discretise the flux divergence term, equation \eqref{eqn:transport} becomes
\begin{equation}
\frac{\partial \phi}{\partial t} =  u \frac{\phi_R  \phi_L}{\Delta x} \label{eqn:fluxdiv}
\end{equation}
where \(\phi_L\) and \(\phi_R\) are approximate values of \(\phi\) at the left and right faces respectively, and \(\Delta x\) is the distance between the faces.
</p>
<figure>
<img src="/images/cubicFit1D.png"/>
<figcaption>The onedimensional cubicFit transport scheme approximates face values \(\phi_L\) and \(\phi_R\) using fourpoint, upwindbiased stencils of cell centre values. Cell centres are marked by grey filled circles and faces are marked by black open circles.</figcaption>
</figure>
<p>
The cubicFit scheme approximates a face value using four neighbouring cell centre values, as shown above. For each face, a cubic polynomial is found that interpolates the four data points,
\begin{equation}
\phi = a_1 + a_2 x + a_3 x^2 + a_4 x^3 \text{.} \label{eqn:cubic}
\end{equation}
By choosing a local coordinate with its origin at the face then the value of \(\phi\) at the face is just \(a_1\), which is calculated as follows.
Assuming a uniform mesh with \(\Delta x = 1\) and choosing the position of \(\phi_{i+1/2}\) to be \(x=0\), equation \eqref{eqn:cubic} is evaluated at the cell centres \(\phi_{i2}, \ldots, \phi_{i+1}\) to form the matrix equation
\begin{align}
\mathbf{B} \mathbf{a} &= \mathbf{\phi} \\
\begin{bmatrix}
1 & 5/2 & 25/4 & 125/8 \\
1 & 3/2 & 9/4 & 27/8 \\
1 & 1/2 & 1/4 & 1/8 \\
1 & 1/2 & 1/4 & 1/8 \\
\end{bmatrix}
\begin{bmatrix}
a_1 \\
a_2 \\
a_3 \\
a_4
\end{bmatrix}
&=
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \text{.}
\end{align}
The unknown coefficients \(\mathbf{a}\) are found by inverting \(\mathbf{B}\) such that \(\mathbf{a} = \mathbf{B}^{1} \mathbf{\phi}\). The inverse matrix is
\begin{align}
\mathbf{B}^{1} =
\frac{1}{48}
\begin{bmatrix}
3 & 15 & 45 & 15 \\
2 & 6 & 42 & 46 \\
12 & 60 & 84 & 36 \\
8 & 24 & 24 & 8
\end{bmatrix} \label{eqn:cubicfitinverse}
\end{align}
Since \(x=0\) was chosen to be the position \(i+1/2\) then
\begin{align}
\phi_{i+1/2} = a_1 =
\frac{1}{16}
\begin{bmatrix}
1 \\ 5 \\ 15 \\ 5
\end{bmatrix}
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi{i+1}
\end{bmatrix} \text{.} \label{eqn:cubicfitfluxcoeffs}
\end{align}
</p>
<h2>Onedimensional finite difference approximation</h2>
<p>
To obtain a highorder version of the onedimensional cubicFit transport scheme I return to the flux divergence term on the righthand side of equation \eqref{eqn:transport} and discretise it using the finite difference method.
Using this method, the flux divergence of cell \(i\) is calculated using a cubic approximation using cell centre values \(\phi_{i2}, \ldots, \phi_{i+1}\). A matrix equation is constructed using equation \eqref{eqn:cubic} evaluated at every cell centre. For convenience, assume that \(\Delta x = 1\) and that \(x=0\) at the cell centre position of \(\phi_i\), hence the matrix equation becomes
\begin{align}
\mathbf{B} \mathbf{a} &= \mathbf{\phi} \\
\begin{bmatrix}
1 & 2 & 4 & 8 \\
1 & 1 & 1 & 1 \\
1 & 0 & 0 & 0\\
1 & 1 & 1 & 1\\
\end{bmatrix}
\begin{bmatrix}
a_1 \\
a_2 \\
a_3 \\
a_4
\end{bmatrix}
&=
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix}
\end{align}
The unknown coefficients \(\mathbf{a}\) are found by calculating the inverse matrix,
\begin{align}
\mathbf{B}^{1} =
\frac{1}{6}
\begin{bmatrix}
0 & 0 & 6 & 0 \\
1 & 6 & 3 & 2 \\
0 & 3 & 6 & 3 \\
1 & 3 & 3 & 1
\end{bmatrix} \text{.}
\end{align}
To calculate the flux divergence I calculate the derivative \(\partial \phi / \partial x = a_2 + 2 a_3 x + 3 a_4 x^2\). Evaluating the flux divergence at \(\phi_i\) where \(x=0\) then \(\partial \phi_i / \partial x = a_2\). Hence I find that the finite difference weighted sum is
\begin{align}
u \frac{\partial \phi_i}{\partial x} = u a_2 =
u \cdot \frac{1}{6}
\begin{bmatrix}
1 \\ 6 \\ 3 \\ 2
\end{bmatrix}
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \label{eqn:fdfluxdiv}
\end{align}
The cubic finite difference approximation given in equation \eqref{eqn:fdfluxdiv} is conservative on uniform meshes. This can be demonstrated by decomposing the weights vector,
\begin{align}
u \cdot \frac{1}{6}
\begin{bmatrix}
1 \\ 6 \\ 3 \\ 2
\end{bmatrix}
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix}
&=
u \cdot
\frac{1}{6}
\left(
\begin{bmatrix}
0 \\ 1 \\ 5 \\ 2
\end{bmatrix}

\begin{bmatrix}
1 \\ 5 \\ 2 \\ 0
\end{bmatrix}
\right)
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \\
&=
u \left(
\frac{1}{6}
\begin{bmatrix}
1 \\ 5 \\ 2
\end{bmatrix}
\cdot
\begin{bmatrix}
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix}

\frac{1}{6}
\begin{bmatrix}
1 \\ 5 \\ 2
\end{bmatrix}
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i
\end{bmatrix} \right) \text{.} \label{eqn:fdfluxcoeffs}
\end{align}
Notice that the flux divergence has been rewritten as the difference between right and left fluxes as in equation \eqref{eqn:fluxdiv}.
<h2>Highorder correction in one dimension</h2>
The highorder correction to the cubicFit scheme is calculated as the difference between the cubic finite difference approximation (equation \ref{eqn:fdfluxcoeffs}) and the uncorrected cubicFit approximation (equation \ref{eqn:cubicfitfluxcoeffs}),
\begin{align}
\mathrm{correction}(\phi_{i+1/2})
&=
\left(
\frac{1}{6}
\begin{bmatrix}
0 \\ 1 \\ 5 \\ 2
\end{bmatrix}

\frac{1}{16}
\begin{bmatrix}
1 \\ 5 \\ 15 \\ 5
\end{bmatrix}
\right)
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \\
&=
\frac{1}{48}
\begin{bmatrix}
3 \\ 7 \\ 5 \\ 1
\end{bmatrix}
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \label{eqn:correction}
\end{align}
which can be decomposed into a linear combination of second derivatives where \(\partial_x^2 \phi_i = \phi_{i1}  2 \phi_i + \phi_{i+1}\),
\begin{align}
&=
\frac{1}{48} \left(
3 \begin{bmatrix}1 \\ 2 \\ 1 \\ 0\end{bmatrix}
+ \begin{bmatrix}0 \\ 1 \\ 2 \\ 1\end{bmatrix}
\right)
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \\
&=
\frac{1}{48}
\left( 3 \partial_x^2 \phi_{i1} + \partial_x^2 \phi_i \right) \text{.}
\end{align}
Applying this correction using the threepoint approximation of the second derivative results in thirdorder convergence on uniform meshes and secondorder convergence on nonuniform meshes.
</p>
<figure>
<img src="/images/cubicFitConvergence.svg"/>
<figcaption>Convergence of uncorrected and corrected versions of the onedimensional cubicFit transport scheme.
A single sine wave is transported once around a periodic domain. The \(\ell_2\) error is calculated as \(\ell_2 = \sqrt{\sum_i{\left(\phi_i  \phi_i^T\right)^2 \Delta x_i} / \sum_i{\left(\phi_i^T\right)^2 \Delta x_i}}\) where \(\phi\) is the numeric solution and \(\phi_T\) is the analytic solution.
The <a href="https://github.com/hertzsprung/skamarockgassmann2011transport">python source code is available on github</a>.</figcaption>
</figure>
<p>
Alternatively, the second derivative can be calculated from equation \eqref{eqn:cubic} such that \(\partial_x^2 \phi = 2a_3 + 6a_4x\) where \(a_3\) and \(a_4\) can be calculated using equation \eqref{eqn:cubicfitinverse}. This approach results in fourthorder convergence on uniform meshes and secondorder convergence on nonuniform meshes, as seen in the convergence plot above.
Curiously, this approach yields the correction
\begin{align}
\mathrm{correction}(\phi_{i+1/2})
&=
\frac{1}{48}
\left(
3
\cdot
\begin{bmatrix}
0 \\ 1 \\ 2 \\ 1
\end{bmatrix}
+
\begin{bmatrix}
1 \\ 4 \\ 5 \\ 2
\end{bmatrix}
\right)
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \\
&=
\frac{1}{48}
\cdot
\begin{bmatrix}
1 \\ 1 \\ 1 \\ 1
\end{bmatrix}
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix}
=
\frac{1}{48}
\left(
\begin{bmatrix}
1 \\ 2 \\ 1 \\ 0
\end{bmatrix}
+
\begin{bmatrix}
0 \\ 1 \\ 2 \\ 1
\end{bmatrix}
\right)
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \\
&=
\frac{1}{48} \left( \partial_x^2 \phi_{i1} + \partial_x^2 \phi_i \right)
\cdot
\begin{bmatrix}
\phi_{i2} \\
\phi_{i1} \\
\phi_i \\
\phi_{i+1}
\end{bmatrix} \text{.}
\end{align}
</p>
<h2>Can highorder be obtained in two dimensions?</h2>
<p>
The onedimensional version of cubicFit exactly interpolates four data points with a cubic polynomial. However, on a twodimensional mesh of rectangular cells, the stencil has four points in the \(x\) direction and three points in the \(y\) direction for a total of twelve data points. The multidimensional polynomial is then
$$
\phi = a_1 + a_2 x + a_3 y + a_4 x^2 + a_5 xy + a_6 y^2 + a_7 x^3 + a_8 x^2 y + a_9 x y^2 \text{.}
$$
Hence, we have an overconstrained system of equations with twelve data points and nine unknown coefficients \(a_1 \ldots a_9\) which is solved using a leastsquares approach.
</p>
<p>In two dimensions, it is no longer clear to me how to formulate a highorder correction, but <a href="https://doi.org/10.1175/MWRD1005056.1">Skamarock and Gassmann 2011</a> used their onedimensional highorder correction in two dimensions by applying it at each cell face. It seems to me that their approach could be applied to cubicFit, though I have yet to have success applying a highorder correction to the <a href="/publications/slantcelladvection.pdf">OpenFOAM cubicFit implementation for arbitrarilystructured meshes</a>. I hope to describe the features of this scheme in a future post.</p>
<h2>Further reading</h2>
<ul>
<li><a href="https://doi.org/10.1175/15200493(2002)130<2459:ANTFVC>2.0.CO;2">A New TerrainFollowing Vertical Coordinate Formulation for Atmospheric Prediction Models</a>, Schär et al. 2002</li>
<li><a href="https://doi.org/10.1175/MWRD1005056.1">Conservative Transport Schemes for Spherical Geodesic Grids: HighOrder Flux Operators for ODEBased Time Integration</a>, Skamarock and Gassmann 2011</li>
</ul>
Fri, 20 Jan 2017 17:00:00 +0000
https://www.datumedge.co.uk/2017/01/20/creatingahighorderfinitevolumescheme/
https://www.datumedge.co.uk/2017/01/20/creatingahighorderfinitevolumescheme/Advection over steep slopes<p>There are many ways to represent terrain in atmospheric models. Terrain following coordinates are in widespread operational use but can suffer from numerical errors near steep slopes. Advection errors, and errors calculating pressure gradients both reduce model accuracy. The cut cell method is often put forward as an alternative which can reduce pressure gradient errors, but the technique may still suffer from advection errors where flow is misaligned with the mesh. Without special treatment, the cut cell method can create very small cells which constrain the timestep when explicit methods are used.</p>
<p>We present a new type of mesh, the slanted cell mesh, which avoids additional timestep constraints by creating cells that are long in the direction of flow. We describe a multidimensional, finite volume advection scheme that is suitable for arbitrary meshes. We discuss a new mathematical technique, the adaptive polynomial fit, which improves advection accuracy at the lower boundary around steep terrain.</p>
<figure>
<img src="/images/slugSlantedCells.png"/>
<figcaption>Comparison of linearUpwind and cubicFit error fields in the “slug” advection test.</figcaption>
</figure>
<p>Results of the slanted cell mesh and multidimensional advection scheme are compared with terrain following and cut cell meshes in a newly formulated test case which advects a stably stratified thermal profile in a prescribed, terrainfollowing wind. The test is designed to mimic thermal flow in the atmosphere and challenges the advection scheme at the lower boundary. We show that the scheme is resilient to grid distortions and misalignment of the flow with the grid. Incorporating the advection scheme into a dynamical core, we compare results with terrain following and cut cell meshes, and demonstrate that the slanted cell mesh also reduces pressure gradient errors.</p>
<ul>
<li><a href="https://doi.org/10.6084/m9.figshare.13502619.v1">PowerPoint slides</a></li>
<li><a href="https://centaur.reading.ac.uk/82281/">A complete description of the cubicFit advection scheme</a></li>
<li>An <a href="http://www.openfoam.org">OpenFOAM</a> implementation of cubicFit is <a href="https://github.com/AtmosFOAM/AtmosFOAM">available on GitHub</a></li>
<li>The slanted cell method is described in <em><a href="https://doi.org/10.1175/MWRD150226.1">Comparison of TerrainFollowing and CutCell Grids Using a Nonhydrostatic Model</a></em></li>
</ul>
<p><em>Presented at the <a href="http://www.ecmwf.int/en/learning/workshopsandseminars/workshopnumericalandcomputationalmethodssimulationallscalegeophysicalflows">ECMWF workshop on numerical and computational methods for simulation of allscale geophysical flows</a>.</em></p>
Mon, 03 Oct 2016 15:45:00 +0000
https://www.datumedge.co.uk/2016/10/03/advectionoversteepslopes/
https://www.datumedge.co.uk/2016/10/03/advectionoversteepslopes/Interactive polynomial fit visualisation<p>This demonstration shows quadratic and cubic polynomials fitted through a set of points using a least squares approach. This technique generalises to higher dimensions and can be used as an interpolation function for solving the flux form <a href="https://en.wikipedia.org/wiki/Advection#The_advection_equation">advection equation</a> on arbitrary meshes.</p>
<p>Drag points to change the fit. Controldrag points to change weights. The polynomial is fitted almost exactly through the filled points (which have large weights) and inexactly through the open points (which have small weights).</p>
<p>Coefficients are displayed next to each point with a colour corresponding to the quadratic and cubic curves. These coefficients are multiplied by the \(y\)values at each point to calculate the interpolated value where the curve intersects the \(y\)axis.
<div id="graph"></div>
<div>
<div id="pointdump"></div>
<div id="weightdump"></div>
</div>
<p>Created with <a href="https://d3js.org/">d3js</a> and <a href="http://www.numericjs.com/">numericjs</a>. The source repository is <a href="https://github.com/hertzsprung/polyvis">available on GitHub</a>.</p>
<script>
var margin = {top: 30, right: 20, bottom: 30, left: 50};
var width = 640;
var height = 480;
function approximate(domain, f) {
var N = 100;
var dx = (domain[1]  domain[0]) / N;
var data = [];
for (var i = 0; i <= N; i++) {
var x = domain[0] + i*dx;
data.push({x: x, y: f(x)});
}
return data;
}
var stencil = [];
stencil.push({x: 1, y: 2.5});
stencil.push({x: 0.6, y: 3});
stencil.push({x: 5, y: 5});
stencil.push({x: 3, y: 1});
var weights = [1e3, 1e3, 1, 1];
var x = d3.scale.linear().domain([5, 1]).range([0, width]);
var y = d3.scale.linear().domain([0, 5]).range([height, 0]);
var line = d3.svg.line()
.x(function(d) { return x(d.x); })
.y(function(d) { return y(d.y); });
var graph = d3.select("#graph")
.append("svg")
.attr("width", width + margin.left + margin.right)
.attr("height", height + margin.top + margin.bottom)
.append("g")
.attr("transform", "translate(" + margin.left + "," + margin.top + ")");
var xAxis = d3.svg.axis().scale(x).tickValues([5, 4, 3, 2, 1, 0, 1]);
graph.append("g")
.attr("class", "axis")
.attr("transform", "translate(0," + height + ")")
.call(xAxis);
var yAxis = d3.svg.axis().scale(y).tickValues([0, 1, 2, 3, 4, 5]).orient("left");
graph.append("g")
.attr("class", "axis")
.attr("transform", "translate(" + x(0) + ",0)")
.call(yAxis);
function dragmove(d) {
var newX = x.invert(d3.event.x);
var newY = y.invert(d3.event.y);
var index = d3.select(this).attr("pointindex");
if (d3.event.sourceEvent.ctrlKey) {
weights[index] = Math.max(1, Math.pow(10, newY));
} else {
stencil[index].x = newX;
stencil[index].y = newY;
}
redraw();
}
function format(a) {
return Math.round(a*100)/100;
}
function redraw() {
var quadFit = fit(stencil, quadratic(), weights);
var cubicFit = fit(stencil, cubic(), weights);
graph.selectAll(".polynomial").remove();
graph.append("path")
.attr("class", "polynomial quad")
.attr("d", line(approximate(x.domain(), quadFit.polynomial)));
graph.append("path")
.attr("class", "polynomial cubic")
.attr("d", line(approximate(x.domain(), cubicFit.polynomial)));
graph.selectAll("circle").remove();
graph.selectAll(".coeff").remove();
graph.selectAll("point").data(stencil).enter()
.append("circle")
.attr("pointindex", function(d, i) { return i; })
.attr("r", 8)
.attr("class", function(d, i) { return i < 2 ? "central" : "peripheral"; })
.attr("cx", function(d) { return x(d.x); })
.attr("cy", function(d) { return y(d.y); })
.style("cursor", "pointer")
.call(drag);
graph.selectAll("coeffs").data(stencil).enter()
.append("text")
.attr("class", "coeff cubic")
.attr("x", function(d) { return x(d.x) + 10; })
.attr("y", function(d) { return y(d.y) + 10; })
.text(function(d, i) { return format(cubicFit.coefficients[i]); });
graph.selectAll("coeffs").data(stencil).enter()
.append("text")
.attr("class", "coeff quad")
.attr("x", function(d) { return x(d.x)  10; })
.attr("y", function(d) { return y(d.y)  10; })
.text(function(d, i) { return format(quadFit.coefficients[i]); });
d3.select("#pointdump").selectAll("div").remove()
d3.select("#pointdump").selectAll("point").data(stencil).enter()
.append("div")
.text(function(d) { return "(" + format(d.x) + ", " + format(d.y) + "), \\"; });
d3.select("#weightdump").selectAll("div").remove()
d3.select("#weightdump").selectAll("weight").data(weights).enter()
.append("div")
.text(function(d) { return format(d) + ", \\"; });
}
var drag = d3.behavior.drag()
.on("drag", dragmove);
redraw();
</script>
Tue, 03 May 2016 19:39:00 +0000
https://www.datumedge.co.uk/2016/05/03/interactivepolynomialfitvisualisation/
https://www.datumedge.co.uk/2016/05/03/interactivepolynomialfitvisualisation/OpenFOAM on Ubuntu 16.04<p>If you want to use the <a href="http://www.openfoam.org/">OpenFOAM</a> 3 on <a href="https://wiki.ubuntu.com/XenialXerus/ReleaseNotes">Ubuntu 16.04 Xenial Xerus</a>, there is <a href="http://www.openfoam.org/mantisbt/view.php?id=2070">no binary package in the OpenFOAM repository</a> at the time of writing.</p>
<p>The source build takes many hours to complete, so you might prefer to workaround the issue as I did:
<ol>
<li>Install the dependencies: <pre>sudo aptget install libopenmpi1.10 csh libscotch5.1 binutilsdev libscotchdev libptscotch5.1</pre></li>
<li>Assuming you have a 64bit architecture, download the binary package for <a href="https://wiki.ubuntu.com/WilyWerewolf/ReleaseNotes">Ubuntu 15.10 Wily Werewolf</a>: <pre>wget O openfoam30_11_amd64.deb https://sourceforge.net/projects/foam/files/foam/ubuntu/dists/wily/main/binaryamd64/openfoam30_11_amd64.deb/download</pre></li>
<li>This package depends upon <a href="https://launchpad.net/ubuntu/+source/openmpi"><code>libopenmpi1.6</code></a> which no longer exists in Ubuntu 16.04, so we must coax OpenFOAM into using the new version. Install the downloaded binary with <pre>sudo dpkg ignoredepends=libopenmpi1.6 i openfoam30_11_amd64.deb</pre></li>
<li>Finally, ensure that <code>libmpi</code> is in the expected place: <pre>sudo ln s /usr/lib/libmpi.so /usr/lib/libmpi.so.1</pre></li>
</ol>
I haven't tested that this installation works with parallel execution, but it's enough to get OpenFOAM running.
</p>
Wed, 27 Apr 2016 08:18:00 +0000
https://www.datumedge.co.uk/2016/04/27/openfoamonubuntu1604/
https://www.datumedge.co.uk/2016/04/27/openfoamonubuntu1604/Beyond Stack Overflow<p>Most researchers I know in <a href="http://www.met.reading.ac.uk/">my department</a> make regular use of <a href="http://stackoverflow.com/tour">Stack Overflow</a>, the question and answer site for programmers. While it is the oldest and most popular, there are <a href="http://stackexchange.com/sites">a family of Q&A sites</a> that closely follow the look and feel of Stack Overflow. More than a dozen of these target <a href="http://stackexchange.com/sites#sciencename">scientific and mathematical topics</a>, and some are particularly relevant to those of us working in meteorology:</p>
<dl>
<dt><a href="http://math.stackexchange.com/"><img class="selogo" src="/images/mathse.png"/>Mathematics</a></dt><dd>Discussions are more rigorous than the mathematics I use in my own work, but this site has helped me with PDEs and calculus, geometric problems, and numerical analyses</dd>
<dt><a href="http://physics.stackexchange.com/"><img class="selogo" src="/images/physicsse.png"/>Physics</a></dt><dd>Perhaps the best place to learn about fluid dynamics, although I haven't used the site very much myself</dd>
<dt><a href="http://stats.stachexchange.com/"><img class="selogo" src="/images/crossvalidatedse.png"/>Cross Validated</a></dt><dd>I have yet to use this site, but it's the place for questions about statistics</dd>
<dt><a href="http://gis.stackexchange.com/"><img class="selogo" src="/images/gisse.png"/>Geographic Information Systems</a></dt><dd>Again, I've not used this site except <a href="http://gis.stackexchange.com/a/109969/68184">to get inspiration for shading relief maps</a> and discuss the <a href="http://twitter.com/search?q=%23endrainbow">merits of rainbow colour scales</a></dd>
</dl>
<p>These sites are already wellestablished, and new questions are often answered within minutes. But there are some other sites that are still in beta, with <a href="http://earthscience.stackexchange.com/">Earth Science</a> and <a href="http://scicomp.stackexchange.com/">Computational Science</a> being the most relevant to meteorologists. These sites have smaller communities and new questions can <a href="http://earthscience.stackexchange.com/unanswered/tagged/?tab=noanswers">go unanswered for days or longer</a>.
Answering a question on Stack Overflow often feels like trying to snipe on an eBay auction, but there is rarely such a race for upvotes on smaller sites.
</p>
<p>Compared to Stack Overflow, I've found it easier to participate in the Earth Science community. As well as answering questions at a more leisurely pace, there are more opportunities to post new questions, too. And I think that my questions and answers tend to gain more attention at Earth Science than those on Stack Overflow.</p>
<p>The Earth Science site is still maturing and needs to attract more scientists to make it credible. At present there are, in my opinion, a disproportionate number of <a href="http://earthscience.stackexchange.com/questions/tagged/hypothetical">hypothetical questions</a> and questions that are <a href="http://earthscience.stackexchange.com/questions/6847/whyhaventweedsovertakentheentireplanet">mostly</a> for <a href="http://earthscience.stackexchange.com/questions/245/isittruethatabutterflyflappingitswingscanresultinatornadoinadistantlocation">fun</a>.
There has been <a href="http://meta.earthscience.stackexchange.com/questions/34/levelofthequestionssofar">debate</a> amongst the community about <a href="http://meta.earthscience.stackexchange.com/questions/114/areweaimingtobeanexpertsite">what types of questions are desirable</a> on the site.</p>
<p>There are many other Stack Exchange sites that are not yet at the public beta stage. These are hosted in ‘<a href="http://area51.stackexchange.com/">Area 51</a>’ and include <a href="http://area51.stackexchange.com/categories/7/science">several with topics relating to sciences, mathematics and teaching</a>. Finally, the Academia Stack Exchange will explain how <a href="http://academia.stackexchange.com/questions/3545/whatdocosupervisorsgetoutofaphd">academics</a>, <a href="http://academia.stackexchange.com/questions/12342/selfplagiarisminphdthesis">students</a>, and <a href="http://academia.stackexchange.com/questions/55665/whatdoesthetypicalworkflowofajournallooklike">journals</a> operate.
</p>
<p><i>You can find my profile on <a href="https://stackoverflow.com/users/150884">Stack Overflow</a> and <a href="https://stackexchange.com/users/50566?tab=accounts">Stack Exchange</a>.</i></p>
<todo>finally, mention academia</todo>
Mon, 04 Apr 2016 15:27:00 +0000
https://www.datumedge.co.uk/2016/04/04/beyondstackoverflow/
https://www.datumedge.co.uk/2016/04/04/beyondstackoverflow/DatumEdge revival<p>This is a revival of the blog previously hosted with Google Blogger. Inspired by <a href="http://www.carlboettiger.info/labnotebook.html">Carl Boettiger’s open lab notebook</a>, and talks from <a href="http://www.climatelabbook.ac.uk/author/ed/">Ed Hawkins</a> and <a href="https://sites.google.com/site/tomsizmur/home">Tom Sizmur</a> about academics engaging with online communities, I have updated this website to include more information about me, documents and software that I've written, and links to my online presence elsewhere.</p>
<p>I hope to start sharing notes on my research as a PhD student in the <a href="http://www.met.reading.ac.uk/">Department of Meteorology at the University of Reading</a>, where I am developing numerical methods for atmospheric flows on arbitrary meshes using the <a href="http://www.openfoam.com/">OpenFOAM computational fluid dynamics library</a>, supervised by <a href="http://www.met.reading.ac.uk/~sws02hs/">Hilary Weller</a> and <a href="http://www.met.reading.ac.uk/~swrmethn/">John Methven</a> at the University of Reading, and <a href="http://www.metoffice.gov.uk/research/ourscientists/dynamics/terrydavies">Terry Davies</a> at the <a href="http://www.metoffice.gov.uk/">UK Met Office</a>.</p>
<p>This site was created using the superb <a href="https://jekyllrb.com/">Jekyll</a> website and blog generator, hosted with <a href="https://pages.github.com/">GitHub Pages</a>. The website source is available on <a href="https://github.com/hertzsprung/hertzsprung.github.io">GitHub</a>, should anyone want to reuse or modifying the site's layout and styling.</p>
Mon, 28 Mar 2016 12:54:00 +0000
https://www.datumedge.co.uk/2016/03/28/datumedgerevival/
https://www.datumedge.co.uk/2016/03/28/datumedgerevival/NetCDF and matplotlib basemap with Python 3Ubuntu 14.04 does not have packages for <a href="https://github.com/Unidata/netcdf4python">NetCDFPython</a> or the <a href="http://matplotlib.org/basemap/">Matplotlib Basemap Toolkit</a> for Python 3. However, both can be installed from source without too much difficulty.
<ol>
<li>Install the prerequisite packages with
<pre>sudo aptget install python3matplotlib python3dev libnetcdfdev libgeosdev libhdf5dev</pre></li>
<li>Download and extract <a href="https://pypi.python.org/pypi/netCDF4#downloads">NetCDFPython</a> and install it with
<pre>sudo python3 setup.py install</pre></li>
<li>Download and extract <a href="http://sourceforge.net/projects/matplotlib/files/matplotlibtoolkits/">Matplotlib Basemap Toolkit</a></li>
<li>If you try and install it, you will see the error
<pre>/usr/bin/ld: cannot find lgeos</pre>
To avoid this, symlink the <code>libgeos</code> shared object
<pre>sudo ln s /usr/lib/libgeos3.4.2.so /usr/lib/libgeos.so</pre>
Then install as before using <pre>sudo python3 setup.py install</pre></li>
</ol>
To check the installation, try import both libraries at the Python 3 prompt:
<script src="https://gist.github.com/hertzsprung/caa0bf35ebdca7fa84b2.js"> </script>
Sun, 14 Sep 2014 15:49:00 +0000
https://www.datumedge.co.uk/2014/09/14/netcdfandmatplotlibbasemapwithpython3/
https://www.datumedge.co.uk/2014/09/14/netcdfandmatplotlibbasemapwithpython3/Upgrading Intel video drivers for Xubuntu 13.04Having purchased a refurbished <a href="http://shop.lenovo.com/us/en/laptops/thinkpad/xseries/x301">ThinkPad X301</a> this month, I set about installing Xubuntu 13.04. So far, the only problem I have had was with the Intel video drivers. I was seeing some screen corruption, especially with <a href="http://packages.ubuntu.com/raring/rxvtunicode256color"><code>rxvt</code></a>:
<figure><img src="/images/corruption.png" /><figcaption>Example of screen corruption with <code>urxvt</code></figcaption></figure>
<a href="https://bugs.freedesktop.org/show_bug.cgi?id=66055">FreeDesktop's bug tracker confirmed</a> that I wasn't alone in having the problem, and this post explains how I fixed it.
<dl>
<dt><code>aptitude purge xserverxorgvideoall</code></dt>
<dd>This is a metapackage that depends upon video drivers for all supported chipsets. You can safely uninstall it without uninstalling all the chipset drivers.</dd>
<dt>Download the latest driver source</dt>
<dd>Fetch the <code>xf86videointel</code> archive from the <a href="http://xorg.freedesktop.org/archive/individual/driver/">XOrg archives</a>; as of August 2013, the latest version is <a href="http://xorg.freedesktop.org/archive/individual/driver/xf86videointel2.21.15.tar.bz">2.21.15</a>.</dd>
<dt>Install a toolchain</dt>
<dd>If you don't already have them, you will need to <code>aptitude install <a href="http://packages.ubuntu.com/raring/buildessential">buildessential</a> <a href="http://packages.ubuntu.com/raring/xserverxorgdev">xserverxorgdev</a></code></dd>
<dt><code>./configure && make && make install</code></dt>
<dd>Installs the Intel video driver into <code>/usr/local</code>. However, X11 does not scan this path by default<a href="#footnote1"><sup>1</sup></a>.</dd>
<dt><code>aptitude purge xserverxorgvideointel</code></dt>
<dd>Removes the older, packaged version of the driver</dd>
<dt><code>ln s /usr/local/lib/xorg/modules/drivers/intel_drv.so /usr/lib/xorg/modules/drivers/</code></dt>
<dd>Symlinks the driver into a path where X11 can discover it</dd>
<dt>Restart X11</dt>
<dd>Drop to a tty with ctrlaltf1 and <code>service lightdm restart</code></dd>
</dl>
<p id="footnote1"><sup>1</sup> If anyone knows how to persuade X11 to search <code>/usr/local/lib/xorg</code>, tell me how in the comments.</p>
The screen corruption bug should be fixed in Ubuntu 13.10 since <a href="http://packages.ubuntu.com/saucy/xserverxorgvideointel">it will include a more recent driver version</a>.
Mon, 26 Aug 2013 19:43:00 +0000
https://www.datumedge.co.uk/2013/08/26/upgradingintelvideodriversforxubuntu1304/
https://www.datumedge.co.uk/2013/08/26/upgradingintelvideodriversforxubuntu1304/