This week I am flying to Japan for a business trip. I have only been there once before for another business trip and I really enjoyed it, so I am really looking forward to going again. Except for the whole 15-hour flight part. That part isn’t so great. But it is good to know that Great Circle Routes can shorten the travel distance (and time) considerably.  Compare the great circle route shown in this picture with the straight heading route from the image below.  Note that it almost passes north of Canada and Alaska, and skirts along the eastern coast of Siberia.  In fact, when we take off, we will be heading close to a northerly direction (approximately 332 degrees).  Along the way, the plane will have to alter its heading to stay on this route.  In fact, as we approach Japan, we will be heading almost due south.  The entire route will be a big sweeping left turn.  Every 30 minutes or so the plane will alter course by about 8 degrees to the left.  This seems a little confusing because the equator is a great circle, and if I flew a path between two cities on the equator, that would indeed be the great circle route and the shortest route.  But I wouldn’t have to change course from either due east or west.  That’s only because the equator just happens to be a great circle that lies on a cardinal heading (east-west).  Also, if you were flying a great circle route that had you going either due north or due south, you would not have to alter your heading during the trip.  In this case you would be flying along a line of longitude.  Other than these two special cases, all other great circle trips require periodic heading alterations to stay on course.  So let’s analyze my trip.

The Newark airport is at 40° 41′ N 074° 10′ W and the Narita airport in Tokyo is at 34° 45′ N 140° 22′ E.

Using regular trigonometry we can calculate the distance between these two airports using the fact that on average a degree of latitude is about 69.1 miles long and a degree of longitude is about 53.0 miles long. So all we do is calculate how many degrees of change in latitude and how many degrees of change in longitude there are between the two points, and multiply it by the mileage factors. Now, this formula makes one big (incorrect) assumption. That is, that the distance between two points long a line of longitude is the same at all latitudes.  In just a minute, we will correct for that. So, using Pythagorean, and taking the north-south change as the “X” and the east-west change as the “Y”, we can calculate the length of the hypotenuse.

We are basically trying to calculate the length of this line:

That is the route you would take if you pointed your plane right at Tokyo and flew straight at it the whole way.  As you know, when you fly on a great circle route, you have to periodically alter your heading  to stay on the route.

$x= 69.1 * (lat2 - lat1)$
$y = 53.0 * (lon2 - lon1)$

For the math to work out, north latitudes are positive and south are negative; east longitudes are positive and west are negative, and we need to convert the degree-minutes-seconds into decimal degrees

$dist = \sqrt{x^2 + y^2}$

Newark, NJ
$lat1 = 40+41/60 = 40.6833$
$lon1 = -74+10/60 = -74.1667$

Tokyo, Japan
$lat2 = 34+45/60 = 34.75$
$lon2 = 140+22/60 = 140.3667$

$x = 69.1 * (34.75 - 40.6833) = -409.9910$
$y = 53.0 * (140.3667 - -74.1667) = 11370.2702$

$dist = \sqrt{11370.2702^2 + -409.9910^2} = 11377.6596 miles$

But now let’s try and adjust for the difference for different distances at different degrees of latitude.

The improved formulas are:

$AvgLat = (lat2 + lat1)/2$
$x = 69.1 * (lat2 - lat1)$
$y = 69.1 * (lon2 - lon1) * cos(AvgLat)$

When taking the cos calculation, make sure you are calculating in degrees mode, not radians.

$dist = sqrt(x^2 + y^2)$

Substituting our values in

$AvgLat = (40.6833 - 34.75) / 2 = 37.7167$
$x = 69.1 * (34.75 - 40.6833) = -409.9910$
$y = 69.1 * (140.3667 - (-74.1667)) * \cos(37.7167) = 11726.6589$

$dist = \sqrt{11726.6589^2 + (-409.9910^2)} = 11733.8239$

Which, interestingly, is higher than the first approximation, yet is considered more accurate. However, it probably is pretty close to the actual distance that would be traversed if that route were indeed chosen. The number does make sense though.  The circumference of the earth is about 24,000 miles.  I am not going quite half way around, so a number a little less than half seems to be about right.  I would not recommend choosing that route though because the great circle route is shorter by several THOUSAND miles. Let’s calculate it.

First, convert all the lats and longs to radians.

Newark, NJ
$lat1 = 40.6833 / 57.29577951 = 0.7101$
$lon1 = -74.1667 / 57.29577951 = -1.2945$

Tokyo, Japan
$lat2 = 34.75 / 57.29577951 = 0.6065$
$lon2 = 140.3667 / 57.29577951 = 2.4499$

The formula for great circle distance is

$3963.0 * arccos[\sin(lat1) * \sin(lat2) + \cos(lat1) * \cos(lat2) * \cos(lon2 - lon1)]$ $3963.0 * arccos[\sin(.7101) * \sin(.6065) + \cos(.7101) * \cos(2.4499-(-1.2945))] = 6788.5935$

For these calculations, you will need to do the trig functions in radians mode.

Google earth calculated 6724.76 miles. It looks like they are off by about 60 miles

Actually, I did some rounding along the way here, and these calculations are very prone to precision errors.

But what a difference!  As you can clearly see, the great circle route is the way to go here.

Finally, even though this is the shortest route, it may not be the fastest route. The airlines are pretty smart about taking other things into consideration, namely the high altitude weather. They like to fly with the wind on their tails to give them a little boost. Sometimes it can be quite significant.

Now, I have not gone into why great circle routes are the shortest routes.  Hopefully you already understand that, but if you are a little rusty on that concept, there is plenty of reading material available on the web.  But as you can see from our simple calculations here, the distance saved is significant by choosing a great circle route.

The formulas for calculating great circle distances are well documented and are known as the spherical laws of cosines.  I used this web page for most of my verification.

In what may be the simplest and most elegant formula of all time, Leonhard Euler discovered the relationship between all of the most elemental and primal constants in our mathematical system.  Sometimes called Euler’s Identity, it isn’t a very useful formula because there aren’t any variables in the equation, but you have to admit, it is amazing.  No other formula can claim to combine and relate so many of the most important numbers is such a simple way.

$e^{i \pi} + 1 = 0$

Look at it.  Ain’t it beautiful?  It’s got:

The constant e (2.718..)

The constant $\pi$ (3.141..)

The constant $i$ ($i$ is the square root of -1)

The constant 1

The constant 0

And the mathematical operations of addition, multiplication, and exponentiation.

One simple equation that pulls it all together.  Who would have thought?  Euler, that’s who.  Good job.  I’m impressed.