Tool chain

Overview of operations

Customization of X-Plane10 is covered in the Installation instructions. This page describes how I created those customization files: airport layouts, nav data and maps, the base mesh, modified terrains, and bitmap versions of the base mesh.

Here is a summary plan for generating airports and scenery for Mars:

1. Use tools/WED to create/update airports layouts.
2. Use tools/apt.sh to consolidate all individual apt.dat files.
3. Use apts -l -b, and the consolidated apt.dat, to create:
   • a shape file that contains airport locations (used when creating maps with QGIS).
   • a shape file that contains airport boundaries (used when creating .hgt files, and in meshscript.txt).
4. Use apts -n -j, and the consolidated apt.dat, to create:
   • a nav.dat file that contains NDB and ILS details.
   • an aptData.js file that contains runway, NDB, and ILS data; for HTML document.
5. Use qgis to generate a map of Mars, showing the location of all airports. [Optional]
6. Use bitmapper to generate files xxxx.dds and xxxx-nrm.png, for rendering distant scenery.
7. Use meshxes, and some suitable climate profile, to generate 176x360 .xes files.
8. Use meshhgt -vM, and MOLA data, to generate 176x360 .hgt files. Flatten airports as well.
9. Use makemesh.sh to create 176x360 base tiles. For each tile:
   1. Run MeshTool tile.txt tile.xes tile.hgt borderData/ tile.dsf
   2. Compress tile.dsf
   3. Move the compressed tile to some directory "Earth nav data/±n0±nn0/"
10. Modify selected .ter and .dds files.
11. Copy "Earth nav data/*" to a subdirectory of X-Plane, and finally test.

All these notes assume (but don't require) a Linux system, with about 200 GB of free disk space and at least 4 GB of RAM. I use GNU g++ for compiling the ~7000 lines of C code developed for this project.

The following steps involve a lot of modifications to core X-Plane resource files. Consequently I recommend creating a separate copy of X-Plane, to be customized for Mars: I put my copy in a directory called M-Plane10/. Furthermore, the instructions below assume that the tool chain files are installed in a sibling directory. This relationship makes it simpler to transfer files between M-Plane10/ and marsPackage/.

|-- M-Plane10/   Contains clone of X-Plane10
|   |-- Aircraft/
|   |-- Global Scenery/
|   |-- tools/
|   :     :
|   |-- Log.txt
|   :     :
|-- marsPackage/  Contains tool chain files
|   |-- bitmapper.c
|   |-- meshhgt.c
|   :     :

As further preparation for what is to follow, you will need to use 'make' to compile the code, and ensure that the MOLA dataset is present. The full MOLA dataset can be downloaded as 16 simple 128 MB height files:

   Download MOLA data by FTP to ./molaData/
   for lat in 88n 44n 00n 44s; do
   for lon in 000 090 180 270; do
       wget pds-geosciences.wustl.edu/mgs/mgs-m-mola-5-megdr-l3-v1/mgsl_300x/meg128/megt${lat}${lon}hb.img
       wget pds-geosciences.wustl.edu/mgs/mgs-m-mola-5-megdr-l3-v1/mgsl_300x/meg128/megt${lat}${lon}hb.lbl   Optional
   done; done

Download Laminar tools

If you already have WED then you can direct it to M-Plane10 by editing the preference file, ~/.WED.prefs. Otherwise, it can be downloaded from Laminar:

cd M-Plane10   Contains clone of X-Plane10
mkdir tools
cd tools
wget https://developer.x-plane.com/tools/worldeditor/wed_lin_150r3.zip
unzip wed_lin_150r3.zip

Get MeshTool and associated XPTools:

cd marsPackage   Contains tool chain files
mkdir dump
wget https://dev.x-plane.com/download/tools/meshtool_lin_300b1.zip
wget https://dev.x-plane.com/download/tools/config_MT3.zip
wget https://dev.x-plane.com/download/tools/xptools_lin_15-3.zip
unzip meshtool_lin_300b1.zip  MeshTool
unzip config_MT3.zip          config/*.txt (45 files)
unzip xptools_lin_15-3.zip    ObjView XGrinder tools/DDSTool tools/DSFTool
rm -rf __MACOSX

The program versions indicated above are what I used. They date from 2015, and were compatible with all versions of X-Plane10. However don't feel bound to those specific versions. There are both newer and older versions of these programs that may be better in some way.

1: tools/WED

The airports in the package are created and maintained with the help of Laminar's World EDitor (WED). I used WED to ensure each airport has (a) a Tower Viewpoint, (b) a light beacon, and (c) an airport boundary. I also checked that elevations were correct, and that the runways were properly labelled. All these are essential for later steps.

Once the ICAO code and runway labels are settled, then suitable entries need to be added to util/aptDat.cpp. The entries comprise ILS frequencies, glideslopes, and suchlike: information that will be output to the navaid file (see step 4 below).

    Example insertion into util/aptDat.cpp
        {"MAAF:17", 11030, 25, "IMAE", "ILS-cat-I", 3.0},
        {"MAAF:35", 11130, 25, "IMAF", "ILS-cat-I", 3.0},

The inherited airports sometimes had taxiways, sometimes not. I didn't add much to these airports' layouts, so they still look rather empty.

Pre-requisites

As explained in the Installation notes, all the Martian airports should be installed under M-Plane10/Custom Scenery/. Then the program WED should be downloaded from Laminar. It doesn't matter where it is put: I installed it to M-Plane10/tools/.

2: tools/apt.sh

The shell script apt.sh consolidates all of the individual apt.dat files created by WED.

#! /bin/bash

# Merges all files 'Custom Scenery/MA??/Earth nav data/apt.dat'
# This script should be located in ~/M_Plane10/tools, and run thus:
#    tools/apt.sh
#    tools/apt.sh > Resources/'default scenery'/'default apt dat'/'Earth nav data'/apt.dat

sceneryPacks='./Custom Scenery'
earthNavData='Earth nav data'
id=0
if [ ! -d "${sceneryPacks}" ]; then echo "ERROR: Run from M-Plane10/"; exit 1; fi

echo "I"
echo "1000 Generated by WorldEditor"
echo

ls -1 -d "${sceneryPacks}"/MA?? |
while read marsDir; do
    let "id = id + 1"
    marsFile=${marsDir}/${earthNavData}/apt.dat
    echo
    tail -n +4 "${marsFile}"  | grep -v '^99$'
done

echo "99"

The consolidated apt.dat is used both as a Resource file by X-Plane10, and as input to the next two steps in the tool chain.

Pre-requisites

The directory M-Plane10/Custom Scenery/ needs to contain all airports in packages named "MAAB", "MACF", etc. Also, the script apt.sh should be located in cd M-Plane10/tools.

Command

cd M-Plane10
tools/apt.sh               This just displays the consolidated data
tools/apt.sh > Resources/'default scenery'/'default apt dat'/'Earth nav data'/apt.dat

3: apts -l -b

The collective apt.dat (created in the previous step) can act as the basis for 'shape' files: files with a format commonly used in spatial mapping. One of the jobs of the program apts is to generate these shape files.

Option -l causes the program to generate shape files containing 'point' information: the latitude/longitude of each airport, along with the airport's ICAO code. See aptShapes/aptMars.*. These files are read by QGIS when creating maps.

Option -b causes the program to generate shape files containing 'polygon' information: the boundary of each airport. See aptShapes/aptBoundary.*. These files are used by meshhgt to define the extent of each airfield. [Airfields are artificially levelled, to ensure that runways are usable.]

Pre-requisites

This program reads M-Plane10/Resources/default scenery/default apt dat/Earth nav data/apt.dat.

Command

Type ./apts -h to see the command line options.

cd marsPackage               This folder must be a sibling to M-Plane10/. It contains all toolchain programs.
./apts -l -b                 Outputs aptShapes/aptMars.* and aptShapes/aptBoundary.*

4: apts -n -j

The collective apt.dat can act as the basis for additional data files. One of the jobs of the program apts is to generate these data files.

Option -n causes the program to generate a file earth_nav.dat. Plausible positions are chosen for NDB, ILS, GS, and DME antennas. Plausible names and frequencies are also assigned. The output file should be copied to M-Plane10/Resources/default data/

Option -j causes the program to generate a file aptData.js. This contains all airport and navaid data, formatted as JavaScript. The advantage of this is that is can be easily imported and rendered by an HTML document.

Pre-requisites

This program reads M-Plane10/Resources/default scenery/default apt dat/Earth nav data/apt.dat. There is additional ILS data hard-coded in util/aptDat.cpp; so remember to update this file after adding or modifying runways in WED.

Command

Type ./apts -h to see the command line options.

./apts -n -j                 Outputs earth_nav.dat and aptData.js
    Move the two files to where they will be used
mv earth_nav.dat ../M-Plane10/Resources/'default data'/
mv aptData.js docs/js/

5: QGIS

QGIS is a GIS program with the ability to load and render raster data. In particular, it can load the 16 raw MOLA files, and colour the pixels according to elevation. Furthermore it can load the shape files generated earlier, so the names and locations of airports can be overlaid on the map.

Maps produced with the help of QGIS might be used as a navigational aid. I put maps into the directory aptMapPDF/. I tried labelling mountains and craters, using the shape files in aptMapReproj/. However those shape files use some other coordinate system (0 to 360, instead of -180 to +180) so it was not successful.

Pre-requisites

The MOLA files megtnnxnnnhb.img need to be downloaded, and placed within ./molaData/, say. [Downloading MOLA data was described at the top of this page.] QGIS can also make use of the shape files produced earlier by apts -l.

Generating a nice PDF map is easy with the configuration file map.qgs.

Command

qgis map.qgs
    1. Project / Print Composers / 'mars1'
    2. Composer / Export as PDF / 'mars/aptMapPDF/mars.pdf'
cp aptMapPDF/mars.pdf docs/images/

6: bitmapper [far | mid]

Apart from the base mesh and overlay DSFs there are two other representations of the scenery. They have lower resolution, and are used to render far distant scenery (latitude resolution ~0.2°), and scenery in the middle distance (latitude resolution ~0.01°). The data for each tile is contained in pairs of files, xxxx.dds and xxxx-nrm.png.

The two members of each pair have complementary roles:

• The xxxx.dds file contains four channels (RGBA):
  • 'RGB' is some texture color. Each component is scaled to the range 0 to 255.
  • 'A' distinguishes land and water (zero for sea/rivers/lakes, 255 for land, and intermediate values along the beaches). It is not opacity. Possibilities are that: (1) it is an unsharp mask used in compositing land and sea textures, or (2) it contains some measure of specularity.
• The xxxx-nrm.png file is more interesting. It too contains four channels (RGBA), however the channels have unconventional roles:
  • 'R' contains the EW component of the surface normal. The quantity has a potential range [-1.0,+1.0], where positive means downhill eastwards. This float is encoded as an 8-bit integer 128 + 127*norm_ew.
  • 'G' contains the NS component of the surface normal. The quantity has a potential range [-1.0,+1.0], where positive means downhill northwards. This float is encoded as an 8-bit integer 128 + 127*norm_ns. [Trust me on the direction. It is true, despite the fact that DSF vertices have normals for which a positive NS component means downhill southwards.]
  • 'B' was a mystery, but I eventually realized the authoritative answer lies in the XPTools source code. The file XESCore/DEMIO.cpp reveals that 'B' is actually the vertical component of the surface normal. Its potential range is [0,1], and is encoded as an 8-bit integer 255*norm_v. Why did I earlier reject this obvious idea?
  • 'A' contains elevations. Each elevation is encoded as an 8-bit integer 255 - int[(418+elevInMetres)/(418+8848)*255]. The implicit range of allowed elevations, -418 – 8848m, is not ideal for Mars. So much of Mars is above or below this range! All we can do is truncate the scaled values, which (unfortunately) creates artificial floors and ceilings. The consequent visual artifacts are discussed below, in the final section.

Bitmaps are 'post-centric' arrays, so the nominal coordinate of each value is well-defined. Each value is the computed from a number of DEM values (since bitmaps are coarse representation of the DEM data) — but which DEM values, exactly? For a given pixel latitude and longitude, should the bitmap value be computed as an average of surrounding elevations, or values to north and east? It is hard to guess the workings of the X-Plane rendering engine. It is also hard to establish this empirically.

Some specs for these pairs of files:

Neighboring tiles overlap. The extreme right column of one tile is identical to column zero of its neighbor. Likewise for the top and bottom edges. This is known as 'post-centric', in the terminology of the DSF format

  display filename.png                     Visual display
  xmag                                     Magnifies, and shows RGB values
  identify -verbose filename.png           Image info
  convert filename.png -alpha Deactivate -color-matrix
        "1.0 0.0 0.0 0.0 0.0 \
         1.0 0.0 0.0 0.0 0.0 \
         1.0 0.0 0.0 0.0 0.0 \
         0.0 0.0 0.0 0.0 0.0 \
         0.0 0.0 0.0 0.0 0.0" filename-R.png  Extract R channel to 8-bit gray-scale
  convert filename.png -alpha Deactivate -color-matrix
        "0.0 1.0 0.0 0.0 0.0 \
         0.0 1.0 0.0 0.0 0.0 \
         0.0 1.0 0.0 0.0 0.0 \
         0.0 0.0 0.0 0.0 0.0 \
         0.0 0.0 0.0 0.0 0.0" filename-G.png  Extract G channel to 8-bit gray-scale
  convert filename.png -alpha Deactivate -color-matrix
        "0.0 0.0 1.0 0.0 0.0 \
         0.0 0.0 1.0 0.0 0.0 \
         0.0 0.0 1.0 0.0 0.0 \
         0.0 0.0 0.0 0.0 0.0 \
         0.0 0.0 0.0 0.0 0.0" filename-B.png  Extract B channel to 8-bit gray-scale
  convert filename.png -alpha Extract filename-A.png  Extract A channel to 8-bit gray-scale

  convert filename.png txt:-                  Convert image to readable ascii
  # ImageMagick pixel enumeration: 1024,1024,255,srgba     Row-wise enumeration, starting at top-left corner
  0,0: (132,130,254,239)  #8482FEEF  srgba(132,130,254,0.937255)
  1,0: (133,126,254,239)  #857EFEEF  srgba(133,126,254,0.937255)
   :            :                 :                :
  1022,1023: (132,131,254,235)  #8483FEEB  srgba(132,131,254,0.921569)
  1023,1023: (128,129,254,236)  #8081FEEC  srgba(128,129,254,0.92549)

  DDSTool --png2dxt5 dummy1 dummy2 in.png out.dds  Convert from .png to .dds
  convert in.dds out.png                           Convert from .dds to .png

X-Plane version 10.36 provided mars{1,2}.dds, but none of the other mid/far distant scenery for Mars. However we can generate our own, using MOLA elevations. For simplicity we choose a uniform color (RGB=172,120,90).

The files generated by bitmapper are:

• For command line option 'far'
  1. docs/mars{1,2}.png used as a figure in the documentation
  2. bitmaps/earth{1,2}.dds for rendering Mars from a high orbit
  3. bitmaps/earth{1,2}-nrm.png to complement the .dds images
• For command line option 'mid'
  1. bitmaps/Earth Orbit Textures/±n0±nn0.dds
  2. bitmaps/Earth Orbit Textures/±n0±nn0-nrm.png

Pre-requisites

This program requires the MOLA elevation dataset molaData/megt00n000hb.img, megt00n090hb.img, ... It also requires the utility program tools/DDSTool, from Laminar's XPTools package.

Command

Type ./bitmapper -h to see the command line options. Then, typically:

./bitmapper far                    About 2 mins
./bitmapper mid                    About 20 mins
rsync -av bitmaps  ../M-Plane10/Resources        Install

7: meshxes

This program generates a complete set of 176x360 .xes files, containing 'climate data' for Mars.

The .xes file format is not documented. However it is fairly easy to dissect a .xes file, and to generate suitable files from scratch.

   Download an arbitrary land class file
   wget https://dev.x-plane.com/update/misc/MeshTool3/+20+120.zip
   unzip ./+20+120.zip                     First unzip
   unzip ./+20+120/+28+121.xes             Second upzip
   Extract definitions
   strings Volumes/Hedge/MT3/+20+120/+28+121.xes > defs.txt
   vim defs.txt                            Eliminate a few lines of junk at the end
   gawk -f defs.awk defs.txt > xesDefs.h   Create a C include file

   BEGIN {n = 0; print "const char *defs[] = {"}
       {print "    \"" $1 "\",   // " n++}
   END {print "};\n#define nDefs (" n ")\n"}

Offset    Size   Example value
000000     4+4   "TOKN" 0002f076
000008   2f06e   NO_VALUE BARE FOREST .......  [6319 such 'definitions']
02f076     4+4   "MAP1" 00000054
02f07e     4+4   "MAP2" 0000004c
02f086    17x4   uint32 map2[17] = {0,0,1,1,1,0,0,0,1,0,0,0,0,0,0,0,0};
02f0ca     4+4   "MSH1" 00000080 
02f0d2     4+4   "mesh" 00000024 
02f0da     7x4   sint32 mesh[7] = {1,1,-1,0,0,0,0};
02f0f6     4+4   "dat1" 00000054
02f0fe    38x2   uint16 dat1[38] = {0,0,0,0,0,0,0,0,      0,0,0,0,0,0,0,0,
                           1,0,0,0x3ff0,0,0,0,0, 0,0,0x8000,0xffff,0,0,0,0,
                           0,0,0,0,0,0};
02f14a     4+4   "DEMd" 00000028
02f152       4   0000007
02f156     7x4   uint32 demd[7] = {9,10,11,25,27,28,29};

02f172     4x4   uint32 type =  9, size = 000e130, dim = 120, dim = 120
02f182     4x8   f64 lat0, lon0, lat1, lon1
02f1a2  120x120  f32 val[120x120]  (e.g. dem_Temperature ~= 25.2 C)

03d2a2     4x4   uint32 type = 10, size = 000e130, dim = 120, dim = 120
03d2b2     4x8   f64 lat0, lon0, lat1, lon1
03d2d2  120x120  f32 val[120x120]  (e.g. dem_TemperatureRange ~= 30.2 C)

04b3d2     4x4   uint32 type = 11, size = 000e130, dim = 120, dim = 120
04b3e2     4x8   f64 lat0, lon0, lat1, lon1
04b402  120x120  f32 val[120x120]  (e.g. dem_Rainfall ~= 214.0 mm)

059502     4x4   uint32 type = 25, size = 0040a094 dim = 1029, dim = 1029
059512     4x8   f64 lat0, lon0, lat1, lon1
059532 1029x1029 f32 val[1029x1029]  (e.g. dem_LandUse = 205.0 == lu_globcover_SPARSE)

463596     4x4   uint32 type = 27, size = 00003a54, dim = 61, dim = 61
4635a6     4x8   f64 lat0, lon0, lat1, lon1
4635c6   61x61   f32 val[61x61]  (e.g. dem_SoilStyle = 4691.0 == SOIL_RED)

466fea     4x4   uint32 type = 28, size = 00000214, dim = 11, dim = 11
466ffa     4x8   f64 lat0, lon0, lat1, lon1
46701a   11x11   f32 val[11x11]  (e.g. dem_AgriStyle = 0.0 == NO_VALUE)

4671fe     4x4   uint32 type = 29, size = 00000214, dim = 11, dim = 11
466ffa     4x8   f64 lat0, lon0, lat1, lon1
46701a   11x11   f32 val[11x11]  (e.g. dem_ClimStyle = 0.0 == NO_VALUE)

467412   File size

Col:  0            1               2               3                4               5               6
-------------------------------------------------------------------------------------------------------------
#TERRAIN_RULE   TERRAIN         ZONING          LAND USE        SOIL_STYLE      AGRI_STYLE      CLIM_STYLE   
#                                                                                                            
-------------------------------------------------------------------------------------------------------------
TERRAIN_RULE    terrain_Airport NO_VALUE        NO_VALUE        NO_VALUE        NO_VALUE        NO_VALUE     
TERRAIN_RULE    NO_VALUE        NO_VALUE        *various*       NO_VALUE        NO_VALUE        CLIM_TUNDRA  
TERRAIN_RULE    NO_VALUE        NO_VALUE        *various*       NO_VALUE        NO_VALUE        CLIM_TUNDRA  
     :             :               :                :              :               :               :


Col:7     8        9       10      11      12      13     14      15       16     17
-----------------------------------------------------------------------------------------
   ELEVATION         SLOPE           TEMP          TEMP RANGE       RAINFALL     WATER   
  MIN     MAX     MIN     MAX     MIN     MAX     MIN     MAX     MIN     MAX     PROX   
-----------------------------------------------------------------------------------------
   0       0       0       0     -999    -8.0      0       0       0       55      0     
   0       0       8      19     -999    -8.0      0       0       0       55      0     
   0       0       0       8     -999    -8.0      0       0       0       55      0     
   :       :       :       :        :      :       :       :       :        :      :


Col:18    19      20      21      22      23               24
------------------------------------------------------------------------
  REL ELEV       ELEV RANGE       LATITUDE                NAME          
 MIN     MAX     MIN     MAX     MIN     MAX                            
------------------------------------------------------------------------
  0        0      0       0       0       80      terrain10/apt_vpol_dry
  0       0.1     0     650       0       80      terrain10/tun_spr_pol_dry_sflat
  0       0.1     0     650       0       80      terrain10/tun_spr_pol_dry_fl   
  :        :      :       :       :        :               :

Note:
 1  In the above '*various*' stands for:
    lu_globcover_BARE_ROCKS,lu_globcover_BARE_CONSOLIDATED,lu_globcover_BARE_SCREE,lu_globcover_SALT,
    lu_globcover_SALT_HEM,lu_globcover_MORAINE,lu_globcover_MORAINE,lu_globcover_MINE
 2  Values 0 and NO_VALUE generally mean 'ignore'.
 3  Mars data extends to 88 degrees of latitude.  To get terrain_rules.txt to work at extreme
    latitudes it is necessary to modify column 23, replacing '80' with '88'.

How MeshTool uses .xes files

The bottom line is that the .xes specifications (plus .hgt values) will help MeshTool to choose the semi-plausible terrain files. (BTW, terrain files are located in Resources/default scenery/1000 world terrain/terrain10/.) The list below is not quite complete, but you get the picture: there is a good spread of terrains selected by MeshTool, Nevertheless, they are standard X-Plane terrains, originally intended for Earth. It would of course be preferable to create terrains that are customized for Mars, however I am not a graphic artist, a geographer, nor Mars geology specialist. Anyone?

    At most latitudes
    lib/g10/terrain10/red_spr_{hot|vhot}_dry_sflat     Rendered as sand
    lib/g10/terrain10/red_spr_{hot|vhot}_dry_fl        Rendered as sand
    lib/g10/terrain10/red_{rock|sand}_{hot|vhot}_dry_steep.ter
    lib/g10/terrain10/red_{rock|sand}_{hot|vhot}_dry_hill.ter
    lib/g10/terrain10/red_{rock|sand}_{hot|vhot}_dry_sflat.ter
    lib/g10/terrain10/red_{rock|sand}_{hot|vhot}_dry_fl_md.ter
    lib/g10/terrain10/red_{rock|sand}_{hot|vhot}_dry_fl_tp.ter
    lib/g10/terrain10/red_{rock|sand}_{hot|vhot}_dry_fl.ter
    lib/g10/terrain10/apt_{hot|vhot}_dry.ter
    At polar latitudes
    lib/g10/terrain10/{rock|sand}_{vpol|vcld}_dry_steep.ter
    lib/g10/terrain10/{rock|sand}_{vpol|vcld}_dry_hill.ter
    lib/g10/terrain10/{rock|sand}_{vpol|vcld}_dry_sflat.ter
    lib/g10/terrain10/{rock|sand}_{vpol|vcld}_dry_fl_md.ter
    lib/g10/terrain10/{rock|sand}_{vpol|vcld}_dry_fl_tp.ter
    lib/g10/terrain10/{rock|sand}_{vpol|vcld}_dry_fl.ter
    lib/g10/terrain10/snow_{vpol|vcld}_dry_flat.ter
    lib/g10/terrain10/snow_{vpol|vcld}_dry_flat_lo.ter
    lib/g10/terrain10/snow_{vpol|vcld}_dry_sflat.ter
    lib/g10/terrain10/snow_{vpol|vcld}_dry_sflat_lo.ter
    lib/g10/terrain10/snow_{vpol|vcld}_dry_hill.ter
    lib/g10/terrain10/snow_{vpol|vcld}_dry_steep.ter
    lib/g10/terrain10/apt_{vpol|vcld}_dry.ter

Pre-requisites

This program requires that landType has already been run, so that the directory TerrainAssignments/ is fully populated. The program landType digests geological land class data for Mars, obtainable from USGS. It reads polygon shape files, rasterizes the data, and writes 176x360 1x1 degree maps to files under TerrainAssignments/. This geological data is referred to by meshxes.

Command

Type ./meshxes -h to see the command line options.

./meshxes                     About 2 hours

8: meshhgt -vM

This program reads MOLA elevation data (the 16 .img files in molaData/) and outputs 63360 .hgt files. Each file covers 1x1 degree, and has a format suitable for use with MeshTool. The hierarchical directory structure of the output matches that of other scenery files: "hgt/Earth nav data/±n0±nn0/±nn±nnn.hgt".

The raw MOLA elevations are interpolated – from 128 points per degree to 300 points per degree – to help MeshTool introduce more spatial variations. This level of interpolation ultimately leads to 17 GB of .dsf files. Finer interpolation results in nicer looking scenery, but then downloading becomes a hurdle.

This program also performs flattening within each airport's boundary: natural height variations around the mean are reduced to 3%. This is intended to make the runway flat enough to use, but not unnaturally flat.

A minor additional job for this program is to build a replacement for Resources/dsf_files. This resource file is 64805 bytes long, comprising: the character 'i', the number 860 as a little-endian 4-byte integer, and 180×360 bytes (0x00 for empty sea, 0x01 implies a DSF tile is available). It can be inspected with tail -c +6 dsf_files | tr '\000\001' ' #' | fold -b -w 180. Row 0 contains longitude -180, row 1 contains longitude -179, etc; and latitudes run from -90 (leftmost) to +89 (rightmost).

Pre-requisites

This program requires

1. 16 MOLA files: molaData/megt00n000hb.img, megt00n090hb.img, ...
2. Boundary shape files: aptShapes/aptBoundary.{shp,shx,dbf}

Command

Type ./meshhgt -h to see the command line options. The option -vM is recommended.

./meshhgt -vM                    About 3 mins
mv dsf_files ../M-Plane10/Resources/

9: makemesh.sh

MeshTool requires a script file, containing supplementary data. The command MESH_SPECS max_added_points max_error prevents MeshTool getting carried away trying to match neighboring tiles, and conforming to the supplied DEM. The command BACKGROUND base_terrain means that terrain is generated automatically (with the help of config/terrain_rules.txt). The command SHAPEFILE_TERRAIN terrain_Airport aptShapes/aptBoundary.shp means that suitable airport terrains will be 'burnt into' the base mesh. This file will serve for all 63360 tiles:

MESH_SPECS 78000 6.5
BACKGROUND terrain_Natural
SHAPEFILE_TERRAIN terrain_Airport aptShapes/aptBoundary.shp

MeshTool is quite easy to run, once all the preparatory work is done. However the shell script makemesh.sh is handy because:

• paths are like hgt/Earth nav data/±n0±nn0/±nn±nnn.hgt, so are tedious to type, and tricky to get right
• MeshTool needs to be called 176x360 = 63360 times
• we also need to compress the output files (with 7zr)

#! /bin/bash

# This runs MeshTool multiple time, to create a full set
# of .dsf files.  Presently these files are not compressed.

# This script accepts  0, 2 or 4 command line arguments
#    makemesh.sh                 -88 to +87    -180 to +179
#    makemesh.sh lat0 lat1       lat0 to lat1  -180 to +179
#    makemesh.sh lat0 lat1 lon0 lon1 
# E.g.
#   ./makemesh.sh -80 -61
#   ./makemesh.sh -20 -11 -60 -51

# We assume that .xes and .hgt files have already been
# created, for example by
#    ./meshxes -vM -dR
#    ./meshhgt -vM

# Enable Ctrl-C; cursor must be in the terminal window
trap exit 2 15

lat0=-88
lat1=87
lon0=-180
lon1=179

if [ $# -eq 0 ]; then
   # Process the full range of lats and lons
   # However runtime would be unreasonably long, so exit
   echo "Generating the full 176x360 dataset will take ~10 days!"
   echo "Type './makemesh.sh -h' to see more sensible options"
   exit 0;
elif [ $# -eq 2 ]; then
  lat0=${1}
  lat1=${2}
elif [ $# -eq 4 ]; then
  lat0=${1}
  lat1=${2}
  lon0=${3}
  lon1=${4}
else 
  echo "Usage:"
  echo "  ${0}                      Generate -88 to 87    & -180 to 179  DISABLED!"
  echo "  ${0} lat0 lat1            Generate lat0 to lat1 & -180 to 179"
  echo "  ${0} lat0 lat1 lon0 lon1  Generate lat0 to lat1 & lon0 to lon1"
  exit 2
fi  

# Simple validity checks
echo "Latitudes: " $lat0 "to" $lat1
echo "Longitudes: " $lon0 "to" $lon1
if [ $lat0 -gt $lat1 ]; then
   echo "ERROR: Latitudes need to be swapped!"
  exit 2
elif [ $lon0 -gt $lon1 ]; then
   echo "ERROR: Longitudes need to be swapped!"
  exit 2
fi
if [ $lat0 -lt -88 -o $lat0 -gt 87 -o $lat1 -lt -88 -o $lat1 -gt 87 ]; then
   echo "ERROR: Latitudes need to be in the range -88 to 87!"
  exit 2
elif [ $lon0 -lt -180 -o $lon0 -gt 179 -o $lon1 -lt -180 -o $lon1 -gt 179 ]; then
   echo "ERROR: Longitudes need to be in the range -180 to 179!"
  exit 2
fi

# Run MeshTool
for((lat=$lat0; lat<=$lat1; lat++)); do
for((lon=$lon0; lon<=$lon1; lon++)); do
   let "latR=((lat+90)/10*10)-90"
   let "lonR=((lon+180)/10*10)-180"
   latRF=$(gawk '{printf "%+03d\n", $0;}' <<< $latR)
   lonRF=$(gawk '{printf "%+04d\n", $0;}' <<< $lonR)
   latF=$(gawk '{printf "%+03d\n", $0;}' <<< $lat)
   lonF=$(gawk '{printf "%+04d\n", $0;}' <<< $lon)

   # Create numeric directory and file names
   d1=${latRF}${lonRF}
   fn=${latF}${lonF}

   # Create command line arguments
   dir="Earth nav data/"${d1}
   xes="xes/"${d1}/${fn}.xes
   hgt="hgt/"${d1}/${fn}.hgt
   out1="./"${fn}.dsf
   out2="Earth nav data/"${d1}/${fn}.dsf

   echo 
   echo
   echo "-----------------------------------------------------------"
   echo ./MeshTool meshscript.txt $xes $hgt dump $out1
   echo "-----------------------------------------------------------"
   mkdir -p "$dir"
   ./MeshTool meshscript.txt $xes $hgt dump $out1
   if [ ! -e $out1 ]; then exit 1; fi
   rm -f "$out2"
   7zr a -mx=9 -mfb=64 -md=32m "$out2" $out1
   rm $out1
done
done

Pre-requisites

This script requires

1. A file meshscript.txt
2. A complete set of .xes files, xes/*/*.xes
3. A complete set of .hgt files, hgt/*/*.hgt

It also makes use of files in ./config/ and ./aptShapes/. N.B. A crucial modification needs to be made to ./config/terrain_rules.txt. By default, rules are limited to latitudes within ±80°, and MeshTool will crash for more extreme latitudes. However the fix is simple: replace '80' with '88' in column 23. [One way to do this is sed 's/80\tterrain10/88\tterrain10/' terrain_rules.txt]

Command

Type ./makemesh -h to see the command line options. The script accepts command line arguments, which enables it to generate only a specific subset of the full 63360 set.

./makemesh.sh lat0 lat1              Do only latitudes lat0 to lat1 (inclusive); longitudes -180 to 179 are implied
./makemesh.sh lat0 lat1 lon0 lon1    Do only latitudes lat0 to lat1, and longitudes lon0 to lon1 (inclusive)
./makemesh.sh 4 4 -38 -38            Example of generation of a single .dsf tile, whose SW corner is at lat=4, lon=-38

Latitude arguments must be integers in the range -88 to 87 (inclusive). Longitude arguments must be integers in the range -180 to 179 (inclusive).

It is possible to use the shell script to generate all 63360 .dsf files — which will take about 10 days. However the smarter thing to do is to run several instances of makemesh.sh simultaneously, to make best use of multiple cores. The commands for 6-way parallel construction of .dsf files are something like:

   This is sometimes necessary, and harmless at other times
   rm -rf dump/*
   Execute these commands in different term windows
   ./makemesh.sh -88 -58
   ./makemesh.sh -57 -29
   ./makemesh.sh -28  -1
   ./makemesh.sh   0  27
   ./makemesh.sh  28  56
   ./makemesh.sh  57  87

With six processes running in parallel the total runtime is about 48 hours.

    7zr e -o/tmp ./"Earth nav data"/-20-060/-12-059.dsf
    ./parseDsf -vv /tmp/-12-059.dsf  | grep terrain10 | sort

10: Modify .ter and .dds files

The 176x360 .xes files (see above) each consist of coarse maps of temperature, rainfall, land usage, etc. Meshtool combines these with config/terrain_rules.txt to choose textures for each triangle of the base mesh.

The file config/terrain_rules.txt contains pointers to .ter files (meta texture files), which themselves contain pointers to .dds files (actual texture files). Both should to be modified to create more Mars-like terrain textures.

Minor edits to .ter files

The stage involving meshxes specified a red, dry, rocky terrain for all tiles. This goes a long way to creating Mars-like scenery. We can further improve the appearance of the terrain by converting flat areas to sand dunes, mimicking the dusty dunes of Mars. Another job is to replace 'apt_tmp_dry', since that terrain is far too green.

The following recipe involves renaming .ter. I didn't investigate editing .ter files, although that might be a good course, if I had .png images for Mars, or if I wanted to use the alternative terrain files in Resources/'default scenery'/'1000 world terrain'/pol9/.

Here is how to make the necessary changes to selected .ter files:

cd ~/M-Plane10/
tex=Resources/'default scenery'/'1000 world terrain'/terrain10
   Dry flat 'spr' terrains → sand
for zone in pol vcld cld; do
    ln -sf sand_${zone}_dry_fl.ter    "${tex}"/spr_${zone}_dry_fl.ter
    ln -sf sand_${zone}_dry_sflat.ter "${tex}"/spr_${zone}_dry_sflat.ter
done
for zone in tmp wrm hot vhot; do
    ln -sf red_sand_${zone}_dry_fl.ter    "${tex}"/red_spr_${zone}_dry_fl.ter
    ln -sf red_sand_${zone}_dry_sflat.ter "${tex}"/red_spr_${zone}_dry_sflat.ter
done

   Temperate airports → warm
ln -sf apt_wrm_dry.ter "${tex}"/apt_tmp_dry.ter

Minor edits to .dds files

By default, sand is rather white in appearance. To make it more Mars-like we can redden the .dds images used for rendering sand. One way is as follows.

    Use convert, described in https://www.imagemagick.org/script/command-line-options.php
mkdir -p dds/soil       New directory, to hold working copy of updated textures
tex=~/M-Plane10/Resources/'default scenery'/'1000 world terrain'/textures10
for f in "${tex}"/soil/sand_*.dds; do
    pathTail=${f##*/}
    convert -modulate 80,140,95 "$f" /tmp/tex.png                Alter brightness,saturation,hue
    tools/DDSTool --png2dxt5 dummy1 dummy2 /tmp/tex.png dds/soil/${pathTail}       .png --> .dds
    echo "Updated ${pathTail}"
done
    Install updated textures
cp dds/soil/*.dds "${tex}"/soil/

Another minor edit is to eliminate tinges of green from four images in Resources/default scenery/1000 world terrain/textures10/soil/, namely red_rock_wrm_dry_sflat_d.dds, red_rock_wrm_dry_steep_d.dds, red_rock_wrm_dry_hill_d.dds, and red_rock_tmp_dry_hill_d.dds. Many terrain files reference these images, so the suggestion of green bushes therein is quite distracting. However we can use ImageMagick to create a mask, then modulate greens into browns.

tex=~/M-Plane10/Resources/'default scenery'/'1000 world terrain'/textures10
for f in "${tex}"/soil/red_rock_*.dds; do
    1. Identify pixels with both green hue and good saturation
    convert "$f" -colorspace HSL -channel Hue,Saturation -separate +channel \
            \( -clone 0 -background none -fuzz 10% +transparent gray20 \)   \    select hue=20 ± 10%
            \( -clone 1 -background none -fuzz 10% -transparent gray20 \)   \    select all except saturation=20 ± 10%
            -delete 0,1 -alpha extract -compose multiply -composite -negate \
            /tmp/mask.png
    2. Make these pixels 50% brighter, 50% less vivid, and rotate hues by -25*(180/100)°: green → brown
    convert "$f" -mask /tmp/mask.png      \
            -modulate 150,50,75 +mask     \          modulate brightness,saturation,hue
            /tmp/tex.png
    3. Compress
    tools/DDSTool --png2dxt5 dummy1 dummy2 /tmp/tex.png dds/soil/${f##*/}     .png --> .dds
done
    Install updated textures
cp dds/soil/*.dds "${tex}"/soil/

We also need to redden the texture that is used for 'cld', 'vcld', 'pol' and 'vpol' airport terrains. The terrain files apt_*dry.ter all reference one texture file ../textures10/apt/apt_cld_dry.dds. So, similarly to the above cases:

tex=~/M-Plane10/Resources/'default scenery'/'1000 world terrain'/textures10
convert -modulate 100,80,90 "${tex}"/apt/apt_cld_dry.dds /tmp/tex.png
tools/DDSTool --png2dxt5 dummy1 dummy2 /tmp/tex.png dds/apt/apt_cld_dry.dds
    Install updated texture
cp dds/apt/*.dds "${tex}"/apt/

11: Installation/distribution

To test the new base mesh, copy the contents of Earth nav data/ to a suitable X-Plane directory. I prefer the first of the two alternatives below:

    rsync -av "Earth nav data" "../M-Plane10/Global Scenery/X-Plane 10 Global Scenery"     The effect is to overwrite the base mesh.  RECOMMENDED
         OR
    rsync -av "Earth nav data" "../M-Plane10/Custom Scenery/Trial-yyyymmdd"                The effect is to override the base mesh with a new custom package

A further step may be to bundle all scenery as zip files, for easier distribution. There are makefile commands for this:

    Assemble files in ./distro/
    make docs.zip            3 MB
    make mars.zip            Makes 9 files mars[1-9].zip, totalling 17 GB
    make bitmaps.zip         680 MB
    make ter.zip             6 kB
    make dds.zip             80 MB
    make apts.zip            220 kB
    make other               280 kB   6 files: apt.dat earth_nav.dat dsf_files sky_colors_mars.png sun_glow_mars.png README

Workspace

Here is the complete environment for building Mars scenery from scratch:

|-- aptMapPDF/
|   |-- various    maps derived with QGIS
|   :     :
|-- aptMapReproj/
|   |-- various    extra shape files containing labels for maps
|   :     :
|-- aptShapes/
|   |-- various    shape files derived from apt.dat
|   :     :
|-- archive/
|   |-- various
|   :     :
|-- config/        from Laminar
|   |-- absolute_terrain.txt
|   |-- terrain_rules.txt
|   :     :
|-- docs/
|   |-- index.html
|   |-- toolchain.html
|   :     :
|-- dump/          workspace for MeshTool
|-- md5/
|   |-- makefile
|   |-- md5
|   |-- md5main.cpp
|   |-- md5main.o
|   |-- Md5Sum.cpp
|   |-- Md5Sum.h
|   `-- Md5Sum.o
|-- tools/         from Laminar
|   |-- DDSTool
|   `-- DSFTool
|-- shapelib-1.3.0/
|   |-- various    utility code
|   :     :
|-- I1802ABC_Mars_global_geology/
|   |-- various    geological data files needed by landtype
|   :     :
|-- SIM3292_MarsGlobalGeologicGIS_20M/
|   |-- various    geological data files needed by landtype
|   :     :
|-- wms2pol_1.56/        http://www.cestomano.com/mas/xplane/
|   |-- wms2pol.py       Generates draped orthophotos
|   `-- wms2pol.ini
|-- dds/      
|   |-- apt/
|   |   `-- apt_cld_dry.dds    just this files
|   `-- soil/
|       |-- red_rock_*.dds     4 such files
|       `-- sand_*.dds         15 such files
|-- molaData/
|   |-- megt00n000hb.img       16 such files
|   `-- megt00n000hb.lbl       16 such files
|-- bitmaps/
|   |-- earth{1,2}.dds
|   |-- earth{1,2}-nrm.png
|   `-- Earth Orbit Textures/
|       |-- ±n0±nn0.dds       18x36 such files
|       `-- ±n0±nn0-nrm.png   18x36 such files
|-- TerrainAssignments/      
|   `-- ±nn±nnn/        18x36 such directories
|       `-- ±nn±nnn.dat       100 such files
|-- hgt/
|   `-- ±nn±nnn/        18x36 such directories
|       `-- ±nn±nnn.hgt       100 such files
|-- xes/
|   `-- ±nn±nnn/        18x36 such directories
|       `-- ±nn±nnn.xes       100 such files
|-- Earth nav data/
|   `-- ±nn±nnn/        18x36 such directories
|       `-- ±nn±nnn.dsf       100 such files
|-- util/
|   |-- aptDat.cpp
|   |-- aptDat.h
|   |-- aptDat.cpp
|   |-- flatten.c
|   |-- hel.cpp
|   |-- hel.h
|   |-- landTypeEncodings.c
|   |-- landTypeEncodings.h
|   |-- makefile
|   |-- mkdirs.c
|   |-- readMola.c
|   |-- readMola2.c
|   |-- rwy.cpp
|   |-- rwy.h
|   |-- writeDsf2.c
|   |-- writeDsf2.o
|   |-- writeXes.c
|   |-- writeXes.o
|   `-- xesDefs.h
|-- apts        #3 and #4 in tool chain
|-- apts.cpp
|-- bitmapper   #6 in tool chain
|-- bitmapper.c
|-- landType    [Companion to marsmesh]
|-- landType.c
|-- makefile
|-- meshscript.txt
|-- makemesh.sh #9 in tool chain
|-- map.qgs
|-- marsmesh    [Creates simple mesh in one step]
|-- marsmesh.c
|-- meshhgt     #8 in tool chain
|-- meshhgt.c
|-- meshxes     #7 in tool chain
|-- meshxes.c
|-- parseDsf    [Diagnostic]
|-- parseDsf.c
|-- MeshTool    from Laminar
|-- ObjView     from Laminar
`-- XGrinder    from Laminar

Work flow

Practical considerations

Disk usage and runtime is something to be aware of.

Limitations

Here are some limitations. Some are limitations to the Mars packages, and others are limitations inherent to X-Plane10. In no particular order:

MeshTool is clever at varying terrain based on topography and climate. However on it own it doesn't entirely solve the quilt-like effect of repeated tiling. Also the transition between different terrains is too abrupt. I guess both these issues can be addressed by using overlay DSFs.

A related problem is with perception of scale. Mountains can be huge on Mars, but — in the absence of medium scale landmarks for reference — they really don't look very impressive. It is a big problem when flying visually, since you can't judge how far and high mountains are. Again, I guess we need overlay DSFs.

As explained above in the section dealing with meshxes, I went along with the pre-existing configuration files used by MeshTool. I specified a dry climate, red soil and rocks; and left it to MeshTool to choose .ter texture files. Tweaks were made to some .ter and .dds files. Nevertheless, the results may not be as accurate as they could be, and I don't know enough about Martian geology to do any better. As pointed out by Michael Gilliat, orthophotos are the answer, once higher resolution images become available.

There is a big problem with middle distance scenery. Reasonable blending occurs between DSFs and the mid-distance scenery. However mid-distance elevations are encoded in the 'A' channel of the pseudo images Earth Orbit Textures/±n0±nn0-nrm.png, and are limited to elevations in the range -418 to 8848m. Unfortunately half of Mars is below the standard datum, so the middle distance is missing for much of Mars. There is a hazy void in the distance, which is suddenly filled in as the plane's progress causes the next DSF tile to load (or when the renderer falls back to using the far-scenery).

While this is an annoying and obvious artifact, it can be worse. In some situations there is a ghostly surface floating in the sky at elevation = -418m, while the ground elevation is far below. There is no real solution to this, with X-Plane as it is. A partial solution to floating translucent surfaces is to uncheck the Orbital texture option in the menu Settings → Rendering options. This turns off the mid-distance scenery.

The middle distance scenery has realistic elevations and shadowing, but has a uniform hue — generic Mars red-brown. To do any better would involve considerable effort, as it would be necessary to digest all .dsf files, and all the .ter and .dds files referenced therein.

Mars has no magnetic field, so we would like magnetic compasses to point to true north, rather than to (Earth's!) magnetic north. Ideally we would customize X-Plane's model for magnetic declination/variation. Unfortunately Earth's field seems to be hard-coded within X-Plane10, precluding any simple tweaks (this topic has been raised in the forum), nor is the dataref sim/flightmodel/position/magnetic_variation writeable. However some instruments have writeable heading datarefs, so perhaps a plugin can implement a fix, of sorts. Meanwhile, if you want to fly a particular heading, then you will have to add (Earth's) magnetic declination to your desired true heading. [You can work out local declination by going to Settings → Data Input & Output, and enabling item 19, 'magnetic compass'.]