Generating an accurate altitude map for DCS/FC

[RvEYoda]

This question goes out to both ED developers but also all forum members. All help is appreciated :).

 

I am thinking of ways to generate an accurate altitude map/moving map for external awacs/instrument/dlink purposes primarily, both fc and dcs.

 

I know that FC/DCS uses a flat world projection of the selected theater and this does not perfectly match for example google earth/similar things all that well: the further you get from the sim map "center"(align point?) the more wrong the map would be. For example tacview, has a very nice looking map with altitude but it quite often looks like you fly through a mountain if you are not close to the map center.

 

That being said, I am not going to give up that easily. I am considering a few ways we migth be able to generate an accurate map.

 

1. Perform some mathematical modifications to the open databases to get them accurate enough with the sim. We must also consider that other regions than black sea/nevada could be used in the future, and not fix ourselves completely to a center at a specific lat/lon.

 

2. If ED would allow us to use what is stored in the sim currently (though this is probably not possible due to IP reasons, still it probably wont hurt to ask). Maybe there is something here? "Eagle Dynamics\DCS A-10C\Bazar\Map". We would need to learn the format though.

 

3. Generate our own database. This wont be easy, but NOT impossible. In the config/export.lua there is a function

LoGetAltitude(x, z) -- (args - 2 : meters, results - 1 : altitude above terrain surface, meters)

which In principle could let us build our own database. I am not sure what kind of data storage/representation is the best to use here for our database, but I would guess that we would not want to use sime kind of altitude table, but rather some form of curve representation, and highly adaptive. Building such a database could be quite...challanging :). Maybe there is someone here with experience in such a subject.

Edited August 1, 2011 by =RvE=Yoda

[RvEYoda]

You dont need an own database. All you need is to find a ways how to implement free available SRTM data (NASA, Google..whatever ..free and accurate) into the sim.

I know, that this has been achieved in FSX with tileproxy (basically on the fly), which does not only layer the photomaps, but also implements nicely the elevation data extracted from google earth.

But how you implement THAT with AWACS or generally with lockon in terms of collisions and radar....no idea, because no clue how the code is made and where to grab it.

 

 

The photorealistic overlay may be distracting, but more importantly is, the elevation data extracted from google earth here. FSX has not that mesh detail. Maybe it can help finding parallel solution approaches.

Tileproxy temporarily saves those datas in FSX, but there should be ways to rip that data as raw data for implementation into any sim.

Performance-wise, i am pretty confident FC2 or DCS can handle that easily.

Actually, i can imagine that with simple solutions DCS products can be optimized more beautifully AND more efficient ...just a feeling in the guts.

 

I think you misunderstood my post. I have no intention of putting anything new into the game.

 

My question is how to map the current mathematical representation that the game has of lat-long-altitudes to a map display on an external monitor/moving map. Because the current implementation is a flat world, you can't just use already existing databases - They will miss, and I require at least that you will see someone flying in a canyon in both the game and the external monitor ;)

Edited August 1, 2011 by =RvE=Yoda

[aaron886]

Yoda... The elevation data in the game is actually a proprietary 3d model format, with multiple LODs. It would be difficult at best to get data from that file. (lsa2 file.)

 

You might be better off using the actual satellite data (perhaps SRTM, yeah,) and finding an efficient way to index elevations. It would be somewhat approximate but probably close enough.

[RvEYoda]

I am working on this project here : http://forums.eagle.ru/showthread.php?t=76804 (It is a lot more finished than that thread suggests ^^), and I want to put some cool features into it. Also it will have a AWACS/GCI/commander's view with the ability to see all datalinked aircraft and their radar returns - and also the ability to share mark points and similar. Pilots, leads, commanders can send and share mission and target point data and messages in real time.

 

Adding a moving map capability to this would be nice, or at least some altitude curves.

Edited August 1, 2011 by =RvE=Yoda

[RvEYoda]

Yoda... The elevation data in the game is actually a proprietary 3d model format, with multiple LODs. It would be difficult at best to get data from that file. (lsa2 file.)

 

You might be better off using the actual satellite data (perhaps SRTM, yeah,) and finding an efficient way to index elevations. It would be somewhat approximate but probably close enough.

 

The question then becomes what kind of database representation would you use, after you first have found a 1:1 mapping to Fc's/DCS's flat world projection. (if you use the getAltitude method you dont have to use the mapping, just decide on storage math)

For example a poor implementation would be interpolated altitudes from 1 km uniform points - nowhere near good enough. Much better would be to use some kind of parametric surface representations of different areas with a somewhat low number of control points. You could also store curves with constant altitude. (easy to draw)

 

What I'm looking for are suggestions on how to solve these mathematical problems.

 

EDIT: I have an idea... which also works good for the getAltitude method. It works by storing N parametric surfaces of variable size, where each surface has certain limitations (a certain optimization for control point density). The generation process goes something like this:

You start in one corner of the world map

 

1. Decide on the smallest level of sub section for a parametric surface. Point scan this sub section with getAltitude and create the required control points which maps best to the point scan. Save sub section.

 

2. Do (1) on all neighbors. If compatible (optimizations say OK), merge sections, otherwise, keep separated.

 

3. Do this all over the map, you get a non-uniform grid of parametric surfaces of different shapes and sizes.

 

This should be close to optimal? :P

Ok so then I have stored it. I can then ask what altitudes are where.....Then comes the real question. How to draw from this ^^. Cause all these steps do 1-2-3, are to create my own getAltitude function. :S. I must also find a way to draw from it. Hmm that might be easier than expected. Maybe it is enough just to define a drawing density, for example 100x100 pixels, call the function 100x100 times each time the map camera changes and draw interpolated values in between.....nice.....that could work :P

Edited August 1, 2011 by =RvE=Yoda

[Case]

So you just want to make an altitude map for different spatial scales?

 

Can you confirm that you are not trying to use the map to determine if two points have a direct line-of-sight?

[RvEYoda]

So you just want to make an altitude map for different spatial scales?

 

What I want to, more precise, is to have a function like getAltitude(x,z), which gives the same answer as the built in lockon lua function, but doesn't require lockon :). (and it must work in the projected coordinate system of lockon). This would allow me to create certain cool moving map features.

 

Preferably I would want a full map setup that maps well to the lockon coordinate projection, but just altitudes are a start (again; we can't use open databases directly since they use a properly curved surface while lockon uses a flat projection, so grabbing altitude data directly from lockon will produce much more accurate results!) . The precise storage model I havent decided on, which is what I want to find a good solution for. So far my best idea is a set of variably sized parametric surfaces, dividing the sim regions into smaller sub regions. A uniform grid is far too imprecise and requires too much memory. Linearly interpolating from a set of altitudes at key positions is actually the same (by definition :P) as the first order parametric surfaces (planes). Though I prefer if we could make it a higher order. If I start working on it I will probably experiment with it in matlab :P. Could be cool!

 

Basically something like

 

function getAltitude(x,z)
  SubSection surface = getSubSection(x,y) // Find the sub section (= parametric surface) belonging to this coordinate
  return surface.getAltitude(x,z) // Calculate the altitude at x,z on the parametric surface that is this sub section
end

 

 

Can you confirm that you are not trying to use the map to determine if two points have a direct line-of-sight?

 

For a later time that could be interesting, since it is a simple thing to do with a getAltitude function, though I have nothing that requires such a thing at the moment.

Edited August 1, 2011 by =RvE=Yoda

[Case]

Ok, if that is what you want to do then the sub region idea would work very well. It sort of resembles Google Maps, which has several imagesets for specific zoom levels.

 

Probably the easiest way would be to store the data in some sort of image format, where in x and y you have the pixel numbers and in z you have the altitude represented as a color (hence it would be a grayscale image). This approach also gives you the added benefit of many libraries that can read and write data and also compress it.

 

I would first generate the images on the smallest scale, and then you could just use Photoshop or similar to rescale the images to the larger scales.

[RvEYoda]

Ok, if that is what you want to do then the sub region idea would work very well. It sort of resembles Google Maps, which has several imagesets for specific zoom levels.

 

Probably the easiest way would be to store the data in some sort of image format, where in x and y you have the pixel numbers and in z you have the altitude represented as a color (hence it would be a grayscale image). This approach also gives you the added benefit of many libraries that can read and write data and also compress it.

 

I would first generate the images on the smallest scale, and then you could just use Photoshop or similar to rescale the images to the larger scales.

 

 

Sry I didnt read your post properly before.

 

Using picture compression, nice! It would be the same way as saving adaptive key points . That might be a very good idea. Save image files and use jpeg/other compression on them to emulate the parametric surfaces...That's not bad :). Thanks for the idea... This is perhaps what I need.

 

1. Split the region into 100x100 pictures.

2. Save each picture with a certain lossy compression level. (some will be super small in storage size, others large, but they will all be the same amount of renderable pixels...)

 

...hmm This is good. Low cpu usage when running and easy implementation both generating the files and loading data from them.

Edited August 1, 2011 by =RvE=Yoda

[Case]

Do you need just one zoom level or multiple?

 

Ideally you would want to render a fixed number of pixels, so that if you are zoomed out you read the data from a different set of images compared to being zoomed in. You would then have many more images for the highest zoom level, but you could work out the administration so you only have to load those that are only needed to show the particular area of the map you are looking at at that time.

Edited August 1, 2011 by Case

[aaron886]

Well if the way we (and ED) make scenery now is any indicator, the elevation data you get from government outlets is exactly what's in game... no morphing it to a different projection. Just displaced vertices via the relief of the elevation images, then optimized to reduce the vertex count.

[RvEYoda]

Do you need just one zoom level or multiple?

 

Ideally you would want to render a fixed number of pixels, so that if you are zoomed out you read the data from a different set of images compared to being zoomed in. You would then have many more images for the highest zoom level, but you could work out the administration so you only have to load those that are only needed to show the particular area of the map you are looking at at that time.

 

Makes sense. If I can't fit all of them reasonably well in memory all at the same time I might make one image grid with lower set of images with less total memory usage for more zoomed out. My initial idea is to not render a fixed number of pixels, but rather sample a fixed number of pixels, and render an interpolation between them, though this might be what you meant.

Edited August 1, 2011 by =RvE=Yoda

[Moa]

For extremely large terrain you do need a decent data structure ('database schema' if you will). Algorithms such as ROAM and variants break the structure into a hierarchy for easy rendering (in this case your 'rendering' could be a GPU shader that does collision detection with terrain to check line-of-sight).

[RvEYoda]

Ofc any help or co-coding would be appreciated :). The results of such a project could be open to anyone to use.

 

The first job would be to gather all data required. I am guessing 100x100 images would be alright, and each image could be perhaps 1000x1000 pixels compressed greyscale? We will see how fast the lockon LUA is ^^. I wonder if we can do the compression in lua or if maybe I should send the raw data and compress it outside.

Edited August 1, 2011 by =RvE=Yoda

[Case]

Makes sense. If I can't fit all of them reasonably well in memory all at the same time I might make one image grid with lower set of images with less total memory usage for more zoomed out. My initial idea is to not render a fixed number of pixels, but rather sample a fixed number of pixels, and render an interpolation between them, though this might be what you meant.

 

Yes, that is what I meant.

 

Just checked, the highest point on the map (Elbrus) is at 5642m, so with a 16bit color depth you're looking at better than 0.1m vertical resolution, which may already be more than you need.

[RvEYoda]

Yes, that is what I meant.

 

Just checked, the highest point on the map (Elbrus) is at 5642m, so with a 16bit color depth you're looking at better than 0.1m vertical resolution, which may already be more than you need.

 

Then 16 bpp uncompress seems perfect :). We could do some trial and error to see what kind of compression levels we can apply on top of that to keep the memory usage somewhat low. Right now though, I'm going to sleep. :S

[m0jo]

3. Generate our own database. This wont be easy, but NOT impossible. In the config/export.lua there is a function

LoGetAltitude(x, z) -- (args - 2 : meters, results - 1 : altitude above terrain surface, meters)

which In principle could let us build our own database. I am not sure what kind of data storage/representation is the best to use here for our database, but I would guess that we would not want to use sime kind of altitude table, but rather some form of curve representation, and highly adaptive. Building such a database could be quite...challanging :). Maybe there is someone here with experience in such a subject.

 

You can already export the elevation data with the ACMI export script. In the file TacviewExportDCS.lua, uncomment the line AcmiData:ExportTerrain(); and launch any mission and wait a few minutes.

 

When the game is finally loaded, you can exit and check the /temp/ directory inside the game main folder. There will be a lot of file named like N45E36.hgt, these file will contain a grid of elevation data in SRTM3 text format.

 

You can reduce the number of point by modifying this python example: Canyon decimation.

[Case]

Great find m0j0!

 

It in fact uses LoGetAltitude, so this is exactly what Yoda wanted to do.

 

I just ran it and in about 5 minutes it generated 1.3GB of hgt files in a 1201x1201 pixel grid covering 1x1 degree.

 

Great stuff!

[RvEYoda]

You can already export the elevation data with the ACMI export script. In the file TacviewExportDCS.lua, uncomment the line AcmiData:ExportTerrain(); and launch any mission and wait a few minutes.

 

When the game is finally loaded, you can exit and check the /temp/ directory inside the game main folder. There will be a lot of file named like N45E36.hgt, these file will contain a grid of elevation data in SRTM3 text format.

 

You can reduce the number of point by modifying this python example: Canyon decimation.

 

I have just one question. Does this take care to use high density in important areas and low density in "flat" areas? - basically what image compression would do. Without this the detail level is probably not so very good.

 

EDIT: Looking at tacview script source code, the answer seems to be NO, see : http://lomac.strasoftware.com/download/misc/DCS/TacviewExportDCS.lua

 

But we can definitely make use of the same principle, we just need a little denser export and add compression/surface fitting on top of it :). I just checked in the game and I think a factor 5 or so in each dimension would be nice to have before applying compression.

 

IF i'm not mistaken, the uncompressed density of the tacview export is:

 

Now we can probably make it 5x more precise by increasing base sample depth and adding intelligent compression/surface fitting. Instead of writing uniformly spaced coordinates directly to a file with lua I send them to a compression tool... Gotta try this

Edited August 2, 2011 by =RvE=Yoda

[Case]

I converted the elevation data that Tacview outputs into 8bit grayscale PNGs (still figuring out how to do 16bit).

 

Here's an example of the 1x1 deg tile containing the Elbrus. The PNG file is 382kb, which is a very big reduction from the 26MB file Tacview wrote.

[RvEYoda]

I converted the elevation data that Tacview outputs into 8bit grayscale PNGs (still figuring out how to do 16bit).

 

Here's an example of the 1x1 deg tile containing the Elbrus. The PNG file is 382kb, which is a very big reduction from the 26MB file Tacview wrote.

 

Nice. 8bits should also be ok. 255 altitude steps is fine for moving map. PNG also is nice, lossless and all :). Still I will want to consider upping the resolution a bit to perhaps 20-50m spacing between points. How much space will all files take if you use 0.25x0.25 instead of 1x1? 16 times larger?

 

Do you have some sample source code for creating the maps? Or maybe, you have you already created all the pngs needed? :p. If so, am I allowed to use them? :)

 

About 16 bit I created a 16 bit greyscale like this in Java:

package imagecompressor;

import java.awt.*;
import java.awt.image.BufferedImage;
import java.awt.image.WritableRaster;
import java.io.*;
import javax.swing.*;

public class Compressor2 extends JPanel {

   private final Image image;

   public Compressor2(final BufferedImage image) {
       this.image = image;
   }

   @Override
   protected void paintComponent(final Graphics g) {
       g.drawImage(image, 0, 0, g.getClip().getBounds().width, g.getClip().getBounds().height, this);
   }

   public static void main(String[] args) throws IOException {

       // Generate a test image (gradient to make sure we got 16 bit depth and not 8)
       final int dataW = 1, dataH = 255 * 255;
       final int[] heightMap = new int[dataW * dataH];
       for (int i = 0; i < dataW * dataH; i++) {
           heightMap[i] = i;
       }
       final BufferedImage bi = new BufferedImage(dataW, dataH, BufferedImage.TYPE_USHORT_GRAY);
       final WritableRaster raster = bi.getRaster();
       raster.setPixels(0, 0, dataW, dataH, heightMap);

       // Render it as a test
       final Compressor2 test = new Compressor2(bi);
       final JFrame f = new JFrame();
       f.setLayout(new java.awt.GridBagLayout());
       f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
       final GridBagConstraints gbc = new GridBagConstraints();
       gbc.fill = java.awt.GridBagConstraints.BOTH;
       gbc.gridx = 0;
       gbc.gridy = 0;
       gbc.weightx = 1;
       gbc.weighty = 1;
       f.add(test, gbc);
       f.setSize(500, 500);
       f.setLocation(100, 100);
       f.setVisible(true);
   }
}

 

It seems like my screen is unable to tell the difference between 16 bit or 8 bit greyscale so :P. (don't they just have 8 bits for color components anyway?). So using 16 bit is prob not necessary... Or am I wrong?

 

can normal monitors display above 8 bit greyscale?

I guess sometimes you could zoom in on an area which only has let's say 1000-2000 m altitudes, then u could zoom the greyscale as well to make use of your stored 16 bit valules, but that could get confusing. Still I guess it could be used for a manual gain/contrast/brightness control..hmm.

Edited August 2, 2011 by =RvE=Yoda

[Case]

Still I will want to consider upping the resolution a bit to perhaps 20-50m spacing between points. How much space will all files take if you use 0.25x0.25 instead of 1x1? 16 times larger?

 

Right now the model is sampled at 1200 points over a degree, so 3 arcsecond spacing which amounts to about 92m spacing between points. The combined file size of all 45 images is only 4.9MB

 

I'll rerun the Tacview script with 1 arcsecond spacing (3600x3600 points), which should increase the size to about 50MB, which should still be workable.

 

 

Do you have some sample source code for creating the maps? Or maybe, you have you already created all the pngs needed? :p. If so, am I allowed to use them? :)

 

Sure, check http://www.mediafire.com/?natolw1ai9mh87n.

Edited August 2, 2011 by Case

[RvEYoda]

Right now the model is sampled at 1200 points over a degree, so 3 arcsecond spacing which amounts to about 92m spacing between points. The combined file size of all 45 images is only 4.9MB

 

I'll rerun the Tacview script with 1 arcsecond spacing (3600x3600 points), which should increase the size to about 50MB, which should still be workable.

 

 

 

 

Sure, check http://www.mediafire.com/?natolw1ai9mh87n.

 

Thank you! I see you did not make any for Nevada, though that is a minor thing. We can do that later :). Looking forward to what the 1 arcs spacing brings! Can I ask you what programming language you do this in? Do you do everything in lua or use something like C/Java for more "rough work"?

Edited August 2, 2011 by =RvE=Yoda

[Case]

rgr I see you did not make any for Nevada, though that is a minor thing. We can do that later :). Looking forward to what the 1 arcs spacing brings!

 

I have the files for Nevada, but they need a different scaling as the highest point is very different.

 

Rerunning Tacview will have to wait till tonight :)

[RvEYoda]

I have the files for Nevada, but they need a different scaling as the highest point is very different.

 

Rerunning Tacview will have to wait till tonight :)

 

Ok. Hmm. I think I might create my own maps anyway, because I want them sampled in an x,z coordinate system, and not lat longs. This is mostly because I do not want to implement new latlong->x,z math every time a new region is introduced. On the downside a saved x,z map means you have to select which one to use, but I find that acceptable. In my lua and in GEAR (if you saw that thread) I no longer have any conversion math but always use lockon's lua's internal variants, and only when I absolutely need to. Java-GEAR never needs to convert (in fact is has no knowledge on how to do so).

 

Well, unless someone has another solution :P. Like, is there a decent way to find these constants:

 

-- ===================
-- 	Constants
-- ===================

zeroX = 5000000.0; -- Real coordinates beginning
zeroZ = 6600000.0;

centerX = 11465000.0 - zeroX; -- Circle center
centerZ = 6500000.0 - zeroZ;

pn40x24_X = 4468608.57 - zeroX; -- point 40dgN : 24dgE
pn40x24_Z = 5730893.72 - zeroZ;

pn48x24_X = 5357858.31 - zeroX; -- point 48dgN : 24dgE
pn48x24_Z = 5828649.53 - zeroZ;

pn40x42_X = 4468608.57 - zeroX; -- point 40dgN : 42dgE
pn40x42_Z = 7269106.20 - zeroZ;

pn48x42_X = 5357858.31 - zeroX; -- ????? 48dgN : 42dgE
pn48x42_Z = 7171350.00 - zeroZ;

-- distances from the circle center to 48dgN and 40dgN
lenNorth = math.sqrt((pn48x24_X-centerX)*(pn48x24_X-centerX) + (pn48x24_Z-centerZ)*(pn48x24_Z-centerZ));
lenSouth = math.sqrt((pn40x24_X-centerX)*(pn40x24_X-centerX) + (pn40x24_Z-centerZ)*(pn40x24_Z-centerZ));
lenN_S = lenSouth - lenNorth;
RealAngleMaxLongitude = math.atan((pn40x24_Z - centerZ)/(pn40x24_X - centerX)) * 180.0 /math.pi;

-- Map bounds. Degrees!
EndWest = 24.0;
EndEast = 42.0;
EndNorth = 48.0;
EndSouth = 40.0;
MiddleLongitude = (EndWest + EndEast) / 2;
ToLengthN_S = (EndNorth - EndSouth) / lenN_S;
ToAngleW_E = (MiddleLongitude - EndWest) / RealAngleMaxLongitude;

 

Can I get these from the game, rather than put them in a file manually?

(So that I can make it working in new regions)

 

.

 

UPDATE 2: No i definitely need it in x,z-space. It also improves performance of my programs, since I render in a plane.

Edited August 2, 2011 by =RvE=Yoda