ME corrupts .miz on save after a certain number of ground units added...

[Evilone]

Hi,
I'm creating a mission with many ground unit groups, all set to 'on road', with each having a fairly long path to follow (~2 hours or so), all on the Syria map - specifically Cyprus.
Once I got to about 50 or so separate groups, I pressed save and got the error in the screen shot.
When I click OK, I am presented with an empty map, as in screen shot 2, and the original file will no longer load - giving the same error if I try.
I reloaded from a previous, successful save that had fewer units (about 48 groups) and tried again, adding the units 8 at a time.  Same thing.  Once I got to a certain number of ground groups the error returns, but only upon saving.
I've tried it on a different computer too - same thing.
I've attached a copy of the corrupted .miz as well.  I'd be interested to know if anyone can open this miz file.  Every time I try it gives the same error and then crashes.
Don't know if its something peculiar to my install or something I'm doing wrong, or if indeed it's a bug.  Looking forward to suggestions.

Thanks in advance.

GMT Training Template 2-1.miz

[Nick_17]

Getting the same error once i try to load your mission.

Log output:

2023-08-30 11:49:31.060 ALERT   LUACOMMON (Main): Error: GUI Error: constant table overflow
GUI debug.traceback: stack traceback:
    [C]: ?
    [C]: in function 'loadstring'
    [string "./Scripts/dictionary.lua"]:919: in function 'getMissionDescription'
    [string "./MissionEditor/modules/FileDialog.lua"]:821: in function 'fillRequiredModules'
    [string "./MissionEditor/modules/FileDialog.lua"]:856: in function 'updateRequiredModules'
    [string "./MissionEditor/modules/FileDialog.lua"]:467: in function 'onSelectFile_'
    [string "./MissionEditor/modules/FileDialog.lua"]:486: in function 'onFileClick'
    [string "./MissionEditor/modules/FileGrid.lua"]:313: in function 'onRowClick'
    [string "./MissionEditor/modules/FileGrid.lua"]:321: in function 'onGridMouseDown'
    [string "./MissionEditor/modules/FileGrid.lua"]:160: in function 'onMouseDown'
    [string "./dxgui/bind/Grid.lua"]:26: in function 'callback'
    [string "./dxgui/bind/Widget.lua"]:299: in function <[string "./dxgui/bind/Widget.lua"]:298>
    [C]: in function 'WidgetSetVisible'
    [string "./dxgui/bind/Widget.lua"]:69: in function 'setVisible'
    [string "./MissionEditor/modules/FileDialog.lua"]:932: in function <[string "./MissionEditor/modules/FileDialog.lua"]:904>
    (tail call): ?
    [string "./MissionEditor/modules/me_toolbar.lua"]:766: in function 'openMissionFileDialog'
    [string "./MissionEditor/modules/me_startEditor.lua"]:61: in function 'onChange'
    [string "./dxgui/bind/Button.lua"]:22: in function 'callback'
    [string "./dxgui/bind/Widget.lua"]:368: in function <[string "./dxgui/bind/Widget.lua"]:363>
2023-08-30 11:49:34.535 WARNING LOG (19876): 3 duplicate message(s) skipped.
2023-08-30 11:49:34.535 ALERT   LUACOMMON (Main): Error: GUI Error: constant table overflow
GUI debug.traceback: stack traceback:
    [C]: ?
    [C]: in function 'loadstring'
    [string "./MissionEditor/modules/me_mission.lua"]:1816: in function 'load'
    [string "./MissionEditor/modules/me_toolbar.lua"]:584: in function 'func'
    [string "./Scripts/UI/ProgressBarDialog.lua"]:141: in function 'updater'
    [string "./Scripts/UpdateManager.lua"]:64: in function <[string "./Scripts/UpdateManager.lua"]:40>
2023-08-30 11:49:45.153 INFO    DXGUI_WIN_ADAPTER (Main): resize main window to [0, 0]

[[HOUNDS] CptTrips]

I ran into that.

A couple of suggestions...

I use a free github account to store off a version of my scripts every day so I never risk losing too much work.

I also save obsessively so I realized right after I added the last unit that caused that.

 I was able to use the DCS Web Editor to open it up and delete that last group I placed.  I could then reopen it in the native ME again. https://discord.gg/MYqwPmd6

What that means is you've reached the limit ground units with on-road routes.  I haven't found a way around it.

[Edit]  Mine was on Cyprus too.  Don't know if that is related.

Edited August 30, 2023 by [16AGR] CptTrips

[NineLine]

Thanks, I will take a look. 

[NineLine]

8 hours ago, Evilone said:

Hi,
I'm creating a mission with many ground unit groups, all set to 'on road', with each having a fairly long path to follow (~2 hours or so), all on the Syria map - specifically Cyprus.
Once I got to about 50 or so separate groups, I pressed save and got the error in the screen shot.
When I click OK, I am presented with an empty map, as in screen shot 2, and the original file will no longer load - giving the same error if I try.
I reloaded from a previous, successful save that had fewer units (about 48 groups) and tried again, adding the units 8 at a time.  Same thing.  Once I got to a certain number of ground groups the error returns, but only upon saving.
I've tried it on a different computer too - same thing.
I've attached a copy of the corrupted .miz as well.  I'd be interested to know if anyone can open this miz file.  Every time I try it gives the same error and then crashes.
Don't know if its something peculiar to my install or something I'm doing wrong, or if indeed it's a bug.  Looking forward to suggestions.

Thanks in advance.

GMT Training Template 2-1.miz 3.34 MB · 2 downloads

I am unable to reproduce it, do you have a mission where you can reproduce it on a regular basis? Such as so many groups is fine, adding 1 more kills it?

[[HOUNDS] CptTrips]

 

Here is a copy of his broken one that I deleted a few groups with the Web Editor.

 

On my machine, I can open it, but adding one more ground group with a long on-road wp and try and save it gives the error.

I'd love to find a fix for this too.

(See attached.)

Test4.miz

Edited August 30, 2023 by [16AGR] CptTrips

[NineLine]

Thanks I will give that a try. 

[Grimes]

It seems to be a problem of lua 5.1 in terms of reaching the maximum table size and erroring when it attempts to load the table string. 

The vast, vast, vast majority of that miz file is taken up by "spans" for each on road route for each group. Basically the editor saves the lines that are drawn on the map representing the path the AI will use, this data is not actually used by the AI. It is used to render the lines quickly in the editor. 

It isn't quite a group/object limit, more of a what is contained within each group data. In the mission I reproduced the problem in I used only 12 groups. To put it in perspective in the mission I reproduced the error with was on Caucuses with a route from Tbilisi to Kerch Straits of about 1050km. It had approx 13500 span entries. On that Cyprus mission there are routes under 150km with 6000 span entries. It is possible Syria is generating more than it needs, but the spans bloating the entries in the miz file has been a problem for a while. 

Not much you can do about it other than avoiding excessive use on on road routes over hilly terrain. If you are using scripting to use those groups as a template of sorts you can use any other formation type, then convert it to onroad when giving the task and the AI will use roads as expected. 

[[HOUNDS] CptTrips]

1 hour ago, Grimes said:

Not much you can do about it other than avoiding excessive use on on road routes over hilly terrain. If you are using scripting to use those groups as a template of sorts you can use any other formation type, then convert it to onroad when giving the task and the AI will use roads as expected. 

 

That is a very interesting idea.  Thanks.  I'll think that one over.

In my case, yes, they are late activated templates that I spawn copies of.  I should be able to change the formation.  I'll experiment with that.

 

[[HOUNDS] CptTrips]

3 hours ago, Grimes said:

Basically the editor saves the lines that are drawn on the map representing the path the AI will use, this data is not actually used by the AI. It is used to render the lines quickly in the editor. 

 

 

Wait.  Wut?  It isn't even needed?  Then editor should purge that.

When I drop a road wp I've always been impressed on how fast it calculated it.  

Doesn't sound like worth bricking missions for something not needed.  It might take an extra 2 seconds to open the mission.

 

[Evilone]

Thanks everyone for putting this into context for me.

I'll re-do the mission using less complicated and shorter road loops, and use a proximity trigger to switch the units on when the users are closer to the AO, so that the ground units don't need to run for such a long time.

Regardless, hopefully it helps ED if indeed there's some 'bloating' or other such data-doubling going on.

In any case, thanks for such a quick, responsive and helpful forum!

[Grimes]

There aren't many instances of data doubling that are quite as bad. Closest would be stuff that may be considered more of a "reference" type of information. A 4 ship of client aircraft each have a unique table entry for custom radio frequencies for example. Each F-18 client has 80 total entries to define 40 radio channel frequencies and modulations. It adds up, but its not quite as insane as the spans. 

[[HOUNDS] CptTrips]

8 hours ago, Grimes said:

There aren't many instances of data doubling that are quite as bad. Closest would be stuff that may be considered more of a "reference" type of information. A 4 ship of client aircraft each have a unique table entry for custom radio frequencies for example. Each F-18 client has 80 total entries to define 40 radio channel frequencies and modulations. It adds up, but its not quite as insane as the spans. 

 

Bloating a mission file is one thing, but bricking a mission is unforgivable.  If it hadn't been for the 3rd party web editor, I would have lost a lot of work.  I wonder how they are able to open it.  Oh, they are problem not using Lua.

So can that data be deleted without harm?  Does that bloat get transferred over to anyone playing on a MP server even if it has no use in gameplay?  Ick.

If it could be culled safely, that would be a nice little util feature for the DWE.

 

Edited August 31, 2023 by [16AGR] CptTrips

[Nick_17]

35 minutes ago, [16AGR] CptTrips said:

 

Bloating a mission file is one thing, but bricking a mission is unforgivable.  If it hadn't been for the 3rd party web editor, I would have lost a work.  I wonder how they are able to open it.  Oh, they are problem not using Lua.

So can that data be deleted without harm?  Does that bloat get transferred over to anyone playing on a MP server even if it has no use in gameplay?  Ick.

If it could be culled safely, that would be a nice little util feature for the DWE.

 

You could also easily open the .miz file with 7zip or winrar and edit the mission file with any text editor to remove some of the groups or units from the table

[[HOUNDS] CptTrips]

2 minutes ago, Nick_17 said:

You could also easily open the .miz file with 7zip or winrar and edit the mission file with any text editor to remove some of the groups or units from the table

Possibly, but sounds risky for average customers to have to go muck around in the internals of the file.

The correct answer is to fix the ME so it doesn't brick files.

 

[Grimes]

9 hours ago, [16AGR] CptTrips said:

 

Bloating a mission file is one thing, but bricking a mission is unforgivable.  If it hadn't been for the 3rd party web editor, I would have lost a lot of work.  I wonder how they are able to open it.  Oh, they are problem not using Lua.

So can that data be deleted without harm?  Does that bloat get transferred over to anyone playing on a MP server even if it has no use in gameplay?  Ick.

If it could be culled safely, that would be a nice little util feature for the DWE.

 

 

Like I said its an issue in lua 5.1 when attempting to run loadstring on a large enough file. Lua 5.1 dates back to 2006 and the up to date version is lua 5.4. Fairly certain the web editor converts the file to json for its own uses. 

Its part of the miz file, so people who connect in MP must download it. However the mission part of a miz file is quite small in terms of file size. Its all text which compresses extremely well. Uncompressed it could be 50mb, but with it compressed its only a few MB. Chances are you have an image or audio file that takes up as much space. 

[[HOUNDS] CptTrips]

15 hours ago, Grimes said:

Like I said its an issue in lua 5.1 when attempting to run loadstring on a large enough file. Lua 5.1 dates back to 2006 and the up to date version is lua 5.4. Fairly certain the web editor converts the file to json for its own uses.

 

So would it be safe to delete those spans so I could save more units with road routes?  

A shame to hobble designers for something the AI doesn't even use.

 

[Edit]  I don't see ED updating Lua versions.  Who knows what chaos that would unleash in everything written up to now.

Edited September 1, 2023 by [16AGR] CptTrips

[[HOUNDS] CptTrips]

Darn.  Was hoping this one had been fixed but not listed.

Is it safe to manually delete the spans out of the file?