Split Update into multiple downloads for core, maps, aircraft and campaign

[shagrat]

Split larger Updates into parts, as in:

Part I - Core Game Updates,

Part II - Kola Map Update,

Part III - Syria Map Update,

Part IV - Aircraft Updates,

Part V - Campaign Updates.

So you can individually select, one or more, and then download, decompress and install, one at a time, or all at once.

This can help to manage disk space during for download, decompression and installation, if you don't have enough disk space.

[joker62]

Great idea. It would also be helpful because I have a slow internet connection: I could download and install the base game, modules and campaigns, then use the latest update, and postpone the map update to a later time. I hope it is possible and that ED is well disposed towards it.

[MAXsenna]

19 minutes ago, joker62 said:

Great idea. It would also be helpful because I have a slow internet connection: I could download and install the base game, modules and campaigns, then use the latest update, and postpone the map update to a later time. I hope it is possible and that ED is well disposed towards it.

No, I don't think that's really possible. Everything must be on the same version, as long as it is installed. 

[shagrat]

@joker62 Yeah, I think installing only one part of the update (core) without updating the map etc. may not be possible.

The idea is to be able to download, say a 20-30GB Core update, decompress and install, download a map update of 50 GB for a map you own, decompress and install, and so on. This way we could reduce the space required for download and installation, as after each part the installer can clean up and delete the temporary files and download.

[MAXsenna]

9 minutes ago, shagrat said:

@joker62 Yeah, I think installing only one part of the update (core) without updating the map etc. may not be possible.

The idea is to be able to download, say a 20-30GB Core update, decompress and install, download a map update of 50 GB for a map you own, decompress and install, and so on. This way we could reduce the space required for download and installation, as after each part the installer can clean up and delete the temporary files and download.

Could be automated sequentially as well. 

[shagrat]

vor 2 Stunden schrieb MAXsenna:

Could be automated sequentially as well. 

Automation requires thorough testing under every conceivable scenario, so only the scenario nobody thought of, breaks the automation and causes issues.

I would rather use a manual selection process, that you can troubleshoot/adapt, if you encounter issues.

You may select all parts and it works fine, but if you encounter an issue, with hardlinks, or redirected folder, or cosmic radiation, you can try again part by part...

[joker62]

14 hours ago, MAXsenna said:

No, I don't think that's really possible. Everything must be on the same version, as long as it is installed.

That's what I've been doing for several years when an update comes out including a map update.

For example, the patch 2.9.8.1107 (september 30, 2024) included an update to the Syria map. I simply disabled and removed the Syria map through Module Manager, then I updated the sim to patch 2.9.8.1107, and then slowly reinstalled the Syria map (in this case updated). I know, I can't fly over Syria until I finish reinstalling the map, and it's also true that reinstalling a map is longer than updating it, but this way I bypass my slow internet connection problem, quickly keeping the sim updated to the latest patch. I just have to be patient before I can fly on the updated map again.

In any case, I don't think ED will take any further charge from such a fragmentation of the patches.

Edited October 29, 2024 by joker62

[MAXsenna]

That's what I've been doing for several years when an update comes out including a map update.
For example, the patch 2.9.8.1107 (september 30, 2024) included an update to the Syria map. I simply disabled and removed the Syria map through Module Manager, then I updated the sim to patch 2.9.8.1107, and then slowly reinstalled the Syria map (in this case updated). I know, I can't fly over Syria until I finish reinstalling the map, and it's also true that reinstalling a map is longer than updating it, but this way I bypass my slow internet connection problem, quickly keeping the sim updated to the latest patch. I just have to be patient before I can fly on the updated map again.
In any case, I don't think ED will take any further charge from such a fragmentation of the patches.

We're talking about the same thing it seems. (You didn't specify that the modules not updated were uninstalled ).
All good then!

Sent from my SM-A536B using Tapatalk

[shagrat]

vor 22 Stunden schrieb joker62:

That's what I've been doing for several years when an update comes out including a map update.

For example, the patch 2.9.8.1107 (september 30, 2024) included an update to the Syria map. I simply disabled and removed the Syria map through Module Manager, then I updated the sim to patch 2.9.8.1107, and then slowly reinstalled the Syria map (in this case updated). 

The thing is, then you unnecessarily removed all files of the map that were not changed by the update and had to download the whole(!) map, again. With map sizes growing, this means you have to juggle a lot of deinstalling and reinstalling unnecessary.

It does make sense to deactivate and remove a map you don't play for a longer time, but to just remove it, to reduce the update size, it's way more efficient to have multiple (smaller) parts, with just the changed files, that you can individually download and unpack/install.

[joker62]

9 hours ago, shagrat said:

The thing is, then you unnecessarily removed all files of the map that were not changed by the update and had to download the whole(!) map, again. With map sizes growing, this means you have to juggle a lot of deinstalling and reinstalling unnecessary.

It does make sense to deactivate and remove a map you don't play for a longer time, but to just remove it, to reduce the update size, it's way more efficient to have multiple (smaller) parts, with just the changed files, that you can individually download and unpack/install.

This is exactly what I was happy about, most likely I expressed myself badly. If it were really possible it would mean a great benefit for all users. Thanks.

[Paganus]

Most map updates are entire maps. Check the size next time. 

I have done the deactivate/move map files - update - restore map files, activate - update - myself to break it up for storage reasons.

I have been wondering if deactivating the map and then just renaming its folder, would do the trick, instead of the time it takes to move files. 

To the OP's point, I agree, maps should be updated separately. At least give an option to not update maps and if necessary deactivate them until updated individually.

 

 

[Dangerzone]

On 10/29/2024 at 9:50 PM, joker62 said:

That's what I've been doing for several years when an update comes out including a map update.

For example, the patch 2.9.8.1107 (september 30, 2024) included an update to the Syria map. I simply disabled and removed the Syria map through Module Manager, then I updated the sim to patch 2.9.8.1107, and then slowly reinstalled the Syria map (in this case updated). I know, I can't fly over Syria until I finish reinstalling the map, and it's also true that reinstalling a map is longer than updating it, but this way I bypass my slow internet connection problem, quickly keeping the sim updated to the latest patch. I just have to be patient before I can fly on the updated map again.

In any case, I don't think ED will take any further charge from such a fragmentation of the patches.

 

There is a slightly more optimized way of doing this. 

If you copy your DCS install (before removing the maps) to another location (doesn't matter if it's on an External HDD, NAS, or anywhere you have the space available), you can add a new text file called "dcs_local_source.txt" to your main DCS install, and put in the first line of that the path to the copied install. 

Then, when the updater needs to update, it will first look at this source for any available files that it needs prior to downloading them. This means that when you update, any files that haven't changed that don't need to be updated will get sourced locally first - and then it will only download the necessary files. This should stop your update from being any larger than necessary. 

 

[MAXsenna]

There is a slightly more optimized way of doing this. 
If you copy your DCS install (before removing the maps) to another location (doesn't matter if it's on an External HDD, NAS, or anywhere you have the space available), you can add a new text file called "dcs_local_source.txt" to your main DCS install, and put in the first line of that the path to the copied install. 
Then, when the updater needs to update, it will first look at this source for any available files that it needs prior to downloading them. This means that when you update, any files that haven't changed that don't need to be updated will get sourced locally first - and then it will only download the necessary files. This should stop your update from being any larger than necessary. 
 

That's a neat trick!

Sent from my SM-A536B using Tapatalk

[Dangerzone]

49 minutes ago, MAXsenna said:

That's a neat trick!

Was very handy when we had 2 installs (Open Beta and Stable) - because often the Stable could rip from the Open Beta version when a stable was released and save a lot of downloading. I still have 2 installs - one for testing and mods, and one for my actual gameplay (I'm blessed enough to have the drive space available). I often will update my test first, so if there are issues (like antivirus breaking stuff) - I'm able to address that before I break my gaming build. 

[MAXsenna]

Was very handy when we had 2 installs (Open Beta and Stable) - because often the Stable could rip from the Open Beta version when a stable was released and save a lot of downloading. I still have 2 installs - one for testing and mods, and one for my actual gameplay (I'm blessed enough to have the drive space available). I often will update my test first, so if there are issues (like antivirus breaking stuff) - I'm able to address that before I break my gaming build. 

Yeah, that it did for me as well with my dual Stable/OB install, but I thought it pulled that info from the registry. I never made no file. It will even do it between Steam and Standalone.

Sent from my SM-A536B using Tapatalk

[Dangerzone]

1 hour ago, MAXsenna said:

Yeah, that it did for me as well with my dual Stable/OB install, but I thought it pulled that info from the registry. I never made no file. It will even do it between Steam and Standalone.

 

I've always used that file. (Can be used across network as well - if you have a copy on another PC, etc where the registry "won't know". 

Interesting though - I had no idea that DCS also scanned the registry for other installs for updating. (I didn't even know that DCS used the registry. I thought all the settings were stored in the saved games directory alone.) There you go. 

[MAXsenna]

I've always used that file. (Can be used across network as well - if you have a copy on another PC, etc where the registry "won't know". 
Interesting though - I had no idea that DCS also scanned the registry for other installs for updating. (I didn't even know that DCS used the registry. I thought all the settings were stored in the saved games directory alone.) There you go. 

All the hidden undocumented jems!

Sent from my SM-A536B using Tapatalk

[Sile]

I think the key problem is the data structure of the maps / terrains. Combined with an inefficient updater. This leads to enormous update sizes.
Like someone said on reddit: "Your doomed if someone slightly adjusted a map texture file."

Even if only a single airfield gets updated - maybe some 1-2 GB - the updater downloads the full 30 - 50 GB file which contains the data.
And it stores it temporarily till the whole update download is complete. There is no concept for larges files. Most of the data doesn't even change.

Today my Updater showed a download size of 146 GB.
After uninstalling South Atlantic it went down to about 110 GB.
Now read the changelog for South Atlantic and examine the file structure in its mods folder.

I mentioned this on EDs discord already, but got a negative answer from a CM:
"I will certainly share your concerns, but I don't think it is possible to change the way it is delivered currently".

[joker62]

10 hours ago, Dangerzone said:

There is a slightly more optimized way of doing this. 

If you copy your DCS install (before removing the maps) to another location (doesn't matter if it's on an External HDD, NAS, or anywhere you have the space available), you can add a new text file called "dcs_local_source.txt" to your main DCS install, and put in the first line of that the path to the copied install. 

Then, when the updater needs to update, it will first look at this source for any available files that it needs prior to downloading them. This means that when you update, any files that haven't changed that don't need to be updated will get sourced locally first - and then it will only download the necessary files. This should stop your update from being any larger than necessary. 

Thanks a lot, great trick.