An open letter to ED from a Stable Release Admin

[Dangerzone]

Respectfully before reading further, can I please ask if you're in the camp of "What's the point of Stable Release, we only need Open Beta", can you please refrain from posting such, as this will just derail the intention of this post. It's not that your opinion doesn't matter, but that the context of this discussion is based on the minority who appreciate Public Beta's vs Stable, and looking for answers that accommodate this mindset. (There's plenty of other topics that relate to that debate where this can be discussed or favor Open Beta users). 

Dear ED,

Our preference has been to use Stable Release and Open Beta as you have intended; Open Beta for testing the new features, and reporting back or working through bugs, and then Stable for release for online servers. Even after you have changed your official recommendation for multiplayers to use Open Beta, we have still appreciated your ongoing stances made in this forum that "Open Beta is for testing, Stable is the Release" and desired to practice multiplayer designing on that basis. 

I'm aware this goes against the majority of what people do, and that's OK. I've never felt the need to follow the pack, nor to convince the pack to change their minds. Having options is good too. In the past there's been plenty of people who prefer/appreciate stable release out there still to warrant having Stable Release servers. The worst normally meant is having to wait 3 or 4 weeks for the new shiny toys (or try and test them out in Open Beta to learn the new features). Where stability was preferred over instant access - this has worked well. It has also meant more stability and less server maintenance/debugging, as well as supporting the smaller group of players that prefer stable release too.  

However, changes in release cycles this year has this no longer being a viable option anymore. It seems that the time between a big feature (aka, new module released, or significant enhancement) released on Open Beta vs Stable is increasing to that of Months now instead of weeks. (I note that there has only been 3 stable releases this year, averaging just under 4 months between stable releases). In comparison, if I look at an average over the 4 years prior, I notice the average was over 10 per year. 

Waiting ~4 months for a long awaited feature is hard, but if that's how long it takes for DCS to become stable, then so be it.

However, compounding this issue now is pressure to switch to Open Beta, not to get access to the new features, but to get access to the new 'fixes' as soon as possible, because Stable was released with bugs that aren't fixed until the next OB patch or two.

As for now, (even from the perspective of a stubborn minded "Stable Release" user), using Stable is no longer a viable option as users who are wanting stability need to update to the next Open Beta to get patch fixes. (It feels as though Stable Release is being released, not because it's Stable, but because "We're now moving onto the next big change, so that's as good as it's going to get". This leaves us switching to Open Beta to get bug fixes). 

This post is not to bash ED. I'm sure that ED are choosing releases in what they believe is the best release cycle with their current development roadmap that I'm either not aware of, or haven't considered. I'm being candid with the hope to serve three purposes:

1. The first to give transparent and informative feedback as to how a server administrator who's wanting to use Beta vs Stable in the way you intend is struggling, with the hope that this may be given more consideration in your next planning/roadmap staff meeting.

2. The second is to ask a question: Is what we have seen this year the new approach for ED now? Or is this a temporary change (due MT and DLSS?) and will we be seeing a return to builds being more finished before being released to stable, along with a more frequent stable release cycle at the start of next year as we have seen in years gone by? 

   (I appreciate with Vulkan on the Horizon too, that this change we've seen this year may be ongoing - and if so - that's OK... as what we really need is transparency and guidance, so we have the right information to base decisions on how to handle this moving forward).

3. The last purpose is to ask for help or guidance from ED in this. I'm aware that numerous administrators are closed beta testers, so they basically get some of their "Beta" vs "Release" testing difference through that, but for the others who aren't closed beta testers, do you have any recommendations on how to handle the current situation? (Apart from just joining the Open Beta server hosters and give up on the desire for more stability, along with time to test new features before release)? 

 

Thank you for the hard work that you continue to pour into DCS. I'm sure that these have been difficult times as large changes have been made to the underlying code set foundation. I look forward to your response and working with you for making the best decisions moving forward for hosting MP servers for those focusing more on Stability.

Respectfully, with Thanks, 

DZ

Edited December 12, 2023 by Dangerzone

[Rudel_chw]

1 hour ago, HC_Official said:

well I pretty much expected this response (nothing) on your post .....

It’s discouraging to see the lack of an official response to such an articulate set of questions, I don’t really fly MP but I understand the problems of the OP and was looking forward to ED answers  

Edited December 21, 2023 by Rudel_chw

[Hotdognz]

ED never officerly respond to post like this, the forum is a user forum to talk about DCS but not really get any information from the developer other than coming soon updates.

[HC_Official]

@Dangerzone I guess you have your answer , as the mods have for sure read this thread but are not going to reply to you, which is sad

[SharpeXB]

This is really an issue for server admins to resolve, not ED. If server owners decided to use the game as intended and run the Stable version more often, players would follow. 
ED can’t get rid of the Open Beta, they need it. And they can’t make the OB single player only, MP needs to be tested as well. But ED could stop actually marketing the OB for multiplayer use on the store page. The OB should be a test version not a de facto release, which is how it’s treated. But in the end they can’t dictate how the players use the game, that’s up to them. It’s a puzzling situation, no other game I’m aware of uses its test version in multiplayer for obvious reasons. 

Edited December 22, 2023 by SharpeXB

[NineLine]

Dear Dangerzone,

First, let me start by saying I am very sorry this thread went unnoticed for so long. It is an important issue and your detailed and thoughtful post deserved a response, finally after discussing it with management, here we are. 

Second, this type of feedback is what we look forward to and makes it tougher that I missed this for so long. The answer will be much shorter but please I hope you do not see it a disregard of all your points but rather that we at ED have been seeing what's been going on running two release versions.

So the answer is this... The Open Beta and Stable system of the released will be fully reworked. The process will be changed and all our customers will receive only one version of DCS. Now I do not have any more details than this right now and of course no timeline. But this is the direction we will be headed in the future. 

Thanks once again for the feedback, hope you and yours have a wonderful Holiday Season!
NineLine/Norm 

[Tank50us]

1 hour ago, NineLine said:

Dear Dangerzone,

First, let me start by saying I am very sorry this thread went unnoticed for so long. It is an important issue and your detailed and thoughtful post deserved a response, finally after discussing it with management, here we are. 

Second, this type of feedback is what we look forward to and makes it tougher that I missed this for so long. The answer will be much shorter but please I hope you do not see it a disregard of all your points but rather that we at ED have been seeing what's been going on running two release versions.

So the answer is this... The Open Beta and Stable system of the released will be fully reworked. The process will be changed and all our customers will receive only one version of DCS. Now I do not have any more details than this right now and of course no timeline. But this is the direction we will be headed in the future. 

Thanks once again for the feedback, hope you and yours have a wonderful Holiday Season!
NineLine/Norm 

So, to clarify, those that grab the Steam version and don't opt into the betas will one day not have to worry about the version differences any longer?

[NineLine]

56 minutes ago, Tank50us said:

So, to clarify, those that grab the Steam version and don't opt into the betas will one day not have to worry about the version differences any longer?

One day instead of 2 versions being available to choose from (not including older versions which should still be available as before) there will only be one choice to install. 

[Johnny Dioxin]

Music to my old ears!

[twistking]

52 minutes ago, NineLine said:

One day instead of 2 versions being available to choose from (not including older versions which should still be available as before) there will only be one choice to install. 

i'm quite happy to hear that. I think the current system is not an optimal solution and that there's potential to make it better for all.

Even though i don't play public MP, i still very much understand @Dangerzone's arguments and my frustration (as a stable-enjoyer) is rooted in - at least- similar issues.

I want to add a completely different and unrelated point of critique though: I have only anecdotical evidence unfortunately but i suspect that having the open beta the "standard" for the (public) MP community significantly reduces the quality of technical feedback (bug reports) from what is supposed to be a testing branch.
I chose stable branch, because i don't like to deal with bugs and issues and very often i still encounter a new bug in a stable release, that is not yet reported on the forums, despite the open beta being available for month at that point. And i'm talkin "proper" bugs: Little things that broke with an update as compared to "complaining" about long standing issues (which i also endulge in a lot).
On other games that have an explicit "testing branch" version before complex updates, i feel that both quantity and quality of user feedback is way better, because users make a conscious decision to download said branch to help with testing and don't just download to get the new shiny stuff.
I bring this up, because i think that there can be a lot of value in having a seperate testing branch, because there are a lot of users who actually want to help with testing. It's just that i'm not one of them... and i think that letting the test branch become the standard for many (most?), will muddy down the feedback quality eventually.

Edited December 22, 2023 by twistking

[Art-J]

^ Hmmm.... I don't know.... The problem is - "our" beta IS the "seperate testing branch". Always has been. Some users are just more interested in testing and reporting properly (they're smaller group I suspect), while the others are more interested in getting access to new content earlier (this is majority I'd hazard a guess). The former will always be active on the forum with more detailed reports, while the latter will not. But that's just a human nature, and In my opinion this proportion will never change. Thus I don't see how creating a yet another, more closed / "elite" separate branch could solve this issue, especially when ED MUST rely on wide scale public testing because of their internal QA resources limitations.

I've no idea how to tackle this problem. They have to persuade/force people somehow to submit bugs in a better, standardized, repeatable and more effective manner. Just posting guidlines in a sticky thread on a forum clearly doesn't work, 'cause some player follow these while the others do not. Maybe some sort of a separate "public" bugtracker database, which won't accept a report entry until player fills all the blanks properly (logs, tracks, screens etc.)?

As for the original post concerns, let's not forget that switching to current trend of updating stable version with one big patch every few months was direct consequence of a period 3-4 years ago when OB updates were thrown out almost every two weeks and stable updates following almost immediately - unfortunately together with a myriad of gamebreaking bugs, which inevitably had to slip through in such haphazard workflow. It just didn't work.

So that's quite a tough nut to crack. Having stable updates more frequently would bring long awaited fixes more frequently... along with higher number of new bugs at the same time. How to find a good balance point?

 

[SharpeXB]

That’s good news. The game will be much better off with a single version for everyone. 

[ignition22]

So.. predictably because too many people are too unreasonable for an open beta to work in a positive way it will become a closed beta. Maybe there will be a way for people to apply to be closed beta testers.  

[Raven (Elysian Angel)]

I like this decision! Thanks ED for taking this thread - and by extension Dangerzone - seriously and responding in the way NineLine did here.
Well done!

I indeed see an extended role for more and properly qualified closed beta testers. Doing away with the concept of 2 branches will bring clarity to many people and will make life easier for many others  

[Dangerzone]

6 hours ago, NineLine said:

Dear Dangerzone,

First, let me start by saying I am very sorry this thread went unnoticed for so long. It is an important issue and your detailed and thoughtful post deserved a response, finally after discussing it with management, here we are. 

Second, this type of feedback is what we look forward to and makes it tougher that I missed this for so long. The answer will be much shorter but please I hope you do not see it a disregard of all your points but rather that we at ED have been seeing what's been going on running two release versions.

So the answer is this... The Open Beta and Stable system of the released will be fully reworked. The process will be changed and all our customers will receive only one version of DCS. Now I do not have any more details than this right now and of course no timeline. But this is the direction we will be headed in the future. 

Thanks once again for the feedback, hope you and yours have a wonderful Holiday Season!
NineLine/Norm 

Hi NineLine,

Firstly, no need to apologise. I'm very grateful that you've taken the time to take my letter to management and that you have responded with information. Thank you!

It's encouraging to see that the challenges server administrators are facing is being considered at management and reworks are in the making. I also appreciate that they are allowing you to be transparent with your response, so we know that there's changes in the making. I fully understand there's no timelines - just knowing that there won't be a continual cycle of what we've seen this year is encouragement enough for now!

I hope you have a wonderful Christmas too, and a safe new year! 

Cheers

DZ

[Dangerzone]

3 hours ago, Art-J said:

^ Hmmm.... I don't know.... The problem is - "our" beta IS the "seperate testing branch". Always has been. Some users are just more interested in testing and reporting properly (they're smaller group I suspect), while the others are more interested in getting access to new content earlier (this is majority I'd hazard a guess). The former will always be active on the forum with more detailed reports, while the latter will not. But that's just a human nature, and In my opinion this proportion will never change. Thus I don't see how creating a yet another, more closed / "elite" separate branch could solve this issue, especially when ED MUST rely on wide scale public testing because of their internal QA resources limitations.

I've no idea how to tackle this problem. They have to persuade/force people somehow to submit bugs in a better, standardized, repeatable and more effective manner. Just posting guidlines in a sticky thread on a forum clearly doesn't work, 'cause some player follow these while the others do not. Maybe some sort of a separate "public" bugtracker database, which won't accept a report entry until player fills all the blanks properly (logs, tracks, screens etc.)?

As for the original post concerns, let's not forget that switching to current trend of updating stable version with one big patch every few months was direct consequence of a period 3-4 years ago when OB updates were thrown out almost every two weeks and stable updates following almost immediately - unfortunately together with a myriad of gamebreaking bugs, which inevitably had to slip through in such haphazard workflow. It just didn't work.

So that's quite a tough nut to crack. Having stable updates more frequently would bring long awaited fixes more frequently... along with higher number of new bugs at the same time. How to find a good balance point?

 

Hi Art,

I understand your hesitation, and you raise good points. You are right, the Open Beta branch has not been used by many (maybe most) for the intended purpose. When it's been released with unfinished bugs, etc - it's been made clear numerous times by ED that it is Open Beta to test out the new features/changes, but many expect it to be stable as they are not running it for Beta purposes. The OB vs Stable 'debate' has not provided any great solutions...

...However, optimistically, I would imagine if Open Beta & Stable goes the way of the dodo and we just have one 'public' release, then public releases will have a higher level of expectations before it's released to the public, as it would no longer be considered a 'test' version. I suspect the result will be less frequent releases, but more stability. 

Now, entering the speculating zone, it may be that ED will open the door for more Closed Beta testers who manage online servers, or play campaigns, or even many who just understand what Beta testing really is about and how to do proper reports. I would imagine with this change, there would be a need for greater scrutiny before public releases, so there could be opportunity there. (Maybe they will look at previous bug reports and vet some people out by previous behavior?  )  This could be the answer to the problem of 'forcing people to submit bugs in a better way', by vetting the Beta testers. The current CB testers no doubt are kept to a higher level of expectation, not only with their bug reports, but also their level of professionalism than some OB users have shown, so it could be a way to reduce the workload, stress and patience on ED's side with beta testing from people who better work with ED's requests?

Personally, I have never applied to be a DCS closed beta tester as I've figured I don't have enough time to give back to be worthy (for lack of a better word) for consideration for CB testing, and I wouldn't want to take a CB slot if I wasn't pulling my weight. Plus... there was an existing beta test option that I have had access to.

But maybe with this future change, there could be provision for people like us who don't have the time to test as thoroughly, but will test on their own servers/missions/scripts and submit bug reports properly to be part of a CB team? 

The simple fact that ED has no more to reveal on this matter gives me the impression that they're still working out the best way to move forward with this themselves, so maybe this is a good opportunity for us to make suggestions as to how this could look, so they can glean from the best suggestions available to help them with their decision process. Either way - it's obvious that the current Open Beta / Stable release cycle is not the ideal approach with the DCS community, and it's encouraging that ED are looking into changes. 

DZ

[Jarmak]

On 12/22/2023 at 3:48 PM, Art-J said:

^ Hmmm.... I don't know.... The problem is - "our" beta IS the "seperate testing branch". Always has been. Some users are just more interested in testing and reporting properly (they're smaller group I suspect), while the others are more interested in getting access to new content earlier (this is majority I'd hazard a guess). The former will always be active on the forum with more detailed reports, while the latter will not. But that's just a human nature, and In my opinion this proportion will never change. Thus I don't see how creating a yet another, more closed / "elite" separate branch could solve this issue, especially when ED MUST rely on wide scale public testing because of their internal QA resources limitations.

I've no idea how to tackle this problem. They have to persuade/force people somehow to submit bugs in a better, standardized, repeatable and more effective manner. Just posting guidlines in a sticky thread on a forum clearly doesn't work, 'cause some player follow these while the others do not. Maybe some sort of a separate "public" bugtracker database, which won't accept a report entry until player fills all the blanks properly (logs, tracks, screens etc.)?

As for the original post concerns, let's not forget that switching to current trend of updating stable version with one big patch every few months was direct consequence of a period 3-4 years ago when OB updates were thrown out almost every two weeks and stable updates following almost immediately - unfortunately together with a myriad of gamebreaking bugs, which inevitably had to slip through in such haphazard workflow. It just didn't work.

So that's quite a tough nut to crack. Having stable updates more frequently would bring long awaited fixes more frequently... along with higher number of new bugs at the same time. How to find a good balance point?

 

This has not been true for a long time now. The purpose of a system is what it does, and the open beta branch has not served as a beta test release in as long as I can remember. It doesn't matter what you call it, if you focus all your marketing and promotion on the open beta as if it is the main product, and go so far as recommend it over your stable version to the multiplayer community, it is your main branch. It has long felt to me like the actual distinction has been one of consumers who are eager for new features vs commercial/institutional customers who require a much higher service level guarantee.

This is the cause of much of the frustration when "it's open beta" is given as an excuse for truly game-breaking bugs; you can't push people to open beta and then scold them when they're upset it doesn't work. 

That said, I don't want to drag ED when they're in the middle of making a good decision. This is a very welcome development and I'm glad to hear it.  

Edited December 24, 2023 by Jarmak

[St4rgun]

IMHO as long as we as OB users are lack of a proper ticketing system with detailed info on each issue this whole OB thing is "questionable". 

Right now it does not matter how good you report any issue even several times: if somehow magaged to reach the "reported" status in the forum there's no further follow ups. I'm literally tired up writing about issues which are several months old. Did it once, twice, third time, fourth time, ... but then simply lost interest when getting no reply, sorry. I don't want to bother the developers anymore.

ED works on their own schedule which is normal, but the situation in the past 2 years made me a passive forum reader and an occassional SP player. It's a pity.

Wake me up if something changed at the foundations.

[Gierasimov]

On 12/22/2023 at 8:18 PM, NineLine said:

One day instead of 2 versions being available to choose from (not including older versions which should still be available as before) there will only be one choice to install. 

I hope you did not just tell us that we all will get cut off the public beta branch, please?

 

[Dangerzone]

6 hours ago, Gierasimov said:

I hope you did not just tell us that we all will get cut off the public beta branch, please?

Maybe you could clarify what your concerns are with going to a single public release please?

ie - Do you actively use the public beta for actual beta testing before deploying missions to a stable release? Or are you more concerned that frequent releases of new features may be a thing of the past as focus comes on having a single public release that is more stable?

The reason I ask is with the intention to try and make this thread as constructive as possible. If you raise what your actual concerns are relating to this new change, it may help in allowing dicsussions/suggestions for ED on what can be done moving forward to alleviate those concerns, if possible. 

[SharpeXB]

20 hours ago, Gierasimov said:

I hope you did not just tell us that we all will get cut off the public beta branch, please?

According to what 9L just wrote, there will no longer be a public beta. So yes, seems rather self explanatory. There is certainly a process to apply for being a closed beta tester so if such things interest you that’s always an option. 

[Gierasimov]

Well, we shall see what does the future hold for DCS in this department then. 

I like Open Beta release frequency, I don't mind testing and  finding bugs.

[NineLine]

On 12/22/2023 at 2:29 PM, ignition22 said:

So.. predictably because too many people are too unreasonable for an open beta to work in a positive way it will become a closed beta. Maybe there will be a way for people to apply to be closed beta testers.  

The importance of the Closed Beta team will increase for sure. Having good people eager to help will also be important. The best way to get noticed is to report bugs as we require, or help others having issues. Be noticeable on the forums for things like this and an invite will come eventually. 

[Firebird1955]

WOW.......22 posts (Multiples by some) and we are going to change the process again.............I must be missiong something.

\

[GGTharos]

On 12/27/2023 at 5:52 PM, NineLine said:

The importance of the Closed Beta team will increase for sure. Having good people eager to help will also be important. The best way to get noticed is to report bugs as we require, or help others having issues. Be noticeable on the forums for things like this and an invite will come eventually. 

Glad to hear you're moving to a single version - that should at least end certain types of arguments.

A bit OT, but related:

Automated testing is a must.  The current problem (and possibly relatively frequent problems) with mission editor triggers and such things for example, are prime candidates for building an automated test harness.   You're in good company not having it so far (Maybe you do, but it appears obvious that this is just not a fact at this time) as there are loads of software out there that don't go through test automation and it really is challenging to automate, but I think you really need that automated regression.

The quality of the software appears to have increased which is great, but unfortunately such bugs are very impactful, highly visible (at least to certain groups) and should be covered by automation because frankly , relying on humans for this tedium is a bad deal.

Edited December 31, 2023 by GGTharos