Beta vs. stable vs. EA — bashing it out

[Tippis]

Since the topic keeps coming up due to the roll-out of 2.5.6, and are causing all kinds of OT deletions and mod headaches wherever it sweeps over the land, I though it might be worth-while trying to create a thread where we bash out the question of what the different releases are really for. This is particularly relevant now that there is talk of having a pre-beta tier for extra-volunteer testing.

 

What's the actual purpose of having a beta version, a stable release, and where do all the early access modules fit into that? And just as importantly, irrespective of the “actual purpose”, how are the different branches really used, and why?

 

On the one hand, beta is — as the name implies — for beta testing.

On the other hand, how many actually use it for that purpose?

On the one hand, stable is obviously supposed to be for those who want a… well… stable ground.

On the other hand, beta has all the latest features and solutions and bugfixes, and is often (but obviously not always) more stable than the stable release.

On the third hand, beta may have the latest features and bugfixes but it also has the latest bugs and broken things.

On the one hand, EA modules by their very nature belong in a beta branch.

On the other hand, stable is full of EA modules in some cases in a worse state than EA modules only available through beta.

On the one hand, stable has the appearance of just being an older beta with a minimised bug count — it's just differently bug-riddled, not bug-free.

On the other hand, how else can fixes be deployed and not completely fracture everything? Is it even possible to roll out individual solutions to a separate branch or do we have to accept that DCS and/or the modules are pretty monolithic in how they can be delivered?

 

So it all comes back to those earlier questions: what is beta for? Is that really how it's being used? If not, why is being used differently? Can that “why” be satisfied in a different way, such that beta can be used for its intended purpose without interfering with the “newest and most improved” use case that (let's face it) even ED is incentivising?

 

It would be particularly interesting to hear the devs' thoughts on this: is the beta branch serving its purpose or has it morphed into something else that is now becoming a problem of its own? Or has it morphed, but it's not actually problem from a dev / feedback standpoint?

[Mr. Big.Biggs]

I think it would be a great Idea for the mods to close this thread before it starts.

Just my opinion.

Regards

[Boosterdog]

I think it would be a great Idea for the mods to close this thread before it starts.

Just my opinion.

Regards

 

Not just yours :)

[Tom Kazansky]

Or we could wait and see what ED's new closed beta system brings and discuss all that afterwards?

[Tippis]

Or we could wait and see what ED's new closed beta system brings and discuss all that afterwards?

Maybe, but that sort if just reinforces the question imo: if they're going to implement a pre-beta, then there's probably a reason for that. Is it precisely because the regular beta has gone of to become something it wasn't initially intended to be? And what will the key distinctions (in terms of the path to “release”) be between the two tiers?

 

Is it just to file off the rough edges so that the beta can serve the purpose it has organically grown into over the years? Is it a difference between obvious-flaw-testing vs. the kind of edge case testing that only a much larger number can readily provide? And where does that leave the stable branch? Will it simply be relegated to a low-maintenance SP campaign only status, whereas more people will actually migrate to the beta now that it, too, will be a bit more stable right out of the gate?

 

That's why I think the whole intent vs. actual use is a pretty interesting topic to chip away at from various perspectives — now, more than ever.

[dburne]

Sheese Open Beta and Stable have been going on how many years now? C'mon!

Hey at least we do not have Open Alpha any longer.

 

The difference between the two options is so obvious even a caveman get it.

:smilewink:

[Tippis]

Sheese Open Beta and Stable have been going on how many years now? C'mon!

Hey at least we do not have Open Alpha any longer.

That's kind of the point, isn't it?

 

They're now doing something quite similar again, and as the discussions surrounding the latest release have shown, there are quite a few different views of what purpose the different branches can, should, and actually do serve. Hence the question: it should be obvious, but it even more obviously isn't, so can the different use cases actually be satisfied in a more constructive way? Is that part of what the closed beta is trying to solve?

[Tom Kazansky]

Is the point: How to get a stable open beta?

 

 

Good luck.

[Tippis]

Is the point: How to get a stable open beta?

It's more like: is beta perceived as and/or slowly turning into a different level of stable?

[randomTOTEN]

Instead of trying to make inferences based on your own experience and discussion with other users, you could check the "Technical Questions" section of the main website, which offers the developers intentions...

 

 

The difference between Stable and OpenBeta

 

Speaking of Open Beta and Release versions, we currently operate two versions of DCS World, the Open Beta version and the Release version (many fewer versions than before!). Because DCS World is a “living” project that changes daily, it’s possible for a project-wide bug to worm itself in and make the game unplayable for all. To prevent this and add a level of protection, we adopted Open Beta and Release versions.

Almost every two weeks (sometimes more with hot fixes) we create a new Open Beta version that comprises the changes from the prior two weeks. Although this first goes through both internal and closed-external testing, it is possible for errors to slip through. In this case, the Open Beta is utilized to identify a “blocking” bug that prevents all customers from playing the game (not just customers of a single module). For those that take part in the Open Beta, we greatly appreciate it and thank you for your testing efforts. If a “blocking bug” is present in the Open Beta, we always have the Release version available. This is why the Release version is sometimes called the Stable version.

As such, the Open Beta version is for us to “field-test” new changes and check for any DCS-wide “Blocking” crashes, and the Release version is the result of the previous Open Beta that does not suffer any “Blocks.” As with Early Access, Open Beta is an option and may not be for everyone.

What is DCS World Early Access?

 

Early Access is an option for you to play this module in an early state, but it will be incomplete with bugs. The time a product remains in Early Access can vary widely based on the scope of the project, technical hurdles, and how complete the module is when it enters Early Access. Eagle Dynamics and all of our third parties strive to make this period as short as possible. An Early Access module can be played on both the Open Beta and Release versions of DCS World. Once the module exits Early Access, you will automatically have the Release version.

Edited March 7, 2020 by randomTOTEN

[Tippis]

Instead of trying to make inferences based on your own experience and discussion with other users, you could check the "Technical Questions" section of the main website, which offers the developers intentions...

…which is not even half of the question, and only tangentially related to the actual key issue at hand, not to mention that it is also potentially outdated now that they've decided to add another testing tier. Hence the OP.

[Arctander]

‘ The time a product remains in Early Access can vary widely ... Eagle Dynamics ... strive to make this period as short as possible’

 

I would contest that this is still accurate, as we have seen where ED have decided to move resources from an EA product to enable another EA product to be released - and simple logic says if one product is being worked on by a defined size team, and that team then have two products to develop and support, the time to complete the first module WILL be longer.

Edited March 7, 2020 by Arctander
Emoticon didn’t work

[Caliboy42]

I can see a few good reasons to close this thread as has been suggested by a couple of members but letting it run wild might serve a higher purpose. Let me give it a shot at changing the trajectory...

Please, God. Let every OB user read this. Amen...

[Baldrick33]

I can see a few good reasons to close this thread as has been suggested by a couple of members but letting it run wild might serve a higher purpose. Let me give it a shot at changing the trajectory..

I was going to agree with your opening line before the but, then I read the rest. Great post!

[jasonbirder]

To me the question is...why do simple "bug-fixes" I'm thinking principally of 3rd party written fixes to DLC campaigns...take so long to make their way to "stable" release?

 

 

EG: Baltic Dragon has written a couple of mission fixes for the A10C TEW/M2000C Red Flag campaigns...

 

 

What is the logic behind having to wait for ED to sort out the new lighting system 2D & VR performance issues etc...big complex problems that will likely take weeks to fix before they're rolled out to the overwhelming majority of users (IE: those not "beta-testing") ?

 

 

I'm not a purchaser -but the same argument could be made about people who purchased the JF-17 and don't have access to it...why are they waiting for substantive issues with the core modules being fixed...when the Plane itself could have been added to stable some time ago...

 

 

If the policy wasn't - until every single thing we've put in the latest Beta release - no matter what/ no matter how complex or how trivial it is works...then nobody can have any of it (except beta testers)

 

 

 

Broad brush it skews/degrades what Open Beta is...99% of people on open beta aren't "Beta Testers" by any stretch of the imagination...its just become the de-facto release for many...

[RuskyV]

As i learned from a previous thread and if correct then you have all the early access modules in the stable version of the sim. If you are having issues with the Beta and its not for you then the best thing to do is install and run the stable version only.

 

The core issue is most people are choosing to run the beta because they want the shiny stuff first, the problem is everyone (including myself) loses sight of the fact that a beta will break every now and then.

 

If more people were on the stable release there would probably be less winning (no disrespect to the OP) about broken stuff. Could ED communicate or do a better job of informing people that the Beta is not as stable as the Released version? Sure they could but they need people to help develop it.

 

Maybe a way forward is to have a Beta version that is an application based that way you could have a better concentration of players on a stable release and then still have a healthy amount of beta testers to help develop the game. It would keep things ticking over and prevent the forums filling up with rants..

[Tippis]

The bugs, undelivered expectations, and everything I've seen since the latest in 2.5.6 release are old, old news. They are baked into the development cycles and the risk involved in bring new software to life. They always will be. We can't wish or order them away.

 

So too will people be with us who cannot manage their expectations. Their expectations. Not ED's. It's not ED's problem. ED and a long list of developers have a part in building unreasonable expectations sometimes. Most of the times it's customers who don't take the time to find out what they're getting in to or only know what the want, right now. Devs also don't deliver, often, though it's usually a matter of falling short rather than failure.

That's the balance and interaction that I think often causes this whole mess. It's old news, as you say, so why is there no learning on either side of the equation? The developers are good at building expectation, and better still at hyping and pre-selling content… and then it gets stuck in beta limbo, perhaps longer than it needs to be. The somewhat incongruous approach they have to moving things from beta to stable, and the equally incongruous approach to what retains the EA label, and how long, just fuels that fire.

 

Maybe it could be as simple as allowing more stuff “release” earlier would take some of the pressure off of the beta and make stable more viable? The current popular example seems to be the Jf-17, which is in many instances more complete and stable than some stuff that's available in the stable branch… so why isn't the Jeff? There's probably some logic to it, but what is it? That's one of those things that could probably be clarified to help manage those expectations.

 

To me the question is...why do simple "bug-fixes" I'm thinking principally of 3rd party written fixes to DLC campaigns...take so long to make their way to "stable" release?

 

[…]

 

What is the logic behind having to wait for ED to sort out the new lighting system 2D & VR performance issues etc...big complex problems that will likely take weeks to fix before they're rolled out to the overwhelming majority of users (IE: those not "beta-testing") ?

Quite. That's why I'm quietly wondering about how monolithic the DCS builds really are. Are there interdependencies that we aren't seeing that keeps smaller facets from being rolled out to stable at a quicker clip, or is it a matter of how dev branches are distributed and then recollected in preparation for a new stable roll-out? Or is it simply that ED's own sprint-release cycle is what dictates how everyone else's fixes are incorporated — i.e. it doesn't matter if if third-party module X was ready 8 weeks ago, because ED has decided that they themselves will go two more sprints before a new stable release is cobbled together?

 

Tthat kind of gap in the understanding of the path to a stable release is probably another factor in why expectations and reality don't quite match: the developer notes that a bug has been identified; that a fix has been created; that the fix will be part of the next release, and indeed it is… for the beta branch. There's no telling when it will arrive in the stable branch (where the bug quite likely also exists and will continue to do so for quite a while).

 

The core issue is most people are choosing to run the beta because they want the shiny stuff first, the problem is everyone (including myself) loses sight of the fact that a beta will break every now and then.

In a funny way, I think maybe the beta branch is a victim of its own success. For the most part, it doesn't break and often ends up being more stable than the stable branch. Add to that the fact that the newest and hype:est (yes, it's a real word, shush! :D) is available on beta for a long time before it reaches stable, and of course people will favour the beta branch.

 

With how good at is at being better than stable, it's also no wonder that people lose sight of the fact that it will break — indeed, that part of its entire purpose is to break.

 

Maybe a way forward is to have a Beta version that is an application based that way you could have a better concentration of players on a stable release and then still have a healthy amount of beta testers to help develop the game.

I'm not entirely sure I understand what you're suggesting. Could you elaborate? It certainly makes sense that, if possible, the beta should be more beta — more clearly and explicitly rolled out for testing than for every-day gameplay — but how on earth do you make that happen? Especially now that it seems like they're slowly letting the beta-as-regular-version philosophy take an even greater hold by having a pre-beta that hopefully gets rid of the ugliest warts before regular beta starts.

[maxTRX]

...(I only wish all my old mates from 3 SQN were back flying.) This is a chance to sharpen old skills, learn new aircraft, and have fun in the community. Flying with what is here today, with all the warts and flaws, is one helluva lot more fun than whining about what isn't...

 

Yep, in spite of all the "persistent" bugs, unfriendly net code, hacks (those were fun:D) back in the Lomac days, who can forget the inter-squadron events, learning or developing tactics or just going through training syllabus... I guess some of this is still going on today.

3 SQN? Was that former 169th? I briefly flew with some of them. Haven't seen any of the callsigns on the forum recently. Some of them probably got renamed:D

[Baldrick33]

Maybe it could be as simple as allowing more stuff “release” earlier would take some of the pressure off of the beta and make stable more viable? The current popular example seems to be the Jf-17, which is in many instances more complete and stable than some stuff that's available in the stable branch… so why isn't the Jeff? There's probably some logic to it, but what is it? That's one of those things that could probably be clarified to help manage those expectations.

It is pretty common practice to promote a beta release to stable once fully tested in a user environment. If you opt in or out of SteamVR or WMR betas they work in exactly the same way as DCS.

 

Adding further branches adds to complexity. The JF-17 may require underlying components in the beta, so adding it in isolation to stable may not work, it would certainly need extensive testing so you would need a beta2 or something to test in a user environment before that gets promoted to stable. Then something else needs a beta3 if it might be introduced independently and you have to merge beta2 and beta3 to beta4 to ensure they work together. It just becomes a nightmare managing so many branches and the risk of adding ever more issues.

[RuskyV]

I'm not entirely sure I understand what you're suggesting. Could you elaborate? It certainly makes sense that, if possible, the beta should be more beta — more clearly and explicitly rolled out for testing than for every-day gameplay — but how on earth do you make that happen? Especially now that it seems like they're slowly letting the beta-as-regular-version philosophy take an even greater hold by having a pre-beta that hopefully gets rid of the ugliest warts before regular beta starts

 

Sure, Maybe its possible to have an enrolment type closed beta that would still be open to all to apply but to have a limited number to be defined by ED so that they would have the right numbers and people making the right reports. This would cut down on the amount of work they must spend hacking through forum posts trying to extract issues from those that are just moaning.

 

This would also insure that the stable version is always the first option, and then upon request and providing there are spaces you can enter the beta.

 

I don't think this will cut down on admin so much as focusing the information in the forums to the right place,

 

this would also benefit the community, in fact i think it would show steam users that there is a larger community than is currently perceived as there are so many on the beta program.

 

Not sure if this clears it up, so much that it probably is opening yet another can of worms..lol

[will-]

Sure, Maybe its possible to have an enrolment type closed beta that would still be open to all to apply but to have a limited number to be defined by ED so that they would have the right numbers and people making the right reports. This would cut down on the amount of work they must spend hacking through forum posts trying to extract issues from those that are just moaning.

 

its called Open Beta. case closed johnson.

[Tippis]

Sure, Maybe its possible to have an enrolment type closed beta that would still be open to all to apply but to have a limited number to be defined by ED so that they would have the right numbers and people making the right reports. This would cut down on the amount of work they must spend hacking through forum posts trying to extract issues from those that are just moaning.

Ah, ok. That sounds a lot like what the seem to be planning, but the details on the new beta haven't exactly been extensive so who knows how it'll end up.

[RuskyV]

who knows how it'll end up.

 

For the better we hope, but i guess time will tell:pilotfly:

[rwbishUP]

Why you demand this thread gets closed?

 

That's what I was wondering.

[Pikey]

OB would be fine if it werent for the fact that most servers run on it and many users want it to.

 

 

Which places this fully in the realm of user driven disatisfaction.

 

 

OB is fairly a victim of it's own success, ED release it usually playable in most respects. The problems arise when issues affect multiplayer, servers, mission creation, or sometimes the plain and obvious issues that affect everyone equally. Then servers roll back, people get annoyed because they have to eithe rplay single player or suffer some bug they dont want.

 

 

 

ED on the other hand love that as the noisy complaints come in thick and fast and they get to see the temperature of the issues fast.

 

 

So, ED dont want to change it, Many people dont want to change it, and server admins are miserable watching progress bars for days on end.

 

 

It's absolutely abused to hell and wrong. We live in OB. and should not, for the want of waiting a bit more we are essentially forcing ourselves to play a less stable/optimised/bugfree game.