DCS world new releases preparation, involving key community users?

[FlightControl]

Hello DCS World community, moderators and especially ED,

 

Please take your time to consider the underlying written.

 

"I wonder whether the community can be more proactively and interactively involved in the testing of new releases of DCS World by ED."

 

The simulator is an enormously complex program, and I can imagine that in order to test for a new release, the ED development team, ED testers are facing enormous challenges to ensure that the product stays bug-free and that defects are properly reported and followed-up.

 

What we're noticing within the community is that there are people who have become very acquainted with the simulator in its mission design aspects and mission editing aspects:

- Designing missions using the DCS world scripting API.

- Making highly complex scripts.

- Having a large set of "mods" made within the product.

 

The amount of users and scripts and missions have grown exponentially, and thus, the challenge for ED to release a bug-free product has grown, and the complexity to keep backward compatibility has also grown also.

It has become very hard for ED to test everything keeping this large community happy.

The challenges for ED are in the following areas:

- Ensure that the promised functionality gets delivered within the expected deadlines.

- New modules are integrated in relation with the partners.

- Partner modules can have bugs, and fixes are required.

- Changes in the DCS world core engine may sometimes mean a large rework, with a high risk for bugs. eg. the planned merged product (version 2.5), is a huge challenge for ED today.

- the DCS world lua scripting engine is used within many missions, bugs appear frequently. Backward compatibility is key, and the functions need to work consistently.

- the mission editor needs to keep up with the changes... and keep backward compability with previous designed missions.

- cope with unexpected crashes during run-time, due to memory errors or other internal bugs in the C++ code...

 

What we've noticed in previous releases is that bugs appear, and are staying within the system for a couple of months, if not years. Some bugs however, get quickly fixed, while others are awaiting to get fixed for a (very) long time...

 

And unfortunately, some of these bugs were only detected after "beta testing", in the already release product... And i see a couple of reasons of that:

 

1. Beta releases are appearing very shortly after each other. The community has a hard time to follow-up...

2. Core changes are done in the product, but are not communicated to some community people who maintain complex scripts...

3. I really wonder how closely the community is involved today in bug fix prioritization and bug fixing.

 

Therefore, i have a couple of suggestions to help ED to make this better:

 

1. Would it make sense to involve some people of the community in "pre-release" testing? Especially major releases should be tested properly.

 

2. Would it make sense to establish a communication channel for testing and test results? A proper logging of bug fixes and follow-up? I don't think the ED forum is a good platform to do that. I think something else is needed.

 

3. There is an ED testers team today: Would it make sense to re-evaluate the team, and see if there are people within the community who can contribute to that team also? Possibly early trying out test builds and provide feedback to the ED test team?

 

So, i want to kick-off this discussion on the forum with the moderators and ED dev team, hopefully this post may help to improve the process.

 

I had an offline conversation with Grimes on this topic, and he advised me to open this discussion towards the community, which i hope this starts off with this thread.

 

FC

[Buzzles]

I think you misunderstand who the people with "ED Test Team" tags on the forum are, as they ARE community members.

 

They're (afaik) unpaid volunteer testers, who are invited into the test team after showing the required knowledge and skills.

 

Read Wag's sticky on it: http://forums.eagle.ru/showthread.php?t=50779

 

They basically already cover most of your post and suggestions 1 and 3.

 

Point 2 is already covered by ED's internal bug tracker. They've just (correctly) decided to not make it public.

 

Also, what is there to discuss on this topic? With all due respect to Grimes, this is the type of thing to be kept as PM's between yourself and a member of ED staff (probably Wags) as ED have chosen their approach and the forums have no input on it.

 

ED have already decided to open up testing, they allow public access to the beta release and any one in the community is free to make bug reports in the correct section.

Edited June 16, 2016 by Buzzles

[cichlidfan]

Yet another pointless poll. Buzzles has quite correctly pointed out why.

[FlightControl]

Yet another pointless poll. Buzzles has quite correctly pointed out why.

 

Everything is alright I understand? Did not mean to offend. If the poll is pointless, then let us forget about it. It was meant proactively. It seems I am yet again pointing useless thoughts.

 

Sent from mTalk on Windows 10 mobile

[hannibal]

thats why theres a stable build and Open Beta... tests join in open beta, weary pilots stay with stable....

[FlightControl]

thats why theres a stable build and Open Beta... tests join in open beta, weary pilots stay with stable....

 

 

 

I know ... I know ... Not complaining. This text is not about complaining, but about where can the process be improved? Are we all convinced that the current beta process,is sufficient? Let me be honest. I don't think it is and don't want to complain about it.

 

Sent from mTalk on Windows 10 mobile

[cichlidfan]

The places the process can be improved are all internal to ED and, as such, are beyond our ability to influence.

[wilbur81]

I think a better poll question might be something like:

 

1. Do you think that ED says "yes" to too many projects without completing essential work first?

 

I do appreciate FlightControl's initiative, though, with this poll.

[Bearfoot]

Folks, I think FlightControl is talking about testing the DCS Scripting Engine specifically. He is currently building a library that provides a really powerful, flexible, and clean object-oriented layer on top of the scripting engine. This will be of major benefit to all of us: directly to anyone who wants to build missions, and indirectly to everyone else who just wants to fly missions.

 

Both the Beta and Alpha versions, by their very name, imply "testing" status. But even the builds that are released here lags by quite a bit with the current development snapshot builds. What FlightControl is asking is for access to the snapshot builds. This is so that he can identify not just issues in the DCS scripting engine as in bugs, but also issues that are not technically bug such as sudden/undocumented DCS scripting engine API or implementation changes that break existing missions. The former would be reported to ED, but the latter (which the primary benefit/purpose for the access) is to allow him to fix his own DCS extensions. This is common development model with operating system software as well as many other types of software.

 

As an example, in the last update (or maybe before?), a bug was introduced that resulted in spawned troops not moving or doing anything. This has been fixed in the internal builds for a long time time now. But it has not been pushed to either the beta, alpha, or stable releases for a while. In the meantime, not only are missions relying on this functionality broken (whether using native code, MIST, CTLD, or MOOSE libraries), but presumably Lua developers such as FlightControl find their continuing development work stalled because they cannot adequately run their test scripts.

 

FlightControl is not asking to change ED's internal code. For Lua developers like FlightControl, having access to the in-house latest builds will allow them to run their own tests for their own DCS Lua libraries and check to see what needs to be updated in their OWN code to keep things up to date, BEFORE the build gets pushed to us either in the alpha, beta, or stable installations. This way, when we get the builds we will simultaneously have access to the updated Lua libraries that work with those builds, so none of our missions get broken :) !

 

As a side benefit, while I am sure that the ED testers work very hard in putting the game through the paces, the fact that libraries like MOOSE, MIST, CTLD, etc. make extensive and intensive use of many of the dark corners of the DCS scripting engine that might rarely be touched otherwise by the testers mean that they might be able to identify problems before they get pushed. But that -- i.e. extended testing of the build -- is a side benefit. The primary objective is to allow third-party scripting engine library developers a preview of upcoming build so that they can fix/improve the libraries.

 

@FlightControl --- I think DIRECTLY contacting Matt Wagner or someone else at ED is the best way to go about this: ask to become a tester, saying that you will focus on the DCS scripting engine. Point out how you have thoroughly documented the DCS Scripting Engine API (e.g. https://github.com/FlightControl-Master/MOOSE/blob/master/Moose%20Development/Dcs/DCSGroup.lua), and so are very familiar with it.

Edited June 16, 2016 by Bearfoot

[cichlidfan]

Folks, I think FlightControl is talking about testing the DCS Scripting Engine specifically.

 

If he is, then he omitted some vital words in his post.

[Bearfoot]

I think a better poll question might be something like:

 

1. Do you think that ED says "yes" to too many projects without completing essential work first?

 

I do appreciate FlightControl's initiative, though, with this poll.

 

 

This has nothing to do with either the poll or any of the issues at hand. IMHO, this sounds like a personal axe being ground. Sorry.

[FlightControl]

Folks, I think FlightControl is talking about testing the DCS Scripting Engine specifically. He is currently building a library that provides a really powerful, flexible, and clean object-oriented layer on top of the scripting engine. This will be of major benefit to all of us: directly to anyone who wants to build missions, and indirectly to everyone else who just wants to fly missions.

 

 

 

Both the Beta and Alpha versions, by their very name, imply "testing" status. But even the builds that are released here lags by quite a bit with the current development snapshot builds. What FlightControl is asking is for access to the snapshot builds. This is so that he can identify not just issues in the DCS scripting engine as in bugs, but also issues that are not technically bug such as sudden/undocumented DCS scripting engine API or implementation changes that break existing missions. The former would be reported to ED, but the latter (which the primary benefit/purpose for the access) is to allow him to fix his own DCS extensions. This is common development model with operating system software as well as many other types of software.

 

 

 

As an example, in the last update (or maybe before?), a bug was introduced that resulted in spawned troops not moving or doing anything. This has been fixed in the internal builds for a long time time now. But it has not been pushed to either the beta, alpha, or stable releases for a while. In the meantime, not only are missions relying on this functionality broken (whether using native code, MIST, CTLD, or MOOSE libraries), but presumably Lua developers such as FlightControl find their continuing development work stalled because they cannot adequately run their test scripts.

 

 

 

FlightControl is not asking to change ED's internal code. For Lua developers like FlightControl, having access to the in-house latest builds will allow them to run their own tests for their own DCS Lua libraries and check to see what needs to be updated in their OWN code to keep things up to date, BEFORE the build gets pushed to us either in the alpha, beta, or stable installations. This way, when we get the builds we will simultaneously have access to the updated Lua libraries that work with those builds, so none of our missions get broken :) !

 

 

 

As a side benefit, while I am sure that the ED testers work very hard in putting the game through the paces, the fact that libraries like MOOSE, MIST, CTLD, etc. make extensive and intensive use of many of the dark corners of the DCS scripting engine that might rarely be touched otherwise by the testers mean that they might be able to identify problems before they get pushed. But that -- i.e. extended testing of the build -- is a side benefit. The primary objective is to allow third-party scripting engine library developers a preview of upcoming build so that they can fix/improve the libraries.

 

 

 

@FlightControl --- I think DIRECTLY contacting Matt Wagner or someone else at ED is the best way to go about this: ask to become a tester, saying that you will focus on the DCS scripting engine. Point out how you have thoroughly documented the DCS Scripting Engine API (e.g. https://github.com/FlightControl-Master/MOOSE/blob/master/Moose%20Development/Dcs/DCSGroup.lua), and so are very familiar with it.

 

 

 

Bearfoot, you posted the right words... My apologies to those I may have offended or annoyed. I meant it well and proactively.

 

 

 

Sven

 

Sent from mTalk on Windows 10 mobile

[NeilWillis]

Perhaps we ought to have a poll to decide whether we want more polls or not?

 

It might go some way to address the issue of pointless polls. :music_whistling:

[FlightControl]

Perhaps we ought to have a poll to decide whether we want more polls or not?

 

 

 

It might go some way to address the issue of pointless polls. :music_whistling:

 

 

 

Good idea, let's do that.

 

Sent from mTalk on Windows 10 mobile

[wilbur81]

This has nothing to do with either the poll or any of the issues at hand. IMHO, this sounds like a personal axe being ground. Sorry.

 

No need to apologize, Bearfoot. You're wrong. :-)

[NeilWillis]

It is good that people are thinking proactively, any way we can assist ED in refining and improving the whole package is always very welcome. My earlier remarks were aimed at the way you assume having a poll will in any way contribute to the furtherment of DCS World.

 

ED ARE listening, but never ever assume this is a democracy! By all means offer your services as a tester, but assuming a poll will add to the suggestion by getting everyone to vote on it is frankly amusing.

 

So, ask yourself, was the poll either necessary, useful, or in any way adding to your well made point? If anything, it cheapens things in that it implies you want go gather a crowd of pitchfork and flaming torch weilding peasants to get ED's attention.

[Wolf Rider]

Perhaps we ought to have a poll to decide whether we want more polls or not?

 

It might go some way to address the issue of pointless polls. :music_whistling:

 

 

Love your reasoning... seem to have also time limited the "remember me Log in" also, seems to be a bit of a 'boys club'

Edited June 17, 2016 by Wolf Rider

[FlightControl]

It is good that people are thinking proactively, any way we can assist ED in refining and improving the whole package is always very welcome. My earlier remarks were aimed at the way you assume having a poll will in any way contribute to the furtherment of DCS World.

 

 

 

ED ARE listening, but never ever assume this is a democracy! By all means offer your services as a tester, but assuming a poll will add to the suggestion by getting everyone to vote on it is frankly amusing.

 

 

 

So, ask yourself, was the poll either necessary, useful, or in any way adding to your well made point? If anything, it cheapens things in that it implies you want go gather a crowd of pitchfork and flaming torch weilding peasants to get ED's attention.

 

 

 

 

 

Thanks for your message. You're right. Having a poll may not have been the best way forward.. My GOD! I feel I've been smashing some windows and opening Pandora's boxes with this post... To be honest, when I was writing the post, it kind of flashed through my mind: "Hey!" There is a poll facility on the forum! Great! Let's try out...". Shouldn't have done that ☹ so again my apologies to anyone I may have upset...

 

 

 

That being said, What Bearfoot wrote is exactly why I started with this.... Track the forum, many examples of disappointed people and people reporting bugs, unexpected ones, in stable release versions. And those posted in beta are logged. And in the updates and change logs we receive, how many of these are fixes are related to logged bugs? Sometimes yes, by really not often.

 

 

 

And while building this framework, I want to help ed, and I think i can, because there are lots of great test missions (and in the make), testing the framework api, while also testing the dcs scripting api, with at least a 85%) test coverage. Many other people who have scripts have similar situation. The thing is, when we go on beta, we are at risk that the code in the framework stops working, and the earlier we can validate and test this in the process the better. Grimes also has pre-test versions...

 

 

 

Sven

 

Sent from mTalk on Windows 10 mobile

[NeilWillis]

I wouldn't worry about upsetting people here, breathing is sufficient at times. I have a feeling that the bug fixing cycle at ED is more down to impact and is placed on a long long to-do list if it only has a relatively minor effect on the overall DCS World experience.

 

I like your positive attitude towards that end, definitely. Testing the more obscure areas of the game is highly relevant, and in effect, just reporting your findings here in a logical and comprehensive way using their preferred bug reporting system will effectively make you a tester - albeit unreconised and unrewarded. ED do carry out head hunting exercises, so don't be surprised if you get a message some day saying come join our testing team.

 

So Sven, keep at it. And never worry about saying what you think, and just accept that upsetting forum members is unavoidable - it doesn't mean what you have to say isn't highly relevant even if the great unwashed take umbrage.

[Kuky]

If you were in same line of business as ED is, you would see that the way they are working and how they have testing sorted out, is a good way.

[sorcer3r]

Question about DCS development and communtiy "participation" is an old one. ED has its own priorities and my lesson was lower your expectations.... drastically. ;)

 

 

1. Beta releases are appearing very shortly after each other. The community has a hard time to follow-up...

 

 

+1

 

Unfortunately DCS open beta releases are not as successful as I was hoping before.

For example broken triggers in final releases are in my opinion a no-go.

[Pikey]

Once they get the version conversion done and some of these multiplayer and multiseat issues done there may be some time to get through the back log of tickets that are triaged, @Flightcontrol, I just dont see this period as a great one for development of anything DCS related, not even modules. I've not seen the backlog but I suspect its pretty huge judging from the snippets in the scripting known issues threads and the bugs ive seen in the last 6 months. I once told someone about Mision editing, its not to do with understanding th eME and scripting, its to do with working round th eknown issues or new issues with imagination.

 

They get through bugs, they get through loads of bugs in fact, you can see that build on build, but frankly I can tell where the priorities are by the types of bugs fixed and the types that are remaining and the scripting engine isn't top drawer right now, all seriousness considered equal. Even some that are reported but get fixed faster because someone moaned harder (I put my hand up on that one)

 

I can understand frustration, i'm sure something like MOOSE would be great if the key issues could be resolved. The fact is that the ED testing team have plenty of hard working technical community contributors in it that sink hours into the games various builds. You couldnt pay me to do that, it would destroy every last enjoyment I have of DCS if I had to spend 6 hours chasing an intermittent landing bug or whatever. Throwing more people at it is not a solution, refreshing them might be but I can assure you they take new guys on and they are very good.

 

This community is a key tester itself, at a level that is right, be proud to be part of what makes DCS and understand it for what it is. You could report 10 Bugs right now, if you really wanted to. Or as a tester you could try to reproduce 10 peoples bugs already done...

[sorcer3r]

Once they get the version conversion done and some of these multiplayer and multiseat issues done there may be some time to get through the back log of tickets that are triaged, ....

 

Some years ago I was told when work on DCS world 1.0 is done things will improve... but yeah.. maybe this time. ;)

[FlightControl]

Once they get the version conversion done and some of these multiplayer and multiseat issues done there may be some time to get through the back log of tickets that are triaged, @Flightcontrol, I just dont see this period as a great one for development of anything DCS related, not even modules. I've not seen the backlog but I suspect its pretty huge judging from the snippets in the scripting known issues threads and the bugs ive seen in the last 6 months. I once told someone about Mision editing, its not to do with understanding th eME and scripting, its to do with working round th eknown issues or new issues with imagination.

 

 

 

 

 

 

 

They get through bugs, they get through loads of bugs in fact, you can see that build on build, but frankly I can tell where the priorities are by the types of bugs fixed and the types that are remaining and the scripting engine isn't top drawer right now, all seriousness considered equal. Even some that are reported but get fixed faster because someone moaned harder (I put my hand up on that one)

 

 

 

 

 

 

 

I can understand frustration, i'm sure something like MOOSE would be great if the key issues could be resolved. The fact is that the ED testing team have plenty of hard working technical community contributors in it that sink hours into the games various builds. You couldnt pay me to do that, it would destroy every last enjoyment I have of DCS if I had to spend 6 hours chasing an intermittent landing bug or whatever. Throwing more people at it is not a solution, refreshing them might be but I can assure you they take new guys on and they are very good.

 

 

 

 

 

 

 

This community is a key tester itself, at a level that is right, be proud to be part of what makes DCS and understand it for what it is. You could report 10 Bugs right now, if you really wanted to. Or as a tester you could try to reproduce 10 peoples bugs already done...

 

 

 

 

 

 

 

As a matter a fact I have reported 10 bugs. Of which 8 are open.

 

 

 

http://forums.eagle.ru/showthread.php?p=2759100

 

 

 

 

 

 

 

Sent from mTalk on Windows 10 mobile

[Seaeagle]

3. There is an ED testers team today: Would it make sense to re-evaluate the team, and see if there are people within the community who can contribute to that team also? Possibly early trying out test builds and provide feedback to the ED test team?

 

The "ED testers team" is exactly recruited from the community and regularly "re-evaluated" - i.e. some people leave, while others join. However, a test team needs to be managed in order to provide a useful service to the developers, so there is a limit as to how many people it is feasible to draw in.

 

What we've noticed in previous releases is that bugs appear, and are staying within the system for a couple of months, if not years. Some bugs however, get quickly fixed, while others are awaiting to get fixed for a (very) long time...

 

That some bugs remain unfixed for ages is usually not down to insufficient testing resources, but rather to how they are prioritised by the "fixing resources" :) .