+- CC Zone - Chip's Challenge Forum (https://forum.bitbusters.club)
+-- Forum: Chip's Challenge (https://forum.bitbusters.club/forum-4.html)
+--- Forum: CCLP Discussions (https://forum.bitbusters.club/forum-15.html)
+--- Thread: CCLP1 Voting Proposals (/thread-1959.html)
CCLP1 Voting Proposals - jblewis - 14-Mar-2013
Hi everyone,
While we're still trying to finalize the details regarding our voting administrator, I'd like to go ahead and present everyone else in our community with a proposal for voting and where CCLP1 will go in the days to come. I've tried to consider the feedback I've received from my previous posts about CCLP1 from months past. That being said, we'd like to present to you the following for consideration and further discussion.
--- 1. PRE-VOTING ELIMINATION ---
CCLP3 had (if memory serves correctly) about 2,030 levels in its initial round of voting, and all were available for the public to play at once. Several hundred of these levels were eliminated after about one month of voting. For CCLP1, one of the key goals we've tried to establish on the staff is to encourage voters to distinguish good levels from great levels. "Great" doesn't necessarily mean ultra-difficult or ridiculously clever. Sometimes, it's all about the little things, too: design quality, the packaging of the familiar in an innovative way, and consideration of new players. And ultimately, we're trying to seek out the levels that are the best fit for a set that's meant to serve as a CC1 replacement for beginners.
Toward that end, we'd like to conduct an elimination prior to the beginning of voting. This is a sensitive issue but one we felt was necessary for a number of reasons:
- Narrowing it down to good levels. During CCLP3 voting, some voters felt like many top-notch levels were shafted. Obviously, this will always happen when there are a lot of good candidates, but while there can't ever be a guarantee that every level in the voting process will receive adequate consideration by every voter, it's much more likely to happen if the field is narrowed down to good levels to begin with - without non-CCLP-worthy material to worry about - so that the great levels can shine a bit more easily.
- Removing a heavy burden from level designers. One of the most overwhelming feelings a designer could ever feel during CCLP3 voting came upon visiting a levelset info page on Mike's site, seeing a bunch of "unequal" levels throughout a set, and feeling the need to correct every one before voting, which was especially frustrating if the designer didn't even build levels with Lynx compatibility or appropriate CCLP1 difficulty in mind to begin with. With a pre-voting elimination, we can focus on encouraging level designers to correct the levels that could actually stand a chance in the voting process.
- Removing a heavy burden from voters. One of the issues that the CCLP3 staff noticed during its voting process was that some community members started playing some levels but then stopped. While we can't completely prevent that here, I think it's much less likely to happen if the number of levels in voting is much less overwhelming to begin with. More ideas on how we can further accomplish that in a bit, though.
- Shifting the focus from levelsets to levels. One of the initiatives we're hoping to try to push for CCLP1 voting is to conduct the voting across level packs of 50 or 100 levels each, randomly picked from the narrowed submission pool. I'll explain more about how we can make this work in Section 2, but one of the reasons why we felt like this was a good idea was because of "levelset bias." In CCLP3 voting, it was very easy for voters to assign ratings to individual levels based solely on who designed the levels or in what set they appeared, primarily because voting was categorized by set. This especially was an issue with the difficulty rating, but we'd like to avoid having it affect the "fun" rating as much as possible, especially for CCLP1. While randomly ordered levels can't completely remove certain biases (for example, a recency bias in which one's opinion of the previous level solved in the pack affects his or her opinion of the current level being played), they can at least encourage the evaluation of levels on each level's own merit, not by who designed the level or what else the designer made.
Please note that we would like the community to be involved as much as possible in the elimination and have the final say concerning what makes it into voting. Toward that end, we'd like to propose the following steps:
- The staff puts levels being considered for the cut on a "marked for elimination" list. The criteria for elimination are pretty simple: (a) excessive difficulty for CCLP1, ( B) excessive triviality (for instance, a level that involves just walking to the exit), © poor design quality that is not on par with the standard established in the original CC, (d) non-Lynx-compatibility that is practically beyond repair, (e) excessive length (especially if the level is extremely monotonous), and (f) any other criteria that defined levels eliminated in the first round of CCLP3 voting except for optimization-related criteria (in other words, we won't eliminate a level because it's not optimization-friendly or contains randomness). To translate our opinions into something more quantifiable, we've been assigning numerical ratings based on fun and beginner accessibility using a -2 to +2 scale for each. Please note: beginner accessibility is NOT difficulty - it's how reasonable a level would be for a beginner at an appropriate place in the set.
- Levels above a certain cutoff point would be placed on a "whitelist." I'm considering 0 and higher for fun, and 1 and higher for beginner accessibility. The staff will confer about the whitelist's content and double-check to ensure everything is in order and that a consensus is reached before proceeding any further. At this point, levels that need compatibility or bust fixes will also be added to the whitelist as well but marked so that they could be fixed. (Based on the increase in knowledge about the Lynx ruleset since CCLP3's release, as well as the large percentage of non-Lynx-compatible levels that fall into criterion (d) described above, I don't think this list of levels to be fixed will be huge.) In the end, I'm hoping the whitelist will contain about 800-900 levels.
- The whitelist and the "marked" list will both be presented to the community publicly. At this point, community members will have a set amount of time to (a) correct any levels in the whitelist that need fixes and ( B) nominate any "marked" levels they wish to see in voting for whitelisting. We may have to come up with some rules to ensure that some designers don't abuse this ability for their own levels, but we can discuss options for how to proceed among the rest of the community. That's what this thread is for. Once this period ends, the remainder of the levels on the "marked" list will be properly eliminated, and the whitelist as it stands will be randomly jumbled to form the level packs.
- Voting will begin.
--- 2. VOTING LEVEL PACKS ---
I've already mentioned above why we'd like to go with this option, but I'd like a few opinions about what everyone believes to be reasonable. How many levels do you think you can complete and reasonably evaluate in about a month's time? 50? 100? Even more? Please bear in mind that CCLP1 voting won't have levels whose difficulty matches some of the most devious triple-digit entries of CCLP3, so solving them shouldn't be terribly hard for most of us veterans.
One issue we've got to decide, though, is what to do about levels that need fixes. Should we cut off all level modifying at a specified date before voting starts, or allow level designers time until the pack in which the level in question is up for voting? I'm leaning toward the former, if only so we can be consistent across the board with respect to making changes to levels and avoid any confusion. Also, the latter option doesn't allow for flexibility in scheduling the pack releases. Speaking of which...
Another issue is how the packs should be released. Should it be done all at once or pack by pack throughout the voting process? There's pros and cons for either option, but personally, I'm more in favor of the latter. From the standpoint of making sure each level in voting receives as much consideration as possible, a small number of levels on which voters can concentrate at a given time will most likely be more rewarding in terms of getting more informed ratings than releasing everything at the same time. And although we would be running the risk of having Pack #1 receive more consideration than the rest by virtue of being first, the staff will take on additional marketing responsibility to promote interest in subsequent packs and create anticipation. Either way, any pack that has been released would remain released and available for voting until voting ends completely, which most likely won't happen until the staff feels like everyone's gotten a chance to play every pack and vote as much as voters see fit.
--- 3. RATINGS USED IN VOTING ---
This was an issue I brought forward several months ago. Back then, I proposed the idea of including a "beginner accessibility" rating (or "BA") instead of difficulty. Since one of our objectives was to create a set friendly to first-time players, we thought that evaluating the level based on how reasonable it was for that group of players at an appropriate place in the set - regardless of actual difficulty - would be more beneficial. But as the discussion continued, I started to wonder if BA might be more useful for the staff to use in deciding what gets cut in a pre-voting elimination, whereas the only rating voters would even have to worry about would be the "fun" rating, in which beginner accessibility and all other qualities could be considered. A couple of proposals:
First, we'd like to ditch the name "fun" and just give this rating the generic term "rating." After some discussion, we started to realize that we needed to encourage voters to evaluate each level as holistically as possible and not just focus on any one concept isolated from everything else: for instance, how unique the concept or presentation is, how great the design is, or how fun the level is to play. All of these factors are important, but we'd like to encourage voters to keep all of them in mind as they vote. I think renaming the rating, along with putting some guidelines on each voting page for making sure one's vote is as informed as possible, can help with that.
Second, we'd like to abandon the difficulty rating altogether. The CCLP1 staff would like to do as much as possible post-voting to ensure that a proper difficulty curve is established throughout the set, and toward that end, we'd like to rate the top 250 or so levels for difficulty ourselves and go from there (more on that in the next section). Not only will this be easier on us, it will be even easier for voters when there's only one rating to worry about!
--- 4. POST-VOTING ASSEMBLY ---
In CCLP2 and CCLP3, the levels inducted into the final set closely resembled what the top 149 levels on the voting results were. It was especially the case in CCLP2. In CCLP3, we had the wildcard round to further refine the results for the "bubble" of levels around the cutoff point, plus we removed some really difficult levels and replaced them with somewhat easier ones for variety's sake. These included Toggle Bust, Blue Blocks, The Haunted House, and Jaywalker. For CCLP1, the staff would like to have the final say in judging what ultimately gets inducted into the set while using the voting results as a guideline. A few reasons why we believe this would be necessary:
- Establishing a proper difficulty curve using the steps described above and ensuring that there aren't too many levels of any particular difficulty.
- Creating as much variety in the set as possible. This was one complaint levied toward CCLP3 that would have been somewhat worse if not for the staff's intervention with the inclusions mentioned above, and we'd like to continue that tradition here with more freedom and flexibility.
- Related to the above point: removing levels that are too similar. This will help ensure that we won't have pairs of levels in CCLP1 like Tool Box / Which One Next?, or Caves / Divide and Conquer, unless there's a very good reason to do so.
Let us know what you think. We'll be sure to listen to everyone's comments and suggestions before we take the next big step. Thanks for your patience while we've been testing!
- J.B.
CCLP1 Voting Proposals - AdrenalinDragon - 15-Mar-2013
Sounds good to me, J.B! I remember on CCLP3 voting feeling overwhelmed by how many levels I had to vote all in one go and missing a lot of them as a result. Leaving decent candidates in is the way to go.
CCLP1 Voting Proposals - random 8 - 15-Mar-2013
Concerning the release of the level packs, I was thinking to release them all at once so voters can go at their own pace, but randomize the order they appear on the webpage every time it's loaded. A slight problem with it, though, is unless someone is willing to leave the page open 24/7, finding packs one hasn't played yet becomes increasingly difficult as one goes through more of the levels. Perhaps use browser cookies somehow?
CCLP1 Voting Proposals - tensorpudding - 15-Mar-2013
I feel that this is all very well-conceived and I applaud the effort that the CCLP1 has put into deciding these things and testing the levels.
If the whitelisting/elimination has been ongoing (the use of tense in "we've been assigning" regarding the ratings), will we be seeing those lists soon?
CCLP1 Voting Proposals - jblewis - 15-Mar-2013
Yes, we're almost done.
CCLP1 Voting Proposals - tensorpudding - 15-Mar-2013
This is good news...I really feel antsy to do something useful for the CCLP1 process.
CCLP1 Voting Proposals - M11k4 - 21-Mar-2013
Thanks for keeping us informed on the plans.
1) I like this idea of narrowing down the levels to vote on. Even compared to the CCLP3 process, we now have a clearer goal in mind and you guys are able to get rid of more levels than was possible last time had it been done that way. The lists for 'marked for elimination' and 'white list' sound like very useful tools to ensure that some levels don't slip by on accident. Like I know in my set there are some poor levels that shouldn't be in the voting really, but if I update them they might have a better chance, and you guys can't obviously evaluate if that will happen or not.
2) I'm starting to like the idea of voting level packs more than I did at first. I still don't like the idea that they are all released at once, nor that they are released individually. Could they be released in waves? Like a third of them at a time? What if I have time to play and vote only towards the start of the process? Or if you release most of them at once, randomizing the order they appear in sounds like a good idea (and you don't have to name them pack1, pack2.. but could go with some kind of code words that don't have an inherent ordering built in, like pack_orange, pack_monkey and so on). (And how about using the file extension ccl?)
In about a month I think I could evaluate about 100 levels if real life does not present large obstacles, but even more if the levels are familiar from earlier. Which is why I'd prefer packs of fifty, but more than one of them available if I have the time to work on more, but also would value the flexibility for you guys of not releasing them all at once. And last time the list of levels that had the least reviews/votes was a very useful tool, so hopefully that idea will be utilized again.
3) Agreed that one rating is enough, and you guys can evaluate the difficulty and appropriateness.
4) This idea is essential to making this set work. I would have suggested about 200 levels, but 250 sounds even better! And if a unique concept that fits well, like a hole filler (as in a certain type of lesson level), you could go even further down the list. Hopefully we will still see some voting for certain spots in the set where needed, like CCLP3 had for the start and cypher levels, as these were a fun way to feel part of the process in addition to the main voting pool.
Keep up the good work! And a warm thank you in advance too!
-Miika
CCLP1 Voting Proposals - rockdet - 21-Mar-2013
As we mentioned, those are probably going to be the first "full level" of the game and the cypher level. We will consider further possibilities as well
CCLP1 Voting Proposals - jblewis - 21-Mar-2013
Great suggestions for the voting packs, Miika. Thanks for sharing! I love "orange" and "monkey" as your examples...though I wonder if maybe we could get away with using CC elements instead. (Unless we have trouble getting people to play the Blob Pack!)
CCLP1 Voting Proposals - starfishgurl1984 - 07-Apr-2013
Everything sounds great to me as well, and I really like Miika's idea of the random level pack names! Great job staff on doing such hard work for this set and its upcomming voting process, all of the ideas you have put forth seem well thought out and a great asset to creating a great beginner friendly CCLP! I am also fine with the voting administrator choices as well. And I do agree that the staff having final say over the voting results to ensure maximum variety in difficulty, level type, style, and size, etc., is a great idea, because we don't need a repeat of CCLP3's two "duplicate" levels again. Keep up the great work!
p.s. is there any criteria for being the final level of the set? Just curious if that needs voting too, unless the staff can find one that's appropriate for that position amongst all of the submissions (after voting), which I am totally fine with too.