Hi everyone,
While we're still trying to finalize the details regarding our voting administrator, I'd like to go ahead and present everyone else in our community with a proposal for voting and where CCLP1 will go in the days to come. I've tried to consider the feedback I've received from my previous posts about CCLP1 from months past. That being said, we'd like to present to you the following for consideration and further discussion.
--- 1. PRE-VOTING ELIMINATION ---
CCLP3 had (if memory serves correctly) about 2,030 levels in its initial round of voting, and all were available for the public to play at once. Several hundred of these levels were eliminated after about one month of voting. For CCLP1, one of the key goals we've tried to establish on the staff is to encourage voters to distinguish good levels from great levels. "Great" doesn't necessarily mean ultra-difficult or ridiculously clever. Sometimes, it's all about the little things, too: design quality, the packaging of the familiar in an innovative way, and consideration of new players. And ultimately, we're trying to seek out the levels that are the best fit for a set that's meant to serve as a CC1 replacement for beginners.
Toward that end, we'd like to conduct an elimination prior to the beginning of voting. This is a sensitive issue but one we felt was necessary for a number of reasons:
- Narrowing it down to good levels. During CCLP3 voting, some voters felt like many top-notch levels were shafted. Obviously, this will always happen when there are a lot of good candidates, but while there can't ever be a guarantee that every level in the voting process will receive adequate consideration by every voter, it's much more likely to happen if the field is narrowed down to good levels to begin with - without non-CCLP-worthy material to worry about - so that the great levels can shine a bit more easily.
- Removing a heavy burden from level designers. One of the most overwhelming feelings a designer could ever feel during CCLP3 voting came upon visiting a levelset info page on Mike's site, seeing a bunch of "unequal" levels throughout a set, and feeling the need to correct every one before voting, which was especially frustrating if the designer didn't even build levels with Lynx compatibility or appropriate CCLP1 difficulty in mind to begin with. With a pre-voting elimination, we can focus on encouraging level designers to correct the levels that could actually stand a chance in the voting process.
- Removing a heavy burden from voters. One of the issues that the CCLP3 staff noticed during its voting process was that some community members started playing some levels but then stopped. While we can't completely prevent that here, I think it's much less likely to happen if the number of levels in voting is much less overwhelming to begin with. More ideas on how we can further accomplish that in a bit, though.
- Shifting the focus from levelsets to levels. One of the initiatives we're hoping to try to push for CCLP1 voting is to conduct the voting across level packs of 50 or 100 levels each, randomly picked from the narrowed submission pool. I'll explain more about how we can make this work in Section 2, but one of the reasons why we felt like this was a good idea was because of "levelset bias." In CCLP3 voting, it was very easy for voters to assign ratings to individual levels based solely on who designed the levels or in what set they appeared, primarily because voting was categorized by set. This especially was an issue with the difficulty rating, but we'd like to avoid having it affect the "fun" rating as much as possible, especially for CCLP1. While randomly ordered levels can't completely remove certain biases (for example, a recency bias in which one's opinion of the previous level solved in the pack affects his or her opinion of the current level being played), they can at least encourage the evaluation of levels on each level's own merit, not by who designed the level or what else the designer made.
Please note that we would like the community to be involved as much as possible in the elimination and have the final say concerning what makes it into voting. Toward that end, we'd like to propose the following steps:
- The staff puts levels being considered for the cut on a "marked for elimination" list. The criteria for elimination are pretty simple: (a) excessive difficulty for CCLP1, ( B) excessive triviality (for instance, a level that involves just walking to the exit), © poor design quality that is not on par with the standard established in the original CC, (d) non-Lynx-compatibility that is practically beyond repair, (e) excessive length (especially if the level is extremely monotonous), and (f) any other criteria that defined levels eliminated in the first round of CCLP3 voting except for optimization-related criteria (in other words, we won't eliminate a level because it's not optimization-friendly or contains randomness). To translate our opinions into something more quantifiable, we've been assigning numerical ratings based on fun and beginner accessibility using a -2 to +2 scale for each. Please note: beginner accessibility is NOT difficulty - it's how reasonable a level would be for a beginner at an appropriate place in the set.
- Levels above a certain cutoff point would be placed on a "whitelist." I'm considering 0 and higher for fun, and 1 and higher for beginner accessibility. The staff will confer about the whitelist's content and double-check to ensure everything is in order and that a consensus is reached before proceeding any further. At this point, levels that need compatibility or bust fixes will also be added to the whitelist as well but marked so that they could be fixed. (Based on the increase in knowledge about the Lynx ruleset since CCLP3's release, as well as the large percentage of non-Lynx-compatible levels that fall into criterion (d) described above, I don't think this list of levels to be fixed will be huge.) In the end, I'm hoping the whitelist will contain about 800-900 levels.
- The whitelist and the "marked" list will both be presented to the community publicly. At this point, community members will have a set amount of time to (a) correct any levels in the whitelist that need fixes and ( B) nominate any "marked" levels they wish to see in voting for whitelisting. We may have to come up with some rules to ensure that some designers don't abuse this ability for their own levels, but we can discuss options for how to proceed among the rest of the community. That's what this thread is for. Once this period ends, the remainder of the levels on the "marked" list will be properly eliminated, and the whitelist as it stands will be randomly jumbled to form the level packs.
- Voting will begin.
--- 2. VOTING LEVEL PACKS ---
I've already mentioned above why we'd like to go with this option, but I'd like a few opinions about what everyone believes to be reasonable. How many levels do you think you can complete and reasonably evaluate in about a month's time? 50? 100? Even more? Please bear in mind that CCLP1 voting won't have levels whose difficulty matches some of the most devious triple-digit entries of CCLP3, so solving them shouldn't be terribly hard for most of us veterans.
One issue we've got to decide, though, is what to do about levels that need fixes. Should we cut off all level modifying at a specified date before voting starts, or allow level designers time until the pack in which the level in question is up for voting? I'm leaning toward the former, if only so we can be consistent across the board with respect to making changes to levels and avoid any confusion. Also, the latter option doesn't allow for flexibility in scheduling the pack releases. Speaking of which...
Another issue is how the packs should be released. Should it be done all at once or pack by pack throughout the voting process? There's pros and cons for either option, but personally, I'm more in favor of the latter. From the standpoint of making sure each level in voting receives as much consideration as possible, a small number of levels on which voters can concentrate at a given time will most likely be more rewarding in terms of getting more informed ratings than releasing everything at the same time. And although we would be running the risk of having Pack #1 receive more consideration than the rest by virtue of being first, the staff will take on additional marketing responsibility to promote interest in subsequent packs and create anticipation. Either way, any pack that has been released would remain released and available for voting until voting ends completely, which most likely won't happen until the staff feels like everyone's gotten a chance to play every pack and vote as much as voters see fit.
--- 3. RATINGS USED IN VOTING ---
This was an issue I brought forward several months ago. Back then, I proposed the idea of including a "beginner accessibility" rating (or "BA") instead of difficulty. Since one of our objectives was to create a set friendly to first-time players, we thought that evaluating the level based on how reasonable it was for that group of players at an appropriate place in the set - regardless of actual difficulty - would be more beneficial. But as the discussion continued, I started to wonder if BA might be more useful for the staff to use in deciding what gets cut in a pre-voting elimination, whereas the only rating voters would even have to worry about would be the "fun" rating, in which beginner accessibility and all other qualities could be considered. A couple of proposals:
First, we'd like to ditch the name "fun" and just give this rating the generic term "rating." After some discussion, we started to realize that we needed to encourage voters to evaluate each level as holistically as possible and not just focus on any one concept isolated from everything else: for instance, how unique the concept or presentation is, how great the design is, or how fun the level is to play. All of these factors are important, but we'd like to encourage voters to keep all of them in mind as they vote. I think renaming the rating, along with putting some guidelines on each voting page for making sure one's vote is as informed as possible, can help with that.
Second, we'd like to abandon the difficulty rating altogether. The CCLP1 staff would like to do as much as possible post-voting to ensure that a proper difficulty curve is established throughout the set, and toward that end, we'd like to rate the top 250 or so levels for difficulty ourselves and go from there (more on that in the next section). Not only will this be easier on us, it will be even easier for voters when there's only one rating to worry about!
--- 4. POST-VOTING ASSEMBLY ---
In CCLP2 and CCLP3, the levels inducted into the final set closely resembled what the top 149 levels on the voting results were. It was especially the case in CCLP2. In CCLP3, we had the wildcard round to further refine the results for the "bubble" of levels around the cutoff point, plus we removed some really difficult levels and replaced them with somewhat easier ones for variety's sake. These included Toggle Bust, Blue Blocks, The Haunted House, and Jaywalker. For CCLP1, the staff would like to have the final say in judging what ultimately gets inducted into the set while using the voting results as a guideline. A few reasons why we believe this would be necessary:
- Establishing a proper difficulty curve using the steps described above and ensuring that there aren't too many levels of any particular difficulty.
- Creating as much variety in the set as possible. This was one complaint levied toward CCLP3 that would have been somewhat worse if not for the staff's intervention with the inclusions mentioned above, and we'd like to continue that tradition here with more freedom and flexibility.
- Related to the above point: removing levels that are too similar. This will help ensure that we won't have pairs of levels in CCLP1 like Tool Box / Which One Next?, or Caves / Divide and Conquer, unless there's a very good reason to do so.
Let us know what you think. We'll be sure to listen to everyone's comments and suggestions before we take the next big step. Thanks for your patience while we've been testing!
- J.B.
While we're still trying to finalize the details regarding our voting administrator, I'd like to go ahead and present everyone else in our community with a proposal for voting and where CCLP1 will go in the days to come. I've tried to consider the feedback I've received from my previous posts about CCLP1 from months past. That being said, we'd like to present to you the following for consideration and further discussion.
--- 1. PRE-VOTING ELIMINATION ---
CCLP3 had (if memory serves correctly) about 2,030 levels in its initial round of voting, and all were available for the public to play at once. Several hundred of these levels were eliminated after about one month of voting. For CCLP1, one of the key goals we've tried to establish on the staff is to encourage voters to distinguish good levels from great levels. "Great" doesn't necessarily mean ultra-difficult or ridiculously clever. Sometimes, it's all about the little things, too: design quality, the packaging of the familiar in an innovative way, and consideration of new players. And ultimately, we're trying to seek out the levels that are the best fit for a set that's meant to serve as a CC1 replacement for beginners.
Toward that end, we'd like to conduct an elimination prior to the beginning of voting. This is a sensitive issue but one we felt was necessary for a number of reasons:
- Narrowing it down to good levels. During CCLP3 voting, some voters felt like many top-notch levels were shafted. Obviously, this will always happen when there are a lot of good candidates, but while there can't ever be a guarantee that every level in the voting process will receive adequate consideration by every voter, it's much more likely to happen if the field is narrowed down to good levels to begin with - without non-CCLP-worthy material to worry about - so that the great levels can shine a bit more easily.
- Removing a heavy burden from level designers. One of the most overwhelming feelings a designer could ever feel during CCLP3 voting came upon visiting a levelset info page on Mike's site, seeing a bunch of "unequal" levels throughout a set, and feeling the need to correct every one before voting, which was especially frustrating if the designer didn't even build levels with Lynx compatibility or appropriate CCLP1 difficulty in mind to begin with. With a pre-voting elimination, we can focus on encouraging level designers to correct the levels that could actually stand a chance in the voting process.
- Removing a heavy burden from voters. One of the issues that the CCLP3 staff noticed during its voting process was that some community members started playing some levels but then stopped. While we can't completely prevent that here, I think it's much less likely to happen if the number of levels in voting is much less overwhelming to begin with. More ideas on how we can further accomplish that in a bit, though.
- Shifting the focus from levelsets to levels. One of the initiatives we're hoping to try to push for CCLP1 voting is to conduct the voting across level packs of 50 or 100 levels each, randomly picked from the narrowed submission pool. I'll explain more about how we can make this work in Section 2, but one of the reasons why we felt like this was a good idea was because of "levelset bias." In CCLP3 voting, it was very easy for voters to assign ratings to individual levels based solely on who designed the levels or in what set they appeared, primarily because voting was categorized by set. This especially was an issue with the difficulty rating, but we'd like to avoid having it affect the "fun" rating as much as possible, especially for CCLP1. While randomly ordered levels can't completely remove certain biases (for example, a recency bias in which one's opinion of the previous level solved in the pack affects his or her opinion of the current level being played), they can at least encourage the evaluation of levels on each level's own merit, not by who designed the level or what else the designer made.
Please note that we would like the community to be involved as much as possible in the elimination and have the final say concerning what makes it into voting. Toward that end, we'd like to propose the following steps:
- The staff puts levels being considered for the cut on a "marked for elimination" list. The criteria for elimination are pretty simple: (a) excessive difficulty for CCLP1, ( B) excessive triviality (for instance, a level that involves just walking to the exit), © poor design quality that is not on par with the standard established in the original CC, (d) non-Lynx-compatibility that is practically beyond repair, (e) excessive length (especially if the level is extremely monotonous), and (f) any other criteria that defined levels eliminated in the first round of CCLP3 voting except for optimization-related criteria (in other words, we won't eliminate a level because it's not optimization-friendly or contains randomness). To translate our opinions into something more quantifiable, we've been assigning numerical ratings based on fun and beginner accessibility using a -2 to +2 scale for each. Please note: beginner accessibility is NOT difficulty - it's how reasonable a level would be for a beginner at an appropriate place in the set.
- Levels above a certain cutoff point would be placed on a "whitelist." I'm considering 0 and higher for fun, and 1 and higher for beginner accessibility. The staff will confer about the whitelist's content and double-check to ensure everything is in order and that a consensus is reached before proceeding any further. At this point, levels that need compatibility or bust fixes will also be added to the whitelist as well but marked so that they could be fixed. (Based on the increase in knowledge about the Lynx ruleset since CCLP3's release, as well as the large percentage of non-Lynx-compatible levels that fall into criterion (d) described above, I don't think this list of levels to be fixed will be huge.) In the end, I'm hoping the whitelist will contain about 800-900 levels.
- The whitelist and the "marked" list will both be presented to the community publicly. At this point, community members will have a set amount of time to (a) correct any levels in the whitelist that need fixes and ( B) nominate any "marked" levels they wish to see in voting for whitelisting. We may have to come up with some rules to ensure that some designers don't abuse this ability for their own levels, but we can discuss options for how to proceed among the rest of the community. That's what this thread is for. Once this period ends, the remainder of the levels on the "marked" list will be properly eliminated, and the whitelist as it stands will be randomly jumbled to form the level packs.
- Voting will begin.
--- 2. VOTING LEVEL PACKS ---
I've already mentioned above why we'd like to go with this option, but I'd like a few opinions about what everyone believes to be reasonable. How many levels do you think you can complete and reasonably evaluate in about a month's time? 50? 100? Even more? Please bear in mind that CCLP1 voting won't have levels whose difficulty matches some of the most devious triple-digit entries of CCLP3, so solving them shouldn't be terribly hard for most of us veterans.
One issue we've got to decide, though, is what to do about levels that need fixes. Should we cut off all level modifying at a specified date before voting starts, or allow level designers time until the pack in which the level in question is up for voting? I'm leaning toward the former, if only so we can be consistent across the board with respect to making changes to levels and avoid any confusion. Also, the latter option doesn't allow for flexibility in scheduling the pack releases. Speaking of which...
Another issue is how the packs should be released. Should it be done all at once or pack by pack throughout the voting process? There's pros and cons for either option, but personally, I'm more in favor of the latter. From the standpoint of making sure each level in voting receives as much consideration as possible, a small number of levels on which voters can concentrate at a given time will most likely be more rewarding in terms of getting more informed ratings than releasing everything at the same time. And although we would be running the risk of having Pack #1 receive more consideration than the rest by virtue of being first, the staff will take on additional marketing responsibility to promote interest in subsequent packs and create anticipation. Either way, any pack that has been released would remain released and available for voting until voting ends completely, which most likely won't happen until the staff feels like everyone's gotten a chance to play every pack and vote as much as voters see fit.
--- 3. RATINGS USED IN VOTING ---
This was an issue I brought forward several months ago. Back then, I proposed the idea of including a "beginner accessibility" rating (or "BA") instead of difficulty. Since one of our objectives was to create a set friendly to first-time players, we thought that evaluating the level based on how reasonable it was for that group of players at an appropriate place in the set - regardless of actual difficulty - would be more beneficial. But as the discussion continued, I started to wonder if BA might be more useful for the staff to use in deciding what gets cut in a pre-voting elimination, whereas the only rating voters would even have to worry about would be the "fun" rating, in which beginner accessibility and all other qualities could be considered. A couple of proposals:
First, we'd like to ditch the name "fun" and just give this rating the generic term "rating." After some discussion, we started to realize that we needed to encourage voters to evaluate each level as holistically as possible and not just focus on any one concept isolated from everything else: for instance, how unique the concept or presentation is, how great the design is, or how fun the level is to play. All of these factors are important, but we'd like to encourage voters to keep all of them in mind as they vote. I think renaming the rating, along with putting some guidelines on each voting page for making sure one's vote is as informed as possible, can help with that.
Second, we'd like to abandon the difficulty rating altogether. The CCLP1 staff would like to do as much as possible post-voting to ensure that a proper difficulty curve is established throughout the set, and toward that end, we'd like to rate the top 250 or so levels for difficulty ourselves and go from there (more on that in the next section). Not only will this be easier on us, it will be even easier for voters when there's only one rating to worry about!
--- 4. POST-VOTING ASSEMBLY ---
In CCLP2 and CCLP3, the levels inducted into the final set closely resembled what the top 149 levels on the voting results were. It was especially the case in CCLP2. In CCLP3, we had the wildcard round to further refine the results for the "bubble" of levels around the cutoff point, plus we removed some really difficult levels and replaced them with somewhat easier ones for variety's sake. These included Toggle Bust, Blue Blocks, The Haunted House, and Jaywalker. For CCLP1, the staff would like to have the final say in judging what ultimately gets inducted into the set while using the voting results as a guideline. A few reasons why we believe this would be necessary:
- Establishing a proper difficulty curve using the steps described above and ensuring that there aren't too many levels of any particular difficulty.
- Creating as much variety in the set as possible. This was one complaint levied toward CCLP3 that would have been somewhat worse if not for the staff's intervention with the inclusions mentioned above, and we'd like to continue that tradition here with more freedom and flexibility.
- Related to the above point: removing levels that are too similar. This will help ensure that we won't have pairs of levels in CCLP1 like Tool Box / Which One Next?, or Caves / Divide and Conquer, unless there's a very good reason to do so.
Let us know what you think. We'll be sure to listen to everyone's comments and suggestions before we take the next big step. Thanks for your patience while we've been testing!
- J.B.