15-Oct-2015, 1:27 PM
@Jeffrey (and everyone else): I think there are a couple of solutions we could look at to address the limitations of categorization. One is more segmented categorization for broad groups. During CCLP1 assembly, for instance, we noticed that there were a bunch of “variety” levels that performed well, and many of them were itemswappers. So in order to make sure that non-itemswappers were represented fairly, we split up “variety” to include a separate itemswapper list with its own levels. The same could be done here depending on what level categories will be overloaded during the submission process - add a secondary dimension. So for instance, if the CCLP4 staff found themselves with tons of mazes, they may choose to divide that up into “open” (Crumbling Point, perhaps, to use Josh as an example) and “closed” mazes (Batteries in a Sock). The second solution is to categorize the categories themselves. We also did this for CCLP1, with categories dedicated to gameplay type, dominant elements, design style, size, and difficulty. Of course, how those lists were filled out was informed by the voting results, and as a result, we had a number of levels that appeared on multiple lists. How we ultimately used the lists was to fill out what was underrepresented from the raw results. Perhaps the CCLP4 staff can ensure that each level appears in at least one “bucket” in each broad category, even if this means creating groups like “no dominant tile” in the tile category. Or set aside certain categories for voting and reserve others for more internal use, like the broad "difficulty" ones.
I think if this categorization approach was used, the CCLP4 staff can’t lean too far in either direction - always using hard numbers to make decisions or always using subjective opinion. There has to be a balance, a constant awareness of what kinds of levels are necessary to round out the set. My concern with leaving the categorization of levels up to the designers is that it shifts responsibility that would be more optimally handled by a few to a large group, and it has the potential to undermine the purpose of working within a submission cap. I’m a little worried that if designers were actively forced to limit the number of levels they submit per “bucket,” particularly when multiple categories are involved, there would be no flexibility within that portion of the process to account for underrepresented or overrepresented buckets as submissions roll in, and designers could attempt to account for those discrepancies on their own before the submission period ends.
Maybe I’m alone in this, but I think if the CCLP4 staff were to institute a submission cap, working within that cap has to involve some kind of realism about the production process. We have to keep in mind that out of 2,000 (so far) submissions, only 149 levels are going to be in the final product. If we were each to envision what the final set would look like, with variety on several counts, just how many levels would a given designer have? Would any one person have 75-100, or even 50? I didn’t mean to imply that selecting one’s submissions had to be a siloed effort. If a designer is unsure about what to submit, of course feedback is an option. But if a CCLP is meant to be a collection of the community’s best work, the point of that effort would be to narrow down one’s own compositions to that which represents him or her the best. Which levels stand out enough to hold a spot in a CCLP? Which levels are the most fun to play? One could argue that voting helps answers these questions, but in reality, we’re dealing with variables and qualities that are difficult to quantify. As mentioned earlier, the “agreeability” factor in voting undermines its effectiveness with respect to addressing these issues. And I don't believe trying to institute measures at that stage to govern variety rather than leaving it in designers' hands will be very effective.
Once the submission period closes, then Ithink the staff can start having a discussion about what types of levels are overrepresented and underrepresented, and how the voting takes place and how all the levels are distributed into category buckets can be informed by that. The staff could use their own judgment about certain levels or categories here, but at least we have some boundaries and delegations of responsibility in place with an understanding of the scope of the project.
I think if this categorization approach was used, the CCLP4 staff can’t lean too far in either direction - always using hard numbers to make decisions or always using subjective opinion. There has to be a balance, a constant awareness of what kinds of levels are necessary to round out the set. My concern with leaving the categorization of levels up to the designers is that it shifts responsibility that would be more optimally handled by a few to a large group, and it has the potential to undermine the purpose of working within a submission cap. I’m a little worried that if designers were actively forced to limit the number of levels they submit per “bucket,” particularly when multiple categories are involved, there would be no flexibility within that portion of the process to account for underrepresented or overrepresented buckets as submissions roll in, and designers could attempt to account for those discrepancies on their own before the submission period ends.
Maybe I’m alone in this, but I think if the CCLP4 staff were to institute a submission cap, working within that cap has to involve some kind of realism about the production process. We have to keep in mind that out of 2,000 (so far) submissions, only 149 levels are going to be in the final product. If we were each to envision what the final set would look like, with variety on several counts, just how many levels would a given designer have? Would any one person have 75-100, or even 50? I didn’t mean to imply that selecting one’s submissions had to be a siloed effort. If a designer is unsure about what to submit, of course feedback is an option. But if a CCLP is meant to be a collection of the community’s best work, the point of that effort would be to narrow down one’s own compositions to that which represents him or her the best. Which levels stand out enough to hold a spot in a CCLP? Which levels are the most fun to play? One could argue that voting helps answers these questions, but in reality, we’re dealing with variables and qualities that are difficult to quantify. As mentioned earlier, the “agreeability” factor in voting undermines its effectiveness with respect to addressing these issues. And I don't believe trying to institute measures at that stage to govern variety rather than leaving it in designers' hands will be very effective.
Once the submission period closes, then Ithink the staff can start having a discussion about what types of levels are overrepresented and underrepresented, and how the voting takes place and how all the levels are distributed into category buckets can be informed by that. The staff could use their own judgment about certain levels or categories here, but at least we have some boundaries and delegations of responsibility in place with an understanding of the scope of the project.