09 March 2012

Two Years of the PL/SQL Challenge, and Looking Ahead

The second anniversary of the PL/SQL Challenge's launch date (8 April) approaches. Two years of daily quizzes, and so much more. Excellent time to reflect on how things have gone and how they can be improved in the future.

Three-plus years ago, I had this wild idea: that PL/SQL developers might like to take quizzes – even daily quizzes, both to learn more about PL/SQL and also to demonstrate to the world (well, mostly to other PL/SQL developers) how much of an expert they are in this database programming language.

Turns out I was right, since two years after starting the PL/SQL Challenge, thousands of developers have played these quizzes, submitting over 450,000 answers to hundreds of quizzes. Along the way, I learned many lessons about what it takes to write high-quality, error-free, interesting and non-trivial quizzes, and to provide a website where players could compete and be ranked. 

I offer below some thoughts on the PL/SQL Challenge and also alternative ideas for competitive play (the daily PL/SQL quiz) and international rankings. I would very much like to hear your own ideas, as well as any feedback you have on mine.

First, What's Coming
We will soon be releasing version 2.2 of the PL/SQL Challenge, with these major new features:

Activity Points

Up through 2.1, the only reflection of your activity on the PL/SQL Challenge was your ranking in the daily PL/SQL quiz, or some other quiz. Of course, the vast majority of players will never find themselves in the top rankings for these quizzes, but they nevertheless spend lots of time and effort at the PL/SQL Challenge improving their skills. In 2.2, we now award points to players for all of their activities on the site (taking quizzes, submitting quizzes, reviewing results, engaging in discussions about quizzes, etc.).

Practice Quizzes

Prior to 2.2, you could take a quiz when it is initially offered, and then review results and answers through the Library afterwards. You could not, however, take past quizzes if you'd missed them, no could you re-take a quiz. Now, you can set up practices based on past quizzes, specific features about which you want to learn more, or your favorite authors. You can also use the unique Autotune feature: the PL/SQL Challenge will automatically create practices for you, based on pre-defined criteria.

Quizbook: Quiz as PDF

One of the most long-sought enhancement requests on the PL/SQL Challenge, the Quizbook feature makes it easy for you to generate a PDF document of one or more quizzes. These can be formatted as an offline test (no answer information included), a player scorecard (showcasing your performance) or a knowledge document, containing all answers and resources. You can save and reuse your Quizbooks, as well as sharing them with other players.

On-Site Discussions

Rather than discuss interesting aspects of quizzes on the PLSQL-Challenge.Blogspot.com blog, you can now start or answer a thread directly on the Quiz Details page. You can also ask for help (it must be related to the topic of the quiz) or even post an objection to the quiz. In other words, if you think there is something wrong with the quiz, you can register your concern on both the post-quiz feedback page or on the Quiz Details page. You can also easily see if another player has already submitted a request for a correction.
Lessons Learned
Here are some of those lessons:
  • You can't write a good quiz in five minutes. Maybe 30 minutes, when you include the time it takes to write verification code, test that code, etc. More like an hour. So, as you might well imagine, I have spent many hours over the past several years writing quizzes (and lots of other developers have, too, since over 200 quizzes have been submitted by others!).
  • Many developers are ready to take the time and make the effort to play each day, but many more probably don't have the time or commitment or discipline to do any more than play occasionally, and use the website to look over quizzes and learn lessons from them.
  • Moving from passive learning (reading documentation or books) to active learning through competition helps identify all sorts of nuances, bugs and areas for improvement in the technology covered by the quizzes.
  • It is extremely difficult to come up with a way to offer quizzes that makes cheating impossible, but also extremely difficult to prove that anyone actually is cheating.
  • It's hard work to support and maintain a website that is available 24x7. Oh yeah.
  • A formal education in music theory is an excellent foundation for people who would like to become programmers. I have been very impressed and proud of how my son, Eli, has been able to come up to speed on SQL, PL/SQL, APEX, JavaScript, CSS and more.
  • There's so much about the Oracle technology stack and running websites that I do not know. The PL/SQL Challenge could never have survived as long as it has without the help of my friends at Apex Evangelists.
  • I sometimes wonder how long I can keep up the flow of five new quizzes each week that are of sufficient quality to be used as a way to rank PL/SQL developers.
Certainly, I could just keep the PL/SQL Challenge going the way it has, but I think that it is worth, after two years of publishing daily quizzes (and more), to contemplate alternatives to the daily quiz and, more broadly, public ranking through competition.

Variations on the Daily Quiz
I offer below ideas for variations on the daily PL/SQL quiz.The motivation behind these variations is that I want (a) lots more people to play quizzes generally and (b) lots more people to compete.
  • Three new quizzes a week: rather than offer a new quiz each weekday, cut back on the volume by publishing a new quiz on, say, Monday, Wednesday and Friday. I will have to provide 40% fewer quizzes and players will not have to commit as much of their time to compete. The big question for players is: how important do you think it is to have a new quiz each day?
  • Keep doing a daily quiz, but offer a mix each week of three new quizzes and two "used" quizzes – played sometime in the past (at least six months, probably, so that they will not be fresh in anyone's mind).
  • Rather than post a new quiz each day, publish all the new quizzes (whether it be three or five) on, say, Sunday. They must be completed by the following Saturday, but you can choose when to take the quizzes. You might have a few hours available on Tuesday afternoon, so you do all the quizzes in one sitting.
Making Rankings Count
Here are some ideas to minimize the chance of cheating and improve the credibility of rankings:
  • Rather than offer rankings on all quizzes taken, define only certain quizzes as competitive. Each player makes a decision to compete and to do so, you must register with a credit card, pay a small fee. If a person wants to compete under two accounts, they will have to go to much greater lengths to hide their identity (and they'll also have to pay more money). Sure it is still possible, but much less likely, I believe.
  • For all other quizzes, you take them, you accumulate points for the effort, but you are not ranked. One advantage of making this distinction is that we can publish many more quizzes that will help you deepen your expertise (and highlight the knowledge of others), since not all quizzes will have to meet the criteria needed for competition. For example, "quick" true/false quizzes don't work well for the daily quiz, but can be very handy in reinforcing knowledge or exposing gaps.
  • Your real name (that provided on your credit card, which must match your registration information) will be displayed in rankings. You can't "hide" behind a player name.
  • Zero tolerance for aberrant scoring: everyone takes qualifier quizzes to verify their performance during the previous month (and, likely, for the quarterly championship as well). Highly aberrant patterns (such as extremely fast and extremely accurate) that cannot be reproduced lead to lifetime expulsion from the competition.'
Your Thoughts?
  • What have you learned from the PL/SQL Challenge? 
  • What do you like best about it? 
  • How do you think it can be improved? 
  • How can we get thousands of Oracle technologists to play? 
  • Why don't your co-workers play now? 
  • What can be done to get your manager to see the value of the PL/SQL Challenge, and actively encourage all members of the team to participate?
Many thanks in advance,
Steven Feuerstein

    13 comments:

    1. Hello Steven,

      Why don't your co-workers play now?
      I just asked my co-working sitting vis-à-vis to me. She said she simply has no time to play daily, if she wants to read and think about the questions thoroughly.
      Another reason is the language. Though most have learned English at school, many have not practiced it since then. I don't think it is possible to make the challenge available in different languages, not only because you would need a translator that really understands the question, but also because even in the english version there have been complaints about the meaning of some phrases.

      Rather than post a new quiz each day, publish all the new quizzes (whether it be three or five) on, say, Sunday
      I like to play the quiz in the morning before I start working and I like that I get a feedback the next day. I still could do so if the questions are published weekly and it would add the flexibility that I can choose to miss a day.

      Each player makes a decision to compete and to do so, you must register with a credit card, pay a small fee
      Keep in mind, that in some parts of the world credit cards are not that common as in the USA, e.g. many people in Europe only have a debit card (EC-card). This would be another barrier to take part.

      What do you like best about it?
      The best is, that there are quizzes that tell you what you don't know or, even better, that your knowing is wrong.

      Regards
      Marcus

      ReplyDelete
    2. Making Rankings Count
      Each player makes a decision to compete and to do so, you must register with a credit card, pay a small fee

      Does this mean that I have to pay for every competition? Why should I do so?

      Your real name ... will be displayed in rankings. You can't "hide" behind a player name
      There are valid reason why someone does not want his name to be displayed. Maybe someone does not want his/her employee to see how good or bad the results are.

      For all other quizzes, you take them, you accumulate points for the effort
      What are these points good for, apart from somthing like the lifetime ranking we already have?

      Regards
      Marcus

      ReplyDelete
    3. Hi Steven,

      Here are some thoughts about how to get more Oracle technologists to play:

      1. One more advantage to join PLSQL challenge: It helps prepare OCP exams.

      When I write the quizzes, some of them remind me of the questions from the OCP exams I took. Besides the advantages the website already promotes, it might add more value to mention it will also help to prepare Oracle certificate exams.

      2. Add anther "Search the library" on the main page.

      When asking my co-workers to participate in PLSQL challenge, they either don't like taking quizzes or are simply too lazy. Some don't like to be timed to answer questions; others just don't like "test/quiz" in general because those words trigger their worst memories.

      By observing the way our team (including me) work our projects, I notice that we will absolutely go out of our way to find answers if we have issues or need ideas. How about meeting the lazy people (including me) half way by adding another search box on the main page? So the non-quiz takers can also benefit from the two-year worth of practical and in-depth materials, and even being "converted" to take part in PLSQL challenge.

      PLSQL challenge currently has a Search in "Library" section, but it's a bit hidden from the general web users. By adding another one on the main page, it might help to draw more potential Oracle technologists to be part of PLSQL challenge.

      Regards,
      Teresa

      ReplyDelete
    4. Steven,

      Some general thoughts, keeping in mind that I greatly appreciate the time and effort you have committed to the PL/SQL Challenge, knowing that it is easier to find fault with things rather than create something new that everyone will like.

      1. Activity Points: What's the point? The original goal with the scoring of quizzes was to rank individuals based on their knowledge (% correct) and timeliness (time to answer) on the quizzes. Will the activity points be separate from the current points used to rank individuals? If you start including information such as discussions, creating quizzes, etc, this seems to confuse the original purpose of the rankings and becomes more similar to the Oracle ACE type rankings on the Oracle Discussion Boards. The vast majority of individuals will not find themselves in the top rankings, regardless of the system used (there can only be 25 indivduals in the top 25).

      2. Including 2 old quizzes per week: I think you underestimate the ability of many of the individuals taking the quizzes on a regular basis to recall past quizzes, especially if they were tricky or if an answer was missed (I know I spend more time analyzing the quizzes where I missed answers than the quizzes were I got all answers correct). Would be an interesting study on whether the performance on old quizzes was higher the second time around.

      3. Requiring a credit card: This is a no go for me. Although having played over 90% of the daily quizzes, I will stop participating the day this becomes a requirement for several reasons. A. I'm not buying anything, so why would I agree to give you my credit card information? B. How secure would the credit card info be? I have no way of knowing. C. Were I to qualify for a quarterly playoff, I know my chances for a top 3 placing are about 0%. With a 0% chance of monetary rewards, outside of potential weekly and monthly drawings, why should I keep my credit card on file with you? If you are really concerned about cheating, why not remove all monetary prizes? Even the threat of quarterly verification has deterred at least 2 individuals from continuing the competition. With no monetary goal, would individuals still cheat for higher rankings, whatever form the new rankings take? If the purpose is to further one's PL/SQL knowledge, is the monetary carrot even required?

      4. Zero tolerance for aberrant scoring: I know this is a tough issue to tackle with no good solutions. Would removing all monetary prizes help? Even though I am playing honestly, it has the appearance that everyone is now in the boat of guilty until proven innocent via the monthly and/or quarterly qualifier quizzes. Is this really the way to build a strong foundation on which to grow this PL/SQL community?

      5. A formal education in music theory is an excellent foundation for people who would like to become programmers: Every once in a while, throw a music theory question into the daily mix, to test whether the opposite is true as well.

      6. Post this blog entry to the PL/SQL Challenge page "Messages" so that it is seen by the group (only 2 responses after a four days indicates not many people have seen the post).

      ReplyDelete
    5. I will respond to your feedback soon, but thanks, Anonymous, for reminding me to post a message about this. Done! Let's see if we get some comments and then I will chime back in with my "two cents."

      ReplyDelete
    6. Hi Steven,
      here are my thoughts on some topics.

      Mixing 3 new and 2 old quizzes in a week.
      I got a feeling that the reason for this suggestion is primarily to relieve you of some work because you just don't have time to produce enough good quizzes.
      If that will help you, let's do it. There's no sense in making you work like a maniac just to keep the PLSQL Challenge going for the rest of us. I very much admire the extraordinary effort you and your team have put into this project, and I think noone can blame you for wanting to slow down a bit.
      I think having some old quizzes in the mix would be ok also for checking whether we have learned something. Don't show us if this is the new or an old quiz before we take it, but it would be great if you could give us the comparison of our "old" and "new" score for the same quiz.
      The ratio of 3/2 (new/old) seems fine to me. If at some point in time other authors also become as productive in writing quizzes as you are, and you suddenly have a lot of quizzes waiting in queue you can change the ratio to 4/1 or even to 5/0 again.

      Introducing the possibility of taking all 5 weekly quizzes at once.
      It would be appropriate to call this "BULK COLLECT quiz taking", don't you think? :-)
      I think this is a great idea. Sometimes in a stressful work week it can be a burden to take the quiz each day, so it would be great to have an opportunity to take them all at once when we have time, or take e.g. 1 on Monday, 2 on Wednesday and 2 on Thursday. There would be a lot less missed quizzes that way.

      Using a credit card as a verification of identity.
      I don't have any issues about security of my credit card data with you. I think you would take care of them as secure as any other institution that I entrusted my credit card data with.
      But, I agree with some concerns that others have stated in their comments:
      Not everyone has a credit card.
      Not everyone is willing to pay anything (even a small symbolic fee). I personally wouldn't mind paying a fee, but I think a lot of players would be put off by that, and we would loose many good players to compete with.
      So, all in all, I would say "no" to the credit card suggestion.

      Having time-limited qualifying questions and comparing that performance to monthy or quarterly player's average seem like a good idea to identify cheaters. I would make those mandatory for the players that have qualified for quarterly playoff. Maybe even make them mandatory on a monthly basis for all the players that ranked in top 100 or 200 players. If you want to continue playing - take the test and prove that you are ok.
      Of course, if you expel someone that doesn't prevent them from registering a new account, but at least they would have to do so every month and because of that couldn't compete effectively.

      Gotta go back to work now. Those PLSQL lines won't write themselves :-)

      Regards,
      Dalibor

      ReplyDelete
    7. Great job, Steven. Please keep going.

      ReplyDelete
    8. Some quick thoughts:

      I like the idea of three quizzes instead of 5 per week. This might entice more people to play, knowing the they aren't committed to 5 per week. Couple that with the ability to play at any time during the week may be even better.

      For those having issues with credit card, would paypal be an option? A lot of podcasters seem to offer the ability to have micropayments, so perhaps a nominal fee to help verify people's identity, without having to publish their real name?

      ReplyDelete
    9. Wow, a long list of excellent questions :) :) :)
      It took me a while to elaborate my thoughts, and here they are,
      posted in several parts because of length limitations.

      Regarding the changes proposed:

      You are so right that it is a hard work to come up day by day with high level quizzes, though, 30 minutes or one hour (only!) for a quiz seems almost like a fantasy :)

      I am really working very hard on my own proposed quizzes, most of the times the idea of a quiz comes in a natural way, without seeking especially for it, but afterwards checking out all the
      different facets of the issue for being 100% precise and correct, elaborating good choices, explanations, a.s.o. is not a few work.

      I really think that much more players could have been involved in proposing quizzes for the competition, including some of those who usually do not rank in the highest positions,
      even if they might need sometimes more help from Steven and the reviewers team in finalizing their quizzes. I am very confident that we all can learn something from the experience of everyone.

      The idea of using past quizzes also can really be a good one, and, as Dalibor said in a previous post, it does not need to be definitely turned into a rigid 3+2 or 4+1 rule, it can be done variably as the need arises.
      I think this is preferable upon reducing the number of quizzes for each week.

      How important it really is for us to have a quiz every day ?
      The answers to this question will probably be extremely variable, and it is also not a few psychology involved ...
      Once you answer a quiz correctly, maybe you would be happy to rest for a day enjoying your accomplishment, but after one on which you were mistaken, you usually prefer to have another one as soon as possible for correcting yourself.

      With any number of quizzes, it will be a good idea to allow for an entire week to answer all the quizzes, but, if possible, to still allow you to receive your own result as soon as possible
      after taking the quiz, that is, even while the quiz is still open for the other players (similarly to the immediate feedback we receive by e-mail for the ToadWorld monthly quiz).

      I think that everybody will agree that the PL/SQL Challenge has matured a lot in these two years, many of today's quizzes are much more difficult that those in the first Playoffs.

      For encouraging more developers to play, it is probably important to maintain a reasonable balance between more difficult and more simple quizzes, to give everybody the feeling that he/she can be successful and that the competition is not reserved for turbo-fast and top-level gurus only.

      As about making some of the quizzes non-competitive: does this mean that only the topmost difficult quizzes will be considered competitive ? If so, then really only the super-gurus will be able to compete...
      I would not go for excluding completely the "short text" quizzes of the kind of "Which of the following
      statements are true regarding ...". As we can often see, such short quizzes are not always as easy as they seem, the percent of answering them correctly is not so much higher than for a longer quiz, I see sometimes people answering correctly a long code quiz and erring on a short TRUE/FALSE one, so, probably the knowledge testified by answering correctly such a short quiz does however have its value.

      The idea of awarding points for all the quiz-related activities is welcome, since it would be a nice recognition for one's entire work and commitment, though, I don't understand completely how those points will be used, I suppose they are not meant to be cumulated in any way with those of the competition, am I right ?

      The possibility of (re)taking quizzes for practicing, as well as having the quizzes available in PDF format are really valuable features, most welcome.

      ( to be continued )

      ReplyDelete
    10. ( part 2 - continuation )

      Regarding the identity issue, I have never understood exactly why a player who is playing correctly
      ( and I am sure that the overwhelming majority do so ) would want to hide behind a nickname, instead of using his real name.
      I remember a statement on the entry page of the PL/SQL Challenge at its first version, welcoming the players by saying "You will be famous !".
      I am pretty sure that inside his inner conscience, everyone enjoys to be appreciated for what he does well, everyone desires "his small fame" to reward his hard work, it is really frustrating to always do things well without anybody ever knowing about it, and, unfortunately, it happens too often in real life, so why should things be the same in PL/SQL Challenge as well ?

      Together with this, I respect anyone's choice to remain anonymous if he wants, in what concerns the publication of his name on the site, but, however, I think that not publishing one's real name does not imply necessarily that the real name should remain unknown to the site back-end database as well.
      If knowing the real name of somebody is a must for lowering the possibility of cheating, I am for it.


      Now to my "less technical" thoughts ..

      What have you learned from the PL/SQL Challenge?
      That you should always pay the same attention to the apparently small details as to those that seem the most significant, that you should never pretend you know everything about a topic, even not about a beginner's one.

      What do you like best about it?
      Each time you revisit a topic, either by reading a book or by taking a quiz, there is always something additional to learn.
      And, if you make a mistake in a quiz, you can always hope to perform better in the next one.

      ( to be continued )

      ReplyDelete
    11. ( part 3 - continued )

      Why don't your co-workers play now?
      and
      What can be done to get your manager to see the value of the PL/SQL Challenge, and actively encourage
      all members of the team to participate?

      These were the most difficult questions to answer ...
      I really don't want to blame anyone, especially not those who are really working diligently day by day to do their work and be always ready in time ... but ... but ... but ...
      For playing, and times X 1000 for being a manager who ENCOURAGES HIS SUBORDINATES to play, you have to be crafted from a completely different material than, let's face it, most people that surround us are... You have TO LOVE doing things well and improve every day, you have to acknowledge that every day can bring along a new opportunity to learn something,
      to discover and enjoy the ability to deepen and master a certain piece of knowledge and, yes, why not ?, TO GAIN RECOGNITION for having done so, even without an immediate financial profit for your organization or for yourself...
      I don't speak of course in everybody's name, but most managers are still very far from knowing and appreciating the value of learning as a central point of one's activity, because they usually don't learn by themselves, there is a common tacit acceptance of the fact that managers have only management responsibilities, and therefore are exempted from having also knowledge ...
      For some reason, knowledge is supposed to be "built in" and "born", and, unfortunately, many people without a high knowledge level succeed in being promoted to managerial positions, and this has an impact if not directly on the accomplishment of the daily tasks (by others), but for sure on the light in which knowledge, quality, dedication and other such abstract notions are looked at or, better say, overlooked.

      In addition, but not less important, for playing you should be willing to work hard just for the sake of success by itself, just the same as one cannot be a true artist if he does not love art for the sake of art, and instead is doing it just for money.

      You mentioned the musical education as a good foundation for becoming a successful software developer. Yes, there is much truth in this, as one who has done both, I can also witness
      it, though I don't state that I am definitely a successful developer ...
      I think that probably the same intrinsic qualities and skills are required for both these activities and a few others, mathematics and music, foreign languages and programming languages have always gone well together, BUT, along with any native skills, having Steven Feuerstein as a mentor of one's professional development allows for different results to be expected than from somebody who should always do everything alone, cope with any difficulties always alone.

      In this context, I cannot but express again my gratitude for the opportunity offered to us through the PL/SQL Challenge to share a part of your mentorship, to find here the support and advice that a worker in a low hierarchical position is completely entitled to expect and find at his managers and HE DOES NOT, as well as the recognition that he is equally entitled to receive for his daily endeavors, dedication, results and successful work from the same managers and HE DOES NOT.

      This is what I love PL/SQL Challenge for, and, in spite of the daily pressure you put on yourself if you really want to be always successful, it has become a sine-qua-non of my professional life, at this point hard to imagine without it :) :)

      Thanks again and again for the excellent work, for being with us and, for some of us, offering the almost only chance for accomplishment and recognition.

      Best Regards,
      Iudith

      ReplyDelete
    12. Coming to this conversation late & have only started taking quizzes, so discount this appropriately, but I suggest the Will Shortz (New York Times crossword puzzle) model:

      1) Be (primarily) the editor of the quiz. I suspect lots of folks will enjoy the challenge of creating great quizzes, and ranking their difficulty. You then can choose, offer constructive revisions, and still write your own (without having to write as many).

      2) A suggestion for getting more to play: stagger the difficulty, Mondays being the easiest up through Fridays being the hardest. Participants can play whichever ones they want (and possibly see their scores broken out by day/difficulty). I think this will encourage new participants to start "playing the Mondays" and then as they gain confidence they'll start checking out the Tuesdays onward.
      Some quiz composers will be able to write easy and medium difficulty questions at first, and then possibly graduate to harder questions as they gain experience.

      Again, I'm not familiar enough with the Challenge to know the satisfying aspects of the Challenge that these suggestions will upset, but just throwing into the suggestion pool.

      ReplyDelete
    13. 1. "Be (primarily) the editor of the quiz" - that's a lovely idea, but at last right now I don't have nearly enough submissions from outside my brain to support the daily quiz. Any ideas on how to encourage more submissions?

      2. Progressive difficulty - I am not sure how I feel about this. It could just lead to many fewer players at the end of the week. :-( Generally, I try to do beginner or intermediate with limited code on Mondays and Fridays, and push the more advanced quizzes into the middle of the week. Rationale: slow start to week on Monday and winding down to weekend on Friday (for most of us).

      ReplyDelete