TL;DR: The INCOSE SE Handbook Study Group successfully completed a six-week program focused on version 5 of the handbook. Regular quizzes provided valuable feedback, and performance tracking showed stable results. Attendance remained high, closing at 93%. A closing survey revealed that 100% of participants found the study group valuable, with 71% recommending it to others. Quizzes were the most valued element, while the Discord server was the least suggesting room for improvement in utilizing the Discord server. Overall, the study group was a success, with positive feedback and high engagement, providing a strong foundation for future iterations.

I recently concluded facilitating the accelerated study group focusing on version 5 of the INCOSE SE Handbook over the space of 6 weeks. I’m a strong believer in continuous improvement and the two pillars that support that objective are data collection and analysis.

I can now share my insights that will inform future study group iterations (next one starts this March 24th).

Quiz Analysis

A key factor in making the study group a success is the deployment of regular quizzes that give that timely feedback to individuals on how they are doing.

As you can see in the first table I am able to track individual quiz performance as well as running averages which I email to each person after each quiz.

Not only do I share individual quiz performance but also the cohort average which you can see I track in Mean Quiz Score graph. As you can see the rolling average was fairly stable ending up at 74.6% with a peak range of 6%. I also track quiz performance at the team level to determine if there is any significant deviation to the mean suggesting a breakdown in team communication and necessitating stepping in to provide additional support. In the below case I felt that there weren’t any sudden deviations and therefore I didn’t step in.

Finally, I track the efficacy of the quiz with Quiz Score Probability Density and Question Score Probability Density graphs and in both graphs I’m looking for deviation to a desired bell shaped profiles. The Quiz Score Probability Density graph is fairly close to what I would like to see (to read it: little more than 20% of all quiz results had a score of 80%). However the Question Score Probability Density graph tells me that I need to have a second look at 9% of questions that only 40% or less people answered correctly as well I need to increase the difficulty on 26% of questions that more than 90% individuals answered correctly.

Attendance

Predictably the attendance was high at the start and did attrite towards the end but overall I’m remaining positive that the study group sessions had enough ‘pull’ to close out the attendance rate at a respectable 93%.

Closing Out Survey

After the study group came to a close I asked the team to fill out a closing out survey. Here are the results (n=7):

100% respondents responded Yes or Absolutely Yes to the question whether the Study Group provided value to them.

71% respondents responded Yes or Absolutely Yes to the question whether they would recommend the Study Group to others. 29% respondents were Not Sure.

As to which elements of the study group gave the most value the respondents were unanimous that the Quizzes gave the most value followed by the (positive) peer pressure from the rest of the cohort to study the handbook. As to the elements that gave the least value, the Discord server came out on top which is somewhat understandable considering I introduced it half way through the study group though I have high hopes that the future study group cohorts can leverage the quick asynchronous nature of Discord to help them with their study.

I recognize the sample size of the survey wasn’t as high as I would have liked, probably because I didn’t allocate time within the last session and instead relied on follow up emails to increase survey participation. However, the closing survey reinforced my belief that quizzes play an important feedback to the cohort and that I am not fully leveraging the Discord server.

Conclusion

Overall, the study group was a success, evidenced by high attendance rates and positive feedback. The insights gained from this program will be instrumental in refining future study groups, ensuring they continue to provide valuable learning experiences for participants. The commitment to continuous improvement and participant engagement has set a strong foundation for future study groups.