Juan-Marc Scrimgeour

Jacques Ophoff
email: jacques.ophoff@uct.ac.za

Lessons Learned from an Organizational Information Security Awareness Campaign

Keywords:

Educating end-users to improve information security awareness plays an important part in securing organizational environments. While best practice standards provide a set of minimum information security awareness controls that should be implemented, little guidance is given on how to implement these controls to ensure the effectiveness of training. This research defined and evaluated a method for implementing an information security awareness campaign (ISAC) within an organization. The method is based on prior research and standards, while assisting the subject in improving their ISAC through the creation of artefacts and measurement techniques. A design science research approach was used with several research cycles to design the method. The method was implemented within an organization and evaluated based on the impact, effectiveness and results of each step, as well as the feedback from participants (two questionnaires were completed by 47 and 36 employees respectively). The research found both positive and negative results. Certain steps within the method proved time consuming and confusing to some participants. Although improvements can be made, the method was found to be adequate as it achieved the required objective within the organization and provided the organization with a risk-based method and visual representation to measure awareness on specific information security awareness topics. The results of the study not only provided value to the organization but provides a validated method for implementing an ISAC which could be applied in other contexts.

Introduction

Research has shown that educating end-users on information security awareness (ISA) plays an important part in securing your environment [START_REF] Daniel Ani | Human Capability Evaluation Approach for Cyber Security in Critical Industrial Infrastructure[END_REF][START_REF] Denning | Control-Alt-Hack: the design and evaluation of a card game for computer security awareness and education[END_REF][START_REF] Yildirim | The Importance of Information Security Awareness for the Success of Business Enterprises[END_REF]. ISA can be defined as "a learning process that sets the stage for training by changing individual and organizational attitudes to realize the importance of security and the adverse consequences of its failure" [4]. Best practice standards provide a set of minimum controls that should be implemented, however, little guidance is given on how to implement these controls to ensure the effectiveness of the training [START_REF] Denning | Control-Alt-Hack: the design and evaluation of a card game for computer security awareness and education[END_REF][START_REF] Aloul | The Need for Effective Information Security Awareness[END_REF][START_REF] Stewart | Death by a thousand facts: Criticising the technocratic approach to information security awareness[END_REF]. The field of ISA is popular; however, most studies focus on improving training or how to implement the training in a different way. There seems to be a lack of studies that focus on how to implement a campaign that meets the stringent requirements set by best practice standards. In addition, research on effectiveness measurement techniques is largely focused on questionnaires, and as a result, questionnaires are the primary means of measuring effectiveness. However, researchers in this field state that the questionnaire technique can be improved upon by supplementing it with hard measures. This research defined and evaluated a method for implementing an ISAC within an organization based on existing research and standards, while assisting the subject in improving their ISAC through the creation of artefacts and measurement techniques. The method used attempts to supplement the widely used questionnaire technique with risk management practices, such as those defined by the NIST [4], "The program and supporting processes to manage information security risk to organizational operations (including mission, functions, image, reputation), organizational assets, individuals, other organizations, and the Nation." This will address the research problem by implementing and evaluating a method for delivering an ISAC within an organization that is measurable and relies on the use of risk management metrics to determine its effectiveness. The current paper reflects on the lessons learned during this ISAC implementation process.

The study was conducted within a financial institution in South Africa. The training provided was created and administered by employees of the target organization (TO) and focused on one specific theme (Acceptable Usage Policy).

Background

Recently several studies have focused on identifying the best way to improve on an ISAC. These studies have varied in approach from simulations [START_REF] Aloul | The Need for Effective Information Security Awareness[END_REF][START_REF] Arachchilage | Security awareness of computer users: A phishing threat avoidance perspective[END_REF][START_REF] Young-Mclear | A White Hat Approach to Identifying Gaps Between Cybersecurity Education and Training: A Social Engineering Case Study[END_REF] to the development of educational material [START_REF] Denning | Control-Alt-Hack: the design and evaluation of a card game for computer security awareness and education[END_REF][START_REF] Pattinson | Managing information security awareness at an Australian bank: a comparative study[END_REF]. When examining factors which influence the effectiveness of ISA the following themes emerge: the end-user's knowledge and skills, the personality of the end-user, and the environment that surrounds the enduser.

There is little guidance of what knowledge should be passed on in ISA. While some standards (e.g. PCI-DSS) state key focus areas for ISA, it is not allencompassing and is normally focused on software developers. In addition, it is suggested that the end-user's personality can influence the effectiveness of ISAC. Studies show that end-users tend to interpret, experience, and perceive the importance of ISA differently based on personality [START_REF] Kajzer | An exploratory investigation of message-person congruence in information security awareness campaigns[END_REF][START_REF] Mccormac | Individual differences and Information Security Awareness[END_REF][START_REF] Pattinson | Adapting Cyber-Security Training to Your Employees[END_REF]. These studies, in how personalities can impact the effectiveness of ISA, have highlighted the importance of how one communicates the content and importance of ISA. Lastly, studies show that context is crucial in the effectiveness of ISA training [START_REF] Yildirim | The Importance of Information Security Awareness for the Success of Business Enterprises[END_REF][START_REF] Tsohou | Analyzing trajectories of information security awareness[END_REF][START_REF] Waly | Improving Organisational Information Security Management: The Impact of Training and Awareness[END_REF]. This aligns with standards and regulations which state that ISA training should be relevant to end-users' role and environment [START_REF] Chaplin | The standard of good practice for in-formation security 2016[END_REF][START_REF] Jordan | The Standard of Good Practice for Information Security 2018[END_REF]. In addition, it has been shown that demographic factors, such as age and educational level, can affect employee security policy awareness and compliance [START_REF] Chua | Impact of employees' demographic characteristics on the awareness and compliance of information security policy in organizations[END_REF]. Effective training in one environment does not guarantee success in another.

Measuring the Effectiveness of ISA Training

Two primary measurement techniques can be identified from prior research: technical measures, in the form of metrics (number of reported phishing mails; incidents due to change, data leakage events, audit findings); and questionnaires or surveys, that test the knowledge of end-users. Technical measures of effectiveness are seldomly reported in academic publications, in most part because obtaining the data required from businesses has proven difficult (if not impossible) as the data owners have security concerns around sharing the data [START_REF] Lebek | Employees' Information Security Awareness and Behavior: A Literature Review[END_REF][START_REF] Poepjes | The development and evaluation of an information security awareness capability model: linking ISO/IEC 27002 controls with awareness importance, capability and risk[END_REF]. Most studies that make use of such measures are case studies using simulated phishing campaigns [START_REF] Aloul | The Need for Effective Information Security Awareness[END_REF][START_REF] Young-Mclear | A White Hat Approach to Identifying Gaps Between Cybersecurity Education and Training: A Social Engineering Case Study[END_REF][START_REF] Pattinson | Managing information security awareness at an Australian bank: a comparative study[END_REF]. In addition, it has been seen that methods also include reporting on incidents, testing employee pre and post training, and collection feedback from stakeholders [START_REF] Alshaikh | An Exploratory Study of Current Information Security Training and Awareness Practices in Organizations[END_REF]. While these studies show that end-users are retaining what they have learnt and have changed their behavior, it only addressed one security concern, albeit an important one. On the other hand, questionnaires are by far the most popular means of measuring the effectiveness of ISACs, with more than three-quarters of studies performed over the past years focusing on the designing and using a variation of a questionnaire to test end-user knowledge of ISA topics. While many questionnaires are self-developed they should ideally be grounded in bestpractice. For example, a formal approach is taken by Poepjes [START_REF] Poepjes | The development and evaluation of an information security awareness capability model: linking ISO/IEC 27002 controls with awareness importance, capability and risk[END_REF], which bases questions and focus areas on the ISO 27001/2 standard.

ISA Capability Model

The Information Security Awareness Capability Model (ISACM) [START_REF] Poepjes | The development and evaluation of an information security awareness capability model: linking ISO/IEC 27002 controls with awareness importance, capability and risk[END_REF] requires that several metrics be obtained to determine what awareness campaigns need to be delivered in the environment. These metrics are Awareness Importance (AI), Awareness Capability (AC) and Awareness Risk (AR). AI is derived from taking an information security best practice standard, extracting the controls and rating their level of importance regarding user awareness of the control within the environment. AC is derived from surveying a random sample of end-users on their understanding of the various controls identified as important in the AI phase. AC can therefore be defined as the end-users' understanding and knowledge around specific awareness controls. AR is then calculated by applying the values obtained to an awareness risk matrix. This matrix is used to determine the AR rating. AR can therefore be defined as the risk identified when comparing the users' AC against the AI score for each awareness control. The results of the ISACM can be used to create a targeted and measurable ISAC.

Implementation Methodology

We created a practical method to implement an ISAC, derived from the ISACM. The method adapted the model to the environment of a target organization (TO). Several instruments were developed in conjunction with TO personnel:

 Control framework: the control framework is a list of information security best practice controls. The framework aligns with the information security framework (ISF) that the TO uses.  Control awareness survey: the survey contains questions to determine the AC value of controls that received an AI rating of moderate to extremely important. The survey should be updated upon any change in the control framework.  Awareness risk matrix: this matrix is used to determine the AR rating and was adapted from the TO risk framework.

Fig

Awareness Importance (AI)

The control framework is a list of information security best practice controls which will be given to Senior Staff members of the IT security department that are stakeholders in the ISAC. Each control is then rated by the IT security team to determine the AI of each control, for each end-user group. The control framework is developed by incorporating the IS/cybersecurity framework of the TO. The NIST [4] defines a cybersecurity framework as "a common language for understanding, managing, and expressing cybersecurity risk both internally and externally. Includes activities to achieve specific cybersecurity outcomes, and references examples of guidance to achieve those outcomes".

Survey Userbase

The control awareness survey is dependent on the results of the control framework.

To not overburden users only the top ten controls, that are deemed moderate to extremely important for each group, are used in the survey. Additionally, the questions are tailored to the identified group's environment. The results of the survey are used to calculate the group's AC.

Calculate Awareness Capability (AC)

The results of the survey are then used to obtain an average score for each control for each group, used as the AC metric. For example, for control X, the survey results for 10 users from group Y are 2, 3, 2, 4, 5, 2, 2, 4, 5 and 2. The scores are added together, resulting in a total of 31 out of 50. This is then divided by 10 (the number of participants) to give an AC score for control X and group Y of 3.1.

Calculate Awareness Risk (AR)

The awareness risk matrix must be applied in the TO environment so that it aligns with the organization's risk framework. This results in an awareness risk matrix that will be used to plot the AI and AC scores, to determine the AR score for each control.

Analyze Results

The AR rating for each control will be used to determine what training should be implemented in the environment for each user group. The control with the highest AR will be used to compile and implement the ISAC strategy for the coming training period. The ISAC strategy is then implemented by the information security team.

Determine Effectiveness

Once the training curriculum addressing the AR has been successfully implemented and is considered complete by the information security team, the same questionnaire that was used to measure AC will be sent to a new sample set of users for each target group (i.e. it does not contain any of the users that answered in the original sample group). The answers from the second sample group will then be reflected on the awareness risk matrix. Should the userbase have learnt anything from the ISAC, their AC score will be higher, therefore reducing the AR score. As the AR score of a control reduces, a new control with a higher AR score will become the new focus for the next ISAC period.

Target Organization Overview

The TO is required by both regulation and legislation to implement ISA within their environment, with the objective to change the behavior of staff to be more security minded. The TO is aligned with international best practices and has chosen the Information Security Forums Standards of Good Practice (SoGP) [START_REF] Chaplin | The standard of good practice for in-formation security 2016[END_REF] as their primary ISF. The responsibility of implementing training within the TO is given to the IT security department where there are multiple security teams, including: information security, cyber security, and access control, to name a few. Senior members of the various teams within the IT security department form part of the ISAC stakeholder group. The TO is required by their board of directors to deploy at least four topics a year, allowing topics to be deployed every three months at a minimum. Training is deployed in multiple ways depending on the level of training or awareness the information security team needs to implement. For structured e-learning style training, the information security team makes use of a learning management system, which deploys the training to the end-user and tracks their progress and completion status. Elearnings are either sourced from an external supplier who is an expert on the topic or created internally by the TO's human resource employee education department that specializes in educating staff, with the information security team playing the role of the subject matter expert.

ISAC Implementation and Lessons Learned

The method was implemented and evaluated in a financial institution in South Africa over a ten-month period in 2018 (January to October). Ethical clearance from the university and permission from the organization was obtained before any data was collected. Implementation and evaluation data were collected in several ways. The first collection method was in the form of artefacts created in Microsoft Excel to capture the responses of the IT security team in a structured manner. The second method was a questionnaire that was sent to a group of end-users to obtain the participants' understanding of the material given to them. The content was obtained from documents and questionnaires previously created by the TO. Lastly, feedback regarding the method's efficacy was collected through observation by the researcher as well as from the IT security team; the team members were asked two questions: "In your opinion, what worked well?" and "In your opinion, what did not work well?". During the research process we endeavored to minimize bias by using established frameworks and questionnaires, discussing preliminary findings for alternative explanations, and being open to contrary evidence. We are confident that bias was sufficiently considered and that the research was conducted to a high ethical standard.

Awareness Importance (AI)

The control framework is a list of information security best practice controls which was given to senior staff members of the IT security department that are stakeholders in the ISAC, to measure AI. The control framework aligns to the 2016 version of the SoGP, as the 2018 version [START_REF] Jordan | The Standard of Good Practice for Information Security 2018[END_REF] was not published until later in the year. The instrument is an adaptation of the instrument used by Poepjes [START_REF] Poepjes | The development and evaluation of an information security awareness capability model: linking ISO/IEC 27002 controls with awareness importance, capability and risk[END_REF]. The stakeholder will rate the AI on a 1 to 5 ordinal scale, with 5 being the highest level of importance.

Data for the control framework was collected by the researcher, with assistance from the information security team. The target audience for the control framework were senior members of the IT security department who are stakeholders in the ISAC. By answering the control framework, these senior members will provide the information security team with their individual understanding of what training is important for each target group (senior management, branch staff, privileged users, contact center and end-users).

The data was collected by sending the control framework to each senior IT security department team member with instructions on how to complete it. Each senior IT security department team member was given two weeks to complete the task. After three months of project prioritization sessions, the control framework was completed by all senior IT security department team members. As an example, Table 1 highlights the top six topics for the end-user group. Lessons learned. Using TO's ISF proved to be very effective, as IT security staff were already familiar with the topics and controls within the framework. As the TO strives to comply with this framework, it meant that most controls within the framework were implemented already. While the translation of this standard into a questionnaire was detailed and comprehensive, determining AI proved to be very difficult, as interpreting the importance and impact of a control can be subjective. Additionally, the length of the control framework proved to be a challenge, as the time taken to complete the AI rating resulted in a slow response rate from participants. Additionally, participants had various degrees of understanding about the SoGP controls. While the information security team believed that all IT security department team members were familiar with the standard, this was not the case, with some participants struggling to understand the control on its own, how one control differentiates from another, or how the control changes and applies depending on the stakeholder group.

Their seemed to be a difference of opinion about the grouping of stakeholder groups with one participant stating that "The one thing that really worked well was using the ISF framework to map out areas that needs to be addressed in the different classes of users. Identifying the user classes and applying what is relatable for them from the ISF framework, ensures that all the areas are covered for the correct audience." While another participant stated that the groupings were inaccurate "There should only be a grouping of three user categories -Senior Management, Privileged users and End-Users. I think although only certain controls might be applicable to certain environments or grouping of users in a certain org [organization] structure it is imperative that the business users as a whole should be made aware of all kinds of risks and attacks a business might face."

A suggestion for improving this step would be to create ISA control groups that are mapped to the ISF. For example, many of the controls, such as data protection controls, were grouped together by the respondents and received similar, if not the same, AI score. A grouping under a control named Data Protection can represent all data protection controls, allowing for one control to supplement many of the smaller controls. This will not only shorten the length of the control framework but allow for more control variety within the various ISACs throughout the year. Additionally, this will provide a single ISAC with multiple topics (each information security control broken up into its smaller ISF controls) as well as make it easier to incorporate other ISFs to the proposed method. The grouped controls would also allow for a more generic definition of a control making it easier for participants not familiar with the ISF of the organization to answer the control framework.

Survey Userbase

While the information security team was awaiting responses on their AI control framework, the information security team started working on creating questions for the AC questionnaire based on each topic within the SoGP. The information security team were provided several artefacts to assist in crafting these questions that included: the HAIS-Q [START_REF] Parsons | Determining employee awareness using the Human Aspects of Information Security Questionnaire (HAIS-Q)[END_REF], questionnaire guidelines; and the AI control framework for formatting. The information security team first attempted to create Likert style questions for each topic within SoGP, which proved to be far more challenging than they expected. While theme-specific questions are easily incorporated into Likert style questions, the topics presented in SoGP were far more complex in nature and did not allow for a strongly agree to strongly disagree Likert scale format. The information security team therefore decided to create questions that were aimed at current processes and procedures in the environment that relate to the SoGP controls. Due to several security incidents and projects throughout the environment, several ISA topics were required to be implemented as a matter of urgency, but conflicted with the topics that were identified in the AI results. It was decided by the TO that they would therefore use the AI results to define the topic for one specific target group, the end-user group. The remaining target groups will receive training based on the current need from the business. While this was not ideal, the risk of not performing the required training for the other target groups was too great to ignore.

The information security team creating the AC questionnaire was informed of the decision and immediately started focusing on delivering the questions that were required for measuring the AC of the top ten topics (out of 132) [START_REF] Chaplin | The standard of good practice for in-formation security 2016[END_REF] for the end-user target group. The team created five questions for each topic, which was set up on Survey Monkey. An invitation with a link to the survey was emailed to a random sample of users. Users were given two weeks and a total of 47 users successfully completed the questionnaire. The information security team felt that this was enough to proceed to the next step. The challenge of competing for employee attention has also been noted in other ISA training contexts [START_REF] Alshaikh | An Exploratory Study of Current Information Security Training and Awareness Practices in Organizations[END_REF].

Lessons learned. Using the results from the AI questionnaire to build a questionnaire targeting the end-users proved to have many challenges. The mapping of the ISF controls to general ISA questions or previously used questions from academic papers was near impossible. This resulted in the information security team having to create the questions themselves, as opposed to relying on past research. While the time taken to build this questionnaire took far longer than expected, the resulting questions can be easily used again in the future with little to no rework.

Calculate Awareness Capability (AC)

Table 2 shows the average score received from the questionnaire. Score ranges from 0 to 5. With a score of 5 meaning the users answered all questions correctly and 0 meaning the user did not answer any of the questions correctly, for that topic. The results show that the end-user target group is unfamiliar with several security controls within their day to day working environment, particularly information classification and handling and the processes and procedures defined in the Acceptable Use Policies (AUP). The sores in Table 1 and 2 is then used to map to the awareness risk matrix to determine the AR score for each control. Lessons learned. The information security team felt the results were a true reflection of the environment with one member of the information security team stating "The one thing that really worked well was using the ISF framework to map out areas that needs to be addressed in the different classes of users. Identifying the user classes and applying what is relatable for them from the ISF framework, ensures that all the areas are covered for the correct audience."

Calculate Awareness Risk (AR)

The TO makes use of an organization-wide risk framework that is developed by the risk management business unit and approved by the board of directors. It examines two main dimensions, impact and likelihood. For the organization impact holds greater value than likelihood, as the TO sees any impact within the short term (three years) as being significant enough to warrant attention. The strategy applied by Poepjes [START_REF] Poepjes | The development and evaluation of an information security awareness capability model: linking ISO/IEC 27002 controls with awareness importance, capability and risk[END_REF] is then applied to this matrix, supplementing likelihood with AC and impact with AI. The result of mapping aggregated scores for AI and AC for each grouping will be an AR rating for each control. The AI and AC scores for the end-user target group were then mapped on the awareness risk matrix. The AC score is inverted to map to the risk matrix. This is done by taking the highest rating and subtracting the current score. For example, "IM1.1 Information Classification and Handling" received an AC score of 1.7 in Table 2, which is subtracted from the highest risk rating of 5, to give the risk matrix AC mapping of 3.3. On the risk matrix the area towards the top left indicates very high risk. Controls in this area require immediate attention and should form part of the primary objectives of the next ISAC strategy. As can be seen "SM1.2 Acceptable Use Policies" and "IM1.1 Information Classification and Handling" are identified as very high risk for the organization.

Lessons learned. The plotting of the results of the AI and AC questionnaire provided a very effective means of illustrating the AR within the environment. Unfortunately, the 5x5 risk matrix results in groups of ARs, requiring the information security team to revert to the raw data to determine the highest AR. Additionally, the awareness risk matrix required the original AC score to be inverted to map to the awareness risk matrix. While this did not affect the results of the AR score, it was an unnecessary complication in an otherwise well-developed step.

Analyze Results

The results from mapping the AI and AC score showed that the largest AR score comes from the topic "SM1.2 Acceptable Use Policies". The information security team agreed that this reflects what they had suspected and used this topic for the ISAC. The information security team then started building an ISAC strategy to increase end-users' awareness of AUP controls within the environment. ISAC Strategy. The team had multiple discussions with various vendors and departments to determine the best way to deliver training for this topic. Due to the topic being organization-specific, generic e-learnings from vendors proved to be inadequate. Recommendations from the human resources employee education department within the organization required creating a very lengthy and complex, custom built, elearning. This too was deemed inadequate, as the team knew from prior training efforts that lengthy e-learnings do not have a high completion rate.

The information security team decided to focus on an awareness campaign with the overall theme of "Protecting the family". The theme was chosen because it aligned with a family-focused product which the organization was rolling out, as well as the organizational culture (communications often referred to 'our family' and 'being part of a great family'). The goal was to instill a security mind-set and a sense of responsibility in employees.

To implement the campaign posters and desktop backgrounds (wallpapers) were designed. Large (A3) posters were displayed in social areas in every building. These posters were intended to be eye-catching, with minimal text but reference to where employees could go to find out more. Smaller (A4) posters were placed behind every bathroom stall door and contained more detailed information. The desktop backgrounds also had minimal text but provided the user with the central call-to-action and link to more information.

The information security team then created a series of emails in line with the style of the posters, with the content being only a few short paragraphs. An email was sent out every two weeks, each tackling a different section of the AUP. The topics were: sharing data, laptop security, policy document awareness and importance, communicating security risks, removable media, backups and shared drives, and disposing of information. Content was in plain English and could be read in under three minutes.

Lessons learned. The visual representation of the AR within the environment made identifying the required training very easy. The information security team felt that it was a true reflection of the current environment, with one member of the information security team stating: "One could gain a true direction for awareness training within an organization".

Building the ISAC strategy proved to be quite challenging, due to the nature of the topic chosen. As the topic focused on AUP, the training needed to be customized to the environment, making it impossible to use generic training from a third party proficient in the topic. While the content was easy for the team to come by, as it came from a company policy, deciding on the method to deploy the training was not. The chosen method was a series of emails supported by a marketing campaign. The information security team could benefit from previous research, such as that done by Pattinson et al. [START_REF] Pattinson | Adapting Cyber-Security Training to Your Employees[END_REF], to determine what method of training is best suited for different end-users.

Determine Effectiveness

Once the information security team had deployed the training to their satisfaction, the questionnaire and email was sent to a further 255 respondents. Respondents were given three weeks to complete the survey with a reminder e-mail sent a week before final closure date. A total of 36 users successfully completed the questionnaire. The average score received from the second questionnaire show that there was a slight increase (from an average score of 1.9 before training to 2.5 after training) in the user's knowledge of the security controls within their day to day working environment. The original AI and post training AC score for the end-user target group was then mapped on the awareness risk matrix. There was a clear move of AC to the right (more capable), making the AR decrease from a very high risk to a high risk.

Lessons learned. Re-performing the survey, as well as calculating the new AC and AR scores was done significantly faster than many of the other steps. The artefacts created at the start of this method made this step very efficient.

The move of the AC of the target group from very high to high proved to be a very effective illustration of the effectiveness of the training implemented. The information security team believed that the method used to train the staff was not the most effective but believed that the slight move was an accurate view of the effectiveness of the ISAC and were therefore pleased with the results.

Conclusions

While we acknowledge the limitations of statistical generalization within a single case study we propose that analytic generalization is possible. Such lessons learned may potentially apply to a variety of contexts. Our study reflects on the practical steps required in an ISAC and the hurdles staff who implement it will have to overcome. These include identifying and prioritizing training topics, creating training material, and effectively measuring learning.

Several limitations should be considered in this study. The TO comes from a highly regulated industry, which requires compliance with many controls. This requirement drives some of the decision-making in this case. Due to compliance requirements and risks prevalent in the environment, the method was only used on a single target group that was considered less risky by the TO. The implementation of this method over a longer period would provide the opportunity to not only collect more data but help improve and evolve the method. Additionally, the content and mechanisms used to deliver training was decided on by TO staff. While TO staff were sure of its effectiveness as a training delivery method, there is no evidence to validate their assumption, other than their experience in the field and knowledge of the environment.

While our study contributes to the validation of the model proposed by Poepjes [START_REF] Poepjes | The development and evaluation of an information security awareness capability model: linking ISO/IEC 27002 controls with awareness importance, capability and risk[END_REF] it also indicates the need for further research. Within the field of ISA training ways to efficiently measure the effectiveness of a campaign are needed. Further research could also investigate the methodology used to conduct the training, and how this impacts the effectiveness of the ISAC. Additionally, the incorporation of hard measures, such as phishing simulation results and other user behavior metrics, could prove effective but more research is required on what those hard measures should be.

Fig. 1 .

 1 Fig. 1. ISAC Methodology

Fig 2

 2 demonstrates the AI and AC mapping.

Fig. 2 .

 2 Fig. 2. End-user awareness risk matrix

  [START_REF] Daniel Ani | Human Capability Evaluation Approach for Cyber Security in Critical Industrial Infrastructure[END_REF] shows how the instruments and metrics were combined in a step-wise approach.

		Control Framework	Control Awareness Survey
	Start	1. Calculate Awareness Importance (AI)	2. Survey Userbase	6. Determine Effectiveness
	3. Calculate Awareness Capability (AC)	4. Calculate Awareness Risk (AR)	5. Analyze Results	ISAC Strategy
		Awareness Risk
		Matrix

Table 1 .

 1 AI for end-users

	SoGP 2016 Controls	Average Score
	PA2.5 Portable Storage Devices	4.3
	SM1.2 Acceptable Use Policies	4.2
	BA2.2 Protection of Spreadsheets	4.2
	PA2.4 Employee-owned Devices	4.0
	PM2.2 Security Awareness Messages	3.8
	IM1.1 Information Classification and Handling	3.8

Table 2 .

 2 AC for end-users

	SoGP 2016 Controls	Average Score
	IM1.1 Information Classification and Handling	1.7
	SM1.2 Acceptable Use Policies	1.9
	BA2.2 Protection of Spreadsheets	2.4
	PA1.2 Office Equipment	2.4
	PA2.4 Employee-owned Devices	2.7
	PA2.3 Mobile Device Connectivity	3.1
	IM2.2 Sensitive Physical Information	3.2
	PM2.2 Security Awareness Messages	3.2
	IM1.2 Information Privacy	3.4
	PA2.5 Portable Storage Devices	3.5

Acknowledgements. This work is based on the research supported wholly / in part by the National Research Foundation of South Africa (Grant Numbers 114838).


Dimitrios Damopoulos

Susanne Wetzel
email: swetzel@stevens.edu

Introducing Research into the Undergraduate Curriculum in Cybersecurity

Keywords: Research, Cybersecurity, Bachelor Degree, Senior Design Course Sequence

In this paper, we describe a new approach for introducing a research experience based on real-world research problems into the two semester Senior Design course sequence of the B.S. in Cybersecurity program at Stevens Institute of Technology as part of the INSuRE consortium. We discuss the necessity of such a research experience in the context of the ever-changing nature of Cybersecurity in that attacks seen and countermeasures developed today may already be different and outdated tomorrow. We also detail the insights gained, challenges encountered, and lessons learned in order to provide the community with the necessary means to use this as a model to implement a research experience as part of other undergraduate Cybersecurity degree programs.

Introduction

In the recent past, an increasing number of four-year institutions in the United States started dedicated undergraduate programs in Cybersecurity. In the past, degree programs focused on Cybersecurity were mostly available on the graduate level-often building on undergraduate degrees in computing, engineering, or business-or Cybersecurity was offered as a possible concentration in the context of other traditional undergraduate degree programs-most prominently in Computer Science, Computer Engineering, Mathematics, or Business.

The main benefit of a dedicated undergraduate degree program in Cybersecurity is in that it allows for a greater focus on Cybersecurity content-both in terms of depth and breadth. Consequently, students have more opportunities and options to select a focus within Cybersecurity to pursue their specific interests. Generally, this approach allows for a more comprehensive education in Cybersecurity and is thus assumed to better prepare students for a career in Cybersecurity-already by means of an undergraduate degree. This is particularly important considering the fact that in the US the majority of students pursuing an advanced degree in computing-focused fields are non-domestic students [START_REF]Why are American Graduate Programs in Math, Statistics and CS Dominated by International Students?[END_REF][START_REF]Foreign Students and Graduate STEM Enrollment[END_REF]. Furthermore, there is a widely reported workforce shortage in Cybersecurity-and providing a comprehensive education only at the graduate level seems unlikely to be sufficient to address this issue [START_REF]New Data Shows Expansion in Cybersecurity Worker Shortage[END_REF][START_REF]US Lawmakers Introduce Bill to Fight Cybersecurity Workforce Shortage[END_REF].

In designing an undergraduate degree program in Cybersecurity, the challenge is not only to find the right balance between depth and breadth in terms of covering the various areas in Cybersecurity but to also ensure that the intrinsic characteristics of the field are properly covered and taught as part of the curriculum. Most importantly, unlike other fields in Science, Technology, Engineering and Mathematics (STEM), Cybersecurity is characterized by the fact that anything and everything may change at an every-increasing pace. Every day brings new challenges and attacks that often require the swift development of novel solutions and countermeasures in order to ensure suitable security guarantees for any and all services, applications, operations, and infrastructure in today's cyber-driven world.

It is in this context that this paper describes an implementation of the socalled Information Security Research and Education (INSuRE) program [START_REF]INSuRE Project[END_REF] as part of the required two semester Senior Design course sequence in the B.S. in Cybersecurity program at Stevens Institute of Technology [START_REF] Wetzel | Cybersecurity at Stevens Institute of Technology[END_REF]. This paper argues that the INSuRE effort-which exposes the students to a real-world research experience-provides an ideal means to teach the skills necessary in order for them to succeed as Cybersecurity professionals. In particular, through INSuRE students gain a unique educational experience which is team-based (recognizing that most Cybersecurity efforts are carried out by teams), multi-/crossdisciplinary (as Cybersecurity generally draws upon many different disciplines that must be properly integrated in order to enable the development of suitable solutions), multi-institutional (as Cybersecurity problems tend to span across multiple organizations), as well as research-oriented and time-scaled (as devising responses to new and ever-changing vulnerabilities and attacks is typically very time-critical) [START_REF]INSuRE Project[END_REF][START_REF] Sherman | The INSuRE Project: CAE-Rs Collaborate to Engage Students in Cybersecurity Research[END_REF][START_REF] Sherman | INSuRE: Collaborating Centers of Academic Excellence Engage Students in Cybersecurity Research[END_REF].

Previously, INSuRE was mostly offered on the graduate level, typically in the context of a one semester elective course. A few institutions, including Stevens, also worked with teams comprised of both graduate and undergraduate students. Furthermore, Dakota State University did a pilot where teams of undergraduate students were paid a stipend instead of earning course credits for their participation in INSuRE [3]. Stevens was the first to carry out INSuRE with teams comprised of all undergraduate students for a duration of two consecutive semesters in the context of a required Senior Design course sequence. Before that, it was not known whether it is feasible to introduce and carry out this kind of research experience in the context of a required two semester sequence on the undergraduate level in an effective manner. Outline: The remainder of the paper is organized as follows: We first provide a brief overview of the INSuRE program in general (Section 2) and then detail its novel implementation in the context of a required two semester Senior Design course sequence (Section 3). This is followed by a discussion of lessons learned and challenged encountered (Section 4).

INSuRE

The INSuRE research collaborative is a self-organizing, cooperative, multidisciplinary, multi-institutional, and multi-level research collaboration of National Centers of Academic Excellence in Cyber Defense Research (CAE-R) and Cyber Defense Education (CAE-CDE) universities [5,[START_REF]INSuRE Project[END_REF][START_REF] Sherman | INSuRE: Collaborating Centers of Academic Excellence Engage Students in Cybersecurity Research[END_REF][START_REF] Sherman | The INSuRE Project: CAE-Rs Collaborate to Engage Students in Cybersecurity Research[END_REF] that "cooperate to engage students in solving applied Cybersecurity research problems". The Center of Academic Excellence in Cyber Defense Research (CAE-R) designation was launched in 2008 with the purpose to increase the understanding of "robust Cyber Defense technology, policy, and practices that will enable our nation to effectively prevent or respond to a catastrophic cyber event" [6]. The CAE-R designation recognizes universities for their institutional excellence in Cybersecurity research. The vision for the program was to "establish a process that will present opportunities for Cyber Defense research centers to drill deeper into much needed solutions to securing the global information grid and provide NSA, DHS, and other federal agencies with insight into academic Cyber Defense programs that can support advanced academic research and development capabilities" [6]. However, until INSuRE was launched, these government agencies did not have an effective means to tap into the expertise within the CAE-Rs-except for working with some faculty or institutions on an individual basis.

Starting with one university and one government agency in Fall 2012, INSuRE has since grown into a collaboration of 20+ universities (holding the CAE-R or CAE-CD designations) with many government agencies and labs (on both the federal and local levels)-which suggest the real-world research problems in Cybersecurity (of national interest and need) and provide technical oversight to the student teams. Over the years, the INSuRE partners have worked on more than 100+ distinct research problems resulting in more than 150 project reports, involving hundreds of students across the participating universitiesmany of whom have subsequently been hired into government jobs. The effort has resulted in refereed conference publications and published data sets (e.g., [START_REF] Alabi | Toward a Data Spillage Prevention Process in Hadoop Using Data Provenance[END_REF][START_REF] Alves | Virtualization of Industrial Control System Testbeds for Cybersecurity[END_REF][START_REF] Nair | OBD SecureAlert: An Anomaly Detection System for Vehicles[END_REF].

Since the beginning, Purdue University has been the central point and coordinating entity of INSuRE. All participating universities follow a common schedule (possibly with minor local adjustments due to the institution's semester schedule). The coordinating faculty at Purdue solicit research problems from the participating government agencies and labs and make them centrally accessible to all participating universities. In turn, universities locally recruit students to participate. Throughout the semesters, there are various milestones that are coordinated across all universities: Bids: The first major milestone that all students have to complete is the bidding on a number of the contributed research problems. In particular, all students are asked to review the descriptions that the Technical Directors provided for the research problems they contributed. For at least two research problems the students are interested in, they are then asked to write and submit a brief (two-to-three paragraph) statement on why they are interested in this problem, what excites them about the problem, what they see as possible directions for their project, and why they feel they are qualified to carry out the project. The INSuRE coordinator reviews the bids submitted by all students from all the participating institutions and assigns the students to research problems. In doing so, the coordinator seeks to strive a balance of covering as many research problems and Technical Directors as possible while recognizing the student's interests and preferences as best as possible. Also, teams are typically formed at a participating institution-while teams across institutions are possible and have been done. Project Proposal: Once project teams have been formed, the teams prepare a project proposal that outlines the team's ideas, directions, milestones, distribution of work, etc. for their project to address the assigned research problem. This also includes an initial literature review. Typically, the teams are in close contact with the Technical Director who contributed the research problem while preparing the project proposal. Oftentimes, Technical Directors phrase the descriptions of their research problems in such a fashion that many different directions and deliverables for a project are possible.

The purpose is to allow the students to develop their own ideas and shape their project. Once completed, the Technical Director reviews the project proposal-possibly asking for changes or additions as needed. Once the Technical Director approves the project proposal, the team starts working on the project, executing its plan. Midterm Deliverables: About half way through the semester, each team prepares a report and a slide deck documenting the current status of their projects. All reports and slide decks are shared with everyone (i.e., all participating institutions and Technical Directors) through the common project platform. In addition, all teams are scheduled to give a midterm presentation-facilitated through WebEx sessions (arranged for by the coordinators at Purdue University). Any participating institution, their teams and students, as well as all Technical Directors may participate in these WebEx sessions. These open sessions provide everyone with a unique learning opportunity on a large variety of projects on current topics (including their state-of-the-art, open issues, results achieved, etc.)-almost like a conference in Cybersecurity that is covering topics across the entire spectrum. End of the Semester Deliverables: Similar to the Midterm Deliverables, the teams prepare reports and slide decks to comprehensively document their results at the end of the semester. Once again, these documents are shared with everyone and the respective final presentations are facilitated through WebEx.

All offline interactions and data sharing are handled through a central platformreferred to as INSuREHub [START_REF]INSuRE Project[END_REF][START_REF] Sherman | INSuRE: Collaborating Centers of Academic Excellence Engage Students in Cybersecurity Research[END_REF][START_REF] Sherman | The INSuRE Project: CAE-Rs Collaborate to Engage Students in Cybersecurity Research[END_REF]. While the coordinators maintain the central components (including the uploading of the research problem sets, semester schedules, etc.), each institution has a dedicated space on INSuREHub where deliverables are uploaded and can be accessed not only by the Technical Directors co-supervising the projects but the INSuRE community at large.

It is important to note that it is not unusual for a Technical Director to offer the same research problem for multiple semesters as a single semester might not be enough time for a team to comprehensively address all facets of a specific research problem. Teams who are assigned to a research problem that has been offered and worked on before have access (through the INSuREHub) to the previous deliverables of these teams (i.e., Project Proposals, Midterm and End of the Semester Deliverables) and may use these as a jump-off point for their own projects. Consequently, there have been research problems that have been worked on for multiple semesters and have seen contributions from various teams leading to a wealth of directions and results over time [START_REF] Sherman | The INSuRE Project: CAE-Rs Collaborate to Engage Students in Cybersecurity Research[END_REF].

In addition to the milestones that are the same across all participating INSuRE institutions, there may be additional ones that individual institutions implement. For example, many (including Stevens) also require the following:

Weekly Dashboards: Each week, the teams are required to submit a two-page slide deck that reviews the status of the project. In particular, as part of their slide deck, the teams discuss the progress they have made during the past week, whether they have encountered any challenges, or whether they have any questions. In addition, each team member reviews what s/he worked on, what s/he individually contributed and delivered to the project, and what s/he is planning in terms of both activities and deliverables for the coming week. The teams also present these dashboards in class-where they also demo the code, review submitted deliverables, and field questions from the instructors and other classmates. The Weekly Dashboards are made available to the Technical Directors in order to keep them updated on the projects. The dashboards also serve as input and basis for the regularly scheduled meetings with the Technical Directors. Regular Check-in with Technical Directors: Regular meetings (e.g., conference calls) are scheduled with the Technical Directors-ideally during the weekly class time(s). As part of these meetings, the students report on their progress, ask questions, and provide the Technical Directors with the opportunity to give input and shape the projects as needed and desired. in Cybersecurity program at Stevens. In particular, the grant included funding to hold in-person meetings of the teams with the Technical Directors of their project. We arranged for the meetings to take place either right before or at the very beginning of the second semester-thus constituting an integral part of the overall realignment process of the project. Depending on the schedules of the Technical Directors, the meetings typically lasted 2-4 hours and included demonstrations and presentations by the teams, as well as indepth discussions with the Technical Directors on the progress made during the past semester and plans for the semester ahead. Optional Extra-credit Assignments: It is not unusual for students to ask for extra-credit assignments in order to boost their grades. To facilitate such requests and yet make it fair to all students in the class, we have designed a set of assignments that all students can take advantage of. In particular, in the first semester we offered a group assignment for the teams to start early with the planning for the second semester of the two semester Senior Design course sequence. In addition, we gave individual team members the opportunity to work on the project over the break in between the two semester and earn extra-credit for doing so in both semesters (for planning the work to be done at the end of the first semester and reporting on the completed work at the beginning of the second semester).

Grading: Most of the deliverables were graded on a per team basis. The only exception was the Weekly Dashboards where the weekly deliverables and planning include a major individual grading component. This was done on purpose in order to give each team member the opportunity to show his/her individual contributions and distinguish himself/herself beyond the team's performance.

Program The two semester Senior Design course sequence at Stevens is to be "the culmination of the undergraduate experience, where knowledge gained in the classroom is applied to a major design project" [8]. In the past, Cybersecurity majors completed their Senior Design course sequence together with the Computer Science majors. However, many Cybersecurity majors expressed great frustration and dissatisfaction as the typical Computer Science Senior Design projects had too little Cybersecurity content. Instead, their efforts had to mostly focus on software development aspects. In order to address this shortcoming, we decided to offer a dedicated Senior Design course sequence for our Cybersecurity majors. The goal was to allow the Cybersecurity majors to apply the Cybersecurity knowledge, skills, and abilities they had acquired in their course work in the context of a comprehensive, challenging, real-world Cybersecurity project. We feel that implementing the Senior Design course sequence as part of INSuRE is especially well-suited to meet this goal.

Discussion: Challenges Encountered and Lessons Learned

In the past, Stevens offered INSuRE in the more traditional setting as it is to be found across most of the institutions participating in the consortium over the years. Specifically, the course was offered as an elective course (either as independent study course or special topics course) with some/all team members pursuing a Cybersecurity degree program or concentration in a computingfocused degree program on the graduate level (for more details see [START_REF]INSuRE Project[END_REF][START_REF] Sherman | INSuRE: Collaborating Centers of Academic Excellence Engage Students in Cybersecurity Research[END_REF][START_REF] Sherman | The INSuRE Project: CAE-Rs Collaborate to Engage Students in Cybersecurity Research[END_REF]). Consequently, there are two major differences to the newly implemented Senior Design sequence in the context of INSuRE:

Elective vs. Required Course(s): For elective courses, students typically choose courses that best meet their expectations-typically including factors such as their own experience, interest in the topic, anticipated workload, instructor, timing, etc. In general, students taking the elective have made a deliberate decision to take the course. This is very different in case of a required course where students have no choice but complete the course in order to meet graduation requirements. In turn, it means that a required course will include a cross-cutting spectrum of students (w.r.t. strengths, weaknesses, and proficiency). Undergraduate vs. Graduate Students: Typically, graduate students are more advanced, have more experience, or more generally are more mature.

While not a must, these traits at the very least can prove very beneficial when working in a team, dealing with the unknowns and challenges of a research problem, and working under tight time constraints.

In this context we now discuss challenges we have encountered and lessons learned in offering the two semester course sequence for the past two academic years. When possible, we indicate changes we introduced for the second teaching in response to the challenges encountered in teaching the sequence for the first time. For some of the challenges we have not yet introduced any countermeasures as they may need more investigation or implementing solutions may also require some non-trivial changes to the Stevens' B.S. in Cybersecurity curriculum [START_REF] Wetzel | Cybersecurity at Stevens Institute of Technology[END_REF].

Typical Student Work Pattern: Many of today's (undergraduate) degree programs in the US seem to be structured such that students can succeed by solely focusing on and working towards the typical course deliverables that come with a (hard) deadline-many of which are now scoped such that students can still complete them rather last minute if they just put in some major effort right before such a deadline. This holds true especially for homeworks and in some cases even exams. This pattern proves a major challenge in the context of this Senior Design course sequence. Specifically, such a research-oriented implementation of the Senior Design course sequence requires that all students in the team consistently work on the project every week. Last minute power work does not prove effective and does not lead to the progress necessary and expected-especially also by the Technical Directors. It proves extremely challenging to get the students to change their work pattern-even though multiple measures have been introduced to at least enforce some of it. This includes that the students must develop a timeline as part of the Project Proposal-which they are required to adjust for each Status Update and as part of the Project Realignment milestone. Furthermore, the Weekly Dashboards are meant to instill some level of regular accountabilityforcing the students to report on the work they have completed in the past week and plan ahead for the coming week. Still, we have continuously experienced that students deviate from these plans-especially at times where they have major deliverables or milestones in other classes (e.g., homeworks or midterm/final exams). Students seem to think that there is enough flexibility in this Senior design course sequence that allows them to make up for working less one week by putting in more time during a subsequent week. Clearly, the students underestimate ripple effects and the dependency as well as impact on other team members.

We have experienced that holding the students accountable to the deliverables they have committed to for certain dates (e.g., by deducting points for not meeting them in time) can lead to a lot of tension and complaints. Students tend to do things their way, i.e., the way they have done it in the past and continue to successfully do it for other courses. While some express their dissatisfaction openly, others only do so in the course assessment at the conclusion of the term.

Senior Design Sequence in the Curriculum: Related to the previous matter is the fact that the two semester Senior Design course sequence merely accounts for one of the five three-credit courses that the students are scheduled to take during each semester of their Senior year. 1 Consequently, there is a natural limit on the amount of time that the students can realistically spend working on their Senior Design project. While the expected time commitment outside of the classroom is 6-9 hours per week for a three credit course, in reality students devote a decreasing amount of time on a course with an increasing number of courses they are enrolled in. Given the experience of the first teaching of the two semester sequence with the "I did not spend much time on the project because it was a busy week with other classes", for the second teaching we increased the focus on the Weekly Dashboards seeking to hold the students more accountable to the timelines they have committed to.

Research Methods: Most of the undergraduate students have no major experience in project work other than software development at a small scale or based on typical textbook problems. Furthermore, the main goto source in case of questions is Wikipedia. Even when provided with research papers along with instructions on how to search for papers they cite and papers they are cited by in order to get a comprehensive picture of the topic, by and large the students seem to lack the patience to do so. Also, we experienced a tremendous resistance to thoroughly studying the papers and documenting findings through brief write-ups that can be reviewed by the whole team. Instead of spending some time to comprehensively explore a topic and available resources, we noticed that students would rather just revert to trying things out or coding something up quickly (possibly also to demonstrate activity and produce some results). At this point we can only speculate what the reason is for this-possibly attesting it also to how students have been conditioned-i.e., regurgitate materials instead of making them transfer knowledge to unknown problems.

Once again, we used the Weekly Dashboards during the second teaching of the two semester course sequence in order to force some change in behavior. Specifically, we made the students report on the papers they have read and make them submit short summaries that also discuss what other literature they found based on what they read and what needs to be reviewed next. Similarly, we have used the Weekly Dashboards and the class time where students present their progress to challenge the rapid prototyping and development directions the students prefer to take. In some cases we have even stepped in and not allowed the students to focus on coding until they finished at least some basic specification or mathematical description of the underlying problem.

As students progressed through the two semester course sequence, we noticed that students acquired more of a critical thinking and adopted a more structured approach.

Expectations of Student, Faculty, and Technical Directors Interactions and Commitments: Having Technical Directors from government agencies and labs suggest real-world research problems and co-supervise the teams (together with the faculty at the respective institutions) is a unique opportunity for the students and faculty alike. All parties can benefit tremendously from the interactions alone. However, we learned that this can also bring about challenges and friction. First and foremost, it is important for all to understand that it is a two semester commitment. We learned that it is important to explicitly address this out at the beginning of the first semester. Furthermore, it is important to align expectations. While some Technical Directors (especially those from government labs) may have extensive experience with summer interns, we have come to realize that it is important to manage expectations with regard to experience (which is different when working with undergraduate students versus graduate students) as well as the time commitment (a full-time intern versus a senior who is carrying a full load of classes and only one of which is the research-based Senior Design course sequence).

In turn, students are used to faculty being rather readily available-if not for an in-person meeting then at least by email. However, this is not necessarily the case for Technical Directors as participating in INSuRE is something that many of them do in addition to their usual job responsibilities. Consequently, it has proven rather challenging at times to manage requests for meetings and such. In particular, scheduling a meeting outside of the weekly class time by itself is a non-trivial challenge as individual team members typically have different class schedules and thus finding a common meeting time that works for the whole team, faculty advisors, and the Technical Director alike is generally extremely difficult.

Lastly, it is important to recognize that different Technical Directors not only have different backgrounds but also have different styles in overseeing projects. Some prefer to be hands-off and let the teams find their way-as a fresh set of minds may discover a promising and successful avenue that may lead to great results. Others are very hands-on and expect to see things done in a specific manner. As faculty we thus sometimes have to mediate matters accordingly.

Problem Selection: Another critical element is the selecting of a suitable research problem for the undergraduate student teams to work on for two semesters. It is a tradeoff between too easy (possibly mostly software development focused) versus too technically challenging (possibly even to the extent that the undergraduate students simply do not yet have the necessary background and the topic is too technical in order for the students to quickly get to the level necessary to successfully master the project). Similarly, it is necessary to pick a research problem that allows for a project that is comprehensive enough to span a sequence of two semesters. In turn, if the problem is too extensive, the challenge is to carve out a piece of suitable size that can be accomplished and mastered in two semesters. In addition, there might be research problems that are so tightly defined that students do not have much room to develop their own ideas. In contrast, there may be problems that are so generally phrased that while the students may find them interesting, they do not know where to start and what to aim for. Overall, the right scoping is a major challenge. For the second teaching, we pre-selected some problems for which the students were allowed to submit bids. We are not sure yet whether we will continue with this kind of an approach in the future. While it was easier for students to master the bid phase, it is possible that some students would have better liked some of the other research problems.

Orthogonal to the above is an observation we made after the bid phase for the second teaching. While we had students bid on research problems enthusiastically, some then were stunned when they were actually assigned to these problems. When asked why this came as a surprise, it became apparent that the students had underestimated that stating interest could result in them being assigned to the problem. Or more generally, it seemed that some students did not fully grasp the magnitude and importance of the problem selection at the onset and its impact not just for a short period of time (say days or weeks as in case of deliverables in other classes) but that instead this constitutes a long lasting commitment for a full academic year.

In general, in both years we have learned a lot and anticipate to continue to learn more in future years. We expect that for some of the encountered problems and challenges we will find good solutions over time that will then manifest themselves in rules and directions. For others, we expect that we will need to cope with them as they appear.

Conclusion and Future Work

Implementing INSuRE in the context of a two semester Senior Design course sequence has been exciting, challenging, and rewarding at the same time. Some of the challenges faced in the first two years were unexpected. Some of the changes introduced after the first year paid off, others still have to be reassessed. We expect that further adjustments and changes will be made based on the teaching of the two semester course sequence in its second year. For sure, we plan to explore different options of identifying potential projects (from the set of all offered projects) for our students to choose from.

Generally speaking, the experience of the past two years provides a positive answer to the question raised above as to whether it is feasible to introduce and carry out this kind of research experience in the context of a required two semester sequence on the undergraduate level in an effective manner. First, there are research problems contributed as part of INSuRE that can be carried out by undergraduate student teams over a sequence of two semester. Second, the contributed research problems are non-trivial thus requiring the students to apply a wide range of knowledge, skills, and abilities they have acquired throughout their course work. Third, it is possible for undergraduate students to make substantial progress and develop exciting new approaches to challenging research problems.

In particular, over the past two years, our teams successfully carried out a number of projects. For example, for the research problem of devising effective countermeasures to phishing, one of our teams developed a new approach based on visual similarity testing. The team implemented a browser plug-in, a backend testing tool, and carried out user studies to better understand user behavior in the context of phishing. Another team worked on the research problem of designing effective counter drone mechanisms. Specifically, the team worked on classifying drones based on characteristics found in the 802.11 traffic of drones networks.

Questions we cannot answer to date include whether teams with (some) graduate students would have been able to develop better solutions (faster), whether two semester projects generally result in more comprehensive solutions than those of one semester projects, etc. In order to answer such questions one would have to break with some key principles of INSuRE. In particular, it would be necessary to assign various kinds of student teams (one semester vs. two semester, all undergraduate vs. all graduate vs. mixed teams) to the same research problem, forcing them to pursue the same approach to allow for the comparability of results. Most likely it would also be necessary to do this for a variety of research problems in order to ensure that the results are not specific to a particular research problem.

Going forward, we intend to carry out a comparative study (w.r.t. effectiveness, difficulties, challenges, and success) with other educational systems which introduce research components into the curriculum in other ways. For example in Germany, students pursuing a Bachelors degree in Cybersecurity or Computer Science at a university are typically required to write a Bachelor thesis. The time frames allotted to complete this requirement vary-in some cases it is three months in others it is up to six months. While working on their thesis, students typically are not enrolled in classes and instead can focus all their time and efforts on the project. Yet, a Bachelor thesis is typically completed on an individual basis. Also, earlier in their studies, students are often required to complete a seminar course in which they may already be introduced to some research methods-in particular in conducting a thorough literature search for a specific topic. We would like to investigate similarities and difference in terms of challenges and opportunities. Questions we seek to answer include: Does working in a group when first being introduced to research make things easier or more difficult? Would increasing the number of credits for the Senior Design courses (and thus decreasing the number of other courses that students would take in parallel) improve the quality and outcomes of the projects? How beneficial is it to expose students to some research methods earlier in their studies-e.g., through a seminar course where students are exposed to current research and are required to conduct a literature search?

3

  Structure of INSuRE in the Context of Senior DesignIn implementing INSuRE in the context of a two semester Senior Design course sequence during the Senior year of our B.S. in Cybersecurity degree program at Stevens Institute of Technology[START_REF] Wetzel | Cybersecurity at Stevens Institute of Technology[END_REF], we had to make some adjustments to the typical deliverables when offering INSuRE as a single course for one semester only (as described above). In particular, this pertains to defining effective means to connect the two semesters-especially regarding the typical beginning of the semester elements such as the Bids and Project Proposal which are not necessary for the second semester. Instead, we included a milestone to connect back with the Technical Directors and added a Project Realignment milestone:Project Status Updates: Instead of the Midterm and End of the Semester Deliverables, we adjusted the two semester schedule to include a total of three Project Status Update milestones, coinciding with the Midterm and End of the Semester Deliverables of the first semester and the Midterm Deliverables of the second semester. For both the second and the third updates, we require that these status updates build on the previous update and that the respective slide decks and reports are updated and extended on correspondingly to properly reflect the project progress since the last status update. Similarly, with each status update we request that the timeline and deliverables going forward are adjusted as needed. Project Realignment: During the second semester of the two semester sequence of the project, we have adjusted the schedule replacing the Bids and the Project Proposal with a Project Realignment milestone. In most cases, this goes hand in hand with the next newly introduced milestone, namely the facilitating-if possible-of an in-person meeting with the Technical Director. Typically, the Project Realignment milestone is scheduled after such in-person meetings with the Technical Directors and after the first few class meetings have been completed in order to give the students enough time to discuss matters and plan the second semester (including other milestones, deliverables, and assignments of responsibilities to team members) accordingly. Project Meeting with Technical Director: We were fortunate that we were awarded a grant to support the development and introduction of INSuRE in the context of the Senior Design course sequence of our B.S.

  Context: Stevens Institute of Technology was one of the first institutions in the US to offer a dedicated degree program in Cybersecurity on the undergraduate level. The program was approved in 2006 and had its first class graduating in 2011.The B.S. in Cybersecurity program at Stevens[START_REF] Wetzel | Cybersecurity at Stevens Institute of Technology[END_REF] includes strong core requirements in Computer Science and Mathematics as well as an extensive set of courses covering a broad spectrum of Cybersecurity aspects (including Cryptography, Privacy, System Security, and Law). In addition, students may select electives from a wide range of Computer Science of Cybersecurity courses.

It is important to note that many of our students elect to overload during their Junior and Senior years-pursuing both a Bachelor and Masters degree program in parallel. Such an increased workload just further exacerbates the problem at hand in terms of time commitment and time management.

Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Acknowledgements: The authors thank the INSuRE consortium-and in particular Melissa Dark and John Springer from Purdue University-for their continued support and guidance.

In part, this work was supported by the CAE Cybersecurity Workforce Education Grant # H98230-17-1-0342. Some of this work was carried out while Susanne Wetzel was serving as Program Director for the Secure and Trustworthy Cyberspace Program at the National Science Foundation. 2