Table of Links

• Abstract and I. Introduction
• II. Related Work
• III. Background and Motivation
• IV. FluentCrypto
• V. Study Setup
• VI. Study Result
• VII. Threats to Validity
• VIII. Conclusion and References

V. STUDY SETUP

We performed a study, outlined in Figure 2, to evaluate FluentCrypto. We asked participants to complete three basic cryptographic tasks using the Node.js crypto API and FluentCrypto. We designed the tasks based on real-world problems found on the Stack Overflow website. We measured the results by asking the participants to record how long they spent on each task, state a difficulty level and explain the impediments they had while working on each task. We also asked the participants to send us their final code pieces for further investigations. We manually inspected their implementation to assess the characteristics of their solutions in terms of both completeness, i.e., delivering what a task requests, and correctness, i.e., providing a secure solution.

A. Tasks

We designed three tasks based on the frequent problems that developers recently asked about hashing, symmetric encryption, and key-pair encryption on the Stack Overflow website.

1) Task 1: We searched for the “hash” and “node.js” tags on Stack Overflow, which resulted in more than 30 active questions within the last week at the time of running this experiment. We skipped the questions that were not related to problems tied with Node.js crypto API. We analyzed the first ten related questions and realized that seven of askers did not know how to use a hash algorithm; one did not have adequate knowledge to understand hashing; and the remaining were related to libraries. We therefore designed the following symmetric encryption task:

The main goal was to comprehend how difficult it is to perform hashing with each API.

2) Task 2: We searched for the “cipher” and “node.js” tags on Stack Overflow, which resulted in 10 relevant active questions within the last week at the time of running this experiment. Five questions were about lack of knowledge in using the API that yield error or incorrect result, two were about problems with concatenating the results from the calls, two questions were related to the correct length of IV, and one was a high-level question about the cipher. Consequently, we formulated the following task:

We were interested to comprehend how difficult it is to perform encryption and decryption with a symmetric key (cipher) in Node.js crypto API and FluentCrypto.

3) Task 3: We searched for “private encrypt” and the “node.js” tags, and the results contained 251 questions. We analyzed the first top ten questions that were relevant. Eight such questions were about how to generate a key pair for asymmetric encryption or how to use the key correctly in the code, one question was about how to set a password for a private key, and one was related to the API’s difficulty. Consequently, we designed the task below:

The aim of this task was to compare how challenging it is to encrypt/decrypt data with a key pair.

B. Participants

In order to find interested participants, we wrote two blog posts that describe how FluentCrypto works. We then called for participants for our study by sharing the blog posts across programming forums, such as Node.js, JavaScript, cryptography forums, and also with the industrial colleagues who had experience with Node.js runtime environment. Eleven people showed their interest in our work and, in the end, eight agreed to voluntary participate in the study. All the participants were from industry. Specifically, four had 2-5, three had 5-10 and one had more than 10 years of experience in programming. All of the participants had experience in working with Node.js and developed at least one server-side system in this environment. Half of the participants expressed that they are “partly knowledgeable” in cryptography, meaning that they had an indefinite opinion about some fields of cryptography. The remaining participants stated that they are knowledgeable in this field, implying that they are familiar with cryptographic APIs and how and where to apply them. None of the participants expressed that they are “very knowledgeable” or “not knowledgeable” in cryptography. Three participants had used the Node.js crypto API prior to attending this study.

C. Research questions

We sought to answer the following research questions:

• RQ1: Does adopting FluentCrypto increase developer efficiency when working on a cryptographic task?

• RQ2: Is FluentCrypto effective in developing secure cryptographic solutions?

The purpose of RQ1 was to understand whether developers can complete a cryptographic task faster when they use FluentCrypto versus when they rely on the native Crypto API in Node.js. In RQ2, we looked at developer solutions through the lens of security to identify whether developers make fewer mistakes when they use FluentCrypto versus when they use the native Node.js Crypto API.

D. Study procedure

Prior to the study, all participants were given an introduction to the basic concepts pertinent to the three cryptographic tasks. Particularly, we made sure that they understand both the meaning and the purpose of “hashing” and “encryption”, and they appreciate the difference between symmetric and asymmetric cryptography. We then asked the participants to fill out a pre-survey to tell us about their years of experience in programming, level of knowledge in cryptography, previous experience with Node.js crypto API, and finally how comfortable they are with the introduced cryptographic concepts.

We attempted to minimize the impact of wrong assumptions with regard to how subjects report their level of knowledge in cryptography by providing them with an explanation of each level with regard to APIs such as Hash, Cipher, Sign, and concepts behind Secret Key, IV and Salt.

• Very knowledgeable: “I am proficient in cryptography; I fully understand these concepts and am able to use these APIs correctly”.

• Knowledgeable: “I am familiar with cryptography fundamentals; I know the purpose of these APIs and fairly understand these concepts, however, I am not confident to develop a secure solution on my own”.

• Partly knowledgeable: “I have vague ideas about cryptography; I know the basics e.g., what these APIs are, but I rarely developed a cryptographic solution”.

• Not knowledgeable: “I am not familiar with cryptography and have not dealt with a cryptographic task”.

Each task comprised two activities: (i) solving the task with the Node.js Crypto API; and (ii) providing a solution using FluentCrypto. The participants could consult the official Node.js documentation or other online information sources such as the Stack Overflow website, but in case of FluentCrypto the only available reference was the project documentation on GitHub.

Participants had to start from the first task and fill out a postsurvey before moving to the next task. We did not specify a deadline for the completion of a task, and the subjects were asked to fill out the survey regardless of whether or not they could complete a task. We collected the following information via a Google form for each activity related to a task:

• Time taken to work on an activity.

• Perceived level of difficulty expressed as a ten-point Likert scale with 1 being the lowest value and 10 being the highest value.

• Feedback esp. regarding problems encountered while working on an activity.

• Code piece developed by the subject.

In addition to satisfaction which is a subjective feeling, we asked the participants to consider a few factors when rating the level of difficulty to perform a task. Specifically, how easily you learned about the API; how easily you could map a task into API elements; how obvious the purpose of the APIs were from their names; and the level to which you were able to find errors in code and debug it.

Finally, subjects were debriefed on the goal of this study and were asked to sign their consent if they agree that we use their anonymised data to report in a publication.

E. Statistical testing

We ran Wilcoxon signed-rank tests to investigate whether there is a significant difference between the participants’ performance when they use Node.js cryptography API versus FluentCrypto. The Wilcoxon signed-rank test is a nonparametric statistical hypothesis test for dependent samples and is used when distributions are not assumed to be normal.

For each test, we formulated a null hypothesis to determine whether there is a significant difference between the two activities in each task, i.e., when using the FluentCrypto versus Node.js cryptography API. If, after testing a null hypothesis, we find that we can reject the null hypothesis with a high confidence (i.e., α= 0.05), we accept its alternative hypothesis.

An example null hypothesis is H0: “there is no significant difference in the efficiency of participants when they rely on Node.js crypto API versus FluentCrypto for developing cryptographic solutions”; and the corresponding alternative hypothesis is H1: “The efficiency of participants significantly increases when they rely on FluentCrypto versus Node.js crypto API for developing cryptographic solutions”. The remaining null and alternative hypotheses are analogous.