Mining Ethical Concerns from Open-Source Project Repositories

Abstract

In the process of open-source software development, ethical concerns arise when development decisions involve conflicting values or the interests of stakeholders. Although previous research has identified certain ethical issues in open-source repositories, such as license violations and incorrect attribution, broader ethical issues remain under-researched. This study employs a mixed-methods approach to examine these issues. In the first phase, the researcher conducted an inductive thematic analysis of 134 comments from GitHub issues and pull requests from multiple open source repositories, identifying 18 types of ethical issues and grouping these issues into six themes. The researcher used Cohen’s kappa and Fleiss’ kappa coefficients to assess inter-rater reliability, and the results indicated substantial agreement. In the second phase, the researcher trained supervised machine learning classifiers using a labeled dataset containing 3,084 comments to enable automatic detection of these types of comments. The best-performing binary classifier achieved a macro F1 score of 0.68 in cross-validation, and the theme-level classifier achieved 0.79. External validation on an unseen repository (ansible/ansible) yielded a binary macro-F1 of 0.75 and a restricted theme-level macro-F1 of 0.53. In the third phase, the classifier trained in the RQ2 phase was applied to 54,605 comments. These comments were collected from five repositories over a 36-month period. The results of the third-phase analysis indicate that ethical concerns are a persistent feature of the open-source development process. Furthermore, significant governance events can lead to a measurable short-term increase in the expression of ethical concerns, and the findings suggest that ethical concern frequency may be more closely associated with structural community factors than with short-term activity fluctuations.