A study of how trust can be assigned in crowdsourced information retrieval

Abstract

For establishing the trustworthiness of information retrieved through crowdsourcing, trust management systems can be used, which compute a trust value used in automated decision making. Using such systems, makes feedback vital since it gives users incentives to do more and better [1]. An instance of the more general problem is experienced by Forza Football, which has a desire to make automated decisions regarding users reported lineups. The purpose of this study is to explore possible ways to determine whether or not information retrieved through crowdsourcing from external entities is trustworthy. For the purpose of this study, the Design Science Research methodology was used, beginning with a literature review for creating awareness of the problem along with gathering requirements from the crowdsourced information retrieval domain, followed by designing the trust management system, conducted through four iterations by evaluating it through validation sessions with relevant participants. The results in this study indicate that using the trust value, to evaluate the trustworthiness of the information retrieved, does not yield similar results as a manual evaluation. When quality is of greatest importance, the data should be compared, for evaluating if several users have provided the same information. In those cases, a threshold can be set for which users’ trust values needs to exceed in order to get their data approved. For lower quality controls, the trust value alone can be used for automated decisions. The feedback should be immediate, containing reasons for why the data was accepted or rejected, not including the trust value.