Browsing by Subject "Crowdsourcing"
Now showing 1 - 11 of 11
Results Per Page
Sort Options
Item Bridging the gap between mobile CPU design and user satisfaction via crowdsourcing(2016-05) Halpern, Matthew Franklin; Janapa Reddi, Vijay; Tiwari, MohitThis report aims to provide an understanding of how the mobile CPU designs have evolved and its influence on end-user satisfaction. To that end, a quantitative performance analysis is conducted across ten cutting-edge mobile CPU designs studied within top-selling off-the-shelf smartphones released over the past seven years. This analysis is then used to guide a large-scale user study spanning over 25,000 participants via crowdsourcing on the Amazon Mechanical Turk service. The user study asks participants to assess the responsiveness of interactive application use cases for a set of current-generation applications (e.g. Angry Birds and FaceBook) and next-generation applications (i.e. face recognition and augmented reality) relative to the performance capabilities of the devices studied. This framework allows us to quantitatively link how the mobile CPU designs studied impacted end-user satisfaction. The study results indicate that mobile CPU designs have exhibited signifiant performance improvements through aggressive core scaling techniques prevalent in desktop CPUs. Just as was observed in desktop CPU design, these same techniques have lead to excessive mobile CPU power consumption. However, from an end-user perspective this power consumption was not without success. Mobile CPUs have evolved to provide satisfactory experiences for the studied current- generation applications. The reason is that many of these applications rely heavily on single-threaded performance. Other, more recent applications, actually multi-thread user-critical parts of the applications, which also demonstrates that multi- core mobile CPUs are an important design consideration – contrary to conventional wisdom. However, looking ahead, the same mobile CPUs where not able to provide satisfactory experiences for many of the next-generation applications studied, questioning the sustainability of these power-hungry design techniques in future mobile CPU designs.Item A collaborative approach to IR evaluation(2014-05) Sheshadri, Aashish; Grauman, Kristen Lorraine, 1979-; Lease, Matthew A.In this thesis we investigate two main problems: 1) inferring consensus from disparate inputs to improve quality of crowd contributed data; and 2) developing a reliable crowd-aided IR evaluation framework. With regard to the first contribution, while many statistical label aggregation methods have been proposed, little comparative benchmarking has occurred in the community making it difficult to determine the state-of-the-art in consensus or to quantify novelty and progress, leaving modern systems to adopt simple control strategies. To aid the progress of statistical consensus and make state-of-the-art methods accessible, we develop a benchmarking framework in SQUARE, an open source shared task framework including benchmark datasets, defined tasks, standard metrics, and reference implementations with empirical results for several popular methods. Through the development of SQUARE we propose a crowd simulation model that emulates real crowd environments to enable rapid and reliable experimentation of collaborative methods with different crowd contributions. We apply the findings of the benchmark to develop reliable crowd contributed test collections for IR evaluation. As our second contribution, we describe a collaborative model for distributing relevance judging tasks between trusted assessors and crowd judges. Based on prior work's hypothesis of judging disagreements on borderline documents, we train a logistic regression model to predict assessor disagreement, prioritizing judging tasks by expected disagreement. Judgments are generated from different crowd models and intelligently aggregated. Given a priority queue, a judging budget, and a ratio for expert vs. crowd judging costs, critical judging tasks are assigned to trusted assessors with the crowd supplying remaining judgments. Results on two TREC datasets show significant judging burden can be confidently shifted to the crowd, achieving high rank correlation and often at lower cost vs. exclusive use of trusted assessors.Item Creative financing & strategies for mixed-income transit oriented development in Dallas, Texas(2013-08) Partovi, Lauren Neda; Wilson, Barbara B. (Barbara Brown)This study evaluates the current environment for mixed-income transit oriented development along DART rail within the city limits of Dallas. A close look at income and racial disparity is used as the foundation for advocating for a more proactive and aggressive approach to the development of affordable units proximate to affordable transportation choices. Assembling financing for mixed-income TOD projects is especially challenging, and multiple layers of federal, state, and city funding mechanisms are required for achieving the capital requirements of the development. Both typical affordable housing funding methods and new and nontraditional funding methods for multifamily housing were researched and evaluated with the intention to propose possibilities for catalyzing development in DART station areas within the City of Dallas that have, to this point, experienced underdevelopment.Item Crowdsourcing and the law(2012-05) Wolfson, Stephen Manuel; Lease, Matthew A.; Howison, JamesWith the development and proliferation of new social and connective technologies, crowdsourcing is becoming a viable method for conducting many types of work. At the same time, however, these developments are progressing more quickly than the law and raising new legal questions that often do not have definite answers yet. This thesis address some of these legal issues that crowdsourcing raises. In this thesis, we begin by addressing four areas that might lead to legal problems in the near future. First, we look at the labor and employment law issues that might arise from online crowdlabor markets like Amazon Mechanical Turk (www.mturk.com) and oDesk (www.odesk.com). Then we discuss inventorship issues under patent law that services like InnoCentive might experience. Next, we consider how data security laws could be problematic for open innovation projects like the Netflix challenge. Finally, we explore potential intellectual property ownership problems under copyright law. After discussing these topics, this thesis then turns to examine in detail the area of crowdfunding. As the name suggests, crowdfunding refers to process of raising money through crowdsourcing. Until recently, one type of crowdfunding known as crowdfinance was largely illegal under federal securities laws. However, the law in this area is starting to change. In this chapter, we look at four different models for crowdfunding: donation, lending, reward/prepurchase, and equity investment. Following that, we consider how federal securities regulation might apply to crowdfunding, particularly the equity investment model. Finally we conduct a content analysis of three legislative proposals to create a limited exemption for crowdfunding in securities law that the U.S. Congress recently considered. Finally, we assess how crowdsourcing platforms use private contracts to bind their users to certain terms and conditions. This chapter begins with a primer on contract law. Then we examine the enforceability of standardized online agreements. Following that, we review several provisions that are common to nearly all crowdsourcing platforms. Finally, we conduct a content analysis of the specific Terms of Use contracts of several crowdsourcing platforms.Item Crowdsourcing construction of information retrieval test collections for conversational speech(2015-05) Zhou, Haofeng; Lease, Matthew A.; Wallace, ByronBuilding a test collection for an ad hoc information retrieval system on conversational speech raises new challenges for researchers. Traditional methods for building test collections are costly, and thus they are not feasible to apply to large scale conversational speech data. Constructing a large test collection on conversational speech with high quality at low cost is challenging. Crowdsourcing may represent a promising approach. Crowd workers tend to be less expensive than professional assessors, and crowd workers can work simultaneously to perform jobs on a large scale. However, despite the benefits of scale and cost, the quality of the results delivered by crowd workers may suffer. This thesis focuses on relevance judging, one of the key components of a test collection. We adopt two crowdsourcing platforms: oDesk and MTurk, use audio clips and various versions of transcripts, conduct multiple experiments under diverse settings, and analyze the results qualitatively and quantitatively. We delve into what factors influence the quality of relevance judgments on conversational speech. We also investigate differences between relevance judgements from experts and crowd workers. This thesis also describes best practices for the design of crowdsourcing tasks to improve crowd workers' performance. Ultimately, these may assist researchers in building high-quality test collections on conversational speech at low cost and scale through crowdsourcing.Item An empirical study of the effects of priming on crowdsourced evaluations of design concepts(2016-05) Pang, Michelle Audrey; Seepersad, Carolyn C.; Crawford, Richard H.As product development teams begin utilizing crowdsourcing as a means of ideation, the evaluation of large numbers of design concepts becomes a time consuming and resource intensive process that drives development activities and impacts the design of the final product created. Crowdsourcing the evaluation of design concepts has been examined in previous work as a means to reduce the demands on expert raters, while achieving similar evaluation results. In prior crowdsourcing studies, successful use of novice evaluators required detailed, in-person training that can be time and cost prohibitive. This thesis research explores the fidelity of a pairwise comparison method for evaluation that requires minimal training of novice raters. In a pilot study the pairwise method for crowdsourcing evaluations is compared with crowdsourced evaluations using non-pairwise rating scales and with the evaluations of expert raters. The analysis of pilot study responses indicates that the pairwise comparison method is a promising alternative to the other methods. Another focus of this thesis is to examine the impact of priming novice raters prior to their evaluations of alternative design concepts. A follow-on study incorporates written and empathic priming strategies to determine their impact on novice raters’ evaluation of concepts. Raters are asked to consider several criteria, including novelty, feasibility, clarity (of the concept), usefulness, ease of use, and overall worthiness of further development. Results offer insight into the criteria that are most relevant to novice raters and the effect of priming on those evaluations. Specifically, empathic priming focused on ergonomics and ease of use is shown to positively influence the raters’ emphasis on those criteria when evaluating concepts.Item Pilot study of crowdsourcing evidence-based practice research for adults with aphasia(2014-05) Rigney, Daniel Yiorgios; Chandrasekaran, BharathThe purpose of this study is to explore crowdsourcing as a research paradigm for creating evidence-based practice research in the field of speech pathology. Using an Internet survey, respondents provided de-identified information about one patient with aphasia they had treated in the previous year. The respondents were then asked to rate the success of treatment. Analysis and grading of the responses was performed to identify which responses were usable for the purpose of planning a treatment for a patient with similar demographics and diagnostic make-up. Results showed that crowdsourcing is a viable research method; however, further refinements to the collection and analysis are required before it can be an effectively used.Item Subjective and objective quality evaluation of synthetic and high dynamic range images(2016-05) Kundu, Debarati; Evans, Brian L. (Brian Lawrence), 1965-; Bovik, Alan C; Fussell, Donald S; Geisler, Wilson S; Ghosh, JoydeepRecent years have seen a huge growth in the acquisition, transmission, and storage of videos. The visual data consists of both natural scenes as well as synthetic scenes, such as animated movies, cartoons and video games. In all these cases, the ultimate goal is to provide the viewers with a satisfactory quality-of-experience. In addition to the traditional 8-bit images, high dynamic range imaging is also becoming popular because of its ability to represent the real world luminances more realistically. Coming up with objective image quality assessment algorithms for these applications is an interesting research problem. In this work, I have developed a synthetic image quality database by introducing varying degrees of different types of distortions and conducted a subjective experiment in order to obtain the ground-truth data. I evaluated the performance of state-of-the-art image quality assessment algorithms (typically meant for natural images) on this database, especially no-reference algorithms that have not been applied to the domain of computer graphics images before. I identified the top-performing algorithms along with analyzing the types of distortions on which the present algorithms show a less impressive performance. For high dynamic range(HDR) images, I have designed two new full-reference image quality assessment algorithms to judge the quality of tonemapped HDR images using statistical features extracted from them. I have also conducted a massive online crowd-sourced subjective test for HDR image artifacts arising from tonemapping, multiple-exposure fusion and post processing. To the best of our knowledge, presently this is the largest HDR image database in the world involving the largest number of source images and most number of human evaluations. Based on the subjective evaluations obtained, I have also proposed machine learning based no-reference image quality assessment algorithms to predict the perceptual quality of HDR images.Item Temporal modeling of crowd work quality for quality assurance in crowdsourcing(2015-12) Jung, Hyun Joon; Lease, Matthew A.; Mooney, Raymond; Bennett, Paul; Fleischmann, Kenneth; Wallace, Byron CWhile crowdsourcing offers potential traction on data collection at scale, it also poses new and significant quality concerns. Beyond the obvious issue of any new methodology being untested and often suffering initial growing pains, crowdsourcing has faced a very particular criticism since its inception: given anonymity of crowd workers, it is questionable whether we can trust their contributions as much as work completed by trusted workers. To relieve this concern, recent studies have proposed a variety of methods. However, while temporal behavioral patterns can be discerned to underlie real crowd work, prior studies have typically modeled worker performance under an assumption that a sequence of model variables is independent and identically distributed (i.i.d). This dissertation focuses on the measurement and prediction of crowd work quality by considering its temporal properties. To better model such temporal worker behavior, we present a time-series prediction model for crowd work quality. This model captures and summarizes past worker label quality, enabling us to better predict the quality of each worker’s next label. Further- more, we propose a crowd assessor model for predicting crowd work quality more accurately. By taking account of multi-dimensional features of a crowd assessor, we aim to build a better quality prediction model of crowd work. Finally, this dissertation explores how the proposed prediction models work under realistic scenarios. In particular, we consider a realistic use case in which limited gold labels are provided for learning our proposed model. For this problem, we leverage instance weighting with soft labels, which takes ac- count of uncertainty of each training instance. Our empirical evaluation with synthetic datasets and a public crowdsourcing dataset has shown that our pro- posed models significantly improve prediction quality of crowd work as well as lead to an acquisition of better quality labels in crowdsourcing.Item Toward a storytelling systems analysis model : a situational analysis of three global crowdsourced documentary media projects(2016-05) Moner, William Joseph; Strover, Sharon; Doty, Philip; Frick, Caroline; Stein, Laura; Straubhaar, JosephThis study investigates three participatory documentary projects that emerged in the 2011 to 2012 time period. Each project utilized crowdsourcing to generate primary source material for their respective endeavors. The projects — Life in a Day (2011), One Day on Earth (2011), and 18 Days in Egypt (2012) — are analyzed through situational analysis, a qualitative analytical framework that builds from grounded theory method, social worlds/arenas theory, and actor-network theory (ANT) to analyze the relationships between human actors, non-human actants, spatial and temporal components, and political economic factors within a situation. Using this method, I created a situational map for each documentary system, finding that each emerges from a distinct economic system where value is determined through different treatments of the “crowd” and its contributed media, data, and stories. Subsequently, using political economy of communication theory (Mosco, 2009) and the concepts of structuration, spatialization, and commodification, I identified several control mechanisms apparent in each of the projects. These control factors – commodity control, spatial control, and structural control – and their subcategories – content and labor control (commodity), technological, temporal, and circulatory control (spatial), and contractual and organizational control (structural) – draw from the analysis of three very different economic systems and storytelling intents. The study offers a preliminary framework for a participatory systems analysis approach to grapple with technological and economic concerns in shared media production spaces.Item Understanding dynamics and resource allocation in social networks(2015-08) Chatterjee, Avhishek; Vishwanath, Sriram; Baccelli, Francois; Sanghavi, Sujay; Shakkottai, Sanjay; Sirbu, Mihai; Varshney, LavWidespread popularity of various online social networks has attracted significant attention of the research community. Research interest in social networks are broadly divided into two categories: understanding the social or human network dynamics and harnessing the social network dynamics to gain economic, business or political advantage using minimal resource. These two research directions fuel each other. Better understanding offers better resource utilization/allocation in harnessing the network and the need for better resource utilization/allocation drives the fundamental research in understanding human networks. This thesis considers important problems in both directions as well as at their intersection. We first study opinion dynamics in social networks. We propose a new stochastic dynamics which generalizes two widely used and complementary models of opinion dynamics, graph-based linear dynamics and bounded confidence dynamics into a single stochastic dynamics. We analytically study the conditions under which such dynamics result in reconciliation or some sort of consensus. Our findings relate well to observed behaviors of societies. The next problem that we consider is related to designing personalized/targeted advertisements or campaigns for social network users. Currently viral marketing or campaigning rely only on the structure of the friendship graph. In reality friends may have different opinions on different topics or issues. It is understood that if opinions regarding a topic were known one could design better targeted campaigns. We propose algorithms which can infer opinions of people by observing their interactions regarding a topic or an issue. As data gathering and computation requires resources, our algorithm is designed to work with fewer such resources for a broad class of social networks and interaction patterns. A recent trend among different businesses is to work with social software providers (e.g., Lithium, Salesforce.com) to engage consumers online and often involve the online crowd directly in developing and running business ideas. This trend, popularly known as crowdsourcing uses human cloud to do jobs that cannot be done by machines. Crowdsourcing has been successfully used to do simple human tasks (Amazon Mechanical Turk), scientific research (fold.it), freelance software development(oDesk) as well as in impacting the lives of people in poverty (Samasource). Many big business houses use crowdsourcing, e.g., Microsoft, Samsung, Intel etc., IBM harness its employee pool using internal crowdsourcing. As employing humans (a.k.a. agents) for jobs, and especially for skilled jobs (like software development, scientific studies) is costly, an efficient job to agent allocation is key to the success of crowdsourcing. Motivated by this, in the last part of the thesis we study efficient resource allocation in skill-based crowdsourcing platforms.