Browsing by Subject "Active learning"
Now showing 1 - 12 of 12
Results Per Page
Sort Options
Item Active learning in cost-sensitive environments(2009-12) Liu, Alexander Yun-chung; Ghosh, JoydeepActive learning techniques aim to reduce the amount of labeled data required for a supervised learner to achieve a certain level of performance. This can be very useful in domains where unlabeled data is easy to obtain but labelling data is costly. In this dissertation, I introduce methods of creating computationally efficient active learning techniques that handle different misclassification costs, different evaluation metrics, and different label acquisition costs. This is accomplished in part by developing techniques from utility-based data mining typically not studied in conjunction with active learning. I first address supervised learning problems where labeled data may be scarce, especially for one particular class. I revisit claims about resampling, a particularly popular approach to handling imbalanced data, and cost-sensitive learning. The presented research shows that while resampling and cost-sensitive learning can be equivalent in some cases, the two approaches are not identical. This work on resampling and cost-sensitive learning motivates a need for active learners that can handle different misclassification costs. After presenting a cost-sensitive active learning algorithm, I show that this algorithm can be combined with a proposed framework for analyzing evaluation metrics in order to create an active learning approach that can optimize any evaluation metric that can be expressed as a function of terms in a confusion matrix. Finally, I address methods for active learning in terms of different utility costs incurred when labeling different types of points, particularly when label acquisition costs are spatially driven.Item Active learning module assessment and the development and testing of a new prototyping planning tool(2014-08) Dunlap, Brock Usher; Crawford, Richard H.This thesis contains the research findings from my participation in two research projects. The first is the development and assessment of Active Learning Modules (ALMs) for engineering students. The ALMs assist students in learning complex Finite Element Analysis (FEA) principles. We measure the effectiveness of the modules by issuing pre- and post-module quizzes and analyze the differences of the quiz scores. Active learning modules are used to meet the needs of all students’ learning styles. Each student who uses an ALM takes a series of learning style assessment quizzes (MBTI, LIS …). We statistically compare the learning styles and quiz scores to ensure all learning styles are improving equally well. In cases where they are not, we created a tool to make suggestions to the ALM developer on how to adjust the ALM to meet the needs of the outlying learning style group(s). Following modification, the implementation and evaluation process of the ALM is repeated. My second area of research focused on the development of a concise prototype strategy development tool. This tool guides engineering product development teams through six critical prototype strategy choices: (1) How many concepts should be prototyped? (2) How many iterations of a concept should be built? (3) Should the prototype be virtual or physical? (4) Should subsystems be isolated? (5) Should the prototype be scaled? (6) Should the design requirements be temporarily relaxed? This list of choices is not comprehensive but served as a starting point for this groundbreaking research. The tool was tested at The University of Texas at Austin and the United States Air Force Academy. Results indicate the method did improve students’ performance across a number of assessment metrics.Item Active learning of an action detector on untrimmed videos(2013-05) Bandla, Sunil; Grauman, Kristen Lorraine, 1979-Collecting and annotating videos of realistic human actions is tedious, yet critical for training action recognition systems. We propose a method to actively request the most useful video annotations among a large set of unlabeled videos. Predicting the utility of annotating unlabeled video is not trivial, since any given clip may contain multiple actions of interest, and it need not be trimmed to temporal regions of interest. To deal with this problem, we propose a detection-based active learner to train action category models. We develop a voting-based framework to localize likely intervals of interest in an unlabeled clip, and use them to estimate the total reduction in uncertainty that annotating that clip would yield. On three datasets, we show our approach can learn accurate action detectors more efficiently than alternative active learning strategies that fail to accommodate the "untrimmed" nature of real video data.Item Active visual category learning(2011-05) Vijayanarasimhan, Sudheendra; Grauman, Kristen Lorraine, 1979-; Dhillon, Inderjit S.; Aggarwal, J K.; Mooney, Raymond J.; Torralba, AntonioVisual recognition research develops algorithms and representations to autonomously recognize visual entities such as objects, actions, and attributes. The traditional protocol involves manually collecting training image examples, annotating them in specific ways, and then learning models to explain the annotated examples. However, this is a rather limited way to transfer human knowledge to visual recognition systems, particularly considering the immense number of visual concepts that are to be learned. I propose new forms of active learning that facilitate large-scale transfer of human knowledge to visual recognition systems in a cost-effective way. The approach is cost-effective in the sense that the division of labor between the machine learner and the human annotators respects any cues regarding which annotations would be easy (or hard) for either party to provide. The approach is large-scale in that it can deal with a large number of annotation types, multiple human annotators, and huge pools of unlabeled data. In particular, I consider three important aspects of the problem: (1) cost-sensitive multi-level active learning, where the expected informativeness of any candidate image annotation is weighed against the predicted cost of obtaining it in order to choose the best annotation at every iteration. (2) budgeted batch active learning, a novel active learning setting that perfectly suits automatic learning from crowd-sourcing services where there are multiple annotators and each annotation task may vary in difficulty. (3) sub-linear time active learning, where one needs to retrieve those points that are most informative to a classifier in time that is sub-linear in the number of unlabeled examples, i.e., without having to exhaustively scan the entire collection. Using the proposed solutions for each aspect, I then demonstrate a complete end-to-end active learning system for scalable, autonomous, online learning of object detectors. The approach provides state-of-the-art recognition and detection results, while using minimal total manual effort. Overall, my work enables recognition systems that continuously improve their knowledge of the world by learning to ask the right questions of human supervisors.Item Bayesian learning methods for neural coding(2013-12) Park, Mi Jung; Pillow, Jonathan W.; Bovik, Alan C. (Alan Conrad), 1958-A primary goal in systems neuroscience is to understand how neural spike responses encode information about the external world. A popular approach to this problem is to build an explicit probabilistic model that characterizes the encoding relationship in terms of a cascade of stages: (1) linear dimensionality reduction of a high-dimensional stimulus space using a bank of filters or receptive fields (RFs); (2) a nonlinear function from filter outputs to spike rate; and (3) a stochastic spiking process with recurrent feedback. These models have described single- and multi-neuron spike responses in a wide variety of brain areas. This dissertation addresses Bayesian methods to efficiently estimate the linear and non-linear stages of the cascade encoding model. In the first part, the dissertation describes a novel Bayesian receptive field estimator based on a hierarchical prior that flexibly incorporates knowledge about the shapes of neural receptive fields. This estimator achieves error rates several times lower than existing methods, and can be applied to a variety of other neural inference problems such as extracting structure in fMRI data. The dissertation also presents active learning frameworks developed for receptive field estimation incorporating a hierarchical prior in real-time neurophysiology experiments. In addition, the dissertation describes a novel low-rank model for the high dimensional receptive field, combined with a hierarchical prior for more efficient receptive field estimation. In the second part, the dissertation describes new models for neural nonlinearities using Gaussian processes (GPs) and Bayesian active learning algorithms in closed-loop neurophysiology experiments to rapidly estimate neural nonlinearities. The dissertation also presents several stimulus selection criteria and compare their performance in neural nonlinearity estimation. Furthermore, the dissertation presents a variation of the new models by including an additional latent Gaussian noise source, to infer the degree of over-dispersion in neural spike responses. The proposed model successfully captures various mean-variance relationships in neural spike responses and achieves higher prediction accuracy than previous models.Item Collaborative information acquisition(2011-12) Kong, Danxia; Saar-Tsechansky, Maytal; Mooney, Raymond; McAlister, Leigh; Konana, Prabhudev; Whinston, Andrew; Shen, Zuo-Jun MaxIncreasingly, predictive models are used to support routine business de- cisions and are integral to the strategic competitive business strategies for a wide range of industries. Most often, data-driven predictive models are in- duced from training data obtained through the businesss routine operations. However, recent research on policies for intelligent information acquisitions suggests that proactive acquisition of information can improve models at a lower cost. Most active information acquisition policies are accuracy centric; they aim to identify acquisitions of training data that are particularly benefi- cial for improving the predictive accuracy of a given model. In practice, however, inferences from a predictive model are often used along with inferences from other predictive models as well as constant factors to inform arbitrarily complex decisions. In this dissertation, I discuss how these settings motivate a new kind of collaborative information acquisition (CIA) policies that exploit knowledge of the decision to allow multiple predictive models to collaboratively prioritize the prospective information acquisitions, so as to best improve the decisions they inform jointly. I present a framework for CIA policies and two specific CIA policies: CIA for binary decisions (CIA-BD), and CIA for top-ranked opportu- nities in terms of expected revenue (CIA-TR). Extensive empirical evaluations of the policies on real-world data suggest that the notion of CIA policies is indeed a valuable one. In particular, I demonstrate that these two new poli- cies lead to superior decision-making performances as compared to those of alternative policies that are either decision-centric or do not allow multiple models to collaboratively prioritize acquisitions. The performance exhibited by the CIA policies suggest that these policies are able to effectively exploit knowledge of the decisions to avoid greedy improvements in accuracy of any individual model informing the decisions; instead, they promote improvements in any one or all of the models when such improvements are likely to benefit the decisions.Item Experiences and engagement levels of entering community college students and returning students(2008-12) De los Reyes, Maria Oralia; Roueche, John E.In order to explore the differences in engagement levels between entering and returning community college students, the researcher analyzed 13,300 surveys from the 2007 Survey of Entering Student Engagement (SENSE) pilot data set utilizing a quantitative methodology. This study focused on analyzing engagement levels of entering and returning students in six constructs: Active and Collaborative Learning, First Day, Student Effort, Student-faculty Interaction, Support for Learners, and Motivation. After the comparison between the two groups was performed, data were disaggregated by eleven groups to further explore differences. Differences in engagement levels were explored in terms of students’ characteristics such as remedial background, age, gender, full-time status, ethnicity, degree seeking, first generation, and traditional status. The results of this study revealed that returning community college students in general, are more engaged in educational practices associated with persistence than entering students. In addition, findings suggest that with the exception of one variable (overall high school grade average), students commonly categorized as “at risk” or “disadvantaged” (developmental, non-traditional, part-time, first generation, minorities) overwhelmingly held higher levels of engagement in all positive engagement variables of the six analyzed constructs. Furthermore, in an analysis of disaggregated data by eleven groups of students, the following was found: o Students who placed in three developmental courses were by far the most highly engaged group in all positive engagement variables of the six constructs. o Students with the highest level of engagement in the two negative variables of the Student Effort construct (skipped class or came to class without completing readings or assignments) were traditional, 18-19 year olds, not-first generation, and non-developmental students. o Students who reported that success courses had helped them to get the knowledge necessary to succeed in college were overwhelmingly disadvantaged students (developmental, non-traditional, females and minorities). o Developmental students showed higher levels of engagement with regard to college services. o Financial aid advising and skill labs (math, reading, and writing) were the two services with the highest number of statistically significant differences throughout the eleven groups. This study was concluded with recommendations for further research and strategies that community college stakeholders could implement to increase student retention.Item Fostering active learning through the use of feedback technologies and collaborative activities in a postsecondary setting(2010-05) Guerrero, Camilo; Robinson, Daniel H.; Borich, Gary D.; Katayama, Andrew D.; Svinicki, Marilla D.; Vaughn, Brandon K.Technology is enjoying an increasingly important role in many collegiate pedagogical designs. Contemporary research has become more focused on the ways that technology can contribute to learning outcomes. These studies provide a critical foundation for educational researchers who seek to incorporate and reap the benefits of new technologies in classroom environments. The aim of the present study is to empirically assess how combining an active, collaborative learning environment with a classroom response system (colloquially called “clickers”) in a postsecondary setting can influence and improve learning outcomes. To this end, the study proposes an instructional design utilizing two feedback response-formats (clickers and flashcards) and two response methods for answering in-class questions (collaborative peer instruction and individual). The theoretical bases that provide the academic structure for the five instructional conditions (control, clicker-response individual, clicker-response peer instruction, flashcard-response individual, and flashcard-response peer instruction) are the generative learning theory and social constructivism. Participants were 171 undergraduate students from an Educational Psychology subject pool from a large Southwest university. The researcher used a two-way analysis of covariance (ANCOVA) with two treatments (response format and collaboration level) as the between-subjects factors; students’ posttest scores as the dependent variable; and pretest scores as the covariate. Results showed no significant main effects; however, the study produced statistically significant findings that there was an interaction effect between the use of clickers and a peer instruction design. To follow up the interaction, the researcher conducted tests of the simple effects of response format within each collaboration condition, with the pretest as the covariate. Results showed that for students who collaborated, clickers were better than flashcards, whereas when students worked individually, there was no difference. This study builds upon existing studies by using a stronger empirical approach with more robust controls to evaluate the effects of a variety of instructional interventions, clicker and flashcard response systems and peer instruction on learning outcomes. It shows that clicker technology might be most effective when combined with collaborative methods. The discussion includes implications, limitations, and directions for future research.Item Interactive image search with attributes(2014-08) Kovashka, Adriana Ivanova; Grauman, Kristen Lorraine, 1979-An image retrieval system needs to be able to communicate with people using a common language, if it is to serve its user's information need. I propose techniques for interactive image search with the help of visual attributes, which are high-level semantic visual properties of objects (like "shiny" or "natural"), and are understandable by both people and machines. My thesis explores attributes as a novel form of user input for search. I show how to use attributes to provide relevance feedback for image search; how to optimally choose what to seek feedback on; how to ensure that the attribute models learned by a system align with the user's perception of these attributes; how to automatically discover the shades of meaning that users employ when applying an attribute term; and how attributes can help learn object category models. I use attributes to provide a channel on which the user of an image retrieval system can communicate her information need precisely and with as little effort as possible. One-shot retrieval is generally insufficient, so interactive retrieval systems seek feedback from the user on the currently retrieved results, and adapt their relevance ranking function accordingly. In traditional interactive search, users mark some images as "relevant" and others as "irrelevant", but this form of feedback is limited. I propose a novel mode of feedback where a user directly describes how high-level properties of retrieved images should be adjusted in order to more closely match her envisioned target images, using relative attribute feedback statements. For example, when conducting a query on a shopping website, the user might state: "I want shoes like these, but more formal." I demonstrate that relative attribute feedback is more powerful than traditional binary feedback. The images believed to be most relevant need not be most informative for reducing the system's uncertainty, so it might be beneficial to seek feedback on something other than the top-ranked images. I propose to guide the user through a coarse-to-fine search using a relative attribute image representation. At each iteration of feedback, the user provides a visual comparison between the attribute in her envisioned target and a "pivot" exemplar, where a pivot separates all database images into two balanced sets. The system actively determines along which of multiple such attributes the user's comparison should next be requested, based on the expected information gain that would result. The proposed attribute search trees allow us to limit the scan for candidate images on which to seek feedback to just one image per attribute, so it is efficient both for the system and the user. No matter what potentially powerful form of feedback the system offers the user, search efficiency will suffer if there is noise on the communication channel between the user and the system. Therefore, I also study ways to capture the user's true perception of the attribute vocabulary used in the search. In existing work, the underlying assumption is that an image has a single "true" label for each attribute that objective viewers could agree upon. However, multiple objective viewers frequently have slightly different internal models of a visual property. I pose user-specific attribute learning as an adaptation problem in which the system leverages any commonalities in perception to learn a generic prediction function. Then, it uses a small number of user-labeled examples to adapt that model into a user-specific prediction function. To further lighten the labeling load, I introduce two ways to extrapolate beyond the labels explicitly provided by a given user. While users differ in how they use the attribute vocabulary, there exist some commonalities and groupings of users around their attribute interpretations. Automatically discovering and exploiting these groupings can help the system learn more robust personalized models. I propose an approach to discover the latent factors behind how users label images with the presence or absence of a given attribute, from a sparse label matrix. I then show how to cluster users in this latent space to expose the underlying "shades of meaning" of the attribute, and subsequently learn personalized models for these user groups. Discovering the shades of meaning also serves to disambiguate attribute terms and expand a core attribute vocabulary with finer-grained attributes. Finally, I show how attributes can help learn object categories faster. I develop an active learning framework where the computer vision learning system actively solicits annotations from a pool of both object category labels and the objects' shared attributes, depending on which will most reduce total uncertainty for multi-class object predictions in the joint object-attribute model. Knowledge of an attribute's presence in an image can immediately influence many object models, since attributes are by definition shared across subsets of the object categories. The resulting object category models can be used when the user initiates a search via keywords such as "Show me images of cats" and then (optionally) refines that search with the attribute-based interactions I propose. My thesis exploits properties of visual attributes that allow search to be both effective and efficient, in terms of both user time and computation time. Further, I show how the search experience for each individual user can be improved, by modeling how she uses attributes to communicate with the retrieval system. I focus on the modes in which an image retrieval system communicates with its users by integrating the computer vision perspective and the information retrieval perspective to image search, so the techniques I propose are a promising step in closing the semantic gap.Item Multiple-instance active learning with online labeling(2013-08) Salmani, Kimia; Sridharan, Mohan; Watson, RichardRobots are designed to automate a variety of processes. However, partial observability and non-determinism in the real world make it expensive for robots to operate automatically. Therefore, in order to fulfill their goal, robots need to be trained using multi-modal sensory cues such as visual and verbal. In this thesis, we move towards incremental learning for robots using the minimum human intervention. The existing framework, called multiple instance active learning, uses a defined static set of unlabeled data to be queried. Our contribution is to make it happen for the trainer to query dynamic unlabeled data by proposing a new active learning method, called Bag Uncertainty. In Bag Uncertainty method, a query is asked when the robot is uncertain regarding a recently entered therefore unlabeled data. This method abet the robot to boost its knowledge regarding a particular object category with adding unseen aspects of it to the trained model. If the human feedback is available, it is answered right away; however, if no human feedback is available, the questions are stored and will be asked with respect to the level of their uncertainty. A query in accordance with an image with highest level of uncertainty will be given high priority to be asked. The queries are asked in a verbal format and the corresponding response is processed with lexical tools to extract verbal features. The particular instance is addressed by extracted feature vector and graphical measures. We show the experimental results by applying this algorithm on a set of natural scenes and object categories selected from a database known as IAPR TC-12 with altered feature vectors regarding each image and its instances.Item A new wave in engineering education: understanding the beat of active learning through innovative tutorial assessment(2009-12) Kaufman, Kristen Kay; Wood, Kristin L.; Crawford, Richard H.Recent efforts in engineering education research have set in motion reform advocating more active learning in the classroom. Active learning centers on the student and consists of pedagogical approaches to address the broad spectrum of educational backgrounds and demographics. In order to further the research focused on active learning products, appropriate and innovative assessment methods must be developed. For this thesis, innovative active learning modules are the focus of the analysis. In total, 12 Finite Element tutorials are designed and assessed using both statistical analysis and confidence interval correlations. Fundamental and informative assessment strategies have been developed to iteratively improve active learning approaches. Results of this process show that the finite element tutorials lead to enhanced student learning that can span across student demographics. Certain cases do exist where unique learning styles or personality types respond more positively to this pedagogical technique than others. Global outcomes are presented to assess these tutorials cumulatively, as active learning products. Finally, the assessment methodology is redesigned into a useful toolkit for educators to follow in furthering efforts of integrating active learning into any engineering classroom.Item Semi-automated annotation and active learning for language documentation(2009-12) Palmer, Alexis Mary; Baldridge, Jason; Erk, Katrin; England, Nora; Mooney, Raymond; Woodbury, AnthonyBy the end of this century, half of the approximately 6000 extant languages will cease to be transmitted from one generation to the next. The field of language documentation seeks to make a record of endangered languages before they reach the point of extinction, while they are still in use. The work of documenting and describing a language is difficult and extremely time-consuming, and resources are extremely limited. Developing efficient methods for making lasting records of languages may increase the amount of documentation achieved within budget restrictions. This thesis approaches the problem from the perspective of computational linguistics, asking whether and how automated language processing can reduce human annotation effort when very little labeled data is available for model training. The task addressed is morpheme labeling for the Mayan language Uspanteko, and we test the effectiveness of two complementary types of machine support: (a) learner-guided selection of examples for annotation (active learning); and (b) annotator access to the predictions of the learned model (semi-automated annotation). Active learning (AL) has been shown to increase efficacy of annotation effort for many different tasks. Most of the reported results, however, are from studies which simulate annotation, often assuming a single, infallible oracle. In our studies, crucially, annotation is not simulated but rather performed by human annotators. We measure and record the time spent on each annotation, which in turn allows us to evaluate the effectiveness of machine support in terms of actual annotation effort. We report three main findings with respect to active learning. First, in order for efficiency gains reported from active learning to be meaningful for realistic annotation scenarios, the type of cost measurement used to gauge those gains must faithfully reflect the actual annotation cost. Second, the relative effectiveness of different selection strategies in AL seems to depend in part on the characteristics of the annotator, so it is important to model the individual oracle or annotator when choosing a selection strategy. And third, the cost of labeling a given instance from a sample is not a static value but rather depends on the context in which it is labeled. We report two main findings with respect to semi-automated annotation. First, machine label suggestions have the potential to increase annotator efficacy, but the degree of their impact varies by annotator, with annotator expertise a likely contributing factor. At the same time, we find that implementation and interface must be handled very carefully if we are to accurately measure gains from semi-automated annotation. Together these findings suggest that simulated annotation studies fail to model crucial human factors inherent to applying machine learning strategies in real annotation settings.