Summary of Computational Learning Theory
Kui Tang December 2012
Check before you submit. Get Turnitin Score Report in 15 Minutes.
Don't risk the 'Red' score. Get the exact same Turnitin report your professor uses. Join 50,000+ students who submitted their essays with confidence this semester.
Prepared from notes and homeworks from Professor Rocco Servedio’s Fall 2012 course in preparation for the final. No guarantees of completeness or correctness. Contents 1 Online Learning 2 1.1 Elementary Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Winnow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 (Randomized) Weighted Majority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.5 General Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 PAC Learning 5 2.1 Probability Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 OLMB to PAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Consistent Hypothesis Finder; Occam’s Razor . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.4 Hardness of 3-Term DNF Learning; Proper and Improper Learning . . . . . . . . . . . . . . . 7 2.5 VC Dimension Lower-Bounds PAC Sample Complexity . . . . . . . . . . . . . . . . . . . . . . 8 3 Bounds for Infinite Hypothesis Classes 8 3.1 Growth Functions and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Sauer’s Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3 Infinite CHF (Method of Double Sampling) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4 Boosting 10 4.1 Three-Stage Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2 Recursive Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.3 AdaBoost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5 Noisy PAC 12 5.1 Statistical Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.2 Example: Monotone Disjunctions Noisy PAC . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.3 Lower Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.3.1 Sample Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.3.2 Noise Rate and Adversarial Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 6 Representation-Independent (Cryptographic) Hardness 13 6.1 Pseudorandom Functions are Hard to Learn . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.2 Discrete Cube Roots are Hard to Learn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1
Background image of page 1
Info iconThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentRight Arrow Icon
1 Online Learning 1.1 Elementary Algorithms To learn a one-decision list, use extended decision (multiple rules; if multiple antecedents are satisfied, arbitrarily choose one) lists are your hypotheses. Begin with all 4 n + 2 rules. Each time a rule fires and returns a wrong answer, such that that rule is responsible for the error, we push the rule down to the next level. A rule is pushed iff a mistake is made (show the converse side), so we have at most O ( n ) mistakes. Decision lists can be encoded as LTFs with exponentially decaying weights. Recall that a matching rule short-circuits the rest of the decision list evaluation. In an LTF with exponentially-decaying weights, the first matching rule overwhelm all subsequent coefficients [HW1.1b]. [HW1.2] Any OLMB algorithm can be made conservative: Suppose algorithm A has mistake bound M . Algorithm A0 predicts with its current hypothesis. If it turned out to be a mistake, A0 gives the current example to A and takes A0 s output hypothesis. The two algorithms make the same number of mistakes on the subsequence that causes mistakes. This is because A only sees this sequence of mistake examples. Therefore, A0 can make no more mistakes than A itself would have. Randomized halving algorithm: if we remove the oblivious adversary, a more efficient algorithm (in expectation) can be obtained: maintain the set of consistent-so-far hypothesis, and on each mistake, pick h uniformly at random from the new set of consistent hypotheses. Then for any fixed target c and training sequence ± x i ² (this is the oblivious adversary assumption), we have E [ M ] ≤ ln |C| + O (1) . Proof idea: order the concepts in CONSIST by the order in which the fixed training sequence will eliminate them
Welcome to our trusted essay writing website with track record among students. We specialize in connecting students in need of high-quality essay assistance with skilled writers who can deliver just that. Explore the ratings of our essay writers and choose the one that best aligns with your requirements. When you rely on our online essay writing service, rest assured that you will receive a top-notch, plagiarism-free A-level paper. Our experienced professionals write each paper from scratch, carefully following your instructions. Request a paper from us and experience 100% originality.


