IR-Lab Project of Yanjun Qi

Compare the three semi-supervised methods and SVM’s experimental performance

· We make a comparison for the three semi-supervised methods we tried.

i. Transductive Support Vector Machine Experimental Results

ii. Harmonic Function Experimental Results

iii. Semi- MLE – EM Experimental Results

· Each methods were tested on 7 UCI data sets :

i. For each data set, there are various labeled set sizes to be tested: {5, 10, 20, 30, 40, 60, 80, 100}. In each case, we compared the performance of SVM transductive and SVM inductive.

ii. For each labeled set size l tested, we perform 10 trials. In each trial, we randomly sample labeled data from the entire dataset, and use a fixed number of items from the rest items as unlabeled data, see data sets for details. If any class is absent from the sampled labeled set, we redo the sampling.

· We use error rate, balanced error rate and AUC area to measure the performance of each case (averaged over 10 trials).

F Balanced error rate (BER = the average of the error rate on positive class examples and the error rate on negative class examples).

If there are fewer positive examples, the errors on positive examples will count more.

F Error rate: errors on positive and negative examples are penalized in the same way.

F The area under the ROC curve (AUC)

· The methods are roughly perform on each data set like the following table:

Data Set	Balanced Error	Error Rate	AUC Score
Data 1	SVM-transductive > Harmonic > SVM > EM-MLE	Harmonic > SVM-Trans > SVM = EM	Harmonic > SVM-transductive ~> SVM > EM-MLE
Data 2	Harmonic > SVM-transductive ~> SVM > EM-MLE	Harmonic > SVM-transductive ~> SVM > EM-MLE	Harmonic > SVM-transductive ~> SVM > EM-MLE
Data 3	Harmonic > SVM-transductive ~> SVM > EM-MLE	Harmonic > SVM ~> SVM- transductive > EM-MLE	Harmonic ~> SVM ~> SVM- transductive > EM-MLE
Data 4	SVM- transductive > Harmonic > SVM > EM-MLE	Harmonic > SVM > EM-MLE > ~SVM- transductive	Harmonic > SVM ~> SVM- transductive > EM-MLE
Data 5	SVM-transductive > SVM ~> Hamonic ~> EM-MLE	SVM-Transductive ~> SVM ~> Harmonic ~> EM-MLE	SVM-Transductive ~> SVM > Harmonic > EM-MLE
Data 6	Harmonic ~> EM-MLE > SVM-Transductive ~> SVM	SVM ~> EM-MLE ~> Harmonic > SVM-Transductive	SVM > EM-MLE > Harmonic > SVM transductive
Data 7	Harmonic > SVM transductive > SVM > EM-MLE	Harmonic > SVM transductive > SVM > EM-MLE	Harmonic ~> SVM transductive ~> SVM >> EM-MLE

- Harmonic and TransductiveSVM perform much better than the EM-MLE method

- TransductiveSVM gives a little help compared to the SVM itself by using the unlabeled data

- Harmonic method overall perform better than TransductiveSVM, much better than EM-MLE.

- The reason for bad performance of EM-MLE

o Compared to the small train set, too many unlabeled data have very big effect on the total data log likelihood

o The covariance matrix is hard to get when too small label set. Must take some ways to reduce the effect of this problem.

- From these performance experiments, we could see that unlabeled data do help in the small train set case somehow.

o But it also happens that sometimes using the unlabeled data degrades the performance of the classification

- From the results on these date set with different class ratio, I feel that the imbalanced distribution is not the main problem for a concrete classification task. If some classification task is found to perform badly under some imbalance distribution, the problem would be most likely caused by the small training set’ size problem.