第三次实现Logistic Regression(c++)_接口
看《我是歌手》第三期回放,张宇唱得实在太烂,还是回来写会儿blog吧。
1. 初衷
前两次实现,第一次的输入特征限于0-1特征,第二次限于实值特征,两者混用的还没尝试过,这次也不打算尝试。前面实现的都是二分类问题,在现实生活中,最经常遇到的还是多分类问题。由二分类器转成多分类器有两种方法:one vs all,or one vs one。前者,对于K个类别,建立K个分类器,每个分类器区分当前类别和其余类别。后者,K个类别需要K(K-1)个分类器,分别区分每两个类别。工程实践中,相信很少有人好事儿选择后者。
这两天再次学习李航老师的《统计学习方法》,翻看逻辑回归那部分章节。书中简略的提到了逻辑回归在多分类上的公式。所以,手痒,动手实现一个。
2. 原理
对于二分类LR模型,最后用sigmoid函数来计算当前类别的概率,sigmoid函数公式为:f(x) = 1.0 / (1.0 + exp(-x))。在实际计算中,为了提升计算精度,往往用他的变形:f(x) = exp(x) / (1.0 + exp(x))。
对于多分类问题,李航老师书中给出的概率计算函数为:f_i(x) = exp(x_i) / (1.0 + sum_j (exp(x_j)) )。大家凑合着看,带有‘_i’或者‘_j’的都是表示下标。
对于SGD的权重更新公式,我没有具体推导。从代码实现上看,在从前的代码上稍作改动即可。
3. 参数
在处理二分类的时候,每一个特征有一个权重,权重形成一个权重向量。在处理K分类的时候,对于每一个特征,在每一个分类中都有一个权重,特征+分类,其权重形成一个权重矩阵。当特征数为N的时候,按刚才描述,权重矩阵大小为K * N。不过,1 * N能处理二分类问题;则(K-1)*N就能处理K类问题,只要增加一个default分类即可。不知道说明白没有,代码如下:
private: // the number of target class int iClassNum; // the number of feature int iFeatureNum; // the theta matrix of iMaxFeatureNum * (iClassNum - 1) // note: for binary class, we need only 1 vector of theta; for multi-class, // iMaxFeatureNum * (iClassNum - 1) is always enough vector< vector<double> > ThetaMatrix;
4. 整体接口
整体函数接口,定义在LogisticRegression.h文件中,如下:
/*********************************************************************************** * Logistic Regression classifier version 0.03 * Implemented by Jinghui Xiao ([email protected] or [email protected]) * Last updated on 2014-1-17 ***********************************************************************************/ #pragma once #include <vector> #include <fstream> #include <iostream> #include <iterator> #include <sstream> #include <algorithm> #include <cmath> using namespace std; // The represetation for a feature and its value, init with ‘-1‘ class FeaValNode { public: int iFeatureId; double dValue; FeaValNode (void); ~FeaValNode (void); }; // The represetation for a sample class Sample { public: // the class index for a sample: 0-1 value, init with ‘-1‘ int iClass; vector<FeaValNode> FeaValNodeVec; Sample (void); ~Sample (void); }; // the minimal float number for smoothing for scaling the input samples #define SMOOTHFATOR 1e-100 // The logistic regression classifier for MULTI-classes class LogisticRegression { public: LogisticRegression(void); ~LogisticRegression(void); // scale all of the sample values and put the result into txt bool ScaleAllSampleValTxt (const char * sFileIn, int iFeatureNum, const char * sFileOut); // train by SGD on the sample file bool TrainSGDOnSampleFile ( const char * sFileName, int iClassNum, int iFeatureNum, // about the samples double dLearningRate, // about the learning int iMaxLoop, double dMinImproveRatio // about the stop criteria ); // train by SGD on the sample file, decreasing dLearningRate during loop bool TrainSGDOnSampleFileEx ( const char * sFileName, int iClassNum, int iFeatureNum, // about the samples double dLearningRate, // about the learning int iMaxLoop, double dMinImproveRatio // about the stop criteria ); // save the model to txt file: the theta matrix with its size bool SaveLRModelTxt (const char * sFileName); // load the model from txt file: the theta matrix with its size bool LoadLRModelTxt (const char * sFileName); // load the samples from file, predict by the LR model bool PredictOnSampleFile (const char * sFileIn, const char * sFileOut, const char * sFileLog); // just for test void Test (void); private: // read a sample from a line, return false if fail bool ReadSampleFrmLine (string & sLine, Sample & theSample); // load all of the samples into sample vector, this is for scale samples bool LoadAllSamples (const char * sFileName, vector<Sample> & SampleVec); // initialize the theta matrix with iClassNum and iFeatureNum bool InitThetaMatrix (int iClassNum, int iFeatureNum); // calculate the model function output for iClassIndex by feature vector double CalcFuncOutByFeaVec (vector<FeaValNode> & FeaValNodeVec, int iClassIndex); // calculate the model function output for all the classes, and return the class index with max probability int CalcFuncOutByFeaVecForAllClass (vector<FeaValNode> & FeaValNodeVec, vector<double> & ClassProbVec); // calculate the gradient and update the theta matrix, it returns the cost double UpdateThetaMatrix (Sample & theSample, vector<double> & ClassProbVec, double dLearningRate); // predict the class for one single sample int PredictOneSample (Sample & theSample); private: // the number of target class int iClassNum; // the number of feature int iFeatureNum; // the theta matrix of iMaxFeatureNum * (iClassNum - 1) // note: for binary class, we need only 1 vector of theta; for multi-class, // iMaxFeatureNum * (iClassNum - 1) is always enough vector< vector<double> > ThetaMatrix; };
增加了一个样本scale函数,用来处理训练和测试样本。增加了另一个SGD训练函数,区别在于学习率随着迭代逐渐衰减。函数实现在LogisticRegression.cpp中,见后续博文。
转载请注明出处:http://blog.csdn.net/xceman1997/article/details/18426073
郑重声明:本站内容如果来自互联网及其他传播媒体,其版权均属原媒体及文章作者所有。转载目的在于传递更多信息及用于网络分享,并不代表本站赞同其观点和对其真实性负责,也不构成任何其他建议。