Using scikit-learn to training an NLP log linear model for NER

Question

I wonder how to use sklearn.linear_model.LogisticRegression to train an NLP log linear model for named-entity recognition (NER).

A typical log-linear model for defines a conditional probability as follows:

with:

• x: the current word
• y: the class of a word being considered
• f: the feature vector function, which maps a word x and a class y to a vector of scalars.
• v: the feature weight vector

Can sklearn.linear_model.LogisticRegression train such a model?

The issue is that features depend on the class.

Answer 1

In scikit-learn 0.16 and higher, you can use the multinomial option for sklearn.linear_model.LogisticRegression to train a log-linear model (a.k.a. MaxEnt classifier, multiclass logistic regression). Currently the multinomial option is supported only by the ‘lbfgs’ and ‘newton-cg’ solvers.

Example with the Iris data set (4 features, 3 classes, 150 samples):

#!/usr/bin/python
# -*- coding: utf-8 -*-

from __future__ import print_function
from __future__ import division

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model, datasets
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report

# Import data 
iris = datasets.load_iris()
X = iris.data # features
y_true = iris.target # labels

# Look at the size of the feature matrix and the label vector:
print('iris.data.shape: {0}'.format(iris.data.shape))
print('iris.target.shape: {0}\n'.format(iris.target.shape))

#  Instantiate a MaxEnt model
logreg = linear_model.LogisticRegression(C=1e5, multi_class='multinomial', solver='lbfgs')

# Train the model
logreg.fit(X, y_true)
print('logreg.coef_: \n{0}\n'.format(logreg.coef_))
print('logreg.intercept_: \n{0}'.format(logreg.intercept_))

# Use the model to make predictions
y_pred = logreg.predict(X)
print('\ny_pred: \n{0}'.format(y_pred))

# Assess the quality of the predictions
print('\nconfusion_matrix(y_true, y_pred):\n{0}\n'.format(confusion_matrix(y_true, y_pred)))
print('classification_report(y_true, y_pred): \n{0}'.format(classification_report(y_true, y_pred)))

The multinomial option for sklearn.linear_model.LogisticRegression was introduced in version 0.16:

• Add multi_class="multinomial" option in :class:linear_model.LogisticRegression to implement a Logistic Regression solver that minimizes the cross-entropy or multinomial loss instead of the default One-vs-Rest setting. Supports lbfgs and newton-cg solvers. By Lars Buitinck_ and Manoj Kumar_. Solver option newton-cg by Simon Wu.