Skip to content
This repository has been archived by the owner on Apr 19, 2021. It is now read-only.

Framework to learn Named Entity Recognition models without labelled data using weak supervision.

Notifications You must be signed in to change notification settings

NorskRegnesentral/weak-supervision-for-NER

Repository files navigation

Weak supervision for NER

BIG FAT WARNING: This codebase is now deprecated and has been replaced by our brand-new skweak framework, please check it out!

Source code associated with the paper "Named Entity Recognition without Labelled Data: a Weak Supervision Approach" accepted to ACL 2020.

Requirements:

You should first make sure that the following Python packages are installed:

  • spacy (version >= 2.2)
  • hmmlearn
  • snips-nlu-parsers
  • pandas
  • numba
  • scikit-learn

You should also install the en_core_web_sm and en_core_web_md models in Spacy.

To run the neural models in ner.py, you need also need pytorch, cupy, keras and tensorflow installed.

To run the baselines, you will also need to have snorkel installed.

Finally, you also need to download the following files and add them to the data directory:

Quick start

You should first convert your corpus to Spacy DocBin format.

Then, to run all labelling functions on your corpus, you can simply:

import annotations
annotator = annotations.FullAnnotator().add_all()
annotator.annotate_docbin('path_to_your_docbin_corpus')

You can then estimate an HMM model that aggregates all sources:

import labelling
hmm = labelling.HMMAnnotator()
hmm.train('path_to_your_docbin_corpus')

And run it on your corpus to get the aggregated labels:

hmm.annotate_docbin('path_to_your_docbin_corpus')

Step-by-step instructions

More detailed instructions with a step-by-step example are available in the Jupyter Notebook Weak Supervision.ipynb. Don't forget to run it using Jupyter to get the visualisation for the NER annotations.