Skip to content

Retrieve PubMed articles, text-mining annotations, or molecular data from >35 Entrez databases via easy to use Python package - built on top of Entrez E-utilities API.

License

Notifications You must be signed in to change notification settings

krassowski/easy-entrez

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

easy-entrez

Tests CodeQL Documentation Status DOI Python

Python REST API for Entrez E-Utilities, aiming to be easy to use and reliable.

Easy-entrez:

  • makes common tasks easy thanks to simple Pythonic API,
  • is typed and integrates well with mypy,
  • is tested on Windows, Mac and Linux across Python 3.7 to 3.12,
  • is limited in scope, allowing to focus on the reliability of the core code,
  • does not use the stateful API as it is error-prone as seen on example of the alternative entrezpy.

Examples

from easy_entrez import EntrezAPI

entrez_api = EntrezAPI(
    'your-tool-name',
    'e@mail.com',
    # optional
    return_type='json'
)

# find up to 10 000 results for cancer in human
result = entrez_api.search('cancer AND human[organism]', max_results=10_000)

# data will be populated with JSON or XML (depending on the `return_type` value)
result.data

See more in the Demo notebook and documentation.

For a real-world example (i.e. used for this publication) see notebooks in multi-omics-state-of-the-field repository.

Fetching genes for a variant from dbSNP

Fetch the SNP record for rs6311:

rs6311 = entrez_api.fetch(['rs6311'], max_results=1, database='snp').data[0]
rs6311

Display the result:

from easy_entrez.parsing import xml_to_string

print(xml_to_string(rs6311))

Find the gene names for rs6311:

namespaces = {'ns0': 'https://www.ncbi.nlm.nih.gov/SNP/docsum'}
genes = [
    name.text
    for name in rs6311.findall('.//ns0:GENE_E/ns0:NAME', namespaces)
]
print(genes)

['HTR2A']

Fetch data for multiple variants at once:

result = entrez_api.fetch(['rs6311', 'rs662138'], max_results=10, database='snp')
gene_names = {
    'rs' + document_summary.get('uid'): [
        element.text
        for element in document_summary.findall('.//ns0:GENE_E/ns0:NAME', namespaces)
    ]
    for document_summary in result.data
}
print(gene_names)

{'rs6311': ['HTR2A'], 'rs662138': ['SLC22A1']}

Obtaining the chromosomal position from SNP rsID number

from pandas import DataFrame

result = entrez_api.fetch(['rs6311', 'rs662138'], max_results=10, database='snp')

variant_positions = DataFrame([
    {
        'id': 'rs' + document_summary.get('uid'),
        'chromosome': chromosome,
        'position': position
    }
    for document_summary in result.data
    for chrom_and_position in document_summary.findall('.//ns0:CHRPOS', namespaces)
    for chromosome, position in [chrom_and_position.text.split(':')]
])

variant_positions
id chromosome position
0 rs6311 13 46897343
1 rs662138 6 160143444

Converting full variation/mutation data to tabular format

Parsing utilities can quickly extract the data to a VariantSet object holding pandas DataFrames with coordinates and alternative alleles frequencies:

from easy_entrez.parsing import parse_dbsnp_variants

variants = parse_dbsnp_variants(result)
variants

<VariantSet with 2 variants>

To get the coordinates:

variants.coordinates
rs_id ref alts chrom pos chrom_prev pos_prev consequence
rs6311 C A,T 13 46897343 13 47471478 upstream_transcript_variant,intron_variant,genic_upstream_transcript_variant
rs662138 C G 6 160143444 6 160564476 intron_variant

For frequencies:

variants.alt_frequencies.head(5)  # using head to only display first 5 for brevity
rs_id allele source_frequency total_count study count
0 rs6311 T 0.44349 2221 1000Genomes 984.991
1 rs6311 T 0.411261 1585 ALSPAC 651.849
2 rs6311 T 0.331696 1486 Estonian 492.9
3 rs6311 T 0.35 14 GENOME_DK 4.9
4 rs6311 T 0.402529 56309 GnomAD 22666

Obtaining the SNP rs ID number from chromosomal position

You can use the query string directly:

results = entrez_api.search(
    '13[CHROMOSOME] AND human[ORGANISM] AND 31873085[POSITION]',
    database='snp',
    max_results=10
)
print(results.data['esearchresult']['idlist'])

['59296319', '17076752', '7336701', '4']

Or pass a dictionary (no validation of arguments is performed, AND conjunction is used):

results = entrez_api.search(
    dict(chromosome=13, organism='human', position=31873085),
    database='snp',
    max_results=10
)
print(results.data['esearchresult']['idlist'])

['59296319', '17076752', '7336701', '4']

The base position should use the latest genome assembly (GRCh38 at the time of writing); you can use the position in previous assembly coordinates by replacing POSITION with POSITION_GRCH37. For more information of the arguments accepted by the SNP database see the entrez help page on NCBI website.

Obtaining amino acids change information for variants in given range

First we search for dbSNP rs identifiers for variants in given region:

dbsnp_ids = (
    entrez_api
    .search(
        '12[CHROMOSOME] AND human[ORGANISM] AND 21178600:21178720[POSITION]',
        database='snp',
        max_results=100
    )
    .data
    ['esearchresult']
    ['idlist']
)

Then fetch the variant data for identifiers:

variant_data = entrez_api.fetch(
    ['rs' + rs_id for rs_id in dbsnp_ids],
    max_results=10,
    database='snp'
)

And parse the data, extracting the HGVS out of summary:

from easy_entrez.parsing import parse_dbsnp_variants
from pandas import Series


def select_protein_hgvs(items):
    return [
        [sequence, hgvs]
        for entry in items
        for sequence, hgvs in [entry.split(':')]
        if hgvs.startswith('p.')
    ]


protein_hgvs = (
    parse_dbsnp_variants(variant_data)
    .summary
    .HGVS
    .apply(select_protein_hgvs)
    .explode()
    .dropna()
    .apply(Series)
    .rename(columns={0: 'sequence', 1: 'hgvs'})
)
protein_hgvs.head()
rs_id sequence hgvs
rs1940853486 NP_006437.3 p.Gly203Ter
rs1940853414 NP_006437.3 p.Glu202Gly
rs1940853378 NP_006437.3 p.Glu202Lys
rs1940853299 NP_006437.3 p.Lys201Thr
rs1940852987 NP_006437.3 p.Asp198Glu

Fetching more than 10 000 entries

Use in_batches_of method to fetch more than 10k entries (e.g. variant_ids):

snps_result = (
    entrez.api
    .in_batches_of(1_000)
    .fetch(variant_ids, max_results=5_000, database='snp')
)

The result is a dictionary with keys being identifiers used in each batch (because the Entrez API does not always return the indentifiers back) and values representing the result. You can use parse_dbsnp_variants directly on this dictionary.

Find PubMed ID from DOI

When searching GWAS catalog PMID is needed over DOI. You can covert one to the other using:

def doi_term(doi: str) -> str:
    """Clean a DOI string by removing URL prefix."""
    doi = (
        doi
        .replace('http://', 'https://')
        .replace('https://doi.org/', '')
    )
    return f'"{doi}"[Publisher ID]'


result = entrez_api.search(
    doi_term('https://doi.org/10.3389/fcell.2021.626821'),
    database='pubmed',
    max_results=1
)
print(result.data['esearchresult']['idlist'])

['33834021']

Installation

Requires Python 3.6+ (though only 3.7+ is tested). Install with:

pip install easy-entrez

If you wish to enable (optional, tqdm-based) progress bars use:

pip install easy-entrez[with_progress_bars]

If you wish to enable (optional, pandas-based) parsing utilities use:

pip install easy-entrez[with_parsing_utils]

Contributing

To build the documentation locally:

pip install -e .[docs]
sphinx-build docs docs/_build
open docs/_build/index.html

Alternatives

You might want to try:

  • biopython.Entrez - biopython is a heavy dependency, but probably good choice if you already use it
  • pubmedpy - provides interesting utilities for parsing the responses
  • entrez - appears to have a comparable scope but quite different API
  • entrezpy - this one did not work well for me (hence this package), but may have improved since