genscrape

A JavaScript library that aids in scraping person data off of genealogy websites. It is designed to be used in a browser extension. Data is output in GEDCOM X format.

Install

npm install --save genscrape

Or use the CDN

<script src="https://unpkg.com/genscrape@latest/dist/genscrape.min.js"></script>

We recommend pinning CDN links to a specific version.

Usage

Scraping is sometimes asynchronous so we chose to implement the EventEmitter interface using EventEmmitter2.

genscrape().on('data', function(data){
  // Do something with the data
});

That's it. Genscrape automatically detects what page you're on, looks up the correct scraper, then does its magic.

Events

data

The data event is fired when genscrape successfully scrapes data from the page. Read more about the data model below.

noMatch

The noMatch event is fired when genscrape is called on a page with a URL that it does not recognize. In other words, it does not have a parser registered for the URL.

noData

The noData event is fired when genscrape is run on a page that a parser is registered for but the parser was unable to find any data.

In cases of one-page apps such as the FamilySearch Family Tree and the findmypast trees, you may see multiple noData and data events as the user browses through the tree.

error

The error event is fired when something unexpected occurs while a scraper is processing, such as a failed AJAX call. There is no standardized format for the errors yet.

Data Model

Data is returned in the GEDCOM X JSON format.

When possible, we mark one person as principal. This isn't possible when viewing marriage records on sites that don't make a distinction between persons in a record. The only currently supported site where this occurs is Open Archives.

Every GEDCOM X document has a SourceDescription, pointed to by the about property of the document, that provides a citation for all data in the document. That SourceDescription points to an Agent, via the repository property, which describes the website the data came from.

{
  "id": "agent",
  "names": [
    {
      "lang": "en",
      "value": "Find A Grave"
    }
  ],
  "homepage": {
    "resource": "https://www.findagrave.com"
  }
}

We populate person's IDs with IDs from the source repository.

Person IDs aren't unique between websites. Thus we add a genscrape Identifier which allow us to better compare two arbitrary records to determine if they come from the same record on the same website.

{
  "id": "65630115",
  "identifiers": {
    "genscrape": [
      "genscrape://findagrave/65630115"
    ]
  }
}

Supported Sites

• Ancestry.com, Ancestry.ca, Ancestry.co.uk, Ancestry.com.au, both historical records and tree profiles.
• BillionGraves
• Find A Grave
• findmypast.com, findmypast.co.uk, findmypast.com.au, findmypast.ie, both historical records and tree profiles.
• FamilySearch.org, both historical records and tree profiles.
• Genealogie Online
• MyHeritage historical records and tree profiles
• Open Archives
• WeRelate
• WikiTree person profiles