Semanticscholar

S2AG Datasets

Download full-corpus datasets from the Semantic Scholar Academic Graph (S2AG) Some python demonstrating usage of the datasets API: r1 =…

Data & analytics#science--math

Fork copies the collection into your workspace, without the publisher's secrets, so you can add your own key and send requests. Star saves it to your starred APIs (Account › Marketplace) and helps others find it.

Overview

Download full-corpus datasets from the Semantic Scholar Academic Graph (S2AG)

Some python demonstrating usage of the datasets API:

r1 = requests.get('https://api.semanticscholar.org/datasets/v1/release').json()
print(r1[-3:])
['2023-03-14', '2023-03-21', '2023-03-28']

r2 = requests.get('https://api.semanticscholar.org/datasets/v1/release/latest').json()
print(r2['release_id'])
2023-03-28

print(json.dumps(r2['datasets'][0], indent=2))
{
"name": "abstracts",
"description": "Paper abstract text, where available. 100M records in 30 1.8GB files.",
"README": "Semantic Scholar Academic Graph Datasets The "abstracts" dataset provides..."
}

r3 = requests.get('https://api.semanticscholar.org/datasets/v1/release/latest/dataset/abstracts').json()
print(json.dumps(r3, indent=2))
{
"name": "abstracts",
"description": "Paper abstract text, where available. 100M records in 30 1.8GB files.",
"README": "Semantic Scholar Academic Graph Datasets The "abstracts" dataset provides...",
"files": [
"https://ai2-s2ag.s3.amazonaws.com/dev/staging/2023-03-28/abstracts/20230331_0..."
]
}

Endpoints 4

Release Data

  • GETList of Available Releases/release/Releases are identified by a date stamp such as "2023-08-01". Each release contains full data for each dataset.
  • GETList of Datasets in a Release/release/:release_idMetadata describing a particular release, including a list of datasets available.
  • GETDownload Links for a Dataset/release/:release_id/dataset/:dataset_nameDatasets are partitioned and stored on S3. Clients can retrieve them by requesting this list of pre-signed download urls and fetching all the partitions.

Incremental Updates

  • GETDownload Links for Incremental Diffs/diffs/:start_release_id/to/:end_release_id/:dataset_nameFull datasets can be updated from one release to another to avoid downloading and processing data that hasn't changed. This method returns a list of all the…

API details

Authentication
None declared in the description
Operations
4 operations on 4 pathsGET 4
Main resources
Release Data 3Incremental Updates 1
Response formats
application/json
Description format
Swagger 2.0, API version 1.0

How to test the S2AG Datasets in Swastik Codes

  1. Open the APISelect “Fork & run in Swastik Codes” on this page. Sign in or create a free account if asked.
  2. Fork it into your workspaceChoose a workspace. You get your own copy of the S2AG Datasets collection, without the publisher's secrets.
  3. Add your credentialsOpen the collection's variables or an environment and enter your API key or token, if the API needs one.
  4. Send a requestPick an endpoint, for example “List of Available Releases” (GET /release/), and press Send. Check the response, add tests and run the whole collection with the runner.

Frequently asked questions

What is the S2AG Datasets?

Download full-corpus datasets from the Semantic Scholar Academic Graph (S2AG) Some python demonstrating usage of the datasets API: r1 =… It is listed under Data & analytics and has 4 endpoints.

Who publishes the S2AG Datasets?

It is listed from Semanticscholar's public API description (an OpenAPI document), not published by Semanticscholar on Swastik Codes. Anyone who controls semanticscholar.org can claim the profile by adding a DNS TXT record.

How do I authenticate with the S2AG Datasets?

Its OpenAPI description does not declare an authentication scheme. Some endpoints may be open; check the provider's documentation before relying on that.

How many endpoints does the S2AG Datasets have?

4 operations on 4 paths (4 GET). The largest groups are Release Data, Incremental Updates.

Is forking the S2AG Datasets free?

Yes. Forking copies the collection into your workspace for free. It has been starred 0 times and forked 0 times.

Suggest an API

Tell us which public API is missing. We check it, then add it here with its endpoints when it has an OpenAPI (Swagger) description.