跳到主要内容

Dataset Usage Guide

AtomGit provides comprehensive dataset management features to help you easily create, share, and use high-quality datasets. This guide will introduce the main operations related to datasets.

Dataset Creation​

Create a New Dataset​

  1. Log in to your AtomGit account
  2. Click "+" > "New Dataset" in the top right corner
  3. Fill in the dataset information:
    • Dataset ID
    • Dataset Name
    • Select a LICENSE template
    • Choose whether to make it public
  4. Select a LICENSE template:
    • PyTorch Creative Commons Attribution Non Commercial 3.0
    • Creative Commons Attribution Non Commercial No Derivatives 3.0
    • Creative Commons Attribution Non Commercial Share Alike 2.0
    • Creative Commons Attribution Non Commercial Share Alike 3.0
    • H Research License
    • Open Model, Data & Weights License Agreement
    • Unknown
  5. Click "Create Dataset" to complete

Dataset Creation Dataset Creation

Dataset Configuration​

Create a dataset-config.yaml file to define the dataset structure:

dataset-name: my-awesome-dataset
version: 1.0.0
type: image-classification
format: 
  - jpg
  - png
structure:
  train: train/
  validation: val/
  test: test/
labels:
  path: labels.csv
  format: csv
  1. Enter keywords in the search box
  2. Use filter conditions:
    • Comprehensive sorting
    • Most popular
    • Most downloads
    • Newest created
    • Recently updated

Supports the following search syntax:

  • type:image - Search by data type
  • size:>1GB - Search by dataset size
  • license:MIT - Search by license
  • language:chinese - Search by dataset language

Dataset Download​

  1. Go to the dataset details page
  2. Click the "Clone" button
  3. Copy the project address
  4. Clone it locally
  5. When using HTTPS protocol, configure and use a personal access token instead of login password for cloning, pushing, etc.
  6. When using SSH protocol, generate an SSH public key on your local machine for cloning, pushing, etc.

Command Line Download​

# Install GitCode CLI
pip install gitcode

# Download the full dataset
gitcode download-dataset username/dataset-name

# Download a specific version
gitcode download-dataset username/dataset-name --version v1.0.0

# Download part of the data
gitcode download-dataset username/dataset-name --split train

Dataset Usage​

openMind​

# You need to install openmind-hub first
pip install openmind-hub

# Set environment variables:
import os
os.environ["OPENMIND_HUB_ENDPOINT"] = "https://hub.gitcode.com"

# Download the dataset
from openmind_hub import snapshot_download
or
from openmind.integrations.datasets import load_dataset token = "***" load_dataset(path="lejurobot/LET_Base_Dataset", streaming=False)

# Apply for an access token by configuring and using a personal access token
token = ""

# If local_dir is not specified, check in ~/.cache/openmind
snapshot_download("BAAI/Infinity-Instruct",  local_dir = './Infinity-Instruct',token = token )

Dataset Version Control​

# Load a specific version
dataset_v1 = load_dataset("username/dataset-name"， version="1.0.0")

# View version history
dataset.version_history()

# Create a new version
dataset.create_version("1.1.0"， description="Added new samples")

Best Practices​

  1. Data Quality Control

    • Perform data cleaning
    • Check data integrity
    • Validate annotation quality

  2. Dataset Documentation

    • Detailed data description
    • Data collection methods
    • Usage restriction statements
    • Privacy considerations

  3. Version Management

    • Semantic version numbers
    • Update logs
    • Change notes

  4. Data Security

    • Data anonymization
    • Access control
    • Compliance checks

Frequently Asked Questions​

Q: How to handle large datasets? A: You can use streaming loading or sharded download features to avoid loading all data at once.

Q: What data formats are supported? A: Common data formats such as CSV, JSON, images, audio, etc., are supported. See the documentation for more details.

Q: How to contribute data? A: You can submit new data through the dataset update feature, or create a dataset branch for collaboration.

Q: What is the storage limit for datasets? A: Free accounts can create datasets up to 10GB, while advanced accounts have larger storage limits.