Nightscout Migrate

A CLI tool to migrate all data from one Nightscout instance to another with resume capability.

Features

• Interactive wizard - Guided prompts for credentials and configuration
• Full data migration - Transfers entries (CGM readings), treatments, profiles, device status, and food database
• Resume capability - Interrupted migrations can be resumed from the last checkpoint
• Fresh install verification - Prevents accidental data overwrites by verifying target is empty
• Progress tracking - Real-time progress display with weekly chunk status
• Target reset - Built-in command to wipe target instance for re-migration

Requirements

• Node.js 18.0.0 or higher (uses native fetch)
• Source Nightscout with API access enabled and read permissions
• Target Nightscout with API access enabled and write permissions (use API_SECRET)
• Both instances must be accessible via HTTPS

Installation

# Clone the repository
git clone https://github.com/imbercal/nightscout-migrate.git
cd nightscout-migrate

# Install dependencies
npm install

# Build the project
npm run build

Usage

Start a New Migration

npm start

The interactive wizard will prompt you for:

1. Source Nightscout URL - The instance to migrate FROM (e.g., https://my-old-ns.herokuapp.com)
2. Source API key - Your API_SECRET or a token with read access
3. Target Nightscout URL - The instance to migrate TO (must be empty)
4. Target API key - Your API_SECRET (requires write access)
5. Start date - How far back to migrate (default: 1 year)

Resume an Interrupted Migration

npm start -- --resume

If a migration is interrupted (Ctrl+C, network failure, etc.), you can resume from where it left off. The tool saves progress after each weekly chunk.

Check Migration Status

npm start -- --status

Shows the current state of an in-progress migration, including which collections are completed and the last migrated date.

Reset Target Instance

npm start -- --reset-target

Deletes all data from a target Nightscout instance. Useful if you need to start the migration over. Use with caution!

How It Works

Migration Order

Collections are migrated in a specific order to maintain data integrity:

1. Profiles - Migrated first as treatments may reference them
2. Entries - CGM/SGV readings (typically the largest dataset)
3. Treatments - Insulin doses, carbs, notes, temp basals, etc.
4. Device Status - Loop/OpenAPS/pump status records
5. Food - Food database entries

Chunking Strategy

• Data is fetched and uploaded in weekly chunks to handle large datasets
• Each chunk uses batches of 25-100 records per API request to avoid payload limits
• Progress is saved after each week completes, enabling resume capability

Batch Sizes

Different collections use different batch sizes to avoid "request entity too large" errors:

Collection	Batch Size
Entries	100
Treatments	100
Device Status	25
Food	100
Profiles	10

Device status records are particularly large due to pump/loop data, hence the smaller batch size.

Authentication

The tool supports Nightscout's API authentication:

• API_SECRET (12+ characters) - Automatically hashed with SHA1
• Access tokens - Used as-is (created via Admin Tools)

For the source, read access is sufficient. For the target, you need write access (typically requires API_SECRET).

State File

Migration progress is stored in .migration-state.json in the current directory:

{
  "sourceUrl": "https://source.example.com",
  "targetUrl": "https://target.example.com",
  "startDate": "2024-01-01T00:00:00.000Z",
  "collections": {
    "profiles": { "status": "completed", "recordsMigrated": 2 },
    "entries": { "status": "in_progress", "lastDate": "2024-06-15T00:00:00.000Z", "recordsMigrated": 52847 },
    "treatments": { "status": "pending", "recordsMigrated": 0 },
    "devicestatus": { "status": "pending", "recordsMigrated": 0 },
    "food": { "status": "pending", "recordsMigrated": 0 }
  },
  "startedAt": "2024-12-01T10:00:00.000Z",
  "lastUpdated": "2024-12-01T10:30:00.000Z"
}

Delete this file to start a fresh migration.

Known Issues & Limitations

General Limitations

• Target must be empty - The tool will abort if the target instance contains any data. Use --reset-target to clear it first.
• No incremental sync - This is a one-time migration tool, not a continuous sync solution.
• No data transformation - Records are copied as-is; no data cleanup or transformation is performed.

API Limitations

• Rate limiting - The tool includes retry logic with exponential backoff, but very large migrations may hit API limits. If this happens, wait a few minutes and use --resume.
• Payload size - Nightscout has request body limits. The tool uses small batch sizes to avoid this, but some device status records may be exceptionally large.

Data Considerations

• Duplicate prevention - The tool doesn't check for duplicates. Running a migration twice without resetting will create duplicate records.
• Profile references - If treatments reference profile names that don't exist in the migrated profiles, those references may be broken.
• Time zones - Timestamps are preserved as-is. Ensure both instances use the same time zone settings.

Troubleshooting

"request entity too large" Error

The batch size is too large for your Nightscout server. This has been addressed with smaller default batch sizes, but if you still encounter it:

1. Check if it's happening with device status (most common)
2. Consider manually reducing batch sizes in src/config.ts

"Invalid API key" Error

• Ensure you're using the correct API_SECRET (check Nightscout's config vars)
• For Heroku: heroku config:get API_SECRET --app your-app-name
• For self-hosted: Check your environment variables

"Target is not a fresh installation" Error

The target Nightscout has existing data. Either:

• Use a different target instance
• Run npm start -- --reset-target to clear the target (data will be lost!)

Migration Stuck or Slow

• Large datasets (years of CGM data) take time - this is normal
• Device status records are particularly slow due to their size
• Use --status to check progress
• If truly stuck, Ctrl+C and --resume to retry

Network Errors

The tool automatically retries failed requests up to 3 times with exponential backoff. If errors persist:

1. Check your internet connection
2. Verify both Nightscout instances are online
3. Wait a few minutes and use --resume

Project Structure

nightscout-migrate/
├── src/
│   ├── index.ts              # CLI entry point
│   ├── config.ts             # Configuration and types
│   ├── api/
│   │   ├── client.ts         # Nightscout API client
│   │   └── types.ts          # API response types
│   ├── migration/
│   │   ├── migrator.ts       # Migration orchestrator
│   │   ├── collections.ts    # Per-collection handlers
│   │   └── progress.ts       # State persistence
│   └── utils/
│       ├── prompts.ts        # Interactive prompts
│       ├── validation.ts     # Validation helpers
│       └── logger.ts         # Console output
├── dist/                     # Compiled JavaScript
├── package.json
├── tsconfig.json
└── README.md

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run npm run build to ensure it compiles
5. Submit a pull request

License

MIT License - see LICENSE file for details.

Disclaimer

This tool is provided as-is. Always ensure you have backups of your Nightscout data before performing migrations. The authors are not responsible for any data loss.

Acknowledgments

• Nightscout - The open-source CGM monitoring project
• The Nightscout community for their invaluable diabetes management tools