CLI

The command-line interface for TopK.

Command-line interface for TopK.

Installation

brew tap topk-io/topk
brew install topk

Quick start

topk login
topk import ./books.parquet --region aws-us-east-1-elastica
topk import postgres://user:pw@host/db 'public.*' --region aws-us-east-1-elastica

topk import discovers a schema, shows the plan, and imports on confirmation — see import.

Authentication

To authenticate, run:

topk login

Commands

login

Log in with TopK CLI:

topk login

If the CLI cannot open your browser automatically, or you prefer to open it manually, use topk login --no-browser to print the login URL.

logout

Remove the saved session:

topk logout

project

list

List projects in the current organization.

topk project list

get

Get a project by ID.

topk project get <PROJECT_ID>

argument	description
<PROJECT_ID>	ID of the project

create

Create a project.

topk project create <NAME>

argument	description
<NAME>	Display name of the new project

delete

Delete a project by ID.

topk project delete <PROJECT_ID> [-y]

argument	description
<PROJECT_ID>	ID of the project
-y, --yes	Skip the confirmation prompt; required when stdin is not a terminal

region

list

List available regions.

topk region list

import

Bulk import into TopK collections. Every run prints the plan as a TOML spec and asks before writing; collections are created right after.

topk import [SOURCE] [OBJECTS]... [OPTIONS]

argument	description
[SOURCE]	Source URL, file path, or glob; also required with --spec or --resume, except for files
[OBJECTS]...	Source objects to import; exact names, globs, or <object>=<collection>
--region <REGION>	Region to import into (env TOPK_REGION); required. See https://docs.topk.io/regions
--to <COLLECTION>	Name the target collection; required when it cannot be derived
--id <COLUMN>	Column to use as the document id (_id); use when it can’t be auto-detected
--dry-run	Print the spec and a sample of documents, without importing
-f, --spec <SPEC>	Read a TOML import spec
-y, --yes	Skip the confirmation prompt; required when stdin is not a terminal
--resume <RUN>	Continue a run that stopped, by the id in its header; -f swaps in an edited spec
--filter <FILTER>	Only read rows matching a filter, in the source’s language: SQL WHERE for files and databases, a JSON query for mongodb/elasticsearch
--limit <LIMIT>	Read at most this many rows per object
--partition <PARTITION>	Import into this partition
--continue-on-error	Skip documents that fail instead of stopping; exits non-zero if any did
-c, --concurrency <N>	Concurrent upserts in flight, budgeted across the whole run (default 16)
--batch-bytes <SIZE>	Bytes of documents per upsert (default 8MiB)
--project-id <PROJECT_ID>	Project to import into with your login; required unless using an API key
--api-key <API_KEY>	Use an API key instead of your login; overrides the TOPK_API_KEY environment variable

Authenticate

Log in, then provide --project-id and --region when importing:

topk login
topk project list
topk import ./books.parquet --project-id <project-id> --region aws-us-east-1-elastica

Use topk project list to find your project ID and topk region list to list regions. Access tokens are cached and renewed automatically. Alternatively, pass --api-key or set TOPK_API_KEY instead of logging in and providing --project-id. Get an API key in the TopK console.

--project-id cannot be combined with an API key (--api-key or TOPK_API_KEY). Unset TOPK_API_KEY to use your login.

Import a database

topk import postgres://user:pw@host/db 'public.*'
topk import mysql://root@host/shop orders
topk import mongodb://host/shop products
topk import es+https://user:[e-mail] 'products*'
topk import postgres://host/db orders --filter "created_at > '2024-01-01'"

Objects are exact names, globs, or <object>=<collection> renames.

Import files

topk import ./books.parquet                                   # .csv .tsv .json(l) .arrow .avro
topk import 's3://bucket/books/*.parquet' --to books          # a glob needs --to
topk import gs://bucket/books.csv                             # az:// hf:// http(s):// too

A single file names its collection.

Copy a TopK collection

topk import topk://aws-us-east-1-elastica/books --to books-v2                        # reindex under a new schema
topk import topk://aws-us-east-1-elastica/books --region aws-eu-central-1-monstera   # copy us → eu

Schema and indexes copy as-is; the copy is additive.

Preview and edit the plan

topk import postgres://host/db orders --dry-run > spec.toml   # spec on stdout, sample documents on stderr
vim spec.toml                                                 # drop columns, fix types, add indexes
topk import "$DB_URL" -f spec.toml --yes                      # run

Keep credentials on the command line or in the environment, never in the spec — the spec is meant to go in git.

Resume a stopped run

Stop a run — ^C, a lost connection — and pick it up where it left off:

topk import postgres://host/db --resume 01J9...               # run id printed at start

Resume skips finished collections and continues the in-flight one from a checkpoint. Without --resume, a re-run re-imports everything — upserts are idempotent.

Sources

source	url	filter	auth
postgres	postgres://	SQL WHERE	in-URL, PGPASSWORD
mysql	mysql://	SQL WHERE	in-URL, MYSQL_PWD
sqlite	sqlite:<path>	SQL WHERE	—
files	path, glob, s3:// r2:// gs:// az:// hf:// http(s)://	SQL WHERE	credential chain; hf auth login; http anonymous
elasticsearch	es:// es+https://, *.cloud.es.io	query DSL	in-URL, ELASTIC_API_KEY / ELASTIC_PASSWORD
mongodb	mongodb://	find document	in-URL, MONGODB_URI
topk	topk://[<key>@]<region>/<collection>	—	URL key, else the run’s API key or access token

Spec

One TOML table per collection — from, id, filter/partition/limit as the flags, and a fields whitelist (only declared fields import):

[books]
from = "public.books"
id = "sku"

[books.fields]
title = { type = "text", index = "semantic" }
year = { type = "int", from = "published_year" }
embedding = { type = "f32_vector", dim = 768, index = { vector = { metric = "cosine" } } }

field key
type	text int float bool bytes timestamp struct *_list *_vector *_matrix *_sparse_vector
from	source column, when the name differs
required	fail the document if missing
truncate	max chars, text only
dim / cols	vector dimension / matrix row width
index	"keyword" "exact" "semantic" "ngram" { vector = { metric = "cosine" | "euclidean" | "dot_product" | "hamming" } } { multi_vector = {} }

Discovery takes the source’s types; declaring a different type converts. Embeddings that arrive as float lists, text ("[1,2,3]") or packed bytes become vectors by declaring f32_vector and dim (f32_matrix and cols for multi-vectors); a sparse vector reads numeric keys or parallel indices/values lists; epoch numbers or date text become timestamp. Conversions are exact or the document fails; a narrower declaration (truncate included) is how loss is accepted.

Failures and limits

A document fails when a value does not convert exactly, its id is missing or empty, or it is over 200 KB. A failure stops the run, ready to --resume; --continue-on-error skips them instead.

• there is no schema migration: an existing collection whose schema differs is rejected
• an upsert replaces the whole document, so a spec that omits an existing field clears it

Global flags

These flags are accepted by every command:

--output

Options:

• text (default)
• json

Updating the CLI

To update CLI to the latest version, run:

brew update
brew upgrade topk