Skip to main content

chapter 1: your first upload

WIP

Set up the upload engine, add files, and track progress.

Goal

By the end of this chapter you will have a working upload pipeline. You will create an upload client, add a file, start the upload, and watch it progress through the state machine phases.

Loading diagram...

Step by Step

Install @gentleduck/upload

npm install @gentleduck/upload
npm install @gentleduck/upload

This is the only package you need. It includes the core engine, React bindings, and all built-in strategies. It works with npm, yarn, pnpm, and bun.

Define your types

Create src/upload.ts. First, define the intent map, cursor map, purpose union, and result type that describe your upload pipeline:

src/upload.ts
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// Intent map: which strategies exist and what data they need
type PhotoIntentMap = {
  post: PostIntent
}
 
// Cursor map: resume state per strategy
type PhotoCursorMap = {
  post: PostCursor
}
 
// Purpose: what kind of upload is this?
type PhotoPurpose = 'photo'
 
// Result: what the backend returns on completion
type PhotoResult = UploadResultBase & {
  url: string
}
src/upload.ts
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// Intent map: which strategies exist and what data they need
type PhotoIntentMap = {
  post: PostIntent
}
 
// Cursor map: resume state per strategy
type PhotoCursorMap = {
  post: PostCursor
}
 
// Purpose: what kind of upload is this?
type PhotoPurpose = 'photo'
 
// Result: what the backend returns on completion
type PhotoResult = UploadResultBase & {
  url: string
}

The intent map defines all upload strategies your app supports. Each key is a strategy name, and the value is the data shape your backend returns. For now we use the built-in PostIntent.

The purpose is a string union that categorizes uploads. PhotoDuck only has 'photo' for now, but a real app might have 'avatar' | 'cover' | 'attachment'.

Implement the UploadApi

The UploadApi is the contract between the upload engine and your backend. It has two required methods: createIntent and complete. For now, we mock them:

src/upload.ts
const api: UploadApi<PhotoIntentMap, PhotoPurpose, PhotoResult> = {
  async createIntent({ purpose, contentType, size, filename }) {
    // In a real app, this calls your backend to get a presigned POST URL
    console.log(`Creating intent for ${filename} (${size} bytes, ${contentType})`)
 
    return {
      strategy: 'post',
      fileId: `file-${Date.now()}`,
      url: 'https://your-bucket.s3.amazonaws.com',
      fields: {
        key: `uploads/${filename}`,
        'Content-Type': contentType,
      },
    }
  },
 
  async complete({ fileId }) {
    // In a real app, this tells your backend the upload finished
    console.log(`Completing upload for ${fileId}`)
    return {
      fileId,
      key: `uploads/${fileId}`,
      url: `https://cdn.example.com/uploads/${fileId}`,
    }
  },
}
src/upload.ts
const api: UploadApi<PhotoIntentMap, PhotoPurpose, PhotoResult> = {
  async createIntent({ purpose, contentType, size, filename }) {
    // In a real app, this calls your backend to get a presigned POST URL
    console.log(`Creating intent for ${filename} (${size} bytes, ${contentType})`)
 
    return {
      strategy: 'post',
      fileId: `file-${Date.now()}`,
      url: 'https://your-bucket.s3.amazonaws.com',
      fields: {
        key: `uploads/${filename}`,
        'Content-Type': contentType,
      },
    }
  },
 
  async complete({ fileId }) {
    // In a real app, this tells your backend the upload finished
    console.log(`Completing upload for ${fileId}`)
    return {
      fileId,
      key: `uploads/${fileId}`,
      url: `https://cdn.example.com/uploads/${fileId}`,
    }
  },
}

createIntent is called when a file enters the pipeline. Your backend decides the strategy, generates presigned URLs, and returns an intent object. The engine then hands this intent to the matching strategy.

complete is called after the bytes are transferred. Your backend can mark the file as uploaded in the database, generate thumbnails, or return metadata.

Create the upload client

Now wire everything together with createUploadClient:

src/upload.ts
import {
  createUploadClient,
  createStrategyRegistry,
  PostStrategy,
  createXHRTransport,
} from '@gentleduck/upload'
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// ... types and api from above ...
 
const strategies = createStrategyRegistry<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>()
strategies.set(PostStrategy<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>())
 
export const uploadClient = createUploadClient<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>({
  api,
  strategies,
  transport: createXHRTransport(),
})
src/upload.ts
import {
  createUploadClient,
  createStrategyRegistry,
  PostStrategy,
  createXHRTransport,
} from '@gentleduck/upload'
import type { UploadApi, UploadResultBase } from '@gentleduck/upload'
import { PostIntent, PostCursor } from '@gentleduck/upload'
 
// ... types and api from above ...
 
const strategies = createStrategyRegistry<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>()
strategies.set(PostStrategy<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>())
 
export const uploadClient = createUploadClient<PhotoIntentMap, PhotoCursorMap, PhotoPurpose, PhotoResult>({
  api,
  strategies,
  transport: createXHRTransport(),
})

createStrategyRegistry creates a registry where you register strategy implementations. PostStrategy() creates the built-in POST strategy for simple file uploads. createXHRTransport() creates a browser-native XHR transport with upload progress support.

Add files and start uploading

Create src/main.ts:

src/main.ts
import { uploadClient } from './upload'
 
// Add files to the pipeline
const file = new File(['hello world'], 'photo.jpg', { type: 'image/jpeg' })
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Listen to progress events
uploadClient.on('upload.progress', (event) => {
  console.log(`Progress: ${event.pct.toFixed(1)}% (${event.uploadedBytes}/${event.totalBytes})`)
})
 
// Listen to completion
uploadClient.on('upload.completed', (event) => {
  console.log(`Upload completed: ${event.localId}`, event.result)
})
 
// Listen to errors
uploadClient.on('upload.error', (event) => {
  console.log(`Upload failed: ${event.localId}`, event.error.message)
})
 
// Start all uploads
uploadClient.dispatch({ type: 'startAll' })
src/main.ts
import { uploadClient } from './upload'
 
// Add files to the pipeline
const file = new File(['hello world'], 'photo.jpg', { type: 'image/jpeg' })
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Listen to progress events
uploadClient.on('upload.progress', (event) => {
  console.log(`Progress: ${event.pct.toFixed(1)}% (${event.uploadedBytes}/${event.totalBytes})`)
})
 
// Listen to completion
uploadClient.on('upload.completed', (event) => {
  console.log(`Upload completed: ${event.localId}`, event.result)
})
 
// Listen to errors
uploadClient.on('upload.error', (event) => {
  console.log(`Upload failed: ${event.localId}`, event.error.message)
})
 
// Start all uploads
uploadClient.dispatch({ type: 'startAll' })

dispatch is the single entry point for all commands. addFiles puts files into the pipeline. startAll begins uploading every file that is in the ready phase.

How the State Machine Works

Every upload item flows through a sequence of phases. The phase determines what the engine is doing with the file at any given moment:

Loading diagram...
PhaseMeaning
validatingFile is being checked against validation rules (size, type)
creating_intentEngine is calling api.createIntent() to get upload instructions
readyIntent received, waiting for user or autoStart to trigger upload
queuedUpload requested, waiting for a concurrency slot
uploadingBytes are being transferred
completingBytes sent, calling api.complete() to finalize
completedUpload finished successfully
errorSomething went wrong (may be retryable)
pausedUpload paused by user (resumable strategies can pick up where they left off)
canceledUpload canceled by user

When you dispatch({ type: 'addFiles', ... }), the engine:

  1. Creates a local ID and fingerprint for each file
  2. Moves the item to validating phase
  3. Runs validation rules (Chapter 5)
  4. On success, moves to creating_intent and calls api.createIntent()
  5. On success, moves to ready with the intent attached

When you dispatch({ type: 'start', localId }) or dispatch({ type: 'startAll' }):

  1. Items in ready move to queued
  2. The scheduler picks items from the queue (respecting maxConcurrentUploads)
  3. The matching strategy's start() method runs, moving the item to uploading
  4. On success, the item moves to completing and the engine calls api.complete()
  5. On success, the item reaches completed

Commands Reference

All user actions go through dispatch(). Here are the available commands:

CommandEffect
{ type: 'addFiles', files: File[], purpose: P }Add files to the pipeline
{ type: 'start', localId: string }Start a single upload
{ type: 'startAll', purpose?: P }Start all ready uploads (optionally filtered by purpose)
{ type: 'pause', localId: string }Pause a single upload
{ type: 'pauseAll', purpose?: P }Pause all active uploads
{ type: 'resume', localId: string }Resume a paused upload
{ type: 'cancel', localId: string }Cancel a single upload
{ type: 'cancelAll', purpose?: P }Cancel all uploads
{ type: 'retry', localId: string }Retry a failed upload
{ type: 'remove', localId: string }Remove an item from state
{ type: 'rebind', localId: string, file: File }Re-attach a File to a persisted paused item

Events Reference

Subscribe to events with uploadClient.on(eventName, callback). The on method returns an unsubscribe function:

const unsub = uploadClient.on('upload.progress', (event) => {
  console.log(event.pct)
})
 
// Later, stop listening
unsub()
const unsub = uploadClient.on('upload.progress', (event) => {
  console.log(event.pct)
})
 
// Later, stop listening
unsub()
EventPayload
file.added{ localId, purpose, file, fingerprint }
file.rejected{ file, reason }
intent.creating{ localId }
intent.created{ localId, intent }
intent.failed{ localId, error, retryable }
upload.started{ localId }
upload.progress{ localId, pct, uploadedBytes, totalBytes }
upload.paused{ localId, cursor }
upload.canceled{ localId }
upload.completing{ localId }
upload.completed{ localId, result, completedBy }
upload.error{ localId, error, retryable }

Reading State with getSnapshot

You can read the current state at any time:

const snapshot = uploadClient.getSnapshot()
 
// snapshot.items is a Map<string, UploadItem>
for (const [localId, item] of snapshot.items) {
  console.log(`${localId}: phase=${item.phase}, purpose=${item.purpose}`)
 
  if (item.phase === 'uploading') {
    console.log(`  progress: ${item.progress.pct.toFixed(1)}%`)
  }
 
  if (item.phase === 'error') {
    console.log(`  error: ${item.error.message} (retryable: ${item.retryable})`)
  }
}
const snapshot = uploadClient.getSnapshot()
 
// snapshot.items is a Map<string, UploadItem>
for (const [localId, item] of snapshot.items) {
  console.log(`${localId}: phase=${item.phase}, purpose=${item.purpose}`)
 
  if (item.phase === 'uploading') {
    console.log(`  progress: ${item.progress.pct.toFixed(1)}%`)
  }
 
  if (item.phase === 'error') {
    console.log(`  error: ${item.error.message} (retryable: ${item.retryable})`)
  }
}

The UploadItem type is a discriminated union on phase. TypeScript narrows the type when you check the phase, giving you access to phase-specific fields like progress, intent, error, and cursor.

Waiting for Uploads

The waitFor method returns a promise that resolves when all given uploads reach a terminal state (completed, error, or canceled):

uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Get the localId from the snapshot
const snapshot = uploadClient.getSnapshot()
const localId = Array.from(snapshot.items.keys())[0]
 
uploadClient.dispatch({ type: 'start', localId })
 
const [outcome] = await uploadClient.waitFor([localId])
 
if (outcome.status === 'completed') {
  console.log('Upload done:', outcome.result)
} else if (outcome.status === 'error') {
  console.log('Upload failed:', outcome.error)
}
uploadClient.dispatch({ type: 'addFiles', files: [file], purpose: 'photo' })
 
// Get the localId from the snapshot
const snapshot = uploadClient.getSnapshot()
const localId = Array.from(snapshot.items.keys())[0]
 
uploadClient.dispatch({ type: 'start', localId })
 
const [outcome] = await uploadClient.waitFor([localId])
 
if (outcome.status === 'completed') {
  console.log('Upload done:', outcome.result)
} else if (outcome.status === 'error') {
  console.log('Upload failed:', outcome.error)
}

Checkpoint

Your project should look like this:

photoduck/
  src/
    upload.ts    -- types + api + client
    main.ts      -- dispatch commands + event listeners
  package.json
  tsconfig.json

Chapter 1 FAQ

Next: Chapter 2: Strategies & Backends