Living Document Notice
Published 2026-09-16. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Hydrating FreeNext from FreeMyData

Summary

Migrating personal records out of cloud platforms often produces unstructured document archives, monolithic JSON blobs, or raw SQLite database dumps. While extraction utilities like the FreeMyData extraction engine unlock raw data silos directly within the browser, converting these heterogenous records into structured note collections requires an automated ingestion and normalization pipeline.

FreeNext integrates directly with FreeMyData outputs through a client-side hydration engine. Running entirely inside local Web Workers, this pipeline parses extracted database rows, maps arbitrary JSON properties into YAML frontmatter schemas, and writes individual CommonMark documents into local vault storage without intermediate server hops.

The Ingestion Pipeline Architecture

The hydration pipeline consumes extracted database streams, applies structural transformations, and serializes files directly into the user-selected filesystem directory.

+-------------------------------------------------------------+
|                     FreeMyData Output                       |
|         (Raw SQLite Tables or Chunked JSON Exports)         |
+-------------------------------------------------------------+
                               |
                               | Streaming Record Batches
                               v
+-------------------------------------------------------------+
|                 FreeNext Hydration Worker                   |
|       +---------------------------------------------+       |
|       |             Schema Field Mapper             |       |
|       +---------------------------------------------+       |
|                              |                              |
|                              v                              |
|       +---------------------------------------------+       |
|       |         Markdown Generator & AST Sanitizer  |       |
|       +---------------------------------------------+       |
+-------------------------------------------------------------+
                               |
                               | Serialized File Batches
                               v
+-------------------------------------------------------------+
|                 Local Vault Storage Directory               |
|            [notes/2026-09-01-entry.md]                      |
|            [attachments/media-001.png]                      |
+-------------------------------------------------------------+

Processing occurs in memory streams, keeping RAM usage capped under 40 megabytes even when processing archives with tens of thousands of records.

Transformation Mapping Matrix

Incoming SaaS exports vary widely in naming conventions and data structures. The hydration engine uses declarative mapping tables to normalize source fields into canonical FreeNext note properties.

This mapping isolates upstream format quirks from the primary vault structure, ensuring all notes adhere to consistent organization rules.

Hydration Pipeline Implementation

The TypeScript worker implementation below illustrates batch stream processing from an extracted SQLite database cursor into individual vault files:

// hydration-worker.ts: Ingestion of FreeMyData tables
export interface IngestionConfig {
  tableName: string;
  idField: string;
  dateField: string;
  titleField: string;
  bodyField: string;
  tagField?: string;
  targetDirectoryHandle: FileSystemDirectoryHandle;
}

export async function hydrateVaultFromCursor(
  records: AsyncIterable<Record<string, unknown>>,
  config: IngestionConfig
): Promise<number> {
  let processedCount = 0;

  for await (const row of records) {
    const rawId = String(row[config.idField] || crypto.randomUUID());
    const title = String(row[config.titleField] || 'Untitled Note');
    const rawDate = row[config.dateField];
    const isoDate = normalizeDate(rawDate);
    const body = String(row[config.bodyField] || '');

    const tags: string[] = [];
    if (config.tagField && row[config.tagField]) {
      const rawTags = String(row[config.tagField]);
      tags.push(...rawTags.split(',').map(t => t.trim()).filter(Boolean));
    }

    const markdownContent = serializeNoteEnvelope({
      id: rawId,
      title,
      date: isoDate,
      tags,
      body,
    });

    const sanitizedFilename = sanitizeFilename(`${isoDate}-${title}.md`);
    await writeVaultFile(config.targetDirectoryHandle, sanitizedFilename, markdownContent);

    processedCount++;
  }

  return processedCount;
}

function normalizeDate(raw: unknown): string {
  if (typeof raw === 'number') {
    return new Date(raw).toISOString().split('T')[0];
  }
  if (typeof raw === 'string') {
    const parsed = Date.parse(raw);
    if (!isNaN(parsed)) {
      return new Date(parsed).toISOString().split('T')[0];
    }
  }
  return new Date().toISOString().split('T')[0];
}

function serializeNoteEnvelope(note: {
  id: string;
  title: string;
  date: string;
  tags: string[];
  body: string;
}): string {
  const yamlLines = [
    '---',
    `id: "${note.id}"`,
    `title: "${note.title.replace(/"/g, '\\"')}"`,
    `date: ${note.date}`,
  ];

  if (note.tags.length > 0) {
    yamlLines.push('tags:');
    for (const tag of note.tags) {
      yamlLines.push(`  - ${tag}`);
    }
  }

  yamlLines.push('---', '', `# ${note.title}`, '', note.body);
  return yamlLines.join('\n');
}

function sanitizeFilename(name: string): string {
  return name.replace(/[/\\?%*:|"<>]/g, '-').slice(0, 120);
}

async function writeVaultFile(
  dir: FileSystemDirectoryHandle,
  filename: string,
  content: string
): Promise<void> {
  const fileHandle = await dir.getFileHandle(filename, { create: true });
  const writable = await fileHandle.createWritable();
  await writable.write(content);
  await writable.close();
}

By connecting extraction and hydration via local pipelines, users migrate personal data out of cloud silos into structured Markdown vaults without vendor lock-in.