Engineering Breakdown
AI-powered SOP generation platform that helps teams turn scattered documents, PDFs, screenshots, and project materials into structured Standard Operating Procedures.
Role: Designed and implemented the backend architecture, document ingestion pipeline, job lifecycle, database model, and AI generation workflow.
Teams receive project materials from different sources: Word documents, PDFs, screenshots, diagrams, and notes. Turning all of that into a consistent SOP is slow, manual, and easy to get wrong when the source material is scattered across multiple files and formats.
Trailwren has two asynchronous workflows. First, uploaded documents are ingested and converted into Markdown. Later, when the user chooses to generate an SOP for a project, the system combines the extracted project documents and runs the AI generation pipeline.
Instead of sending raw PDFs, DOCX files, screenshots, and project materials directly into the AI flow, Trailwren extracts a consistent Markdown representation through a dedicated document extraction service. That gives the SOP generator cleaner input and avoids repeatedly parsing the same files.
Documents are processed individually as they are uploaded, but SOP generation only starts when the user chooses to generate for a project. This lets users add or replace documents incrementally without triggering unnecessary AI work.
Document extraction is independent of the upload request. After the file and metadata are stored, goroutines and channels dispatch extraction work to a dedicated extraction service without blocking additional uploads.
Generating an SOP depends on multiple extracted documents and an AI model that may take significant time. Asynq keeps this work out of the HTTP request lifecycle and gives the system queueing, retries, and worker-based execution.
Each generation request creates a PostgreSQL job record so users can monitor progress independently of the original request and the system can recover from failures more predictably.
Project ownership checks, document records, and job updates need to stay consistent. Transactions help prevent partial database state when validation, record creation, or status updates fail.
Requests are checked against project ownership before documents are created or SOP jobs are started, so users can only operate on resources that belong to them.
Worker execution and job lifecycle events are logged so asynchronous processing can be traced during development and production troubleshooting.
My initial implementation treated generation like a straightforward request: receive the project materials, process the documents, call the AI model, and return the SOP. That worked for small inputs, but it exposed timeout and user experience problems as generation became slower. Splitting the system into ingestion and generation pipelines made the work easier to reason about, retry, and scale.
Background workers allow the API to return quickly while extraction and generation continue independently. Worker concurrency can be tuned without scaling the HTTP API, which lets compute-heavy document and AI work scale separately from request handling.
Trailwren favors explicit job records and background workers over a simpler synchronous request flow. That adds more moving parts, but it gives the system clearer state, better retry behavior, and a cleaner separation between user-facing requests and long-running document or AI processing work.