NovaNote

The problem

When LLMs first came out, they unlocked an ocean of opportunities for text-based processes. One of the most obvious high-value use cases was to combine the already existing Speech-to-Text (STT) technology with the power of text generation to autonomously generate meeting minutes from audio recordings.

While working at Novatix, we organised a discovery workshop with C-level executives and key stakeholders for a Swiss pension and insurance institution and surfaced this painful process: producing detailed meeting minutes from long general assembly meetings (sessions of four to five hours!), legally and organizationally sensitive, requiring capture of each participant’s interventions, voting decisions, and resolutions. The existing process took multiple hours, often spanning over days or weeks of work and still produced inconsistent results.

The initial plan was to build on Microsoft Power Apps to integrate into their existing Microsoft 365 ecosystem. We abandoned that approach quickly after hitting a hard ceiling: the platform’s 15 MB file upload limit made it unusable for multi-hour audio recordings. Besides, the developer experience on Power Apps is not great, so quitting it was a rather easy choice. That constraint forced us to build something custom, which turned out to be the right call: what started as a single-client project matured into one of Novatix’s flagship solutions, now deployed in production across more than 30 enterprise clients.

The task at hand demanded more than simple transcription. General assemblies are structured legal proceedings. Meeting minutes must attribute every intervention to the correct speaker, preserve the logical flow of debate, reflect decisions with precision, and be formatted according to the organization’s conventions. Off-the-shelf transcription tools don’t handle speaker attribution at this fidelity. General-purpose note-takers lack the configurability to respect each client’s workflow, vocabulary, and compliance requirements.

Additional constraints shaped the architecture:

• Data residency: Most clients required all processing to remain in Switzerland or the EU, meaning model and infrastructure selection had to be configurable at the deployment level.
• Deployment diversity: Clients ranged from Azure-hosted cloud deployments to on-premises OpenStack installations on InfoManiak (a Swiss sovereign cloud provider), requiring a cloud-agnostic infrastructure.
• Variable audio quality: Recordings went from high-quality boardroom setups to compressed Teams recordings.
• User adoption: The tool had to be usable by non-technical staff, with minimal friction and multilingual support.

My role

As lead developer and architect of a team of seven, I owned the end-to-end technical vision and oversaw every significant feature, architectural decision, and integration. Some of my responsibilities included:

• Designing the overall system architecture: multi-tenant API, async task pipeline, provider abstraction layers, and cloud-agnostic storage
• Creating the STT and LLM integration strategy, including the fallback logic and per-task model routing
• Overseeing the infrastructure design (Terraform on Azure, Docker Compose, multi-environment deployment packaging)
• Developing the multi-tenant data model and organization/team access control system
• Reviewing and approving all major pull requests and directing the technical growth of the team throughout
• Keeping an up-to-date backlog, prioritizing and delegating tasks across the team through recurrent sprint-like meetings
• Driving the product’s evolution from single-client project to a reusable, sellable Novatix solution

Technical approach

From audio recording to ready-to-share meeting minutes

To get from the initial audio file to the final document ready to be shared with colleagues, NovaNote was composed of various workflows and processes, some fully autonomous and others requiring human verification.

1. STT provider abstraction with fallback

Transcription is the first and most critical step. Rather than commit to one Speech-to-Text vendor, we designed a system that abstracts over four providers: Speechmatics, Azure Speech (standard and fast modes), and Gladia. We also had an internal benchmark to compare STT models not only on Word Error Rate (the “golden standard” for STT models), but also diarization accuracy, which is the act of attributing each utterance to the correct speaker (and is seldom evaluated on existing benchmarks!). This way we can use the first provider as primary model, and the other serve as fallbacks in case the first one fails.

For each client, we set a specific profile which defines the STT providers to use, LLM models, language settings, and summarization behavior. This abstraction was what allowed us to serve 30+ clients with radically different compliance requirements and infrastructure constraints from a single codebase. For example, Swiss-residency clients were routed to the available models in Azure Switzerland while performance-first clients could use Speechmatics for STT (EU processing) and the latest LLMs, not yet available for Swiss-based inference on Azure.

Internally, we discussed multiple times the idea of developing our own STT pipeline, using open-weight Whisper for transcription and a diarization processing based on PyAnnote, deploying on dedicated GPUs geographically located in Switzerland. While this would allow us to remove external providers (such as Speechmatics and Gladia) and improve the sovereignty of the solution, it was also a lot more work and would add the cost of maintaining our custom STT stack, including GPU provisioning. Therefore, the decision was made to postpone this development until a client would request this level of governance and would be willing to cover the costs of development.

2. Speaker attribution and diarization

For this project, the other important parameter in STT processing was diarization. How accurately an utterance can be assigned to the correct speaker matters as much as identifying the correct words, especially in meetings holding legally important material. However, for compliance reasons, specific voice profiles were not created, so NovaNote could identify a speaker by their voice tone but couldn’t suggest an actual name for them.

In NovaNote, we created a workflow which exposes a speaker attribution workflow where auto-detected speakers can be confirmed, merged, or manually corrected before summarization runs. A suggest_speakers task uses the LLM with meeting context to propose speaker identities based on naming patterns in the transcript, reducing manual correction effort, especially for recurring meetings with known participants.

For example, in this (simplified) sequence:

“Speaker_A: I agree with this decision. What about you, John?

Speaker_B: I do as well, Anna.”

NovaNote would suggest Speaker_A to be Anna, and speaker_B to be John.

Additionally, we extracted tiny voice snippets (holding no sensitive information) for each identified speaker, allowing the user to play the audio and recognize the participant by their voice.

Despite all this, one of the hardest problems we faced was that audio diarization quality varies significantly across providers and recording conditions. To further mitigate this issue, we started working on a speaker recognition feature where:

• Users could record themselves or other people and assign names to each recorded identity to improve future recognition by the model and automatically suggest the correct name, even if it wasn’t deductible in the transcript
• The audio snippets and user input for speaker attribution would be used to generate and improve vocal identities of the users over time

All of this was compliant with the fact of not storing any sensitive information on the STT provider, where the only accepted workflow was a quick transit of the recording to generate a transcription, successively deleted.

3. Section-by-section summarization

Meeting content is not summarized as a monolith. The system uses meeting templates: structured schemas of sections that the user can customize (e.g., “Decisions taken”, “Discussion items”, “Upcoming tasks”) and fills each section independently using the LLM. This produces coherent, well-organized meeting minutes rather than a wall of text. Additionally, we circumvent the main limitation of LLMs (attention) by forcing to generate one single section on a given topic at a time. With the most recent SOTA reasoning models, this may not seem like too much of an issue, but in 2024 (when NovaNote was first being used), asking an LLM to generate 5-10 pages of text on a wide range of topics from a 4 hour recording was an impossible task! The model’s attention mechanism would be all over the place, every section would feel like a lazy summary and getting the LLM to follow specific instructions (on tone, style and format) would get you nowhere.

Also, we are not bound by the maximum token number a model can generate, and can create virtually infinite content. And finally, this made it possible for users to define specific instructions for every section of their recurring templates: for example: “For the section ‘Upcoming Tasks’, always use a bullet point format including a brief description of the task, the person responsible and if possible, a deadline”.

Not to mention that this behavior allowed us to use lighter and faster (and cheaper!) models such as GPT-4o with acceptable drops in accuracy.

The final workflow looked like this:

Other notable features

AI writing assist and NoteBot

Beyond transcription, users can invoke AI writing assistance directly in a rich markdown editor (MDXEditor) for reformulation, expansion, or tone adjustment of any section. This is different than asking ChatGPT or Copilot to “rewrite this section”, because we would use the relevant context from the transcription in order to add or change parts in the meeting minutes. LLM capabilities minus the hallucinations.

NoteBot is a contextual AI assistant that answers questions grounded in the meeting content, enabling users to query specific decisions or interventions without re-reading the full transcript. Eventually NoteBot would become more general, living in the homepage and allowing to answer questions about any meeting or even pulling informations from multiple meetings together to answer a specific question (eg. “Trace the evolution of project X from its conception in March to its deployment in December”).

Both features route through the same LLM abstraction and respect the client’s model configuration.

Cloud-agnostic storage

Audio files and generated assets are stored via CloudPathLib, which provides a uniform interface over Azure Blob Storage, AWS S3, and local filesystem. Storage backend is selected at deployment time through environment configuration, requiring no code changes between cloud and on-premises deployments.

Multi-tenant data model and access control

The data model is organized around organizations, teams, and meetings. Role-based access is enforced at the API layer, with team-level sharing of meetings and organization-level administration. Authentication supports Azure Entra ID for enterprise clients embedded in the Microsoft 365 ecosystem or other providers through Keycloak.

Infrastructure and deployment packaging

Infrastructure is managed with Terraform targeting Azure (Container Apps, PostgreSQL Flexible Server, Azure Blob, Entra ID app registration, autoscaling policies). Docker Compose handles local and on-premises deployments. The same application was deployed on InfoManiak OpenStack for Swiss sovereign cloud clients without modification to the application layer — only the Terraform target and storage backend changed.

And more…

As NovaNote evolved, more and more features were integrated to accommodate each client’s requests and ideas. Here are some more things we did:

1. Microsoft Teams webhook integration Meetings recorded in Teams are automatically ingested, transcribed, and processed. With error-retry logic. This was huge for Microsoft ecosystem clients.
2. DOCX export Critical for enterprise workflows, people need the output in Word, not just in the browser.
3. Meeting presets Users can save reusable meeting configurations: participants, language, template, vocabularies, section length, etc. A UX feature mainly for recurring meetings (e.g. a weekly board meeting always has the same setup).
4. Fine-grained meeting sharing Meetings can be shared with specific users at defined access levels to allow to collaborate on meeting minutes.
5. PDF agenda extraction Upload a PDF and the system extracts the agenda items from it automatically. Useful for formal meetings that already have a printed agenda.
6. Usage statistics dashboard Model usage tracking, token consumption, and estimated price per minutes. Enterprise billing/reporting feature.

Tech stack

Outcome

• 8+ hours → under 20 minutes for producing a complete, structured meeting minutes document, reported consistently across client organizations
• 30+ enterprise clients in production, across Azure, InfoManiak (Swiss sovereign cloud), and on-premises OpenStack deployments
• Full data residency compliance achieved for every client, including Swiss-only and EU-only configurations, without application-layer changes
• Minimal vendor lock-in at every layer: STT, LLM, storage, and cloud infrastructure are all replaceable through configuration
• 5+ LLM model endpoints configurable per deployment, with per-task routing and automatic fallback
• 4 STT provider integrations with runtime fallback, no single provider failure can break the transcription pipeline
• Average of 1 CHF per meeting minutes generated, although this could vary a lot depending on length of meeting and generated text

What I’d do differently

NovaNote started as a project for a single client and then evolved into one of Novatix’s flagship solutions. For every new client there would be new requests and new features to develop. Combining specific feature development, NovaNote’s general updates and all the various deployment and compliance modalities was not always a simple task.

If I were to start over, I would spend more time researching the market and realizing that the demand for a Swiss-based meeting minutes platform was this important. It would have allowed us to predict NovaNote’s growth and design its foundations around a “SaaS-like” product rather than a client-specific web app. We could have gained a lot of time if we developed things like feature flags, automated releases and updates, processing pipeline profiles and multi-tenant / multi-infra deployments from the beginning. But, as a wise man’s mother once said:

“Life is like a box of chocolates, you never know what you’re going to get!”.