Notes from Working on Multi-Tenant Architecture

17 min

infrastructure

Around this time last summer, I was working on okiyani.com. More specifically, in July, I was focused on a set of multi-tenant architecture challenges that felt both unique and fun, and I thought I’d share them.

A bit of context

Okiynai is a startup that I and my friend, @golden, were working on for the majority of 2025. It was an AI-powered e-commerce platform where you can build your storefront just like Shopify, and have a dashboard to manage your stocks, SKUs, and the usual e-commerce stuff, and you had AI in all of those parts. Like an AI builder for the storefront. And an AI store manager. Now unfortunately, it did fail, and we decided to open source most of the stuff on GitHub. But they did not include the cool infrastructure parts, and that’s what this blog post is about.

Domains setup overview

okiynai.com was the main platform, there you have your dashboard. api.okiynai.com is obvious1. *.okiynai.shop was where storefronts of tenants lived. custom domain were connected through a CNAME pointing to shops.okiynai.shop and routed to the correct storefront based on the incoming Host header.

A little cookie trait

When I first started working on authentication, tenant storefronts still lived under *.okiynai.com, before we eventually moved them to *.okiynai.shop. This was one of the first pieces of our multi-tenant infrastructure, and at the time, we had no idea how many browser and cookie behaviours would come with it.

So, to reproduce that setup locally, we ran the Nextjs app on localhost:3000 and accessed each storefront through *.localhost:3000.

To route those tenant subdomains to storefronts, we could've used nginx as a reverse proxy, but we went with a simple solution from Nextjs middleware:

// src/middleware.ts

if (subdomain) {
	// rewrite to the subdomain page
	let response = NextResponse.rewrite(new URL(`/shops/${subdomain}${pathname}`, request.url));
	response.cookies.set('subdomain', subdomain, {
		httpOnly: false,
		sameSite: 'lax',
		path: '/',
	});
	response.headers.set('x-shop-name', subdomain);
	return response;
}

Either way, the browser will see that we are sending requests from tenantX.localhost:3000.

Now you might think that this setup is correct, mirroring what we have in on the prod, but in reality it's far from the truth.

See, there are two cookie properties that we care about here: Domain and SameSite.

Cookies are host-only by default. So if api.okiynai.com sets a cookie without a Domain property, that cookie will only be sent back to api.okiynai.com. But if it sets Domain=okiynai.com, the cookie can be sent to okiynai.com and all of its subdomains.

This does not mean that sibling subdomains can directly set cookies for each other. hossam.okiynai.com cannot set a cookie specifically for golden.okiynai.com. What it can do is set one for their shared parent, okiynai.com, which causes the browser to send it to both of them.

Then there is SameSite, which controls whether the browser sends a cookie when a request originates from another site.

Strict only sends the cookie in same-site contexts. Lax, which is the default in modern browsers, also allows it during some cross-site top-level GET navigations, but not regular cross-site API calls. None allows the cookie to be sent in cross-site contexts, but it requires the cookie to also be marked as Secure.

A site is determined using the scheme and the publicly registerable domain. Subdomains and ports do not make separate sites. But that did not exist in this setup, because localhost is not a publicly registerable domain. localhost and tenant.localhost were therefore treated as different sites.

And that caused two issues:

  1. In production, all of our original *.okiynai.com tenants shared the same parent. A cookie scoped to Domain=okiynai.com was sent to every tenant, while api.okiynai.com, being a sibling, could not set a cookie for one specific tenant. Locally, every tenant.localhost appeared to have isolated authentication, so I completely missed this behaviour.
  2. Since tenant.localhost and the API on localhost were different sites, SameSite=Lax and SameSite=Strict cookies were not sent with normal API requests. So the local setup could not test the authentication flow at all.

A Production-Like Local Setup

As we saw, keeping every storefront under *.okiynai.com meant they all shared the same parent domain, which made tenant-scoped authentication difficult. This eventually led us to get okiynai.shop and move the storefronts to *.okiynai.shop.

Now that production had two separate sites: okiynai.com for the platform and API, and okiynai.shop for the storefronts, we needed our local environment to reproduce that same boundary.

So we stopped using localhost and switched to reserved testing domains. Nginx would receive requests for those domains and proxy them to the applications already running locally:

# Redirect the main app and API from HTTP to HTTPS
server {
	listen 8080;
	server_name devokiynai.test;
	return 301 https://$host:8443$request_uri;
}

server {
	listen 8080;
	server_name api.devokiynai.test;
	return 301 https://$host:8443$request_uri;
}

# Redirect storefronts from HTTP to HTTPS
server {
	listen 8080;
	server_name *.devokiynai.example;
	return 301 https://$host:8443$request_uri;
}

# Main Next.js app
server {
	listen 8443 ssl;
	server_name devokiynai.test;
	
	ssl_certificate ssl/localhost.crt;
	ssl_certificate_key ssl/localhost.key;
	
	location / {
	  proxy_pass http://localhost:3000;
	  proxy_set_header Host $host;
	  proxy_set_header X-Forwarded-Proto $scheme;
	}
}

# Express API
server {
	listen 8443 ssl;
	server_name api.devokiynai.test;
	
	ssl_certificate ssl/localhost.crt;
	ssl_certificate_key ssl/localhost.key;
	
	location / {
	  proxy_pass http://localhost:3007;
	  proxy_set_header Host $host;
	  proxy_set_header X-Forwarded-Proto $scheme;
	}
}

# Tenant storefronts
server {
	listen 8443 ssl;
	server_name *.devokiynai.example;
	
	ssl_certificate ssl/localhost.crt;
	ssl_certificate_key ssl/localhost.key;
	
	location / {
	  proxy_pass http://localhost:3000;
	  proxy_set_header Host $host;
	  proxy_set_header X-Forwarded-Proto $scheme;
	}
}

So our production domains mapped to the local setup like this:

okiynai.com       => devokiynai.test:8443
api.okiynai.com   => api.devokiynai.test:8443
*.okiynai.shop    => *.devokiynai.example:8443

Authentication Across Tenant Domains

With the local environment finally matching production, we could start developing our auth.

Email and password

For email-password authentication, the user submits a form directly to the expressjs backend, including a hidden field that has the tenant domain as a value. The Expressjs backend then verfies the user credentials then generates a short-lived exchange token23. Then it returns an auto-submitting HTML form, the idea of using this here is to perform a browser top-level navigation, so that when go to the tenant's Nextjs backend's callback endpoint. After that it's a simple exchnge token and verify it server-to-server communication between the Nextjs and Expressjs backend, and we just return the response from the Nextjs backend that we are on that has the Set-Cookie header.

The Expressjs side handled the initial sign-in verfication, token generation, the browser navigation. And the later token exchange:

// routes/sso.ts

router.post("/signin", async (req, res) => {
	const { email, password, tenant_domain } = req.body;
	
	const tenant = await findTenantByDomain(tenant_domain);
	if (!tenant) {
		return res.status(400).json({
			error: "Invalid tenant domain",
		});
	}
	
	const currentUser = await findUserByEmail(email);
	if (!currentUser) {
		return res.status(401).json({
			error: "Invalid email or password",
		});
	}
	
	const passwordIsValid = await verifyPassword(currentUser, password);
	
	if (!passwordIsValid) {
		return res.status(401).json({
			error: "Invalid email or password",
		});
	}
	
	const exchangeToken = generateExchangeToken(currentUser.id, tenant.domain, currentUser.email);
	
	return res.type("html").send(`
		<form id="ssoForm" method="POST" action="https://${tenant.domain}/api/auth/callback">
			<input type="hidden" name="code" value="${exchangeToken}">
			<input type="hidden" name="tenant" value="${tenant.domain}">
		</form>

		<script>
			document.getElementById("ssoForm").submit();
		</script>
	`);
});

router.post("/exchange", async (req, res) => {
	const { code, tenant_domain } = req.body;
	const decoded = validateExchangeToken(code);
	
	if (decoded.tenant !== tenant_domain) {
		return res.status(400).json({
			error: "Tenant domain mismatch"
		});
	}
	
	const wasConsumed = await consumeExchangeToken(decoded.nonce);
	if (!wasConsumed) {
		return res.status(401).json({
			error: "Invalid or already used exchange token",
		});
	}
	
	const sessionToken = generateSessionToken();
	await createSession(sessionToken, decoded.sub);
	
	return res.json({
		data: { session_token: sessionToken }
	});
});

The callback endpoint on the tenant's Nextjs backend completed the flow:

// api/auth/callback/route.ts

export async function POST(request: NextRequest) {
	const params = new URLSearchParams(await request.text());

	const code = params.get("code");
	const tenant = params.get("tenant");

	const exchangeResponse = await fetch(`${backendUrl}/api/v1/sso/exchange`, {
		method: "POST",
		headers: {
			"Content-Type": "application/json"
		},
		body: JSON.stringify({
			code,
			tenant_domain: tenant
		})
	});

	const { data } = await exchangeResponse.json();

	const response = NextResponse.redirect(`https://${request.headers.get("host")}/`);

	response.headers.set("Set-Cookie", `session=${data.session_token}; HttpOnly; Secure; SameSite=Lax; Path=/`);

	return response;
}

Google OAuth

Google OAuth only changed the first half of the flow.

Google requires the callback URL to be registered beforehand, so we couldn't use a different callback for every tenant subdomain and custom domain. The OAuth flow had to begin and end at the centralized Expressjs API.

The storefront therefore sent the browser to the central Google sign-in route with sso=true and its tenant domain:

// components/auth/index.tsx

function buildGoogleSsoUrl(redirectUrl: string) {
	const url = new URL("/api/v1/auth/signin/google", getDirectApiUrl());
	
	const tenant = new URL(redirectUrl);
	
	url.searchParams.set("sso", "true");
	url.searchParams.set("tenant_domain", tenant.host);
	
	return url.toString();
}

Expressjs generated the usual CSRF state and PKCE code verifier. In SSO mode, it also encoded the tenant domain into the OAuth state and stored the SSO context temporarily in cookies:

// routes/auth.ts

router.get("/signin/google", async (req, res) => {
	const isSSO = req.query.sso === "true";
	const tenantDomain = req.query.tenant_domain;
	
	const csrfState = generateState();
	const codeVerifier = generateCodeVerifier();
	
	let state = csrfState;
	
	if (isSSO && tenantDomain) {
		const encodedTenant = Buffer.from(tenantDomain).toString("base64");
	
		state = `${csrfState}|${encodedTenant}`;
	}
	
	const googleUrl = await google.createAuthorizationURL(state, codeVerifier, {
		scopes: ["profile", "email"]
	});
	
	res.cookie("google_oauth_state", csrfState, cookieOptions);
	res.cookie("google_oauth_code_verifier", codeVerifier, cookieOptions);
	res.cookie("sso_tenant_domain", tenantDomain, cookieOptions);
	
	return res.redirect(googleUrl.toString());
});

The browser then went through the normal Google OAuth flow and returned to the centralized callback:

Once Expressjs verified Google's response and identified the user, the rest was identical to email-password authentication. It generated the same exchange token and returned the same auto-submitting form:

// routes/auth.ts

router.get("/callback/google", async (req, res) => {
	const tenantDomain = req.cookies.sso_tenant_domain;
	
	// Verify the OAuth state and PKCE code verifier,
	// exchange Google's code and find or create the user.
	
	const exchangeToken = generateExchangeToken(user.id, tenantDomain, user.email);
	
	return res.type("html").send(`
		<form id="ssoForm" method="POST" action="https://${tenantDomain}/api/auth/callback">
			<input type="hidden" name="code" value="${exchangeToken}">
			<input type="hidden" name="tenant" value="${tenantDomain}">
		</form>

		<script>
			document.getElementById("ssoForm").submit();
		</script>
	`);
});

From here, we performed that same browser navigation, and the rest is the same as username-password auth.

This flow worked, but it left Nextjs sitting between the browser and Expressjs for the final exchange, adding what now appears to be an unnecessary extra round trip, which brings us to how I would build it today

Looking back at it now

While writing this blog, I was discussing this solution with Codex and suggesting alternatives, and we found out that this flow could've been waaay simpler. For example, we could've used an Nginx routing rule like this:

upstream express_backend {
  server nerv:3007;
}

upstream nextjs_frontend {
  server frontend:3000;
}

server {
  listen 443 ssl;
  server_name *.okiynai.shop;

  # Custom-domain server blocks could use the same locations.

  location /api/ {
	  proxy_pass http://express_backend;

	  proxy_set_header Host $host;
	  proxy_set_header X-Forwarded-Host $host;
	  proxy_set_header X-Forwarded-Proto $scheme;
	  proxy_set_header X-Real-IP $remote_addr;
	  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }

  location / {
	  proxy_pass http://nextjs_frontend;

	  proxy_set_header Host $host;
	  proxy_set_header X-Forwarded-Host $host;
	  proxy_set_header X-Forwarded-Proto $scheme;
  }
}

So that we could talk directly to the Expressjs backend. That way, we wouldn't need to use the Nextjs backend as a proxy. Nginx would act as the reverse proxy instead.

That would've made the username-password authentication waaay simpler.

Though Google OAuth would still need one extra hop. Because Google requires the OAuth callback URL to be registered beforehand. We cannot register every tenant subdomain and arbitrary custom domains, so the callback still had to go through one stable, centralized URL.

But even then, we could've kept the flow only until the centralized callback returned the auto-submitting form. That form would perform a top-level navigation to something like https:// tenant.okiynai.shop/api/oauth/complete. Nginx would route that request straight back to the same Expressjs backend, which could verify and consume the short-lived exchange code, create the session, and respond directly with a Set-Cookie header and a redirect to /. So we would still have the browser navigation required to reach the tenant domain, but without the additional Nextjs-to-Expressjs exchange.

Custom Domains Onboarding

This is more of an honorable mention part. Unlike the authentication flow, this one did not require the same kind of architectural thinking. It was mostly a known sequence of steps: verify DNS, serve the ACME challenge, issue the certificate, write the final Nginx config, and reload.

Also, by the time I worked on this part, I had already learned a decent bit about DNS, SSL, Nginx, Cloudflare, and the general cloud infrastructure side of things. So unlike the earlier auth days, this was less of a "learn as you go" situation, and more of just applying the pieces step by step.

The start is the user-facing flow, it was simple:

  1. The merchant adds their custom domain.
  2. We tell them to create a CNAME record in their domain registrar4.
  3. That CNAME should point to shops.okiynai.shop.
  4. Once DNS is verified, we provision SSL.
  5. Then Nginx starts serving that domain as a real storefront domain.

The backend first verified that the domain actually pointed to us:

// services/domain/domainService.ts

export const CNAME_TARGET = 'shops.okiynai.shop';

export async function verifyDNS(domain: string) {
	const records = await resolveCname(domain);

	const pointsToUs = records.some(
		record =>
			record.toLowerCase() === CNAME_TARGET.toLowerCase() ||
			record.toLowerCase().endsWith(`.${CNAME_TARGET.toLowerCase()}`)
	);

	if (pointsToUs) {
		return {
			verified: true,
			message: 'DNS verified successfully',
			records,
		};
	}

	return {
		verified: false,
		message: `Your domain is pointing to ${records[0]} instead of ${CNAME_TARGET}.`,
		records,
	};
}

Once DNS was verified, we kicked off the SSL provisioning workflow in the background:

// routes/manager/domain.ts

const dnsResult = await verifyDNS(domain.domain);

if (dnsResult.verified) {
	await updateCustomDomainStatus(domainId, 'dns_verified', {
		dnsVerified: true,
		verifiedAt: new Date(),
	});

	fullSSLProvisioningWorkflow(
		domain.domain,
		shopSubdomain,
		async (status, errorMessage) => {
			if (status === 'active') {
				await updateCustomDomainStatus(domainId, 'active', {
					sslIssued: true,
				});
			} else if (status === 'error') {
				await updateCustomDomainStatus(domainId, 'error', {
					errorMessage,
				});
			} else {
				await updateCustomDomainStatus(domainId, status as any);
			}
		}
	);

	await updateCustomDomainStatus(domainId, 'provisioning_ssl');
}

The infrastructure part was basically a small Nginx config generator.

The first config was HTTP-only. This existed before the SSL certificate was issued, because Let's Encrypt needs to hit /.well-known/acme-challenge/... over HTTP to verify that we actually control the domain.

// services/domain/infrastructureService.ts

export function generateHttpConfig(
	domain: string,
	shopSubdomain: string
): string {
	return `# Auto-generated config for ${domain}
# Stage: HTTP-only (awaiting SSL)

server {
	listen 80;
	server_name ${domain};

	# Let's Encrypt ACME challenge
	location /.well-known/acme-challenge/ {
		root /var/www/certbot;
	}

	# Temporary redirect to subdomain while SSL is being provisioned
	location / {
		return 302 https://${shopSubdomain}.okiynai.shop$request_uri;
	}
}
`;
}

So while SSL was still being provisioned, the custom domain was not fully active yet. It could answer the ACME challenge, and everything else was temporarily redirected back to the normal *.okiynai.shop storefront.

After writing that config, we reloaded Nginx, then ran Certbot:

// services/domain/infrastructureService.ts

export async function reloadNginx() {
	const testResult = await testNginxConfig();

	if (!testResult.success) {
		return {
			success: false,
			error: `Config test failed: ${testResult.error}`,
		};
	}

	await execAsync(`docker exec ${CONFIG.NGINX_CONTAINER} nginx -s reload`);

	return { success: true };
}

export async function provisionSSLCertificate(domain: string) {
	const cmd = `docker exec ${CONFIG.CERTBOT_CONTAINER} certbot certonly \
		--webroot \
		-w /var/www/certbot \
		-d ${domain} \
		--email ${CONFIG.ADMIN_EMAIL} \
		--agree-tos \
		--non-interactive \
		--expand`;

	await execAsync(cmd, { timeout: 120000 });

	return { success: true };
}

One important detail here is how the backend was controlling the Nginx and Certbot containers.

The Expressjs backend itself was running inside Docker, but it still needed to reload the custom-domains Nginx container and run Certbot commands. So the backend container had access to the Docker CLI, and on the VPS we mounted the host Docker socket into it.

In docker-compose.yml, it looked roughly like this:

nerv: # our Expressjs backend
	volumes:
	  - ./custom-domains/nginx/conf.d:/home/avramcohen/apps/okiynai/custom-domains/nginx/conf.d
	  - /var/run/docker.sock:/var/run/docker.sock
	environment:
	  CUSTOM_DOMAINS_BASE: /home/avramcohen/apps/okiynai/custom-domains
	  NGINX_CONTAINER: nginx-custom-domains
	  CERTBOT_CONTAINER: certbot-custom-domains

This pattern is called Docker-outside-of-Docker. The backend was not running another Docker daemon inside itself. It was just using the mounted Docker socket to talk to the host Docker daemon.

The VPS structure looked roughly like this:

apps/okiynai/
├── docker-compose.yml
└── custom-domains/
    ├── nginx/
    │   ├── nginx.conf
    │   └── conf.d/
    │       └── shop.example.com.conf
    └── certbot/
        ├── www/
        └── conf/
            ├── live/
            ├── archive/
            └── renewal/

The backend wrote one Nginx config file per custom domain under custom-domains/nginx/conf.d.

Once Certbot succeeded, we replaced the temporary HTTP config with the real HTTPS config:

// services/domain/infrastructureService.ts

export function generateHttpsConfig(domain: string): string {
	return `# Auto-generated config for ${domain}
# Stage: HTTPS (SSL active)

server {
	listen 80;
	server_name ${domain};

	location /.well-known/acme-challenge/ {
		root /var/www/certbot;
	}

	location / {
		return 301 https://$host$request_uri;
	}
}

server {
	listen 443 ssl http2;
	server_name ${domain};

	ssl_certificate /etc/letsencrypt/live/${domain}/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/${domain}/privkey.pem;

	location / {
		proxy_pass http://${CONFIG.FRONTEND_CONTANIER_NAME}:${CONFIG.APP_PORT};
		proxy_set_header Host $host;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Proto $scheme;
	}
}
`;
}

And the full workflow was just those steps glued together:

// services/domain/infrastructureService.ts

export async function fullSSLProvisioningWorkflow(
	domain: string,
	shopSubdomain: string,
	onStatusChange?: (status: string, error?: string) => Promise<void>
) {
	const httpConfig = generateHttpConfig(domain, shopSubdomain);
	await writeNginxConfig(domain, httpConfig);
	await reloadNginx();

	await onStatusChange?.('provisioning_ssl');

	const sslResult = await provisionSSLCertificate(domain);
	if (!sslResult.success) {
		await onStatusChange?.('error', sslResult.error);
		return sslResult;
	}

	const httpsConfig = generateHttpsConfig(domain);
	await writeNginxConfig(domain, httpsConfig);
	await reloadNginx();

	await onStatusChange?.('active');

	return {
		success: true,
		output: `SSL provisioned and nginx configured for ${domain}`,
	};
}

Here's it in a diagram:

User adds custom domain
        ↓
User creates CNAME to shops.okiynai.shop
        ↓
Backend verifies DNS
        ↓
Backend writes temporary HTTP Nginx config
        ↓
Nginx reloads
        ↓
Certbot verifies the ACME challenge
        ↓
Backend writes HTTPS Nginx config
        ↓
Nginx reloads again
        ↓
Domain becomes active

Some core parts were missing here, like renewals, but this was mainly a proof-of-concept, and it came after we decided to shutdown okiynai. I only did it because I thought this is gonig to be a fun technical exercise.