Someone is going to submit a fork bomb to your platform. Someone else will try to read /etc/passwd, open a reverse shell, or mine crypto on your dime. If you want to build your own online judge platform, the coding challenges are the easy part. The moment you run untrusted code from strangers, security stops being a feature and becomes the whole product.
Get the isolation right and everything else, the API, the database, the editor, is just plumbing. Get it wrong and one clever submission takes down your cluster.
TL;DR:
- An online judge accepts user code, runs it against hidden test cases, and returns a verdict (Accepted, Wrong Answer, TLE, and so on).
- The core challenge is code execution isolation: running untrusted code without letting it escape, steal data, or exhaust your machine.
- Use an asynchronous architecture: an API to accept submissions, a queue to buffer them, and a pool of sandboxed workers to judge them.
- Docker gives you a fast start, but a shared kernel is a real escape risk. Firecracker microVMs give each submission its own kernel.
- You can stand up a working online judge sandbox in a weekend by leaning on proven tools like Judge0 and per-VM isolation.
Building Your Own Online Judge: Why the Sandbox Is the Hard Part
An online judge compiles and executes the code you submit, feeds it prepared inputs, captures the output, and compares it against an expected answer within a time and memory limit. If your program blows past the time limit, you get a TLE (Time Limit Exceeded) verdict. Match the expected output inside the limits and you get Accepted.
That flow, described plainly in this low-level design writeup, sounds simple. The trap is the word "execute."
You are running arbitrary code written by people you will never meet, some of whom actively want to break your system. As one engineer who has operated a semi-large online judge for four years puts it, the first hard requirement when you build your own online judge platform is a sandbox to run user code in. And it has to be cheap enough that servers do not bankrupt you.
The sandbox is not a component of an online judge. It is the reason the rest of the system is allowed to exist.
The Architecture You Actually Need
Almost every serious design converges on the same shape. Judging is slow, bursty, and dangerous, so you never run it inline with an HTTP request.
Here is the pattern that holds up in production:
- API layer. A user submits code. The API records the submission in your database, drops it onto a queue, and returns a token the client can poll later. That async handoff, described well in this LeetCode-style system design, keeps your web tier fast when 500 people submit at once during a contest.
- Queue. A message queue buffers submissions so a traffic spike becomes a longer wait, not a crash. It also decouples your API from your judging capacity, so you can scale workers independently.
- Code execution engine. A pool of workers polls the queue, pulls a submission, runs it inside a sandbox, and parses the result (exit code, runtime, memory, stdout).
- Database. PostgreSQL is the common pick for users, problems, test cases, and submission history. Node.js is a popular API layer, though the language barely matters here.
For the backend of a LeetCode clone, a microservices split (auth, problems, submissions, judging) pays off once you have contests and real traffic. If you are learning, a modular monolith is fine. Do not let architecture astronomy delay you from shipping.
(Image: a flow diagram showing submission moving from API to queue to sandboxed worker to database and back as a verdict token, alt: 'online judge platform architecture with queue and code execution sandbox')
If you would rather not write the judging engine yourself, Judge0 is an open-source, sandboxed code execution system with an HTTP JSON API and a Python SDK. It handles many languages out of the box. Many teams wrap Judge0 for execution and write only the problem, contest, and user layers themselves.
How to Run User Code Safely in the Cloud Without Getting Owned
This is where the money is. To run user code safely in the cloud, assume every submission is hostile and design the blast radius accordingly.
Your sandbox needs to enforce, at minimum:
- CPU and wall-clock time limits, so an infinite loop dies instead of hogging a core forever.
- Memory limits, so a submission cannot allocate its way into an OOM kill of the whole host.
- No network access by default. A judged program has zero business making outbound calls.
- A read-only, throwaway filesystem scoped to that one run.
- Dropped privileges and restricted syscalls, so the process cannot escalate or touch the host.
Docker is the go-to for a fast start because it is easy to deploy, and most tutorials, including this MEAN-stack online judge build, reach for containers when you want to build your own online judge platform. But be honest about the trade-off. Docker containers share the host kernel. A security study of online judge sandboxes catalogs real attack methods against exactly this model. One kernel vulnerability and a container escape puts an attacker on your host, next to every other user's code.
The strongest isolation gives each submission its own kernel, not just its own namespace. That is what microVMs do. Firecracker boots a lightweight VM with its own Linux kernel, so you get VM-grade isolation at close-to-container speed.
This is the model Krova Cloud is built on when you choose to build your own online judge platform. Every Cube is a Firecracker microVM with its own kernel inside a hardened per-cube jailer, no public IP by default, and full root SSH. You can spin up a fresh microVM per judging worker, so a container escape has nowhere to go. It is trapped in a VM with no network exposure. If you are weighing this against running your own hosts, our breakdown of tenant isolation and why your SaaS architecture needs it covers the same defense-in-depth thinking.
The Mistakes That Sink Most LeetCode Clone Backends
I have watched more than one weekend project turn into an incident. The failure modes rhyme.
Running the judge inside your API process. One infinite loop and your web server is frozen for everyone. Always push execution to isolated workers behind a queue.
Trusting Docker alone as your security boundary. It is a good sandbox for honest mistakes, a weak one for a determined attacker. Layer it: drop capabilities, disable networking, run rootless, and put a VM boundary underneath if you are exposing this publicly.
No resource ceilings. Without hard CPU, memory, and process limits, a single submission (or a fork bomb) can starve the host. Set limits per run and enforce them at the kernel or hypervisor level, not in application code.
Leaving state between runs. Reusing a container or VM lets one submission plant a file the next one reads. Destroy and recreate the sandbox per submission. With fast microVM boots and per-minute billing that stops the instant a Cube powers off, disposable isolation is cheap enough to actually do.
Over-engineering before you have users. You do not need Kubernetes to judge 50 submissions a day. Start simple, isolate hard, scale when the queue backs up.
Ship a Working Online Judge Sandbox This Weekend
Here is a concrete path that gets you from zero to judging real submissions fast:
- Model the domain. Users, problems, hidden test cases, submissions. PostgreSQL, one schema. Our guide on self-hosting Postgres on an isolated microVM is a clean starting point.
- Build the submit endpoint. Accept code plus language, store it, enqueue a job, return a token. Poll for the verdict.
- Stand up execution. Self-host Judge0 for the engine, or write a thin worker that runs each submission in a fresh, network-disabled sandbox with strict CPU and memory limits.
- Add the isolation layer. Put each worker (or each run) inside its own Firecracker microVM so a breakout has its own kernel and no public IP to phone home from.
- Wire the verdict flow. Compare captured stdout to expected output within limits, write Accepted / WA / TLE back to the database, surface it to the client.
Do steps 1 through 3 first to feel the loop working. Do step 4 before you let a single stranger near it. Isolation is not the last feature you add, it is the foundation you build on.
FAQ
Is Docker Enough to Run User-Submitted Code Safely?
For honest mistakes and casual use, yes. For a public platform facing determined attackers, no. Docker containers share the host kernel, so a kernel exploit can lead to a container escape. Layer strict resource limits, dropped privileges, and disabled networking on top, and put a microVM boundary underneath for a real security perimeter.
What Is the Best Sandbox for an Online Judge?
The best online judge sandbox gives each submission its own kernel and no network access, then destroys itself after the run. Firecracker microVMs deliver that VM-grade code execution isolation while still booting quickly, which is why platforms like Krova Cloud use the same technology.
Do I Have to Write the Code Execution Engine from Scratch?
No. Judge0 is an open-source, sandboxed execution system with an HTTP JSON API and a Python SDK that handles many languages already. Most teams building a LeetCode clone backend wrap Judge0 for execution and write only the problem, contest, and user-management layers themselves.
Why Use a Queue Instead of Running the Judge Inline?
Judging is slow and bursty, especially during contests. A queue lets your API accept submissions instantly and return a token, while a separate pool of sandboxed workers judges at its own pace. That keeps your web tier responsive and lets you scale judging capacity independently.
