Building my own screen-recording app

For a recent job interview I had to record my screen completing a code challenge.

The obvious solution would be to grab a tool like loom but they require payment for recording video. So I thought to myself, I’ve got ffmpeg, a browser, node, an LLM and 2 hours time, I can build this myself 😅

Here’s a video of the final output (without any sound for this demo)

I’ve worked with MediaCapture API before, so to add some extra challenge I built the frontend with Svelte.

Table of contents

A simple screen recorder with svelte

To get started with Svelte we’ll start with a simple boilerplate by using the CLI

npm create svelte@latest 

Similar to Vue.JS templates, Svelte works with scripts, styles and templates in a single file.

Let’s set up some state in our src/App.svelte file!
In svelte this is as simple as creating a let statement:

<script lang="ts">
  let screenStream, webcamStream;
  let screenRecorder, webcamRecorder;

Let’s a write a function to record!
For now we can take the audio stream from the webcam and combine it into the screen stream.

  async function startRecording() {
    screenStream = await navigator.mediaDevices.getDisplayMedia({
      video: true,
      audio: true,
    });
    webcamStream = await navigator.mediaDevices.getUserMedia({
      video: true,
      audio: true,
    });
 
    // Combine the screen stream with the audio from the webcamStream
    const tracks = [
      ...screenStream.getVideoTracks(),
      ...webcamStream.getAudioTracks(),
    ];
    const combinedStream = new MediaStream(tracks);
    
    // Stop all tracks from both streams
    screenStream.getTracks().forEach((track) => track.stop());
    webcamStream.getTracks().forEach((track) => track.stop());
  }
</script>

In the template we show a very simple Record / Stop Recording button with the final video output files.

<main>
  <button on:click={startRecording}>Start Recording</button>
  <button on:click={stopRecording}>Stop Recording</button>
 
  {#if screenVideoUrl}
    <div>
      <h2>Screen Recording</h2>
      <video src={screenVideoUrl} controls width="320"></video>
      <a href={screenVideoUrl} download="screen-recording.mp4"
        >Download Screen Recording</a
      >
    </div>
  {/if}
 
  {#if webcamVideoUrl}
    <div>
      <h2>Webcam Recording</h2>
      <video src={webcamVideoUrl} controls width="320"></video>
      <a href={webcamVideoUrl} download="webcam-recording.mp4"
        >Download Webcam Recording</a
      >
    </div>
  {/if}
</main>

And that’s basically it 🎉
We’ve got a working application that can record the screen as well as a webcam feed.

Improvements

While this basic application functions, there were some improvements that really made the result work for me 🤓

Moving the webcam feed into a Picture in Picture window

I wanted to move my webcam frame into a separate window, so I will never block any content from the viewer.

I like the approach described in this article about composing the desktop & webcam streams together into a <canvas /> element and saving the output as a video file, but while approach looks very cool, it is lacking the context of the webcam position.

It also introduces another layer where things might go out of sync with the audio as well as a performance overhead of another composition layer.

But we can solve this by using the requestPictureInPicture method on the a video.

We separate the webcam stream from the desktop stream again, and just open it in a separate Picture in Picture window.

navigator.mediaDevices
  .getUserMedia({ video: true })
  .then((stream) =>     {
    const videoElement = document.createElement("video");
    videoElement.srcObject = stream;
    videoElement
      .play()
      .then(() => {
        videoElement.requestPictureInPicture();
      })
  });

Here’s a demo of the result:

You won’t get a nice circle as loom, but for my purposes this was good enough.

Tuning the Codec

As the codec for the recorded screen I’ve used these options:

new MediaRecorder(combinedStream, {
    mimeType: "video/webm;codecs=vp9",
    videoBitsPerSecond: 4 * 1024 * 1024,
});

This gave me the best results with acceptable performance.

Apparently the newer AV1 codec gives you an even better output quality, but It’s very CPU intensive and was unusable on my machine.

Fixing the color profile under Linux

On my linux machine the colors of the recordings looked washed out.

I could fix this by forcing a color profile via the chrome flags:
chrome://flags/#force-color-profile

For my machine forcing them to sRGB led to the best results.

Streaming the video directly to my hard drive

While saving the video locally is fine, I would like to be in control of any recorded data at all time. When the browser crashes or I close it accidentally, the whole stream would be lost 😢

As a simple solution I’ve set up a WebSockets Server that accepts the video stream and writes it directly to a file.

import { WebSocketServer } from "ws";
import fs from "fs";
import { randomUUID } from "node:crypto";
 
const wss = new WebSocketServer({ port: 8080 });
 
wss.on("connection", (ws) => {
  // Ensure unique files on every new recording by giving them a UUID
  const id = randomUUID();
  const fileName = `${id}.mp4`;
  const writeStream = fs.createWriteStream(fileName);
 
  console.log(`Client connected, creating stream file`, fileName);
 
  // Accept chunks from the clients and write them to the file
  ws.on("message", (message) => {
    console.log("Received chunk");
    writeStream.write(message);
  });
 
  ws.on("close", () => {
    console.log("Client disconnected");
    writeStream.end();
  });
});

And on the client we simply connect to our server when we start a recording and close the session when stopping.

// Svelte state for the socket
let socket: WebSocket;
 
async function startRecording() {
  // Connect to the socket when starting the recording
  socket = new WebSocket("ws://localhost:8080"); 
}

Whenever we’ve got new data we send it to the server.

screenRecorder.ondataavailable = (event) => {
  if (event.data.size > 0 && socket.readyState === WebSocket.OPEN) {  
    socket.send(event.data);  
  }  
}

We’ll also define how often we want to push chunks to the server, in our case every second.

screenRecorder.start(1000);

The only thing left is to close the WebSocket connection when we stop the recording.

screenRecorder.onstop = () => {
  socket.close();  
}

Now whenever we finish a recording the file already sits at the root of our repository, no more downloading needed!

Processing the recorded files

Depending on your display resolution the files size is most likely to quite large.

For my machine with a 1920x1200 resolution I’m getting around 3 GB of data for 1 Hour of video.

We can optimize the final result using ffmpeg:

ffmpeg -i input.mp4 -vcodec libx264 -preset veryfast -b:v 1000k -maxrate 1000k -bufsize 2000k -vf "scale=-1:720" -acodec aac -b:a 128k output.mp4

This will optimize our video so we can upload it to a server to share.

With these properties I could reduce the file size to 365 Mb 🏋️

Tuning Audio Settings

To get a better output for the audio you could try these settings, although I did encounter some cutting of short voice responses.

const audioConstraints = {
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
  },
};
const audioStream =
  await navigator.mediaDevices.getUserMedia(audioConstraints);

That’s a wrap!

Have a look a the source code here:
https://github.com/floscr/svelte-screen-record