End-To-End Testing For Developer Tutorials

The Problem:

Maintaining developer tutorials is a time-consuming but essential responsibility for anyone who maintains technical documentation.

Consider a new developer who is eager to dive into a new tool or library. They find your tutorial, follow the first set of instructions, and immediately hit a roadblock—the command they just copied no longer works. This scenario is frustrating for the developer and reflects poorly on your product. A tutorial written even one month ago might already be obsolete if it depends on rapidly evolving technologies.

However, updating an extensive catalog of developer tutorials, meticulously testing each tutorial step-by-step, and ensuring each works as expected can take days to weeks. The more complete the tutorial, the more likely you are to rely on several tools that may have frequent breaking changes, making this task unmanageable.

The Solution:

What if there was a streamlined and reliable process for you to test that all of your tutorials work?

A way to feel secure that each command is up-to-date and new users aren't feeling frustrated?

What if you could test an entire suite of tutorials simultaneously?

What if it also worked with existing documentation frameworks like Docusarus?

One possible solution is to utilize a testing framework like Playwright to run end-to-end tests. By implementing end-to-end testing for tutorials, you can transform the maintenance of technical documentation from a daunting task to a streamlined, reliable process.

Here's how it could work:

Embedding Test Metadata: Before each step in your tutorial, you can embed a hidden  element with a unique ID and custom properties describing the step a user would take. This metadata serves as the backbone for your automated tests. You can fully customize the actions available.
Automating Test Execution: Using Playwright, you can set up a test to execute the steps from the  elements in the order they appear in the tutorial. This mimics a real user performing these steps, building the project locally as they would.
Ensuring Accuracy and Functionality: You can run the project locally to test its full functionality, or the final output files can be compared to a source of truth.

How does this look in practice?

Let's use a very simple tutorial as an example. Here is what the tutorial might look like in markdown format with  elements added for each step:

# Example Tutorial

## Make a new folder

Make a new folder called `project-folder`:

<span
id="create-project-folder"
data-name="runCommand"
></span>

```sh
mkdir project-folder
```

## Create a new React project

Initialize a new React project.

<span
id="create-react-app"
data-name="runCommand"
data-command-folder="tests-output/project-folder"
></span>

```sh
npx create-react-app my-app
```

## Add a "Hello World" button

Replace your `src/App.js` file with the code below:

<span
id="modify-app-file"
data-name="writeToFile"
data-filepath="tests-output/project-folder/my-app/src/App.js"
></span>

```jsx
import { useState } from 'react';

function App() {
  const [showSecret, setShowSecret] = useState(false);
  return (
    <div>
      <button onClick={() => setShowSecret(true)}>
        Show Secret Text
      </button>

      {showSecret && <p>Hello World</p>}
    </div>
  );
}

export default App;
```

Now you have a React app that shows a secret message!

To take this to the next level, let's add some testing for the complete project. The steps should mimic what the user will do.

## Testing the app

To run the app locally, run the command below:

<span
id="run-app"
data-name="runCommand"
data-pre-command="pnpm pm2 start 'PORT=4000 BROWSER=none <COMMAND>' --name 'react-app' --cwd ./tests-output/project-folder/my-app"
></span>

```sh
npm start
```

<span
id="wait-until-loaded"
data-name="wait"
data-timeout="8000"
></span>

<span
id="visit-app"
data-name="goToUrl"
data-url="http://localhost:4000/"
></span>

<span
id="click-button"
data-name="clickButtonFromText"
data-button-text="Show Secret Text"
></span>

<span
id="click-button"
data-name="findText"
data-find-text="Hello World"
></span>

With this, the full functionality of the tutorial is set up to be tested.

Now, in a Playwright test, you can find all of the  elements on the page that have a data-name attribute and execute the steps as a user would.

For example, for a runCommand step, the test would look for the "copy-to-clipboard" button immediately following the  element to find the code or command that a user would copy.

To see the complete working example configured with Docusarus, you can clone the example below or view the repository on GitHub.

git clone https://github.com/sarahschwartz/tutorial-testing-demo.git

Implementing Testing in Your Tutorial

To implement this method of testing in your tutorials:

1. Setup Your Testing Environment: Install Playwright and set up a basic testing script. Ensure your environment mirrors what your readers will use.

2. Embed Testing Metadata: Modify your tutorials to include hidden  elements before each step the user should perform and for each step needed to validate the project. The element should contain the name of the type of action the test should perform and any other relevant information needed.

3. Write the Test: Create a Playwright script to find these  elements and execute the associated actions by extracting the HTML attributes. Include error handling to manage command failures or unexpected outputs.

4. Integrate and Automate: Integrate this testing into your continuous integration (CI) pipeline, such as GitHub Actions, to run tests automatically whenever tutorials are updated or at regular intervals.

For more information on how to use Playwright, refer to the Playwright documentation.

Sarah Schwartz's Blog