Shortest: AI-Powered Natural Language End-to-End Testing Framework

Introduction

Shortest is an advanced AI-powered end-to-end testing framework designed to streamline quality assurance through natural language. It enables developers and QA engineers to write comprehensive tests using plain English, abstracting away complex automation details. Built on top of Playwright, Shortest provides robust browser automation capabilities, enhanced by the intelligence of the Anthropic Claude API for dynamic test execution.

Key features include:

• Natural language-driven E2E testing.
• AI-powered test execution using Anthropic Claude API.
• Foundation on Playwright for reliable browser interaction.
• Seamless GitHub integration, including 2FA support.
• Email validation capabilities with Mailosaur.

Installation

Getting started with Shortest is straightforward. Use the shortest init command to quickly set up the framework in your project.

npx @antiwork/shortest init

This command automates several steps:

• Installs the @antiwork/shortest package as a dev dependency if not already present.
• Creates a default shortest.config.ts file with boilerplate configuration.
• Generates a .env.local file (if absent) with placeholders for essential environment variables, such as ANTHROPIC_API_KEY.
• Adds .env.local and .shortest/ to your .gitignore file.

After installation, configure your shortest.config.ts file and provide your Anthropic API key.

import type { ShortestConfig } from "@antiwork/shortest";

export default {
  headless: false,
  baseUrl: "http://localhost:3000",
  browser: {
    contextOptions: {
      ignoreHTTPSErrors: true
    },
  },
  testPattern: "**/*.test.ts",
  ai: {
    provider: "anthropic",
  },
} satisfies ShortestConfig;

Examples

Shortest offers a flexible and powerful way to define your tests.

Basic Test

Create test files following the pattern specified in your configuration, for example, app/login.test.ts:

import { shortest } from "@antiwork/shortest";

shortest("Login to the app using email and password", {
  username: process.env.GITHUB_USERNAME,
  password: process.env.GITHUB_PASSWORD,
});

Using Callback Functions

You can extend tests with callback functions for additional assertions or logic after the browser execution.

import { shortest } from "@antiwork/shortest";
import { db } from "@/lib/db/drizzle";
import { users } from "@/lib/db/schema";
import { eq } from "drizzle-orm";

shortest("Login to the app using username and password", {
  username: process.env.USERNAME,
  password: process.env.PASSWORD,
}).after(async ({ page }) => {
  const clerkId = await page.evaluate(() => {
    return window.localStorage.getItem("clerk-user");
  });

  if (!clerkId) {
    throw new Error("User not found in database");
  }

  const [user] = await db
    .select()
    .from(users)
    .where(eq(users.clerkId, clerkId))
    .limit(1);

  expect(user).toBeDefined();
});

Lifecycle Hooks

Execute code before and after test suites or individual tests using lifecycle hooks.

import { shortest } from "@antiwork/shortest";

shortest.beforeAll(async ({ page }) => {
  // Setup before all tests
});

shortest.beforeEach(async ({ page }) => {
  // Setup before each test
});

shortest.afterEach(async ({ page }) => {
  await page.close();
});

shortest.afterAll(async ({ page }) => {
  // Teardown after all tests
});

Chaining Tests

Shortest supports chaining tests for sequential execution or reusable test flows.

// Sequential test chain
shortest([
  "user can login with email and password",
  "user can modify their account-level refund policy",
]);

// Combine flows with spread operator
const loginAsLawyer = "login as lawyer with valid credentials";
const allAppActions = ["send invoice to company", "view invoices"];
shortest([loginAsLawyer, ...allAppActions]);

API Testing

Test API endpoints using natural language descriptions or explicit requests.

const req = new APIRequest({
  baseURL: API_BASE_URI,
});

shortest(
  "Ensure the response contains only active users",
  req.fetch({
    url: "/users",
    method: "GET",
    params: new URLSearchParams({
      active: true,
    }),
  }),
);

Or a simpler natural language approach:

shortest(`
  Test the API GET endpoint ${API_BASE_URI}/users with query parameter { "active": true }
  Expect the response to contain only active users
`);

Why Use Shortest?

Shortest revolutionizes the testing process by making it more accessible and efficient. Its natural language interface allows non-technical team members to understand and even contribute to test definitions, fostering better collaboration. The integration of AI, specifically Anthropic Claude, enables more dynamic and intelligent test execution, adapting to UI changes and reducing maintenance overhead. With its Playwright foundation, Shortest ensures reliable and robust end-to-end testing across various browsers. Features like GitHub 2FA login and Mailosaur email validation address common real-world testing challenges, making it a comprehensive solution for modern web applications.

Links

• GitHub Repository: https://github.com/antiwork/shortest
• NPM Package: https://www.npmjs.com/package/@antiwork/shortest
• Video Tutorial: https://github.com/antiwork/shortest/issues/143#issuecomment-2564488173
• CI Setup Example: https://github.com/antiwork/shortest/blob/main/.github/workflows/shortest.yml