Boot protocol #1398

adamziel · 2024-05-15T09:26:56Z

Playground needs a boot function with this signature: bootWordPress :: Configuration -> Promise<PHPRequestHandler>. It would be a single point of entry for all Playground-based applications on all platforms to get WordPress up and running. See the discussion for more context.

Related resources

Boot flow diagram sketch – some details are missing but it shows the idea pretty well.
Playground Boot/Export Protocol #1379
Playground Boot: Align the boot process between remote.html and CLI #1389
v1 WordPress Boot Protocol #1390
importWordPressFiles contains a lot of boot-like logic

What will that unlock?

Reliable Playground CLI that can run any WordPress project
Reliable OPFS („browser storage”) that doesn’t break whenever db.php changes
Reliable and fast native fs integration that only stores wp-content locally without the rest of WP core (writing all the WordPress core files is super slow)
Migrating wp-now and the VS Code extension to a Playground CLI backend and solving most issues it suffers from
Groundwork for zip exports and imports.
Simpler, more consistent codebase
Inform Blueprints v2 development

Related issues this will solve:

Browser storage: Unhelpful crash message when outdated code is stored #1121

Version 1 scope

A `bootWordPress` function

There are tactical guidelines all over the discussion, for example this comment about conflicts between mount options and installing WordPress options. Refer to the discussion frequently when working on this.

Boot PHP
- Configuration would include a runtime-specific PHP factory function to create a PHPRequestHandler, setup runtime rotation, and use the right PHP version.
- Set the default php.ini path
- Set php.ini overrides
- Set sub-process spawn handler
Boot PHPRequestHandler
- Setup PHP process factory
- Accept a site URL
Setup VFS
- Create the right /internal directory structure to preload mu-plugins etc. from
- Mount any resources (NODEFS, OPFS, Native FS handle, ZIP etc.)
- Recursive copy files from given resources (NODEFS, OPFS, Native FS handle, extract ZIPs etc.)
- Mounting and copying could happen at one of three stages: before setting up WordPress, before setting up the database, after setting up the database
- Proxy /tmp, /internal/shared, and documentRoot to the primary PHP instance of the PHPRequestHandler
Setup WordPress files
- Create any custom mu-plugins in /internal/shared/preload
- Put ca certs in /internal/shared/ca.certs (if provided)
- Unzip a "base snapshot" at /wordpress if needed. It could be an official WordPress release, a minified build, or a "complete Playground export" with wp-admin etc, but it can't be a "partial Playground export" with just wp-content.
- Create docroot/../wp-config.php if docroot/wp-config.php is missing. If docroot is not in /wordpress, e.g. when mapping all local paths to vfs, then create wp-config.php in docroot
- Set SITE_URL, HOME_URL
- Generate secrets constants if needed
- Assert WordPress core files are in place
Setup the database
- Configuration specifies SQLite, MySQL, or "custom"
- Boot function either preloads the SQLite plugin, expects MySQL connection details already provided in wp-config.php, or expects to find a custom db.php file
- If SQLite, source the database either from wp-content, if we’re allowed to modify local files, or from /internal/shared/.ht.sqlite if we’re not allowed to.
- Assert DB is accessible
Installs WordPress
- checks is_blog_installed()
- If yes, upgrade the database and we're done
- Otherwise run the WordPress installation wizard
- Run database upgrade
- Assert is_blog_installed()

Platform-level enhancements

Migrate Playground CLI, Playground web, and Blueprints unit tests to this new boot function.
Generate the minified playground.wordpress.net build using Playground CLI. Remove the SQLite integration plugin from it. Still ship a pre-installed .ht.sqlite database. On web boot install the sqlite plugin in /internal. Ensure that previous ZIP exports and NativeFS mounts continue to work on playground.wordpress.net, ditto for new ZIP exports and NativeFS mounts.
- Not required, but it would be lovely to replace the WordPress-building Dockerfile with a Blueprint.
Playground CLI: Don't create /wordpress/wp-config.php on boot #1407 + an in-browser counterpart
Get rid of wpContentFilesExcludedFromExport
A utility function similar to withPhpIniValues(callback) to disable networking for the WordPress installation request
Decide how to handle the configureErrorLogging platform-level mu-plugin – it creates debug.log file in wp-content which means it potentially modifies a local mount.
Align GitHub import/export handlers with zip import/export handlers and boot handlers. Remove a lot of special casing and excluding „core files” from different types of exports (zip, GitHub, native fs, local directories), reuse GitHub import/export path mapping logic in all these scenarios
Remove NodePHP and WebPHP classes #1399 to simplify the Configuration shape
Remove the importWordPressFiles and runWpInstallationWizard Blueprint steps as all these operations will be handled by the boot flow. Leave a no-op fallback in place and log a warning when it's used.

Related PRs:

Nice to haves

Update the default php.ini values baked into WebAssembly PHP to something more useful for all runtimes, e.g. don't disable_functions, allow network calls, set the memory limit to 256M etc. Basically compile with the same default values we're setting in runtime now.
Replace the entire WordPress PR previewer Blueprint with a single artifact URL since the Boot flow would handle the installation, SQLite setup, db upgrades etc. cc @dmsnell

Assumptions

The behavior is low-level and explicit. We might use sensible default to ease the setup, but the boot function should avoid implicit, magic behaviors.
Error early when mutually incompatible options are used (example). This could be a runtime check, a type-level check, or both.
fetch() requests to get the Base Snapshot, SQLite plugin etc. may be started before the bootWordPress() call so the configuration should accept deferred resources.
Running the Blueprint is not a part of boot. The Blueprint runner assumes the filesystem already exists.
Boot avoids modifying the mounted files as much as possible. Two exceptions could be:
- Writing to .ht.sqlite database – unless we'd offer a way of moving it to /internal/shared
- Copying wp-config-sample.php to wp-config.php

Out of scope for v1 (future work)

Migrating wp-now
OverlayFS
Other mounting modes
- One-way mounts (local changes are copied to Playground but not the other way around
- Read-only modes (error on write attempt)
- Timer/watcher for syncing local files to VFS
Site transfer protocol
Importing data from data.sql and rewrite stale URLs on the fly. V1 would only accept input data in form of a .ht.sqlite database.

The text was updated successfully, but these errors were encountered:

…1401) Writes sunrise.php to the `/internal` directory instead of `/wordpress` to ensure setting up a multisite won't pollute local website when mounted. Related to [Boot Protocol](#1398) ## Testing Instructions Confirm the CI checks pass.

adamziel · 2024-05-15T21:49:57Z

The only two files Playground modifies during the boot sequence now are:

`wp-config.php‘, only when it’s originally missing. It could be also moved one level above the document root and WordPress would still pick it up.
.ht.sqlite, to install WordPress. It could be sourced from /internal if needed.

Prevents creating a `wp-config.php` file in WordPress document root. Instead, creates it one level higher as WordPress falls back to loading it from there. Related to #1398. Similar to #1382 and #1401. ## Testing instructions Run `bun packages/playground/cli/src/cli.ts server --login` and confirm it loads an installed WordPress.

With this PR, you can mount directories in Playground CLI before WordPress is installed. This enables creating the entire file structure in a local directory instead of VFS. This is just the first step. Eventually we'll have more `mount` switches and the name `mount` will just default to one of the options. ## Testing instructions Run this and confirm WordPress got installed in the local directory: ```shell bun packages/playground/cli/src/cli.ts run-blueprint --mount-before-install=`pwd`/new-wordpress-site:/wordpress ll new-wordpress-site ``` Related to #1398

Do not merge – explorations in progress Uses Playground CLI to produce a minified WordPress build, replacing the custom bash-based WordPres installation flow that relied on wget, unzip, sed, wp-cli etc. This is to dogfood Playground CLI and put the boot protocol to a practical use. With this PR, CLI Playground produces a minified WordPress version that is later used to boot the web Playground. Eventually, the minification logic could be just a Blueprint instead of a Dockerfile. For now, though, I didn't explore that. ## Remaining work To ship this PR, remote.html needs to use the same boot logic as the CLI Playground and, e.g., install the SQLite integration plugin in the `/internal` directory. In its current state, this PR just ships a minified WordPress without that plugin. ## Testing instructions Rebuild the latest WordPress with ```shell rm -rf packages/playground/wordpress-builds/public/wp-6.5 npx nx bundle-wordpress:major-and-beta playground-wordpress-builds ``` Then run `npm run dev` and confirm the local Playground loads without any issues. Related: #1398

Related to Boot protocol #1398 Ships the SQLite database integration in WordPress builds to make it accessible in remote.html and enable installing it separately in WordPress builds that don't ship it by default. No testing instructions – this only adds new code that's not used for anything yet.

Related to Boot protocol #1398 Ships the SQLite database integration in WordPress builds to make it accessible in remote.html and enable installing it separately in WordPress builds that don't ship it by default. To test, run `rm packages/playground/wordpress-builds/src/sqlite-database-integration/sqlite-database-integration.zip` and then confirm this command refreshes the local SQLite database zip: ``` nx bundle-sqlite-database playground-wordpress-builds ```

…ld (without the SQLite integration plugin) Uses Playground CLI to produce a minified WordPress build, replacing the custom bash-based WordPress installation flow that relied on wget, unzip, sed, wp-cli etc. This is to dogfood Playground CLI and put the boot protocol to a practical use. With this PR, CLI Playground produces a minified WordPress version that is later used to boot the web Playground. Depends on #1423 #1423 #1425 ## Testing instructions Rebuild the latest WordPress with ```shell rm -rf packages/playground/wordpress-builds/public/wp-6.5 npx nx bundle-wordpress:major-and-beta playground-wordpress-builds ``` Then run npm run dev and confirm the local Playground loads without any issues. Related: #1398

…d php.setPhpIniEntry() (#1423) Refactors PHP.ini management: * Removes the `php.setPhpIniEntry()` methods that only worked correctly when called before initializing the PHP runtime. * Hardcodes the php.ini path to `/internal/shared/php.ini`. The developer is welcome to override that file as needed, but they can't customize the path. Once changing the path becomes important, let's add that feature at the [Boot Protocol](#1398) level. * Provides `getPhpIniEntries()`, `setPhpIniEntries()`, and `withPHPIniValues()` helpers that work at any point of the PHP runtime lifecycle. * Moves a list of the default php.ini values from `php_wasm.c` to `base-php.ts` where it can be customized without rebuilding the wasm module. ## What problem is it solving? Working with php.ini is highly problematic as you could never be sure whether your `setPhpIni*` call would be effective. Then, at the API level, the `setPhpIni*` methods were also excluded from some TypeScript types but not from the others. ## Remaining work * Rebuild all the PHP modules ## Testing Instructions Confirm the CI tests pass.

…if there's no custom db.php inside wp-content (#1424) This PR always preloads the SQLite plugin in remote.html, even if the provided WordPress already ships one. The preloaded plugin will only be used if no `wp-content/db.php` drop-in plugin is found in the filesystem. This change sets the stage for: * #1426, which ships a minified WordPress build without the SQLite plugin baked-in. * #1425, which runs the WordPress installation wizard if the site isn't installed yet Related to [Boot Protocol](#1398) ## Problem solved We want the official WordPress release `.zip` to just work in Playground. Currently it doesn't because remote.html expects a pre-processed build with the SQLite plugin already baked-in. This PR, in conjunction with #1425 and #1426, remove that requirement. ## Testing Instructions Confirm the CI checks pass.

## What is this PR doing? Runs the WordPress installation process in remote.html when the loaded WordPress instance isn't installed yet. This is to enable the official WordPress release `.zip` to just work in Playground. Currently it doesn't because remote.html expects a pre-installed WordPress build with a pre-populated SQLite database already baked-in. This PR, in conjunction with #1425 and #1426, remove that requirement. Related to [Boot Protocol](#1398) ## Testing Instructions Confirm the CI checks pass.

@brandonpayton

Implements a `bootWordPress()` function aiming to provide a reliable and consistent boot pipeline for every Playground-based app, whether it's a browser app, server app, a VS Code extension, or anything else. As a nice side-effect, Playground can now boot not just from the custom minified WordPress builds, but also from the official WordPress releases! Related issue: #1398 ## How does it work? `bootWordPress()` ties together all the pieces from different Playground-based apps. [Boot flow diagram](#1379 (comment)) demonstrates the idea. A tl;dr pipeline is: * Mount resources (stage=before WordPress files) * Unzip WordPress, if that's requested * Mount resources (stage=before database setup) * Setup SQLite, MySQL (TODO), or rely on a mounted database * Run WordPress installer, if the site isn't installed yet Here's a usage example in a Node.js app: ```ts const requestHandler = await bootWordPress({ siteUrl: absoluteUrl, createPhpInstance() { return new NodePHP(); }, createPhpRuntime: async () => await NodePHP.loadRuntime(compiledBlueprint.versions.php), wordPressZip, sqliteIntegrationPluginZip: fetchSqliteIntegration(monitor), sapiName: 'cli', createFiles: { '/internal/shared/ca-bundle.crt': rootCertificates.join('\n'), }, phpIniEntries: { 'openssl.cafile': '/internal/shared/ca-bundle.crt', allow_url_fopen: '1', disable_functions: '', }, hooks: { async beforeWordPressFiles(php) { if (args.mountBeforeInstall) { mountResources(php, args.mountBeforeInstall); } }, }, }); ``` ## Testing instructions * Confirm all the CI checks work * Load the latest nightly **official WordPress build** into Playground: http://localhost:5400/website-server/?php=8.0&wp=http://localhost:5400/plugin-proxy.php?url=https://wordpress.org/nightly-builds/wordpress-latest.zip * Load the official WordPress 6.5 release into Playground: http://localhost:5400/website-server/?php=8.0&wp=http://localhost:5400/plugin-proxy.php?url=https://wordpress.org/latest.zip * Confirm Playground CLI still works: Run bun packages/playground/cli/src/cli.ts server --login and confirm it loads an installed WordPress. ## Follow-up work * Replace manual calls to `setupPlatformLevelMuPlugins()` in unit tests with `bootWordPress()` * Add unit tests to confirm boot works with minified WordPress build, official WordPress release, SQLite, MySQL. Test all the `BootOptions` and confirm PHP.ini directives are used, constants are created etc. * Remove the `createPhpInstance()` argument in favor of using just the `PHP` class – once #1457 merges * Figure out storing `debug.log` in the `configureErrorLogging` mu-plugin – right now it's stored in `/wordpress/wp-content` * Support more database modes: MySQL and `custom` where Playground relies on `db.php` brought in from * Replace hooks with a Mount data type that would have different implementations, like OPFS, Native FS, and Node FS – explorations started in #1457 * Once the API settles, document the usage for developers cc @brandonpayton @bgrgicak

@wojtekn

… "PHP" class (#1457) > [!WARNING] > This is breaking change! Review the description below to assess the impact on your app. Consolidates the `BasePHP`, `NodePHP`, and `WebPHP` classes into a single `PHP` class. This refactor reduces redundancy and clarifies the PHP handling process across different environments. ## Examples ### NodePHP **Before** ```ts import { NodePHP } from '@php-wasm/node'; const php = await NodePHP.load('8.0'); ``` **After** ```ts import { loadNodeRuntime } from '@php-wasm/node'; import { PHP } from '@php-wasm/universal'; const php = new PHP(await loadNodeRuntime('8.0')); ``` ### WebPHP **Before** ```ts import { WebPHP } from '@php-wasm/web'; const php = await WebPHP.loadRuntime('8.0'); ``` **After** ```ts import { loadWebRuntime } from '@php-wasm/web'; import { PHP } from '@php-wasm/universal'; const php = new PHP(await loadWebRuntime('8.0')); ``` ### Mounting **Before** ```ts php.useHostFilesystem(); // or php.mount( '/home/users/adam/my-dir', '/my-dir-in-vfs'); ``` **After** ```ts import { NodeFSMount, useHostFilesystem } from '@php-wasm/node'; useHostFilesystem( php ); php.mount( new NodeFSMount( '/home/users/adam/my-dir' ), '/my-dir-in-vfs' ); ``` Closes #1399 ## Motivation First, the public API surface of all the PHP classes and interfaces is [overly complex](#514). This PR is a step towards simplifying it. Second, in the [Boot Protocol](#1398), PR the `bootWordPress()` function requires separate `createPHP` and `createPHPRuntime` callbacks just because Playground uses different classes in node.js and in the browser. With this PR, `createPHP` callback will no longer be needed anymore as `bootWordPress()` will just always create a `new PHP()` instance. ## Public API Changes - Removes `BasePHP`, `NodePHP`, and `WebPHP` classes in favor of a single PHP class exported from `@php-wasm/universal` - `php.useHostFilesystem()` was removed in favor of a decoupled function `useHostFilesystem( php )` available in `@php-wasm/node`. - `PHP.mount()` signature changed from `mount( hostPath: string, vfsPath: string )` to `mount( mountable: Mountable, vfsPath: string )` - Moves WebPHPEndpoint from `@php-wasm/web` to `@php-wasm/universal` and renames it to `PHPWorker`. That class isn't specific to the web and could be easily used by Node workers. - Removes the `IsomorphicLocalPHP` interface. The PHP class is now the source of truth for all derived worker, client, etc. implementations. - Removes the `createPhpInstance()` option from `bootWordPress( options )` in favor of always using the PHP class. - Updates to Documentation: Adjusted examples and references in the documentation to reflect the new `PHP` class. - Refactoring of Test Suites: Updated tests to work with the new class structure. ## Testing instructions Confirm all the CI checks pass – this PR includes an extensive refactor of all the tests. Heads up @wojtekn @fluiddot @sejas

…ld (without the SQLite integration plugin) (#1426) Uses Playground CLI to produce the minified WordPress builds: ```bash $ bun \ packages/playground/cli/src/cli.ts run-blueprint \ --wp=https://wordpress.org/6.5.zip \ --mount-before-install=./wordpress-site:/wordpress ``` Those new builds ships **without** the SQLite database integration plugin but **with** the SQLite database file containing all the data and tables created by a vanilla WordPress installation. Advantages: * Less code to maintain – Playground CLI replaces a custom bash script relying on wget, unzip, sed, wp-cli etc. * More testing for our internal tools – we're dogfooding Playground CLI and the boot protocol in the critical code path. ## Testing instructions * Confirm the CI checks passed – this is about as good of a testing as we can get. * Run `npm run dev` and confirm the local Playground loads without any issues. Run `bun packages/playground/cli/src/cli.ts server --login` and confirm it loads an installed WordPress. Related: #1398

adamziel added [Type] Enhancement New feature or request [Feature] Boot Flow labels May 15, 2024

adamziel added this to the Innovative Developer Tools milestone May 15, 2024

adamziel mentioned this issue May 15, 2024

Blueprints: Write sunrise.php to /internal in enableMultisite step #1401

Merged

adamziel modified the milestones: Innovative Developer Tools, Boot Protocol May 15, 2024

adamziel mentioned this issue May 15, 2024

Playground CLI: Don't create /wordpress/wp-config.php on boot #1407

Merged

adamziel mentioned this issue May 16, 2024

CLI: Distinguish between mount and mountBeforeInstall options #1410

Merged

This was referenced May 16, 2024

Browser storage: Unhelpful crash message when outdated code is stored #1121

Open

Playground Block: Simplify running Core and Gutenberg PRs WordPress/playground-tools#264

Open

adamziel mentioned this issue May 16, 2024

Remote.html: Use Playground CLI to produce minified WordPress build #1416

Closed

4 tasks

adamziel mentioned this issue May 17, 2024

Website: Ship the SQLite database integration plugin #1418

Merged

adamziel mentioned this issue May 18, 2024

WordPress Build: Use Playground CLI to produce minified WordPress build (without the SQLite integration plugin) #1426

Merged

adamziel mentioned this issue May 23, 2024

v1 WordPress Boot Protocol #1390

Merged

adamziel mentioned this issue May 23, 2024

Breaking: PHP: Remove NodePHP and WebPHP classes in favor of a single "PHP" class #1457

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Boot protocol #1398

Boot protocol #1398

adamziel commented May 15, 2024 •

edited

adamziel commented May 15, 2024

Boot protocol #1398

Boot protocol #1398

Comments

adamziel commented May 15, 2024 • edited

Related resources

What will that unlock?

Version 1 scope

A bootWordPress function

Platform-level enhancements

Nice to haves

Assumptions

Out of scope for v1 (future work)

adamziel commented May 15, 2024

adamziel commented May 15, 2024 •

edited

A `bootWordPress` function