eventcatalog build timeout with large amount of domains/events/services

[razafinl]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Description

I am testing eventcatalog with a few hundreds Kafka topics and 20/30 applications.

I define a domain and a service per application. And attach each topic as event to the corresponding domain/application.

When I build with npm build for such setup, the build fails due to timeout.

Defining the same services into the root folder (not within any domain) allows the build to complete successfully. However, that means the application serice is no more attached to its domain/application.

How can I manage the timeout so that I can build with my expected setup ?

Steps to reproduce

• Generate 20/30 domains.
• For domain,
  • Define 1 service (README.md)
  • Define 1 to 10 events (README.md + schema.avsc)
• run npm run build

Expected behavior

Build completed and ability to launch with npm start.

Actual behavior

> Build error occurred
Error: Collecting page data for [object Object] is still timing out after 2 attempts. See more info here https://nextjs.org/docs/messages/page-data-collection-timeout
    at onRestart (/home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/build/index.js:594:35)
    at Worker.isPageStatic (/home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/lib/worker.js:48:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Span.traceAsyncFn (/home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/trace/trace.js:79:20)
    at async /home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/build/index.js:706:52
    at async Span.traceAsyncFn (/home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/trace/trace.js:79:20)
    at async Promise.all (index 7)
    at async /home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/build/index.js:653:17
    at async Span.traceAsyncFn (/home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/trace/trace.js:79:20)
    at async /home/razafinl/eventcatalog/eventcatalogdev/node_modules/next/dist/build/index.js:618:170

Your environment

• EventCatalog version used: 1.1.0

• Environment name and version (e.g. Chrome 89, Node.js 16.4): Firefox 102.12.0esr (32-bit), Node.js 21.1.0

• Operating system and version (e.g. Ubuntu 20.04.2 LTS): WSL2, Ubuntu 22.04.3 LTS

[boyney123]

Interesting issue, never seen this before, but never tried so many events etc.

Interesting idea from the NextJS docs. https://nextjs.org/docs/messages/page-data-collection-timeout,`Increase the timeout by changing the config.staticPageGenerationTimeout configuration option (default 60 in seconds).` wonder if this can help.

In the .eventcatalog-core folder, there is a next.config.js file that is used for the project. Try and change this value

module.exports = {
  // time in seconds of no pages generating during static
  // generation before timing out
  staticPageGenerationTimeout: 1000,
}

To see if this helps?

[bandantonio]

Good news: Thanks for this amazing tool 🤘 I was looking for it for years!

Bad news: I'm having the same issue.

I'm trying to migrate from our custom solution to document events and currently I ended up with the following structure

• 60 domains
• 60 services
• 390 events

Each of the domains has the following structure (auto-generated via a script):

.
└── DomainName
    ├── events
    │   └── eventName
    │       ├── index.md
    │       └── schema.json
    ├── services
    │   └── serviceName
    │       └── index.md
    └── index.md

When I'm starting a build, it fails on timeout with the following error:

warn  - Restarted collecting page data for [object Object] because it took more than 60 seconds
warn  - See more info here https://nextjs.org/docs/messages/static-page-generation-timeout
warn  - Restarted collecting page data for [object Object] because it took more than 60 seconds

> Build error occurred
Error: Collecting page data for [object Object] is still timing out after 2 attempts. See more info here https://nextjs.org/docs/messages/page-data-collection-timeout

The only stable build I was able to produce is:

• 14 domains
• 19 services
• 176 events

(407 static pages in total, there were also lots of warning about exceeding the 128 Kb threshold and performance degradation)

Without defining domains, I can create the full list of events and services (825 static pages in total)

---

Your suggestion did help, but the process was extremely slow - about 50 minutes (~10 for collecting data and ~20 for generating static pages (1303 in total), whereas the last 150 pages took ~20 minutes) . There were also lots or errors: Failed to get service by name, and warnings about exceeding the 128 Kb threshold).

I will check my script to ensure there are no broken paths or names but it's pretty strange that 800+ pages were generated in ~2 minutes, and 1300 made my Mac an airplane for 50 minutes :)

---

Generation details

info - Generating static pages (1303/1303)

info - Finalizing page optimization

Route (pages) Size First Load JS
┌ ○ / 995 B 89.1 kB
├ /_app 0 B 88.1 kB
├ ○ /404 199 B 88.3 kB
├ ● /domains (1756989 ms) 3.45 kB 91.5 kB
├ ● /domains/[domain] (2049049 ms) 3.69 kB 171 kB
├ ├ /domains/Domain1 (193923 ms)
├ ├ /domains/Domain2 (148871 ms)
├ ├ /domains/Domain3 (134091 ms)
├ ├ /domains/Domain4 (124463 ms)
├ ├ /domains/Domain5 (114692 ms)
├ ├ /domains/Domain6 (106060 ms)
├ ├ /domains/Domain7 (105153 ms)
├ └ [+52 more paths] (avg 21573 ms)
├ ● /domains/[domain]/events/[name] (2025374 ms) 320 B 877 kB
├ ├ /domains/Domain8/events/event1 (6941 ms)
├ ├ /domains/Domain9/events/event2 (6042 ms)
├ ├ /domains/Domain2/events/event3 (4209 ms)
├ ├ /domains/Domain2/events/event4 (4193 ms)
├ ├ /domains/Domain10/events/event5 (4167 ms)
├ ├ /domains/Domain2/events/event6 (4152 ms)
├ ├ /domains/Domain11/events/event7 (4078 ms)
├ └ [+538 more paths] (avg 3702 ms)
├ ● /domains/[domain]/events/[name]/logs (1980496 ms) 1.84 kB 178 kB
├ ├ /domains/Domain1/events/event8/logs (4777 ms)
├ ├ /domains/Domain1/events/event9/logs (4702 ms)
├ ├ /domains/Domain1/events/event10/logs (4701 ms)
├ ├ /domains/Domain1/events/event11/logs (4693 ms)
├ ├ /domains/Domain1/events/event12/logs (4488 ms)
├ ├ /domains/Domain1/events/event13/logs (4451 ms)
├ ├ /domains/Domain3/events/event14/logs (4320 ms)
├ └ [+538 more paths] (avg 3621 ms)
├ ● /domains/[domain]/events/[name]/v/[version] 344 B 877 kB
├ ● /domains/[domain]/services/[name] (8408 ms) 304 B 254 kB
├ ├ /domains/Domain9/services/service1
├ ├ /domains/Domain8/services/service2
├ ├ /domains/Domain1/services/service3
├ └ [+92 more paths]
├ ● /events (1777047 ms) 4.44 kB 259 kB
├ ● /events/[name] 251 B 877 kB
├ ● /events/[name]/logs 1.81 kB 178 kB
├ ● /events/[name]/v/[version] 360 B 877 kB
├ ● /overview 2.99 kB 91.1 kB
├ ● /services (3991 ms) 3.98 kB 259 kB
├ ● /services/[name] (573 ms) 237 B 254 kB
├ ├ /services/service4
├ ├ /services/service5
├ ├ /services/service6
├ └ [+47 more paths]
├ ○ /users 1.21 kB 89.3 kB
├ ● /users/[id] 3.2 kB 246 kB
└ ● /visualiser (2608810 ms) 3.41 kB 168 kB

• First Load JS shared by all 96.5 kB
  ├ chunks/framework-1b912105ce057a0c.js 46.6 kB
  ├ chunks/main-1e0049fa30f96a34.js 32.1 kB
  ├ chunks/pages/_app-085c3a570d581179.js 6.61 kB
  ├ chunks/webpack-d1232b074761d079.js 2.77 kB
  └ css/b5a7db9f08d9bb82.css 8.37 kB

---

My env:

• EventCatalog: 1.1.1
• macOS Sonoma 14.1.1
• Safari 17.1, Node 16.16.0, 20.8.0

The related thread in Discord (There are also some suggestions to enhance config capabilities)

[imrafaelmerino]

@bandantonio did you manage to solve this or implement a workaround!? thanks in advance

[bandantonio]

@imrafaelmerino Nope, I postponed integration with EventCatalog due to this limitation. As I mentioned, in my setup, one of the possible workarounds was not to use (define) domains. In this way, everything worked and the generation process was reasonably fast.

The workaround mentioned by @boyney123 did help a bit but the problem is that the next.config.js and .eventcatalog-core folder are created in the runtime, so you have no direct access to this config. The eventcatalog.config.js also doesn't have mapping to the needed parameters. I believe it's rather a critical issue but still no solution 😞

[imrafaelmerino]

@bandantonio Thanks for the quick replay. Sitting tight for version 2!

[boyney123]

Hey

Yeah sorry about that, I need a better test for loads more domains and events. Part of v2 I have used a new file system libary which hopefully speeds things up. The problem is, is everything is build time and the more pages = more load time.....

I often think about what if we moved to server side rendering, this would allow us to render on the fly and have less build times ofc, but would require EventCatalog to be run on a server somewhere rather than static files.

A few folks still want static rendered catalogs, so I'm hesitant to move to server side rendering, but it does open the door to better integrations in the future....

Although I imagine it's annoying, I do appreciate the patience while I look at this too.

I'm hoping soon I get to work more on EC.....! Watch this space.