feat: Clean cache command#1394
Open
d3xter666 wants to merge 260 commits into
Open
Conversation
getIntegrity tests still need to be updated
Memory adapter and Proxy reader called the deprecated getStatInfo() to check the nodir option, which triggered a deprecation log via @ui5/logger. This caused spurious verbose log entries that leaked into downstream tests using global logger mocks.
… Module Project._addReadersForWriter used unshift which reversed the stage priority order, causing earlier stages (e.g. replaceCopyright) to shadow later stages (e.g. replaceVersion) in the result reader chain. Change to push so that later stages have higher priority, matching ComponentProject behavior.
…hed tags The build manifest version was bumped to "1.0" but the version check in ProjectBuildContext only accepted "0.1" and "0.2", causing pre-built dependencies to be rebuilt from scratch instead of being skipped. Also ensure getProjectResourceTagCollection() applies cached stage tags before returning the collection, matching the pattern already used by getResourceTagCollection().
Add tests for previously uncovered modules: MonitoredReader, MonitoredReaderWriter, MonitoredResourceTagCollection, and Proxy reader. Also extend existing tests for ResourceTagCollection, resourceFactory, and AbstractReaderWriter to cover missing functions. Functions coverage rises from 81% to 95%.
Add targeted tests to bring @ui5/project coverage above the configured thresholds (95% statements, 90% branches, 95% functions, 95% lines). Also fix AVA timeout for parallel test execution and remove dead code module utils/sanitizeFileName.js. New test files: - build/cache/utils.js, index/TreeNode.js, index/ResourceIndex.js - build/BuildReader.js, helpers/WatchHandler.js, helpers/ProjectBuilderOutputStyle.js - ui5framework/maven/CacheMode.js Extended existing tests: - HashTree.js, SharedHashTree.js, CacheManager.js - specifications/Project.js, types/Component.js, types/Library.js - types/Module.js, types/ThemeLibrary.js, extensions/Task.js
The CacheManager is a singleton per cache directory. When multiple concurrent builds share the same singleton (e.g. in parallel test execution), the first build to finish would close the database and break any other build still using it. Track the number of active consumers via a reference count that is incremented on each create() call and decremented on close(). The underlying SQLite database is only closed when the last consumer releases its reference.
… BuildServer When source files are modified while a build is in progress (detected by condition rather than a fatal error. This fixes a race condition where the build takes close to or exceeds 500ms, causing the integration test's file writes to land during source revalidation. - Add SourceChangedDuringBuildError for typed error handling - Handle it alongside AbortBuildError in BuildServer (re-queue projects) - Reset source index state on error to prevent infinite retry loops - Pass abort signal to allTasksCompleted() so watcher-triggered aborts take precedence over the source-change error
Replace fragile setTimeout(0) with deterministic promise-based signals to wait for the handler's async work to complete. On Node 22, esmock's off-thread ESM loader hooks can take more than one microtask tick to resolve dynamic imports, causing afterEach's esmock.purge() to nullify the mock cache before the handler's imports resolve. Use handlerReady (resolves on first stdout write) for most tests, and openCalled (resolves when the open stub fires) for --open tests which have an additional dynamic import after stdout output.
… handles On Windows, SQLite kept the cache.db file handle open after close, preventing subsequent test runs and concurrent builds from cleaning their output directories (fs.rm fails with EBUSY). The root cause is twofold: WAL-mode SQLite retains file mappings until a checkpoint, and the CacheManager singleton was not being released by all consumers. Resolve the issue with: - Add `PRAGMA wal_checkpoint(TRUNCATE)` before close in BuildCacheStorage to flush WAL content back to the main database and release file handles cleanly. This single change resolves the underlying handle leak across cache tests and integration tests without per-test workarounds. - Close the CacheManager at the end of `buildToTarget()`, after all deferred disk writes have completed. - Expose a public `ProjectBuilder#closeCacheManager()` method and call it from `BuildServer.destroy()` so long-running consumers release the database handle on shutdown. - Add a `#destroyed` flag to BuildServer to prevent new builds from being triggered after destroy(). Clear pending build queue and timeout before awaiting active build to avoid races with setTimeout-based queue processing. - Document the lifecycle on `ProjectBuilder#build`, `#buildToTarget`, and `#closeCacheManager`: `buildToTarget` closes the CacheManager itself; consumers of `build` must call `closeCacheManager` when done. - Close singleton instances in CacheManager `create()` test, make `after.always` rimraf best-effort in cache tests (test/tmp is already cleaned at start of each run), and add maxRetries to rimraf in integration tests for any residual Windows handle-release delays.
Add tests for previously untested error-handling paths: - lib/server.js: HTTP server error event handler, buildServer error event forwarding, and the close() rejection handler when buildServer.destroy() fails. - middleware/serveIndex: byGlob rejection. - sslUtil: chmod failure callbacks when fixing outdated file permissions.
…e during index re-init After a SourceChangedDuringBuildError, the source index is nulled and state reset to RESTORING_PROJECT_INDICES. If projectSourcesChanged or dependencyResourcesChanged was called before initSourceIndex ran again, the state was incorrectly transitioned to REQUIRES_UPDATE, causing initSourceIndex to skip initialization and a subsequent NPE on the null source index.
… and missing error handler Close CacheManager in a finally block within buildToTarget() so the SQLite database is released even when the build throws. Previously, builds that threw (e.g. SourceChangedDuringBuildError) leaked a refcount, keeping the DB memory-mapped on Windows and preventing cleanup by subsequent tests. Add an error event handler to the JSDoc child process spawn. Without it, a failed spawn on Windows would never emit close, hanging the promise indefinitely.
…esource.setStream Streams from older userland packages like readable-stream@2 (used by replacestream) have .pipe but no Symbol.asyncIterator, causing "stream is not async iterable" errors when the resource content is later consumed by node:stream/consumers. Also handle WHATWG TransformStream objects by unwrapping to their readable side.
The custom task reads dependencies via taskUtil.getProject().getReader() which bypasses the monitored dependencies reader. These reads are not tracked by ResourceRequestManager, so dependency changes don't invalidate the application's result cache.
Now only used in @ui5/fs
…c-generate
Replace type expressions unsupported by JSDoc's Catharsis parser:
- Union types inside Record literals in generics ({string|number|...})
- TypeScript-style import() type syntax
- Tuple types ([T, U]) inside generics
- Optional property syntax (?) inside Record generics
…ware configuration (#1367) JIRA: CPOUI5FOUNDATION-1206 --------- Co-authored-by: Copilot <copilot@github.com>
matchResourceMetadataStrict and the duplicated fast path in HashTree#upsertResources Phase 1 returned "unchanged" whenever a file's mtime matched the cached value, without consulting size or content. External operations that preserve mtime - cp -p, tar -x, rsync -t, and atomic rename of a temp file with copied stat - therefore passed silently as cache hits and produced stale build output. Verify size before the mtime-based short-circuit can return true. This matches the cheap-path semantics used by rsync's -t quick-check and git's stat-bundle compare, and closes the gap without forcing an integrity hash on every comparison. For FS-loaded resources getSize() resolves synchronously from statInfo, so the only added cost is one microtask per comparison. Remove the duplicated mtime fast path from HashTree#upsertResources and route every existing resource through Phase 2, which delegates to the now-corrected matchResourceMetadataStrict and already handles tag-only updates.
Extend matchResourceMetadataStrict to compare the resource inode against the cached inode. When both sides expose an inode and they disagree, the file has been replaced (atomic rename, cp, tar extraction, ...) — the mtime+size short-circuit is suppressed and the comparison falls through to the integrity check. Inode is now always populated by createResourceIndex; the optional includeInode flag is removed. When either side has no inode (virtual resources, caches written before this change), the inode check is skipped and behavior matches the previous mtime+size path. Falling through to integrity rather than returning false avoids unnecessary task re-execution when the replacement preserves content (e.g. an idempotent rsync). When the replacement also alters content, the integrity check correctly classifies the resource as changed, closing a stale-cache blind spot that mtime+size alone could not detect.
…changed Drop the unused matchResourceMetadata variant (no longer used) and rename matchResourceMetadataStrict to isResourceUnchanged.
Remove unused firstTruthy method, drop the stale local ResourceMetadata typedef (use existing type from HashTree.js), and strip the UI5_CACHE_PERF instrumentation.
createStat() previously returned a synthetic statInfo with ino: 0, which Resource.clone() forwarded as statInfo into the new instance. The constructor's `this.#inode ??= statInfo.ino` then set the clone's inode to 0, so memory-only resources read through Memory.byPath / Memory.byGlob advertised a fabricated inode. That defeated the inode short-circuit in the build cache's isResourceUnchanged whenever cached metadata was later compared against an FS-backed read of the same path. Set ino to undefined in createStat() so memory-only resources and their clones report getInode() === undefined, leaving the cache to fall back on size + lastModified + integrity. Add Resource tests covering inode propagation through construction, clone, and setBuffer for both memory-only and FS-backed resources.
Expose ui5DataDir as a programmatic option on ProjectGraph.build(), plumbed through ProjectBuilder, BuildContext, and CacheManager.create(). Previously the build cache location could only be configured via the UI5_DATA_DIR env var or the ~/.ui5rc config file, both of which are process-global and unsuitable for parallel test isolation. Builder integration tests now route the SQLite-backed build cache into a per-test directory via a new isolatedUi5DataDir helper. This fixes intermittent ERR_SQLITE_ERROR "database is locked" failures on Windows CI caused by parallel AVA workers contending on the shared ~/.ui5/buildCache database, and removes the UI5_DATA_DIR fallback from the builder ava.config.js.
d3xter666
force-pushed
the
feat-clean-cache
branch
from
3491dad to
83f2262
Compare
d3xter666
force-pushed
the
feat-clean-cache
branch
from
83f2262 to
d52c4ad
Compare
RandomByte
force-pushed
the
feat/incremental-build-4
branch
from
7c59782 to
444977d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The command completely cleans the cache by removing the cache files as well as cleaning up the SQLite records.
It does not wipe out the SQLite DB file(s)
JIRA: CPOUI5FOUNDATION-891