feat: organization-scoped event custom fields with management UI and filter integration#1815
Draft
ejsmith wants to merge 7 commits into
Draft
feat: organization-scoped event custom fields with management UI and filter integration#1815ejsmith wants to merge 7 commits into
ejsmith wants to merge 7 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds custom field support for events by updating the event model and repository logic, removing legacy query visitor code, and updating test infrastructure to support the new behavior.
- Removed obsolete EventFieldsQueryVisitor usage and file.
- Introduced automatic custom field creation in the event repository.
- Updated the PersistentEvent model to implement virtual custom fields and modified test configurations accordingly.
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/Exceptionless.Tests/Search/PersistentEventQueryValidatorTests.cs | Removed legacy query visitor call in tests. |
| tests/Exceptionless.Tests/Search/EventIndexTests.cs | Updated repository query to include organization filtering. |
| tests/Exceptionless.Tests/Migrations/FixDuplicateStacksMigrationTests.cs | Set log level to Trace for migration tests. |
| tests/Exceptionless.Tests/AppWebHostFactory.cs | Added Kibana container configuration. |
| src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs | Removed unused visitor implementation. |
| src/Exceptionless.Core/Repositories/EventRepository.cs | Added auto-creation of custom fields and related tenant logic. |
| src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs | Removed legacy dynamic mapping for event index custom fields. |
| src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs | Added index creation for custom fields. |
| src/Exceptionless.Core/Models/PersistentEvent.cs | Updated Idx property type and implemented IHaveVirtualCustomFields. |
Files not reviewed (1)
- src/Exceptionless.Core/Exceptionless.Core.csproj: Language not supported
| public IDictionary<string, object?> Idx { get; set; } = new DataDictionary(); | ||
|
|
||
| object? IHaveVirtualCustomFields.GetCustomField(string name) => Data?[name]; | ||
| IDictionary<string, object?> IHaveVirtualCustomFields.GetCustomFields() => Data ?? []; |
There was a problem hiding this comment.
Using an empty array literal '[]' as a fallback for an IDictionary<string, object?> is invalid. Replace '[]' with an appropriate empty dictionary initializer (e.g. 'new DataDictionary()').
Suggested change
| IDictionary<string, object?> IHaveVirtualCustomFields.GetCustomFields() => Data ?? []; | |
| IDictionary<string, object?> IHaveVirtualCustomFields.GetCustomFields() => Data ?? new DataDictionary(); |
Member
There was a problem hiding this comment.
Fixed in commit a5e340a - GetCustomFields() returns new DataDictionary() when Data is null. The [] fallback was removed entirely.
niemyjski
changed the title
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…filter integration Add organization-level custom field management that allows premium users to define typed, searchable event fields (keyword, string, numeric, date, boolean) with full lifecycle management including soft-delete, index slot reuse, and saved view protection. Key changes: - Custom field CRUD on organization controller with validation and plan gating - EventCustomFieldService for document change handling and field lifecycle - Premium feature detection in saved views (UsesPremiumFeatures flag) - Filter integration with operator support per field type - Angular UI: management dialog, filter picker integration - Svelte UI: custom fields management and filter components - Comprehensive test coverage (107+ custom field tests) - Max 20 active fields per organization, slot-based index allocation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
niemyjski
force-pushed
the
feature/custom-fields
branch
from
0a8855a to
b277880
Compare
…seline, fix Prettier - Restore accidentally deleted ControllerManifestTests.cs (was removed during squash) - Generate controller-manifest.json baseline with custom fields endpoints included - Fix Prettier formatting in sidebar.svelte Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…heck Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…finedAsync The previous baseline was missing PostPredefinedAsync because UPDATE_SNAPSHOTS ran against a stale build. Rebuilt and regenerated correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es, ContainsKey pattern - PersistentEvent.GetCustomFields(): consolidate value-type guard into the .Where() predicate - EventCustomFieldService: use .OfType<PersistentEvent>() to replace the outer .Where() null filter and the redundant inner .Where(d => d is not null) in the document loop - CustomFieldIndexingTests: fix regex type names (integer→int, boolean→bool; add string and float) - CustomFieldApiTests: fix 'integer' index type to 'int' (integer is not a supported type) - CustomFieldApiTests: replace ContainsKey+indexer with TryGetValue for SessionHasError assertion The 'integer' index type bug was silent: ConvertValue returns null for unknown types so the field would be stored but never actually indexed, with no error surfaced to the user. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ring
ConvertToKeyword and ConvertToString were calling value.ToString() without
a format provider. On servers with a non-en-US locale (e.g. de-DE), float,
double, and decimal values produce '1,5' instead of '1.5', breaking
Elasticsearch keyword-field searches because the stored value and the query
value use different decimal separators. DateTime values were also formatted
with the current thread culture instead of ISO 8601.
Fix: introduce a FormatInvariant helper that uses CultureInfo.InvariantCulture
for numeric types and the roundtrip 'O' ISO-8601 format for DateTime/DateTimeOffset.
The bool.ToString() behaviour ("True"/"False") is preserved for backwards
compatibility with already-indexed data.
Add regression tests that explicitly set CultureInfo to de-DE and assert
that the output always uses an invariant decimal separator. Also add tests
for long coercion in ConvertToBool, and for DateTime/DateTimeOffset keyword
formatting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds opt-in, organization-scoped custom fields for event data — letting users promote arbitrary
Datadictionary keys into Elasticsearch-indexed fields for use in search filters, saved views, and analytics dashboards.This is a fully additive change with no breaking API or schema changes.
What Changed
Backend
PersistentEventIHaveVirtualCustomFieldsto connect eventDatato Foundatio's pooled-slot indexing infrastructureEventIndexAddStandardCustomFieldTypes()— removes dynamic templates (no longer needed)EventCustomFieldServiceIStartupActionthat wires the document-changing pipeline hook; provisions system fields (sessionend,haserror) per org; validates field names (ASCII, no@prefix, max length); converts and type-checks values before slot writeOrganizationController{id}/event-custom-fields: GET list, POST create, PATCH update, DELETE soft-delete, GET candidates from eventRemoveCustomFieldWorkItemHandlerCustomFieldOptionsMaxFieldsPerOrganization = 20,MaxSlotChurn = 100CopySimpleDataToIdxActionandEventFieldsQueryVisitor— replaced by first-class field definitionsAPI Endpoints (new)
Plan gating: Free-plan organizations receive
426 Upgrade Requiredon POST.Reserved names: Any field starting with
@is rejected (reserved for system use).Active quota: Maximum 20 active fields per organization (soft-deleted fields do not count toward the quota; they do occupy their slot).
Deletion Policy
Deletion is deliberately two-phase to prevent slot reuse corruption:
IsDeleted = true, removes the field from the UI, blocks new events from indexing into this slot. Enqueues a cleanup work item.Saved-view protection: DELETE returns
409 Conflictwith field name if any saved view filter references the field (checks bothidx.{field}anddata.{field}patterns).Frontend (Svelte 5)
/organization/{id}/custom-fieldsUpgradeDialog)keyword→ equals/not-equals/exists/missing;string→ contains/exists/missing;numeric/date→ range/gt/gte/lt/lte/exists/missing;bool→ true/false/exists/missing)Architecture Docs
See
docs/custom-fields.mdfor the full architecture, slot system internals, lifecycle, deletion policy, operator support, FAQ, and churn analysis.Test Evidence
CustomFieldApiTestsEventCustomFieldServiceTestsCustomFieldIndexingTestsPersistentEventCustomFieldsTestsKey test scenarios covered
@-prefix, too long, non-ASCII)IsDeleted=true→ 404)idx.{field}anddata.{field}patterns)sessionend,haserror) on first createlong→float(range guard), date → UTC, bool coercionKnown Limitations
Breaking Changes
None. All changes are additive. Existing events, queries, saved views, and API clients are unaffected.
Original Requirements
Custom Fields System for Exceptionless
Allow organizations to define indexed fields from event data properties, enabling rich search, filter, and analytics capabilities for user-defined data.
Core Requirements
keyword,string,int,long,double,float,bool,dateManagement UI Requirements
Filter Operator Requirements
keyword: equals, not-equals, exists, missingstring/full text: contains/search, exists, missingnumeric/date: equals, range, gt, gte, lt, lte, exists, missingbool: true, false, exists, missingDeletion Policy
System Fields
sessionend(date) andhaserror(bool) provisioned per org automatically