fix: classify unsupported format patterns as Unsupported in CometFromUnixTime

[marvelshan]

Which issue does this PR close?

Closes #4575

Rationale for this change

CometFromUnixTime incorrectly reports non-default datetime format patterns as Incompatible instead of Unsupported. This is misleading because:

• Incompatible implies the expression can run natively if the user opts in via spark.comet.expr.allowIncompatible=true, but non-default formats cannot be executed natively at all — the convert() method returns None for them regardless of configuration.
• The GenerateDocs tool uses getIncompatibleReasons() and getUnsupportedReasons() to produce the public Compatibility Guide. Classifying format limitations as "incompatible" misleads users into thinking they can enable support by setting allowIncompatible=true.
• This was identified as item Set up RAT checks in CI #3 in Fix issues found when documenting current expression incompatibilities #4074, which tracks expression support-level misclassifications discovered during the docs generation work in docs: add compatibility documentation to all expressions #4067.

What changes are included in this PR?

• Split getIncompatibleReasons() to only contain the timestamp range difference (which is a true incompatibility — native can execute but results may differ at boundaries)
• Added getUnsupportedReasons() to document format pattern limitations (native cannot execute these at all)
• Updated getSupportLevel(expr) to dynamically return Unsupported for non-default format patterns and Incompatible for the default pattern, matching the actual behavior in convert()

How are these changes tested?

Existing tests cover the fallback behavior. The convert() method already returns None for non-default formats, and getSupportLevel() now aligns with that behavior. The GenerateDocs output will correctly classify format limitations under "Unsupported" rather than "Incompatible".

[coderfender]

nit : might be an unwanted change ?

[coderfender]

We might wnat to add some SQL tests to verify the fallback and make sure the plan is annotated

[andygrove]

I think this change may prevent the codegen dispatch path from working.

[andygrove]

This query was previously working (presumably due to codegen dispatch). This seems like a regression.

[andygrove]

Thanks @marvelshan. The instinct here is good. Separating the genuine timestamp-range incompatibility from the format-pattern caveat is the right idea, and pulling the format text out of getIncompatibleReasons() so that it only describes what allowIncompatible=true actually changes (the native to_char path) is a real improvement. The from_unix_time_enabled.sql cleanup is nice too, since the default pattern really does run natively under allowIncompatible=true and asserting that with query is more accurate.

A few things I would like to work through before this lands.

1. The code keeps this Incompatible, not Unsupported. The title and the description say getSupportLevel now returns Unsupported for non-default patterns, but both branches return Incompatible:

override def getSupportLevel(expr: FromUnixTime): SupportLevel = {
  if (expr.format != Literal(TimestampFormatter.defaultPattern())) {
    Incompatible(Some("Only the default datetime pattern ... is supported natively"))
  } else {
    Incompatible(None)
  }
}

I actually think keeping it Incompatible is correct. CometFromUnixTime is a CodegenDispatchFallback, so non-default formats are handled by the JVM codegen dispatcher rather than being truly unsupported. If this returned Unsupported, the framework would skip the dispatcher and force a full fall back to Spark, which would be a regression. So the code looks right, it is the title and the "What changes are included" section that need to be updated to match.

2. getUnsupportedReasons() will read as self-contradictory in the docs. GenerateDocs calls getUnsupportedReasons() unconditionally and never consults getSupportLevel, so this entry renders under the heading "The following cases are not supported by Comet:", while the bullet text itself says non-default patterns are handled by the codegen dispatcher. So the generated page says a case is not supported and handled in the same sentence, and that does not match runtime behavior where non-default formats run in Comet via the dispatcher by default. date_format has the same native-allow-list-plus-dispatcher shape and documents it in getCompatibleNotes() (datetime.scala:715). Could the format caveat move there instead? That keeps getIncompatibleReasons() as just the timestamp range, which is the part of this change I like, and avoids the misleading "Unsupported" heading.

3. The non-default test downgrade looks inconsistent. In from_unix_time.sql, from_unixtime(t, 'yyyy-MM-dd') moved from query to query spark_answer_only, but the default-pattern from_unixtime(t) stayed query. At allowIncompatible=false both take the same codegen-dispatch path, so if the default-pattern one passes as native, the non-default one should too. Could those stay query so they keep asserting the dispatcher holds them in Comet? If query actually fails for the non-default case, that would be worth surfacing, since it would mean the dispatcher is not covering it the way the new reason text claims.

4. Minor. getSupportLevel calls TimestampFormatter.defaultPattern() with parens while convert() just below calls defaultPattern without. The parens form matches the Spark definition, so it is fine, but the two call sites reading differently in the same object is worth tidying. Also a heads up that #4693 also edits this serde's getSupportLevel and getIncompatibleReasons for collation, so whichever of the two merges second will need a rebase.

The getIncompatibleReasons() split and the enabled-test fixes are solid. The main rework is routing the format caveat through getCompatibleNotes() rather than getUnsupportedReasons(), and squaring the title and description with the code.

Disclosure: this review was assisted by an AI tool (Anthropic Claude via Claude Code). I read the diff, the linked issues, and the related serde code before forming these comments, and I take responsibility for them.