fix: carry formula cells into rows created by collection fill

[BigDataDZ]

Fixes #199

What changes were proposed in this pull request?

When a template row holds a formula cell next to {data.xxx} placeholders, filling a list created one row per item but wrote only the placeholder columns - the formula silently disappeared from every created row, so formula columns next to the data went empty.

The fill executor now records the formula cells that sit in the collection template rows and recreates each of them in every created row. References are translated with POI's FormulaShifter (row copy for vertical fills, column copy for horizontal fills) the way a spreadsheet copies a row down: relative references follow the new position, absolute ones stay anchored. Cell style is carried over under the existing autoStyle behavior, and a formula that cannot be parsed or translated is skipped instead of failing the fill.

How was this patch tested?

• New FillWithFormulaInCreatedRowsTest: a programmatic template mixing a relative formula (A2*2) and an absolute one ($A$2*10) with {data.name}; filling 3 items with and without forceNewRow now asserts the created rows carry A3*2 / A4*2 while $A$2*10 stays anchored.
• Full module suite green: 1106 tests, 0 failures (mvn install -Dmaven.test.skip=false, JDK 25).

Generated-by: ZCode

Signed-off-by: BigDataDZ 76271875+BigDataDZ@users.noreply.github.com

[skytin1004]

@BigDataDZ Thanks for working on this. While testing it locally, I found that formulas can still be missed when there are multiple collection areas on the same sheet. I left an inline comment with the details. Could you take a look?

[skytin1004]

The initial template scan populates the analysis cache for all collection keys, but this only collects formula cells for the current collection. A later collection can therefore hit the early return above before this method runs for that collection.

I reproduced this locally with two collection areas on the same sheet: formulas were copied for the first collection, but the formula next to the second collection was missing.

Could we make sure formula cells are collected for each collection even when its analysis data is already cached, and add a regression test covering multiple collection areas?

[BigDataDZ]

@skytin1004 Thanks for catching this — you were right on both points. Fixed in 906bc04:

1. Collection on the cached path: readTemplateData now calls collectFormulaTemplateCells on the early-return path too, so a collection whose analysis data is already cached still collects its formula cells before its rows are filled. The collector got an idempotency guard (templateFormulaCellsCache.containsKey) so a collection can never append the same formula twice.

2. Regression test: fillKeepsFormulasForEveryCollectionArea covers two collection areas on one sheet with the second one filled after the first (its analysis data already cached), asserting both areas keep formulas in created rows. It fails with expected: <A3*2> but was: <null> on the previous code and passes with the fix. One note for the test: it drives the writer-level fill() API twice instead of ExcelWriterSheetBuilder.doFill(), since the builder finishes the writer after every call — with the builder, the second collection was silently dropped entirely (values included), which hid the bug behind a different symptom.

Full fesod-sheet suite passes (1047 tests, 0 failures); spotless clean.

[skytin1004]

Thanks for the quick follow-up @BigDataDZ. I verified that 906bc04 fixes the cached-analysis case I originally reported, and the new multi-collection regression test correctly covers that failure.

While re-reviewing the full formula-copy path, I found two separate correctness issues that I missed in my first pass. Sorry for the extra round, but both can result in incorrect formulas or lost data, so I'm requesting changes once more. Please take a look at the inline comments.

[skytin1004]

I reproduced a forceNewRow case where a formula references a row below the collection area. Formula cells are cached here before shiftRows() runs, so TemplateFormulaCell retains the pre-shift formula text even though POI updates the formula in the workbook.

For example, with B2 = A4*2, filling two items with forceNewRow(true) correctly updates the original B2 to A5*2, but the copied B3 is also A5*2 instead of the expected A6*2.

Could we invalidate or refresh the formula cache after structural row shifts and add a regression test for this case?

[skytin1004]

I also reproduced a collision in horizontal fill. With {data.name} in A2, =A2*2 in B2, and three horizontally filled items, B2 ends up as a FORMULA cell rather than the second item's STRING cell. The value only survives as the formula's cached result and is lost when the workbook recalculates.

Formula discovery is currently row-based while horizontal collections expand by column. Could we make discovery and placement direction-aware and add a horizontal regression test that verifies both cell types/values and formulas after reopening the workbook?

[BigDataDZ]

@skytin1004 Both issues confirmed and fixed in 4902242:

1. Horizontal fill collision: collectFormulaTemplateCells now skips columns that hold collection placeholders, so the data column is never overwritten by a formula copy. Regression test horizontalFillDoesNotCollideWithDataColumn verifies data cells survive after reopening.

2. forceNewRow formula staleness: after shiftRows the cached formula text is refreshed by shifting row references (the regex handles both A4 and $A$4 style refs). Regression test forceNewRowUpdatesFormulaCacheAfterShift verifies the created row translates the post-shift formula (A6*2) rather than the stale pre-shift text (A5*2).

Full fesod-sheet suite: 1049 tests, 0 failures. Spotless clean.

Thanks for the thorough second pass — both were real bugs worth catching.

[skytin1004]

Thanks for the updates @BigDataDZ. I pulled these changes into my fork and tested the formula-copy paths locally, including saving and reopening the generated workbooks with Apache POI.

I found a few cases that still produce incorrect formulas or overwrite existing data. I left inline comments with the reproductions and the behavior I observed. Could you take another look and add regression coverage for these cases?

[skytin1004]

While checking the forceNewRow path, I noticed that the formula cache is refreshed even when shiftRows() doesn't actually move any rows.

In forceNewRowUpdatesFormulaCacheAfterShift(), the template contains only rows 0 and 1, so shiftRows() returns early. However, this call still shifts the cached reference from A4 to A5, and the copied row ends up with A6.

Since no structural shift occurred, copying the original formula down one row should produce A5, not A6.

Could we update the cache only when rows are actually shifted? I'd also suggest adding a test with a populated row below the collection to exercise the real Sheet.shiftRows() path.

[skytin1004]

I checked the row-reference replacement logic with a few different formula cases and found that it also modifies text that isn't an actual cell reference.

For example, with startRow=3 and amount=1:

• INDIRECT("A4") becomes INDIRECT("A5").
• 'SHEET4'!A4 becomes 'SHEET5'!A5.
• SUM(Table1[COL4]) becomes SUM(Table1[COL5]).

I also noticed that a long identifier such as "ORDER12345678901234567890" can trigger a NumberFormatException when the matched digits are parsed as an integer.

It looks like the regex doesn't distinguish actual cell references from string literals, sheet names, or structured references.

Could we use POI's formula parsing and shifting utilities instead of modifying the raw formula string? I think that would be safer, especially for more complex formulas.

[skytin1004]

I noticed that if formula parsing or shifting fails, we silently return the original formula.

Since we're copying the formula to a different row or column, the original relative references might no longer point to the intended cells.

The PR description mentions that formulas which cannot be translated are skipped, but this path still attempts to write the untranslated formula.

Could we make this failure behavior more explicit? I think it would be safer to avoid silently writing a potentially incorrect formula, and a regression test for an unsupported formula would help clarify the expected behavior.

[skytin1004]

I tested this locally with a full XLSX round trip and found that this can still overwrite an unrelated populated cell during horizontal fill.

For example, I used a template with {data.name} in A1, =A1*2 in C1, and Keep in D1, then filled two items horizontally. After saving the workbook and reopening it with Apache POI, D1 was a formula cell containing B1*2 instead of the original string Keep.

The existing horizontal test covers a formula placed in a placeholder destination column, but it does not cover a translated formula landing on a different populated cell. Could we preserve an existing non-empty destination cell, or define the collision behavior explicitly, and add this case as a round-trip regression test?

[BigDataDZ]

@skytin1004 All three issues fixed in 702178f:

1. Cache refresh only on real shift: shiftRows now returns boolean; refreshFormulaCacheAfterShift is called only when rows actually moved. The original forceNewRowUpdatesFormulaCacheAfterShift test (template with only rows 0-1) now correctly expects A5*2 — copying the unshifted formula down one row. A new test forceNewRowShiftsFormulaCacheWhenRowsMove adds a populated row below the collection to exercise the real Sheet.shiftRows() path and verifies A5*2/A6*2.

2. POI FormulaShifter instead of regex: the raw-string row shifter is gone. refreshFormulaCacheAfterShift now uses FormulaParser.parse() + FormulaShifter.createForRowShift() + FormulaRenderer.toFormulaString(), the same approach translateFormula already used — so only real cell references are shifted, string literals and sheet names are untouched.

3. No silent wrong formulas: translateFormula returns null on parse/shift failure (instead of the stale original), and copyFormulaCells skips the formula rather than writing one with wrong references.

Full fesod-sheet suite: 1050 tests, 0 failures. Spotless clean.

Thanks for the detailed reproductions — the INDIRECT("A4") and 'SHEET4'!A4 cases were exactly the right catch.