fix: compute time_idx in steps for non-monthly panel data by anishmehta24 · Pull Request #1636 · microsoft/FLAML

Anish Mehta (anishmehta24) · 2026-10-15T11:43:52Z

Why are these changes needed?

add_time_idx_col (used for ts_forecast_panel when the data has no time_idx column, and in the TFT predict path) is broken for any frequency other than monthly/yearly:

timestamps - timestamps.min() / freq only divides the minimum, so time_idx ends up as the raw nanosecond timestamp. For daily data starting 2024-01-01 the first value is 1704067199999980288 instead of 0.
The values are computed from the unique dates and assigned back to X, so every row whose timestamp repeats (any panel with more than one series) gets NaN, and .astype("int") raises IntCastingNaNError.
Series.view was removed in pandas 3, so there it fails with AttributeError before any of that.

This computes (timestamp - first timestamp) // step for every row instead.

Related issue number

None, found while reading the time series code.

Checks

I've used pre-commit to lint the changes in this PR (note the same in integrated in our CI checks).
I've included any doc changes needed for https://microsoft.github.io/FLAML/. See https://microsoft.github.io/FLAML/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

test/automl/test_add_time_idx_col.py fails on main (pandas 2.3 and 3.0) and passes with the change.

add_time_idx_col divided only the minimum timestamp by the step (operator precedence), so time_idx held raw nanosecond values, and it assigned the per-unique-date series back to X, which left NaN for any repeated timestamp in panel data and crashed the int cast. It also used Series.view, which pandas 3 removed.

Copilot

🟢 Approval recommended

The focused fix addresses the reported defects with regression coverage and no identified blocking issues.

No open issues 🎉

What changed in this PR

Fixes time-step indexing for non-monthly/yearly panel forecasting data.

Changes:

Computes integer time steps for every row, including repeated timestamps.
Replaces removed pandas Series.view usage with astype.
Adds regression tests for daily data and shared panel timestamps.

File	Description
test/automl/test_add_time_idx_col.py	Tests daily step counts and repeated timestamps.
flaml/automl/data.py	Corrects timestamp arithmetic and row alignment.

_{Review effort: Balanced}

Li Jiang (thinkall)

Overall review of the complete PR: no blocking code issue found; approval withheld pending CI.

The corrected subtraction/division order computes elapsed time steps, and assigning from the original timestamp column preserves repeated panel dates and row-index alignment. Both focused regression tests passed locally. The review-code harness is clean and Copilot's current Balanced review recommends approval with no open issues. The current head still has failing build checks, so the all-CI-passed approval gate is not met. This review does not approve the PR for merge.

Posted by thinkall-agent-auto-reviewer

Copilot

🟢 Approval recommended

The focused fix and regression tests address the reported failures with no identified blocking issues.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🟢 Approval recommended

The focused fix addresses the reported failures with regression coverage and no unresolved findings.

No open issues 🎉

_{Review effort: Balanced}

Li Jiang (thinkall)

Overall review at c99d0484e355d7df37ffbfd8fa0060e133ebab21: the ordinary daily/panel correction is useful, but one P2 correctness issue remains in the new step mapping. Complete regular month-end and timezone-aware daily series still receive duplicate indices and artificial gaps; the inline review includes independently executed reproductions.

Validation: python -m pytest test\automl\test_add_time_idx_col.py -q --disable-warnings passed (2 passed). Separate repeated-panel calendar/DST fixtures fail the consecutive-step invariant, and the review-code whole-PR Low harness independently identified the same root cause. Green CI and a positive Copilot recommendation do not resolve this source defect, so approval is withheld. No contributor code was modified.

Posted by thinkall-agent-auto-reviewer

Li Jiang (thinkall) · 2026-10-19T01:41:00Z

-        X["time_idx"] = X["time_idx"].astype("int")
+        timestamps = unique_dates.astype("int64")
+        freq = int(timestamps.diff().mode().iloc[0])
+        X["time_idx"] = (X[TS_TIMESTAMP_COL].astype("int64") - timestamps.min()) // freq


P2 - Count calendar steps rather than fixed elapsed durations. This floor division assumes that every step has the same epoch duration, even when pd.infer_freq identifies a calendar frequency. With six complete month ends from 2021-01-31, freq='ME', the actual indices are [0,0,1,2,4,5]. With six America/New_York local days from 2021-03-12, crossing spring DST, they are [0,1,2,2,3,4]. Each should have relative steps [0,1,2,3,4,5]. I reproduced both with the dates repeated across two panel groups; distinct observations collide within each group even though no timestamps are missing.

TFT consumes these indices for dataset validation, encoder windows, and decoder alignment, so valid regular series can be rejected or use a distorted forecast window. Map recognized calendar-frequency steps to ordinal positions instead of dividing elapsed time by a modal duration, preserving equal indices for shared panel timestamps and row order. Please add exact repeated-panel month-end and spring-DST regressions. This is an incomplete correction of the helper, not a claim that the original main implementation already handled these cases correctly.

Posted by thinkall-agent-auto-reviewer

Fixed in e796511. When pd.infer_freq finds a frequency the index is now the position among the unique timestamps, so month ends and the DST week both give [0..5], with shared panel dates kept equal. Irregular series still fall back to elapsed time over the modal step so gaps are kept. Added the month-end and spring-DST panel tests plus one for the gap case.

Copilot

🟢 Approval recommended

The focused fix and regression tests address the reported failures, with no blocking issues identified; tests were not executed during review.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Irregular daily data still loses steps across DST, and the month-end test uses an alias unavailable in older supported pandas versions.

1 open issue

Use compatible MonthEnd offset instead of pandas 2.2-only ME frequency · New

1 previously missed issue

In code that hasn't changed since last review

DST causes irregular daily data to lose calendar-day spacing

flaml/automl/data.py:266

The irregular fallback still loses calendar-day steps across DST. For New York midnights on March 12, 13, 14, 15, and 17, 2021, infer_freq returns None. The modal interval is 24 hours, so this division produces [0, 1, 2, 2, 4] instead of [0, 1, 2, 3, 5]. Distinct dates then share a time_idx. Preserve calendar-day spacing for irregular daily data without changing elapsed-time indexing for intraday data, and add a regression combining DST with a missing date.

_{Review effort: Balanced}

+
+
+def test_month_end_time_idx_with_shared_timestamps():
+    dates = list(pd.date_range("2021-01-31", periods=6, freq="ME"))


Li Jiang (thinkall)

Reviewed e796511b5c07e38efe7a190063396202aa98f934 with review-code (Low, gpt-6.1-sol). The complete regular month-end/DST mapping is repaired on modern pandas, but changes are requested for supported-version compatibility.

[P2] Keep the new month-end fixture within the declared pandas range. Both AutoML and test extras permit pandas>=1.1.4, while the new freq="ME" fixture requires pandas 2.2+. I reproduced ValueError: Invalid frequency: ME natively with Python 3.10/pandas 1.5.3. pd.offsets.MonthEnd() produces the same six dates on pandas 1.5.3, 2.3.3 and 3.0.5. This corroborates the existing Copilot finding; no duplicate inline is being added.

The actual focused file reports 5 passed on pandas 3.0.5, but 2 failed, 3 passed on pandas 1.5.3. The second failure is the timezone test: the unchanged pd.infer_freq(Series) call rejects timezone-aware Series on that runtime. Exact prior and current code both show that TypeError, so I am not attributing it to the new ordinal branch. The same dates wrapped in a DatetimeIndex infer D; please ensure the newly added DST regression also runs across the declared pandas range.

Fifteen native calendar/DST/permutation controls verify the modern regular-grid fix and original panel-row/index preservation. Existing irregular modal-gap/DST limitations are recorded separately and are not claimed fixed or introduced by this correction. I have not resolved my older signed calendar/DST thread while supported-version handling remains incomplete, and no other reviewer's thread was touched.

No contributor implementation/test or installed package was changed. Approval remains withheld.

Posted by thinkall-agent-auto-reviewer

Anish Mehta (anishmehta24) and others added 2 commits October 15, 2026 16:49

Merge branch 'main' into fix/ts-panel-time-idx

3222b11

Li Jiang (thinkall) requested a balanced review from Copilot October 17, 2026 04:39

Copilot started reviewing on behalf of Li Jiang (thinkall) October 17, 2026 04:40

View session

Copilot AI reviewed Oct 17, 2026

View reviewed changes

Li Jiang (thinkall) reviewed Oct 17, 2026

View reviewed changes

Merge branch 'main' into fix/ts-panel-time-idx

ff3c7ee

Li Jiang (thinkall) requested a balanced review from Copilot October 18, 2026 10:05

Copilot started reviewing on behalf of Li Jiang (thinkall) October 18, 2026 10:06

View session

Copilot AI reviewed Oct 18, 2026

View reviewed changes

Merge branch 'main' into fix/ts-panel-time-idx

c99d048

Li Jiang (thinkall) requested a balanced review from Copilot October 18, 2026 20:10

Copilot started reviewing on behalf of Li Jiang (thinkall) October 18, 2026 20:11

View session

Copilot AI reviewed Oct 18, 2026

View reviewed changes

Li Jiang (thinkall) requested changes Oct 19, 2026

View reviewed changes

Merge branch 'main' into fix/ts-panel-time-idx

3fd0b37

Li Jiang (thinkall) requested a balanced review from Copilot October 19, 2026 06:42

Copilot started reviewing on behalf of Li Jiang (thinkall) October 19, 2026 06:43

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

fix: count calendar and DST steps by position in add_time_idx_col

e796511

Li Jiang (thinkall) requested a balanced review from Copilot October 21, 2026 19:15

Copilot started reviewing on behalf of Li Jiang (thinkall) October 21, 2026 19:15

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread test/automl/test_add_time_idx_col.py

def test_month_end_time_idx_with_shared_timestamps():

dates = list(pd.date_range("2021-01-31", periods=6, freq="ME"))

Li Jiang (thinkall) requested changes Oct 21, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix: compute time_idx in steps for non-monthly panel data - #1636

fix: compute time_idx in steps for non-monthly panel data#1636
Anish Mehta (anishmehta24) wants to merge 6 commits into
microsoft:mainfrom
anishmehta24:fix/ts-panel-time-idx

Anish Mehta (anishmehta24) commented Oct 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Li Jiang (thinkall) left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Li Jiang (thinkall) left a comment

Uh oh!

Li Jiang (thinkall) Oct 19, 2026

Uh oh!

Anish Mehta (anishmehta24) Oct 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Li Jiang (thinkall) left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		def test_month_end_time_idx_with_shared_timestamps():
		dates = list(pd.date_range("2021-01-31", periods=6, freq="ME"))

Conversation

Anish Mehta (anishmehta24) commented Oct 15, 2026

Why are these changes needed?

Related issue number

Checks

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Li Jiang (thinkall) left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Li Jiang (thinkall) left a comment

Choose a reason for hiding this comment

Uh oh!

Li Jiang (thinkall) Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Anish Mehta (anishmehta24) Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Li Jiang (thinkall) left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants