Feat/firehose configurable resources - #121

Closed

mabdh wants to merge 142 commits into

raystack:mainfrom

goto:feat/firehose-configurable-resources

Commits on Mar 2, 2023

fix: kube deployment char limit
spy16
committed
ecf98ee

Commits on Mar 7, 2023

feat: add deployment_id feature
spy16
committed
d48ba77

Commits on Mar 13, 2023

Merge pull request #1 from goto/fix/bugs
spy16
authored
5d3d7ad

Commits on Mar 14, 2023

refactor: rename to goto (#2)
ishanarya0
authored
4867aa2
fix: golanci config (#3)
ishanarya0
authored
3b338aa

Commits on Mar 15, 2023

fix: kube config handling (#4)
spy16
authored
4d909e6
fix: deployment id name (#5)
spy16
authored
e96dcf9

Commits on Mar 16, 2023

Fix/generic (#6)
spy16
authored
202e4ad
fix: change odpf reference to goto (#7)
ishanarya0
authored
e758108

Commits on Mar 17, 2023

generic refactors (#8)
spy16
authored
26d5226
fix: firehose image name (#9)
spy16
authored
ab5b2dc

Commits on Mar 21, 2023

refactor: re-implement firehose module (#10)
spy16
authored
022f7a4
fix: use previous telegraf conf (#11)
spy16
authored
4b6a238
fix: length validation issue (#12)
spy16
authored
152a191

Commits on Mar 24, 2023

fix: goreleaser commit author (#14)
ishanarya0
authored
f66d8bf
feat: add stop flag support (#13)
spy16
authored
fade8c7

Commits on Mar 25, 2023

feat: add stop flag support (#13) (#15)
spy16
authored
226fd7f

Commits on Mar 27, 2023

fix: revision listing db query (#16)
spy16
authored
cffb11f
fix: expose chart values type (#17)
spy16
authored
b93cbb8

Commits on Mar 30, 2023

feat: strip trailing colon in image tag, add telegraf config template
spy16
authored
b063bba
fix: telegraf config format (#19)
spy16
authored
fb2202d
feat: add support for dynamic telegraf labels (#20)
spy16
authored
eacb30f
fix: use go map for telegraf image spec (#21)
spy16
authored
9a37814
fix: use map to fix yaml marshal (#22)
spy16
authored
52671a6
fix: set deployment-id length limit as 63 (#23)
spy16
authored
2b799d7
fix: revert 63 limit to 53 due helm limitation (#24)
spy16
authored
3a368d2
feat: refactor release name to be consistent with old format (#25)
spy16
authored
d3e42c0
fix: helm release name limit (#26)
spy16
authored
cbc5e88

Commits on Apr 3, 2023

feat: implement deployment label templating (#27)
spy16
authored
dc4b7ca

Commits on Apr 4, 2023

fix: expose name in firehose module (#28)
spy16
authored
199def5

Commits on Apr 5, 2023

refactor: simplify helm package (#29)
spy16
authored
00fc0d6
feat: add labels to list-resource api (#30)
spy16
authored
a354d34

Commits on Apr 6, 2023

feat: merge action and existing labels (#31)
spy16
authored
84ec3cc
fix: remove name label to prevent length issues (#32)
spy16
authored
4538cb7

Commits on Apr 11, 2023

feat: add jdbc sink type (#33)
ishanarya0
authored
71766cb

Commits on Apr 12, 2023

feat: allow custom usage for each firehose (#34)
ishanarya0
authored
4e03452

Commits on Apr 18, 2023

feat: add tolerations feature (#35)

spy16
and
StewartJingga
authored
b013688

Commits on Apr 21, 2023

fix: move volume values outside firehose key (#36)

spy16
and
ishanarya0
authored
a5f3898
fix: add secret key (#37)
ishanarya0
authored
d3193e7

Commits on Apr 27, 2023

feat: add toleration to kube module config (#38)
ishanarya0
authored
0919cc5

Commits on May 9, 2023

feat: add support for sink specific requests & limits (#39)
spy16
authored
74d17e4

Commits on May 17, 2023

feat: add firehose base config (#40)
ishanarya0
authored
7e4caa7

Commits on Jul 17, 2023

Commits on Jul 19, 2023

feat: stoptime in firehose start (#46)
ishanarya0
authored
f3642c1

Commits on Jul 25, 2023

refactor: public ScaleParams & StartParams (#47)
ishanarya0
authored
cadc134

Commits on Jul 26, 2023

feat: add urn to deployment labels (#48)
ishanarya0
authored
2217ff1

Commits on Jul 27, 2023

Commits on Jul 28, 2023

fix: initialize as empty arrays (#51)
ishanarya0
authored
aac1e7f

Commits on Jul 31, 2023

feat: firehose update to take the latest driver configs (#52)
ishanarya0
authored
0cb3d28

Commits on Aug 7, 2023

fix: update stopped status (#53)
ishanarya0
authored
b66407d

Commits on Aug 16, 2023

feat: add logging request body (#54)
ishanarya0
authored
01dc9fe

Commits on Aug 21, 2023

feat: mounts as secrets (#55)
ishanarya0
authored
b58f67f
fix: mount secrets (#56)
ishanarya0
authored
43d5983
fix: key name (#57)
ishanarya0
authored
63bc82b

Commits on Aug 24, 2023

fix: add bigtable cred path (#58)
ishanarya0
authored
198c661
fix: remove skipping empty values for env variables (#59)
ishanarya0
authored
e0da338

Commits on Aug 28, 2023

fix: mount path to be independent of dep id (#60)
ishanarya0
authored
bc5e389

Commits on Aug 30, 2023

feat: return only running & non-terminating pods (#61)
ishanarya0
authored
4397eb2
feat: get resource to persist output (#62)
ishanarya0
authored
305fb47

Commits on Aug 31, 2023

feat: add name as extra label (#64)
ishanarya0
authored
f82209f

Commits on Sep 4, 2023

feat: support google service account for gke usage (#63)
mabdh
authored
32547ef

Commits on Sep 7, 2023

test: payloads sent to LOG, BQ and GCS/BLOB sinks (#66)
ishanarya0
authored
851bee4

Commits on Sep 11, 2023

fix: add delay before executing reset (#69)
ishanarya0
authored
4137fee

Commits on Sep 14, 2023

fix: job name should be within 63 char limit (#70)
ishanarya0
authored
9773fcd
fix: remove -firehose from job name (#71)
ishanarya0
authored
ee9a105

Commits on Sep 20, 2023

feat: old reset strategy (#72)
ishanarya0
authored
2c4ef1f

Commits on Sep 21, 2023

fix: empty body while logging (#67)
ishanarya0
authored
4fdfd50

Commits on Sep 27, 2023

feat: add global logger (#75)
lavkesh
authored
876f246
feat: add global logger (#76)
lavkesh
authored
e6b56cc

Commits on Sep 29, 2023

feat: adding job module (#74)

lavkesh
and
mabdh
authored
f0f1a8c

Commits on Oct 3, 2023

feat: add shared volume (#77)
lavkesh
authored
44b26ce

Commits on Oct 4, 2023

Commits on Oct 6, 2023

fix: init container issue (#79)
ishanarya0
authored
b0bff0f

Commits on Oct 9, 2023

feat: on delete cascade in tags table (#80)
ishanarya0
authored
8222c63

Commits on Oct 12, 2023

feat: add joins in list query (#82)
ishanarya0
authored
ae35b4b

Commits on Oct 13, 2023

feat: optional spec configs (#83)
ishanarya0
authored
76eba64

Commits on Oct 20, 2023

fix: allow overiding user passed resource & limits (#84)
ishanarya0
authored
4d0e906

Commits on Nov 2, 2023

feat: add sink based kube namespace selection (#86)
ishanarya0
authored
2a0d67d
feat: templatize telegraf config output (#87)
ishanarya0
authored
8579741

Commits on Nov 10, 2023

fix: remove validation on resource name (#89)
ishanarya0
authored
d0f20ea

Commits on Nov 15, 2023

chore: if namespace is empty take from default (#85)
lavkesh
authored
f2387ca

Commits on Nov 22, 2023

Namespace conf (#90)
lavkesh
authored
16ffdac
Namespace should be taken from resource conf (#91)
lavkesh
authored
3089ca9

Commits on Dec 13, 2023

feat: make kube deployment timeout, extendInterval etc configurable (#92)
ishanarya0
authored
168a45d

Commits on Feb 23, 2024

fix: avoid typecheck and upgrade linter version (#94)

manishdangi98
and
Manish Dangi
authored
69f16e6
feat: upgrade lint action (#95)
ishanarya0
authored
f77ad4f
Update .golangci.yml (#96)
ishanarya0
authored
bd86c45
feat: exclude rules for typecheck linter (#97)
ishanarya0
authored
db0fd9a
feat(firehose): add HTTPV2 sink type (#93)
abhishekv24
authored
d4ec916

Commits on Mar 7, 2024

feat: inc writeTimeout (#98)
ishanarya0
authored
b1a2680

Commits on Apr 22, 2024

feat: helm release config validation during plan (#99)
ishanarya0
authored
35322f1

Commits on May 8, 2024

feat: list resources pagination (#100)
FemiNoviaLina
authored
e252bc6

Commits on May 14, 2024

fix: take pagination default value from config (#101)
FemiNoviaLina
authored
ef1cee9

Commits on Jul 11, 2024

feat: module cli (#102)
FemiNoviaLina
authored
1778229

Commits on Jul 30, 2024

feat: overridable firehose image repository (#103)
mabdh
authored
136b628

Commits on Aug 13, 2024

test: e2e-test (#104)
FemiNoviaLina
authored
9077875

Commits on Aug 23, 2024

feat: modular worker (#105)
FemiNoviaLina
authored
3fcd8c4

Commits on Aug 27, 2024

fix: change worker config to map (#106)
FemiNoviaLina
authored
d947041

Commits on Aug 28, 2024

feat: enable opentelemetry for worker (#108)
FemiNoviaLina
authored
417f10e

Commits on Sep 9, 2024

feat: add kafka resource type (#109)
FemiNoviaLina
authored
80d5637

Commits on Sep 10, 2024

feat: kafka module config merge (#111)
FemiNoviaLina
authored
eb750e2

Commits on Oct 16, 2024

feat: add dry run (#116)

ishanarya0
and
Ishan Arya
authored
bac23e0

Commits on Nov 14, 2024

feat: allow maxcompute sink (#118)
FemiNoviaLina
authored
46a524e

Commits on Dec 24, 2024

feat: add dagger (#115)

ishanarya0
and
Ishan Arya
authored
37c2a81

Commits on Jan 7, 2025

feat: kube resource support namespace (#121)
mabdh
authored
c124014

Commits on Jan 13, 2025

fix: flink parallelism update (#122)

ishanarya0
and
Ishan Arya
authored
77eba57
Flink parallelism (#123)

ishanarya0
and
Ishan Arya
authored
6828b8d

Commits on Jan 28, 2025

fix: fix key to SCHEMA_REGISTRY_STENCIL_URLS (#124)

ishanarya0
and
Ishan Arya
authored
953741b

Commits on Feb 27, 2025

feat: add metrics for res status change (#126)

ishanarya0
and
Ishan Arya
authored
5862a9e

Commits on Feb 28, 2025

chore: add debug logs (#128)

ishanarya0
and
Ishan Arya
authored
0fa074c
chore: counter logs (#129)

ishanarya0
and
Ishan Arya
authored
163ee15

Commits on Mar 4, 2025

feat(dagger): add dagger bucket url config (#127)
FemiNoviaLina
authored
eab3919

Commits on Mar 14, 2025

feat: entropy metrics (#130)

ishanarya0
and
Ishan Arya
authored
2e13b12

Commits on Apr 18, 2025

fix: change go struct to go map (#131)

ishanarya0
and
Ishan Arya
authored
54aaba1
fix: add empty check (#132)

ishanarya0
and
Ishan Arya
authored
e882108

Commits on Apr 21, 2025

feat: add helm max history configuration (#133)

ishanarya0
and
Ishan Arya
authored
6b32950

Commits on Apr 30, 2025

fix: add concurrency handling (#134)
ishanarya0
authored
8bc083e

Commits on May 5, 2025

feat: old reset strategy (#135)

ishanarya0
and
Ishan Arya
authored
6e6a6fb

Commits on May 15, 2025

feat: add burst and qps config (#137)

ishanarya0
and
Ishan Arya
authored
a1f9295
feat: log failure response (#138)

ishanarya0
and
Ishan Arya
authored
a9be12e

Commits on Jun 25, 2025

feat: update flink resource (#139)

ayushi0014
and
Ayushi Sharma
authored
67d75ea

Commits on Aug 25, 2025

feat: enable firehose keda autoscaler (#140)
FemiNoviaLina
authored
c85f698

Commits on Aug 26, 2025

feat(firehose): add keys to keda metadata template config (#141)
FemiNoviaLina
authored
496fc60

Commits on Sep 3, 2025

feat: kubernetes affinity config (#142)
FemiNoviaLina
authored
4d2cdd5

Commits on Sep 4, 2025

feat: add desired status in firehose state output (#143)
mabdh
authored
d61dcf9

Commits on Sep 10, 2025

feat(firehose): support taint and affinity for autoscaler (#144)
mabdh
authored
bdfeda6
fix: toleration affinity mode kube (#146)
mabdh
authored
7928d5e

Commits on Sep 11, 2025

feat(firehose): sink based autoscaler config (#145)
FemiNoviaLina
authored
96dac2c

Commits on Sep 15, 2025

feat(autoscaler): add config validation (#147)
mabdh
authored
995f4d3

Commits on Sep 16, 2025

feat(firehose): kafka scaler support multiple topic (#148)
FemiNoviaLina
authored
ffa4242

Commits on Sep 17, 2025

fix: sink based keda config (#149)
FemiNoviaLina
authored
71cfbe6

Commits on Oct 1, 2025

feat(firehose): allow keda trigger override from resource config (#150)
FemiNoviaLina
authored
47c3a39

Commits on Oct 10, 2025

feat: add default ttl to remove job after finished (#152)

ayushi0014
and
Ayushi Sharma
authored
e293ef5

Commits on Oct 13, 2025

feat: update job ttl to remove job (#153)

ayushi0014
and
Ayushi Sharma
authored
e5a8465

Commits on Oct 15, 2025

feat: integrate with otel (#151)

mabdh
and
FemiNoviaLina
authored
168093e

Commits on Jul 7, 2026

feat(firehose): declare limits/requests in config schema

mabdh
and
claude
committed
672bb69