Skip to content

Commit 4e6cf4e

Browse files
committed
Merge main into fix-cloudfetch-zero-duration
Signed-off-by: Maha Rana Yadavalli <271375718+maharanay22@users.noreply.github.com>
2 parents 88ae5e2 + 559f1d3 commit 4e6cf4e

9 files changed

Lines changed: 795 additions & 9 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Release History
22

33
# Unreleased
4+
- Fix inline parameters (`use_inline_params=True`) so a sequence directly after `IN` renders as a list again: `x IN %(ids)s` renders as `x IN (1,2,3)` instead of `x IN ARRAY(1,2,3)`, which Spark rejects. Inside the list, tuples render as rows for a multi-column `IN`, and a `set` or `frozenset` is accepted again. Sequences anywhere else still render as `ARRAY(...)` ([#629](https://github.com/databricks/databricks-sql-python/issues/629))
45
- Fix: a CloudFetch download that finished faster than the clock resolution (common on Windows) no longer fails the fetch with `ZeroDivisionError` while logging its download speed. Download time is now measured with `time.perf_counter()`.
56

67
# 4.9.0 (2026-11-04)

‎docs/parameters.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,25 @@ Output:
235235
SELECT field FROM table WHERE field IN (1,2,3,4,5)
236236
```
237237

238+
The marker must directly follow `IN` (or `NOT IN`), as in `IN %(value_list)s`. Anywhere else, including inside parentheses you write yourself such as `IN (%(value_list)s)`, a sequence is rendered as an `ARRAY(...)` literal.
239+
240+
The connector finds `IN` by reading the query text, not by parsing SQL. It ignores an `IN` at the end of a `--` comment line, but an `IN` inside a string literal, directly before a marker, is still treated as the keyword.
241+
242+
Inside the list, a tuple is rendered as a row and a list as an `ARRAY(...)` literal. For a multi-column `IN`, pass tuples:
243+
244+
```python
245+
query = "SELECT * FROM table WHERE (id, tag) IN %(pairs)s"
246+
parameters = {"pairs": [(1, "a"), (2, "b")]}
247+
```
248+
249+
```sql
250+
SELECT * FROM table WHERE (id, tag) IN ((1,'a'),(2,'b'))
251+
```
252+
253+
Spark does not convert the types of the values inside a row, so each value must have the same type as its column. For example, the literal `1` is an `INT` and does not match a `BIGINT` column.
254+
255+
A `set` or `frozenset` can also be passed directly after `IN`. Its items are sorted, which usually keeps the query text the same from run to run. Items without a consistent order, such as `float("nan")`, can still change position. Anywhere else a set raises `ProgrammingError`, because a set has no order to keep in an `ARRAY(...)` literal.
256+
238257
**Note**: this behavior is not specified by PEP-249 and only works due to the specific implementation of our inline renderer.
239258

240259
### Migrating to native parameters

‎poetry.lock‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎src/databricks/sql/client.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,6 @@
4646
from databricks.sql.backend.databricks_client import DatabricksClient
4747
from databricks.sql.utils import (
4848
ParamEscaper,
49-
inject_parameters,
5049
transform_paramstyle,
5150
ColumnTable,
5251
ColumnQueue,
@@ -1211,8 +1210,7 @@ def _prepare_inline_parameters(
12111210
The list is always empty because native parameters are never used under the inline approach
12121211
"""
12131212

1214-
escaped_values = self.escaper.escape_args(params)
1215-
rendered_statement = inject_parameters(stmt, escaped_values)
1213+
rendered_statement = self.escaper.render_inline(stmt, params)
12161214

12171215
return rendered_statement, NO_NATIVE_PARAMS
12181216

‎src/databricks/sql/utils.py‎

Lines changed: 277 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,27 @@
11
from __future__ import annotations
2-
from typing import Any, Dict, List, Optional, Tuple, Union, Sequence, TYPE_CHECKING
2+
from typing import (
3+
Any,
4+
Callable,
5+
Dict,
6+
List,
7+
Optional,
8+
Set,
9+
Tuple,
10+
Union,
11+
Sequence,
12+
TYPE_CHECKING,
13+
)
314

415
from dateutil import parser
516
import datetime
617
import decimal
718
from abc import ABC, abstractmethod
819
from collections import OrderedDict, namedtuple
9-
from collections.abc import Mapping
20+
from collections.abc import Mapping, Set as AbstractSet
1021
from decimal import Decimal
1122
from enum import Enum
1223
import re
24+
import secrets
1325

1426
import lz4.frame
1527

@@ -557,6 +569,160 @@ def user_friendly_error_message(self, no_retry_reason, attempt, elapsed):
557569
return user_friendly_error_message + ". " + error_context
558570

559571

572+
# A probe token directly after the IN keyword (also matches NOT IN). Where Spark's grammar accepts a
573+
# value directly after IN, in the IN predicate and in PIVOT, it requires a parenthesised list, so
574+
# `IN ARRAY(...)` is a syntax error. (The one other place, `position(a IN b)`, takes a string and
575+
# rejects an ARRAY too.) Requiring the marker to follow IN directly leaves `IN (%(p)s)` alone: there
576+
# the caller's parentheses already form the list and an ARRAY value is a valid list element.
577+
#
578+
# This matches the SQL text rather than parsing it, so it is a heuristic. It skips an IN that ends
579+
# a `--` comment line (see `_in_ends_line_comment`), but an IN inside a string literal or a block
580+
# comment can still count as the keyword when a marker follows it.
581+
_IN_MARKER_PROBE = re.compile(r"\bIN\s*\x00([0-9a-f]+):(\d+)\x00", re.IGNORECASE)
582+
583+
584+
def _in_ends_line_comment(text: str, match: re.Match[str]) -> bool:
585+
"""Whether the IN that `match` found looks like it is in a `--` comment, with the marker on a
586+
later line.
587+
588+
This is a check on the text, not a parse. In that case the marker is most likely live code, so
589+
it keeps its ARRAY rendering. A marker on the same line as a commented IN is usually inside the
590+
comment too, so it is not checked. A `--` inside a string literal earlier on the line also
591+
counts, which falls back to the ARRAY rendering.
592+
"""
593+
if not re.search(r"[\r\n]", match.group(0)):
594+
return False
595+
line_start = max(
596+
text.rfind("\n", 0, match.start()), text.rfind("\r", 0, match.start())
597+
)
598+
return "--" in text[line_start + 1 : match.start()]
599+
600+
601+
def _renders_as_array(value: Any) -> bool:
602+
"""Whether ParamEscaper.escape_item renders `value` as an ARRAY literal."""
603+
return isinstance(value, Sequence) and not isinstance(value, str)
604+
605+
606+
def _is_set(value: Any) -> bool:
607+
"""Whether `value` is a set, frozenset or dict key view. A set has no order, so it can only
608+
render as an IN list, never as an ARRAY."""
609+
return isinstance(value, AbstractSet)
610+
611+
612+
def _renders_as_in_list(value: Any) -> bool:
613+
"""Whether `value` renders as an IN list when its marker directly follows IN.
614+
615+
bytes and bytearray are sequences of ints, so escape_item renders them as ARRAY literals of
616+
byte values. They are left out here, so `x IN %s` with bytes still fails loudly as it does on
617+
main, instead of quietly comparing x with the byte values.
618+
"""
619+
if isinstance(value, (bytes, bytearray)):
620+
return False
621+
return _renders_as_array(value) or _is_set(value)
622+
623+
624+
def _in_list_items(value: Any) -> Any:
625+
"""Return the items to render for an IN list, with a set's items sorted.
626+
627+
Sorting usually makes the SQL text the same from run to run, but not always: items without a
628+
consistent order, such as `float("nan")` among other numbers, can still come out in a
629+
different order. The IN list matches the same rows either way.
630+
"""
631+
if not _is_set(value):
632+
return value
633+
try:
634+
return sorted(value)
635+
except TypeError:
636+
# The items do not compare with each other, for example None and an int.
637+
return sorted(value, key=repr)
638+
639+
640+
class _InListOnly:
641+
"""Stands in for an escaped set, which can only be rendered as an IN list.
642+
643+
`render_inline` rejects a set at any other marker before formatting. Raising here as well
644+
keeps an unescaped set from ever reaching the SQL text.
645+
"""
646+
647+
def __init__(self, value: Any) -> None:
648+
self.value = value
649+
650+
def __str__(self) -> str:
651+
raise ProgrammingError("Unsupported object {}".format(self.value))
652+
653+
__repr__ = __str__
654+
655+
656+
class _MarkerProbe:
657+
"""Formats as a token that identifies the marker it was substituted into."""
658+
659+
def __init__(self, nonce: str, index: int) -> None:
660+
self._token = f"\x00{nonce}:{index}\x00"
661+
662+
def __str__(self) -> str:
663+
return self._token
664+
665+
__repr__ = __str__
666+
667+
668+
class _NumberedLookups:
669+
"""Mapping stand-in for %-formatting that numbers its lookups.
670+
671+
%-formatting looks up each %(name)s marker once, left to right, so lookup n is always the n-th
672+
named marker of the operation. `value_for(n, name)` supplies the value substituted there.
673+
"""
674+
675+
def __init__(
676+
self, value_for: Callable[[int, str], Any], escaped: Dict[str, Any]
677+
) -> None:
678+
self._value_for = value_for
679+
self._escaped = escaped
680+
self._count = 0
681+
682+
def __getitem__(self, name: str) -> Any:
683+
index = self._count
684+
self._count += 1
685+
return self._value_for(index, name)
686+
687+
def __str__(self) -> str:
688+
# A bare %s marker formats the whole mapping; keep rendering the escaped dict there.
689+
return str(self._escaped)
690+
691+
__repr__ = __str__
692+
693+
694+
def _reject_sets_outside_in_lists(
695+
parameters: Any, in_list_markers: Set[int], marker_keys: Dict[int, Any]
696+
) -> None:
697+
"""Raise the error `escape_args` raises for a set, unless the set is used and every marker
698+
it fills renders as an IN list."""
699+
items = (
700+
parameters.items() if isinstance(parameters, dict) else enumerate(parameters)
701+
)
702+
for key, value in items:
703+
if not _is_set(value):
704+
continue
705+
markers = {index for index, k in marker_keys.items() if k == key}
706+
if not markers or not markers <= in_list_markers:
707+
raise ProgrammingError("Unsupported object {}".format(value))
708+
709+
710+
def _format_markers(
711+
operation: str, escaped: Any, value_for: Callable[[int, Any], Any]
712+
) -> str:
713+
"""%-format `operation`, taking the value of marker n from `value_for(n, key)`.
714+
715+
`escaped` holds the escaped parameter values. `key` is the marker's name when it is a dict and
716+
the marker's position otherwise.
717+
"""
718+
values: Any
719+
if isinstance(escaped, dict):
720+
values = _NumberedLookups(value_for, escaped)
721+
else:
722+
values = tuple(value_for(i, i) for i in range(len(escaped)))
723+
return inject_parameters(operation, values)
724+
725+
560726
# Taken from PyHive
561727
class ParamEscaper:
562728
_DATE_FORMAT = "%Y-%m-%d"
@@ -571,6 +737,115 @@ def escape_args(self, parameters):
571737
else:
572738
raise ProgrammingError("Unsupported param format: {}".format(parameters))
573739

740+
def render_inline(self, operation: str, parameters) -> str:
741+
"""Render `parameters` into the markers of `operation` as inline SQL literals.
742+
743+
Values are escaped as `escape_args` escapes them, with one exception: a sequence at a
744+
marker that directly follows IN (e.g. `x IN %(ids)s`) renders as the parenthesised list
745+
Spark requires there, `x IN (1,2,3)`, instead of `x IN ARRAY(1,2,3)`, which Spark rejects.
746+
The choice is made per marker, so a parameter used both after IN and elsewhere keeps its
747+
ARRAY rendering at the other markers. Inside the list, tuples render as rows and lists as
748+
ARRAY literals.
749+
750+
IN is found by matching the SQL text, not by parsing it (see `_IN_MARKER_PROBE`), so an
751+
unusual query, such as one with IN inside a string literal just before a marker, can
752+
still get a list where an ARRAY was meant.
753+
754+
A set (or frozenset, or dict key view) is accepted only where every marker it fills
755+
directly follows IN, and its items are sorted. Anywhere else it raises the same
756+
`ProgrammingError` as `escape_args`, because a set has no order to put into an ARRAY.
757+
"""
758+
escaped = self._escape_values(parameters)
759+
values = parameters.values() if isinstance(parameters, dict) else parameters
760+
if not any(_renders_as_in_list(value) for value in values):
761+
# Only a sequence or a set can render as an IN list, so without one render exactly
762+
# as before.
763+
return inject_parameters(operation, escaped)
764+
765+
in_list_markers, marker_keys = self._markers_following_in(
766+
operation, parameters, escaped
767+
)
768+
_reject_sets_outside_in_lists(parameters, in_list_markers, marker_keys)
769+
770+
def render_marker(index: int, key: Any) -> Any:
771+
if index in in_list_markers:
772+
return self.escape_in_list(_in_list_items(parameters[key]))
773+
return escaped[key]
774+
775+
return _format_markers(operation, escaped, render_marker)
776+
777+
def _escape_values(self, parameters):
778+
"""Escape like `escape_args`, except that a set becomes an `_InListOnly` placeholder
779+
instead of raising, so `render_inline` can decide where it is allowed."""
780+
781+
def escape(value: Any) -> Any:
782+
return _InListOnly(value) if _is_set(value) else self.escape_item(value)
783+
784+
if isinstance(parameters, dict):
785+
return {k: escape(v) for k, v in parameters.items()}
786+
elif isinstance(parameters, (list, tuple)):
787+
return tuple(escape(x) for x in parameters)
788+
else:
789+
raise ProgrammingError("Unsupported param format: {}".format(parameters))
790+
791+
def _markers_following_in(
792+
self, operation: str, parameters, escaped
793+
) -> Tuple[Set[int], Dict[int, Any]]:
794+
"""Return the numbers of the markers that directly follow IN and receive a value that can
795+
render as an IN list, and the parameter key that each marker number refers to.
796+
797+
Markers are numbered as `_format_markers` numbers them. To find them, the operation is
798+
formatted once through the same %-formatting as the real substitution, so `%%` and
799+
positional markers line up exactly. Each value that can render as an IN list (see
800+
`_renders_as_in_list`) is replaced by a probe token. Numbers keep their value, so
801+
markers such as `%d` still format, and escaped strings are replaced by an empty string
802+
literal, so the text of a value cannot change the result. If the operation does not
803+
format, both results are empty.
804+
"""
805+
# The nonce stops text that only looks like a probe token from being taken for one.
806+
nonce = secrets.token_hex(8)
807+
marker_keys: Dict[int, Any] = {}
808+
809+
def probe_marker(index: int, key: Any) -> Any:
810+
# A missing name raises KeyError here, as it does when rendering.
811+
escaped_value = escaped[key]
812+
marker_keys[index] = key
813+
value = parameters[key]
814+
if _renders_as_in_list(value):
815+
return _MarkerProbe(nonce, index)
816+
# Replace text with '' so the content of a value cannot change the result. A string
817+
# of two or more characters formats under exactly the markers '' does. A one-character
818+
# string can only be a single-digit Decimal, which %c accepts, so it is kept.
819+
if isinstance(escaped_value, str) and len(escaped_value) > 1:
820+
return "''"
821+
return escaped_value
822+
823+
try:
824+
probed_operation = _format_markers(operation, escaped, probe_marker)
825+
except (KeyError, TypeError, ValueError):
826+
# Rendering raises the same error: a probe only replaces a sequence, and a marker that
827+
# cannot format a probe (such as %d) cannot format the sequence's ARRAY string either.
828+
# A set gets the ProgrammingError escape_args raises, since none of its markers count.
829+
return set(), {}
830+
831+
in_list_markers = {
832+
int(match.group(2))
833+
for match in _IN_MARKER_PROBE.finditer(probed_operation)
834+
if match.group(1) == nonce
835+
and not _in_ends_line_comment(probed_operation, match)
836+
}
837+
return in_list_markers, marker_keys
838+
839+
def escape_in_list(self, item):
840+
return "(" + ",".join(map(str, map(self._escape_in_list_element, item))) + ")"
841+
842+
def _escape_in_list_element(self, item):
843+
# Inside an IN list a tuple is a row, as in `(a, b) IN ((1, 2), (3, 4))`. A list stays an
844+
# ARRAY literal so ARRAY columns can still be matched, as in `arr IN (ARRAY(1, 2))`.
845+
if isinstance(item, tuple):
846+
return self.escape_in_list(item)
847+
return self.escape_item(item)
848+
574849
def escape_number(self, item):
575850
return item
576851

‎tests/e2e/test_driver.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -559,6 +559,8 @@ def test_get_tables(self):
559559

560560
def test_get_columns(self):
561561
with self.cursor({}) as cursor:
562+
cursor.execute("SELECT current_catalog()")
563+
catalog_name = cursor.fetchone()[0]
562564
table_name = "table_{uuid}".format(uuid=str(uuid4()).replace("-", "_"))
563565
table_names = [table_name + "_1", table_name + "_2"]
564566

@@ -573,7 +575,11 @@ def test_get_columns(self):
573575
"array('item1', 'item2', 'item3') as col_5)".format(table)
574576
)
575577

576-
cursor.columns(schema_name="defa%", table_name=table_name + "%")
578+
cursor.columns(
579+
catalog_name=catalog_name,
580+
schema_name="defa%",
581+
table_name=table_name + "%",
582+
)
577583
cols = cursor.fetchall()
578584
cols_desc = cursor.description
579585

0 commit comments

Comments
 (0)