Skip to content

Add Create Multipart Data keyword for streamed multipart/form-data uploads - #415

Open
florianmorineau wants to merge 7 commits into
MarketSquare:masterfrom
florianmorineau:multipart-form-data
Open

florianmorineau wants to merge 7 commits into
MarketSquare:masterfrom
florianmorineau:multipart-form-data

Conversation

@florianmorineau

@florianmorineau florianmorineau commented Oct 25, 2026 •

Copy link
Copy Markdown

Closes #409
Fixes #408

Problem

Sending a multipart/form-data body with a large file and form fields is not possible out of the box today:

  • files= reads every file into memory and offers no simple way to set form fields, repeated field names, filenames or content types from Robot Framework.
  • Passing a requests_toolbelt.MultipartEncoder as data= does upload, but the keyword then fails while logging the request: format_data_to_log_string calls len() and slices the body, which raises TypeError: object of type 'MultipartEncoder' has no len() (or ... is not subscriptable once __len__ is added, as reported in Issue with Multipart Form-Data in Robot Framework (Large File + Form Fields) #408). The user also has to build the Content-Type header with the boundary by hand.

Changes

The commits can be reviewed one by one:

  1. fix: log streaming request bodies (Issue with Multipart Form-Data in Robot Framework (Large File + Form Fields) #408): format_data_to_log_string logs repr() for any body that is not str/bytes (generators, multipart encoders...), like it already does for file descriptors. No new dependency.
  2. feat: new keyword Create Multipart Data fields=None files=None files_first=False (add New Keyword managment of Multipart form data #409), returning a streamed body to pass as data= to any request keyword (POST, PUT, PATCH, on session or session-less):
    • fields: dictionary or list of (name, value) pairs (repeated names kept), dictionary/list values are sent as JSON strings, other values as strings.
    • files: dictionary name=path, or list of (name, path[, content_type[, filename]]) or of dictionaries {name, path, content_type?, filename?}; a list of paths can be given in place of path to send several files with the same name. Filename defaults to the base name and content type is guessed with mimetypes.
    • Parts are sent in the given order, fields first; files_first=${True} sends the files first for servers that expect them before the fields.
    • When data is a multipart encoder, the Content-Type header (with boundary) is set automatically unless explicitly passed in the request headers, and the files it holds are closed after the request, like Get File For Streaming Upload. Session headers such as Authorization: Bearer ... are kept, and a session default Content-Type: application/json doesn't break the upload.
    • requests-toolbelt is an optional dependency: pip install "robotframework-requests[multipart]" (also added to the test extra). A clear ImportError explains how to install it.
  3. ci: run the Python 3.8 job on ubuntu-24.04: ubuntu-latest is now Ubuntu 26.04, which has no Python 3.8 for actions/setup-python, so the job failed at "Set up Python" and cancelled the matrix. Independent from the feature, feel free to drop it if you prefer to handle the CI separately.
  4. test: compare an uploaded file with its raw bytes, so the acceptance test also passes on Windows checkouts with CRLF line endings.
  5. doc: keyword documentation in sections (form fields, files formats table, order of the parts, sending the body, examples) and a README section; doc/RequestsLibrary.html regenerated with doc/generate_doc.sh (Robot Framework 6.0.2 as the current file).
  6. fix: resend the whole body on 307/308 redirects and retries. The body can be rewound to its start (tell() / seek(0)), which urllib3 uses on retries (e.g. a PUT after a connection reset, retried by default by Create Session), and it is iterable, which requests needs to record the body position and rewind it before following a redirect. Without it, the redirected request announced the same Content-Length with an empty body.
  7. test: acceptance tests for a 307 redirect, files described by dictionaries and the error messages for invalid arguments.
${headers}=      Create Dictionary    Authorization=Bearer ${token}
Create Session   api    https://example.com    headers=${headers}
${settings}=     Create Dictionary    everything=${True}
${fields}=       Create Dictionary    action=add    settings=${settings}
${files}=        Create Dictionary    document=${CURDIR}/report.pdf
${multipart}=    Create Multipart Data    fields=${fields}    files=${files}
${resp}=         POST On Session    api    /upload    data=${multipart}

Tests

  • Unit tests: utests/test_multipart.py (fields/files formats, list of paths, duplicates, part order, content type guessing, invalid input, files closed on error, missing dependency, automatic/explicit Content-Type, rewind and stream contracts used by urllib3 and requests, and a PUT On Session retried after a connection reset in the middle of the body that must resend the whole body) and log tests for streaming bodies.
  • Acceptance tests in atests/test_post_multipart.robot: POST/PUT on session with fields + files, repeated fields and files, files first with a list of paths, files described by dictionaries, a 50 KB file with fields (the Issue with Multipart Form-Data in Robot Framework (Large File + Form Fields) #408 case), a 307 redirect resending the whole body, explicit Content-Type kept, session Authorization header kept, clear errors for invalid arguments.
  • 100% line and branch coverage of the new and changed code; the acceptance tests alone cover 94% of the new module.
  • Python 3.8 and 3.14: pytest utests 95 passed, robot atests 205 passed, flake8 OK. The log fix commit alone (without requests-toolbelt): 68 / 196 passed.
  • Also checked outside of this PR:
    • against three real multipart servers protected by a Bearer token (FastAPI / python-multipart, Express / busboy and multer, ASP.NET Core 8 with IFormFile and MultipartReader): same parts received as with the custom libraries this keyword replaces in our projects (names, filenames, content types, content hashes, order), accented names included;
    • raw bytes on the wire validated against RFC 7578 / RFC 2046 and byte identical to those libraries;
    • 100 MB and 1 GB uploads: received intact by the three servers with under 1 MB of extra client memory, where files= and the custom library need the whole file size in memory;
    • every example of the keyword documentation run as written, with the part headers captured on the wire;
    • each of 10 injected defects (including a non rewindable and a non iterable body) is caught by the tests of this PR.

🤖 Generated with Claude Code

florianmorineau and others added 2 commits October 25, 2026 12:22
Request logging called len() and sliced the body to truncate it, which
raised a TypeError with streaming bodies such as a requests_toolbelt
MultipartEncoder ("has no len()" or "is not subscriptable" for bodies
over the log limit), failing the keyword after the request was sent.
Any body that is not str or bytes is now logged with repr(), like file
descriptors already were.

Fixes MarketSquare#408

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ata uploads

- New `Create Multipart Data` keyword builds a streamed multipart body from
  form fields and files (repeated names, list of paths for one name, JSON
  values, filename and content type override or guess, optional files
  first order), using the optional requests-toolbelt dependency
  (`pip install robotframework-requests[multipart]`)
- The body can be rewound to its start, so session retries (PUT after a
  connection reset) and redirects resend it whole
- Requests keywords set the multipart Content-Type with its boundary
  automatically, keeping session headers, and close the files held by the
  encoder once sent
- Keyword documentation regenerated

Closes MarketSquare#409

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.01%. Comparing base (3f83f5d) to head (c465654).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #415      +/-   ##
==========================================
+ Coverage   85.38%   91.01%   +5.63%     
==========================================
  Files           9       10       +1     
  Lines         479      590     +111     
  Branches       49       74      +25     
==========================================
+ Hits          409      537     +128     
+ Misses         65       51      -14     
+ Partials        5        2       -3     
Flag Coverage Δ
acceptance 86.61% <91.96%> (+1.22%) ⬆️
unit 68.47% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

florianmorineau and others added 5 commits October 25, 2026 12:54
ubuntu-latest is now Ubuntu 26.04, for which actions/setup-python has no
Python 3.8 build, so the job failed at "Set up Python" and cancelled the
rest of the matrix. Python 3.8 builds are still available for 24.04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows, git checks out atests/data.json with CRLF line endings and
`Get File` normalizes them to LF, while the server echoes the bytes it
received, so the comparison failed on an invisible \r. The upload is
now compared with the file bytes, which is what the keyword sends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Structure the keyword documentation in sections (form fields, files,
order of the parts, sending the body, examples), describe each files
format in a table with copyable examples, and expand the README section
with the formats and an example on a Bearer token session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
requests only records the position of a body to rewind it on redirects
when the body is iterable, which MultipartEncoder is not: after a 307
the second request announced the same Content-Length with an empty body
and the server waited for it. The encoder is now iterable, so requests
prepares it as a stream and rewinds it before following the redirect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in atests

Acceptance tests for a 307 redirect that must resend the whole body,
files described by dictionaries with a non string field value, and the
error messages for invalid fields, invalid files and a missing file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add New Keyword managment of Multipart form data Issue with Multipart Form-Data in Robot Framework (Large File + Form Fields)

1 participant