Repository navigation
Add Create Multipart Data keyword for streamed multipart/form-data uploads - #415
Open
florianmorineau wants to merge 7 commits into
Open
florianmorineau wants to merge 7 commits into
florianmorineau wants to merge 7 commits into
Conversation
Request logging called len() and sliced the body to truncate it, which
raised a TypeError with streaming bodies such as a requests_toolbelt
MultipartEncoder ("has no len()" or "is not subscriptable" for bodies
over the log limit), failing the keyword after the request was sent.
Any body that is not str or bytes is now logged with repr(), like file
descriptors already were.
Fixes MarketSquare#408
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ata uploads - New `Create Multipart Data` keyword builds a streamed multipart body from form fields and files (repeated names, list of paths for one name, JSON values, filename and content type override or guess, optional files first order), using the optional requests-toolbelt dependency (`pip install robotframework-requests[multipart]`) - The body can be rewound to its start, so session retries (PUT after a connection reset) and redirects resend it whole - Requests keywords set the multipart Content-Type with its boundary automatically, keeping session headers, and close the files held by the encoder once sent - Keyword documentation regenerated Closes MarketSquare#409 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #415 +/- ##
==========================================
+ Coverage 85.38% 91.01% +5.63%
==========================================
Files 9 10 +1
Lines 479 590 +111
Branches 49 74 +25
==========================================
+ Hits 409 537 +128
+ Misses 65 51 -14
+ Partials 5 2 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ubuntu-latest is now Ubuntu 26.04, for which actions/setup-python has no Python 3.8 build, so the job failed at "Set up Python" and cancelled the rest of the matrix. Python 3.8 builds are still available for 24.04. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows, git checks out atests/data.json with CRLF line endings and `Get File` normalizes them to LF, while the server echoes the bytes it received, so the comparison failed on an invisible \r. The upload is now compared with the file bytes, which is what the keyword sends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Structure the keyword documentation in sections (form fields, files, order of the parts, sending the body, examples), describe each files format in a table with copyable examples, and expand the README section with the formats and an example on a Bearer token session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
requests only records the position of a body to rewind it on redirects when the body is iterable, which MultipartEncoder is not: after a 307 the second request announced the same Content-Length with an empty body and the server waited for it. The encoder is now iterable, so requests prepares it as a stream and rewinds it before following the redirect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in atests Acceptance tests for a 307 redirect that must resend the whole body, files described by dictionaries with a non string field value, and the error messages for invalid fields, invalid files and a missing file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #409
Fixes #408
Problem
Sending a
multipart/form-databody with a large file and form fields is not possible out of the box today:files=reads every file into memory and offers no simple way to set form fields, repeated field names, filenames or content types from Robot Framework.requests_toolbelt.MultipartEncoderasdata=does upload, but the keyword then fails while logging the request:format_data_to_log_stringcallslen()and slices the body, which raisesTypeError: object of type 'MultipartEncoder' has no len()(or... is not subscriptableonce__len__is added, as reported in Issue with Multipart Form-Data in Robot Framework (Large File + Form Fields) #408). The user also has to build theContent-Typeheader with the boundary by hand.Changes
The commits can be reviewed one by one:
fix:log streaming request bodies (Issue with Multipart Form-Data in Robot Framework (Large File + Form Fields) #408):format_data_to_log_stringlogsrepr()for any body that is notstr/bytes(generators, multipart encoders...), like it already does for file descriptors. No new dependency.feat:new keywordCreate Multipart Datafields=None files=None files_first=False(add New Keyword managment of Multipart form data #409), returning a streamed body to pass asdata=to any request keyword (POST,PUT,PATCH, on session or session-less):fields: dictionary or list of(name, value)pairs (repeated names kept), dictionary/list values are sent as JSON strings, other values as strings.files: dictionaryname=path, or list of(name, path[, content_type[, filename]])or of dictionaries{name, path, content_type?, filename?}; a list of paths can be given in place ofpathto send several files with the same name. Filename defaults to the base name and content type is guessed withmimetypes.files_first=${True}sends the files first for servers that expect them before the fields.datais a multipart encoder, theContent-Typeheader (with boundary) is set automatically unless explicitly passed in the requestheaders, and the files it holds are closed after the request, likeGet File For Streaming Upload. Session headers such asAuthorization: Bearer ...are kept, and a session defaultContent-Type: application/jsondoesn't break the upload.requests-toolbeltis an optional dependency:pip install "robotframework-requests[multipart]"(also added to thetestextra). A clearImportErrorexplains how to install it.ci:run the Python 3.8 job onubuntu-24.04:ubuntu-latestis now Ubuntu 26.04, which has no Python 3.8 foractions/setup-python, so the job failed at "Set up Python" and cancelled the matrix. Independent from the feature, feel free to drop it if you prefer to handle the CI separately.test:compare an uploaded file with its raw bytes, so the acceptance test also passes on Windows checkouts with CRLF line endings.doc:keyword documentation in sections (form fields, files formats table, order of the parts, sending the body, examples) and a README section;doc/RequestsLibrary.htmlregenerated withdoc/generate_doc.sh(Robot Framework 6.0.2 as the current file).fix:resend the whole body on 307/308 redirects and retries. The body can be rewound to its start (tell()/seek(0)), which urllib3 uses on retries (e.g. aPUTafter a connection reset, retried by default byCreate Session), and it is iterable, which requests needs to record the body position and rewind it before following a redirect. Without it, the redirected request announced the sameContent-Lengthwith an empty body.test:acceptance tests for a 307 redirect, files described by dictionaries and the error messages for invalid arguments.Tests
utests/test_multipart.py(fields/files formats, list of paths, duplicates, part order, content type guessing, invalid input, files closed on error, missing dependency, automatic/explicit Content-Type, rewind and stream contracts used by urllib3 and requests, and aPUT On Sessionretried after a connection reset in the middle of the body that must resend the whole body) and log tests for streaming bodies.atests/test_post_multipart.robot: POST/PUT on session with fields + files, repeated fields and files, files first with a list of paths, files described by dictionaries, a 50 KB file with fields (the Issue with Multipart Form-Data in Robot Framework (Large File + Form Fields) #408 case), a 307 redirect resending the whole body, explicit Content-Type kept, sessionAuthorizationheader kept, clear errors for invalid arguments.pytest utests95 passed,robot atests205 passed, flake8 OK. The log fix commit alone (without requests-toolbelt): 68 / 196 passed.IFormFileandMultipartReader): same parts received as with the custom libraries this keyword replaces in our projects (names, filenames, content types, content hashes, order), accented names included;files=and the custom library need the whole file size in memory;🤖 Generated with Claude Code