Fil is a pluggable file storage abstraction for Elixir.

Adapters: Local, Memory, S3, AzureBlob, Null.

Mix.install([
  {:fil, "~> 0.2"}
])

disk = Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")

disk
|> Fil.write!("hello.txt", "World")
|> Fil.read!()
#=> "World"

With adapter: Fil.Adapter.S3, a bucket and credentials, the same code writes to S3.

To try Fil without a project, run the tour in Livebook.

Fil is in early development, and the API may still change a lot.

Features and goals

• One API for many kinds of storage. The same behaviour on every adapter, and cp and rename work across disks.
• Just values. A disk is a plain value: no application config, no registry, nothing to supervise. It works in a script or a Livebook with Mix.install/1.
• Pluggable. Anything that isn't about where files are stored is a plugin, such as setting content types or logging.
• Few dependencies. Req, NimbleOptions, MIME and Telemetry, plus Plug if you serve files. Cloud adapters use Req instead of their own SDKs.
• Streaming. Files can be read and written as streams on every disk, so large files don't have to fit in memory.
• Integrations. Each one is compiled only when its dependency is in your project:
  • Plug: Fil.Plug serves public and signed URLs from your endpoint or router, and sends files from controllers.
  • Phoenix LiveView: Fil.LiveView stores uploads on any disk, also straight from the browser, and has an upload field for forms (see the Phoenix guide).
  • Ecto: Fil.Ecto.Ref stores refs in Ecto schemas.
  • Livebook: Fil.Kino browses a disk, uploads files to it, and adds a smart cell that builds one.
• URLs on every disk. Public and signed GET and PUT URLs, from S3 and Azure themselves or from Fil.Plugin.URL and Fil.Plug.
• Safe by default. Paths can't climb out of the disk root, if_exists: :error never replaces a file, and S3 verifies checksums.
• Errors you can act on. Each error, such as Fil.NotFoundError or Fil.UnavailableError, says what to do next and is the same on every adapter.
• Telemetry. Every operation emits Telemetry events with its duration, bytes and error, for metrics, logs and traces (see Fil.Telemetry).

Concepts

Fil uses the function names of Elixir's File module (read, write, stream, stat, ls, cp, rename, rm, rm_rf), but on every adapter they behave like an object store:

• write creates missing parent directories
• rm on a missing file succeeds
• ls on a missing directory returns an empty list
• paths are always relative to the disk root, and . is the root
• a path that climbs above the root fails with a Fil.InvalidRequestError

The contract in Fil.Adapter lists every difference from File. Every function that can fail returns {:ok, result} or {:error, error}, with an error from the Errors section below.

Disks

A disk says where files are stored: an adapter and its options. It's a plain value, so there's no application config and nothing to add to your supervision tree (Fil starts one process of its own, for the Memory adapter). Every function takes a disk as its first argument and a path relative to the disk's root:

disk = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")

{:ok, _} = Fil.write(disk, "hello.txt", "World")

Fil.read(disk, "hello.txt")
#=> {:ok, "World"}

Refs

A Fil.Ref is a single value for a file on a disk, built with Fil.ref/2. Every function that takes a disk and a path as two arguments also takes a ref as one argument in their place:

hello = Fil.ref(disk, "hello.txt")
#=> #Fil.Ref<local:hello.txt>

{:ok, _} = Fil.write(hello, "World")

Fil.read(hello)
#=> {:ok, "World"}

Actions on files return the ref they acted on, which keeps cross-disk code short:

with {:ok, report} <- Fil.write(s3, "reports/q3.pdf", pdf),
     {:ok, _backup} <- Fil.cp(report, Fil.ref(local, "backups/q3.pdf")) do
  {:ok, report}
end

The bang variants return the ref itself instead of {:ok, ref}, so you can pipe one call into the next:

disk
|> Fil.write!("hello.txt", "World")
|> Fil.cp!("backup/hello.txt")
|> Fil.read!()
#=> "World"

Plugins

Plugins attach to a disk and see every operation on it. Fil ships one that sets the content type from the file extension, so S3 serves reports/q3.pdf as application/pdf:

s3 =
  Fil.disk(adapter: Fil.Adapter.S3, bucket: "my-bucket", region: "eu-central-1")
  |> Fil.Plugin.ContentType.attach()

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)

For a disk built from config, Fil.disk/1 takes plugins as data: plugins: [{Fil.Plugin.ContentType, :call, []}].

Your own plugin is a function. It gets the operation, calls next to run the rest, and returns the result:

local =
  Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")
  |> Fil.attach(:log, fn op, next, _opts ->
    IO.puts("#{op.name} #{op.path}")
    next.(op)
  end)

The plugins guide explains how to write plugins: matching on operations, changing content, handling errors and answering without the adapter.

Errors

Every error is an exception struct that says what you can do about it, and means the same on every disk:

case Fil.read(disk, "report.txt") do
  {:ok, content} -> content
  {:error, %Fil.NotFoundError{}} -> nil
  {:error, %Fil.UnavailableError{}} -> :retry_later
end

The structs contain the operation, the path, the disk and what the storage reported (:reason), so a log line says which file failed and why. Each error has its own page under Errors in the docs, and Errors in Fil.Adapter explains how adapters use them.

Every function that can fail has a bang variant that returns the bare result and raises the same struct instead.

Usage

The installation guide is the full setup for an application: a module for your disks, the config for each environment (local disks in development, memory disks in tests and S3 in production), signed URLs and tests. This section is only a quick tour of the API.

Build one disk per kind of storage:

local = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")

s3 =
  Fil.disk(
    adapter: Fil.Adapter.S3,
    bucket: "my-bucket",
    region: "eu-central-1",
    access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
    secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
  )

Every operation works the same on every disk:

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)
{:ok, pdf} = Fil.read(report)
{:ok, stat} = Fil.stat(report)
{:ok, reports} = Fil.ls(s3, "reports/", recursive: true)
{:ok, backup} = Fil.cp(report, Fil.ref(local, "backups/q3.pdf"))
{:ok, _} = Fil.rm(report)

Large files don't have to fit in memory. stream returns a file's content as a stream of chunks, and write takes a stream as well as a binary, so a file can go from one disk to another piece by piece. {:file, path} streams a local file:

{:ok, video} = Fil.stream(s3, "videos/intro.mp4")
{:ok, _} = Fil.write(local, "cache/intro.mp4", video)

{:ok, _} = Fil.write(s3, "backups/db.dump", {:file, "db.dump"})

With a local file, a File.stream! of bytes or a stream from stream, Fil knows the size, and S3 sends the stream as it's read, in one request. For other streams, pass size: if you know it, or S3 uploads the stream in parts of 8 MiB (the local and memory disks don't need the size).

offset: and length: read part of a file, and the storage reads only that part (S3 with a Range header):

{:ok, header} = Fil.read(s3, "videos/intro.mp4", length: 4096)
{:ok, rest} = Fil.stream(s3, "videos/intro.mp4", offset: 4096)

url returns the public URL of a file, signed_url an expiring one, and signed_upload an expiring URL with the headers to send, so clients can download or upload a file directly instead of going through your application code. S3 and Azure serve their URLs themselves. For local and in-memory disks, Fil.Plugin.URL builds them and Fil.Plug serves them from your application, with etags, 304s and range requests, so browsers can cache files and seek in videos:

{:ok, logo_url} = Fil.url(s3, "logo.png")
{:ok, url} = Fil.signed_url(report, expires_in: 900)
{:ok, %{url: upload_url, headers: headers}} = Fil.signed_upload(s3, "inbox/new.bin")

With checksum:, a write sends a checksum of the content. S3 rejects the upload if what it received doesn't match, and stores the checksum with the object, so later reads can check it:

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf, checksum: :sha256)
{:ok, pdf} = Fil.read(report, verify_checksum: true)
{:ok, %Fil.Stat{checksum: {:sha256, checksum}}} = Fil.stat(report, checksum: :sha256)

To create a file only if it doesn't exist yet, pass if_exists: :error. If the file is already there, nothing is overwritten and the write returns a Fil.AlreadyExistsError. That makes a simple lock: whoever creates the file first runs the job.

case Fil.write(s3, "jobs/today.lock", "started", if_exists: :error) do
  {:ok, _lock} -> :run_the_job
  {:error, %Fil.AlreadyExistsError{}} -> :someone_else_won
end

tmp returns a ref to a file in a temporary directory, which is removed when the calling process exits. It's a local file, so a copy from any disk gives a tool the path it needs:

report = Fil.tmp("report.pdf")
{:ok, _} = Fil.cp(Fil.ref(s3, "reports/q3.pdf"), report)
{text, 0} = System.cmd("pdftotext", [Fil.Tmp.path(report), "-"])

Development

mix test runs the unit tests, which need no network. The integration tests run the conformance suite against RustFS (S3) and Azurite (Azure Blob Storage), started with Docker Compose:

docker compose up -d
mix test.integration

FIL_S3_ENDPOINT, FIL_S3_ACCESS_KEY_ID, FIL_S3_SECRET_ACCESS_KEY and FIL_S3_REGION point the integration tests at another S3 endpoint, and FIL_AZURE_BLOB_ENDPOINT, FIL_AZURE_BLOB_ACCOUNT and FIL_AZURE_BLOB_ACCOUNT_KEY at another Azure account.

Acknowledgments

Fil builds on ideas from these projects:

• Flysystem (PHP): the scope, one API across many kinds of storage
• Req: the API design, with disks as plain values and plugins that attach to them
• Plug: plugins as small units you add to a disk, each doing one thing to every operation that passes through

Thanks to their authors and contributors.