The Ruby 4.0 CI job measures library line coverage with SimpleCov and submits coverage/.resultset.json to Coveralls using GitHub's automatic token. Other matrix jobs do not submit duplicate reports. Run COVERAGE=true bundle install followed by COVERAGE=true bundle exec rake spec for a local report. The default-branch badge requires Coveralls to accept an upstream master build; configuring this workflow does not activate the external service.
Typical as_json definitions may involve lots of database point queries and method calls. When returning collections of objects, a single call may yield hundreds of database queries that can take seconds. This library mitigates the problem by implementing a module called CachedJson.
CachedJson enables returning multiple JSON formats and versions from a single class and provides some rules for yielding embedded or referenced data. It then uses a scheme where fragments of JSON are cached for a particular (class, id) pair containing only the data that doesn't involve references/embedded documents. To get the full JSON for an instance, CachedJson will combine fragments of JSON from the instance with fragments representing the JSON for its references. In the best case, when all of these fragments are cached, this falls through to a few cache lookups followed by a couple Ruby hash merges to create the JSON.
Using Mongoid::CachedJson we were able to cut our JSON API average response time by about a factor of 10.
Add mongoid-cached-json to your Gemfile.
gem 'mongoid-cached-json'
Include Mongoid::CachedJson in your models.
class Gadget
include Mongoid::CachedJson
field :name
field :extras
belongs_to :widget
json_fields \
name: {},
extras: { properties: :public }
end
class Widget
include Mongoid::CachedJson
field :name
has_many :gadgets
json_fields \
name: {},
gadgets: { type: :reference, properties: :public }
endInvoke as_json.
widget = Widget.first
# the `:short` version of the JSON, `gadgets` not included
widget.as_json
# equivalent to the above
widget.as_json(properties: :short)
# `:public` version of the JSON, `gadgets` returned with `:short` JSON, no `:extras`
widget.as_json(properties: :public)
# `:all` version of the JSON, `gadgets` returned with `:all` JSON, including `:extras`
widget.as_json(properties: :all)By default Mongoid::CachedJson will use an instance of ActiveSupport::Cache::MemoryStore in a non-Rails and Rails.cache in a Rails environment. You can configure it to use any other cache store.
Mongoid::CachedJson.configure do |config|
config.cache = ActiveSupport::Cache::FileStore.new
endThe default JSON version returned from as_json is :unspecified. If you wish to redefine this, set Mongoid::CachedJson.config.default_version.
Mongoid::CachedJson.configure do |config|
config.default_version = :v2
endMongoid::CachedJson supports the following options:
:hide_as_child_json_whenis an optional function that omits the child key fromas_jsonparent objects when it returns true, eg.cached_json hide_as_child_json_when: lambda { |instance| ! instance.secret? }. A missing relationship remains anilvalue.
Mongoid::CachedJson field definitions support the following options:
:definitioncan be a symbol or an anonymous function, eg.description: { definition: :name }ordescription: { definition: lambda { |instance| instance.name } }:typecan be:reference, required for referenced objects:propertiescan be one of:short,:public,:all, in this order:versioncan be a single version for this field to appear in:versionscan be an array of versions for this field to appear in:cachedefaults totrue; set it tofalseto recompute a field on every serialization (see below):reference_propertiescan be one of:short,:public,:all, default will select the reference properties format dynamically (see below)
Use cache: false for a field whose value should be evaluated on every as_json call, even when the rest of the object's JSON comes from the cache. The field is excluded from the cached fragment and merged into the response after that fragment is fetched. Its definition, property/version filters, and configured transformations still apply.
class Product
include Mongoid::Document
include Mongoid::CachedJson
field :name
field :stock, type: Integer
json_fields \
name: {},
stock: { cache: false }
end
product = Product.create!(name: 'Widget', stock: 10)
product.as_json # => { name: 'Widget', stock: 10 }
# Unsaved changes illustrate which fields are recomputed on a cache hit.
product.name = 'Renamed widget'
product.stock = 9
product.as_json # => { name: 'Widget', stock: 9 }The cached name stays unchanged until its cache entry expires or is invalidated (for example, by saving the product); stock is read from the current model instance each time. This option does not reload the model from MongoDB: reload a stale instance yourself if another process has changed its stored values.
For a type: :reference field, cache: false bypasses reads and writes of the referenced object's own JSON fragment through that relationship. The parent's other fields remain cached:
class Order
include Mongoid::Document
include Mongoid::CachedJson
field :number
belongs_to :product
json_fields \
number: {},
product: { type: :reference, cache: false }
end
order = Order.create!(number: '123', product: product)
order.as_json # => { number: '123', product: { name: 'Renamed widget', stock: 9 } }The product reference is recomputed even if the product has a cached JSON fragment from another serialization. This does not disable caching on the Product model itself: product.as_json and other relationships to that product still use their normal cache policies. References nested inside the product also retain their own cache policies; the bypass is not recursive.
Use json_fields :all, display_name: { definition: :display_name } to expand every Mongoid field declared so far (including _id) into an ordinary JSON field, then merge explicit definitions. An explicit definition overrides the generated definition for the same symbol key; for example, json_fields :all, secret: { properties: :all } restricts that field to the :all representation.
Declare Mongoid fields before json_fields :all. Later declarations, dynamic attributes, and associations are not added automatically; declare references explicitly with type: :reference. This is opt-in and may expose sensitive stored data or foreign keys, so prefer an explicit allowlist for public APIs. Runtime as_json(methods: ...) selection remains unsupported: declare computed methods in json_fields.
When calling as_json on a model that contains references to other models the value of the :properties option passed into the as_json call will be chosen as follows:
- Use the value of the
:reference_propertiesoption, if specified. - For
:shortJSON, use:short. - For
:publicJSON, use:public. - For
:allJSON, use:all.
The dynamic selection where :public generates :short references allows to return smaller embedded collections, while :all allows to fetch deep data. Another way of looking at this is to say that a field in a :short JSON appears in collections, a field declared in the :public JSON appears for all users and the field declared in the :all JSON appears for object owners only.
To override this behavior and always return the :short JSON for a child reference, use :reference_properties. In the following example we would want Person.as_json(properties: :all) to return the social security number for that person, but not for all their friends.
class Person
include Mongoid::Document
include Mongoid::CachedJson
field :name
field :ssn
has_and_belongs_to_many :friends, class_name: 'Person'
json_fields \
name: {},
ssn: { properties: :all },
friends: { properties: :public, reference_properties: :short }
endYou can set an optional version or versions attribute on JSON fields. Consider the following definition where the first version defined :name, then split it into :first, :middle and :last in version :v2 and introduced a date of birth in :v3.
class Person
include Mongoid::Document
include Mongoid::CachedJson
field :first
field :last
def name
[ first, middle, last ].compact.join(' ')
end
json_fields \
first: { versions: [ :v2, :v3 ] },
last: { versions: [ :v2, :v3 ] },
middle: { versions: [ :v2, :v3 ] },
born: { versions: :v3 },
name: { definition: :name }
endperson = Person.create(first: 'John', middle: 'F.', last: 'Kennedy', born: 'May 29, 1917')
person.as_json # { name: 'John F. Kennedy' }
person.as_json(version: :v2) # { first: 'John', middle: 'F.', last: 'Kennedy', name: 'John F. Kennedy' }
person.as_json(version: :v3) # { first: 'John', middle: 'F.', last: 'Kennedy', name: 'John F. Kennedy', born: 'May 29, 1917' }Declare json_fields cache_version: 2, name: {} to add a model-specific schema namespace to its cache keys. Increment it when changing fields, computed values, or transformations in a deployment. This is independent of the API version: passed to as_json; callers need not change. Omitting cache_version preserves existing keys.
New code stops reading old fragments, including referenced fragments of that model. Old entries are not deleted and remain until eviction or a cache-store TTL. The gem does not set a default TTL; configure expires_in on your cache store if needed. Updates expire only the current namespace. For rolling deployments or rollbacks, do not reuse a namespace containing stale values: coordinate invalidation or choose another version.
You can define global transformations on all JSON values with Mongoid::CachedJson.config.transform. Each transformation must return a value. In the following example we extend the JSON definition with an application-specific :trusted field and encode any content that is not trusted.
class Widget
include Mongoid::Document
include Mongoid::CachedJson
field :name
field :description
json_fields \
name: { trusted: true },
description: {}
end Mongoid::CachedJson.config.transform do |field, definition, value|
trusted = !!definition[:trusted]
trusted ? value : CGI.escapeHTML(value)
endTaking part in the Mongoid::CachedJson json_fields scheme is optional: you can still write as_json methods where it makes sense.
You can set Mongoid::CachedJson.config.disable_caching = true. It may be a good idea to set it to ENV['DISABLE_JSON_CACHING'], in case this turns out not to be The Solution To All Of Your Performance Problems (TM).
To bypass shared-cache reads and writes for one model while retaining its JSON fields, versions, and property hierarchy, declare json_fields disable_caching: true, name: {}. This applies when the model is serialized directly, in collections, or as a reference. Other models, including its parent and referenced children, keep their own caching policy. Changes to an uncached child are visible even when its parent's scalar fields are cached.
Normal model updates and destroys expire that document's JSON fragments through Mongoid callbacks. Bulk or raw collection updates that bypass model callbacks cannot invalidate the cache automatically. Explicitly expire each affected document after such a write:
User.where(id: user_id).update_all(status: 'active')
User.find(user_id).expire_cached_jsonConcurrent writes have a separate boundary: an older in-memory instance can repopulate a fragment after another instance has saved newer values and expired the cache. Reload stale instances before serializing them, and expire their fragments if a stale writer may already have repopulated the cache:
user.reload
user.expire_cached_json
user.as_jsonReloading and expiring are not an atomic cross-process operation. This gem does not provide a distributed revision or compare-and-set protocol for cache stores. Applications requiring strict consistency across concurrent writers should coordinate persistence and cache invalidation themselves or disable caching for the affected data.
This library overrides as_json, hence testing JSON results can be done at model level.
describe 'as_json' do
before :each do
@person = Person.create!(first: 'John', last: 'Doe')
end
it 'returns name' do
expect(@person.as_json(properties: :public)[:name]).to eql 'John Doe'
end
endIt's also common to test the results of the API using the Pathy library.
describe 'as_json' do
before :each do
person = Person.create!(first: 'John', last: 'Doe')
end
it 'returns name' do
get "/api/person/#{person.id}"
expect(response.body.at_json_path('name')).to eql 'John Doe'
end
endCache is invalidated by calling :expire_cached_json on an instance.
describe 'updating a person' do
before :each
@person = Person.create!(name: 'John Doe')
end
it 'invalidates cache' do
expect(@person).to receive(:expire_cached_json)
@person.update_attributes!(name: 'updated')
end
endYou may also want to use this RSpec matcher.
describe 'updating a person' do
it 'invalidates cache' do
expect do
@person.update_attributes!(name: 'updated')
end.to invalidate @person
end
endThis gem implements two interesting optimizations.
Consider an array of Mongoid instances, each with numerous references to other objects. It's typical to see such instances reference the same object. Mongoid::CachedJson first collects all JSON references, then resolves them after suppressing duplicates. This significantly reduces the number of cache queries.
Various cache stores, including Memcached, support bulk read operations. The Dalli gem exposes this via the read_multi method. Mongoid::CachedJson will always invoke read_multi where available, which significantly reduces the number of network roundtrips to the cache servers.
See CONTRIBUTING.
MIT License, see LICENSE for details.
(c) 2012-2014 Artsy and Contributors