Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@
pending, when the page doesn't load in time

### Changed
- `page.on(:request)`, `on(:auth)` and `on(:dialog)` callbacks form a chain: they run in order and the rest are skipped
once one continues, responds to, aborts, accepts or dismisses, and an unhandled request is continued. They receive
only the object, not `index` and `total`, use `handled?` to check whether a previous callback took it. A callback
that holds a request to answer later gets it continued, and one registered after the handler no longer sees it
- `page.cookies` is the cookies the page sees, the ones sent to its URL and its frames' URLs, or `each(urls:)`. It
returned every cookie in the context, use `browser.cookies` or `browser.contexts[page.context_id].cookies` for that.
`set` and `remove` default to the page's host, `clear` keeps other domains, `store`/`load` are on the context only
Expand All @@ -30,6 +34,9 @@
`go_to` instead

### Fixed
- `page.on(:request, &:continue)` never continued the request, it called `continue(index, total)` and the error was
dropped. Symbol procs and lambdas taking one argument get only the object now, for `:request`, `:response`, `:auth`
and `:dialog`
- `AuthRequest#abort` raised `BrowserError` and left the page hanging, it cancels the auth challenge now
- `InterceptedRequest#navigation_request?` and `AuthRequest#navigation_request?` always returned `nil`, they are true
for document requests now
Expand Down
3 changes: 3 additions & 0 deletions docs/16-dialogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ Accept dialog with given text or default prompt if applicable

Dismiss dialog

Callbacks run in the order they were added and the rest are skipped once one accepts or dismisses the dialog. A dialog
nobody handled is accepted with a warning.

```ruby
page.on(:dialog) do |dialog|
if dialog.match?(/bla-bla/)
Expand Down
14 changes: 12 additions & 2 deletions docs/7-network.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,10 @@ Subscribes to a CDP event, or to one of the following pseudo-events. Returns a
subscription id you can pass to `off` to unsubscribe.

* `:request` an intercepted request, see `intercept` below. Requires `network.intercept` to be set up first.
Callbacks run in the order they were added and the rest are skipped once one handles the request.
* `:response` a request's response once it's fully loaded, see below. Works without `intercept`.
* `:auth` a proxy/basic auth challenge, see `authorize` below. Requires `network.intercept` to be set up first.
Works like `:request`, an unhandled challenge gets Chrome's default response.
* `:dialog` a JS dialog (`alert`/`confirm`/`prompt`), see [Dialogs](./16-dialogs.md).
* anything else is passed straight through as a raw CDP event name, e.g. `page.on("Page.frameNavigated") { |params| ... }`.

Expand Down Expand Up @@ -133,8 +135,16 @@ Set request interception for given options. This method is only sets request
interception, you should use `on` callback to catch requests and abort or
continue them.

Every paused request has to be continued, aborted or responded to by one of your `on(:request)` callbacks, including
ones that only log or inspect it. A request nobody handles stays paused, and the page waits for it until it times out.
`on(:request)` callbacks run in the order they were added and receive only the request. The first one that continues,
responds to or aborts it handles it and the rest are skipped, so a callback that doesn't care about a request just
returns. A request no callback handled is continued.

```ruby
page.on(:request) { |request| request.abort if request.match?(/ads/) }
page.on(:request) { |request| request.respond(body: "stub") if request.match?(/api/) }
```

`request.handled?` tells whether it was already handled.

* options `Hash`
* :pattern `String` \* by default
Expand Down
4 changes: 2 additions & 2 deletions docs/9-proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ end
## Intercepting requests behind a proxy

A proxy with `:user` and `:password` is authorized through request interception, so Ferrum continues every request
for you as long as nothing else subscribes to `on(:request)`. Once you subscribe, or set `network.blacklist` or
`network.whitelist`, your code takes over and has to continue or abort every request:
nothing else handles. Your `on(:request)` callbacks, `network.blacklist` and `network.whitelist` run first and can
continue, respond to or abort a request, the rest are continued:

```ruby
browser = Ferrum::Browser.new(proxy: { host: "x.x.x.x", port: "8800", user: "user", password: "pa$$" })
Expand Down
2 changes: 1 addition & 1 deletion lib/ferrum/client/subscriber.rb
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ def off(event, id)
# @return [Boolean]
#
def subscribed?(event)
@on.key?(event)
@on[event].to_a.any?
end

#
Expand Down
12 changes: 12 additions & 0 deletions lib/ferrum/dialog.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,16 @@ def initialize(page, params)
@page = page
@message = params["message"]
@default_prompt = params["defaultPrompt"]
@handled = false
end

#
# Whether the dialog was accepted or dismissed.
#
# @return [Boolean]
#
def handled?
@handled
end

#
Expand All @@ -36,6 +46,7 @@ def accept(prompt_text = nil)
options = { accept: true }
response = prompt_text || default_prompt
options.merge!(promptText: response) if response
@handled = true
@page.command("Page.handleJavaScriptDialog", slowmoable: true, **options)
end

Expand All @@ -54,6 +65,7 @@ def accept(prompt_text = nil)
# browser.go_to("https://google.com")
#
def dismiss
@handled = true
@page.command("Page.handleJavaScriptDialog", slowmoable: true, accept: false)
end

Expand Down
71 changes: 44 additions & 27 deletions lib/ferrum/interceptable.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# frozen_string_literal: true

require "ferrum/interceptable/chain"

module Ferrum
# Shared by {Page} and {Worker}: subscribes to the CDP events behind the
# `:request` (Fetch-domain request interception), `:response` (a request's
Expand All @@ -10,48 +12,40 @@ module Ferrum
module Interceptable
# Subscribes to a CDP event, or to `:request`/`:response`/`:auth`/`:console`.
#
# `:request` and `:auth` callbacks form a chain: they run in the order they
# were added, and the rest are skipped as soon as one of them handles the
# object (`continue`, `respond`, `abort`). An object nobody handled is
# continued, an auth challenge gets Chrome's default response.
#
# @param [Symbol, String] name
#
# @yieldparam [Network::InterceptedRequest, Network::Exchange, Network::AuthRequest, Console::Message, Hash] arg
# For `:request`, the intercepted {Network::InterceptedRequest}. For
# `:response`, the finished {Network::Exchange}. For `:auth`, the
# {Network::AuthRequest} challenge. For `:console`, the new
# {Console::Message}. For any other (raw CDP) event name, the event's
# params `Hash`.
#
# @yieldparam [Integer] index
# This callback's position among all callbacks currently registered for
# the event. Not passed for `:console`.
#
# @yieldparam [Integer] total
# Total number of callbacks currently registered for the event. Not
# passed for `:console`.
# params `Hash`, followed by the callback's index and the total number
# of callbacks registered for the event.
#
# @return [Integer]
# The subscription id, used to unsubscribe via {#off}.
#
# @example
# page.on(:console) { |message| warn(message) if message.error? }
#
# @example
# page.on(:request) { |r| r.abort if r.match?(/ads/) }
# page.on(:request) { |r| r.respond(body: "stub") if r.match?(/api/) }
def on(name, &block)
case name
when :console
console.on(&block)
when :request
client.on("Fetch.requestPaused") do |params, index, total|
request = Network::InterceptedRequest.new(client, params)
exchange = network.find_or_build_exchange(request.network_id)
exchange.intercepted_request = request
block.call(request, index, total)
end
when :request, :auth
chain(name).add(&block)
when :response
client.on("Network.loadingFinished") do |params, index, total|
client.on("Network.loadingFinished") do |params|
exchange = network.select(params["requestId"]).last
block.call(exchange, index, total) if exchange
end
when :auth
client.on("Fetch.authRequired") do |params, index, total|
request = Network::AuthRequest.new(self, params)
block.call(request, index, total)
block.call(exchange) if exchange
end
else
client.on(name, &block)
Expand All @@ -70,12 +64,10 @@ def off(name, id)
case name
when :console
console.off(id)
when :request
client.off("Fetch.requestPaused", id)
when :request, :auth
chain(name).remove(id)
when :response
client.off("Network.loadingFinished", id)
when :auth
client.off("Fetch.authRequired", id)
else
client.off(name, id)
end
Expand All @@ -89,5 +81,30 @@ def off(name, id)
def subscribed?(event)
client.subscribed?(event)
end

private

def chain(name)
@chains ||= Concurrent::Hash.new
@chains[name] ||= build_chain(name)
end

def build_chain(name)
case name
when :request
Chain.new(client, "Fetch.requestPaused",
build: lambda do |params|
request = Network::InterceptedRequest.new(client, params)
exchange = network.find_or_build_exchange(request.network_id)
exchange.intercepted_request = request
request
end,
default: lambda(&:continue))
when :auth
Chain.new(client, "Fetch.authRequired",
build: ->(params) { Network::AuthRequest.new(self, params) },
default: ->(request) { request.continue(authChallengeResponse: { response: "Default" }) })
end
end
end
end
99 changes: 99 additions & 0 deletions lib/ferrum/interceptable/chain.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# frozen_string_literal: true

module Ferrum
module Interceptable
#
# An ordered chain of handlers for an event that has to be answered
# exactly once, e.g. a paused request or a dialog. Handlers run in the
# order they were added and the chain stops at the first one that handles
# the event object, so a handler that doesn't want the object just
# returns. An object nobody handled is passed to the default action.
#
class Chain
# @param [#on, #off] client
# Where the event comes from.
#
# @param [String] event
# The CDP event name.
#
# @param [#call] build
# Turns the event params into the object handlers receive. The object
# has to respond to `handled?`.
#
# @param [#call] default
# Called with the object when no handler handled it.
#
# @param [Boolean] persistent
# Keep listening to the event while no handlers are left, so the
# default action still applies.
def initialize(client, event, build:, default:, persistent: false)
@client = client
@event = event
@build = build
@default = default
@handlers = Concurrent::Hash.new
@sequence = Concurrent::AtomicFixnum.new
@subscription = nil
@persistent = persistent
@mutex = Mutex.new
@mutex.synchronize { subscribe } if persistent
end

#
# Adds a handler to the end of the chain.
#
# @return [Integer]
# The id to pass to {#remove}.
#
def add(&block)
id = @sequence.increment
@mutex.synchronize do
@handlers[id] = block
subscribe
end
id
end

#
# Removes a handler, and stops listening to the event once no handlers
# are left.
#
# @param [Integer] id
# The id returned by {#add}.
#
# @return [void]
#
def remove(id)
@mutex.synchronize do
@handlers.delete(id)
next if @persistent || @handlers.any? || @subscription.nil?

@client.off(@event, @subscription)
@subscription = nil
end
end

private

def subscribe
@subscribe ||= @client.on(@event) { |params| run(params) }
end

def run(params)
object = @build.call(params)
@handlers.each_value do |handler|
break if object.handled?

call(handler, object)
end
@default.call(object) unless object.handled?
end

def call(handler, object)
handler.call(object)
rescue StandardError => e
warn("Ferrum: #{@event} callback raised #{e.class}: #{e.message}\n #{e.backtrace&.first}")
end
end
end
end
22 changes: 9 additions & 13 deletions lib/ferrum/network.rb
Original file line number Diff line number Diff line change
Expand Up @@ -318,19 +318,15 @@ def authorize(user:, password:, type: :server, &block)

@page.on(:request, &block)

@page.on(:auth) do |request, index, total|
if request.auth_challenge?(type)
response = authorized_response(@authorized_ids[type],
request.request_id,
user, password)

@authorized_ids[type] << request.request_id
request.continue(authChallengeResponse: response)
elsif index + 1 < total
next # There are other callbacks that can handle this
else
request.abort
end
@page.on(:auth) do |request|
next unless request.auth_challenge?(type)

response = authorized_response(@authorized_ids[type],
request.request_id,
user, password)

@authorized_ids[type] << request.request_id
request.continue(authChallengeResponse: response)
end
end

Expand Down
11 changes: 11 additions & 0 deletions lib/ferrum/network/auth_request.rb
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,16 @@ def initialize(page, params)
@frame_id = params["frameId"]
@resource_type = params["resourceType"]
@request = params["request"]
@handled = false
end

#
# Whether the challenge was answered or canceled.
#
# @return [Boolean]
#
def handled?
@handled
end

#
Expand Down Expand Up @@ -73,6 +83,7 @@ def match?(pattern)
#
def continue(**options)
options = options.merge(requestId: request_id)
@handled = true
@page.command("Fetch.continueWithAuth", **options)
end

Expand Down
Loading
Loading