diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c4f956b..7936d4b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,13 @@ ## [Unreleased](https://github.com/rubycdp/ferrum/compare/v0.20.1...main) ## ### Added +- `Ferrum::Deadline.within(seconds) { ... }` bounds every call the block makes with one budget, `page.go_to`, `cookies` + and `evaluate` included. `go_to` raises `NavigationTimeoutError` once it runs out, whatever `on_timeout` is +- `timeout:` on `go_to`, `refresh`, `back`, `forward`, `click` and `wait_for_navigation` bounds the whole call with a single + `Ferrum::Deadline` +- `context.target(id)` returns the context's target with that id +- `timeout:` on `keyboard.type`, `node.type` and `mouse.move` bounds all the keys or steps together, the page `timeout` + by default, raise it for a long text - `cookies[name, domain:, path:]` picks one of the cookies sharing a name - `context.cookies` with every cookie in the context whatever its domain, `browser.cookies` is the default context's - `network.emulate(user_agent:, accept_language:, platform:, metadata:)` overrides what the page reports in the request @@ -10,6 +17,22 @@ pending, when the page doesn't load in time ### Changed +- `client.command` needs a `timeout:` and has no `async: true`, use `client.notify` to send without waiting, and + `browser.command` defaults to the browser's `timeout` +- `:protocol_timeout` is ignored and warns, as do `browser.protocol_timeout`, `browser.protocol_timeout=` and + `FERRUM_PROTOCOL_TIMEOUT`: internal commands have a fixed 5 second timeout and the connection is closed once a + write to the browser stalls for 30 seconds +- Creating pages and contexts follows `timeout` +- `wait:` of `click`, `mouse.up` and `mouse.down` is renamed to `navigation_wait:`, `wait:` still works and warns +- `Mouse::CLICK_WAIT` and `Page::GOTO_WAIT` are merged into `Page::NAVIGATION_WAIT`, set by `FERRUM_NAVIGATION_WAIT`, + `FERRUM_CLICK_WAIT` and `FERRUM_GOTO_WAIT` still work and warn +- `page.command` doesn't take `wait:` anymore, wrap it in `page.wait_for_navigation` to wait for a navigation it starts. + Passing `wait:` or `async:` to a command raises `ArgumentError`, they would be sent to the browser as parameters +- `page.event` is `page.idle` and holds a `Ferrum::Concurrency::Cell`, not a `Concurrent::Event` +- `browser.targets` and `context.targets` return a copy of the hash, it doesn't follow the targets that come and go +- `go_to`, `refresh` and the script methods no longer restart the timeout for each step, one `timeout` covers the whole call +- `click`, `mouse.move`, `keyboard.type`, `back`, `forward`, `wait_for_selector` and node scripts share one `timeout` across the + commands they send, they restarted it for each - `page.on(:request)`, `on(:auth)` and `on(:dialog)` callbacks form a chain: they run in order and the rest are skipped once one continues, responds to, aborts, accepts or dismisses, and an unhandled request is continued. They receive only the object, not `index` and `total`, use `handled?` to check whether a previous callback took it. A callback diff --git a/docs/10-mouse-keyboard.md b/docs/10-mouse-keyboard.md index 0702ec67..076e01d9 100644 --- a/docs/10-mouse-keyboard.md +++ b/docs/10-mouse-keyboard.md @@ -30,8 +30,9 @@ Click given coordinates, fires mouse move, down and up events. * :x `Integer` * :y `Integer` * :delay `Float` defaults to 0. Delay between mouse down and mouse up events - * :wait `Numeric` defaults to `Mouse::CLICK_WAIT`. Seconds to watch for a navigation started by the click, then + * :navigation_wait `Numeric` defaults to `Page::NAVIGATION_WAIT`. Seconds to watch for a navigation started by the click, then wait for it to load. `0` returns right after the click. + * :timeout `Numeric` how long the click and the page load it starts may take together, the page timeout by default * :button `Symbol` :left | :right, defaults to :left * :count `Integer` defaults to 1 * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`. @@ -56,8 +57,11 @@ Mouse up for given coordinates. * :count `Integer` defaults to 1 * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`. If omitted, uses the keys held by `keyboard.hold`, or none outside of it + * :navigation_wait `Numeric` seconds to watch for a navigation started by releasing the button, then wait for it + to load, defaults to 0 + * :timeout `Numeric` how long the event and the page load it starts may take together, the page timeout by default -#### move(x:, y:, steps: 1, modifiers: nil) : `Mouse` +#### move(x:, y:, steps: 1, modifiers: nil, timeout: nil) : `Mouse` Mouse move to given x and y. @@ -67,6 +71,7 @@ Mouse move to given x and y. * :steps `Integer` defaults to 1. Sends intermediate mousemove events. * :modifiers `Integer` bitfield for key modifiers. See `keyboard.modifiers`. If omitted, uses the keys held by `keyboard.hold`, or none outside of it + * :timeout `Numeric` how long all the steps may take together, the page timeout by default ## Keyboard @@ -86,12 +91,14 @@ Dispatches a keyup event. * key `String` | `Symbol` Name of key such as "b", :enter, :backspace. Raises `ArgumentError` for a modifier, use `hold` instead -#### type(\*keys) : `Keyboard` +#### type(\*keys, timeout: nil) : `Keyboard` Sends a keydown, keypress/input, and keyup event for each character in the text. * text `String` | `Array | Array` A text to type into a focused element, `[:Shift, "s"], "tring"` +* timeout `Numeric` how long typing all the keys may take together, the page timeout by default. Raise it for a long + text #### hold(\*keys) { } : `Object` diff --git a/docs/18-node.md b/docs/18-node.md index 4b89da05..51bb7a70 100644 --- a/docs/18-node.md +++ b/docs/18-node.md @@ -111,10 +111,11 @@ Removes focus from the node. page.at_css("input[name='q']").focus.blur ``` -#### type(\*keys) : `Node` +#### type(\*keys, timeout: nil) : `Node` Types the keys with the page's keyboard, into whatever is focused, so it's usually chained after `focus` or `click`. See -[keyboard](https://github.com/rubycdp/ferrum#keyboard) for what a key can be. +[keyboard](https://github.com/rubycdp/ferrum#keyboard) for what a key can be. `timeout` bounds typing all the keys together, +the page timeout by default. ```ruby page.at_css("input[name='q']").focus.type("Ruby", :Enter) @@ -135,12 +136,15 @@ moving. * :position `Symbol` `:top` | `:center`, what the offset is measured from, the top-left corner of the node or its center, `:top` by default * :delay `Float` delay between mouse down and mouse up, `0` by default - * :wait `Numeric` seconds to watch for a navigation started by a left click, then wait for it to load, - `Mouse::CLICK_WAIT` by default. Ignored for `:right` and `:double`. `0` returns right after the click. + * :navigation_wait `Numeric` seconds to watch for a navigation started by a left click, then wait for it to load, + `Page::NAVIGATION_WAIT` by default. Ignored for `:right` and `:double`. `0` returns right after the click. + * :timeout `Numeric` how long finding the node's position, the click and the page load it starts may take together, + the page timeout by default ```ruby page.at_css("a").click -page.at_css("a").click(wait: 0) +page.at_css("a").click(navigation_wait: 0) +page.at_css("a").click(timeout: 30) page.at_css("a").click(mode: :right) page.at_css("a").click(keys: [:ctrl, :shift]) page.at_css("canvas").click(offset: { x: 10, y: 20 }) diff --git a/docs/2-customization.md b/docs/2-customization.md index 0a1b5bf8..1c948e67 100644 --- a/docs/2-customization.md +++ b/docs/2-customization.md @@ -27,8 +27,11 @@ Ferrum::Browser.new(options) written to it, see [Logging](#logging). * `:slowmo` (Integer | Float) - Set a delay in seconds to wait before sending command. Useful companion of headless option, so that you have time to see changes. - * `:timeout` (Numeric) - The number of seconds we'll wait for a response when - communicating with browser. Default is 5. + * `:timeout` (Numeric) - How long a call waits for the browser, in seconds, `FERRUM_DEFAULT_TIMEOUT` or 5 by + default. `go_to`, `refresh` and `wait_for_navigation` wait for the page to load within it. A call that only sends + commands, like `evaluate` or `at_css`, waits for the browser's answer. A call that runs out raises + `Ferrum::TimeoutError`. Pass `timeout:` to `go_to`, `refresh` or `wait_for_navigation` to change it for one + call, or wrap calls in `Ferrum::Deadline.within` to bound them together, see [Timeouts](#timeouts). * `:js_errors` (Boolean) - Deprecated, will be removed in 0.22. When true, JavaScript errors get re-raised in Ruby. Uncaught exceptions are in the [console](/docs/ferrum/javascript#console), use `page.console.select { |m| m.source == :exception }` or `page.on(:console)` instead. @@ -185,3 +188,23 @@ Ferrum::Browser.new(browser_path: "/Applications/Google Chrome.app/Contents/MacO # Add custom browser options Ferrum::Browser.new(browser_options: { "disable-web-security" => nil }) ``` + +## Timeouts + +Every call that talks to the browser can raise `Ferrum::TimeoutError`, and `Ferrum::DeadBrowserError` when the browser is +gone, even where nothing looks like a request, like `page.cookies.each`. Each call waits at most the `:timeout` option. + +To bound a group of calls by one budget, wrap them in `Ferrum::Deadline.within`. Every call made in the block by the +same thread shares what is left of it, and a call can only shorten it, never extend it: + +```ruby +Ferrum::Deadline.within(10) do + page.go_to("https://example.com") + page.cookies.to_a + page.evaluate("document.title") +end +# => raises Ferrum::TimeoutError from the first call that runs out of the 10 seconds +``` + +Blocks nest, the shorter budget wins. Other threads, such as event callbacks, don't inherit it, open a block of their own +there. diff --git a/docs/3-navigation.md b/docs/3-navigation.md index 98a444b3..c9472ad8 100644 --- a/docs/3-navigation.md +++ b/docs/3-navigation.md @@ -12,27 +12,36 @@ Navigate page to. configuring driver. * options `Hash` * :on_timeout `Symbol` What to do when the timeout is reached: `:ignore` (default) returns `nil`, - `:raise` raises `Ferrum::NavigationTimeoutError` naming the requests still pending. + `:raise` raises `Ferrum::NavigationTimeoutError` naming the requests still pending. It raises with `:ignore` too + once an enclosing `Ferrum::Deadline.within` has run out. + * :timeout `Numeric` How long the whole navigation may take, sending the request and loading the page together, + the page timeout by default. ```ruby page.go_to("https://github.com/") page.go_to("https://github.com/", on_timeout: :raise) +page.go_to("https://github.com/", timeout: 10) ``` -#### back +#### back(timeout: nil) Navigate to the previous page in history. +* timeout `Numeric` how long the history lookup and the page load may take together, the page timeout by default + ```ruby page.go_to("https://github.com/") page.at_xpath("//a").click page.back +page.back(timeout: 10) ``` -#### forward +#### forward(timeout: nil) Navigate to the next page in history. +* timeout `Numeric` how long the history lookup and the page load may take together, the page timeout by default + ```ruby page.go_to("https://github.com/") page.at_xpath("//a").click @@ -40,10 +49,12 @@ page.back page.forward ``` -#### refresh +#### refresh(timeout: nil) Reload current page. +* timeout `Numeric` how long the whole reload may take, the page timeout by default + ```ruby page.go_to("https://github.com/") page.refresh @@ -58,22 +69,23 @@ page.go_to("https://github.com/") page.stop ``` -#### wait_for_navigation(wait = 0.1) { } : `Boolean` +#### wait_for_navigation(navigation_wait = 0.1, timeout: nil) { } : `Boolean` -Runs the block and, if it starts a navigation within `wait` seconds, waits for the new page to load. Returns `false` -if it doesn't load within the page timeout. `click` already does this, use it for navigations started from +Runs the block and, if it starts a navigation within `navigation_wait` seconds, waits for the new page to load. Returns `false` +if it doesn't load within `timeout`. `click` already does this, use it for navigations started from JavaScript. -* wait `Numeric` seconds to watch for a navigation after the block, `FERRUM_CLICK_WAIT` or `0.1` by default +* navigation_wait `Numeric` seconds to watch for a navigation after the block, `FERRUM_NAVIGATION_WAIT` or `0.1` by default +* timeout `Numeric` how long the block and the navigation may take together, the page timeout by default ```ruby page.wait_for_navigation { page.at_css("a").evaluate("this.click()") } # => true page.url # => the link's page ``` -#### wait_for_navigation!(wait = 0.1) { } +#### wait_for_navigation!(navigation_wait = 0.1, timeout: nil) { } -Same as `wait_for_navigation`, but raises `Ferrum::TimeoutError` if the new page doesn't load within the page timeout. +Same as `wait_for_navigation`, but raises `Ferrum::TimeoutError` if the new page doesn't load within `timeout`, the page timeout by default. ```ruby page.wait_for_navigation! { page.at_css("a").evaluate("this.click()") } diff --git a/lib/ferrum.rb b/lib/ferrum.rb index db746feb..1fd0e1da 100644 --- a/lib/ferrum.rb +++ b/lib/ferrum.rb @@ -2,10 +2,13 @@ require "concurrent-ruby" require "ferrum/version" -require "ferrum/utils/event" require "ferrum/utils/thread" require "ferrum/utils/platform" require "ferrum/utils/elapsed_time" +require "ferrum/deadline" +require "ferrum/concurrency/cell" +require "ferrum/concurrency/awaitable_map" +require "ferrum/concurrency/dispatcher" require "ferrum/utils/attempt" require "ferrum/utils/deprecate" require "ferrum/errors" diff --git a/lib/ferrum/accessibility.rb b/lib/ferrum/accessibility.rb index a7dd2753..1c1a851b 100644 --- a/lib/ferrum/accessibility.rb +++ b/lib/ferrum/accessibility.rb @@ -36,6 +36,8 @@ def node_for(node) # @param [Boolean] fetch_relatives # @return [Array] # + # @raise [TimeoutError] + # def partial_tree(node:, fetch_relatives: false) nodes = node.page.command("Accessibility.getPartialAXTree", nodeId: node.node_id, @@ -50,6 +52,8 @@ def partial_tree(node:, fetch_relatives: false) # @param [String, nil] frame_id # @return [Array] # + # @raise [TimeoutError] + # def snapshot(depth: nil, frame_id: nil) params = { depth: depth, frameId: frame_id }.compact build(@page.command("Accessibility.getFullAXTree", **params)["nodes"]) @@ -63,6 +67,8 @@ def snapshot(depth: nil, frame_id: nil) # @param [Ferrum::Node, nil] node Scope the query to this node's subtree. # @return [Array] # + # @raise [TimeoutError] + # def query(name: nil, role: nil, node: nil) page = node ? node.page : @page params = { accessibleName: name, role: role }.compact @@ -76,6 +82,8 @@ def query(name: nil, role: nil, node: nil) # @param [String, nil] frame_id # @return [AXNode, nil] # + # @raise [TimeoutError] + # def root(frame_id: nil) params = { depth: 1, frameId: frame_id }.compact build(@page.command("Accessibility.getFullAXTree", **params)["nodes"]).first @@ -84,6 +92,8 @@ def root(frame_id: nil) # Enables the Accessibility domain, activating live AX tree change events. # # @return [self] + # + # @raise [TimeoutError] def enable @page.command("Accessibility.enable") self @@ -92,6 +102,8 @@ def enable # Disables the Accessibility domain. # # @return [self] + # + # @raise [TimeoutError] def disable @page.command("Accessibility.disable") self diff --git a/lib/ferrum/browser.rb b/lib/ferrum/browser.rb index 49eb5764..fe03ae3e 100644 --- a/lib/ferrum/browser.rb +++ b/lib/ferrum/browser.rb @@ -47,9 +47,8 @@ class Browser attr_reader :client, :process, :contexts, :options - delegate %i[timeout timeout= protocol_timeout protocol_timeout= - base_url base_url= default_user_agent default_user_agent= extensions add_extension] => :options - delegate %i[command] => :client + delegate %i[timeout timeout= protocol_timeout protocol_timeout= base_url base_url= default_user_agent + default_user_agent= extensions add_extension] => :options # # Initializes the browser. @@ -89,13 +88,11 @@ class Browser # changes. # # @option options [Numeric] :timeout (5) - # The number of seconds we'll wait for a response when communicating - # with the browser: navigations, JS evaluation, DOM queries, dispatching input, etc. - # - # @option options [Numeric] :protocol_timeout (5) - # The number of seconds we'll wait for an individual internal CDP - # bookkeeping call to respond, e.g. `Target.createTarget`, - # `Target.attachToTarget`. These normally resolve in milliseconds. + # How long a call waits for the browser, in seconds: navigations, JS evaluation, DOM queries, dispatching + # input, etc. `go_to`, `refresh` and `wait_for_navigation` wait for the page to load within it. A call that + # only sends commands waits for the browser's answer. A call that runs out raises {TimeoutError}. + # `FERRUM_DEFAULT_TIMEOUT` sets the default. Pass `timeout:` to `go_to`, `refresh` or `wait_for_navigation` to + # change it for one call, or wrap calls in {Deadline.within} to bound them together. # # @option options [Boolean] :js_errors # Deprecated, will be removed in Ferrum 0.22. When true, JavaScript @@ -168,6 +165,8 @@ def initialize(options = nil) # @return [Ferrum::Page] # Created page. # + # @raise [TimeoutError] + # def create_page(new_context: false, proxy: nil) page = if new_context || proxy params = {} @@ -195,6 +194,8 @@ def create_page(new_context: false, proxy: nil) # # Closes browser tabs opened by the `Browser` instance. # + # @raise [TimeoutError] + # # @example # # connect to a long-running Chrome process # browser = Ferrum::Browser.new(url: 'http://localhost:9222') @@ -256,11 +257,34 @@ def close command("Browser.close") end + # + # Sends a CDP command to the browser. + # + # @param [String] method + # The CDP method name, e.g. `"Browser.getVersion"`. + # + # @param [Numeric, Deadline, nil] timeout + # How long to wait for the response, the browser's `timeout` by default. + # + # @param [Hash] params + # The command's parameters. + # + # @return [Hash] + # The command's result. + # + # @raise [TimeoutError] + # + def command(method, timeout: nil, **params) + client.command(method, timeout: timeout || self.timeout, **params) + end + # # Gets the version information from the browser. # # @return [VersionInfo] # + # @raise [TimeoutError] + # # @since 0.13 # def version diff --git a/lib/ferrum/browser/options.rb b/lib/ferrum/browser/options.rb index 84ba93c2..1c83bda2 100644 --- a/lib/ferrum/browser/options.rb +++ b/lib/ferrum/browser/options.rb @@ -15,18 +15,23 @@ class Options BROWSER_HOST = "127.0.0.1" WINDOW_SIZE = [1024, 768].freeze BASE_URL_SCHEMA = %w[http https].freeze - DEFAULT_TIMEOUT = ENV.fetch("FERRUM_DEFAULT_TIMEOUT", 5).to_i - DEFAULT_PROTOCOL_TIMEOUT = ENV.fetch("FERRUM_PROTOCOL_TIMEOUT", DEFAULT_TIMEOUT).to_i + DEFAULT_TIMEOUT = ENV.fetch("FERRUM_DEFAULT_TIMEOUT", 5).to_f PROCESS_TIMEOUT = ENV.fetch("FERRUM_PROCESS_TIMEOUT", 10).to_i + PROTOCOL_TIMEOUT_REMOVED = "is ignored and will be removed in Ferrum 0.22. Internal commands have a fixed " \ + "timeout, :timeout bounds everything else." + warn "Ferrum: FERRUM_DEBUG is deprecated, use FERRUM_LOG=* instead" if ENV.key?("FERRUM_DEBUG") + if ENV.key?("FERRUM_PROTOCOL_TIMEOUT") + Utils::Deprecate.warn("FERRUM_PROTOCOL_TIMEOUT #{PROTOCOL_TIMEOUT_REMOVED}") + end attr_reader :window_size, :logger, :log, :ws_max_receive_size, :js_errors, :base_url, :slowmo, :pending_connection_errors, :url, :ws_url, :env, :process_timeout, :browser_path, :save_path, :proxy, :port, :host, :headless, :incognito, :dockerize, :browser_options, :ignore_default_browser_options, :xvfb, :flatten, :extensions - attr_accessor :timeout, :protocol_timeout, :default_user_agent + attr_accessor :timeout, :default_user_agent def initialize(options = nil) @options = Hash(options&.dup) @@ -34,7 +39,7 @@ def initialize(options = nil) @port = @options.fetch(:port, BROWSER_PORT) @host = @options.fetch(:host, BROWSER_HOST) @timeout = @options.fetch(:timeout, DEFAULT_TIMEOUT) - @protocol_timeout = @options.fetch(:protocol_timeout, DEFAULT_PROTOCOL_TIMEOUT) + warn_protocol_timeout if @options.key?(:protocol_timeout) @window_size = @options.fetch(:window_size, WINDOW_SIZE) @js_errors = fetch_deprecated( :js_errors, @@ -69,6 +74,29 @@ def initialize(options = nil) @browser_options = @options.fetch(:browser_options, {}).freeze end + # + # Deprecated, will be removed in Ferrum 0.22. Internal commands have a fixed timeout. + # + # @return [Integer] + # {Client::PROTOCOL_TIMEOUT}. + # + def protocol_timeout + warn_protocol_timeout + Client::PROTOCOL_TIMEOUT + end + + # + # Deprecated, will be removed in Ferrum 0.22. The value is ignored. + # + # @param [Numeric] value + # + # @return [Numeric] + # + def protocol_timeout=(value) + warn_protocol_timeout + value + end + # # Sets the base URL relative navigations are resolved against. # @@ -137,6 +165,10 @@ def to_h private + def warn_protocol_timeout + Utils::Deprecate.warn(":protocol_timeout #{PROTOCOL_TIMEOUT_REMOVED}", user_code: true) + end + def fetch_deprecated(name, instead) return false unless @options.key?(name) diff --git a/lib/ferrum/client.rb b/lib/ferrum/client.rb index 69fe4bd3..590d70e9 100644 --- a/lib/ferrum/client.rb +++ b/lib/ferrum/client.rb @@ -1,8 +1,6 @@ # frozen_string_literal: true -require "concurrent-ruby" require "forwardable" -require "ferrum/client/subscriber" require "ferrum/client/web_socket" require "ferrum/utils/thread" @@ -45,22 +43,35 @@ def initialize(client, session_id) # @param [String] method # The CDP method name, e.g. `"Page.navigate"`. # - # @param [Boolean] async - # Whether to send the command without waiting for a response. + # @param [Numeric, Deadline] timeout + # How long to wait for this command's response. See {Client#send_message}. # # @param [Hash] params # The command's parameters. # - # @param [Numeric, nil] timeout - # How long to wait for this command's response, overriding - # {Browser::Options#protocol_timeout}. See {Client#send_message}. + # @return [Hash] + # The command's result. # - # @return [Boolean, Hash] - # `true` when sent asynchronously, otherwise the command's result. + # @raise [TimeoutError] # - def command(method, async: false, timeout: nil, **params) + def command(method, timeout:, **params) message = build_message(method, params) - @client.send_message(message, async: async, timeout: timeout) + @client.send_message(message, timeout: timeout) + end + + # + # Sends a CDP command scoped to this session without waiting for a response. + # + # @param [String] method + # The CDP method name, e.g. `"Runtime.runIfWaitingForDebugger"`. + # + # @param [Hash] params + # The command's parameters. + # + # @return [true] + # + def notify(method, **params) + @client.send_notification(build_message(method, params)) end # @@ -122,12 +133,12 @@ def method_missing(name, ...) # # Removes all event callbacks registered for this session from the - # underlying client's subscriber. + # underlying client's dispatcher. # # @return [void] # def close - @client.subscriber.clear(session_id: session_id) + @client.dispatcher.clear { |k| k.match?(session_id) } end private @@ -144,24 +155,27 @@ def event_name(event) # # The low-level CDP client. Owns the {WebSocket} connection to the browser, # assigns command ids, matches responses back to their pending commands and - # dispatches incoming events to the {Subscriber}. {SessionClient} builds on + # dispatches incoming events to the {Concurrency::Dispatcher}. {SessionClient} builds on # top of it to scope commands/events to a particular target's session. # class Client extend Forwardable - delegate %i[protocol_timeout protocol_timeout=] => :options + PROTOCOL_TIMEOUT = 5 + INTERRUPTIONS = %w[Fetch.requestPaused Fetch.authRequired].freeze - attr_reader :ws_url, :options, :subscriber + delegate %i[timeout] => :options + + attr_reader :ws_url, :options, :dispatcher def initialize(ws_url, options) @command_id = 0 @command_id_mutex = Mutex.new @ws_url = ws_url @options = options - @pendings = Concurrent::Hash.new - @ws = WebSocket.new(ws_url, options.ws_max_receive_size, options.log, options.protocol_timeout) - @subscriber = Subscriber.new + @responses = Concurrency::AwaitableMap.new + @ws = WebSocket.new(ws_url, options.ws_max_receive_size, options.log) + @dispatcher = Concurrency::Dispatcher.new start end @@ -172,74 +186,86 @@ def initialize(ws_url, options) # @param [String] method # The CDP method name, e.g. `"Target.createTarget"`. # - # @param [Boolean] async - # Whether to send the command without waiting for a response. + # @param [Numeric, Deadline] timeout + # How long to wait for this command's response. See {#send_message}. # # @param [Hash] params # The command's parameters. # - # @param [Numeric, nil] timeout - # How long to wait for this command's response, overriding - # {Browser::Options#protocol_timeout}. See {#send_message}. - # - # @return [Boolean, Hash] - # `true` when sent asynchronously, otherwise the command's result. + # @return [Hash] + # The command's result. # - def command(method, async: false, timeout: nil, **params) + def command(method, timeout:, **params) message = build_message(method, params) - send_message(message, async: async, timeout: timeout) + send_message(message, timeout: timeout) + end + + # + # Sends a CDP command to the browser-wide session without waiting for a response. + # + # @param [String] method + # The CDP method name, e.g. `"Target.closeTarget"`. + # + # @param [Hash] params + # The command's parameters. + # + # @return [true] + # + def notify(method, **params) + send_notification(build_message(method, params)) end # - # Sends a raw CDP message over the websocket. Synchronous calls block - # until a matching response arrives, or `timeout` elapses, defaulting to - # `protocol_timeout` (delegated to {Browser::Options#protocol_timeout}). - # That default is the transport-level budget for internal CDP bookkeeping - # (e.g. `Target.createTarget`). {Page#command} overrides - # this back to `timeout`, or a caller-supplied budget (e.g. `#pdf`/ - # `#screenshot`'s own `timeout:` argument), for the user-facing commands - # it issues -- some of which (e.g. `Page.navigate`, `Page.printToPDF`) - # rely on their own response latency to detect a stuck operation. + # Sends a raw CDP message over the websocket and blocks until a matching + # response arrives or `timeout` elapses. The caller says how long that is, + # so the budget it runs under is visible. Pass the {Deadline} of the + # surrounding operation to share its budget; inside {Deadline.within} the + # budget is also bound by the one that block sets. # # @param [Hash] message # The message to send, must include an `:id` key. # - # @param [Boolean] async - # Whether to return immediately instead of waiting for a response. - # - # @param [Numeric, nil] timeout + # @param [Numeric, Deadline] timeout # How long to wait for the response, counting the time the message waits - # to be sent. Defaults to `protocol_timeout`. + # to be sent. # - # @return [Boolean, Hash] - # `true` when sent asynchronously, otherwise the parsed `"result"` - # from the response. + # @return [Hash] + # The parsed `"result"` from the response. # - def send_message(message, async:, timeout: nil) - timeout ||= protocol_timeout + def send_message(message, timeout:) + deadline = Deadline.wrap(timeout) - if async - @ws.send_message(message, timeout) - true - else - raise dead_browser_error if @ws.messages.closed? - - pending = Concurrent::IVar.new - @pendings[message[:id]] = pending - begin - @ws.send_message(message, timeout) - data = pending.value!(timeout) - ensure - @pendings.delete(message[:id]) - end + raise dead_browser_error if @ws.messages.closed? - raise dead_browser_error if data.nil? && @ws.messages.closed? - raise TimeoutError.new(message[:method], session_id: message[:sessionId]) unless data - - error, response = data.values_at("error", "result") - raise_browser_error(error) if error - response + @responses.expect(message[:id]) + begin + @ws.send_message(message, deadline) + data = @responses.await(message[:id], deadline) + ensure + @responses.delete(message[:id]) end + + raise dead_browser_error if data.nil? && @ws.messages.closed? + raise TimeoutError.new(message[:method], session_id: message[:sessionId]) unless data + + error, response = data.values_at("error", "result") + raise_browser_error(error) if error + response + end + + # + # Sends a raw CDP message over the websocket without waiting for a + # response, which is dropped when it arrives. Nothing bounds it: a browser + # that stops reading is detected by the write watchdog. + # + # @param [Hash] message + # The message to send, must include an `:id` key. + # + # @return [true] + # + def send_notification(message) + @ws.send_message(message) + true end # @@ -252,7 +278,7 @@ def send_message(message, async:, timeout: nil) # The subscription id, used to unsubscribe via {#off}. # def on(event, &) - @subscriber.on(event, &) + @dispatcher.on(event, &) end # @@ -267,7 +293,7 @@ def on(event, &) # @return [void] # def off(event, id) - @subscriber.off(event, id) + @dispatcher.off(event, id) end # Whether there's at least one callback registered for the event. @@ -277,7 +303,7 @@ def off(event, id) # # @return [Boolean] def subscribed?(event) - @subscriber.subscribed?(event) + @dispatcher.subscribed?(event) end # @@ -295,16 +321,16 @@ def session(session_id) # # Closes the underlying websocket, drops pending commands and stops - # the message-processing thread and subscriber. + # the message-processing thread and dispatcher. # # @return [void] # def close @ws.close # Give a thread some time to handle a tail of messages - @pendings.clear + @responses.close @thread.kill unless @thread.join(1) - @subscriber.close + @dispatcher.close @ws.force_close end @@ -316,7 +342,7 @@ def close def inspect "#<#{self.class} " \ "@command_id=#{@command_id.inspect} " \ - "@pendings=#{@pendings.inspect} " \ + "@responses=#{@responses.inspect} " \ "@ws=#{@ws.inspect}>" end @@ -331,7 +357,12 @@ def inspect # # @return [Hash] # + # @raise [ArgumentError] + # When given the removed `async:`, which would otherwise be sent to the browser as a parameter. + # def build_message(method, params) + raise ArgumentError, "async: is removed, use notify to send #{method} without waiting" if params.key?(:async) + { method: method, params: params }.merge(id: next_command_id) end @@ -344,9 +375,9 @@ def start break unless message if message.key?("method") - @subscriber << message + dispatch(message) else - @pendings[message["id"]]&.set(message) + @responses.fulfill(message["id"], message) end end rescue StandardError => e @@ -355,14 +386,21 @@ def start warn("Ferrum: dispatch stopped, #{e.class}: #{e.message}\n #{e.backtrace&.first}") @ws.messages.close ensure - release_pendings + release_commands end end + # Request interception events take the priority lane so they aren't delayed behind the others. + def dispatch(message) + method, session_id, params = message.values_at("method", "sessionId", "params") + lane = INTERRUPTIONS.include?(method) ? :priority : :regular + @dispatcher.publish(SessionClient.event_name(method, session_id), params, lane: lane) + end + # Nothing is going to answer the commands still in flight once the socket # is gone, release them instead of making each wait out its timeout. - def release_pendings - @pendings.each_value { |pending| pending.try_set(nil) } + def release_commands + @responses.close end def dead_browser_error @@ -370,7 +408,7 @@ def dead_browser_error end # Locked so two concurrent commands never share an id and read each - # other's responses from @pendings. + # other's responses from @responses. def next_command_id @command_id_mutex.synchronize { @command_id += 1 } end diff --git a/lib/ferrum/client/subscriber.rb b/lib/ferrum/client/subscriber.rb deleted file mode 100644 index 000201cc..00000000 --- a/lib/ferrum/client/subscriber.rb +++ /dev/null @@ -1,145 +0,0 @@ -# frozen_string_literal: true - -module Ferrum - class Client - # - # Dispatches incoming CDP events to registered callbacks. Messages are - # queued and processed on dedicated threads, with `Fetch.requestPaused` - # and `Fetch.authRequired` given priority so request interception isn't - # delayed behind other events. - # - class Subscriber - INTERRUPTIONS = %w[Fetch.requestPaused Fetch.authRequired].freeze - - def initialize - @regular = Queue.new - @priority = Queue.new - @on = Concurrent::Hash.new - - start - end - - # - # Enqueues an incoming CDP message for dispatch to subscribers. - # `Fetch.requestPaused`/`Fetch.authRequired` messages jump the regular - # queue so request interception isn't delayed behind other events. - # - # @param [Hash] message - # The raw CDP message, as parsed from the websocket. - # - # @return [void] - # - def <<(message) - if INTERRUPTIONS.include?(message["method"]) - @priority.push(message) - else - @regular.push(message) - end - end - - # - # Registers a callback for a CDP event. - # - # @param [String] event - # The event key, as built by {SessionClient.event_name}. - # - # @return [Integer] - # The callback's index within the event's callback list, used to - # unsubscribe via {#off}. - # - def on(event, &block) - @on[event] ||= Concurrent::Array.new - @on[event] << block - @on[event].index(block) - end - - # - # Unregisters a callback for a CDP event. - # - # @param [String] event - # The event key, as built by {SessionClient.event_name}. - # - # @param [Integer] id - # The callback's index, as returned by {#on}. - # - # @return [Boolean] - # - def off(event, id) - @on[event].delete_at(id) - true - end - - # - # Whether there's at least one callback registered for the event. - # - # @param [String] event - # The event key, as built by {SessionClient.event_name}. - # - # @return [Boolean] - # - def subscribed?(event) - @on[event].to_a.any? - end - - # - # Stops the regular and priority dispatch threads. - # - # @return [void] - # - def close - @regular_thread&.kill - @priority_thread&.kill - end - - # - # Removes all callbacks registered for a given session. - # - # @param [String] session_id - # The session id to match against registered event keys. - # - # @return [void] - # - def clear(session_id:) - @on.delete_if { |k, _| k.match?(session_id) } - end - - private - - def start - @regular_thread = Utils::Thread.spawn do - loop do - message = @regular.pop - break unless message - - call(message) - end - end - - @priority_thread = Utils::Thread.spawn do - loop do - message = @priority.pop - break unless message - - call(message) - end - end - end - - def call(message) - method, session_id, params = message.values_at("method", "sessionId", "params") - event = SessionClient.event_name(method, session_id) - - total = @on[event]&.size.to_i - @on[event]&.each_with_index do |block, index| - # In case of multiple callbacks we provide current index and total - block.call(params, index, total) - rescue StandardError => e - # A raising callback used to terminate the dispatch thread, and with it every - # event for the rest of the browser's life: targets stopped being registered - # and any page created later raised `NoSuchTargetError`. Report and carry on. - warn("Ferrum: #{event} callback raised #{e.class}: #{e.message}\n #{e.backtrace&.first}") - end - end - end - end -end diff --git a/lib/ferrum/client/web_socket.rb b/lib/ferrum/client/web_socket.rb index 196d5571..e2cde999 100644 --- a/lib/ferrum/client/web_socket.rb +++ b/lib/ferrum/client/web_socket.rb @@ -25,6 +25,7 @@ class WebSocket WEBSOCKET_BUG_SLEEP = 0.05 DEFAULT_PORTS = { "ws" => 80, "wss" => 443 }.freeze WRITE_CHUNK_SIZE = 64 * 1024 + WRITE_TIMEOUT = 30 ESCAPE_OR_LONE_SURROGATE = /\\u[dD][89abAB]\h{2}\\u[dD][c-fC-F]\h{2}|\\u([dD][89a-fA-F]\h{2})|\\./ attr_reader :url, :messages @@ -33,7 +34,7 @@ class WebSocket # Why Ferrum closed the connection itself, `nil` if it didn't. attr_reader :close_reason - def initialize(url, max_receive_size, log, write_timeout = nil) + def initialize(url, max_receive_size, log, write_timeout = WRITE_TIMEOUT) @url = url @log = log @write_timeout = write_timeout @@ -118,7 +119,7 @@ def on_close(_event) # @param [Hash] data # The message to send, must include an `:id` key. # - # @param [Numeric, nil] timeout + # @param [Numeric, Deadline, nil] timeout # How long the frame may wait in the queue. A frame still queued after # that is dropped, nobody waits for its response anymore. `nil` keeps it # until it's written. @@ -127,7 +128,7 @@ def on_close(_event) # def send_message(data, timeout = nil) json = data.to_json - deadline = Utils::ElapsedTime.monotonic_time + timeout if timeout + deadline = Deadline.wrap(timeout) if timeout drive(deadline) { @driver.text(json) } @log&.sent(data, self) end @@ -195,7 +196,7 @@ def start_writer @writer = Utils::Thread.spawn do while (frame = @outgoing.pop) data, deadline = frame - write_frame(data) unless deadline && Utils::ElapsedTime.monotonic_time > deadline + write_frame(data) unless deadline&.expired? end rescue EOFError, Errno::ECONNRESET, Errno::EPIPE, Errno::EBADF, IOError # rubocop:disable Lint/ShadowedException # The browser went away, nothing to report. diff --git a/lib/ferrum/concurrency/awaitable_map.rb b/lib/ferrum/concurrency/awaitable_map.rb new file mode 100644 index 00000000..8f5a2c44 --- /dev/null +++ b/lib/ferrum/concurrency/awaitable_map.rb @@ -0,0 +1,171 @@ +# frozen_string_literal: true + +module Ferrum + module Concurrency + # + # A thread-safe hash that threads can wait on: {#await} returns the value of a key as soon as it is stored, at + # once if it already is, so it doesn't matter whether the waiter or the writer comes first. + # + # A writer that only answers waiters, like a response to a command that may have nobody waiting, uses + # {#expect} on the waiting side and {#fulfill} on the writing side: values nobody expects are dropped. + # + # @example + # map = Ferrum::Concurrency::AwaitableMap.new + # Thread.new { map["a"] = 1 } + # map.await("a", 5) # => 1 + # + class AwaitableMap + def initialize + @mutex = Mutex.new + @condition = ConditionVariable.new + @map = {} + @expected = {} + @closed = false + end + + # + # @param [Object] key + # + # @return [Object, nil] + # + def [](key) + @mutex.synchronize { @map[key] } + end + + # + # Stores the value and wakes the threads waiting for the key. + # + # @param [Object] key + # + # @param [Object] value + # + # @return [Object] + # + def []=(key, value) + @mutex.synchronize do + @map[key] = value + @condition.broadcast + end + value + end + + # + # Announces that a thread is going to wait for the key, so {#fulfill} keeps its value. + # + # @param [Object] key + # + # @return [Object] + # The key. + # + def expect(key) + @mutex.synchronize { @expected[key] = true } + key + end + + # + # @param [Object] key + # + # @return [Boolean] + # + def expecting?(key) + @mutex.synchronize { @expected.key?(key) } + end + + # + # Stores the value and wakes the waiters, only if the key was expected. + # + # @param [Object] key + # + # @param [Object] value + # + # @return [Boolean] + # Whether the value was stored. + # + def fulfill(key, value) + @mutex.synchronize do + next false unless @expected.key?(key) + + @map[key] = value + @condition.broadcast + true + end + end + + # + # Removes the key and the expectation of it. + # + # @param [Object] key + # + # @return [Object, nil] + # The removed value. + # + def delete(key) + @mutex.synchronize do + @expected.delete(key) + @map.delete(key) + end + end + + # + # Wakes every waiter for good: once closed, {#await} doesn't block and returns what is stored. + # + # @return [void] + # + def close + @mutex.synchronize do + @closed = true + @condition.broadcast + end + end + + # + # @param [Object] key + # + # @return [Boolean] + # + def key?(key) + @mutex.synchronize { @map.key?(key) } + end + + # + # @return [Hash] + # A copy of the stored pairs. + # + def to_h + @mutex.synchronize { @map.dup } + end + + # + # @return [Array] + # + def values + @mutex.synchronize { @map.values } + end + + # + # Blocks until the key is stored or the timeout is reached. + # + # @param [Object] key + # + # @param [Numeric, Deadline] timeout + # + # @return [Object, nil] + # The value, `nil` when the timeout is reached. + # + def await(key, timeout) + deadline = Deadline.wrap(timeout) + @mutex.synchronize do + @condition.wait(@mutex, deadline.remaining) until @map.key?(key) || @closed || deadline.expired? + @map[key] + end + end + + # + # @return [String] + # + def inspect + @mutex.synchronize { @map.inspect } + end + end + end +end diff --git a/lib/ferrum/concurrency/cell.rb b/lib/ferrum/concurrency/cell.rb new file mode 100644 index 00000000..9cdc8345 --- /dev/null +++ b/lib/ferrum/concurrency/cell.rb @@ -0,0 +1,128 @@ +# frozen_string_literal: true + +module Ferrum + module Concurrency + # + # A thread-safe slot that threads wait on: it holds a value once set, can be cleared and set again, and wakes + # every waiter when it is set or closed. It serves as a one-shot response, the latest value of something that + # changes, or a latch, and every wait is bounded by a {Deadline}. + # + # @example + # cell = Ferrum::Concurrency::Cell.new + # Thread.new { cell.set(42) } + # cell.wait(5) # => true + # cell.value # => 42 + # + class Cell + # How many times the cell has been cleared. + # + # @return [Integer] + attr_reader :generation + + # + # Builds a cell that already holds the value. + # + # @param [Object] value + # + # @return [Cell] + # + def self.set(value) + new.tap { |c| c.set(value) } + end + + def initialize + @mutex = Mutex.new + @condition = ConditionVariable.new + @set = false + @closed = false + @value = nil + @generation = 0 + end + + # + # Stores the value and wakes the waiters. + # + # @param [Object] value + # + # @return [Object] + # + def set(value) + @mutex.synchronize do + @value = value + @set = true + @condition.broadcast + end + value + end + + # + # Empties the cell and counts it as a new generation. + # + # @return [Integer] + # The new generation. + # + def clear + @mutex.synchronize do + @value = nil + @set = false + @generation += 1 + end + end + + # + # Whether a value is stored. + # + # @return [Boolean] + # + def set? + @mutex.synchronize { @set } + end + + # + # The stored value, `nil` when the cell is empty. + # + # @return [Object, nil] + # + def value + @mutex.synchronize { @value } + end + + # + # Wakes the waiters for good: a closed cell never blocks, so a waiter learns at once that nothing is coming. + # + # @return [void] + # + def close + @mutex.synchronize do + @closed = true + @condition.broadcast + end + end + + # + # Whether the cell is closed. + # + # @return [Boolean] + # + def closed? + @mutex.synchronize { @closed } + end + + # + # Blocks until a value is stored, the cell is closed or the timeout is reached. + # + # @param [Numeric, Deadline] timeout + # + # @return [Boolean] + # Whether a value is stored. + # + def wait(timeout) + deadline = Deadline.wrap(timeout) + @mutex.synchronize do + @condition.wait(@mutex, deadline.remaining) until @set || @closed || deadline.expired? + @set + end + end + end + end +end diff --git a/lib/ferrum/concurrency/dispatcher.rb b/lib/ferrum/concurrency/dispatcher.rb new file mode 100644 index 00000000..c615c101 --- /dev/null +++ b/lib/ferrum/concurrency/dispatcher.rb @@ -0,0 +1,148 @@ +# frozen_string_literal: true + +module Ferrum + module Concurrency + # + # Delivers published payloads to the callbacks registered for a key. Every lane is a queue with its own + # consumer thread, so payloads of one lane are delivered in order and a slow callback never holds up another + # lane. A raising callback is reported and doesn't stop the delivery. + # + # @example + # dispatcher = Ferrum::Concurrency::Dispatcher.new(lanes: %i[regular priority]) + # dispatcher.on("greeting") { |payload, index, total| puts payload } + # dispatcher.publish("greeting", "hello", lane: :priority) + # + class Dispatcher + DEFAULT_LANES = %i[regular priority].freeze + + # + # @param [Array] lanes + # Names of the lanes, the first one is the default. + # + # @param [#call, nil] on_error + # Receives the key and the error a callback raised, warns when not given. + # + def initialize(lanes: DEFAULT_LANES, on_error: nil) + @queues = lanes.to_h { |l| [l, Queue.new] } + @on_error = on_error || method(:warn_error) + @on = Concurrent::Hash.new + @sequence = Concurrent::AtomicFixnum.new + + start + end + + # + # Enqueues a payload for delivery to the callbacks registered for the key at the time it is delivered. + # + # @param [String] key + # + # @param [Object] payload + # + # @param [Symbol] lane + # The first lane when not given. + # + # @return [void] + # + # @raise [KeyError] + # When there is no such lane. + # + def publish(key, payload, lane: @queues.each_key.first) + @queues.fetch(lane).push([key, payload]) + end + + # + # Registers a callback for a key. It receives the payload, its own index among the callbacks of the key and + # their total number. + # + # @param [String] key + # + # @return [Integer] + # The callback's id, used to unsubscribe via {#off}. Ids are never reused, so removing a callback doesn't + # change the others'. + # + def on(key, &block) + id = @sequence.increment + @on[key] ||= Concurrent::Array.new + @on[key] << [id, block] + id + end + + # + # Unregisters a callback. + # + # @param [String] key + # + # @param [Integer] id + # The callback's id, as returned by {#on}. + # + # @return [Boolean] + # + def off(key, id) + @on[key]&.delete_if { |i, _| i == id } + true + end + + # + # Whether there's at least one callback registered for the key. + # + # @param [String] key + # + # @return [Boolean] + # + def subscribed?(key) + @on[key].to_a.any? + end + + # + # Stops the consumer threads. + # + # @return [void] + # + def close + @threads.each(&:kill) + end + + # + # Removes the callbacks of every key the block returns a truthy value for. + # + # @yieldparam [String] key + # + # @return [void] + # + def clear + @on.delete_if { |k, _| yield(k) } + end + + private + + def start + @threads = @queues.values.map do |queue| + Utils::Thread.spawn do + loop do + item = queue.pop + break unless item + + call(*item) + end + end + end + end + + def call(key, payload) + total = @on[key]&.size.to_i + @on[key]&.each_with_index do |(_, block), index| + block.call(payload, index, total) + rescue StandardError => e + # A raising callback used to terminate the dispatch thread, and with it every + # event for the rest of the browser's life: targets stopped being registered + # and any page created later raised `NoSuchTargetError`. Report and carry on. + @on_error.call(key, e) + end + end + + def warn_error(key, error) + warn("Ferrum: #{key} callback raised #{error.class}: #{error.message}\n #{error.backtrace&.first}") + end + end + end +end diff --git a/lib/ferrum/context.rb b/lib/ferrum/context.rb index 2e833ed0..902b78a8 100644 --- a/lib/ferrum/context.rb +++ b/lib/ferrum/context.rb @@ -13,15 +13,14 @@ module Ferrum class Context POSITION = %i[first last].freeze - attr_reader :id, :targets + attr_reader :id def initialize(client, contexts, id, implicit: false) @id = id @implicit = implicit @client = client @contexts = contexts - @targets = Concurrent::Hash.new - @pendings = Concurrent::Map.new + @targets = Concurrency::AwaitableMap.new end # Whether this is the browser's implicit context, the one it puts its @@ -52,13 +51,24 @@ def cookies # Connects to and returns the {#default_target}'s page. # # @return [Page] + # + # @raise [TimeoutError] def page default_target.page end + # The targets in this context by id, as they are at the time of the call. + # + # @return [Hash{String => Target}] + def targets + @targets.to_h + end + # All page targets in this context, connected to as {Page}s. # # @return [Array] + # + # @raise [TimeoutError] def pages @targets.values.select(&:page?).map(&:page) end @@ -67,6 +77,8 @@ def pages # leaving out the ones we have no session with, e.g. detached ones. # # @return [Array] + # + # @raise [TimeoutError] def workers @targets.values.select { |t| (t.worker? || t.shared_worker?) && t.session_id }.map(&:worker) end @@ -99,6 +111,8 @@ def windows(pos = nil, size = 1) # @param [Hash] options # # @return [Page] + # + # @raise [TimeoutError] def create_page(**options) target = create_target target.page = target.build_page(**options) @@ -110,17 +124,13 @@ def create_page(**options) # @return [Target] # # @raise [NoSuchTargetError] + # + # @raise [TimeoutError] def create_target options = { browserContextId: @id } unless implicit? - target_id = @client.command("Target.createTarget", url: "about:blank", **Hash(options))["targetId"] - - new_pending = Concurrent::IVar.new - pending = @pendings.put_if_absent(target_id, new_pending) || new_pending - resolved = pending.value(@client.protocol_timeout) - raise NoSuchTargetError unless resolved - - @pendings.delete(target_id) - @targets[target_id] + deadline = Deadline.wrap(@client.timeout) + response = @client.command("Target.createTarget", timeout: deadline, url: "about:blank", **Hash(options)) + @targets.await(response["targetId"], deadline) || raise(NoSuchTargetError) end # Registers a target discovered via a CDP `Target.*` event, or updates @@ -138,15 +148,11 @@ def create_target # @return [Target] def add_target(params:, session_id: nil, attached: false) new_target = Target.new(@client, session_id, params, implicit: implicit?) - target = @targets[new_target.id] ||= new_target + target = @targets[new_target.id] || new_target # on first iteration session_id may be nil, then if session is present here we must set it to the target target.session_id = session_id if session_id && (attached || target.session_id.nil?) @default_target ||= target - - new_pending = Concurrent::IVar.new - pending = @pendings.put_if_absent(target.id, new_pending) || new_pending - pending.try_set(true) - target + @targets[target.id] = target end # Updates a known target's params, e.g. on `Target.targetInfoChanged`. @@ -182,13 +188,16 @@ def delete_target(target_id) # @return [Target] # # @raise [NoSuchTargetError] if the context has no such target. + # + # @raise [TimeoutError] def attach_target(target_id) target = @targets[target_id] raise NoSuchTargetError unless target return target if target.session_id @contexts.manually_attached(target_id) - session = @client.command("Target.attachToTarget", targetId: target_id, flatten: true) + session = @client.command("Target.attachToTarget", timeout: Client::PROTOCOL_TIMEOUT, + targetId: target_id, flatten: true) target.session_id = session["sessionId"] target end @@ -215,17 +224,29 @@ def close_targets_connection # implicit context cannot be disposed, see {#implicit?}. # # @return [Boolean, nil] + # + # @raise [TimeoutError] def dispose @contexts.dispose(@id) end + # The target with the given id. + # + # @param [String] target_id + # + # @return [Target, nil] + # `nil` when the context has no such target. + def target(target_id) + @targets[target_id] + end + # Whether a target with the given id is known in this context. # # @param [String] target_id # # @return [Boolean] def target?(target_id) - !!@targets[target_id] + !!target(target_id) end # Debug representation of the context, including its known targets. diff --git a/lib/ferrum/contexts.rb b/lib/ferrum/contexts.rb index c9a68d07..86a4e042 100644 --- a/lib/ferrum/contexts.rb +++ b/lib/ferrum/contexts.rb @@ -79,8 +79,10 @@ def find_by(target_id:) # Keyword arguments forwarded to `Target.createBrowserContext`. # # @return [Context] + # + # @raise [TimeoutError] def create(**options) - response = @client.command("Target.createBrowserContext", **options) + response = @client.command("Target.createBrowserContext", timeout: @client.timeout, **options) context_id = response["browserContextId"] context = Context.new(@client, self, context_id) @contexts[context_id] = context @@ -106,7 +108,7 @@ def dispose(context_id) begin context.close_targets_connection - @client.command("Target.disposeBrowserContext", browserContextId: context.id) + @client.command("Target.disposeBrowserContext", timeout: @client.timeout, browserContextId: context.id) ensure @contexts.delete(context_id) end @@ -130,7 +132,7 @@ def close_connections # @raise [TimeoutError, BrowserError] def reset error = nil - context_ids = @client.command("Target.getBrowserContexts")["browserContextIds"] + context_ids = @client.command("Target.getBrowserContexts", timeout: @client.timeout)["browserContextIds"] @default_context = nil if context_ids.include?(@default_context&.id) (@contexts.keys & context_ids).each do |id| dispose(id) @@ -218,7 +220,7 @@ def rearm_auto_attach(session_id, type) return unless RECURSIVE_AUTO_ATTACH_TYPES.include?(type) client = @client.session(session_id) - client.command("Target.setAutoAttach", autoAttach: true, waitForDebuggerOnStart: true, flatten: true, async: true) + client.notify("Target.setAutoAttach", autoAttach: true, waitForDebuggerOnStart: true, flatten: true) subscribe_attached_target(client) end @@ -240,9 +242,9 @@ def handle_attach(target, session_id, params, parent) def add_extensions_and_resume(session_id, waiting) extensions = @client.options.extensions session = @client.session(session_id) - session.command("Page.enable", async: true) if extensions.any? + session.notify("Page.enable") if extensions.any? extensions.each do |source| - session.command("Page.addScriptToEvaluateOnNewDocument", async: true, source: source) + session.notify("Page.addScriptToEvaluateOnNewDocument", source: source) end resume(session_id) if waiting end @@ -275,29 +277,30 @@ def connect_worker(target) end def resume(session_id) - @client.session(session_id).command("Runtime.runIfWaitingForDebugger", async: true) + @client.session(session_id).notify("Runtime.runIfWaitingForDebugger") end def forget_session(target_id, session_id) - target = find_by(target_id: target_id)&.targets&.[](target_id) + target = find_by(target_id: target_id)&.target(target_id) target.session_id = nil if target&.session_id == session_id end def silent_detach(session_id, parent) Utils::Thread.spawn do - @client.session(session_id).command("Runtime.runIfWaitingForDebugger") - parent.command("Target.detachFromTarget", sessionId: session_id) + @client.session(session_id).command("Runtime.runIfWaitingForDebugger", timeout: Client::PROTOCOL_TIMEOUT) + parent.command("Target.detachFromTarget", timeout: Client::PROTOCOL_TIMEOUT, sessionId: session_id) rescue Error nil end end def discover - @client.command("Target.setDiscoverTargets", discover: true) + @client.command("Target.setDiscoverTargets", timeout: Client::PROTOCOL_TIMEOUT, discover: true) end def auto_attach - @client.command("Target.setAutoAttach", autoAttach: true, waitForDebuggerOnStart: true, flatten: true) + @client.command("Target.setAutoAttach", timeout: Client::PROTOCOL_TIMEOUT, autoAttach: true, + waitForDebuggerOnStart: true, flatten: true) end def add_context(context_id) @@ -320,8 +323,8 @@ def add_context(context_id) # # @return [Context, nil] def find_implicit_context - created_ids = @client.command("Target.getBrowserContexts")["browserContextIds"] - target_infos = @client.command("Target.getTargets")["targetInfos"] + created_ids = @client.command("Target.getBrowserContexts", timeout: Client::PROTOCOL_TIMEOUT)["browserContextIds"] + target_infos = @client.command("Target.getTargets", timeout: Client::PROTOCOL_TIMEOUT)["targetInfos"] implicit_id = (target_infos.filter_map { |t| t["browserContextId"] } - created_ids).first return unless implicit_id diff --git a/lib/ferrum/cookies.rb b/lib/ferrum/cookies.rb index f16ed798..2b1b8117 100644 --- a/lib/ferrum/cookies.rb +++ b/lib/ferrum/cookies.rb @@ -32,10 +32,13 @@ def initialize(client, context_id) # @return [Enumerator] # If no block is given, an Enumerator object will be returned. # + # @raise [TimeoutError] + # def each return enum_for(__method__) unless block_given? - cookies = @client.command("Storage.getCookies", browserContextId: @context_id)["cookies"] + response = @client.command("Storage.getCookies", timeout: @client.timeout, browserContextId: @context_id) + cookies = response["cookies"] cookies.each do |c| yield Cookie.new(c) @@ -78,6 +81,8 @@ def each # # @raise [BrowserError] if neither `:domain`, `:url` nor `:base_url` is set. # + # @raise [TimeoutError] + # # @example # browser.cookies.set(name: "stealth", value: "omg", domain: "google.com") # => true # @@ -113,6 +118,8 @@ def set(options) # # @return [true] # + # @raise [TimeoutError] + # # @example # browser.cookies.remove(name: "stealth", domain: "google.com") # => true # @@ -136,11 +143,13 @@ def remove(name:, **options) # # @return [true] # + # @raise [TimeoutError] + # # @example # browser.cookies.clear # => true # def clear - @client.command("Storage.clearCookies", browserContextId: @context_id) + @client.command("Storage.clearCookies", timeout: @client.timeout, browserContextId: @context_id) true end @@ -149,6 +158,8 @@ def clear # # @return [Integer] # + # @raise [TimeoutError] + # # @example # browser.cookies.store # => Integer # @@ -161,6 +172,8 @@ def store(path = "cookies.yml") # # @return [true] # + # @raise [TimeoutError] + # # @example # browser.cookies.load # => true # @@ -173,7 +186,7 @@ def load(path = "cookies.yml") private def write(cookie) - @client.command("Storage.setCookies", cookies: [cookie], browserContextId: @context_id) + @client.command("Storage.setCookies", timeout: @client.timeout, cookies: [cookie], browserContextId: @context_id) end def default_domain diff --git a/lib/ferrum/deadline.rb b/lib/ferrum/deadline.rb new file mode 100644 index 00000000..367ef35e --- /dev/null +++ b/lib/ferrum/deadline.rb @@ -0,0 +1,140 @@ +# frozen_string_literal: true + +module Ferrum + # + # A point in time by which an operation has to finish, on the monotonic + # clock. One deadline is shared by every step of an operation, so the whole + # operation is bounded by a single budget instead of each step getting its + # own. + # + # A deadline can be set for a block with {.within}, then every call made in it, by the same thread, runs under + # whatever is left of it. A call can only shorten that budget, never extend it. + # + # @example + # deadline = Ferrum::Deadline.after(10) + # deadline.remaining # => 9.99 + # deadline.expired? # => false + # + # @example + # Ferrum::Deadline.within(10) do + # page.go_to("https://example.com") + # page.cookies.to_a + # page.evaluate("document.title") + # end + # + class Deadline + THREAD_KEY = :ferrum_deadline + + # + # The deadline set by the enclosing {.within} block of this thread. + # + # @return [Deadline, nil] + # + def self.current + Thread.current.thread_variable_get(THREAD_KEY) + end + + # + # Runs the block with every Ferrum call made by this thread bound by the same budget. A block inside another + # one can only shorten it. Other threads, event callbacks for instance, don't inherit it. + # + # @param [Numeric, Deadline] timeout + # + # @return [Object] + # What the block returns. + # + # @raise [TimeoutError] + # From the calls in the block once the budget is used up. + # + def self.within(timeout) + previous = current + Thread.current.thread_variable_set(THREAD_KEY, wrap(timeout)) + yield + ensure + Thread.current.thread_variable_set(THREAD_KEY, previous) + end + + # + # Runs the block outside the budget of the enclosing {.within} block, for cleanup that has to reach the browser + # even when the budget is used up, like releasing a held key. The calls in it follow their own timeouts. + # + # @api private + # + # @return [Object] + # What the block returns. + # + def self.suspend + previous = current + Thread.current.thread_variable_set(THREAD_KEY, nil) + yield + ensure + Thread.current.thread_variable_set(THREAD_KEY, previous) + end + + # + # Starts a deadline that expires in `seconds`. + # + # @param [Numeric] seconds + # + # @return [Deadline] + # + def self.after(seconds) + new(Utils::ElapsedTime.monotonic_time + seconds) + end + + # + # Returns a {Deadline} as is, and starts one for a number of seconds. Within a {.within} block it returns + # whichever of the two expires first. + # + # @param [Deadline, Numeric] timeout + # + # @return [Deadline] + # + def self.wrap(timeout) + deadline = timeout.is_a?(self) ? timeout : after(timeout) + ambient = current + ambient ? deadline.min(ambient) : deadline + end + + # + # @param [Float] at + # Monotonic time at which the deadline expires. + # + def initialize(at) + @at = at + end + + # + # Seconds left before the deadline expires, never negative. + # + # @return [Float] + # + def remaining + [@at - Utils::ElapsedTime.monotonic_time, 0.0].max + end + + # + # The one of the two deadlines that expires first. + # + # @param [Deadline] other + # + # @return [Deadline] + # + def min(other) + at <= other.at ? self : other + end + + # + # Whether the deadline has expired. + # + # @return [Boolean] + # + def expired? + remaining.zero? + end + + protected + + attr_reader :at + end +end diff --git a/lib/ferrum/dialog.rb b/lib/ferrum/dialog.rb index 08d92305..2a5e054a 100644 --- a/lib/ferrum/dialog.rb +++ b/lib/ferrum/dialog.rb @@ -31,6 +31,8 @@ def handled? # # @param [String, nil] prompt_text # + # @raise [TimeoutError] + # # @example # browser = Ferrum::Browser.new # browser.on(:dialog) do |dialog| @@ -53,6 +55,8 @@ def accept(prompt_text = nil) # # Dismiss dialog. # + # @raise [TimeoutError] + # # @example # browser = Ferrum::Browser.new # browser.on(:dialog) do |dialog| diff --git a/lib/ferrum/downloads.rb b/lib/ferrum/downloads.rb index c3f08fcb..2fb03e48 100644 --- a/lib/ferrum/downloads.rb +++ b/lib/ferrum/downloads.rb @@ -13,7 +13,7 @@ class Downloads def initialize(page) @page = page - @event = Utils::Event.new.tap(&:set) + @idle = Concurrency::Cell.set(true) @files = {} end @@ -29,7 +29,7 @@ def files # # Waits until the current download finishes. # - # @param [Integer] timeout + # @param [Numeric, Deadline] timeout # How long to wait in seconds. # # @yield @@ -38,10 +38,11 @@ def files # @return [void] # def wait(timeout = 5) - @event.reset + deadline = Deadline.wrap(timeout) + generation = @idle.clear yield if block_given? - @event.wait(timeout) - @event.set + @idle.wait(deadline) + @idle.set(true) if generation == @idle.generation end # @@ -58,6 +59,8 @@ def wait(timeout = 5) # @raise [ArgumentError] # @raise [Ferrum::Error] # + # @raise [TimeoutError] + # def set_behavior(save_path:, behavior: :allow) raise ArgumentError unless VALID_BEHAVIOR.include?(behavior.to_sym) raise Error, "supply absolute path for `:save_path` option" unless Pathname.new(save_path.to_s).absolute? @@ -87,7 +90,7 @@ def subscribe # def subscribe_download_will_begin @page.on("Browser.downloadWillBegin") do |params| - @event.reset + @idle.clear @files[params["guid"]] = params end end @@ -103,9 +106,9 @@ def subscribe_download_progress case params["state"] when "completed", "canceled" - @event.set + @idle.set(true) else - @event.reset + @idle.clear end end end diff --git a/lib/ferrum/errors.rb b/lib/ferrum/errors.rb index d2c47262..e7f35ffc 100644 --- a/lib/ferrum/errors.rb +++ b/lib/ferrum/errors.rb @@ -74,8 +74,7 @@ def message "Timed out waiting for #{awaited}. It's possible that this happened " \ "because something took a very long time (for example a page load " \ "was slow). If so, setting the :timeout option to a higher value might " \ - "help. If this happened on an internal protocol call instead, try " \ - "raising :protocol_timeout." + "help." end private diff --git a/lib/ferrum/frame.rb b/lib/ferrum/frame.rb index affa06a1..f971f5ce 100644 --- a/lib/ferrum/frame.rb +++ b/lib/ferrum/frame.rb @@ -62,7 +62,7 @@ def initialize(id, page, parent_id = nil) @page = page @parent_id = parent_id @lifecycle_events = [] - @execution_id = Concurrent::MVar.new + @execution_id = Concurrency::Cell.new @isolated_worlds = Concurrent::Map.new end @@ -86,6 +86,8 @@ def state=(value) # # @return [String] # + # @raise [TimeoutError] + # # @example # browser.go_to("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe") # frame = browser.frames[1] @@ -100,6 +102,8 @@ def url # # @return [String] # + # @raise [TimeoutError] + # # @example # browser.go_to("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe") # frame = browser.frames[1] @@ -159,6 +163,8 @@ def parent # # @param [String] html # + # @raise [TimeoutError] + # # @example # browser.go_to("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe") # frame = browser.frames[1] @@ -182,15 +188,16 @@ def content=(html) # an error if an execution id hasn't been set yet, if id is set # returns immediately. # + # @param [Numeric, Deadline, nil] timeout + # How long to wait for the id, defaults to the page's `timeout`. + # # @return [Integer] # # @raise [NoExecutionContextError] # - def execution_id! - value = @execution_id.borrow(@page.timeout, &:itself) - raise NoExecutionContextError if value.instance_of?(Object) - - value + def execution_id!(timeout = nil) + @execution_id.wait(timeout || @page.timeout) + @execution_id.value || raise(NoExecutionContextError) end # @@ -200,10 +207,7 @@ def execution_id! # @return [Integer, nil] # def execution_id - value = @execution_id.value - return if value.instance_of?(Object) - - value + @execution_id.value end # @@ -217,9 +221,9 @@ def execution_id # def execution_id=(value) if value.nil? - @execution_id.try_take! - else - @execution_id.try_put!(value) + @execution_id.clear + elsif !@execution_id.set? + @execution_id.set(value) end end @@ -264,7 +268,7 @@ def inspect "@loader_id=#{@loader_id.inspect} " \ "@lifecycle_events=#{@lifecycle_events.inspect} " \ "@state=#{@state.inspect} " \ - "@execution_id=#{@execution_id.inspect}>" + "@execution_id=#{execution_id.inspect}>" end end end diff --git a/lib/ferrum/frame/dom.rb b/lib/ferrum/frame/dom.rb index 5b53e031..088d74c0 100644 --- a/lib/ferrum/frame/dom.rb +++ b/lib/ferrum/frame/dom.rb @@ -70,6 +70,8 @@ module DOM # @return [String] # The window's current URL. # + # @raise [TimeoutError] + # def current_url Utils::Deprecate.warn( "#current_url will be removed in Ferrum 0.22. Use page.url, or frame.page.url from a frame, " \ @@ -87,6 +89,8 @@ def current_url # @return [String] # The window's current title. # + # @raise [TimeoutError] + # def current_title Utils::Deprecate.warn( "#current_title will be removed in Ferrum 0.22. Use page.title, or frame.page.title from a frame, " \ @@ -102,6 +106,8 @@ def current_title # The serialized `` declaration, or +nil+ if the # document has none. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://example.com") # browser.doctype # => "" @@ -116,6 +122,8 @@ def doctype # @return [String] # The HTML source of the current page. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://google.com/") # browser.body # => '... @@ -131,6 +139,8 @@ def body # The element in which the window is embedded, `nil` for the main frame # and for a frame from a different origin than its parent. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe") # frame = browser.frames.last @@ -153,6 +163,8 @@ def frame_element # @return [Array] # The matching nodes. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://github.com/") # browser.xpath("//a[@aria-label='Issues you created']") # => [Node] @@ -187,6 +199,8 @@ def xpath(selector, within: nil) # @return [Node, nil] # The matching node. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://github.com/") # browser.at_xpath("//a[@aria-label='Issues you created']") # => Node @@ -214,6 +228,8 @@ def at_xpath(selector, within: nil) # @return [Array] # The matching nodes. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://github.com/") # browser.css("a[aria-label='Issues you created']") # => [Node] @@ -241,6 +257,8 @@ def css(selector, within: nil) # @return [Node, nil] # The matching node. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://github.com/") # browser.at_css("a[aria-label='Issues you created']") # => Node @@ -271,7 +289,7 @@ def at_css(selector, within: nil) # @param [Node, nil] within # The parent node to search within. # - # @param [Float, Integer] timeout + # @param [Numeric, Deadline] timeout # How long to keep retrying, in seconds. # # @param [Float, Integer] interval @@ -283,6 +301,8 @@ def at_css(selector, within: nil) # @raise [ArgumentError] # If neither or both of `css`/`xpath` are given. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://github.com/") # browser.wait_for_selector(css: "a[aria-label='Issues you created']") # => Node @@ -293,15 +313,17 @@ def wait_for_selector(css: nil, xpath: nil, within: nil, timeout: @page.timeout, selector, method_name = css ? [css, :at_css] : [xpath, :at_xpath] raise ArgumentError, "pass either `css:` or `xpath:`" unless selector - start = Utils::ElapsedTime.monotonic_time + deadline = Deadline.wrap(timeout) - until (node = send(method_name, selector, within: within)) - return nil if Utils::ElapsedTime.timeout?(start, timeout) + Deadline.within(deadline) do + until (node = poll_selector(method_name, selector, within, deadline)) + return nil if deadline.expired? - sleep(interval) - end + sleep(interval) + end - node + node + end end # @@ -315,6 +337,8 @@ def wait_for_selector(css: nil, xpath: nil, within: nil, timeout: @page.timeout, # # @param [String] type # + # @raise [TimeoutError] + # # @example # browser.add_script_tag(url: "http://example.com/stylesheet.css") # => true # @@ -341,6 +365,8 @@ def add_script_tag(url: nil, path: nil, content: nil, type: "text/javascript") # # @param [String, nil] content # + # @raise [TimeoutError] + # # @example # browser.add_style_tag(content: "h1 { font-size: 40px; }") # => true # @@ -357,6 +383,16 @@ def add_style_tag(url: nil, path: nil, content: nil) true end + + private + + def poll_selector(method_name, selector, within, deadline) + send(method_name, selector, within: within) + rescue TimeoutError + raise unless deadline.expired? + + nil + end end end end diff --git a/lib/ferrum/frame/runtime.rb b/lib/ferrum/frame/runtime.rb index 323bd5e1..1d323fd3 100644 --- a/lib/ferrum/frame/runtime.rb +++ b/lib/ferrum/frame/runtime.rb @@ -203,6 +203,8 @@ def evaluate(expression, *positional, **keywords) # # @return [RemoteObject, Node, Object] # + # @raise [TimeoutError] + # # @example # list = page.evaluate_handle("document.querySelectorAll('li')") # page.evaluate("Array.from(nodes).map(n => n.textContent)", nodes: list) @@ -222,6 +224,8 @@ def evaluate_handle(expression, *positional, **keywords) # @return [Boolean] # Always `true`. # + # @raise [TimeoutError] + # # @example # browser.execute("window.scrollBy(0, 100)") # => true # browser.execute(<<~JS, url: "/next") @@ -249,6 +253,8 @@ def execute(expression, *positional, **keywords) # # @return [Object] # + # @raise [TimeoutError] + # def evaluate_in(object, expression, positional, keywords, mode: :value) run(expression, positional, keywords, mode: mode, on: object) end @@ -265,7 +271,8 @@ def run(expression, positional, keywords, mode:, on: nil) script = Script.new(expression) named = keywords.except(:timeout) params, values = arguments_for(script, positional, named) - call(wrap(script, params, mode), values, on: on, mode: mode, timeout: script_timeout(keywords[:timeout])) + deadline = script_deadline(keywords[:timeout]) + Deadline.within(deadline) { call(wrap(script, params, mode), values, on: on, mode: mode, deadline: deadline) } end # @@ -323,8 +330,8 @@ def wrap(script, params, mode) # Waiting for the response is what bounds the script, so a page that # replaces `Promise` or `setTimeout`, or throttles timers, can't defeat # it. - def script_timeout(timeout) - (timeout || @page.timeout).to_f + def script_deadline(timeout) + Deadline.wrap(timeout || @page.timeout) end # @@ -336,18 +343,18 @@ def script_timeout(timeout) # `:value` resolves the result into Ruby, `:handle` returns a # {RemoteObject}, and `:none` discards it. # - def call(declaration, values, on:, mode:, timeout:) + def call(declaration, values, on:, mode:, deadline:) # do not rescue -> retry if we operate on an existing node errors = on ? [] : [NodeNotFoundError, NoExecutionContextError] serialization = mode == :value ? DEEP_SERIALIZATION : SHALLOW_SERIALIZATION response = Utils::Attempt.with_retry(errors: errors, max: INTERMITTENT_ATTEMPTS, wait: INTERMITTENT_SLEEP) do - call_function(declaration, values, call_target(on), serialization, timeout) + call_function(declaration, values, call_target(on, deadline), serialization, deadline) end handle_error(response, declaration) result = response["result"] - result = await_promise(result, serialization, timeout) if thenable?(result) + result = await_promise(result, serialization, deadline) if thenable?(result) case mode when :none then nil when :handle then handle_remote_object(result) @@ -355,9 +362,9 @@ def call(declaration, values, on:, mode:, timeout:) end end - def call_function(declaration, values, target, serialization, timeout) + def call_function(declaration, values, target, serialization, deadline) @page.command("Runtime.callFunctionOn", - timeout: timeout, slowmoable: true, serializationOptions: serialization, + timeout: deadline, slowmoable: true, serializationOptions: serialization, functionDeclaration: declaration, arguments: prepare_args(values), **target) rescue TimeoutError raise ScriptTimeoutError @@ -374,9 +381,9 @@ def thenable?(result) # Awaited outside the retries: by now the script has run, and running # it again after a navigation would repeat what it did. Awaiting an # object that isn't thenable returns it as it is. - def await_promise(promise, serialization, timeout) + def await_promise(promise, serialization, deadline) response = @page.command("Runtime.callFunctionOn", - timeout: timeout, awaitPromise: true, serializationOptions: serialization, + timeout: deadline, awaitPromise: true, serializationOptions: serialization, functionDeclaration: "function() { return this }", objectId: promise["objectId"]) handle_error(response) response["result"] @@ -392,13 +399,13 @@ def raise_deep_error(error) raise ResultTooDeepError, error.response end - def call_target(on) + def call_target(on, deadline) if on.is_a?(RemoteObject) { objectId: on.remote_id } elsif on { objectId: resolve_node(on) } else - { executionContextId: execution_id! } + { executionContextId: execution_id!(deadline) } end end diff --git a/lib/ferrum/headers.rb b/lib/ferrum/headers.rb index eb6428b5..c63367f0 100644 --- a/lib/ferrum/headers.rb +++ b/lib/ferrum/headers.rb @@ -33,6 +33,8 @@ def get # # @return [true] # + # @raise [TimeoutError] + # def set(headers) reset_overrides(keep: headers) @headers = {} @@ -44,6 +46,8 @@ def set(headers) # # @return [true] # + # @raise [TimeoutError] + # def clear reset_overrides @headers = {} @@ -61,6 +65,8 @@ def clear # # @return [true] # + # @raise [TimeoutError] + # def add(headers, permanent: true) if headers["Referer"] @page.referrer = headers["Referer"] diff --git a/lib/ferrum/isolated_world.rb b/lib/ferrum/isolated_world.rb index 08fefcdf..a32fc937 100644 --- a/lib/ferrum/isolated_world.rb +++ b/lib/ferrum/isolated_world.rb @@ -31,7 +31,7 @@ def initialize(frame, name) @frame = frame @page = frame.page @name = name - @execution_id = Concurrent::MVar.new + @execution_id = Concurrency::Cell.new end # @@ -40,9 +40,10 @@ def initialize(frame, name) # # @return [Integer] # - def execution_id! + def execution_id!(timeout = nil) execution_id || begin - id = @page.command("Page.createIsolatedWorld", frameId: @frame.id, worldName: @name)["executionContextId"] + response = @page.command("Page.createIsolatedWorld", timeout: timeout, frameId: @frame.id, worldName: @name) + id = response["executionContextId"] self.execution_id = id execution_id || id end @@ -54,10 +55,7 @@ def execution_id! # @return [Integer, nil] # def execution_id - value = @execution_id.value - return if value.instance_of?(Object) - - value + @execution_id.value end # @@ -70,9 +68,9 @@ def execution_id # def execution_id=(value) if value.nil? - @execution_id.try_take! - else - @execution_id.try_put!(value) + @execution_id.clear + elsif !@execution_id.set? + @execution_id.set(value) end end diff --git a/lib/ferrum/keyboard.rb b/lib/ferrum/keyboard.rb index 63d386c0..b352633e 100644 --- a/lib/ferrum/keyboard.rb +++ b/lib/ferrum/keyboard.rb @@ -53,6 +53,8 @@ def initialize(page) # # @raise [ArgumentError] If the key is a modifier, use {#hold} instead. # + # @raise [TimeoutError] + # def down(key) reject_modifier(key) key = normalize_keys(Array(key)).first @@ -71,6 +73,8 @@ def down(key) # # @raise [ArgumentError] If the key is a modifier, use {#hold} instead. # + # @raise [TimeoutError] + # def up(key) reject_modifier(key) key = normalize_keys(Array(key)).first @@ -85,15 +89,26 @@ def up(key) # @param [Array] keys # The text to type into a focused element, `[:Shift, "s"], "tring"`. # + # @param [Numeric, Deadline, nil] timeout + # How long typing all the keys may take together, the page timeout by default. Raise it for a long text. + # # @return [self] # - def type(*keys) + # @raise [TimeoutError] + # + # @example + # page.keyboard.type("Hello World") + # page.keyboard.type(long_text, timeout: 30) + # + def type(*keys, timeout: nil) keys = normalize_keys(Array(keys)) - keys.each do |key| - type = key[:text] ? "keyDown" : "rawKeyDown" - @page.command("Input.dispatchKeyEvent", type: type, **key) - @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key) + Deadline.within(timeout || @page.timeout) do + keys.each do |key| + type = key[:text] ? "keyDown" : "rawKeyDown" + @page.command("Input.dispatchKeyEvent", type: type, **key) + @page.command("Input.dispatchKeyEvent", slowmoable: true, type: "keyUp", **key) + end end self @@ -112,6 +127,8 @@ def type(*keys) # # @raise [ArgumentError] If no block is given or a key isn't a modifier. # + # @raise [TimeoutError] + # # @example # page.keyboard.hold(:shift) do # page.mouse.move(x: 10, y: 10).down.move(x: 200, y: 200, steps: 10).up @@ -174,7 +191,7 @@ def release_modifiers(pressed, quietly: false) while (name = pressed.pop) @held_modifiers &= ~MODIFIERS[name] begin - dispatch_modifier("keyUp", name) + Deadline.suspend { dispatch_modifier("keyUp", name) } rescue Error raise unless quietly end diff --git a/lib/ferrum/mouse.rb b/lib/ferrum/mouse.rb index eeea806a..31f6f343 100644 --- a/lib/ferrum/mouse.rb +++ b/lib/ferrum/mouse.rb @@ -8,7 +8,6 @@ module Ferrum # can be composed while reporting consistent state to Chrome. # class Mouse - CLICK_WAIT = ENV.fetch("FERRUM_CLICK_WAIT", 0.1).to_f BUTTON_MASKS = { "none" => 0, "left" => 1, @@ -34,6 +33,8 @@ def initialize(page) # @param [Integer] y # The vertical pixel value that you want to scroll by. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://www.google.com/search?q=Ruby+headless+driver+for+Capybara") # browser.mouse.scroll_by(0, 400) @@ -54,6 +55,8 @@ def scroll_by(x, y) # The pixel along the vertical axis of the document that you want # displayed in the upper left. # + # @raise [TimeoutError] + # # @example # browser.go_to("https://www.google.com/search?q=Ruby+headless+driver+for+Capybara") # browser.mouse.scroll_to(0, 400) @@ -72,10 +75,16 @@ def scroll_to(x, y) # @param [Float] delay # Delay between mouse down and mouse up events. # - # @param [Numeric] wait + # @param [Numeric] navigation_wait # Seconds to watch for a navigation started by the click, then wait for it # to load. `0` returns as soon as the click is dispatched. # + # @param [Numeric, nil] wait + # Deprecated, renamed to `navigation_wait`. + # + # @param [Numeric, Deadline, nil] timeout + # How long the click and the page load it starts may take together, the page timeout by default. + # # @param [Hash{Symbol => Object}] options # Additional keyword arguments. # @@ -90,13 +99,19 @@ def scroll_to(x, y) # # @return [self] # - def click(x:, y:, delay: 0, wait: CLICK_WAIT, **options) - move(x: x, y: y, modifiers: options[:modifiers]) - down(**options) - sleep(delay) - # Potential wait because if some network event is triggered then we have - # to wait until it's over and frame is loaded or failed to load. - up(wait: wait, **options) + # @raise [TimeoutError] + # + def click(x:, y:, delay: 0, navigation_wait: Page::NAVIGATION_WAIT, wait: nil, timeout: nil, **options) # rubocop:disable Metrics/ParameterLists + navigation_wait = Utils::Deprecate.renamed_keyword(:wait, :navigation_wait, wait, navigation_wait) + deadline = Deadline.wrap(timeout || @page.timeout) + Deadline.within(deadline) do + move(x: x, y: y, modifiers: options[:modifiers]) + down(**options) + sleep(delay) + # Potential wait because if some network event is triggered then we have + # to wait until it's over and frame is loaded or failed to load. + up(navigation_wait: navigation_wait, timeout: deadline, **options) + end self end @@ -117,6 +132,8 @@ def click(x:, y:, delay: 0, wait: CLICK_WAIT, **options) # # @return [self] # + # @raise [TimeoutError] + # def down(**options) tap { mouse_event(type: "mousePressed", **options) } end @@ -136,8 +153,19 @@ def down(**options) # Bitfield for key modifiers. See `keyboard.modifiers`. # If omitted, uses the keys held by `keyboard.hold`, or none outside of it. # + # @option options [Numeric] :navigation_wait (0) + # Seconds to watch for a navigation started by releasing the button, then wait for it to load. + # + # @option options [Numeric] :wait + # Deprecated, renamed to `:navigation_wait`. + # + # @option options [Numeric, Deadline] :timeout + # How long the event and the page load it starts may take together, the page timeout by default. + # # @return [self] # + # @raise [TimeoutError] + # def up(**options) tap { mouse_event(type: "mouseReleased", **options) } end @@ -156,27 +184,34 @@ def up(**options) # Bitfield for key modifiers. See `keyboard.modifiers`. # If omitted, uses the keys held by `keyboard.hold`, or none outside of it. # + # @param [Numeric, Deadline, nil] timeout + # How long all the steps may take together, the page timeout by default. + # # @return [self] # - def move(x:, y:, steps: 1, modifiers: nil) + # @raise [TimeoutError] + # + def move(x:, y:, steps: 1, modifiers: nil, timeout: nil) from_x = @x from_y = @y @x = x @y = y button = pressed_button - steps.times do |i| - new_x = from_x + ((@x - from_x) * ((i + 1) / steps.to_f)) - new_y = from_y + ((@y - from_y) * ((i + 1) / steps.to_f)) - - @page.command("Input.dispatchMouseEvent", - slowmoable: true, - type: "mouseMoved", - x: new_x, - y: new_y, - button: button, - buttons: @buttons, - modifiers: modifiers || @page.keyboard.held_modifiers) + Deadline.within(timeout || @page.timeout) do + steps.times do |i| + new_x = from_x + ((@x - from_x) * ((i + 1) / steps.to_f)) + new_y = from_y + ((@y - from_y) * ((i + 1) / steps.to_f)) + + @page.command("Input.dispatchMouseEvent", + slowmoable: true, + type: "mouseMoved", + x: new_x, + y: new_y, + button: button, + buttons: @buttons, + modifiers: modifiers || @page.keyboard.held_modifiers) + end end self @@ -184,12 +219,21 @@ def move(x:, y:, steps: 1, modifiers: nil) private - def mouse_event(type:, button: :left, count: 1, modifiers: nil, wait: 0) + def mouse_event(type:, button: :left, count: 1, modifiers: nil, navigation_wait: 0, wait: nil, # rubocop:disable Metrics/ParameterLists + timeout: nil) + navigation_wait = Utils::Deprecate.renamed_keyword(:wait, :navigation_wait, wait, navigation_wait) button = validate_button(button) register_event_button(type, button) options = { x: @x, y: @y, type: type, button: button, buttons: @buttons, clickCount: count, modifiers: modifiers || @page.keyboard.held_modifiers } - @page.command("Input.dispatchMouseEvent", wait: wait, slowmoable: true, **options) + return @page.command("Input.dispatchMouseEvent", slowmoable: true, **options) unless navigation_wait.positive? + + deadline = Deadline.wrap(timeout || @page.timeout) + result = nil + @page.wait_for_navigation!(navigation_wait, timeout: deadline) do + result = @page.command("Input.dispatchMouseEvent", slowmoable: true, timeout: deadline, **options) + end + result end def validate_button(button) diff --git a/lib/ferrum/network.rb b/lib/ferrum/network.rb index 90c5462e..87679a66 100644 --- a/lib/ferrum/network.rb +++ b/lib/ferrum/network.rb @@ -62,7 +62,7 @@ def initialize(page) # @param [Float] duration # Sleep for given amount of time and check again. # - # @param [Float] timeout + # @param [Numeric, Deadline] timeout # During what time we try to check idle. # # @return [Boolean] @@ -73,10 +73,10 @@ def initialize(page) # browser.network.wait_for_idle # => false # def wait_for_idle(connections: 0, duration: 0.05, timeout: @page.timeout) - start = Utils::ElapsedTime.monotonic_time + deadline = Deadline.wrap(timeout) until idle?(connections) - return false if Utils::ElapsedTime.timeout?(start, timeout) + return false if deadline.expired? sleep(duration) end @@ -185,6 +185,8 @@ def status # # @return [true] # + # @raise [TimeoutError] + # # @example # traffic = browser.network.traffic # => [] # browser.go_to("https://github.com/") @@ -213,6 +215,8 @@ def clear(type) # One or more patterns matched against the request's URL, see # {InterceptedRequest#match?}. # + # @raise [TimeoutError] + # # @example # browser.network.blacklist = /jquery/ # browser.go_to("https://example.com/") @@ -232,6 +236,8 @@ def blacklist=(patterns) # One or more patterns matched against the request's URL, see # {InterceptedRequest#match?}. # + # @raise [TimeoutError] + # # @example # browser.network.whitelist = /example/ # browser.go_to("https://example.com/") @@ -252,6 +258,8 @@ def whitelist=(patterns) # @param [Symbol, nil] resource_type # One of the [resource types](https://chromedevtools.github.io/devtools-protocol/tot/Network#type-ResourceType) # + # @raise [TimeoutError] + # # @example # browser = Ferrum::Browser.new # browser.network.intercept @@ -301,6 +309,8 @@ def intercept(pattern: "*", resource_type: nil, request_stage: nil, handle_auth_ # @yieldparam [Request] request # An HTTP request. # + # @raise [TimeoutError] + # # @example # browser.network.authorize(user: "login", password: "pass") { |req| req.continue } # browser.go_to("http://example.com/authenticated") @@ -396,7 +406,7 @@ def build_exchange(id) end # `Network.requestWillBeSent` and `Fetch.requestPaused` are handled on - # separate threads (see `Client::Subscriber`), so the "find the existing + # separate threads (see `Concurrency::Dispatcher`), so the "find the existing # exchange for this id or build a new one" check has to be atomic, # otherwise both threads can race past the `select` before either has # appended, and end up building two exchanges for the same request. @@ -433,6 +443,8 @@ def find_or_build_exchange(id) # * `"wimax"` # * `"other"` # + # @raise [TimeoutError] + # # @example # browser.network.emulate_network_conditions(connection_type: "cellular2g") # browser.go_to("https://github.com/") @@ -455,6 +467,8 @@ def emulate_network_conditions(offline: false, latency: 0, # # Activates offline mode for a page. # + # @raise [TimeoutError] + # # @example # browser.network.offline_mode # browser.go_to("https://github.com/") @@ -502,6 +516,8 @@ def offline_mode # @raise [ArgumentError] on any other option, or a `:metadata` that is # missing a key or has an unknown one. # + # @raise [TimeoutError] + # # @example # browser.network.emulate(platform: "Win32", accept_language: "de-DE") # browser.go_to("https://github.com/") @@ -532,6 +548,8 @@ def emulate(**overrides) # # Toggles ignoring cache for each request. If true, cache will not be used. # + # @raise [TimeoutError] + # # @example # browser.network.cache(disable: true) # diff --git a/lib/ferrum/network/intercepted_request.rb b/lib/ferrum/network/intercepted_request.rb index 0bf4e11a..9dea0470 100644 --- a/lib/ferrum/network/intercepted_request.rb +++ b/lib/ferrum/network/intercepted_request.rb @@ -101,7 +101,7 @@ def respond(**options) options = options.merge(body: Base64.strict_encode64(options.fetch(:body, ""))) if has_body @status = :responded - @client.command("Fetch.fulfillRequest", async: true, **options) + @client.notify("Fetch.fulfillRequest", **options) end # @@ -128,7 +128,7 @@ def respond(**options) def continue(**options) options = options.merge(requestId: request_id) @status = :continued - @client.command("Fetch.continueRequest", async: true, **options) + @client.notify("Fetch.continueRequest", **options) end # @@ -139,7 +139,7 @@ def continue(**options) # def abort @status = :aborted - @client.command("Fetch.failRequest", async: true, requestId: request_id, errorReason: "BlockedByClient") + @client.notify("Fetch.failRequest", requestId: request_id, errorReason: "BlockedByClient") end # diff --git a/lib/ferrum/network/response.rb b/lib/ferrum/network/response.rb index 0f48596e..1dba5139 100644 --- a/lib/ferrum/network/response.rb +++ b/lib/ferrum/network/response.rb @@ -126,6 +126,8 @@ def body_size=(size) # # @raise [Ferrum::BrowserError] # + # @raise [TimeoutError] + # def body! @body ||= begin body, encoded = @page.command("Network.getResponseBody", requestId: id) @@ -139,6 +141,8 @@ def body! # # @return [String, nil] # + # @raise [TimeoutError] + # def body body! rescue Ferrum::BrowserError diff --git a/lib/ferrum/node.rb b/lib/ferrum/node.rb index aebc8985..1347e9e4 100644 --- a/lib/ferrum/node.rb +++ b/lib/ferrum/node.rb @@ -54,6 +54,8 @@ def initialize(frame, properties) # # @return [Hash{String => Object}] # + # @raise [TimeoutError] + # # @example # page.at_css("a").description["nodeName"] # => "A" # @@ -67,6 +69,8 @@ def description # # @return [String] # + # @raise [TimeoutError] + # # @example # page.at_css("#nav").tag_name # => "div" # @@ -100,6 +104,8 @@ def node_id # # @return [Boolean] # + # @raise [TimeoutError] + # # @example # page.at_css("a").node? # => true # page.xpath("//a/text()").first.node? # => false @@ -115,6 +121,8 @@ def node? # # @return [Frame, nil] # + # @raise [TimeoutError] + # # @example # frame = page.at_xpath("//iframe").frame # => Frame # frame.at_css("//a[text() = 'Log in']") # => Node @@ -128,6 +136,8 @@ def frame # # @return [self] # + # @raise [TimeoutError] + # # @example # input = page.at_css("input[name='q']") # input.focus @@ -140,6 +150,8 @@ def focus # find out, and leaves it focused when it can. # # @return [Boolean] + # + # @raise [TimeoutError] def focusable? focus true @@ -164,6 +176,8 @@ def focusable? # # @raise [NodeMovingError] # + # @raise [TimeoutError] + # # @example # page.at_css("#modal").wait_for_stop_moving # # => [[50.2, 390.9, 203.2, 390.9, 203.2, 411.9, 50.2, 411.9]] @@ -186,6 +200,8 @@ def wait_for_stop_moving(delay: MOVING_WAIT_DELAY, attempts: MOVING_WAIT_ATTEMPT # # @return [Boolean] # + # @raise [TimeoutError] + # # @example # page.at_css(".animated").moving? # => true # page.at_css("#nav").moving? # => false @@ -200,6 +216,8 @@ def moving?(delay: MOVING_CHECK_DELAY) # # @return [self] # + # @raise [TimeoutError] + # # @example # page.at_css("input[name='q']").focus.blur # @@ -214,13 +232,19 @@ def blur # @param [Array] keys # The keys to type, e.g. `"Input"`, `[:Shift, "s"], "tring"`. # + # @param [Numeric, Deadline, nil] timeout + # How long typing all the keys may take together, the page timeout by default. + # # @return [self] # + # @raise [TimeoutError] + # # @example # input.focus.type("Input") + # input.focus.type(long_text, timeout: 30) # - def type(*keys) - tap { page.keyboard.type(*keys) } + def type(*keys, timeout: nil) + tap { page.keyboard.type(*keys, timeout: timeout) } end # @@ -249,34 +273,49 @@ def type(*keys) # @param [Float] delay # Delay between mouse down and mouse up events. # - # @param [Numeric] wait + # @param [Numeric] navigation_wait # Seconds to watch for a navigation started by a left click, then wait for # it to load. Ignored for `:right` and `:double`, which never wait. `0` # returns as soon as the click is dispatched. # + # @param [Numeric, nil] wait + # Deprecated, renamed to `navigation_wait`. + # + # @param [Numeric, Deadline, nil] timeout + # How long finding the node's position, the click and the page load it starts may take together, the page + # timeout by default. + # # @return [self] # + # @raise [TimeoutError] + # # @example # page.at_css("a").click - # page.at_css("a").click(wait: 0) + # page.at_css("a").click(timeout: 30) + # page.at_css("a").click(navigation_wait: 0) # page.at_css("a").click(mode: :right) # page.at_css("a").click(keys: [:ctrl, :shift]) # page.at_css("canvas").click(offset: { x: 10, y: 20 }) # page.at_css("canvas").click(offset: { x: 10, y: 20, position: :center }) # - def click(mode: :left, keys: [], offset: {}, delay: 0, wait: Mouse::CLICK_WAIT) - x, y = find_position(**offset) - modifiers = page.keyboard.modifiers(keys) - - # `:right` and `:double` pass `wait: 0` to preserve the historical - # no-network-wait default of `Mouse#up` and `Mouse#down` - case mode - when :right - page.mouse.click(x:, y:, modifiers:, delay:, button: :right, wait: 0) - when :double - page.mouse.click(x:, y:, modifiers:, delay:, count: 2, wait: 0) - when :left - page.mouse.click(x:, y:, modifiers:, delay:, wait:) + def click(mode: :left, keys: [], offset: {}, delay: 0, navigation_wait: Page::NAVIGATION_WAIT, wait: nil, # rubocop:disable Metrics/ParameterLists + timeout: nil) + navigation_wait = Utils::Deprecate.renamed_keyword(:wait, :navigation_wait, wait, navigation_wait) + deadline = Deadline.wrap(timeout || page.timeout) + Deadline.within(deadline) do + x, y = find_position(**offset) + modifiers = page.keyboard.modifiers(keys) + + # `:right` and `:double` pass `navigation_wait: 0` to preserve the historical + # no-network-wait default of `Mouse#up` and `Mouse#down` + case mode + when :right + page.mouse.click(x:, y:, modifiers:, delay:, button: :right, navigation_wait: 0, timeout: deadline) + when :double + page.mouse.click(x:, y:, modifiers:, delay:, count: 2, navigation_wait: 0, timeout: deadline) + when :left + page.mouse.click(x:, y:, modifiers:, delay:, navigation_wait:, timeout: deadline) + end end self @@ -285,6 +324,8 @@ def click(mode: :left, keys: [], offset: {}, delay: 0, wait: Mouse::CLICK_WAIT) # Not currently implemented. # # @raise [Ferrum::NotImplementedError] always + # + # @raise [TimeoutError] def hover raise Ferrum::NotImplementedError end @@ -294,6 +335,8 @@ def hover # # @return [self] # + # @raise [TimeoutError] + # # @example # page.at_css("#footer").scroll_into_view # @@ -309,6 +352,8 @@ def scroll_into_view # # @return [Boolean] # + # @raise [TimeoutError] + # # @example # page.at_css("#footer").in_viewport? # => false # page.at_css("#footer").scroll_into_view.in_viewport? # => true @@ -337,6 +382,8 @@ def in_viewport?(of: nil) # # @return [Hash{String => Object}] # + # @raise [TimeoutError] + # # @example # page.at_css("input[type=file]").select_file("/path/to/file.png") # @@ -357,6 +404,8 @@ def select_file(value) # # @return [Node, nil] # + # @raise [TimeoutError] + # # @example # page.at_xpath("//nav").at_xpath(".//a") # => Node # @@ -372,6 +421,8 @@ def at_xpath(selector) # # @return [Node, nil] # + # @raise [TimeoutError] + # # @example # page.at_css("form").at_css("input[name='q']") # => Node # @@ -387,6 +438,8 @@ def at_css(selector) # # @return [Array] # + # @raise [TimeoutError] + # # @example # page.at_css("ul").xpath(".//li") # => [Node] # @@ -402,6 +455,8 @@ def xpath(selector) # # @return [Array] # + # @raise [TimeoutError] + # # @example # page.at_css("ul").css("li") # => [Node] # @@ -414,6 +469,8 @@ def css(selector) # # @return [String] # + # @raise [TimeoutError] + # # @example # page.at_css("a > h3").text # => "rubycdp/ferrum: Ruby Chrome/Chromium driver - GitHub" # @@ -427,6 +484,8 @@ def text # # @return [String] # + # @raise [TimeoutError] + # # @example # #

Hello hiddenworld

# page.at_css("p").text # => "Hello hiddenworld" @@ -442,6 +501,8 @@ def inner_text # # @return [Object] # + # @raise [TimeoutError] + # # @example # page.at_css("input[name='q']").value # => "ferrum" # @@ -456,6 +517,8 @@ def value # # @return [Object] # + # @raise [TimeoutError] + # # @example # page.at_css("input").property("value") # => "Foo" # @@ -473,6 +536,8 @@ def property(name) # # @return [String, nil] # + # @raise [TimeoutError] + # # @example # page.at_css("input").attribute("value") # => "Foo" # @@ -487,6 +552,8 @@ def attribute(name) # # @raise [JavaScriptError] when the node is not a `select`. # + # @raise [TimeoutError] + # # @example # page.at_css("select#title").selected.map(&:text) # => ["Mrs"] # @@ -515,6 +582,8 @@ def selected # # @return [self] # + # @raise [TimeoutError] + # # @example # page.at_xpath("//*[select]").select(["1"]) # => Node (select) # page.at_xpath("//*[select]").select(["text"], by: :text) # => Node (select) @@ -555,6 +624,8 @@ def select(*values, by: :value) # # @return [Object] # + # @raise [TimeoutError] + # # @example # page.at_css("input").evaluate("this.value") # page.at_css("input").evaluate("this.value + suffix", suffix: "!") @@ -573,6 +644,8 @@ def evaluate(expression, *positional, **keywords) # @return [RemoteObject, Node, Object] # A {Node} for a node and the value itself for a primitive. # + # @raise [TimeoutError] + # # @example # page.at_css("form").evaluate_handle("this.dataset") # => RemoteObject # page.at_css("form").evaluate_handle("this") # => Node @@ -592,6 +665,8 @@ def evaluate_handle(expression, *positional, **keywords) # @return [Boolean] # Always `true`. # + # @raise [TimeoutError] + # # @example # page.at_css("input").execute("this.value = text", text: "hello") # @@ -642,6 +717,8 @@ def inspect # # @return [(Float, Float)] # + # @raise [TimeoutError] + # # @example # page.at_css("input").find_position # => [126.7, 401.4] # page.at_css("input").find_position(x: 5, y: 5) # => [55.2, 395.9] @@ -662,6 +739,8 @@ def find_position(x: nil, y: nil, position: :top) # # @return [Hash{String => String}] # + # @raise [TimeoutError] + # # @example # page.at_css("a").computed_style["display"] # => "inline" # @@ -676,6 +755,8 @@ def computed_style # # @return [Accessibility::AXNode, nil] # + # @raise [TimeoutError] + # # @example # page.at_css("a").axnode.role # => "link" # @@ -688,6 +769,8 @@ def axnode # # @return [Hash{String => Object}] # + # @raise [TimeoutError] + # # @example # page.at_css("#ad").remove # @@ -699,6 +782,8 @@ def remove # # @return [Boolean] # + # @raise [TimeoutError] + # # @example # node = page.at_css("#ad") # node.remove diff --git a/lib/ferrum/page.rb b/lib/ferrum/page.rb index 884f34ee..e6b95470 100644 --- a/lib/ferrum/page.rb +++ b/lib/ferrum/page.rb @@ -29,7 +29,9 @@ module Ferrum # JavaScript evaluation methods are ultimately delegated to from {Browser}. # class Page - GOTO_WAIT = ENV.fetch("FERRUM_GOTO_WAIT", 0.1).to_f + NAVIGATION_WAIT = Utils::Deprecate.renamed_env( + "FERRUM_NAVIGATION_WAIT", %w[FERRUM_CLICK_WAIT FERRUM_GOTO_WAIT], 0.1 + ) ON_TIMEOUT = %i[ignore raise].freeze extend Forwardable @@ -48,7 +50,7 @@ class Page include Interceptable attr_accessor :referrer - attr_reader :context_id, :target_id, :event, :tracing + attr_reader :context_id, :target_id, :idle, :tracing # Client connection. # @@ -109,7 +111,7 @@ def initialize(client, context_id:, target_id:, proxy: nil) @frames = Concurrent::Map.new @main_frame = Frame.new(nil, self) - @event = Utils::Event.new.tap(&:set) + @idle = Concurrency::Cell.set(true) @document_version = 0 self.proxy = proxy @@ -138,6 +140,12 @@ def initialize(client, context_id:, target_id:, proxy: nil) # What to do when the page doesn't load in time: `:ignore` returns `nil`, # `:raise` raises {NavigationTimeoutError}. `nil` follows the deprecated # `:pending_connection_errors` option, which raises {PendingConnectionsError}. + # Once the budget of an enclosing {Deadline.within} is used up, `:ignore` + # raises {NavigationTimeoutError} too. + # + # @param [Numeric, Deadline, nil] timeout + # How long the whole navigation may take, from sending the request to the + # page loading. Defaults to the page's `timeout`. # # @return [String, nil] # The navigated frame's id, or `nil` when the timeout is reached. @@ -154,15 +162,20 @@ def initialize(client, context_id:, target_id:, proxy: nil) # @example # page.go_to("https://github.com/") # page.go_to("/slow", on_timeout: :raise) + # page.go_to("/slow", timeout: 10) # - def go_to(url = nil, on_timeout: nil) + def go_to(url = nil, on_timeout: nil, timeout: nil) # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity unless on_timeout.nil? || ON_TIMEOUT.include?(on_timeout) raise ArgumentError, "on_timeout should be one of #{ON_TIMEOUT.inspect}, got #{on_timeout.inspect}" end options = { url: combine_url!(url) } options.merge!(referrer: referrer) if referrer - response = command("Page.navigate", wait: GOTO_WAIT, **options) + deadline = Deadline.wrap(timeout || self.timeout) + response = nil + wait_for_navigation!(NAVIGATION_WAIT, timeout: deadline) do + response = command("Page.navigate", timeout: deadline, **options) + end error_text = response["errorText"] # https://cs.chromium.org/chromium/src/net/base/net_error_list.h if error_text && error_text != "net::ERR_ABORTED" # Request aborted due to user action or download raise StatusError.new(options[:url], "Request to #{options[:url]} failed (#{error_text})") @@ -171,12 +184,12 @@ def go_to(url = nil, on_timeout: nil) response["frameId"] rescue TimeoutError on_timeout ||= @options.pending_connection_errors ? :pending_connections : :ignore - return if on_timeout == :ignore + return if on_timeout == :ignore && !Deadline.current&.expired? pendings = network.traffic.select(&:pending?).map(&:url).compact - raise NavigationTimeoutError.new(options[:url], pendings) if on_timeout == :raise + raise PendingConnectionsError.new(options[:url], pendings) if on_timeout == :pending_connections - raise PendingConnectionsError.new(options[:url], pendings) + raise NavigationTimeoutError.new(options[:url], pendings) end alias goto go_to alias go go_to @@ -190,7 +203,7 @@ def go_to(url = nil, on_timeout: nil) # page.close # => true # def close - client.command("Target.closeTarget", async: true, targetId: @target_id) + client.notify("Target.closeTarget", targetId: @target_id) close_connection true @@ -220,6 +233,7 @@ def close_connection # a desktop and leaves touch emulation as is. Turning touch off resets hover and pointer to the detected input # devices, which undoes `--blink-settings` overrides until the next navigation # + # @raise [TimeoutError] def set_viewport(width: 0, height: 0, scale_factor: 0, mobile: nil) command("Emulation.setTouchEmulationEnabled", enabled: mobile, maxTouchPoints: 1) unless mobile.nil? @@ -247,6 +261,8 @@ def set_viewport(width: 0, height: 0, scale_factor: 0, mobile: nil) # # @return [Hash{String => Object}] # + # @raise [TimeoutError] + # # @example # page.resize(width: 1024, height: 768) # @@ -270,6 +286,7 @@ def resize(width: nil, height: nil, fullscreen: false) # # This doesn't prevent users evaluate JavaScript with Ferrum. # + # @raise [TimeoutError] def disable_javascript command("Emulation.setScriptExecutionDisabled", value: true) end @@ -280,6 +297,8 @@ def disable_javascript # @return [(Integer, Integer)] # The left, top coordinates of the window. # + # @raise [TimeoutError] + # # @example # page.position # => [10, 20] # @@ -298,6 +317,8 @@ def position # @option options [Integer] :top # The number of pixels from the top of the screen. # + # @raise [TimeoutError] + # # @example # page.position = { left: 10, top: 20 } # @@ -324,6 +345,8 @@ def position=(options) # @option options [String] :window_state # The window state. Default to normal. Allowed Values: normal, minimized, maximized, fullscreen # + # @raise [TimeoutError] + # # @example # page.window_bounds = { left: 10, top: 20, width: 1024, height: 768, window_state: "normal" } # @@ -332,7 +355,7 @@ def window_bounds=(bounds) window_state = options.delete(:window_state) bounds = { windowState: window_state, **options }.compact - client.command("Browser.setWindowBounds", windowId: window_id, bounds: bounds) + command("Browser.setWindowBounds", windowId: window_id, bounds: bounds) end # @@ -340,11 +363,13 @@ def window_bounds=(bounds) # # @return [Hash{String => (Integer, String)}] # + # @raise [TimeoutError] + # # @example # page.window_bounds # => { "left": 0, "top": 1286, "width": 10, "height": 10, "windowState": "normal" } # def window_bounds - client.command("Browser.getWindowBounds", windowId: window_id).fetch("bounds") + command("Browser.getWindowBounds", windowId: window_id).fetch("bounds") end # @@ -352,28 +377,40 @@ def window_bounds # # @return [Integer] # + # @raise [TimeoutError] + # # @example # page.window_id # => 1 # def window_id - client.command("Browser.getWindowForTarget", targetId: target_id)["windowId"] + command("Browser.getWindowForTarget", targetId: target_id)["windowId"] end # # Reloads the current page. # + # @param [Numeric, Deadline, nil] timeout + # How long the whole reload may take. Defaults to the page's `timeout`. + # + # @raise [TimeoutError] + # # @example # page.go_to("https://github.com/") # page.refresh # - def refresh - command("Page.reload", wait: timeout, slowmoable: true) + def refresh(timeout: nil) + deadline = Deadline.wrap(timeout || self.timeout) + wait_for_navigation!(NAVIGATION_WAIT, timeout: deadline) do + command("Page.reload", timeout: deadline, slowmoable: true) + end end alias reload refresh # # Stop all navigations and loading pending resources on the page. # + # @raise [TimeoutError] + # # @example # page.go_to("https://github.com/") # page.stop @@ -385,65 +422,80 @@ def stop # # Navigates to the previous URL in the history. # + # @param [Numeric, Deadline, nil] timeout + # How long the history lookup and the page load may take together, the page timeout by default. + # + # @raise [TimeoutError] + # # @example # page.go_to("https://github.com/") # page.at_xpath("//a").click # page.back # - def back - history_navigate(delta: -1) + def back(timeout: nil) + history_navigate(delta: -1, timeout: timeout) end # # Navigates to the next URL in the history. # + # @param [Numeric, Deadline, nil] timeout + # How long the history lookup and the page load may take together, the page timeout by default. + # + # @raise [TimeoutError] + # # @example # page.go_to("https://github.com/") # page.at_xpath("//a").click # page.back # page.forward # - def forward - history_navigate(delta: 1) + def forward(timeout: nil) + history_navigate(delta: 1, timeout: timeout) end # # Blocks until the page reloads or the timeout is reached. # - # @param [Numeric] timeout + # @param [Numeric, Deadline] timeout # Maximum time in seconds to wait for a reload event. # # @example # page.wait_for_reload # def wait_for_reload(timeout = 1) - @event.reset if @event.set? - @event.wait(timeout) - @event.set + deadline = Deadline.wrap(timeout) + generation = @idle.clear if @idle.set? + @idle.wait(deadline) + @idle.set(true) if generation == @idle.generation end # # Runs the block and, if it starts a navigation of the main frame within - # `wait` seconds, waits for that navigation to finish, e.g. after a click - # on a link made from JavaScript. + # `navigation_wait` seconds, waits for that navigation to finish, e.g. after + # a click on a link made from JavaScript. # - # @param [Numeric] wait + # @param [Numeric] navigation_wait # How many seconds to watch for a navigation after the block returns. # + # @param [Numeric, Deadline, nil] timeout + # How long the block and the navigation it starts may take together. + # Defaults to the page's `timeout`. + # # @yield The code that may start a navigation. # # @return [Boolean] - # `false` when a navigation started but didn't finish within the page's - # `timeout`. + # `false` when a navigation started but didn't finish in time. # # @example # page.wait_for_navigation { page.at_css("a").evaluate("this.click()") } # => true # - def wait_for_navigation(wait = Mouse::CLICK_WAIT) - iteration = @event.reset + def wait_for_navigation(navigation_wait = NAVIGATION_WAIT, timeout: nil) + deadline = Deadline.wrap(timeout || self.timeout) + generation = @idle.clear yield - @event.wait(wait) - iteration == @event.iteration || @event.wait(timeout) + @idle.wait([navigation_wait, deadline.remaining].min) + generation == @idle.generation || @idle.wait(deadline) end # @@ -463,6 +515,8 @@ def wait_for_navigation!(...) # # @return [Boolean] # + # @raise [TimeoutError] + # # @example # page.bypass_csp # => true # page.go_to("https://github.com/ruby-concurrency/concurrent-ruby/blob/master/docs-source/promises.in.md") @@ -490,6 +544,8 @@ def bypass_csp(enabled: true) # @return [String] # The script identifier, for {#remove_script_on_new_document}. # + # @raise [TimeoutError] + # # @example # page.evaluate_on_new_document <<~JS # Object.defineProperty(navigator, "languages", { @@ -509,6 +565,8 @@ def evaluate_on_new_document(source, world_name: nil, run_immediately: false) # # @return [true] # + # @raise [TimeoutError] + # def remove_script_on_new_document(identifier) command("Page.removeScriptToEvaluateOnNewDocument", identifier: identifier) true @@ -520,6 +578,8 @@ def remove_script_on_new_document(identifier) # # @return [Boolean] # + # @raise [TimeoutError] + # # @example # page.activate # => true # @@ -529,38 +589,36 @@ def activate end # - # Sends a CDP command to the browser and optionally waits for network - # activity on the main frame to settle before returning. + # Sends a CDP command to the browser. To wait for a navigation it starts, + # wrap it in {#wait_for_navigation}. # # @param [String] method # The CDP method name, e.g. `"Page.navigate"`. # - # @param [Numeric] wait - # How many seconds to wait for a network event on the main frame after - # the command is sent. `0` disables waiting. - # # @param [Boolean] slowmoable # Whether to sleep for `Browser::Options#slowmo` seconds before sending # the command. # - # @param [Numeric, nil] timeout + # @param [Numeric, Deadline, nil] timeout # Overrides the timeout this command's response is bound by. Defaults # to the page's `timeout`. Callers with their own budget (e.g. `#pdf`/ # `#screenshot`) pass it explicitly. # # @return [Hash{String => Object}] # + # @raise [TimeoutError] + # + # @raise [ArgumentError] + # When given the removed `wait:` or `async:`, which would otherwise be sent to the browser as parameters. + # # @example - # page.command("Page.navigate", url: "https://github.com/") + # page.command("Runtime.evaluate", expression: "1 + 1") # - def command(method, wait: 0, slowmoable: false, timeout: nil, **params) - sleep(@options.slowmo) if slowmoable && @options.slowmo.positive? - timeout ||= self.timeout - return client.command(method, timeout:, **params) unless wait.positive? + def command(method, slowmoable: false, timeout: nil, **params) + raise ArgumentError, "wait: is removed, wrap #{method} in wait_for_navigation instead" if params.key?(:wait) - result = nil - wait_for_navigation!(wait) { result = client.command(method, timeout:, **params) } - result + sleep(@options.slowmo) if slowmoable && @options.slowmo.positive? + client.command(method, timeout: Deadline.wrap(timeout || self.timeout), **params) end # Subscribes to a CDP event, or to `:dialog`, `:request`, `:response`, @@ -613,9 +671,11 @@ def use_authorized_proxy? # @return [Integer, Boolean] # The root node id, or `true` when sent asynchronously. # + # @raise [TimeoutError] + # def document_node_id(async: false) result = if async - client.command("DOM.getDocument", async: true, depth: 0) + client.notify("DOM.getDocument", depth: 0) else command("DOM.getDocument", depth: 0).dig("root", "nodeId") end @@ -686,18 +746,18 @@ def prepare_page document_node_id end - def history_navigate(delta:) - history = command("Page.getNavigationHistory") + def history_navigate(delta:, timeout: nil) + deadline = Deadline.wrap(timeout || self.timeout) + history = command("Page.getNavigationHistory", timeout: deadline) index, entries = history.values_at("currentIndex", "entries") entry = entries[index + delta] return unless entry # Potential wait because of network event - command("Page.navigateToHistoryEntry", - wait: Mouse::CLICK_WAIT, - slowmoable: true, - entryId: entry["id"]) + wait_for_navigation!(NAVIGATION_WAIT, timeout: deadline) do + command("Page.navigateToHistoryEntry", timeout: deadline, slowmoable: true, entryId: entry["id"]) + end end def combine_url!(url_or_path) diff --git a/lib/ferrum/page/animation.rb b/lib/ferrum/page/animation.rb index 1abdacdd..0c4bd430 100644 --- a/lib/ferrum/page/animation.rb +++ b/lib/ferrum/page/animation.rb @@ -13,6 +13,8 @@ module Animation # # @return [Number] # + # @raise [TimeoutError] + # def playback_rate command("Animation.getPlaybackRate")["playbackRate"] end @@ -22,6 +24,8 @@ def playback_rate # # @param [Number] value # + # @raise [TimeoutError] + # # @example # browser = Ferrum::Browser.new # browser.playback_rate = 2000 diff --git a/lib/ferrum/page/cookies.rb b/lib/ferrum/page/cookies.rb index d17f9161..686e83a4 100644 --- a/lib/ferrum/page/cookies.rb +++ b/lib/ferrum/page/cookies.rb @@ -29,6 +29,8 @@ def initialize(page) # @return [Enumerator] # If no block is given, an Enumerator object will be returned. # + # @raise [TimeoutError] + # # @example # page.cookies.each { |cookie| puts cookie.name } # page.cookies.each(urls: ["https://accounts.google.com"]).map(&:name) # => ["SID"] @@ -53,6 +55,8 @@ def each(urls: nil) # # @return [Boolean] # + # @raise [TimeoutError] + # # @example # page.cookies.set(name: "stealth", value: "omg") # => true # @@ -82,6 +86,8 @@ def set(options) # # @return [true] # + # @raise [TimeoutError] + # # @example # page.cookies.remove(name: "stealth") # => true # @@ -102,6 +108,8 @@ def remove(name:, **options) # # @return [true] # + # @raise [TimeoutError] + # # @example # page.cookies.clear # => true # diff --git a/lib/ferrum/page/frames.rb b/lib/ferrum/page/frames.rb index 447a61df..9e63f80f 100644 --- a/lib/ferrum/page/frames.rb +++ b/lib/ferrum/page/frames.rb @@ -104,7 +104,7 @@ def subscribe_frame_detached frame.execution_id = nil else @frames.delete(params["frameId"]) - @event.set if @main_frame.idle? && frame&.idle? == false && frames.size == 1 + @idle.set(true) if @main_frame.idle? && frame&.idle? == false && frames.size == 1 end end end @@ -125,7 +125,7 @@ def subscribe_frame_started_loading on("Page.frameStartedLoading") do |params| frame = @frames[params["frameId"]] frame.state = :started_loading if frame - @event.reset + @idle.clear end end @@ -148,14 +148,14 @@ def subscribe_frame_stopped_loading # tree, and we save it in a variable because if we call that again root # node will change the id and all subsequent nodes have to change id too. if @main_frame.id == params["frameId"] - @event.set if idling? + @idle.set(true) if idling? document_node_id(async: true) end frame = @frames[params["frameId"]] frame&.state = :stopped_loading - @event.set if idling? + @idle.set(true) if idling? end end @@ -181,14 +181,14 @@ def subscribe_frame_lifecycle_events # `init` and `networkIdle`. Without it `go_to` would wait for such iframe until it raises timeout. if event["name"] == "networkIdle" && !frame.main? frame.state = :stopped_loading - @event.set if idling? + @idle.set(true) if idling? end end end def subscribe_navigated_within_document on("Page.navigatedWithinDocument") do - @event.set if idling? + @idle.set(true) if idling? end end @@ -198,7 +198,7 @@ def subscribe_request_will_be_sent # Document, Stylesheet, Image, Media, Font, Script, TextTrack, XHR, # Fetch, EventSource, WebSocket, Manifest, SignedExchange, Ping, # CSPViolationReport, Other - @event.reset if params["frameId"] == @main_frame.id && params["type"] == "Document" + @idle.clear if params["frameId"] == @main_frame.id && params["type"] == "Document" end end diff --git a/lib/ferrum/page/screencast.rb b/lib/ferrum/page/screencast.rb index e78aac9a..26ff8eff 100644 --- a/lib/ferrum/page/screencast.rb +++ b/lib/ferrum/page/screencast.rb @@ -61,6 +61,8 @@ module Screencast # @yieldparam session_id [Integer] # Frame number. # + # @raise [TimeoutError] + # # @example # require "base64" # @@ -93,6 +95,8 @@ def start_screencast(**opts) end # Stops sending frames. + # + # @raise [TimeoutError] def stop_screencast command("Page.stopScreencast") end diff --git a/lib/ferrum/page/screenshot.rb b/lib/ferrum/page/screenshot.rb index 21e4f0c9..b4b52337 100644 --- a/lib/ferrum/page/screenshot.rb +++ b/lib/ferrum/page/screenshot.rb @@ -75,6 +75,8 @@ module Screenshot # {DEFAULT_RENDER_TIMEOUT} since a full-page capture is a known slow # outlier among CDP commands. # + # @raise [TimeoutError] + # # @example # page.go_to("https://google.com/") # @@ -130,6 +132,8 @@ def screenshot(timeout: DEFAULT_RENDER_TIMEOUT, **opts) # @option opts [Float] :paper_height # Sets the paper's height. # + # @raise [TimeoutError] + # # @note # See other [native options](https://chromedevtools.github.io/devtools-protocol/tot/Page#method-printToPDF) you # can pass. @@ -157,6 +161,8 @@ def pdf(timeout: DEFAULT_RENDER_TIMEOUT, **opts) # @param [String, nil] path # The path to save a file on the disk. # + # @raise [TimeoutError] + # # @example # page.go_to("https://google.com/") # page.mhtml(path: "google.mhtml") # => 87742 @@ -174,6 +180,8 @@ def mhtml(path: nil) # @return [(Integer, Integer)] # The width, height of the viewport. # + # @raise [TimeoutError] + # # @example # page.viewport_size # => [1024, 768] # @@ -189,6 +197,8 @@ def viewport_size # # @return [Float] # + # @raise [TimeoutError] + # # @example # page.device_pixel_ratio # => 1.0 # @@ -205,6 +215,8 @@ def device_pixel_ratio # @return [(Integer, Integer)] # The scroll width, scroll height of the document. # + # @raise [TimeoutError] + # # @example # page.document_size # => [1024, 4000] # diff --git a/lib/ferrum/page/tracing.rb b/lib/ferrum/page/tracing.rb index be8acadb..d584f0c1 100644 --- a/lib/ferrum/page/tracing.rb +++ b/lib/ferrum/page/tracing.rb @@ -34,7 +34,7 @@ def initialize(page) # @param [:binary, :base64] encoding # Encode output as Base64 or plain text. # - # @param [Float, nil] timeout + # @param [Numeric, Deadline, nil] timeout # Wait until file streaming finishes in the specified time or raise # error. # @@ -51,10 +51,13 @@ def initialize(page) # When `path` is specified returns `true` and stores trace data into # file. # + # @raise [TimeoutError] + # def record(path: nil, encoding: :binary, timeout: nil, trace_config: nil, screenshots: false) @path = path @encoding = encoding - @pending = Concurrent::IVar.new + @pending = Concurrency::Cell.new + @failure = nil trace_config ||= DEFAULT_TRACE_CONFIG.dup if screenshots @@ -68,7 +71,10 @@ def record(path: nil, encoding: :binary, timeout: nil, trace_config: nil, screen yield stop - @pending.value!(timeout || @page.timeout) + @pending.wait(timeout || @page.timeout) + raise @failure if @failure + + @pending.value end private @@ -89,7 +95,8 @@ def subscribe_tracing_complete @pending.set(stream_handle(event["stream"])) rescue StandardError => e - @pending.fail(e) + @failure = e + @pending.close end @subscribed_tracing_complete = true diff --git a/lib/ferrum/remote_object.rb b/lib/ferrum/remote_object.rb index 542a1835..de9e3bb5 100644 --- a/lib/ferrum/remote_object.rb +++ b/lib/ferrum/remote_object.rb @@ -50,6 +50,8 @@ def initialize(runtime, page, result) # # @return [Object] # + # @raise [TimeoutError] + # def value @runtime.evaluate_in(self, "this", [], {}) end @@ -59,6 +61,8 @@ def value # # @return [void] # + # @raise [TimeoutError] + # def release @page.command("Runtime.releaseObject", objectId: @remote_id) nil diff --git a/lib/ferrum/target.rb b/lib/ferrum/target.rb index 882ef420..aa483e2f 100644 --- a/lib/ferrum/target.rb +++ b/lib/ferrum/target.rb @@ -48,6 +48,8 @@ def connected? # Connects to and returns this target's {Page}. # # @return [Page] + # + # @raise [TimeoutError] def page @page ||= build_page end @@ -58,6 +60,8 @@ def page # # @raise [NoSuchTargetError] if there's no session with the target, e.g. a service worker # not attached to with {Context#attach_target}. + # + # @raise [TimeoutError] def worker @worker ||= build_worker end @@ -209,10 +213,21 @@ def maybe_sleep_if_new_window # Sends a CDP command through {#client}. # - # @return [Boolean, Hash] - # `true` when sent asynchronously, otherwise the command's result. - def command(...) - client.command(...) + # @param [String] method + # The CDP method name. + # + # @param [Numeric, Deadline, nil] timeout + # How long to wait for the response, the browser's `timeout` by default. + # + # @param [Hash] params + # The command's parameters. + # + # @return [Hash] + # The command's result. + # + # @raise [TimeoutError] + def command(method, timeout: nil, **params) + client.command(method, timeout: timeout || client.timeout, **params) end private diff --git a/lib/ferrum/utils/deprecate.rb b/lib/ferrum/utils/deprecate.rb index b231e487..c616f80e 100644 --- a/lib/ferrum/utils/deprecate.rb +++ b/lib/ferrum/utils/deprecate.rb @@ -46,6 +46,56 @@ def warn(message, user_code: false) Kernel.warn(message) end + # + # Picks the value of a keyword argument that was renamed: warns and returns the old one when it is given, + # otherwise returns the current one. + # + # @param [Symbol] old + # The old keyword. + # + # @param [Symbol] new + # The new keyword. + # + # @param [Object, nil] value + # What was passed as the old keyword, `nil` when nothing was. + # + # @param [Object] current + # The value of the new keyword. + # + # @return [Object] + # + def renamed_keyword(old, new, value, current) + return current if value.nil? + + warn(":#{old} is renamed to :#{new} and will be removed in Ferrum 0.22.", user_code: true) + value + end + + # + # Reads a float from an environment variable that replaced older ones: warns and falls back to the first old + # one that is set when the new one is not. + # + # @param [String] name + # The current variable. + # + # @param [Array] legacy + # The variables it replaced. + # + # @param [Float] default + # Used when none is set. + # + # @return [Float] + # + def renamed_env(name, legacy, default) + return ENV[name].to_f if ENV.key?(name) + + old = legacy.find { |n| ENV.key?(n) } + return default unless old + + warn("#{old} is replaced by #{name} and will be removed in Ferrum 0.22.") + ENV[old].to_f + end + # # Forgets which warnings have already been printed. Only useful in tests. # diff --git a/lib/ferrum/utils/event.rb b/lib/ferrum/utils/event.rb deleted file mode 100644 index 116c6cc5..00000000 --- a/lib/ferrum/utils/event.rb +++ /dev/null @@ -1,33 +0,0 @@ -# frozen_string_literal: true - -module Ferrum - module Utils - # - # A {Concurrent::Event} subclass that additionally tracks how many - # times it has been reset, via {#iteration}. - # - class Event < Concurrent::Event - # - # Number of times the event has been reset. - # - # @return [Integer] - # - def iteration - synchronize { @iteration } - end - - # - # Marks the event as unset and increments the iteration counter. - # - # @return [Integer] - # - def reset - synchronize do - @iteration += 1 - @set = false if @set - @iteration - end - end - end - end -end diff --git a/lib/ferrum/worker.rb b/lib/ferrum/worker.rb index bb69a61b..04c4cf9a 100644 --- a/lib/ferrum/worker.rb +++ b/lib/ferrum/worker.rb @@ -26,7 +26,7 @@ def initialize(client, target_id:, url:, type: "worker") @type = type @options = client.options @page = self - @execution_id = Concurrent::MVar.new + @execution_id = Concurrency::Cell.new @network = Network.new(self) @console = Console.new(self) @@ -53,10 +53,21 @@ def timeout # Sends a CDP command through {#client}. # - # @return [Boolean, Hash] - # `true` when sent asynchronously, otherwise the command's result. - def command(...) - client.command(...) + # @param [String] method + # The CDP method name. + # + # @param [Numeric, Deadline, nil] timeout + # How long to wait for the response, the worker's `timeout` by default. + # + # @param [Hash] params + # The command's parameters. + # + # @return [Hash] + # The command's result. + # + # @raise [TimeoutError] + def command(method, timeout: nil, **params) + client.command(method, timeout: timeout || self.timeout, **params) end # Workers have a single execution context and no navigable document, so @@ -71,9 +82,9 @@ def main_frame # @return [Boolean] def close if type == "worker" - client.command("Runtime.evaluate", async: true, expression: "self.close()") + client.notify("Runtime.evaluate", expression: "self.close()") else - client.command("Target.closeTarget", async: true, targetId: target_id) + client.notify("Target.closeTarget", targetId: target_id) end close_connection @@ -121,18 +132,16 @@ def prepare command("Runtime.runIfWaitingForDebugger") end - def execution_id! - value = @execution_id.borrow(timeout, &:itself) - raise NoExecutionContextError if value.instance_of?(Object) - - value + def execution_id!(timeout = nil) + @execution_id.wait(timeout || self.timeout) + @execution_id.value || raise(NoExecutionContextError) end def execution_id=(value) if value.nil? - @execution_id.try_take! - else - @execution_id.try_put!(value) + @execution_id.clear + elsif !@execution_id.set? + @execution_id.set(value) end end end diff --git a/sig/concurrent/async.rbs b/sig/concurrent/async.rbs deleted file mode 100644 index bb57c0dc..00000000 --- a/sig/concurrent/async.rbs +++ /dev/null @@ -1,10 +0,0 @@ -module Concurrent - class IVar - def initialize: (?untyped value, ?opts: Hash[Symbol, untyped]) -> void - def value: (?Numeric? timeout) -> untyped - def value=: (untyped value) -> untyped - def complete?: () -> bool - def incomplete?: () -> bool - def wait: (?Numeric? timeout) -> self - end -end diff --git a/sig/concurrent/event.rbs b/sig/concurrent/event.rbs deleted file mode 100644 index 4e017c82..00000000 --- a/sig/concurrent/event.rbs +++ /dev/null @@ -1,9 +0,0 @@ -module Concurrent - class Event - def initialize: () -> void - def set: () -> bool - def reset: () -> bool - def wait: (?Numeric? timeout) -> bool - def set?: () -> bool - end -end diff --git a/sig/ferrum/browser.rbs b/sig/ferrum/browser.rbs index d2b40bef..ff259833 100644 --- a/sig/ferrum/browser.rbs +++ b/sig/ferrum/browser.rbs @@ -22,7 +22,7 @@ module Ferrum def timeout=: (::Numeric) -> ::Numeric - def protocol_timeout: () -> ::Numeric + def protocol_timeout: () -> ::Integer def protocol_timeout=: (::Numeric) -> ::Numeric @@ -37,7 +37,7 @@ module Ferrum def add_extension: (String | Hash[Symbol, String] extension) -> ::Array[String] - def command: (String method, **untyped params) -> Hash[String, untyped] + def command: (String method, ?timeout: (::Numeric | Deadline)?, **untyped params) -> Hash[String, untyped] def reset: () -> void diff --git a/sig/ferrum/browser/options.rbs b/sig/ferrum/browser/options.rbs index 28bc6967..3f12b09b 100644 --- a/sig/ferrum/browser/options.rbs +++ b/sig/ferrum/browser/options.rbs @@ -9,17 +9,19 @@ module Ferrum BASE_URL_SCHEMA: ::Array[String] - DEFAULT_TIMEOUT: ::Integer - - DEFAULT_PROTOCOL_TIMEOUT: ::Integer + DEFAULT_TIMEOUT: ::Float PROCESS_TIMEOUT: ::Integer + PROTOCOL_TIMEOUT_REMOVED: String + attr_reader window_size: ::Array[::Integer] attr_accessor timeout: ::Numeric - attr_accessor protocol_timeout: ::Numeric + def protocol_timeout: () -> ::Integer + + def protocol_timeout=: (::Numeric value) -> ::Numeric attr_accessor default_user_agent: String? @@ -71,7 +73,6 @@ module Ferrum @port: String @host: String @timeout: ::Numeric - @protocol_timeout: ::Numeric @window_size: ::Array[::Integer] @js_errors: bool @headless: bool @@ -110,6 +111,8 @@ module Ferrum private + def warn_protocol_timeout: () -> void + def fetch_deprecated: (Symbol name, String instead) -> bool def read_extension: (String | Hash[Symbol, String] extension) -> String diff --git a/sig/ferrum/client.rbs b/sig/ferrum/client.rbs index fc1dcb36..c0277b53 100644 --- a/sig/ferrum/client.rbs +++ b/sig/ferrum/client.rbs @@ -10,7 +10,9 @@ module Ferrum def initialize: (Client client, String session_id) -> void - def command: (String method, ?async: bool, ?timeout: ::Numeric?, **untyped params) -> (bool | Hash[String, untyped]) + def command: (String method, timeout: (::Numeric | Deadline), **untyped params) -> Hash[String, untyped] + + def notify: (String method, **untyped params) -> true def on: (String event) { (Hash[String, untyped]) -> void } -> Integer @@ -32,25 +34,32 @@ module Ferrum end class Client + PROTOCOL_TIMEOUT: Integer + INTERRUPTIONS: Array[String] + extend Forwardable attr_reader ws_url: (String | ::Addressable::URI) attr_reader options: Browser::Options - attr_reader subscriber: Client::Subscriber + attr_reader dispatcher: Concurrency::Dispatcher @command_id: ::Integer @ws_url: (String | ::Addressable::URI) @options: Browser::Options - @pendings: ::Concurrent::Hash[::Integer, ::Concurrent::IVar] + @responses: Concurrency::AwaitableMap[::Integer, Hash[String, untyped]] @ws: Client::WebSocket - @subscriber: Client::Subscriber + @dispatcher: Concurrency::Dispatcher @thread: ::Thread def initialize: ((String | ::Addressable::URI) ws_url, Browser::Options options) -> void - def command: (String method, ?async: bool, ?timeout: ::Numeric?, **untyped params) -> (bool | Hash[String, untyped]) + def command: (String method, timeout: (::Numeric | Deadline), **untyped params) -> Hash[String, untyped] + + def notify: (String method, **untyped params) -> true - def send_message: (Hash[Symbol, untyped] message, async: bool, ?timeout: ::Numeric?) -> (bool | Hash[String, untyped]) + def send_message: (Hash[Symbol, untyped] message, timeout: (::Numeric | Deadline)) -> Hash[String, untyped] + + def send_notification: (Hash[Symbol, untyped] message) -> true def on: (String event) { (Hash[String, untyped]) -> void } -> Integer @@ -66,15 +75,16 @@ module Ferrum def build_message: (String method, Hash[Symbol, untyped] params) -> Hash[Symbol, untyped] - def protocol_timeout: () -> ::Numeric + def timeout: () -> ::Numeric - def protocol_timeout=: (::Numeric value) -> ::Numeric private def start: () -> void - def release_pendings: () -> void + def dispatch: (Hash[String, untyped] message) -> void + + def release_commands: () -> void def dead_browser_error: () -> DeadBrowserError diff --git a/sig/ferrum/client/subscriber.rbs b/sig/ferrum/client/subscriber.rbs deleted file mode 100644 index 43a172c2..00000000 --- a/sig/ferrum/client/subscriber.rbs +++ /dev/null @@ -1,34 +0,0 @@ -module Ferrum - class Client - class Subscriber - INTERRUPTIONS: Array[String] - - @regular: ::Queue - @priority: ::Queue - @on: ::Concurrent::Hash[String, ::Concurrent::Array[Proc]] - @regular_thread: ::Thread? - @priority_thread: ::Thread? - - def initialize: () -> void - - def <<: (Hash[String, untyped] message) -> void - - def on: (String event) { (Hash[String, untyped], Integer, Integer) -> void } -> Integer - - def off: (String event, Integer id) -> bool - - def subscribed?: (String event) -> bool - - def close: () -> void - - def clear: (session_id: String) -> void - - private - - def start: () -> void - - def call: (Hash[String, untyped] message) -> void - end - end -end - diff --git a/sig/ferrum/client/web_socket.rbs b/sig/ferrum/client/web_socket.rbs index 2b1f7b86..7c583fc8 100644 --- a/sig/ferrum/client/web_socket.rbs +++ b/sig/ferrum/client/web_socket.rbs @@ -4,6 +4,7 @@ module Ferrum WEBSOCKET_BUG_SLEEP: Float DEFAULT_PORTS: Hash[String, Integer] WRITE_CHUNK_SIZE: Integer + WRITE_TIMEOUT: Integer attr_reader url: (String | ::Addressable::URI) attr_reader messages: ::Queue @@ -22,7 +23,7 @@ module Ferrum @encoded: Array[String] @close_reason: String? - def initialize: ((String | ::Addressable::URI) url, Integer? max_receive_size, Log? log, ?::Numeric? write_timeout) -> void + def initialize: ((String | ::Addressable::URI) url, Integer? max_receive_size, Log? log, ?::Numeric write_timeout) -> void def on_open: (untyped event) -> void @@ -30,7 +31,7 @@ module Ferrum def on_close: (untyped event) -> void - def send_message: (Hash[Symbol, untyped] data, ?::Numeric? timeout) -> void + def send_message: (Hash[Symbol, untyped] data, ?(::Numeric | Deadline)? timeout) -> void def write: (String data) -> void @@ -40,9 +41,9 @@ module Ferrum private - def drive: (?::Float? deadline) { () -> void } -> void + def drive: (?Deadline? deadline) { () -> void } -> void - def queue_encoded: (?::Float? deadline) -> void + def queue_encoded: (?Deadline? deadline) -> void def start_writer: () -> void diff --git a/sig/ferrum/concurrency/awaitable_map.rbs b/sig/ferrum/concurrency/awaitable_map.rbs new file mode 100644 index 00000000..1454420f --- /dev/null +++ b/sig/ferrum/concurrency/awaitable_map.rbs @@ -0,0 +1,37 @@ +module Ferrum + module Concurrency + class AwaitableMap[K, V] + @mutex: Mutex + @condition: ConditionVariable + @map: Hash[K, V] + @expected: Hash[K, bool] + @closed: bool + + def initialize: () -> void + + def []: (K key) -> V? + + def []=: (K key, V value) -> V + + def expect: (K key) -> K + + def expecting?: (K key) -> bool + + def fulfill: (K key, V value) -> bool + + def delete: (K key) -> V? + + def close: () -> void + + def key?: (K key) -> bool + + def to_h: () -> Hash[K, V] + + def values: () -> Array[V] + + def await: (K key, (Numeric | Deadline) timeout) -> V? + + def inspect: () -> String + end + end +end diff --git a/sig/ferrum/concurrency/cell.rbs b/sig/ferrum/concurrency/cell.rbs new file mode 100644 index 00000000..8fbba417 --- /dev/null +++ b/sig/ferrum/concurrency/cell.rbs @@ -0,0 +1,32 @@ +module Ferrum + module Concurrency + class Cell + @mutex: Mutex + @condition: ConditionVariable + @set: bool + @closed: bool + @value: untyped + @generation: Integer + + attr_reader generation: Integer + + def self.set: (untyped value) -> Cell + + def initialize: () -> void + + def set: (untyped value) -> untyped + + def clear: () -> Integer + + def set?: () -> bool + + def value: () -> untyped + + def close: () -> void + + def closed?: () -> bool + + def wait: ((Numeric | Deadline) timeout) -> bool + end + end +end diff --git a/sig/ferrum/concurrency/dispatcher.rbs b/sig/ferrum/concurrency/dispatcher.rbs new file mode 100644 index 00000000..fdc6c17b --- /dev/null +++ b/sig/ferrum/concurrency/dispatcher.rbs @@ -0,0 +1,35 @@ +module Ferrum + module Concurrency + class Dispatcher + DEFAULT_LANES: Array[Symbol] + + @queues: Hash[Symbol, ::Queue] + @on_error: ^(String, StandardError) -> void + @on: ::Concurrent::Hash[String, ::Concurrent::Array[[Integer, Proc]]] + @sequence: ::Concurrent::AtomicFixnum + @threads: Array[::Thread] + + def initialize: (?lanes: Array[Symbol], ?on_error: (^(String, StandardError) -> void)?) -> void + + def publish: (String key, untyped payload, ?lane: Symbol) -> void + + def on: (String key) { (untyped, Integer, Integer) -> void } -> Integer + + def off: (String key, Integer id) -> bool + + def subscribed?: (String key) -> bool + + def close: () -> void + + def clear: () { (String) -> boolish } -> void + + private + + def start: () -> void + + def call: (String key, untyped payload) -> void + + def warn_error: (String key, StandardError error) -> void + end + end +end diff --git a/sig/ferrum/context.rbs b/sig/ferrum/context.rbs index 896f0a75..e3a61362 100644 --- a/sig/ferrum/context.rbs +++ b/sig/ferrum/context.rbs @@ -3,14 +3,14 @@ module Ferrum POSITION: ::Array[:first | :last] attr_reader id: String - attr_reader targets: ::Concurrent::Hash[String, Target] + + def targets: () -> Hash[String, Target] @id: String @implicit: bool @client: Client @contexts: Contexts - @targets: ::Concurrent::Hash[String, Target] - @pendings: ::Concurrent::Map[String, ::Concurrent::IVar] + @targets: Concurrency::AwaitableMap[String, Target] @default_target: Target? @cookies: Cookies @@ -50,6 +50,8 @@ module Ferrum def dispose: () -> bool? + def target: (String target_id) -> Target? + def target?: (String target_id) -> bool def inspect: () -> ::String diff --git a/sig/ferrum/deadline.rbs b/sig/ferrum/deadline.rbs new file mode 100644 index 00000000..6308a46b --- /dev/null +++ b/sig/ferrum/deadline.rbs @@ -0,0 +1,27 @@ +module Ferrum + class Deadline + THREAD_KEY: Symbol + + def self.current: () -> Deadline? + + def self.within: [T] ((Deadline | Numeric) timeout) { () -> T } -> T + + def self.suspend: [T] () { () -> T } -> T + + def self.after: (Numeric seconds) -> Deadline + + def self.wrap: (Deadline | Numeric timeout) -> Deadline + + def initialize: (Float at) -> void + + def min: (Deadline other) -> Deadline + + def remaining: () -> Float + + def expired?: () -> bool + + protected + + attr_reader at: Float + end +end diff --git a/sig/ferrum/downloads.rbs b/sig/ferrum/downloads.rbs index 648bbd55..1b79ca03 100644 --- a/sig/ferrum/downloads.rbs +++ b/sig/ferrum/downloads.rbs @@ -3,14 +3,14 @@ module Ferrum VALID_BEHAVIOR: ::Array[:deny | :allow | :allowAndName | :default] @page: Page - @event: Utils::Event + @idle: Concurrency::Cell @files: ::Hash[String, Hash[String, untyped]] def initialize: (Page) -> void def files: () -> ::Array[Hash[String, untyped]] - def wait: (?::Integer timeout) ?{ () -> void } -> void + def wait: (?(::Numeric | Deadline) timeout) ?{ () -> void } -> void def set_behavior: (save_path: String, ?behavior: :deny | :allow | :allowAndName | :default) -> void diff --git a/sig/ferrum/frame.rbs b/sig/ferrum/frame.rbs index a28d776d..f8f47d6f 100644 --- a/sig/ferrum/frame.rbs +++ b/sig/ferrum/frame.rbs @@ -38,7 +38,7 @@ module Ferrum alias set_content content= - def execution_id!: () -> untyped + def execution_id!: (?(::Numeric | Deadline)? timeout) -> untyped def execution_id: () -> (nil | untyped) diff --git a/sig/ferrum/frame/dom.rbs b/sig/ferrum/frame/dom.rbs index cbe53c40..aa8de369 100644 --- a/sig/ferrum/frame/dom.rbs +++ b/sig/ferrum/frame/dom.rbs @@ -25,7 +25,9 @@ module Ferrum def at_css: (untyped selector, ?within: untyped?) -> untyped - def wait_for_selector: (?css: untyped?, ?xpath: untyped?, ?within: untyped?, ?timeout: untyped, ?interval: untyped) -> untyped + def poll_selector: (Symbol method_name, String selector, untyped within, Deadline deadline) -> untyped + + def wait_for_selector: (?css: untyped?, ?xpath: untyped?, ?within: untyped?, ?timeout: (::Numeric | Deadline), ?interval: untyped) -> untyped def add_script_tag: (?url: untyped?, ?path: untyped?, ?content: untyped?, ?type: ::String) -> untyped diff --git a/sig/ferrum/frame/runtime.rbs b/sig/ferrum/frame/runtime.rbs index 776c58e7..6ed93ed4 100644 --- a/sig/ferrum/frame/runtime.rbs +++ b/sig/ferrum/frame/runtime.rbs @@ -34,17 +34,17 @@ module Ferrum def wrap: (Script script, ::Array[::String] params, ::Symbol mode) -> ::String - def script_timeout: (Numeric? timeout) -> ::Float + def script_deadline: ((Numeric | Deadline)? timeout) -> Deadline - def call: (::String declaration, ::Array[untyped] values, on: (Node | RemoteObject)?, mode: ::Symbol, timeout: ::Float) -> untyped + def call: (::String declaration, ::Array[untyped] values, on: (Node | RemoteObject)?, mode: ::Symbol, deadline: Deadline) -> untyped - def call_function: (::String declaration, ::Array[untyped] values, ::Hash[::Symbol, untyped] target, ::Hash[::Symbol, untyped] serialization, ::Float timeout) -> untyped + def call_function: (::String declaration, ::Array[untyped] values, ::Hash[::Symbol, untyped] target, ::Hash[::Symbol, untyped] serialization, Deadline deadline) -> untyped def thenable?: (::Hash[::String, untyped] result) -> bool - def await_promise: (::Hash[::String, untyped] promise, ::Hash[::Symbol, untyped] serialization, ::Float timeout) -> ::Hash[::String, untyped] + def await_promise: (::Hash[::String, untyped] promise, ::Hash[::Symbol, untyped] serialization, Deadline deadline) -> ::Hash[::String, untyped] - def call_target: ((Node | RemoteObject)? on) -> ::Hash[::Symbol, untyped] + def call_target: ((Node | RemoteObject)? on, Deadline deadline) -> ::Hash[::Symbol, untyped] def handle_error: (untyped response, ?::String? declaration) -> void diff --git a/sig/ferrum/isolated_world.rbs b/sig/ferrum/isolated_world.rbs index 9f4ff5aa..e528f8a3 100644 --- a/sig/ferrum/isolated_world.rbs +++ b/sig/ferrum/isolated_world.rbs @@ -8,11 +8,11 @@ module Ferrum attr_reader page: Page - @execution_id: ::Concurrent::MVar + @execution_id: Concurrency::Cell def initialize: (Frame frame, String name) -> void - def execution_id!: () -> Integer + def execution_id!: (?(::Numeric | Deadline)? timeout) -> Integer def execution_id: () -> Integer? diff --git a/sig/ferrum/keyboard.rbs b/sig/ferrum/keyboard.rbs index 57f1a4a6..2d2efdf6 100644 --- a/sig/ferrum/keyboard.rbs +++ b/sig/ferrum/keyboard.rbs @@ -15,7 +15,7 @@ module Ferrum def up: ((String | Symbol) key) -> self - def type: (*(String | Symbol | ::Array[String | Symbol])) -> self + def type: (*(String | Symbol | ::Array[String | Symbol]), ?timeout: (::Numeric | Deadline)?) -> self def hold: [T] (*(String | Symbol) keys) { () -> T } -> T diff --git a/sig/ferrum/mouse.rbs b/sig/ferrum/mouse.rbs index 628011db..11418577 100644 --- a/sig/ferrum/mouse.rbs +++ b/sig/ferrum/mouse.rbs @@ -1,7 +1,5 @@ module Ferrum class Mouse - CLICK_WAIT: ::Float - BUTTON_MASKS: ::Hash[String, ::Integer] @page: Page @@ -15,17 +13,17 @@ module Ferrum def scroll_to: (::Numeric x, ::Numeric y) -> self - def click: (x: ::Numeric, y: ::Numeric, ?delay: ::Numeric, ?wait: ::Numeric, ?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer) -> self + def click: (x: ::Numeric, y: ::Numeric, ?delay: ::Numeric, ?navigation_wait: ::Numeric, ?wait: ::Numeric?, ?timeout: (::Numeric | Deadline)?, ?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer) -> self - def down: (?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer) -> self + def down: (?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer, ?navigation_wait: ::Numeric, ?wait: ::Numeric?, ?timeout: (::Numeric | Deadline)?) -> self - def up: (?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer, ?wait: ::Numeric) -> self + def up: (?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer, ?navigation_wait: ::Numeric, ?wait: ::Numeric?, ?timeout: (::Numeric | Deadline)?) -> self - def move: (x: ::Numeric, y: ::Numeric, ?steps: ::Integer, ?modifiers: ::Integer?) -> self + def move: (x: ::Numeric, y: ::Numeric, ?steps: ::Integer, ?modifiers: ::Integer?, ?timeout: (::Numeric | Deadline)?) -> self private - def mouse_event: (type: String, ?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer?, ?wait: ::Numeric) -> Hash[String, untyped] + def mouse_event: (type: String, ?button: Symbol, ?count: ::Integer, ?modifiers: ::Integer?, ?navigation_wait: ::Numeric, ?wait: ::Numeric?, ?timeout: (::Numeric | Deadline)?) -> Hash[String, untyped] def validate_button: (Symbol button) -> String diff --git a/sig/ferrum/network.rbs b/sig/ferrum/network.rbs index 4ff15c70..4d08bc66 100644 --- a/sig/ferrum/network.rbs +++ b/sig/ferrum/network.rbs @@ -19,7 +19,7 @@ module Ferrum def initialize: (untyped page) -> void - def wait_for_idle: (?connections: ::Integer, ?duration: ::Float, ?timeout: untyped) -> untyped + def wait_for_idle: (?connections: ::Integer, ?duration: ::Float, ?timeout: (::Numeric | Deadline)) -> untyped def idle?: (?::Integer connections) -> untyped diff --git a/sig/ferrum/node.rbs b/sig/ferrum/node.rbs index bd9d77fe..0cf28141 100644 --- a/sig/ferrum/node.rbs +++ b/sig/ferrum/node.rbs @@ -47,9 +47,9 @@ module Ferrum def blur: () -> self - def type: (*untyped keys) -> self + def type: (*untyped keys, ?timeout: (::Numeric | Deadline)?) -> self - def click: (?mode: ::Symbol, ?keys: ::Array[::Symbol], ?offset: ::Hash[::Symbol, untyped], ?delay: ::Numeric, ?wait: ::Numeric) -> self + def click: (?mode: ::Symbol, ?keys: ::Array[::Symbol], ?offset: ::Hash[::Symbol, untyped], ?delay: ::Numeric, ?navigation_wait: ::Numeric, ?wait: ::Numeric?, ?timeout: (::Numeric | Deadline)?) -> self def hover: () -> bot diff --git a/sig/ferrum/page.rbs b/sig/ferrum/page.rbs index 1884aa85..e190b823 100644 --- a/sig/ferrum/page.rbs +++ b/sig/ferrum/page.rbs @@ -1,6 +1,6 @@ module Ferrum class Page - GOTO_WAIT: ::Float + NAVIGATION_WAIT: ::Float extend Forwardable @@ -14,7 +14,7 @@ module Ferrum attr_accessor referrer: String? attr_reader context_id: String? attr_reader target_id: String - attr_reader event: Utils::Event + attr_reader idle: Concurrency::Cell attr_reader tracing: Page::Tracing attr_reader mouse: Mouse attr_reader keyboard: Keyboard @@ -33,7 +33,7 @@ module Ferrum @options: Browser::Options @frames: ::Concurrent::Map[String, Frame] @main_frame: Frame - @event: Utils::Event + @idle: Concurrency::Cell @mouse: Mouse @keyboard: Keyboard @headers: Headers @@ -49,7 +49,7 @@ module Ferrum def context: () -> Context? - def go_to: (?String? url, ?on_timeout: (:ignore | :raise)?) -> String? + def go_to: (?String? url, ?on_timeout: (:ignore | :raise)?, ?timeout: (::Numeric | Deadline)?) -> String? alias goto go_to alias go go_to @@ -69,20 +69,20 @@ module Ferrum def window_bounds=: (Hash[Symbol, (::Integer | String)] bounds) -> Hash[String, untyped] - def refresh: () -> void + def refresh: (?timeout: (::Numeric | Deadline)?) -> void alias reload refresh def stop: () -> Hash[String, untyped] - def back: () -> void + def back: (?timeout: (::Numeric | Deadline)?) -> void - def forward: () -> void + def forward: (?timeout: (::Numeric | Deadline)?) -> void - def wait_for_reload: (?::Numeric timeout) -> void + def wait_for_reload: (?(::Numeric | Deadline) timeout) -> void - def wait_for_navigation: (?::Numeric wait) { () -> void } -> bool + def wait_for_navigation: (?::Numeric navigation_wait, ?timeout: (::Numeric | Deadline)?) { () -> void } -> bool - def wait_for_navigation!: (?::Numeric wait) { () -> void } -> void + def wait_for_navigation!: (?::Numeric navigation_wait, ?timeout: (::Numeric | Deadline)?) { () -> void } -> void def bypass_csp: (?enabled: bool) -> Hash[String, untyped] @@ -94,7 +94,7 @@ module Ferrum def set_window_bounds: (Hash[Symbol, (::Integer | String)] bounds) -> Hash[String, untyped] - def command: (String method, ?wait: ::Numeric?, ?slowmoable: bool, ?timeout: ::Numeric?, **untyped) -> Hash[String, untyped] + def command: (String method, ?slowmoable: bool, ?timeout: (::Numeric | Deadline)?, **untyped) -> Hash[String, untyped] def on: (:dialog) { (Dialog) -> void } -> Integer | (:console) { (Console::Message) -> void } -> Integer @@ -123,7 +123,7 @@ module Ferrum def prepare_page: () -> void - def history_navigate: (::Integer delta) -> Hash[String, untyped] + def history_navigate: (delta: ::Integer, ?timeout: (::Numeric | Deadline)?) -> void def combine_url!: (String? url) -> String diff --git a/sig/ferrum/page/tracing.rbs b/sig/ferrum/page/tracing.rbs index 6b060e27..4e708fd4 100644 --- a/sig/ferrum/page/tracing.rbs +++ b/sig/ferrum/page/tracing.rbs @@ -9,9 +9,12 @@ module Ferrum DEFAULT_TRACE_CONFIG: { includedCategories: untyped, excludedCategories: untyped } + @pending: Concurrency::Cell + @failure: StandardError? + def initialize: (untyped page) -> void - def record: (?path: untyped?, ?encoding: ::Symbol, ?timeout: untyped?, ?trace_config: untyped?, ?screenshots: bool) { () -> untyped } -> untyped + def record: (?path: untyped?, ?encoding: ::Symbol, ?timeout: (::Numeric | Deadline)?, ?trace_config: untyped?, ?screenshots: bool) { () -> untyped } -> untyped private diff --git a/sig/ferrum/target.rbs b/sig/ferrum/target.rbs index a8b908dc..e007a138 100644 --- a/sig/ferrum/target.rbs +++ b/sig/ferrum/target.rbs @@ -61,7 +61,7 @@ module Ferrum def maybe_sleep_if_new_window: () -> void - def command: (*untyped) -> (bool | Hash[String, untyped]) + def command: (String method, ?timeout: (::Numeric | Deadline)?, **untyped params) -> Hash[String, untyped] private diff --git a/sig/ferrum/utils/deprecate.rbs b/sig/ferrum/utils/deprecate.rbs index e58cf030..c12c2901 100644 --- a/sig/ferrum/utils/deprecate.rbs +++ b/sig/ferrum/utils/deprecate.rbs @@ -5,6 +5,10 @@ module Ferrum def self?.warn: (::String message, ?user_code: bool) -> void + def self?.renamed_env: (::String name, Array[::String] legacy, ::Float default) -> ::Float + + def self?.renamed_keyword: (Symbol old, Symbol new, untyped value, untyped current) -> untyped + def self?.reset!: () -> void def self?.skip?: (::String path, bool user_code) -> bool diff --git a/sig/ferrum/utils/event.rbs b/sig/ferrum/utils/event.rbs deleted file mode 100644 index b6fc27cd..00000000 --- a/sig/ferrum/utils/event.rbs +++ /dev/null @@ -1,11 +0,0 @@ -module Ferrum - module Utils - class Event < ::Concurrent::Event - @iteration: ::Integer - - def iteration: () -> ::Integer - - def reset: () -> ::Integer - end - end -end diff --git a/sig/ferrum/worker.rbs b/sig/ferrum/worker.rbs index 04f24258..29c3869c 100644 --- a/sig/ferrum/worker.rbs +++ b/sig/ferrum/worker.rbs @@ -24,7 +24,7 @@ module Ferrum def timeout: () -> ::Numeric - def command: (*untyped) -> (bool | Hash[String, untyped]) + def command: (String method, ?timeout: (::Numeric | Deadline)?, **untyped params) -> Hash[String, untyped] def main_frame: () -> nil @@ -40,7 +40,7 @@ module Ferrum def prepare: () -> void - def execution_id!: () -> untyped + def execution_id!: (?(::Numeric | Deadline)? timeout) -> untyped def execution_id=: (untyped value) -> untyped end diff --git a/spec/browser/options_spec.rb b/spec/browser/options_spec.rb deleted file mode 100644 index b37966a8..00000000 --- a/spec/browser/options_spec.rb +++ /dev/null @@ -1,25 +0,0 @@ -# frozen_string_literal: true - -describe Ferrum::Browser::Options do - describe "#protocol_timeout" do - it "defaults to a low value, sufficient for internal CDP bookkeeping" do - options = described_class.new - - expect(options.protocol_timeout).to eq(Ferrum::Browser::Options::DEFAULT_PROTOCOL_TIMEOUT) - end - - it "does not shrink when :timeout is lowered" do - options = described_class.new(timeout: 0.1) - - expect(options.timeout).to eq(0.1) - expect(options.protocol_timeout).to eq(Ferrum::Browser::Options::DEFAULT_PROTOCOL_TIMEOUT) - end - - it "is configurable independently of :timeout" do - options = described_class.new(timeout: 10, protocol_timeout: 60) - - expect(options.timeout).to eq(10) - expect(options.protocol_timeout).to eq(60) - end - end -end diff --git a/spec/client_spec.rb b/spec/client_spec.rb index 8396926e..791f4cbe 100644 --- a/spec/client_spec.rb +++ b/spec/client_spec.rb @@ -22,6 +22,11 @@ def evaluate_in_background(page, expression, timeout:) .to raise_error(Ferrum::TimeoutError, /a response to Page.navigate \(session .+\)/) end + it "rejects the removed async: instead of sending it to the browser" do + expect { browser.command("Browser.getVersion", async: true) } + .to raise_error(ArgumentError, "async: is removed, use notify to send Browser.getVersion without waiting") + end + it "raises NoSuchTargetError for an unknown target" do expect { page.command("Target.activateTarget", targetId: "missing") } .to raise_error(Ferrum::NoSuchTargetError) @@ -29,7 +34,7 @@ def evaluate_in_background(page, expression, timeout:) end context "when the connection is dead" do - let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 10, protocol_timeout: 10) } + let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 10) } after { remote.quit } @@ -66,14 +71,16 @@ def evaluate_in_background(page, expression, timeout:) end context "when the browser stops reading", skip: Ferrum::Utils::Platform.windows? do - let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 1, protocol_timeout: 1) } + let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 1) } + + before { stub_const("Ferrum::Client::WebSocket::WRITE_TIMEOUT", 1) } after do Process.kill("CONT", remote.process.pid) remote.quit end - it "closes the connection once the browser takes no data for protocol_timeout", :launch do + it "closes the connection once the browser takes no data for the write timeout", :launch do page = remote.create_page Process.kill("STOP", remote.process.pid) @@ -97,9 +104,9 @@ def evaluate_in_background(page, expression, timeout:) end end - context "when the browser stops reading for less than protocol_timeout", + context "when the browser stops reading for less than the write timeout", skip: Ferrum::Utils::Platform.windows? do - let(:remote) { Ferrum::Browser.new(base_url: base_url, protocol_timeout: 5) } + let(:remote) { Ferrum::Browser.new(base_url: base_url) } after do Process.kill("CONT", remote.process.pid) @@ -125,7 +132,7 @@ def evaluate_in_background(page, expression, timeout:) end describe "#on" do - let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 3, protocol_timeout: 3) } + let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 3) } after { remote.quit } @@ -144,7 +151,7 @@ def evaluate_in_background(page, expression, timeout:) end describe "#close" do - let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 10, protocol_timeout: 10) } + let(:remote) { Ferrum::Browser.new(base_url: base_url, timeout: 10) } after { remote.quit } diff --git a/spec/context_spec.rb b/spec/context_spec.rb index b44a969b..b55d9e1d 100644 --- a/spec/context_spec.rb +++ b/spec/context_spec.rb @@ -139,7 +139,7 @@ target = wait_for_target(&:service_worker?) wait_for do - browser.client.command("Target.getTargets")["targetInfos"] + browser.command("Target.getTargets")["targetInfos"] .select { |info| info["type"] == "service_worker" }.map { |info| info["attached"] } end.to eq([false]) expect(target.session_id).to be_nil @@ -248,8 +248,8 @@ expect(context.dispose).to be(true) expect(browser.contexts[context.id]).to be_nil - expect(browser.client.command("Target.getBrowserContexts")["browserContextIds"]).not_to include(context.id) - target_ids = browser.client.command("Target.getTargets")["targetInfos"].map { |info| info["targetId"] } + expect(browser.command("Target.getBrowserContexts")["browserContextIds"]).not_to include(context.id) + target_ids = browser.command("Target.getTargets")["targetInfos"].map { |info| info["targetId"] } expect(target_ids).not_to include(page.target_id) end @@ -266,6 +266,21 @@ end end + describe "#target" do + it "returns the target with the given id" do + context = browser.contexts.create + page = context.create_page + + expect(context.target(page.target_id).page).to be(page) + end + + it "returns nil for a target of another context" do + context = browser.contexts.create + + expect(context.target(browser.contexts.create.create_page.target_id)).to be_nil + end + end + describe "#target?" do it "returns true for a target of the context" do context = browser.contexts.create diff --git a/spec/contexts_spec.rb b/spec/contexts_spec.rb index 4e25e91c..1002399f 100644 --- a/spec/contexts_spec.rb +++ b/spec/contexts_spec.rb @@ -113,7 +113,7 @@ context = browser.contexts.create expect(context).not_to be_implicit - expect(browser.client.command("Target.getBrowserContexts")["browserContextIds"]).to include(context.id) + expect(browser.command("Target.getBrowserContexts")["browserContextIds"]).to include(context.id) end end @@ -124,7 +124,7 @@ expect(browser.contexts.dispose(context.id)).to be(true) expect(browser.contexts[context.id]).to be_nil - expect(browser.client.command("Target.getBrowserContexts")["browserContextIds"]).not_to include(context.id) + expect(browser.command("Target.getBrowserContexts")["browserContextIds"]).not_to include(context.id) end it "returns nil for an unknown context" do diff --git a/spec/cookies_spec.rb b/spec/cookies_spec.rb index 50618367..f2fa2da9 100644 --- a/spec/cookies_spec.rb +++ b/spec/cookies_spec.rb @@ -2,6 +2,12 @@ describe Ferrum::Cookies do describe "#each" do + it "raises once the surrounding Deadline.within block is used up" do + Ferrum::Deadline.within(0) do + expect { browser.cookies.each { nil } }.to raise_error(Ferrum::TimeoutError) + end + end + context "when a block is given" do it "yields each cookie" do browser.go_to("/set_cookie") diff --git a/spec/frame/dom_spec.rb b/spec/frame/dom_spec.rb index 6102e970..22bf1abc 100644 --- a/spec/frame/dom_spec.rb +++ b/spec/frame/dom_spec.rb @@ -372,6 +372,15 @@ end describe "#wait_for_selector" do + it "returns nil at the timeout when a poll hangs" do + page.go_to("/with_js") + hang_commands + started = Ferrum::Utils::ElapsedTime.monotonic_time + + expect(page.wait_for_selector(css: "#nope", timeout: 0.3)).to be_nil + expect(Ferrum::Utils::ElapsedTime.elapsed_time(started)).to be < 0.8 + end + it "returns the node immediately when it's already present" do page.go_to("/with_js") diff --git a/spec/keyboard_spec.rb b/spec/keyboard_spec.rb index 27aafef1..505339ff 100644 --- a/spec/keyboard_spec.rb +++ b/spec/keyboard_spec.rb @@ -36,6 +36,27 @@ describe "#type" do before { browser.go_to("/type") } + it "raises when the commands it sends take longer than the timeout together" do + browser.at_css("#empty_input").focus + + with_timeout(0.5) do + slow_commands(0.3) + expect { browser.keyboard.type("Hello") }.to raise_error(Ferrum::TimeoutError) + end + end + + it "types a text that takes longer than the page timeout within the given timeout" do + input = browser.at_css("#empty_input") + input.focus + + with_timeout(0.5) do + slow_commands(0.1) + browser.keyboard.type("Hello", timeout: 5) + end + + expect(input.value).to eq("Hello") + end + it "types text into the focused element" do input = browser.at_css("#empty_input").focus @@ -88,6 +109,12 @@ expect(browser.keyboard.held_modifiers).to eq(0) end + it "releases the modifier when the block outlives the surrounding Deadline.within" do + Ferrum::Deadline.within(0.3) { browser.keyboard.hold(:shift) { sleep(0.5) } } + + expect(browser.evaluate("events")).to eq(%w[keydown:true:false keyup:false:false]) + end + it "presses nested modifiers once" do browser.keyboard.hold(:shift) do browser.keyboard.hold(:shift, :ctrl) { browser.mouse.move(x: 10, y: 10) } diff --git a/spec/mouse_spec.rb b/spec/mouse_spec.rb index 662e6a2e..e692b2c1 100644 --- a/spec/mouse_spec.rb +++ b/spec/mouse_spec.rb @@ -17,6 +17,39 @@ def record_mouse_buttons end describe "#click" do + it "bounds the click by the given timeout" do + browser.go_to("/click_coordinates") + + slow_commands(0.3) + + expect { browser.mouse.click(x: 100, y: 150, timeout: 0.5) }.to raise_error(Ferrum::TimeoutError) + end + + it "lets the given timeout exceed the page timeout" do + browser.go_to("/click_coordinates") + + with_timeout(0.5) do + slow_commands(0.3) + expect { browser.mouse.click(x: 100, y: 150, timeout: 5) }.not_to raise_error + end + end + + it "raises when the commands it sends take longer than the timeout together" do + browser.go_to("/click_coordinates") + + with_timeout(0.5) do + slow_commands(0.3) + expect { browser.mouse.click(x: 100, y: 150) }.to raise_error(Ferrum::TimeoutError) + end + end + + it "warns when given the deprecated wait:" do + browser.go_to("/click_coordinates") + + expect { browser.mouse.click(x: 100, y: 150, wait: 0) } + .to output(/:wait is renamed to :navigation_wait/).to_stderr + end + it "supports clicking precise coordinates" do browser.go_to("/click_coordinates") browser.mouse.click(x: 100, y: 150) @@ -154,6 +187,16 @@ def record_mouse_buttons end describe "#up" do + it "takes the deprecated wait: as navigation_wait:" do + Ferrum::Utils::Deprecate.reset! + record_mouse_buttons + + expect { browser.mouse.down.up(wait: 0.1) }.to output(/:wait is renamed to :navigation_wait/).to_stderr + expect(browser.evaluate("events")).to eq(%w[mousedown:0:1 mouseup:0:1]) + ensure + Ferrum::Utils::Deprecate.reset! + end + it "releases the left button by default" do record_mouse_buttons @@ -180,6 +223,24 @@ def record_mouse_buttons end describe "#move" do + it "raises when the commands it sends take longer than the timeout together" do + browser.go_to("/click_coordinates") + + with_timeout(0.5) do + slow_commands(0.3) + expect { browser.mouse.move(x: 100, y: 150, steps: 3) }.to raise_error(Ferrum::TimeoutError) + end + end + + it "moves in steps that take longer than the page timeout within the given timeout" do + browser.go_to("/click_coordinates") + + with_timeout(0.5) do + slow_commands(0.3) + expect { browser.mouse.move(x: 100, y: 150, steps: 3, timeout: 5) }.not_to raise_error + end + end + it "sends modifiers" do browser.go_to("/simple") browser.execute(<<~JS) diff --git a/spec/node_spec.rb b/spec/node_spec.rb index 57f4ed7f..244d8f5e 100644 --- a/spec/node_spec.rb +++ b/spec/node_spec.rb @@ -138,6 +138,35 @@ end describe "#click" do + it "bounds the click by the given timeout" do + browser.go_to("/with_js") + node = browser.at_css("#remove") + + slow_commands(0.3) + + expect { node.click(timeout: 0.5) }.to raise_error(Ferrum::TimeoutError) + end + + it "lets the given timeout exceed the page timeout" do + browser.go_to("/with_js") + node = browser.at_css("#remove") + + with_timeout(0.5) do + slow_commands(0.3) + expect { node.click(timeout: 5) }.not_to raise_error + end + end + + it "raises when the commands it sends take longer than the timeout together" do + browser.go_to("/with_js") + node = browser.at_css("#remove") + + with_timeout(0.5) do + slow_commands(0.3) + expect { node.click }.to raise_error(Ferrum::TimeoutError) + end + end + it "hovers an element before clicking it" do browser.go_to("/with_js") @@ -156,14 +185,20 @@ expect(browser.network.traffic.length).to be <= 5 end - it "returns without waiting for the navigation to load when wait is 0" do + it "warns when given the deprecated wait:" do + browser.go_to("/link_with_ping") + + expect { browser.at_css("a").click(wait: 0) }.to output(/:wait is renamed to :navigation_wait/).to_stderr + end + + it "returns without waiting for the navigation to load when navigation_wait is 0" do browser.go_to("/") browser.execute(<<~JS) document.body.innerHTML = 'Slow' JS started_at = Ferrum::Utils::ElapsedTime.monotonic_time - browser.at_css("#slow").click(wait: 0) + browser.at_css("#slow").click(navigation_wait: 0) expect(Ferrum::Utils::ElapsedTime.monotonic_time - started_at).to be < 1 expect(browser.network.wait_for_idle(timeout: 5)).to be(true) @@ -1249,6 +1284,15 @@ def expect_select_to_clear(options) describe "#evaluate" do before { browser.go_to("/index") } + it "bounds looking up the node by the given timeout" do + node = browser.at_xpath(".//a") + hang_commands + started = Ferrum::Utils::ElapsedTime.monotonic_time + + expect { node.evaluate("this.tagName", timeout: 0.3) }.to raise_error(Ferrum::TimeoutError) + expect(Ferrum::Utils::ElapsedTime.elapsed_time(started)).to be < 0.8 + end + it "binds this to the node" do expect(browser.at_xpath(".//a").evaluate("this.getAttribute('href')")).to eq("js_redirect") end diff --git a/spec/page_spec.rb b/spec/page_spec.rb index cd3b4d65..a2206534 100644 --- a/spec/page_spec.rb +++ b/spec/page_spec.rb @@ -57,6 +57,30 @@ end end + it "bounds the navigation by the given timeout instead of the page timeout" do + started = Ferrum::Utils::ElapsedTime.monotonic_time + + expect { page.go_to("/really_slow", timeout: 0.5, on_timeout: :raise) } + .to raise_error(Ferrum::NavigationTimeoutError) + expect(Ferrum::Utils::ElapsedTime.elapsed_time(started)).to be < 2 + end + + it "is bound by the surrounding Deadline.within block" do + started = Ferrum::Utils::ElapsedTime.monotonic_time + + Ferrum::Deadline.within(0.5) do + expect { page.go_to("/really_slow", on_timeout: :raise) }.to raise_error(Ferrum::NavigationTimeoutError) + end + expect(Ferrum::Utils::ElapsedTime.elapsed_time(started)).to be < 2 + end + + it "raises once the surrounding Deadline.within block runs out" do + Ferrum::Deadline.within(0.5) do + expect { page.go_to("/really_slow") }.to raise_error(Ferrum::NavigationTimeoutError) + expect { page.go_to("/") }.to raise_error(Ferrum::NavigationTimeoutError) + end + end + it "rejects an unknown on_timeout" do expect { browser.go_to("/", on_timeout: :wait) } .to raise_error(ArgumentError, "on_timeout should be one of [:ignore, :raise], got :wait") @@ -160,6 +184,12 @@ expect(page.evaluate("window.marker")).to be_nil expect(page.body).to include("slow page") end + + it "raises when the given timeout is already used up" do + page.go_to("/slow") + + expect { page.refresh(timeout: 0) }.to raise_error(Ferrum::TimeoutError) + end end describe "#stop" do @@ -220,6 +250,35 @@ end describe "#back" do + it "raises when the history query and the navigation take longer than the timeout together" do + page.go_to("/simple") + page.go_to("/with_js") + + with_timeout(0.5) do + slow_commands(0.3) + expect { page.back }.to raise_error(Ferrum::TimeoutError) + end + end + + it "bounds the history query and the navigation by the given timeout" do + page.go_to("/simple") + page.go_to("/with_js") + + slow_commands(0.3) + + expect { page.back(timeout: 0.5) }.to raise_error(Ferrum::TimeoutError) + end + + it "lets the given timeout exceed the page timeout" do + page.go_to("/simple") + page.go_to("/with_js") + + with_timeout(0.5) do + slow_commands(0.3) + expect { page.back(timeout: 5) }.not_to raise_error + end + end + it "goes back when history state has been pushed" do page.go_to @@ -232,6 +291,16 @@ end describe "#forward" do + it "bounds the history query and the navigation by the given timeout" do + page.go_to("/simple") + page.go_to("/with_js") + page.back + + slow_commands(0.3) + + expect { page.forward(timeout: 0.5) }.to raise_error(Ferrum::TimeoutError) + end + it "goes forward when history state is used" do page.go_to @@ -282,6 +351,10 @@ expect(page.wait_for_navigation { page.execute("location.href = '/really_slow'") }).to be(false) end + + it "returns false when the navigation doesn't finish within the given timeout" do + expect(page.wait_for_navigation(timeout: 0.5) { page.execute("location.href = '/really_slow'") }).to be(false) + end end describe "#wait_for_navigation!" do @@ -375,20 +448,15 @@ expect(result.dig("result", "value")).to eq(2) end - it "waits for the navigation the command starts with wait:" do - page.go_to - loaded = false - page.on("Page.loadEventFired") { loaded = true } - - page.command("Page.navigate", wait: 1, url: base_url("/slow")) - - expect(loaded).to be(true) - end - it "raises when the response doesn't come within timeout:" do expect { page.command("Runtime.evaluate", timeout: 0.5, expression: "new Promise(() => {})", awaitPromise: true) } .to raise_error(Ferrum::TimeoutError) end + + it "rejects the removed wait: instead of sending it to the browser" do + expect { page.command("Page.navigate", wait: 1, url: base_url("/")) } + .to raise_error(ArgumentError, "wait: is removed, wrap Page.navigate in wait_for_navigation instead") + end end describe "#timeout=" do diff --git a/spec/support/global_helpers.rb b/spec/support/global_helpers.rb index a70570b9..e645e8df 100644 --- a/spec/support/global_helpers.rb +++ b/spec/support/global_helpers.rb @@ -68,6 +68,20 @@ def with_timeout(new_timeout) browser.timeout = old_timeout end + def slow_commands(delay) + allow(browser.page.client).to receive(:command).and_wrap_original do |original, *args, **options| + sleep(delay) + original.call(*args, **options) + end + end + + def hang_commands + allow(browser.page.client).to receive(:command) do |*, timeout:, **| + sleep(Ferrum::Deadline.wrap(timeout).remaining) + raise Ferrum::TimeoutError + end + end + def with_external_browser(host: "127.0.0.1", port: 32_001, incognito: true) options = Ferrum::Browser::Options.new(host: host, port: port, window_size: [1400, 1400], headless: true, incognito: incognito) diff --git a/spec/unit/browser/options_spec.rb b/spec/unit/browser/options_spec.rb new file mode 100644 index 00000000..7d59804f --- /dev/null +++ b/spec/unit/browser/options_spec.rb @@ -0,0 +1,49 @@ +# frozen_string_literal: true + +require "rbconfig" + +describe Ferrum::Browser::Options do + before { Ferrum::Utils::Deprecate.reset! } + after { Ferrum::Utils::Deprecate.reset! } + + describe ".new" do + it "warns that :protocol_timeout is ignored" do + expect { described_class.new(protocol_timeout: 10) } + .to output(/:protocol_timeout is ignored.*called from #{__FILE__}/m).to_stderr + end + end + + describe "#protocol_timeout" do + it "warns and returns the fixed timeout of internal commands" do + options = described_class.new + value = nil + + expect { value = options.protocol_timeout }.to output(/:protocol_timeout is ignored/).to_stderr + expect(value).to eq(Ferrum::Client::PROTOCOL_TIMEOUT) + end + end + + describe "#protocol_timeout=" do + it "warns and leaves the timeout as it is" do + options = described_class.new(timeout: 3) + + expect { options.protocol_timeout = 10 }.to output(/:protocol_timeout is ignored/).to_stderr + expect(options.timeout).to eq(3) + end + end + + describe "#timeout" do + def default_timeout(env) + script = 'require "ferrum"; print Ferrum::Browser::Options.new.timeout' + IO.popen(env, [RbConfig.ruby, "-I", "lib", "-e", script], &:read) + end + + it "keeps the fraction of FERRUM_DEFAULT_TIMEOUT" do + expect(default_timeout("FERRUM_DEFAULT_TIMEOUT" => "0.5")).to eq("0.5") + end + + it "defaults to 5 seconds" do + expect(default_timeout("FERRUM_DEFAULT_TIMEOUT" => nil)).to eq("5.0") + end + end +end diff --git a/spec/unit/client_spec.rb b/spec/unit/client_spec.rb index 15875d90..cdec7732 100644 --- a/spec/unit/client_spec.rb +++ b/spec/unit/client_spec.rb @@ -51,51 +51,33 @@ def +(other) describe "#send_message" do before do - client.instance_variable_set(:@pendings, Concurrent::Hash.new) + client.instance_variable_set(:@responses, Ferrum::Concurrency::AwaitableMap.new) end - it "waits up to protocol_timeout, independent of the page-level timeout" do - options = Ferrum::Browser::Options.new(timeout: 100, protocol_timeout: 0.05) - client.instance_variable_set(:@options, options) - client.instance_variable_set( - :@ws, instance_double(Ferrum::Client::WebSocket, send_message: true, messages: Queue.new) - ) - - expect do - client.send_message(client.build_message("Target.createTarget", {}), async: false) - end.to raise_error(Ferrum::TimeoutError) - end - - it "does not time out early just because the page-level timeout is low" do - options = Ferrum::Browser::Options.new(timeout: 0.05, protocol_timeout: 100) - client.instance_variable_set(:@options, options) - - # Simulates the response arriving after the (low) page-level timeout has - # already elapsed, but well inside the (high) protocol timeout. + it "waits for a response that arrives within the given timeout" do ws = instance_double(Ferrum::Client::WebSocket, messages: Queue.new) allow(ws).to receive(:send_message) do |message| Thread.new do sleep 0.1 - client.instance_variable_get(:@pendings)[message[:id]].set({ "result" => { "ok" => true } }) + client.instance_variable_get(:@responses).fulfill(message[:id], { "result" => { "ok" => true } }) end true end client.instance_variable_set(:@ws, ws) message = client.build_message("Target.createTarget", {}) - expect(client.send_message(message, async: false)).to eq({ "ok" => true }) + expect(client.send_message(message, timeout: 5)).to eq({ "ok" => true }) end - it "honors an explicit per-call timeout: over protocol_timeout" do - options = Ferrum::Browser::Options.new(protocol_timeout: 100) - client.instance_variable_set(:@options, options) + it "times out once the given timeout is up" do + client.instance_variable_set(:@options, Ferrum::Browser::Options.new(timeout: 100)) client.instance_variable_set( :@ws, instance_double(Ferrum::Client::WebSocket, send_message: true, messages: Queue.new) ) message = client.build_message("Page.navigate", {}) expect do - client.send_message(message, async: false, timeout: 0.05) + client.send_message(message, timeout: 0.05) end.to raise_error(Ferrum::TimeoutError) end @@ -106,8 +88,20 @@ def +(other) client.instance_variable_set(:@ws, ws) message = client.build_message("Page.navigate", {}) - expect { client.send_message(message, async: false) }.to raise_error(Ferrum::DeadBrowserError) - expect(client.instance_variable_get(:@pendings)).to be_empty + expect { client.send_message(message, timeout: 5) }.to raise_error(Ferrum::DeadBrowserError) + expect(client.instance_variable_get(:@responses)).not_to be_expecting(message[:id]) + end + end + + describe "#notify" do + it "sends the message without waiting, whatever the surrounding deadline" do + ws = instance_double(Ferrum::Client::WebSocket) + allow(ws).to receive(:send_message) + client.instance_variable_set(:@ws, ws) + + Ferrum::Deadline.within(0) { expect(client.notify("Page.enable")).to be(true) } + + expect(ws).to have_received(:send_message).with(hash_including(method: "Page.enable")) end end end diff --git a/spec/unit/concurrency/awaitable_map_spec.rb b/spec/unit/concurrency/awaitable_map_spec.rb new file mode 100644 index 00000000..140c52a0 --- /dev/null +++ b/spec/unit/concurrency/awaitable_map_spec.rb @@ -0,0 +1,117 @@ +# frozen_string_literal: true + +describe Ferrum::Concurrency::AwaitableMap do + subject(:map) { described_class.new } + + describe "#await" do + it "returns the value at once when the key is stored" do + map["a"] = 1 + + expect(map.await("a", 0)).to eq(1) + end + + it "returns nil when the timeout is reached" do + expect(map.await("a", 0.05)).to be_nil + end + + it "wakes up when another thread stores the key" do + Thread.new do + sleep(0.05) + map["a"] = 42 + end + + expect(map.await("a", 5)).to eq(42) + end + + it "keeps waiting while other keys are stored" do + Thread.new do + sleep(0.05) + map["b"] = 1 + sleep(0.05) + map["a"] = 2 + end + + expect(map.await("a", 5)).to eq(2) + end + + it "takes a deadline" do + expect(map.await("a", Ferrum::Deadline.after(0.05))).to be_nil + end + end + + describe "#fulfill" do + it "stores the value of an expected key" do + map.expect("a") + + expect(map.fulfill("a", 1)).to be(true) + expect(map["a"]).to eq(1) + end + + it "drops the value of a key nobody expects" do + expect(map.fulfill("a", 1)).to be(false) + expect(map).not_to be_key("a") + end + + it "drops the value once the key was deleted" do + map.expect("a") + map.delete("a") + + expect(map.fulfill("a", 1)).to be(false) + end + + it "wakes up a thread waiting for the key" do + map.expect("a") + Thread.new do + sleep(0.05) + map.fulfill("a", 42) + end + + expect(map.await("a", 5)).to eq(42) + end + end + + describe "#expecting?" do + it "is true for an expected key until it is deleted" do + map.expect("a") + expect(map).to be_expecting("a") + + map.delete("a") + expect(map).not_to be_expecting("a") + end + end + + describe "#close" do + it "wakes up every waiter" do + waiters = Array.new(2) { Thread.new { map.await("a", 5) } } + sleep(0.05) + + map.close + + expect(waiters.map(&:value)).to eq([nil, nil]) + end + + it "makes await return at once" do + map.close + + expect(map.await("a", 5)).to be_nil + end + end + + describe "#delete" do + it "removes the key and returns its value" do + map["a"] = 1 + + expect(map.delete("a")).to eq(1) + expect(map).not_to be_key("a") + end + end + + describe "#values" do + it "returns the stored values" do + map["a"] = 1 + map["b"] = 2 + + expect(map.values).to eq([1, 2]) + end + end +end diff --git a/spec/unit/concurrency/cell_spec.rb b/spec/unit/concurrency/cell_spec.rb new file mode 100644 index 00000000..7ab7645b --- /dev/null +++ b/spec/unit/concurrency/cell_spec.rb @@ -0,0 +1,79 @@ +# frozen_string_literal: true + +describe Ferrum::Concurrency::Cell do + subject(:cell) { described_class.new } + + describe ".set" do + it "builds a cell that holds the value" do + cell = described_class.set(1) + + expect(cell).to be_set + expect(cell.value).to eq(1) + end + end + + describe "#wait" do + it "returns true at once when a value is stored" do + cell.set(1) + + expect(cell.wait(0)).to be(true) + end + + it "returns false when the timeout is reached" do + expect(cell.wait(0.05)).to be(false) + end + + it "wakes up when another thread sets a value" do + Thread.new do + sleep(0.05) + cell.set(42) + end + + expect(cell.wait(5)).to be(true) + expect(cell.value).to eq(42) + end + + it "wakes up every waiter when the cell is closed" do + waiters = Array.new(2) { Thread.new { cell.wait(5) } } + sleep(0.05) + + cell.close + + expect(waiters.map(&:value)).to eq([false, false]) + end + + it "doesn't block once closed" do + cell.close + + expect(cell.wait(5)).to be(false) + end + + it "takes a deadline" do + expect(cell.wait(Ferrum::Deadline.after(0.05))).to be(false) + end + end + + describe "#clear" do + it "empties the cell" do + cell.set(1) + + cell.clear + + expect(cell).not_to be_set + expect(cell.value).to be_nil + end + + it "returns the next generation" do + expect { cell.clear }.to change(cell, :generation).by(1) + end + end + + describe "#set" do + it "replaces the stored value" do + cell.set(1) + cell.set(2) + + expect(cell.value).to eq(2) + end + end +end diff --git a/spec/unit/concurrency/dispatcher_spec.rb b/spec/unit/concurrency/dispatcher_spec.rb new file mode 100644 index 00000000..ccb74bfc --- /dev/null +++ b/spec/unit/concurrency/dispatcher_spec.rb @@ -0,0 +1,115 @@ +# frozen_string_literal: true + +describe Ferrum::Concurrency::Dispatcher do + subject(:dispatcher) { described_class.new } + + after { dispatcher.close } + + describe "#publish" do + it "delivers the payload to the callbacks of the key" do + called = Queue.new + dispatcher.on("greeting") { |payload| called << payload } + + dispatcher.publish("greeting", "hello") + + expect(called.pop).to eq("hello") + end + + it "passes the index of the callback and the total" do + called = Queue.new + dispatcher.on("greeting") { |_, index, total| called << [index, total] } + dispatcher.on("greeting") { |_, index, total| called << [index, total] } + + dispatcher.publish("greeting", "hello") + + expect([called.pop, called.pop]).to eq([[0, 2], [1, 2]]) + end + + it "delivers in order within a lane" do + called = Queue.new + dispatcher.on("count") { |payload| called << payload } + + 3.times { |i| dispatcher.publish("count", i) } + + expect(Array.new(3) { called.pop }).to eq([0, 1, 2]) + end + + it "delivers on a lane while another one is blocked" do + release = Queue.new + called = Queue.new + dispatcher.on("slow") { release.pop } + dispatcher.on("fast") { called << :fast } + + dispatcher.publish("slow", nil, lane: :regular) + dispatcher.publish("fast", nil, lane: :priority) + + expect(called.pop).to eq(:fast) + release << :go + end + + it "keeps delivering after a callback raises" do + called = Queue.new + errors = [] + dispatcher = described_class.new(on_error: ->(key, e) { errors << [key, e.message] }) + dispatcher.on("boom") { raise "failed" } + dispatcher.on("boom") { called << :second } + + dispatcher.publish("boom", nil) + + expect(called.pop).to eq(:second) + expect(errors).to eq([%w[boom failed]]) + dispatcher.close + end + + it "raises for an unknown lane" do + expect { dispatcher.publish("greeting", "hello", lane: :urgent) }.to raise_error(KeyError) + end + end + + describe "#on" do + it "returns a different id for every callback" do + ids = Array.new(3) { dispatcher.on("greeting") { nil } } + + expect(ids.uniq.size).to eq(3) + end + end + + describe "#off" do + it "removes the callback with the given id after an earlier one is gone" do + first = dispatcher.on("greeting") { nil } + second = dispatcher.on("greeting") { nil } + + dispatcher.off("greeting", first) + dispatcher.off("greeting", second) + + expect(dispatcher).not_to be_subscribed("greeting") + end + + it "keeps the other callbacks" do + called = Queue.new + first = dispatcher.on("greeting") { called << :first } + dispatcher.on("greeting") { called << :second } + + dispatcher.off("greeting", first) + dispatcher.publish("greeting", nil) + + expect(called.pop).to eq(:second) + end + + it "does nothing for a key nobody subscribed to" do + expect(dispatcher.off("greeting", 1)).to be(true) + end + end + + describe "#clear" do + it "removes the callbacks of the keys the block selects" do + dispatcher.on("a:1") { nil } + dispatcher.on("b:2") { nil } + + dispatcher.clear { |k| k.end_with?("1") } + + expect(dispatcher).not_to be_subscribed("a:1") + expect(dispatcher).to be_subscribed("b:2") + end + end +end diff --git a/spec/unit/contexts_spec.rb b/spec/unit/contexts_spec.rb index 5c688deb..32f590ed 100644 --- a/spec/unit/contexts_spec.rb +++ b/spec/unit/contexts_spec.rb @@ -7,16 +7,16 @@ other = browser.contexts.create allow(browser.client).to receive(:command).and_call_original allow(browser.client).to receive(:command) - .with("Target.disposeBrowserContext", browserContextId: stuck.id) + .with("Target.disposeBrowserContext", timeout: anything, browserContextId: stuck.id) .and_raise(Ferrum::TimeoutError) expect { browser.reset }.to raise_error(Ferrum::TimeoutError) expect(browser.contexts[stuck.id]).to be_nil expect(browser.contexts[other.id]).to be_nil - expect(browser.client.command("Target.getBrowserContexts")["browserContextIds"]).not_to include(other.id) + expect(browser.command("Target.getBrowserContexts")["browserContextIds"]).not_to include(other.id) allow(browser.client).to receive(:command) - .with("Target.disposeBrowserContext", browserContextId: stuck.id) + .with("Target.disposeBrowserContext", timeout: anything, browserContextId: stuck.id) .and_raise(Ferrum::BrowserError, { "message" => "Disposal of browser context #{stuck.id} is already pending" }) expect { browser.reset }.not_to raise_error diff --git a/spec/unit/deadline_spec.rb b/spec/unit/deadline_spec.rb new file mode 100644 index 00000000..5c1f78f0 --- /dev/null +++ b/spec/unit/deadline_spec.rb @@ -0,0 +1,127 @@ +# frozen_string_literal: true + +describe Ferrum::Deadline do + describe ".after" do + it "starts a deadline that expires after the given seconds" do + deadline = described_class.after(0.05) + + expect(deadline).not_to be_expired + sleep(0.06) + expect(deadline).to be_expired + end + end + + describe ".wrap" do + it "starts a deadline for a number of seconds" do + expect(described_class.wrap(10).remaining).to be_within(0.5).of(10) + end + + it "returns a deadline as is" do + deadline = described_class.after(10) + + expect(described_class.wrap(deadline)).to be(deadline) + end + end + + describe ".within" do + it "bounds the deadlines made in the block" do + described_class.within(0.05) do + expect(described_class.wrap(10).remaining).to be < 1 + end + end + + it "doesn't change a deadline that expires sooner" do + described_class.within(10) do + expect(described_class.wrap(0.05).remaining).to be < 1 + end + end + + it "shares the deadline passed in when it expires first" do + deadline = described_class.after(0.05) + + described_class.within(10) { expect(described_class.wrap(deadline)).to be(deadline) } + end + + it "only shortens the deadline of an enclosing block" do + described_class.within(0.05) do + described_class.within(10) { expect(described_class.current.remaining).to be < 1 } + end + end + + it "restores the previous deadline afterwards" do + described_class.within(10) { nil } + + expect(described_class.current).to be_nil + end + + it "restores the previous deadline when the block raises" do + expect { described_class.within(10) { raise ArgumentError } }.to raise_error(ArgumentError) + expect(described_class.current).to be_nil + end + + it "returns what the block returns" do + expect(described_class.within(10) { :done }).to eq(:done) + end + + it "isn't inherited by other threads" do + seen = described_class.within(10) { Thread.new { described_class.current }.value } + + expect(seen).to be_nil + end + end + + describe ".suspend" do + it "lifts the deadline of the enclosing block" do + described_class.within(0.05) do + described_class.suspend { expect(described_class.wrap(10).remaining).to be > 9 } + end + end + + it "restores the deadline afterwards" do + described_class.within(10) do + described_class.suspend { nil } + + expect(described_class.current.remaining).to be_within(0.5).of(10) + end + end + + it "returns what the block returns" do + expect(described_class.suspend { :done }).to eq(:done) + end + end + + describe "#min" do + it "returns the deadline that expires first" do + sooner = described_class.after(1) + later = described_class.after(10) + + expect(sooner.min(later)).to be(sooner) + expect(later.min(sooner)).to be(sooner) + end + end + + describe "#remaining" do + it "counts down" do + deadline = described_class.after(10) + first = deadline.remaining + + sleep(0.02) + + expect(deadline.remaining).to be < first + end + + it "is zero once the deadline has expired" do + expect(described_class.after(-1).remaining).to eq(0.0) + end + end + + describe "#expired?" do + it "is false before the deadline" do + expect(described_class.after(10)).not_to be_expired + end + + it "is true after the deadline" do + expect(described_class.after(-1)).to be_expired + end + end +end diff --git a/spec/unit/downloads_spec.rb b/spec/unit/downloads_spec.rb new file mode 100644 index 00000000..728e0dc6 --- /dev/null +++ b/spec/unit/downloads_spec.rb @@ -0,0 +1,23 @@ +# frozen_string_literal: true + +describe Ferrum::Downloads do + describe "#wait" do + subject(:downloads) { described_class.allocate } + + let(:idle) { Ferrum::Concurrency::Cell.set(true) } + + before { downloads.instance_variable_set(:@idle, idle) } + + it "marks downloads idle again when none began" do + downloads.wait(0.05) + + expect(idle).to be_set + end + + it "leaves a download in flight as it is when the timeout is reached" do + downloads.wait(0.05) { idle.clear } + + expect(idle).not_to be_set + end + end +end diff --git a/spec/unit/page_spec.rb b/spec/unit/page_spec.rb new file mode 100644 index 00000000..d45e42f5 --- /dev/null +++ b/spec/unit/page_spec.rb @@ -0,0 +1,28 @@ +# frozen_string_literal: true + +describe Ferrum::Page do + describe "#wait_for_reload" do + subject(:page) { described_class.allocate } + + let(:idle) { Ferrum::Concurrency::Cell.set(true) } + + before { page.instance_variable_set(:@idle, idle) } + + it "marks the page idle again when no reload came" do + page.wait_for_reload(0.05) + + expect(idle).to be_set + end + + it "leaves the page loading when a navigation began meanwhile" do + Thread.new do + sleep(0.02) + idle.clear + end + + page.wait_for_reload(0.1) + + expect(idle).not_to be_set + end + end +end diff --git a/spec/unit/utils/deprecate_spec.rb b/spec/unit/utils/deprecate_spec.rb new file mode 100644 index 00000000..20483bcb --- /dev/null +++ b/spec/unit/utils/deprecate_spec.rb @@ -0,0 +1,38 @@ +# frozen_string_literal: true + +describe Ferrum::Utils::Deprecate do + describe ".renamed_env" do + around do |example| + keys = %w[FERRUM_NEW_WAIT FERRUM_OLD_WAIT] + saved = keys.to_h { |k| [k, ENV.fetch(k, nil)] } + keys.each { |k| ENV.delete(k) } + example.run + saved.each { |k, v| ENV[k] = v } + end + + def read + described_class.renamed_env("FERRUM_NEW_WAIT", %w[FERRUM_OLD_WAIT], 0.1) + end + + it "returns the default when none is set" do + expect(read).to eq(0.1) + end + + it "returns the current variable without warning" do + ENV["FERRUM_NEW_WAIT"] = "0.5" + expect { expect(read).to eq(0.5) }.not_to output.to_stderr + end + + it "returns the old variable and warns about it" do + ENV["FERRUM_OLD_WAIT"] = "0.3" + described_class.reset! + expect { expect(read).to eq(0.3) }.to output(/FERRUM_OLD_WAIT is replaced by FERRUM_NEW_WAIT/).to_stderr + end + + it "prefers the current variable over the old one" do + ENV["FERRUM_NEW_WAIT"] = "0.5" + ENV["FERRUM_OLD_WAIT"] = "0.3" + expect(read).to eq(0.5) + end + end +end