Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion lib/datadog/statsd/sender.rb
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ def flush(sync: false)
def rendez_vous
# could happen if #start hasn't be called
return unless message_queue
# no companion thread to answer, e.g. it could not be re-created
return unless sender_thread && sender_thread.alive?

# Initialize and get the thread's sync queue
queue = (@thread_class.current[:statsd_sync_queue] ||= @queue_class.new)
Expand Down Expand Up @@ -81,6 +83,9 @@ def add(message)
@message_queue = nil
message_buffer.reset
start
# the companion thread could not be re-created, don't queue
# messages that nothing will ever send
return if @done

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make rendezvous bypass the failed restart queue

When Thread.new fails during the restart and the caller subsequently invokes the public Statsd#sync_with_outbound_io, this return leaves the queue allocated by start non-nil even though it has no consumer. Forwarder#sync_with_outbound_io calls Sender#rendez_vous, whose message_queue guard therefore passes before it enqueues a reply queue and blocks on queue.pop forever; because this patch now hides the ThreadError and lets the caller continue, the failed-start state needs to be visible to rendez_vous (or that method must also check @done/the worker).

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, that's reachable once #add no longer raises. Fixed in a9df8e1: #rendez_vous now returns unless the sender thread is alive, the same check #flush already makes, so sync_with_outbound_io doesn't block on a queue nobody reads. Added a spec that calls rendez_vous after the failed restart under a 1 s timeout; it times out without the fix.

@flush_timer.start if @flush_timer && @flush_timer.stop?
}
end
Expand All @@ -106,7 +111,13 @@ def start
@sender_thread.name = "Statsd Sender" unless Gem::Version.new(RUBY_VERSION) < Gem::Version.new('2.3')
rescue ThreadError => e
@logger.debug { "Statsd: Failed to start sender thread: #{e.message}" } if @logger
@mx.synchronize { @done = true }
# #add calls #start while already holding @mx, and Ruby mutexes
# are not re-entrant
if @mx.owned?
@done = true
else
@mx.synchronize { @done = true }
end
end

@flush_timer.start if @flush_timer
Expand Down
29 changes: 29 additions & 0 deletions spec/statsd/sender_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,35 @@
end
end
end

context 'when the sender thread died and a new one cannot be started' do
let(:thread_class) { class_double(Thread) }
let(:dead_thread) { Thread.new {}.tap(&:join) }

before do
allow(thread_class).to receive(:new).and_return(dead_thread)
subject.start
allow(thread_class).to receive(:new).and_raise(ThreadError, "can't create Thread")
allow(message_buffer).to receive(:reset)
end

after do
subject.stop
end

it 'drops the messages without raising' do
expect do
subject.add('message')
subject.add('message')
end.not_to raise_error
expect(subject.send(:message_queue).length).to eq 0
end

it 'does not wait for the missing companion thread in rendez_vous' do
subject.add('message')
Timeout.timeout(1) { subject.rendez_vous }
end
end
end

describe '#flush' do
Expand Down