Can I just have a try without buying sidekiq pro #108
Comments
|
Shuttle doesn't currently work without pro, but we will keep your suggestion in mind. In the future, we can provide a lite version which doesn't require pro. |
|
Really Appreciate that! wait good news! |
|
Hi, I went so happy when I found Shuttle, but when I saw the price of Sidekiq Pro (U$ 750/year), my happiness went away ... I think a "lite" version will bring interest from users and developers. |
|
For what it's worth, Shuttle was originally written for Sidekiq, not Sidekiq Pro. In order to accomplish the job dependency features that Shuttle requires, I wrote my own Sidekiq-based job chaining and dependency system. Now I am apparently not as good at concurrency as @mperham, so the code didn't work very well and would often fail in weird ways. This is why we ultimately switched to Sidekiq Pro. The original commit moving from Sidekiq to Pro is not in the open-source repo, but it is in the Square internal repo. I've attached it here so that you can use it as a starting point in creating a non-pro version of the code, if you wish. commit 78aaa71c1fef4ae125e17487973f1dfde8b8da1a
Author: Tim Morgan <[email protected]>
Date: Thu Mar 20 13:36:03 2014 -0700
Replace SidekiqWorkerTracking with Sidekiq Pro batches feature
* Workers relating to importing a commit are now all grouped under the same
batch.
* Because Sidekiq batches does not support nesting, we cannot have BlobImporters
batch up KeyCreators, and CommitImporters batch up BlobImporters. Therefore,
all three of these job types are under one batch. When that batch finishes,
the Commit is marked as not loading _and_ all its blobs are marked as not
loading.
* To facilitate this, a new association has been introduced, `blobs_commits`,
that associates a Commit with its Blobs.
* Operations relating to counting and listing workers have been altered to use
`Sidekiq::Batch::Stats`.
* The ability to clear workers has been removed. This doesn't seem to exist
in Sidekiq Pro.
diff --git a/app/controllers/commits_controller.rb b/app/controllers/commits_controller.rb
index 4fa6236..5925782 100644
--- a/app/controllers/commits_controller.rb
+++ b/app/controllers/commits_controller.rb
@@ -21,13 +21,12 @@ class CommitsController < ApplicationController
before_filter :authenticate_user!, except: [:manifest, :localize]
before_filter :monitor_required, except: [:show, :manifest, :localize]
- before_filter :admin_required, only: :clear
before_filter :find_project
before_filter :find_commit, except: [:create, :manifest, :localize]
respond_to :html, :json, only: [:show, :create, :update, :destroy, :import,
- :sync, :match, :redo, :clear, :recalculate]
+ :sync, :match, :redo, :recalculate]
# Renders JSON information about a Commit and its translation progress.
#
@@ -175,7 +174,9 @@ class CommitsController < ApplicationController
# | `locale` | The RFC 5646 identifier for a locale. |
def import
- CommitImporter.perform_once @commit.id, locale: params[:locale]
+ @commit.import_batch.jobs do
+ CommitImporter.perform_once @commit.id, locale: params[:locale]
+ end
respond_with @commit, location: nil
end
@@ -196,7 +197,9 @@ class CommitsController < ApplicationController
# | `id` | The SHA of a Commit. |
def sync
- CommitImporter.perform_once @commit.id
+ @commit.import_batch.jobs do
+ CommitImporter.perform_once @commit.id
+ end
respond_with @commit, location: nil
end
@@ -218,30 +221,9 @@ class CommitsController < ApplicationController
# | `id` | The SHA of a Commit. |
def redo
- @commit.update_attribute(:loading, true)
- CommitImporter.perform_once @commit.id, force: true
- respond_with @commit, location: project_commit_url(@project, @commit)
- end
-
- # Removes all workers from the loading list, marks the Commit as not loading,
- # and recalculates Commit statistics if the Commit was previously loading.
- # This method should be used to fix "stuck" Commits.
- #
- # Routes
- # ------
- #
- # * `POST /projects/:project_id/commits/:id/clear`
- #
- # Path Parameters
- # ---------------
- #
- # | | |
- # |:-------------|:-----------------------|
- # | `project_id` | The slug of a Project. |
- # | `id` | The SHA of a Commit. |
-
- def clear
- @commit.clear_workers!
+ @commit.import_batch.jobs do
+ CommitImporter.perform_once @commit.id, force: true
+ end
respond_with @commit, location: project_commit_url(@project, @commit)
end
diff --git a/app/models/blob.rb b/app/models/blob.rb
index 6c8d58e..c04d7df 100644
--- a/app/models/blob.rb
+++ b/app/models/blob.rb
@@ -23,6 +23,7 @@
# |:----------|:-----------------------------------------------------|
# | `project` | The {Project} whose repository this blob belongs to. |
# | `keys` | The {Key Keys} found in this blob. |
+# | `commits` | The {Commit Commits} with this blob. |
#
# Fields
# ======
@@ -33,13 +34,13 @@
# | `loading` | If `true`, one or more workers is currently importing {Key Keys} for this blob. This defaults to `true` to ensure that a Blob's Keys are loaded at least once before the cached result is used. |
class Blob < ActiveRecord::Base
- include SidekiqWorkerTracking
-
self.primary_keys = :project_id, :sha_raw
belongs_to :project, inverse_of: :blobs
has_many :blobs_keys, foreign_key: [:project_id, :sha_raw], inverse_of: :blob, dependent: :delete_all
has_many :keys, through: :blobs_keys
+ has_many :blobs_commits, foreign_key: [:project_id, :sha_raw], inverse_of: :blob, dependent: :delete_all
+ has_many :commits, through: :blobs_commits
extend GitObjectField
git_object_field :sha,
@@ -97,6 +98,7 @@ class Blob < ActiveRecord::Base
# @private
def inspect(default_behavior=false)
return super() if default_behavior
- "#<#{self.class.to_s} #{sha}>"
+ state = loading? ? 'loading' : 'cached'
+ "#<#{self.class.to_s} #{sha} (#{state})>"
end
end
diff --git a/app/models/blobs_commit.rb b/app/models/blobs_commit.rb
new file mode 100644
index 0000000..b3432d4
--- /dev/null
+++ b/app/models/blobs_commit.rb
@@ -0,0 +1,29 @@
+# Copyright 2013 Square Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Join table between {Blob} and {Commit}. Indicates what blobs are accessible
+# from what commits.
+#
+# Associations
+# ------------
+#
+# | | |
+# |:---------|:--------------------------------|
+# | `blob` | The {Blob} found in the Commit. |
+# | `commit` | The {Commit} with the Blob. |
+
+class BlobsCommit < ActiveRecord::Base
+ belongs_to :blob, foreign_key: [:project_id, :sha_raw], inverse_of: :blobs_commits
+ belongs_to :commit, inverse_of: :blobs_commits
+end
diff --git a/app/models/commit.rb b/app/models/commit.rb
index 2bab2dd..cd1d379 100644
--- a/app/models/commit.rb
+++ b/app/models/commit.rb
@@ -45,6 +45,7 @@ require 'fileutils'
# | `user` | The {User} that submitted this Commit for translation. |
# | `keys` | All the {Key Keys} found in this Commit. |
# | `translations` | The {Translation Translations} found in this Commit. |
+# | `blobs` | The {Blob Blobs} found in this Commit. |
#
# Properties
# ==========
@@ -68,11 +69,13 @@ require 'fileutils'
# |:-------------------|:-------------------------------------------------------------------|
# | `description` | A user-submitted description of why we are localizing this commit. |
# | `pull_request_url` | A user-submitted URL to the pull request that is being localized. |
+# | `author` | The name of the commit author. |
+# | `author_email` | The email address of the commit author. |
+# | `import_batch_id` | The ID of the Sidekiq batch of import jobs. |
class Commit < ActiveRecord::Base
extend RedisMemoize
include CommitTraverser
- include SidekiqWorkerTracking
# @return [true, false] If `true`, does not perform an import after creating
# the Commit. Use this to avoid the overhead of making an HTTP request and
@@ -81,16 +84,19 @@ class Commit < ActiveRecord::Base
belongs_to :project, inverse_of: :commits
belongs_to :user, inverse_of: :commits
- has_many :commits_keys, inverse_of: :commit, dependent: :destroy
+ has_many :commits_keys, inverse_of: :commit, dependent: :delete_all
has_many :keys, through: :commits_keys
has_many :translations, through: :keys
+ has_many :blobs_commits, inverse_of: :commit, dependent: :delete_all
+ has_many :blobs, through: :blobs_commits
include HasMetadataColumn
has_metadata_column(
description: {allow_nil: true},
- author: {allow_nil: true},
+ author: {allow_nil: true},
author_email: {allow_nil: true},
- pull_request_url: {allow_nil: true}
+ pull_request_url: {allow_nil: true},
+ import_batch_id: {allow_nil: true}
)
include Tire::Model::Search
@@ -146,9 +152,7 @@ class Commit < ActiveRecord::Base
before_save :set_loaded_at
before_create :set_author
- after_commit(on: :create) do |commit|
- CommitImporter.perform_once(commit.id) unless commit.skip_import
- end
+ after_commit :initial_import, on: :create
after_commit :compile_and_cache_or_clear, on: :update
after_update :update_touchdown_branch
after_commit :update_stats_at_end_of_loading, on: :update, if: :loading_state_changed?
@@ -333,22 +337,17 @@ class Commit < ActiveRecord::Base
def import_strings(options={})
raise CommitNotFoundError, "Commit no longer exists: #{revision}" unless commit!
- blobs = project.blobs.includes(:project) # preload blobs for performance
-
- # add us as one of the workers, to prevent the commit from prematurely going
- # ready; let's just invent a job ID for us
- job_id = SecureRandom.uuid
- add_worker! job_id
+ import_batch.jobs do
+ update_attribute :loading, true
+ blobs = project.blobs.includes(:project) # preload blobs for performance
- # clear out existing keys so that we can import all new keys
- keys.clear unless options[:locale]
- # perform the recursive import
- traverse(commit!) do |path, blob|
- import_blob path, blob, options.merge(blobs: blobs)
+ # clear out existing keys so that we can import all new keys
+ commits_keys.delete_all unless options[:locale]
+ # perform the recursive import
+ traverse(commit!) do |path, blob|
+ import_blob path, blob, options.merge(blobs: blobs)
+ end
end
-
- # this will also kick of stats recalculation for inline imports
- remove_worker! job_id
end
# Returns a commit object used to interact with Git.
@@ -549,14 +548,32 @@ class Commit < ActiveRecord::Base
# @private
def redis_memoize_key() to_param end
- # @return [true, false] True if there are cached Sidekiq job IDs of
- # in-progress BlobImporters that do not actually exist anymore.
+ # @return [Sidekiq::Batch, nil] The batch of Sidekiq workers performing the
+ # current import, if any.
- def broken?
- cached_jids = Shuttle::Redis.smembers("import:#{revision}")
- return false if cached_jids.empty?
- actual_jids = self.class.workers.map { |w| w['jid'] }
- (cached_jids & actual_jids).empty? # none of the cached JIDs actually exist anymore
+ def import_batch
+ if import_batch_id
+ Sidekiq::Batch.new(import_batch_id)
+ else
+ batch = Sidekiq::Batch.new
+ batch.description = "Import Commit #{id} (#{revision})"
+ batch.on :success, ImportFinisher, commit_id: id
+ update_attribute :import_batch_id, batch.bid
+ batch
+ end
+ rescue Sidekiq::Batch::NoSuchBatch
+ update_attribute :import_batch_id, nil
+ retry
+ end
+
+ # @return [Sidekiq::Batch::Status, nil] Information about the batch of Sidekiq
+ # workers performing the current import, if any.
+
+ def import_batch_status
+ import_batch_id ? Sidekiq::Batch::Status.new(import_batch_id) : nil
+ rescue Sidekiq::Batch::NoSuchBatch
+ update_attribute :import_batch_id, nil
+ retry
end
# Returns whether we should skip a key for this particular commit, given the
@@ -668,7 +685,14 @@ class Commit < ActiveRecord::Base
imps.each do |importer|
importer = importer.new(blob_object, path, self)
- if options[:force]
+ # we can't do a force import on a loading blob -- if we delete all the
+ # blobs_keys while another sidekiq job is doing the import, when that job
+ # finishes the blob will unset loading, even though the former job is still
+ # adding keys to the blob. at this point a third import (with force=false)
+ # might start, see the blob as not loading, and then do a fast import of
+ # the cached keys (even though not all keys have been loaded by the second
+ # import).
+ if options[:force] && !loading?
blob_object.blobs_keys.delete_all
blob_object.update_column :loading, true
end
@@ -681,10 +705,7 @@ class Commit < ActiveRecord::Base
if options[:inline]
BlobImporter.new.perform importer.class.ident, project.id, blob.sha, path, id, options[:locale].try!(:rfc5646)
else
- shuttle_jid = SecureRandom.uuid
- blob_object.add_worker! shuttle_jid
- add_worker! shuttle_jid
- BlobImporter.perform_once(importer.class.ident, project.id, blob.sha, path, id, options[:locale].try!(:rfc5646), shuttle_jid)
+ BlobImporter.perform_once importer.class.ident, project.id, blob.sha, path, id, options[:locale].try!(:rfc5646)
end
end
end
@@ -700,4 +721,15 @@ class Commit < ActiveRecord::Base
# calculate readiness and stats.
CommitStatsRecalculator.new.perform id
end
+
+ #TODO there's a bug in Rails core that causes this to be run on update as well
+ # as create. sigh.
+ def initial_import
+ return if @_start_transaction_state[:id] # fix bug in Rails core
+ unless skip_import || loading?
+ import_batch.jobs do
+ CommitImporter.perform_once id
+ end
+ end
+ end
end
diff --git a/app/views/commits/show.slim b/app/views/commits/show.slim
index 597366b..71cc666 100644
--- a/app/views/commits/show.slim
+++ b/app/views/commits/show.slim
@@ -21,10 +21,14 @@
= "#{@commit.revision[0, 6]}"
span.separator /
- if @commit.loading?
- - current_status = "Loading with #{pluralize(@commit.list_workers.count, 'Job')}"
+ - if @commit.import_batch_status
+ | Currently Loading With #{pluralize_with_delimiter @commit.import_batch_status.pending, 'Job'} Remaining
+ - else
+ | Loading Stalled!
+ - elsif @commit.ready?
+ | Currently Ready
- else
- - current_status = (@commit.ready? ? 'Ready' : 'Translating')
- = "Currently #{current_status}"
+ | Currently Translating
span.separator /
- if @commit.completed_at
= "First Completed #{time_ago_in_words(@commit.completed_at)} ago"
@@ -85,12 +89,6 @@ hr.divider
| Use this if you need to manually reimport a commit
.controls
= button_to 'Reimport', redo_project_commit_path(@project, @commit)
- - if current_user.admin? and @commit.loading?
- .control-group
- label.description-label
- | Use this to clear workers for hanging commits
- .controls
- = button_to 'Clear Workers', clear_project_commit_url(@project, @commit)
.control-group
label.description-label
diff --git a/app/views/home/index.slim b/app/views/home/index.slim
index 801bf2b..8a5cb65 100644
--- a/app/views/home/index.slim
+++ b/app/views/home/index.slim
@@ -129,12 +129,17 @@ hr.divider
- else
/ Progress
- if commit.loading?
- td Importing (#{pluralize(commit.list_workers.count, 'Job')})
+ td
+ | Importing
+ - if commit.import_batch_status
+ | (#{pluralize_with_delimiter commit.import_batch_status.pending, 'Job'} Remaining)
+ - else
+ | (stalled!)
- elsif commit.ready?
td Ready for Download
- else
- strings_remaining = commit.translations_pending(*@locales) + commit.translations_new(*@locales)
- td = "Translating #{pluralize(strings_remaining, 'String')}"
+ td = "Translating #{pluralize_with_delimiter strings_remaining, 'String'}"
/ Translation/Monitor Button
= render partial: 'table_action_button', locals: { commit: commit }
diff --git a/app/views/search/keys.js.coffee.erb b/app/views/search/keys.js.coffee.erb
index 542543e..6360bf7 100644
--- a/app/views/search/keys.js.coffee.erb
+++ b/app/views/search/keys.js.coffee.erb
@@ -112,4 +112,4 @@ $(window).ready ->
window.onpopstate = ->
prefillForm()
- submitSearch()
\ No newline at end of file
+ submitSearch()
diff --git a/app/views/search/translations.js.coffee.erb b/app/views/search/translations.js.coffee.erb
index 1992e63..0df72a3 100644
--- a/app/views/search/translations.js.coffee.erb
+++ b/app/views/search/translations.js.coffee.erb
@@ -69,4 +69,4 @@ $(window).ready ->
window.onpopstate = ->
renewSearch() unless firstTime
- firstTime = false
\ No newline at end of file
+ firstTime = false
diff --git a/app/workers/blob_importer.rb b/app/workers/blob_importer.rb
index f69b94f..7eb74b1 100644
--- a/app/workers/blob_importer.rb
+++ b/app/workers/blob_importer.rb
@@ -29,7 +29,7 @@ class BlobImporter
# existing translations from. If `nil`, the base locale is imported as
# base translations.
- def perform(importer, project_id, sha, path, commit_id, rfc5646_locale, shuttle_jid=nil)
+ def perform(importer, project_id, sha, path, commit_id, rfc5646_locale)
commit = Commit.find_by_id(commit_id)
locale = rfc5646_locale ? Locale.from_rfc5646(rfc5646_locale) : nil
project = Project.find(project_id)
@@ -38,7 +38,9 @@ class BlobImporter
if blob.blob!.nil?
# for whatever reason sometimes the blob is not accessible; try again in
# 5 minutes
- BlobImporter.perform_in(5.minutes, importer, project_id, sha, path, commit_id, rfc5646_locale, shuttle_jid)
+ commit.import_batch.jobs do
+ BlobImporter.perform_in 5.minutes, importer, project_id, sha, path, commit_id, rfc5646_locale
+ end
return
end
@@ -49,9 +51,6 @@ class BlobImporter
commit: commit,
locale: locale,
inline: jid.nil?
-
- blob.remove_worker! shuttle_jid
- commit.remove_worker! shuttle_jid
end
include SidekiqLocking
diff --git a/app/workers/key_creator.rb b/app/workers/key_creator.rb
index edc64ce..1f92dfd 100644
--- a/app/workers/key_creator.rb
+++ b/app/workers/key_creator.rb
@@ -45,12 +45,8 @@ class KeyCreator
key_objects.map(&:id).uniq.each { |k| @blob.blobs_keys.where(key_id: k).find_or_create! }
if @commit
- key_objects.reject! { |key| skip_key?(key) }
self.class.update_key_associations key_objects, @commit
end
-
- @blob.remove_worker! shuttle_jid
- @commit.remove_worker! shuttle_jid if @commit
end
# Given a set of keys, bulk-updates their commits-keys associations and
@@ -60,6 +56,7 @@ class KeyCreator
# @param [Commit] commit A Commit these keys are associated with.
def self.update_key_associations(keys, commit)
+ keys.reject! { |key| skip_key?(key, commit) }
keys.map(&:id).uniq.each { |k| commit.commits_keys.where(key_id: k).find_or_create! }
# key.commits has been changed, need to update associated ES fields
@@ -112,16 +109,15 @@ class KeyCreator
# Determines if we should skip this key using both the normal key exclusions
# and the .shuttle.yml key exclusions
- def skip_key?(key)
- skip_key_due_to_project_settings?(key) || skip_key_due_to_branch_settings?(key)
+ def self.skip_key?(key, commit)
+ skip_key_due_to_project_settings?(key, commit.project) || skip_key_due_to_branch_settings?(key, commit)
end
- def skip_key_due_to_project_settings?(key)
- @blob.project.skip_key?(key.key, @blob.project.base_locale)
+ def self.skip_key_due_to_project_settings?(key, project)
+ project.skip_key?(key.key, project.base_locale)
end
- def skip_key_due_to_branch_settings?(key)
- return false unless @commit
- @commit.skip_key?(key.key)
+ def self.skip_key_due_to_branch_settings?(key, commit)
+ commit.skip_key?(key.key)
end
end
diff --git a/config/routes.rb b/config/routes.rb
index da45a48..3113e44 100644
--- a/config/routes.rb
+++ b/config/routes.rb
@@ -40,7 +40,7 @@ Shuttle::Application.routes.draw do
resources :projects do
resources :commits, only: [:show, :create, :update, :destroy] do
member do
- post :import, :sync, :redo, :clear, :recalculate
+ post :import, :sync, :redo, :recalculate
get :manifest, :localize
end
diff --git a/db/migrate/20140320053508_create_blobs_commits.rb b/db/migrate/20140320053508_create_blobs_commits.rb
new file mode 100644
index 0000000..ab6cca1
--- /dev/null
+++ b/db/migrate/20140320053508_create_blobs_commits.rb
@@ -0,0 +1,31 @@
+# Copyright 2013 Square Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class CreateBlobsCommits < ActiveRecord::Migration
+ def up
+ execute <<-SQL
+ CREATE TABLE blobs_commits(
+ project_id INTEGER NOT NULL,
+ sha_raw BYTEA NOT NULL,
+ commit_id INTEGER NOT NULL REFERENCES commits(id) ON DELETE CASCADE,
+ FOREIGN KEY (project_id, sha_raw) REFERENCES blobs(project_id, sha_raw) ON DELETE CASCADE,
+ PRIMARY KEY (project_id, sha_raw, commit_id)
+ )
+ SQL
+ end
+
+ def down
+ drop_table :blobs_commits
+ end
+end
diff --git a/db/structure.sql b/db/structure.sql
index bef994d..075b757 100644
--- a/db/structure.sql
+++ b/db/structure.sql
@@ -41,6 +41,17 @@ CREATE TABLE blobs (
--
+-- Name: blobs_commits; Type: TABLE; Schema: public; Owner: -; Tablespace:
+--
+
+CREATE TABLE blobs_commits (
+ project_id integer NOT NULL,
+ sha_raw bytea NOT NULL,
+ commit_id integer NOT NULL
+);
+
+
+--
-- Name: blobs_keys; Type: TABLE; Schema: public; Owner: -; Tablespace:
--
@@ -551,6 +562,14 @@ ALTER TABLE ONLY users ALTER COLUMN id SET DEFAULT nextval('users_id_seq'::regcl
--
+-- Name: blobs_commits_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace:
+--
+
+ALTER TABLE ONLY blobs_commits
+ ADD CONSTRAINT blobs_commits_pkey PRIMARY KEY (project_id, sha_raw, commit_id);
+
+
+--
-- Name: blobs_keys_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace:
--
@@ -797,6 +816,22 @@ CREATE UNIQUE INDEX users_unlock_token ON users USING btree (unlock_token);
--
+-- Name: blobs_commits_commit_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
+--
+
+ALTER TABLE ONLY blobs_commits
+ ADD CONSTRAINT blobs_commits_commit_id_fkey FOREIGN KEY (commit_id) REFERENCES commits(id) ON DELETE CASCADE;
+
+
+--
+-- Name: blobs_commits_project_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
+--
+
+ALTER TABLE ONLY blobs_commits
+ ADD CONSTRAINT blobs_commits_project_id_fkey FOREIGN KEY (project_id, sha_raw) REFERENCES blobs(project_id, sha_raw) ON DELETE CASCADE;
+
+
+--
-- Name: blobs_keys_key_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
--
@@ -999,3 +1034,5 @@ INSERT INTO schema_migrations (version) VALUES ('20140228025058');
INSERT INTO schema_migrations (version) VALUES ('20140306064700');
INSERT INTO schema_migrations (version) VALUES ('20140311011156');
+
+INSERT INTO schema_migrations (version) VALUES ('20140320053508');
diff --git a/lib/import_finisher.rb b/lib/import_finisher.rb
new file mode 100644
index 0000000..e4f1ebb
--- /dev/null
+++ b/lib/import_finisher.rb
@@ -0,0 +1,16 @@
+# Contains hooks run by Sidekiq upon completion of an import batch.
+
+class ImportFinisher
+
+ # Run by Sidekiq after an import batch finishes successfully. Unsets the
+ # {Commit}'s `loading` flag (thus running post-import hooks), and unsets the
+ # loading flag on all associated {Blob Blobs}.
+
+ def on_success(_status, options)
+ commit = Commit.find(options['commit_id'])
+ commit.update_attributes loading: false, import_batch_id: nil
+ # commit.blobs.update_all loading: false
+ blob_shas = commit.blobs_commits.pluck(:sha_raw)
+ Blob.where(project_id: commit.project_id, sha_raw: blob_shas).update_all loading: false
+ end
+end
diff --git a/lib/importer/base.rb b/lib/importer/base.rb
index 4a1d2c8..b2315d9 100644
--- a/lib/importer/base.rb
+++ b/lib/importer/base.rb
@@ -95,6 +95,10 @@ module Importer
@blob = blob
@commit = commit
@file = File.new(path, nil, nil)
+
+ if @commit
+ blob.blobs_commits.where(commit_id: @commit.id).find_or_create!
+ end
end
# Scans the Blob for localizable strings, and creates or updates
@@ -283,6 +287,8 @@ module Importer
end
def import_by_parsing_blob
+ @blob.update_attribute :loading, true
+
load_contents
@keys = Array.new
@@ -293,14 +299,20 @@ module Importer
end
# then spawn jobs to create those keys
- @keys.in_groups_of(100, false) do |keys|
- if inline
+ if inline
+ @keys.in_groups_of(100, false) do |keys|
KeyCreator.new.perform @blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys
- else
- shuttle_jid = SecureRandom.uuid
- @blob.add_worker! shuttle_jid
- @commit.add_worker!(shuttle_jid) if @commit
- KeyCreator.perform_async(@blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys, shuttle_jid)
+ end
+ elsif @commit
+ bulk_args = @keys.in_groups_of(100, false).map do |keys|
+ [@blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys]
+ end
+ @commit.import_batch.jobs do
+ Sidekiq::Client.push_bulk 'class' => KeyCreator, 'args' => bulk_args
+ end
+ else
+ @keys.in_groups_of(100, false) do |keys|
+ KeyCreator.perform_async @blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys
end
end
end
@@ -308,7 +320,7 @@ module Importer
# Used when this blob was imported as part of an earlier commit; just
# associates the cached list of keys for that blob with the new commit
def import_by_using_cached_keys
- KeyCreator.update_key_associations @blob.keys, @commit
+ KeyCreator.update_key_associations @blob.keys.to_a, @commit
end
# array indexes are stored in brackets
diff --git a/lib/sidekiq_locking.rb b/lib/sidekiq_locking.rb
index b679f4f..5aab039 100644
--- a/lib/sidekiq_locking.rb
+++ b/lib/sidekiq_locking.rb
@@ -49,7 +49,7 @@ module SidekiqLocking
end
def mutex(*args)
- Redis::Mutex.new(lock_name(*args), :expire => 1.hour)
+ Redis::Mutex.new(lock_name(*args), expire: 1.hour)
end
end
end
diff --git a/lib/sidekiq_worker_tracking.rb b/lib/sidekiq_worker_tracking.rb
deleted file mode 100644
index 855e1a2..0000000
--- a/lib/sidekiq_worker_tracking.rb
+++ /dev/null
@@ -1,97 +0,0 @@
-# Copyright 2014 Square Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Adds the ability to track Sidekiq workers performing an oepration on an object
-# and update the value of a `loading` column when all workers have completed.
-#
-# To use this module, your model must have an attribute named `loading`. When
-# you add a worker that performs an operation on your model, you should pass its
-# JID to the {#add_worker!} method. The model's `loading` column will be set to
-# `false`, and will remain `false` until all such workers have completed.
-#
-# @example
-# class MyModel
-# include SidekiqWorkerTracking
-# attr_accessor :loading
-#
-# def perform_operation
-# 10.times do
-# add_worker! OperationPerformer.perform_async(...)
-# end
-# end
-#
-# def all_operations_completed?
-# !loading
-# end
-# end
-
-module SidekiqWorkerTracking
- # Adds a worker to the loading list. This object, if not already loading, will
- # be marked as loading until this and all other added workers call
- # {#remove_worker!}.
- #
- # @param [String] jid A unique identifier for this worker.
-
- def add_worker!(jid)
- self.loading = true
- save!
- Shuttle::Redis.sadd worker_set_key, jid
- end
-
- # Removes a worker from the loading list. This object will not be marked as
- # loading if this was the last worker. Also recalculates Commit statistics if
- # this was the last worker.
- #
- # @param [String] jid A unique identifier for this worker.
- # @see #add_worker!
-
- def remove_worker!(jid)
- if jid.nil?
- return
- end
-
- unless Shuttle::Redis.srem(worker_set_key, jid)
- Squash::Ruby.record "Failed to remove worker", object: self, jid: jid, unimportant: true
- end
-
- loading = (Shuttle::Redis.scard(worker_set_key) > 0)
-
- self.loading = loading
- save!
- end
-
- # Returns all workers from the loading list
-
- def list_workers
- Shuttle::Redis.smembers worker_set_key
- end
-
- # Removes all workers from the loading list, marks the Commit as not loading,
- # and recalculates Commit statistics if the Commit was previously loading.
- # This method should be used to fix "stuck" Commits.
-
- def clear_workers!
- Shuttle::Redis.del worker_set_key
- if loading?
- self.loading = false
- save!
- end
- end
-
- private
-
- def worker_set_key
- "loading:#{self.class.to_s}:#{self.id}"
- end
-end
diff --git a/lib/tasks/maintenance.rake b/lib/tasks/maintenance.rake
index 9e36ba9..e5a3ad1 100644
--- a/lib/tasks/maintenance.rake
+++ b/lib/tasks/maintenance.rake
@@ -13,11 +13,6 @@
# limitations under the License.
namespace :maintenance do
- desc "Locate hung commits and clear their worker queue"
- task fix_hung_commits: :environment do
- Commit.where(loading: true).select(&:broken?).each(&:clear_workers!)
- end
-
desc "Locates lockfiles that no longer refer to active processes and clears them"
task clear_stale_lockfiles: :environment do
workers = Shuttle::Redis.smembers('workers').select { |w| w.start_with? Socket.gethostname }
diff --git a/spec/controllers/search_controller_spec.rb b/spec/controllers/search_controller_spec.rb
index a5e39c2..271b41b 100644
--- a/spec/controllers/search_controller_spec.rb
+++ b/spec/controllers/search_controller_spec.rb
@@ -77,10 +77,11 @@ describe SearchController do
expect(response.status).to eql(200)
results = JSON.parse(response.body)
expect(results.size).to eql(2)
- expect(results.first['copy']).to eql('foo term1 bar')
- expect(results.first['locale']['rfc5646']).to eql('fr')
+ results.sort_by! { |r| r['locale']['rfc5646'] }
expect(results.last['copy']).to eql('foo term1 bar')
- expect(results.last['locale']['rfc5646']).to eql('en')
+ expect(results.last['locale']['rfc5646']).to eql('fr')
+ expect(results.first['copy']).to eql('foo term1 bar')
+ expect(results.first['locale']['rfc5646']).to eql('en')
end
it "should respond with a 422 if the locale is unknown" do
diff --git a/spec/factories/blobs_commits.rb b/spec/factories/blobs_commits.rb
new file mode 100644
index 0000000..12d6e63
--- /dev/null
+++ b/spec/factories/blobs_commits.rb
@@ -0,0 +1,6 @@
+FactoryGirl.define do
+ factory :blobs_commit do
+ association :blob
+ association :commit
+ end
+end
diff --git a/spec/factories/blobs_keys.rb b/spec/factories/blobs_keys.rb
index 023107a..bd8b8c9 100644
--- a/spec/factories/blobs_keys.rb
+++ b/spec/factories/blobs_keys.rb
@@ -14,7 +14,7 @@
FactoryGirl.define do
factory :blobs_key do
- association :blobs
- association :keys
+ association :blob
+ association :key
end
end
diff --git a/spec/models/blobs_commit_spec.rb b/spec/models/blobs_commit_spec.rb
new file mode 100644
index 0000000..ed0762d
--- /dev/null
+++ b/spec/models/blobs_commit_spec.rb
@@ -0,0 +1,4 @@
+require 'spec_helper'
+
+describe BlobsCommit do
+end
diff --git a/spec/models/commit_spec.rb b/spec/models/commit_spec.rb
index 6832813..c4f19df 100644
--- a/spec/models/commit_spec.rb
+++ b/spec/models/commit_spec.rb
@@ -550,6 +550,26 @@ describe Commit do
@project.commit! 'HEAD'
expect(@project.keys.map(&:importer).uniq.sort).to eql(Importer::Base.implementations.map(&:ident).sort - %w(yaml))
end
+
+ it "should remove appropriate keys when reimporting after changed settings" do
+ commit = @project.commit!('HEAD')
+ expect(commit.keys.map(&:original_key)).to include('root')
+
+ @project.update_attribute :key_exclusions, %w(roo*)
+ commit.import_strings
+ expect(commit.keys(true).map(&:original_key)).not_to include('root')
+ end
+
+ it "should only associate relevant keys with a new commit when cached blob importing is being use3d" do
+ @project.update_attribute :key_exclusions, %w(skip_me)
+ commit = @project.commit!('HEAD')
+ blob = commit.blobs.first
+ red_herring = FactoryGirl.create(:key, key: 'skip_me')
+ FactoryGirl.create :blobs_key, key: red_herring, blob: blob
+
+ commit.import_strings
+ expect(commit.keys(true)).not_to include(red_herring)
+ end
end
describe "#all_translations_entered_for_locale?" do |
|
Thanks RISCfuture for pointing out a way. I hope soon small teams and solo professionals can enjoy using Shuttle. |
|
Trying to deploy on centos failed. |
longboardcat
pushed a commit
that referenced
this issue
May 25, 2016
…data-column-in-commits to master * commit '873fbf9cd8fdbfdf4aff15b814741d15df499b81': Deprecate has_metadata_column in Commit model
|
Any progress on a lite version of Shuttle? Buying a software license for $950/year is a big bummer, there has to be another way. |
|
Super interested also to try first! |
|
Can Shuttle work without Sidekiq Pro now? It's so expensive to buy Sidekiq Pro!!! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I want to deploy this tool in my team, but I am newbiew here for ruby , any advice for trying use this under lower requiment of sidekiq will be appreicated
The text was updated successfully, but these errors were encountered: