Nicolas B., Ruby Backend Developer Zola 2015-07-15T00:00:00+00:00 https://nicolas-brousse.fr/tags/sidekiq/atom.xml 2015-07-15T00:00:00+00:00 2015-07-15T00:00:00+00:00 Nicolas B. https://nicolas-brousse.fr/articles/2015-07-15-test-1-sidekiq-on-separate-servers/ <img src="https://nicolas-brousse.fr/processed_images/banner.361201b45685ca07.jpg" alt="" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/banner.361201b45685ca07.jpg" width="1280" height="512" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/banner.97e7b0ea090816f9.jpg 640w, https://nicolas-brousse.fr/processed_images/banner.f7e1b98d978384fc.jpg 784w, https://nicolas-brousse.fr/processed_images/banner.36c72b2d3bddf572.jpg 1280w, https://nicolas-brousse.fr/processed_images/banner.66d3022e3055f746.jpg 1920w" loading="lazy"> <p>These last days I tried to use Sidekiq on different servers than the Rails app one.</p> <h2 id="first-what-is-sidekiq">First, What is Sidekiq?<a class="post-anchor" href="#first-what-is-sidekiq" aria-label="Anchor link for: first-what-is-sidekiq"><span aria-hidden="true">#</span></a> </h2> <p><a rel="noopener noreferrer external" target="_blank" href="http://sidekiq.org/">Sidekiq</a> is a <q>Simple, efficient background processing for Ruby</q>.</p> <p>With more words, it’s a library who allows you to create job classes. A job contains the code pieces you want to perform later. It’s a kind of code decentralization.</p> <p>And in this test the goal is to try running the job in different server than the web server.</p> <h2 id="why">Why?<a class="post-anchor" href="#why" aria-label="Anchor link for: why"><span aria-hidden="true">#</span></a> </h2> <p>Why is it better to separate worker of web server?<br /> Imagine you need to do processes, so you use memory and CPU. In this case that means you take web server resources. So your web app could be slower.</p> <p>By using different servers, it uses separate server resources. If process needs time, it takes the time it needs. And you could eventually decide to use a bigger server later.<br /> So it’s just easier to scale web and worker servers separately with you VPS or cloud hosting provider.</p> <h2 id="test-in-pictures">Test in pictures<a class="post-anchor" href="#test-in-pictures" aria-label="Anchor link for: test-in-pictures"><span aria-hidden="true">#</span></a> </h2> <p>To do this test I used <strong>Vagrant</strong> with 3 VMs. One for the Rails <code>web</code> server and two others for Sidekiq named <code>worker-01</code> + <code>worker-02</code>.</p> <p>For the Rails app code I simply used the one of the <a rel="noopener noreferrer external" target="_blank" href="http://railscasts.com/episodes/366-sidekiq/">Railscast #366</a> by <a rel="noopener noreferrer external" target="_blank" href="https://twitter.com/rbates">Ryan Bates</a>.<br /> I only converted the Sidekiq worker into a Rails ActiveJob.</p> <p>The example is a simple snippets application.</p> <h3 id="1-set-data-into-form">1. Set data into form<a class="post-anchor" href="#1-set-data-into-form" aria-label="Anchor link for: 1-set-data-into-form"><span aria-hidden="true">#</span></a> </h3> <p><img src="https://nicolas-brousse.fr/processed_images/img-1.5caa85a74a167c37.png" alt="Form" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/img-1.5caa85a74a167c37.png" width="1280" height="769" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/img-1.d4f8f22f617575a0.png 640w, https://nicolas-brousse.fr/processed_images/img-1.158211d3b5011337.png 784w, https://nicolas-brousse.fr/processed_images/img-1.d01f5346399f5f5f.png 1280w" loading="lazy"> This is the “homepage” and also the form to send the code of our snippet.</p> <h3 id="2-after-data-posted">2. After data posted<a class="post-anchor" href="#2-after-data-posted" aria-label="Anchor link for: 2-after-data-posted"><span aria-hidden="true">#</span></a> </h3> <p><img src="https://nicolas-brousse.fr/processed_images/img-2.b8058e4222d8dfc0.png" alt="Datas posted" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/img-2.b8058e4222d8dfc0.png" width="1280" height="769" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/img-2.f040a6b1f5330880.png 640w, https://nicolas-brousse.fr/processed_images/img-2.7112e5b930798a61.png 784w, https://nicolas-brousse.fr/processed_images/img-2.ba6af315ed7ab9f1.png 1280w" loading="lazy"> After posted our snippet we could see the code. Currently the code is still raw. We have to wait the job is processed.</p> <h3 id="3-after-worker-processed">3. After worker processed<a class="post-anchor" href="#3-after-worker-processed" aria-label="Anchor link for: 3-after-worker-processed"><span aria-hidden="true">#</span></a> </h3> <p><img src="https://nicolas-brousse.fr/processed_images/img-3.f54a079b6fec2af4.png" alt="Datas processed" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/img-3.f54a079b6fec2af4.png" width="1280" height="769" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/img-3.3862dc1d604ccc56.png 640w, https://nicolas-brousse.fr/processed_images/img-3.03e13d6274eae6fe.png 784w, https://nicolas-brousse.fr/processed_images/img-3.c18ac9026c45f6f4.png 1280w" loading="lazy"> Now the worker processed the job. The code is full of colors. And we could see that the IP is a different one and the hostname as well.</p> <h3 id="4-retry-a-new-once">4. Retry a new once<a class="post-anchor" href="#4-retry-a-new-once" aria-label="Anchor link for: 4-retry-a-new-once"><span aria-hidden="true">#</span></a> </h3> <p><img src="https://nicolas-brousse.fr/processed_images/img-4.3a61260042689898.png" alt="Datas posted after retry" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/img-4.3a61260042689898.png" width="1280" height="769" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/img-4.aee7a18c4fb16c1d.png 640w, https://nicolas-brousse.fr/processed_images/img-4.589dabdcfa2e769f.png 784w, https://nicolas-brousse.fr/processed_images/img-4.5e432d0ac2eb2c80.png 1280w" loading="lazy"> In my example I instantiate two worker servers. So I do others tests.</p> <h3 id="5-after-worker-processed-on-a-different-worker">5. After worker processed on a different worker<a class="post-anchor" href="#5-after-worker-processed-on-a-different-worker" aria-label="Anchor link for: 5-after-worker-processed-on-a-different-worker"><span aria-hidden="true">#</span></a> </h3> <p><img src="https://nicolas-brousse.fr/processed_images/img-5.c4445e56f8f62b24.png" alt="Datas processed after retry" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/img-5.c4445e56f8f62b24.png" width="1280" height="769" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/img-5.8cf6ec6647795d01.png 640w, https://nicolas-brousse.fr/processed_images/img-5.084cb7c5f5f974c4.png 784w, https://nicolas-brousse.fr/processed_images/img-5.8aee9c879fbccff4.png 1280w" loading="lazy"> In this new test the IP is not the same than the other time. It’s not the same worker server who processed the job.</p> <h3 id="sidekiq-admin">Sidekiq admin<a class="post-anchor" href="#sidekiq-admin" aria-label="Anchor link for: sidekiq-admin"><span aria-hidden="true">#</span></a> </h3> <p><img src="https://nicolas-brousse.fr/processed_images/img-6.293292b93cf5dceb.png" alt="Sidekiq admin" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/img-6.293292b93cf5dceb.png" width="1280" height="770" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/img-6.c8505d5a98fff9ad.png 640w, https://nicolas-brousse.fr/processed_images/img-6.a467c4167ea4ea00.png 784w, https://nicolas-brousse.fr/processed_images/img-6.4571224d643b8eb9.png 1280w" loading="lazy"> We have the list of the Sidekiq processes who are working. The two (or more) workers are listed here :)</p> <h2 id="serveurs-configuration">Serveurs configuration<a class="post-anchor" href="#serveurs-configuration" aria-label="Anchor link for: serveurs-configuration"><span aria-hidden="true">#</span></a> </h2> <p>Let see with scheme how it looks for the servers configuration.</p> <h3 id="my-current-test-with-vagrant">My current test with Vagrant<a class="post-anchor" href="#my-current-test-with-vagrant" aria-label="Anchor link for: my-current-test-with-vagrant"><span aria-hidden="true">#</span></a> </h3> <p>For the test I did with Vagrant, it looked like this.</p> <img src="https://nicolas-brousse.fr/processed_images/configuration-vagrant.a9d5e3078aa8ccd5.png" alt="Vagrant configuration" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/configuration-vagrant.a9d5e3078aa8ccd5.png" width="661" height="469" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="https://nicolas-brousse.fr/processed_images/configuration-vagrant.e6d0cd7b8cc2dd05.png 640w" loading="lazy"> <p>So we have the <code>web</code> server who also contains the Redis server and the Rails app. Because this is a test, I run the application in development mode with Rack. The application is exactly the same used for the 3 servers. So I used an Sqlite as a file for simplicity.</p> <p>For the <code>worker</code>s it uses the same Redis server and database. And I run the Sidekiq root process as a daemon.</p> <h3 id="production-example">Production example<a class="post-anchor" href="#production-example" aria-label="Anchor link for: production-example"><span aria-hidden="true">#</span></a> </h3> <p>So, what about a production configuration? It could be something like this.</p> <img src="https://nicolas-brousse.fr/processed_images/configuration-production-example.40a7cfc01ee30012.png" alt="Example of production configuration" data-zoomable="true" data-zoom-src="https://nicolas-brousse.fr/processed_images/configuration-production-example.40a7cfc01ee30012.png" width="495" height="908" sizes="(min-width: 920px) 784px, (min-width: 700px) calc(82vw + 46px), calc(100vw - 40px)" srcset="" loading="lazy"> <p>And have separate <code>db</code> and <code>redis</code> servers. <code>web</code> servers only contain the application with the http server. If you want multiple <code>web</code> servers, you have to install a proxy server as HAProxy.</p> <h2 id="code-sources">Code sources<a class="post-anchor" href="#code-sources" aria-label="Anchor link for: code-sources"><span aria-hidden="true">#</span></a> </h2> <p>If you want to test it, <a rel="noopener noreferrer external" target="_blank" href="https://github.com/nicolas-brousse/1-sidekiq-separate-servers">all the code is available on GitHub</a>.<br /> After clone or download you only have to install Vagrant (if you haven’t yet) and just <code>vagrant up</code>.</p> <p>Have fun 😄!</p> <hr /> <p>Sources:</p> <ul> <li><a rel="noopener noreferrer external" target="_blank" href="http://stackoverflow.com/questions/18265427/is-it-possible-to-run-sidekiq-on-a-separate-host-from-the-rails-host">Stackoverflow question about Sidekiq on separate server</a></li> <li><a rel="noopener noreferrer external" target="_blank" href="http://docs.vagrantup.com/v2/multi-machine/">Vagrant multi-machine documentation</a></li> <li><a rel="noopener noreferrer external" target="_blank" href="http://sidekiq.org/">Sidekiq website</a></li> <li><a rel="noopener noreferrer external" target="_blank" href="http://railscasts.com/episodes/366-sidekiq/">‘Railscast #366 Sidekiq’</a></li> <li><a rel="noopener noreferrer external" target="_blank" href="https://github.com/nicolas-brousse/1-sidekiq-separate-servers">Source code</a></li> </ul>