atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>javasgl</title>
  
  <subtitle>step by step</subtitle>
  <link href="/atom.xml" rel="self"/>
  
  <link href="https://javasgl.github.io/"/>
  <updated>2020-08-14T15:36:07.047Z</updated>
  <id>https://javasgl.github.io/</id>
  
  <author>
    <name>javasgl</name>
    
  </author>
  
  <generator uri="http://hexo.io/">Hexo</generator>
  
  <entry>
    <title>ES7.8.1 集群搭建踩坑记录</title>
    <link href="https://javasgl.github.io/elasticsearch-cluster/"/>
    <id>https://javasgl.github.io/elasticsearch-cluster/</id>
    <published>2020-08-14T14:44:15.000Z</published>
    <updated>2020-08-14T15:36:07.047Z</updated>
    
    <content type="html"><![CDATA[<p>时隔两年多时间，再次接触 elasticsearch,准备在测试环境搭建一套 ES 集群用来测试</p><p>之前使用ES还是在其1.4.x~2.0.x左右的时候，现在 ES 版本已经发展到了7.x以上了(中间经历过于kibana同步版本号,所以版本变动比较大)</p><h2 id="单机安装"><a href="#单机安装" class="headerlink" title="单机安装"></a>单机安装</h2><p>大致安装步骤和之前的版本差别不大,依然保持了其开箱即用的特性,只需要很少的配置即可</p><p>单机很简单，修改集群名称，节点名称,数据及日志目录,以及监听绑定的ip,然后直接运行可执行文件即可</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">注：下载的打包文件中，附带一个openjdk可以直接使用，如果使用 supervisor 来管理 es, 可以使用</span><br><span class="line">supervisor 的 environment 功能，来设置 JAVA_HOME 环境变量</span><br><span class="line"></span><br><span class="line">配置如下，ES_HOME 即 elasticsearch 解压后文件夹所在目录</span><br><span class="line"></span><br><span class="line">[es]</span><br><span class="line"><span class="attribute">command</span>=<span class="variable">$ES_HOME</span>/bin/elasticsearch</span><br><span class="line"><span class="attribute">directory</span>=<span class="variable">$ES_HOME</span></span><br><span class="line"><span class="attribute">environment</span>=JAVA_HOME=$ES_HOME/jdk</span><br></pre></td></tr></table></figure><p>单机没什么问题，很顺畅的就启动起来了，通过 curl <a href="http://ip:es_port" target="_blank" rel="noopener">http://ip:es_port</a> 可以得到很熟悉的返回：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">&#123;</span><br><span class="line">  <span class="attr">"name"</span> : <span class="string">"node1"</span>,</span><br><span class="line">  <span class="attr">"cluster_name"</span> : <span class="string">"cluster-name"</span>,</span><br><span class="line">  <span class="attr">"cluster_uuid"</span> : <span class="string">"2c2jICSxTHK_jUcJjaBucw"</span>,</span><br><span class="line">  <span class="attr">"version"</span> : &#123;</span><br><span class="line">    <span class="attr">"number"</span> : <span class="string">"7.8.1"</span>,</span><br><span class="line">    <span class="attr">"build_flavor"</span> : <span class="string">"default"</span>,</span><br><span class="line">    <span class="attr">"build_type"</span> : <span class="string">"tar"</span>,</span><br><span class="line">    <span class="attr">"build_hash"</span> : <span class="string">"b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89"</span>,</span><br><span class="line">    <span class="attr">"build_date"</span> : <span class="string">"2020-07-21T16:40:44.668009Z"</span>,</span><br><span class="line">    <span class="attr">"build_snapshot"</span> : <span class="literal">false</span>,</span><br><span class="line">    <span class="attr">"lucene_version"</span> : <span class="string">"8.5.1"</span>,</span><br><span class="line">    <span class="attr">"minimum_wire_compatibility_version"</span> : <span class="string">"6.8.0"</span>,</span><br><span class="line">    <span class="attr">"minimum_index_compatibility_version"</span> : <span class="string">"6.0.0-beta1"</span></span><br><span class="line">  &#125;,</span><br><span class="line">  <span class="attr">"tagline"</span> : <span class="string">"You Know, for Search"</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><h2 id="组建集群"><a href="#组建集群" class="headerlink" title="组建集群"></a>组建集群</h2><p>既然单机的起来的，那如法炮制，将配置文件复制一份，修改了其 node name, scp到另外一台机器上，准备好 supervisor 配置，准备开始组建两个节点组成的集群</p><p>满以为不会有什么问题，启动之后，发现，第二个节点，怎么也加入不到第一个节点之中<br><a id="more"></a></p><p>问题出现了，看了看文档，发现自 es7.x版本以后，更新换代了一个集群协调的功能工具 zen, 简化了很多配置项，组建集群时，<br>基本上就需要配置两个：</p><blockquote><p>discovery.seed_hosts, cluster.initial_master_nodes</p></blockquote><p>seed_hosts: 启动当前节点时，用于发现其他节点的初始列表<br>initial_master_nodes: 初始的候选master节点列表</p><p>最开始的时候，这两个均配置了两台机器的ip,即[node1,node2],分别启动节点后，发现后启动的节点，怎么也加入不到之前启动的节点中，组成集群</p><p>两个节点，脑裂了~~~~</p><p>网上搜了很多解决方案，绝大多数是说不该监听0或者0.0.0.0，遂调整配置，再次启动，发现依然无法解决两个节点脑裂问题</p><p>后来仔细研读了下，文档中对 initial_master_nodes 的说明，发现了问题：</p><blockquote><p>initial_master_nodes 中配置了当前节点，即意味着，当前节点也是可以候选的可用的master节点，但其实，当前节点都还没启动，它是没有资格去竞选成为master的，所以导致第二个节点启动后，自己独立成群，无法与已有节点组成集群</p></blockquote><p>问题找到了，修改第二个节点的配置文件，initial_master_nodes 仅指定第一个节点为可用的候选 master节点</p><p>随后启动节点二，启动正常，节点二和节点1成功组成一个集群</p><blockquote><p>后来，将 network.host 改为监听 0.0.0.0,两者依然可以组成集群，可见，网传的不能监听0.0.0.0的说法是不成立的</p></blockquote>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;时隔两年多时间，再次接触 elasticsearch,准备在测试环境搭建一套 ES 集群用来测试&lt;/p&gt;
&lt;p&gt;之前使用ES还是在其1.4.x~2.0.x左右的时候，现在 ES 版本已经发展到了7.x以上了(中间经历过于kibana同步版本号,所以版本变动比较大)&lt;/p&gt;
&lt;h2 id=&quot;单机安装&quot;&gt;&lt;a href=&quot;#单机安装&quot; class=&quot;headerlink&quot; title=&quot;单机安装&quot;&gt;&lt;/a&gt;单机安装&lt;/h2&gt;&lt;p&gt;大致安装步骤和之前的版本差别不大,依然保持了其开箱即用的特性,只需要很少的配置即可&lt;/p&gt;
&lt;p&gt;单机很简单，修改集群名称，节点名称,数据及日志目录,以及监听绑定的ip,然后直接运行可执行文件即可&lt;/p&gt;
&lt;figure class=&quot;highlight routeros&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;6&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;7&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;8&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;9&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;注：下载的打包文件中，附带一个openjdk可以直接使用，如果使用 supervisor 来管理 es, 可以使用&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;supervisor 的 environment 功能，来设置 JAVA_HOME 环境变量&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;配置如下，ES_HOME 即 elasticsearch 解压后文件夹所在目录&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;[es]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;attribute&quot;&gt;command&lt;/span&gt;=&lt;span class=&quot;variable&quot;&gt;$ES_HOME&lt;/span&gt;/bin/elasticsearch&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;attribute&quot;&gt;directory&lt;/span&gt;=&lt;span class=&quot;variable&quot;&gt;$ES_HOME&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;attribute&quot;&gt;environment&lt;/span&gt;=JAVA_HOME=$ES_HOME/jdk&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;
&lt;p&gt;单机没什么问题，很顺畅的就启动起来了，通过 curl &lt;a href=&quot;http://ip:es_port&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;http://ip:es_port&lt;/a&gt; 可以得到很熟悉的返回：&lt;br&gt;&lt;figure class=&quot;highlight json&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;6&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;7&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;8&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;9&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;10&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;11&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;12&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;13&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;14&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;15&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;16&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;17&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  &lt;span class=&quot;attr&quot;&gt;&quot;name&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;node1&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  &lt;span class=&quot;attr&quot;&gt;&quot;cluster_name&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;cluster-name&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  &lt;span class=&quot;attr&quot;&gt;&quot;cluster_uuid&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;2c2jICSxTHK_jUcJjaBucw&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  &lt;span class=&quot;attr&quot;&gt;&quot;version&quot;&lt;/span&gt; : &amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;number&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;7.8.1&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;build_flavor&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;default&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;build_type&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;tar&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;build_hash&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;build_date&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;2020-07-21T16:40:44.668009Z&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;build_snapshot&quot;&lt;/span&gt; : &lt;span class=&quot;literal&quot;&gt;false&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;lucene_version&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;8.5.1&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;minimum_wire_compatibility_version&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;6.8.0&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;attr&quot;&gt;&quot;minimum_index_compatibility_version&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;6.0.0-beta1&quot;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  &amp;#125;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  &lt;span class=&quot;attr&quot;&gt;&quot;tagline&quot;&lt;/span&gt; : &lt;span class=&quot;string&quot;&gt;&quot;You Know, for Search&quot;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;h2 id=&quot;组建集群&quot;&gt;&lt;a href=&quot;#组建集群&quot; class=&quot;headerlink&quot; title=&quot;组建集群&quot;&gt;&lt;/a&gt;组建集群&lt;/h2&gt;&lt;p&gt;既然单机的起来的，那如法炮制，将配置文件复制一份，修改了其 node name, scp到另外一台机器上，准备好 supervisor 配置，准备开始组建两个节点组成的集群&lt;/p&gt;
&lt;p&gt;满以为不会有什么问题，启动之后，发现，第二个节点，怎么也加入不到第一个节点之中&lt;br&gt;
    
    </summary>
    
      <category term="elasticsearch" scheme="https://javasgl.github.io/categories/elasticsearch/"/>
    
    
      <category term="elasticsearch" scheme="https://javasgl.github.io/tags/elasticsearch/"/>
    
      <category term="cluster" scheme="https://javasgl.github.io/tags/cluster/"/>
    
  </entry>
  
  <entry>
    <title>修改 git submodule</title>
    <link href="https://javasgl.github.io/git-submodule/"/>
    <id>https://javasgl.github.io/git-submodule/</id>
    <published>2020-03-22T09:24:51.000Z</published>
    <updated>2020-03-22T09:50:26.428Z</updated>
    
    <content type="html"><![CDATA[<h2 id="初始"><a href="#初始" class="headerlink" title="初始"></a>初始</h2><p><code>git</code> 项目中，可以使用以下命令为项目添加<code>submodule</code></p><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">git submodule add -<span class="selector-tag">b</span> branch_name git@github<span class="selector-class">.com</span>:javasgl/xxx<span class="selector-class">.git</span> module_path</span><br></pre></td></tr></table></figure><p>执行以上命令后，会在当前项目下生成一个<code>.gitmodules</code> 文件，内容如下</p><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[submodule <span class="string">"module_path"</span>]</span><br><span class="line">path = module_path</span><br><span class="line">url = git@github<span class="selector-class">.com</span>:javasgl/xxx.git</span><br><span class="line">branch = branch_name</span><br></pre></td></tr></table></figure><p>同时也将修改 <code>.git/config</code>文件<br><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[submodule <span class="string">"module_path"</span>]</span><br><span class="line">url = git@github<span class="selector-class">.com</span>:javasgl/xxx.git</span><br><span class="line">active = true</span><br></pre></td></tr></table></figure></p><p>此时，执行以下命令即可从远程拉取相关代码到<code>submodule</code>中:<br><figure class="highlight armasm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="symbol">git</span> <span class="keyword">submodule </span>update --init</span><br><span class="line"></span><br><span class="line">// 可以使用 git <span class="keyword">submodule </span>deinit 重新初始化</span><br></pre></td></tr></table></figure></p><p>执行之后，会在 <code>.git/modules/module_path/</code> 下看到 <code>submodeule</code>相关的代码</p><figure class="highlight asciidoc"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="title">.git/modules</span></span><br><span class="line">└── module<span class="emphasis">_path</span></span><br></pre></td></tr></table></figure><h2 id="变更"><a href="#变更" class="headerlink" title="变更"></a>变更</h2><p>有时候，对于已经添加了submodule的项目，如果想更换 <code>submodule</code> 的地址，那个这就比较麻烦，目前<code>git</code>并没有直接提供相关命令,需要进行一系列的手工操作</p><p>简单记录如下，备查</p><a id="more"></a><p>比如，想把 <code>submodule</code> 的地址换为 <code>github.com/xxx/xxx.git</code></p><ol><li><p>将 <code>submodule</code> 移除 <code>git</code> 版本控制</p><figure class="highlight ceylon"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">git rm <span class="keyword">module</span><span class="number">_p</span>ath</span><br></pre></td></tr></table></figure></li><li><p>移除 <code>.git/config</code> <code>.gitmodules</code> 中的配置</p></li><li><p>删除 <code>.git/modules/</code>下相关代码</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">rm .git<span class="regexp">/modules/m</span>odule_path</span><br></pre></td></tr></table></figure></li><li><p>执行之前的初始化，可以重新<code>add</code>新的<code>submodule</code></p><figure class="highlight armasm"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="symbol">git</span> <span class="keyword">submodule </span><span class="keyword">add </span>github.com/xxx/xxx.git xxx_path</span><br></pre></td></tr></table></figure></li></ol>]]></content>
    
    <summary type="html">
    
      &lt;h2 id=&quot;初始&quot;&gt;&lt;a href=&quot;#初始&quot; class=&quot;headerlink&quot; title=&quot;初始&quot;&gt;&lt;/a&gt;初始&lt;/h2&gt;&lt;p&gt;&lt;code&gt;git&lt;/code&gt; 项目中，可以使用以下命令为项目添加&lt;code&gt;submodule&lt;/code&gt;&lt;/p&gt;
&lt;figure class=&quot;highlight stylus&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;git submodule add -&lt;span class=&quot;selector-tag&quot;&gt;b&lt;/span&gt; branch_name git@github&lt;span class=&quot;selector-class&quot;&gt;.com&lt;/span&gt;:javasgl/xxx&lt;span class=&quot;selector-class&quot;&gt;.git&lt;/span&gt; module_path&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;
&lt;p&gt;执行以上命令后，会在当前项目下生成一个&lt;code&gt;.gitmodules&lt;/code&gt; 文件，内容如下&lt;/p&gt;
&lt;figure class=&quot;highlight stylus&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;[submodule &lt;span class=&quot;string&quot;&gt;&quot;module_path&quot;&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	path = module_path&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	url = git@github&lt;span class=&quot;selector-class&quot;&gt;.com&lt;/span&gt;:javasgl/xxx.git&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	branch = branch_name&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;
&lt;p&gt;同时也将修改 &lt;code&gt;.git/config&lt;/code&gt;文件&lt;br&gt;&lt;figure class=&quot;highlight stylus&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;[submodule &lt;span class=&quot;string&quot;&gt;&quot;module_path&quot;&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	url = git@github&lt;span class=&quot;selector-class&quot;&gt;.com&lt;/span&gt;:javasgl/xxx.git&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	active = true&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;此时，执行以下命令即可从远程拉取相关代码到&lt;code&gt;submodule&lt;/code&gt;中:&lt;br&gt;&lt;figure class=&quot;highlight armasm&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;symbol&quot;&gt;git&lt;/span&gt; &lt;span class=&quot;keyword&quot;&gt;submodule &lt;/span&gt;update --init&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;// 可以使用 git &lt;span class=&quot;keyword&quot;&gt;submodule &lt;/span&gt;deinit 重新初始化&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;执行之后，会在 &lt;code&gt;.git/modules/module_path/&lt;/code&gt; 下看到 &lt;code&gt;submodeule&lt;/code&gt;相关的代码&lt;/p&gt;
&lt;figure class=&quot;highlight asciidoc&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;title&quot;&gt;.git/modules&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;└── module&lt;span class=&quot;emphasis&quot;&gt;_path&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;
&lt;h2 id=&quot;变更&quot;&gt;&lt;a href=&quot;#变更&quot; class=&quot;headerlink&quot; title=&quot;变更&quot;&gt;&lt;/a&gt;变更&lt;/h2&gt;&lt;p&gt;有时候，对于已经添加了submodule的项目，如果想更换 &lt;code&gt;submodule&lt;/code&gt; 的地址，那个这就比较麻烦，目前&lt;code&gt;git&lt;/code&gt;并没有直接提供相关命令,需要进行一系列的手工操作&lt;/p&gt;
&lt;p&gt;简单记录如下，备查&lt;/p&gt;
    
    </summary>
    
      <category term="git" scheme="https://javasgl.github.io/categories/git/"/>
    
    
      <category term="git" scheme="https://javasgl.github.io/tags/git/"/>
    
      <category term="submodule" scheme="https://javasgl.github.io/tags/submodule/"/>
    
  </entry>
  
  <entry>
    <title>缓存常见问题处理</title>
    <link href="https://javasgl.github.io/handle-cache-problems/"/>
    <id>https://javasgl.github.io/handle-cache-problems/</id>
    <published>2020-02-19T01:42:12.000Z</published>
    <updated>2020-02-19T03:00:32.191Z</updated>
    
    <content type="html"><![CDATA[<p>缓存使用过过程中常见问题基本是以下几个问题：</p><ul><li>缓存穿透问题 </li><li>缓存并发问题 </li><li>热点数据问题</li><li>数据一致问题</li></ul><p>这些问题不是独立的，一般都会存在因果关系或者同时存在</p><p>下面列举一下问题原因及基本解决思路</p><h2 id="缓存穿透-过期"><a href="#缓存穿透-过期" class="headerlink" title="缓存穿透/过期"></a>缓存穿透/过期</h2><p>缓存穿透问题是指请求数据在缓存中没有找到，导致请求直接打到数据库的情况</p><p>这种场景下，大致有两类原因可以导致这类问题：<br><a id="more"></a></p><ul><li>1.缓存中没有命中，查询数据库，数据库无此数据，不会去更新缓存，这样导致下次同样的请求都会重复这样的请求，每次都会请求数据库</li><li>2.由于缓存过期时间设置不合理，导致瞬时大量缓存过期，大量请求直接打到数据库</li></ul><p>原因 1 的解决方案:<br>为数据设置默认值，即使从数据库没有查询到，也设置缓存，之后数据有值了去主动更新缓存</p><blockquote><p>这样虽然可以避免查询一直不存在的数据而导致的缓存穿透问题，但是也容易引起恶意攻击，攻击方式：构造大量无效的条件去查询，这样导致缓存中大量的存在默认缓存，导致缓存占用量急剧增大而引起缓存驱逐,针对这种方式，可以为默认缓存设置一些比较短的过期时间来稍微缓解这一问题,只是缓解，无法完全解决这个攻击</p></blockquote><p>原因 2 的解决方案：</p><blockquote><p>不要设置统一的过期时间，为过期时间添加随机数,让数据过期时间分散开，不要集中在同一时间过期</p></blockquote><h2 id="缓存并发"><a href="#缓存并发" class="headerlink" title="缓存并发"></a>缓存并发</h2><p>缓存并发问题是指多个请求请求同一个数据，而且都没有从缓存中获取到数据，并从数据库查询后都去更新缓存，导致缓存重复更新问题<br>大多数场景下，这个问题影响并不大，只是会产生一些重复的数据库查询和重复更新缓存<br>但是如果有些缓存不能重复更新，比如一些自动设置的时间、计数器等，那么重复更新会带来问题</p><p>这个问题的解决方案：</p><blockquote><p>使用<code>CAS</code>方案，即设置缓存之前先检查是否已经设置了，如果设置过，则不作任何操作<br>如果是使用<code>redis</code>,则可以直接使用其<code>SETNX</code>命令</p></blockquote><h2 id="热点数据"><a href="#热点数据" class="headerlink" title="热点数据"></a>热点数据</h2><p>绝大多数系统中，频繁访问的数据只是一小部分，而受硬件、成本等因素，把所有的数据都放在缓存中也不太现实</p><blockquote><p>这个问题，可以使用日志分析、访问计数等方式统计热点数据，基于历史访问情况刷新<code>TOP-N</code>数据到缓存中，并且不定时的去更新这个<code>TOP-N</code>列表</p></blockquote><h2 id="数据一致问题"><a href="#数据一致问题" class="headerlink" title="数据一致问题"></a>数据一致问题</h2><p>这个问题一般比较难弄，只要使用缓存，则必然可能出现缓存不一致的问题,这个问题和使用缓存更新策略有很大的关系，具体问题需要根据不同的缓存更新方式来定制化解决，不能一概而论</p><blockquote><p>这个问题基本思路是使用补偿机制，即使用定时、不定时巡检的方式，刷新缓存数据，保证数据一致性</p></blockquote>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;缓存使用过过程中常见问题基本是以下几个问题：&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;缓存穿透问题 &lt;/li&gt;
&lt;li&gt;缓存并发问题 &lt;/li&gt;
&lt;li&gt;热点数据问题&lt;/li&gt;
&lt;li&gt;数据一致问题&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;这些问题不是独立的，一般都会存在因果关系或者同时存在&lt;/p&gt;
&lt;p&gt;下面列举一下问题原因及基本解决思路&lt;/p&gt;
&lt;h2 id=&quot;缓存穿透-过期&quot;&gt;&lt;a href=&quot;#缓存穿透-过期&quot; class=&quot;headerlink&quot; title=&quot;缓存穿透/过期&quot;&gt;&lt;/a&gt;缓存穿透/过期&lt;/h2&gt;&lt;p&gt;缓存穿透问题是指请求数据在缓存中没有找到，导致请求直接打到数据库的情况&lt;/p&gt;
&lt;p&gt;这种场景下，大致有两类原因可以导致这类问题：&lt;br&gt;
    
    </summary>
    
      <category term="redis" scheme="https://javasgl.github.io/categories/redis/"/>
    
    
      <category term="cache" scheme="https://javasgl.github.io/tags/cache/"/>
    
      <category term="redis" scheme="https://javasgl.github.io/tags/redis/"/>
    
      <category term="memcache" scheme="https://javasgl.github.io/tags/memcache/"/>
    
  </entry>
  
  <entry>
    <title>mysql5.7/mariadb Generated Columns</title>
    <link href="https://javasgl.github.io/mysql-generated-columns/"/>
    <id>https://javasgl.github.io/mysql-generated-columns/</id>
    <published>2020-02-18T00:31:45.000Z</published>
    <updated>2020-02-18T01:57:28.905Z</updated>
    
    <content type="html"><![CDATA[<p>mysql 5.7 以后带来了一个新的功能，即 “Generated Columns”,字面意思可翻译为 “生成列”，顾名思义，这列的数据不能被直接插入或者更新，而是依据一定的规则自动生成。</p><p>Generated Columns 定义方式如下:<br><figure class="highlight prolog"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">col_name data_type [<span class="symbol">GENERATED</span> <span class="symbol">ALWAYS</span>] <span class="symbol">AS</span> (expr)</span><br><span class="line">  [<span class="symbol">VIRTUAL</span> | <span class="symbol">STORED</span>] [<span class="symbol">NOT</span> <span class="symbol">NULL</span> | <span class="symbol">NULL</span>]</span><br><span class="line">  [<span class="symbol">UNIQUE</span> [<span class="symbol">KEY</span>]] [[<span class="symbol">PRIMARY</span>] <span class="symbol">KEY</span>]</span><br><span class="line">  [<span class="symbol">COMMENT</span> <span class="string">'string'</span>]</span><br></pre></td></tr></table></figure></p><p>注：</p><blockquote><p>默认是 VIRTUAL,即 虚拟列，不实际占用存储空间，在读取数据时即时计算而来；<br>STORED 则与之相反，它会生成真实的数据列，和正常的列无异，占用存储空间;<br>两者在限制上各有细微区别，具体可查看文末 mysql/mariadb 链接阅读详情</p></blockquote><p>以下样例,展示了自动生成指定时间字段的月份:<br><a id="more"></a><br><code>c_month</code> 的值通过对<code>c_time</code>进行内置函数<code>month</code>调用自动得到,插入语句中无需指定<code>c_month</code>的值<br><figure class="highlight asciidoc"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">mysql&gt; CREATE TABLE generated<span class="emphasis">_columns_</span>test (c<span class="emphasis">_time datetime,c_</span>month int generated always as(month(c<span class="emphasis">_time)));</span></span><br><span class="line"><span class="emphasis">Query OK, 0 rows affected (0.19 sec)</span></span><br><span class="line"><span class="emphasis"></span></span><br><span class="line"><span class="emphasis">mysql&gt; desc generated_</span>columns<span class="emphasis">_test;</span></span><br><span class="line"><span class="emphasis">+---------+----------+------+-----+---------+-------------------+</span></span><br><span class="line"><span class="emphasis">| Field   | Type     | Null | Key | Default | Extra             |</span></span><br><span class="line"><span class="emphasis">+---------+----------+------+-----+---------+-------------------+</span></span><br><span class="line"><span class="emphasis">| c_</span>time  | datetime | YES  |     | NULL    |                   |</span><br><span class="line">| c<span class="emphasis">_month | int(11)  | YES  |     | NULL    | VIRTUAL GENERATED |</span></span><br><span class="line"><span class="emphasis">+---------+----------+------+-----+---------+-------------------+</span></span><br><span class="line"><span class="emphasis">2 rows in set (0.01 sec)</span></span><br><span class="line"><span class="emphasis"></span></span><br><span class="line"><span class="emphasis">mysql&gt; insert into generated_</span>columns<span class="emphasis">_test(c_</span>time)values (now());</span><br><span class="line">Query OK, 1 row affected (0.00 sec)</span><br><span class="line"></span><br><span class="line">mysql&gt; select * from generated<span class="emphasis">_columns_</span>test;</span><br><span class="line"><span class="code">+---------------------+</span>---------+</span><br><span class="line">| c<span class="emphasis">_time              | c_</span>month |</span><br><span class="line"><span class="code">+---------------------+</span>---------+</span><br><span class="line">| 2020-02-18 09:33:42 |       2 |</span><br><span class="line"><span class="code">+---------------------+</span>---------+</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure></p><p>是不是很熟悉？以前这样的需求，一般需要在程序中处理好月份后和 <code>c_time</code> 一并插入数据库，是 <code>generated columns</code> 则可以做自动生成,将程序逻辑后移到数据库层面了。</p><p>可以设想一下，<code>generated columns</code> 可用于哪些场景？</p><ul><li>根据订单创建时间，自动填充默认过期时间？</li><li>根据发货时间，自动填充有效期？</li><li>……</li></ul><p>当然，<code>generated columns</code>并非银弹，不是所有的业务逻辑都适合运用它，不应该把本该程序处理的部分业务交给数据库去处理，这将给系统维护带来隐患，这和不推荐使用存储过程和自定义函数一样的道理</p><p>总体来说，<code>generated columns</code> 虽然有诸多限制，但总体上使用起来还是很方便的，具体的文档可以查看：</p><ul><li>mysql <a href="https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html" target="_blank" rel="noopener">generated columns</a></li><li>mariadb <a href="https://mariadb.com/kb/en/generated-columns/" target="_blank" rel="noopener">generated columns</a></li></ul>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;mysql 5.7 以后带来了一个新的功能，即 “Generated Columns”,字面意思可翻译为 “生成列”，顾名思义，这列的数据不能被直接插入或者更新，而是依据一定的规则自动生成。&lt;/p&gt;
&lt;p&gt;Generated Columns 定义方式如下:&lt;br&gt;&lt;figure class=&quot;highlight prolog&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;col_name data_type [&lt;span class=&quot;symbol&quot;&gt;GENERATED&lt;/span&gt; &lt;span class=&quot;symbol&quot;&gt;ALWAYS&lt;/span&gt;] &lt;span class=&quot;symbol&quot;&gt;AS&lt;/span&gt; (expr)&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  [&lt;span class=&quot;symbol&quot;&gt;VIRTUAL&lt;/span&gt; | &lt;span class=&quot;symbol&quot;&gt;STORED&lt;/span&gt;] [&lt;span class=&quot;symbol&quot;&gt;NOT&lt;/span&gt; &lt;span class=&quot;symbol&quot;&gt;NULL&lt;/span&gt; | &lt;span class=&quot;symbol&quot;&gt;NULL&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  [&lt;span class=&quot;symbol&quot;&gt;UNIQUE&lt;/span&gt; [&lt;span class=&quot;symbol&quot;&gt;KEY&lt;/span&gt;]] [[&lt;span class=&quot;symbol&quot;&gt;PRIMARY&lt;/span&gt;] &lt;span class=&quot;symbol&quot;&gt;KEY&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;  [&lt;span class=&quot;symbol&quot;&gt;COMMENT&lt;/span&gt; &lt;span class=&quot;string&quot;&gt;&#39;string&#39;&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;注：&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;默认是 VIRTUAL,即 虚拟列，不实际占用存储空间，在读取数据时即时计算而来；&lt;br&gt;STORED 则与之相反，它会生成真实的数据列，和正常的列无异，占用存储空间;&lt;br&gt;两者在限制上各有细微区别，具体可查看文末 mysql/mariadb 链接阅读详情&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;以下样例,展示了自动生成指定时间字段的月份:&lt;br&gt;
    
    </summary>
    
      <category term="mysql" scheme="https://javasgl.github.io/categories/mysql/"/>
    
    
      <category term="mysql" scheme="https://javasgl.github.io/tags/mysql/"/>
    
      <category term="sql" scheme="https://javasgl.github.io/tags/sql/"/>
    
      <category term="mariadb" scheme="https://javasgl.github.io/tags/mariadb/"/>
    
  </entry>
  
  <entry>
    <title>永久修改 mac mac files/proc 限制</title>
    <link href="https://javasgl.github.io/mac-max-limit/"/>
    <id>https://javasgl.github.io/mac-max-limit/</id>
    <published>2019-10-11T06:19:49.000Z</published>
    <updated>2019-10-11T06:39:44.000Z</updated>
    
    <content type="html"><![CDATA[<p>mac 升级后的某一天，本地跑程序时发现提示 <code>open too many files</code>, 竟然提示打开文件数过多。</p><p>使用<code>ulimit -n</code> 查看，最大的文件数限制为 256 ，这明显太小了。 使用<code>launchctl limit</code> 查看,结果如下:</p><figure class="highlight lsl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">~/codes/backupAnyThing/blog(master*) » launchctl limit</span><br><span class="line">cpu         unlimited      unlimited</span><br><span class="line">filesize    unlimited      unlimited</span><br><span class="line">data        unlimited      unlimited</span><br><span class="line">stack       <span class="number">8388608</span>        <span class="number">67104768</span></span><br><span class="line">core        <span class="number">0</span>              unlimited</span><br><span class="line">rss         unlimited      unlimited</span><br><span class="line">memlock     unlimited      unlimited</span><br><span class="line">maxproc     <span class="number">1064</span>           <span class="number">1064</span></span><br><span class="line">maxfiles    <span class="number">256</span>            <span class="number">256</span></span><br></pre></td></tr></table></figure><p>解决方案大致上有三种：</p><ul><li><p>命令行中执行 <code>ulimit -n xxx</code>, 但是仅对当前 session 生效，新开 session 或者重启后会恢复为系统值 </p></li><li><p>在 .bashrc / .zshrc 等脚本中加上 <code>ulimit -n xxxx</code>, 这样每次新开 session 即可自动设置最大文件描述符显示</p></li><li><p>加入到 launchd 服务中, 永久修改</p></li></ul><p>这里记录下第三种方案的步骤</p><p>创建两个 plist文件，民间名分别为 <code>limit.maxproc.plist</code>, <code>limit.maxfiles.plist</code>, 内容如下:</p><p>file: limit.maxfiles.plist<br><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="php"><span class="meta">&lt;?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"UTF-8"</span><span class="meta">?&gt;</span></span></span><br><span class="line"><span class="meta">&lt;!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&gt;</span></span><br><span class="line">  <span class="tag">&lt;<span class="name">plist</span> <span class="attr">version</span>=<span class="string">"1.0"</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">dict</span>&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>Label<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">string</span>&gt;</span>limit.maxfiles<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>ProgramArguments<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">array</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>launchctl<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>limit<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>maxfiles<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>200000<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>200000<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;/<span class="name">array</span>&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>RunAtLoad<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">true</span>/&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>ServiceIPC<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">false</span>/&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">dict</span>&gt;</span></span><br><span class="line">  <span class="tag">&lt;/<span class="name">plist</span>&gt;</span></span><br></pre></td></tr></table></figure></p><p>file: limit.maxproc.plist</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">  <span class="php"><span class="meta">&lt;?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"UTF-8"</span><span class="meta">?&gt;</span></span></span><br><span class="line"><span class="meta">&lt;!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&gt;</span></span><br><span class="line">  <span class="tag">&lt;<span class="name">plist</span> <span class="attr">version</span>=<span class="string">"1.0"</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">dict</span>&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>Label<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">string</span>&gt;</span>limit.maxproc<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>ProgramArguments<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">array</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>launchctl<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>limit<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>maxproc<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>20480<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">          <span class="tag">&lt;<span class="name">string</span>&gt;</span>20480<span class="tag">&lt;/<span class="name">string</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;/<span class="name">array</span>&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>RunAtLoad<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">true</span> /&gt;</span></span><br><span class="line">      <span class="tag">&lt;<span class="name">key</span>&gt;</span>ServiceIPC<span class="tag">&lt;/<span class="name">key</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">false</span> /&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">dict</span>&gt;</span></span><br><span class="line">  <span class="tag">&lt;/<span class="name">plist</span>&gt;</span></span><br></pre></td></tr></table></figure><p>将以上两个文件移动到 <code>/Library/LaunchDaemons</code> 目录下，确保两个文件的权限如下：</p><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-tag">-rw-r--r--</span>  1 <span class="selector-tag">root</span>  <span class="selector-tag">wheel</span>   592<span class="selector-tag">B</span> <span class="selector-tag">Sep</span> 29 16<span class="selector-pseudo">:39</span> <span class="selector-tag">limit</span><span class="selector-class">.maxfiles</span><span class="selector-class">.plist</span></span><br><span class="line"><span class="selector-tag">-rw-r--r--</span>  1 <span class="selector-tag">root</span>  <span class="selector-tag">wheel</span>   589<span class="selector-tag">B</span> <span class="selector-tag">Sep</span> 29 16<span class="selector-pseudo">:40</span> <span class="selector-tag">limit</span><span class="selector-class">.maxproc</span><span class="selector-class">.plist</span></span><br></pre></td></tr></table></figure><p>重启系统，重启后自动生效。</p><p>注: 如果只需要修改 <code>max files</code>, 仅添加 <code>limit.maxfiles.plist</code> 文件即可。</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;mac 升级后的某一天，本地跑程序时发现提示 &lt;code&gt;open too many files&lt;/code&gt;, 竟然提示打开文件数过多。&lt;/p&gt;
&lt;p&gt;使用&lt;code&gt;ulimit -n&lt;/code&gt; 查看，最大的文件数限制为 256 ，这明显太小了。 使用&lt;code&gt;l
      
    
    </summary>
    
      <category term="os" scheme="https://javasgl.github.io/categories/os/"/>
    
    
      <category term="mac" scheme="https://javasgl.github.io/tags/mac/"/>
    
      <category term="limit" scheme="https://javasgl.github.io/tags/limit/"/>
    
  </entry>
  
  <entry>
    <title>angular ng-repeat 双向数据绑定实现动态增删DOM</title>
    <link href="https://javasgl.github.io/angular-repeat-bind-object-of-arrays/"/>
    <id>https://javasgl.github.io/angular-repeat-bind-object-of-arrays/</id>
    <published>2018-11-14T03:44:50.000Z</published>
    <updated>2018-11-14T04:16:53.000Z</updated>
    
    <content type="html"><![CDATA[<p>在实现一个需要动态增删的form表单需求时，<code>MVVM</code>的思维模式下，很容易就想到使用数组的添加和删除来实现DOM元素的动态添加和删除。</p><a id="more"></a><h3 id="代码"><a href="#代码" class="headerlink" title="代码"></a>代码</h3><p>首先定义一个空数组,<code>push</code>几条初始数据(或者在相关按钮的<code>click</code>事件中去对这个数组进行增删)，然后在页面上使用<code>ng-repeat</code>和<code>ng-model</code>在循环中绑定这个数组的数据。<br><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">item= &#123;</span><br><span class="line">name:<span class="string">'default value'</span></span><br><span class="line">&#125;</span><br><span class="line">$scope.items= []</span><br><span class="line">$scope.items.push(item)</span><br><span class="line">$scope.items.push(item)</span><br></pre></td></tr></table></figure></p><p>html页面如下：<br><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">div</span> <span class="attr">ng-repeat</span>=<span class="string">"item in items track by $index"</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">input</span> <span class="attr">type</span>=<span class="string">"text"</span> <span class="attr">ng-model</span>=<span class="string">"item.name"</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">div</span>&gt;</span></span><br></pre></td></tr></table></figure></p><h3 id="问题"><a href="#问题" class="headerlink" title="问题"></a>问题</h3><p>以上代码看似没有问题，但是实际结果却并不预期。页面渲染完成后，修改两个<code>input</code>中的任何一个，数组中的两个元素的数据都保持了一致变动。尝试了一下几种方案之后均无法实现单独绑定:<br><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">方案1：</span><br><span class="line"><span class="tag">&lt;<span class="name">div</span> <span class="attr">ng-repeat</span>=<span class="string">"item in items track by $index"</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">input</span> <span class="attr">type</span>=<span class="string">"text"</span> <span class="attr">ng-model</span>=<span class="string">"items[$index].name"</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">div</span>&gt;</span></span><br><span class="line"></span><br><span class="line">方案2：</span><br><span class="line"><span class="tag">&lt;<span class="name">div</span> <span class="attr">ng-repeat</span>=<span class="string">"item in items track by $index"</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">input</span> <span class="attr">type</span>=<span class="string">"text"</span> <span class="attr">ng-model</span>=<span class="string">"$parent.items[$index].name"</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">div</span>&gt;</span></span><br></pre></td></tr></table></figure></p><p>后来怀疑是代码中<code>push(item)</code>的问题，每次<code>push</code>的时候，添加的都是同一个<code>item</code>,这样数组中的每一个元素引用的都是内存中的同一个<code>item</code>的地址，所以修改任何元素，都会影响内存中的同一个<code>item</code>对象，从而导致所有的数组元素的值都同步一致修改了。</p><h3 id="解决"><a href="#解决" class="headerlink" title="解决"></a>解决</h3><p>基于以上思路，产生了两种解决方案：<br>方案一，每次<code>push</code>的时候，添加一个空对象到数组中，这样避免每个元素引用同一个内存对象<br><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$scope.items.push(&#123;&#125;)</span><br><span class="line">$scope.items.push(&#123;&#125;)</span><br><span class="line">$scope.items.push(&#123;&#125;)</span><br><span class="line">......</span><br></pre></td></tr></table></figure></p><p>方案二，每次<code>push</code>的时候，使用 <code>angular.copy</code>重新复制出一个新对象出来添加到数组中<br><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">item= &#123;</span><br><span class="line">name:<span class="string">'default value'</span></span><br><span class="line">&#125;</span><br><span class="line">$scope.items.push(angular.copy(item))</span><br><span class="line">$scope.items.push(angular.copy(item))</span><br><span class="line">$scope.items.push(angular.copy(item))</span><br><span class="line">......</span><br></pre></td></tr></table></figure></p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;在实现一个需要动态增删的form表单需求时，&lt;code&gt;MVVM&lt;/code&gt;的思维模式下，很容易就想到使用数组的添加和删除来实现DOM元素的动态添加和删除。&lt;/p&gt;
    
    </summary>
    
      <category term="javascript" scheme="https://javasgl.github.io/categories/javascript/"/>
    
    
      <category term="angular" scheme="https://javasgl.github.io/tags/angular/"/>
    
      <category term="js" scheme="https://javasgl.github.io/tags/js/"/>
    
      <category term="ng-repeat" scheme="https://javasgl.github.io/tags/ng-repeat/"/>
    
      <category term="mvvm" scheme="https://javasgl.github.io/tags/mvvm/"/>
    
      <category term="ng-model" scheme="https://javasgl.github.io/tags/ng-model/"/>
    
  </entry>
  
  <entry>
    <title>用 Go Plugin 构建模块化系统</title>
    <link href="https://javasgl.github.io/build-module-system-use-go-plugin/"/>
    <id>https://javasgl.github.io/build-module-system-use-go-plugin/</id>
    <published>2018-07-05T08:48:13.000Z</published>
    <updated>2018-07-05T09:23:03.000Z</updated>
    
    <content type="html"><![CDATA[<p>自从 golang 1.8以后，提供了一个 Plugin 的机制使得 golang能够加载 so 动态链接库文件。<br>这样可以对外发布动态链接库而不需要把源码共享给对方去进行编译,还可以做到按需加载相应的插件来实现动态开启插件相应的功能。</p><p>官方文档在此：<a href="https://golang.org/pkg/plugin/" target="_blank" rel="noopener">https://golang.org/pkg/plugin/</a> </p><a id="more"></a><p>编写一个 Plugin 基本有以下几步:</p><p>1.Plguin 需要有自己的 main package<br>2.编译的时候，使用 <code>go build -buildmode=plugin file.go</code> 来编译<br>3.使用 <code>plugin.Open(path string)</code> 来打开.so文件，同一插件只能打开一次，重复打开会报错<br>4.使用 <code>plugin.LookUp(name string)</code> 来获取插件中对外暴露的方法或者类型<br>5.使用类型断言，断言后执行相应的方法</p><p>main.go<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"plugin"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line"></span><br><span class="line">p, err := plugin.Open(<span class="string">"plugin.so"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line"><span class="built_in">panic</span>(err)</span><br><span class="line">&#125;</span><br><span class="line">m, err := p.Lookup(<span class="string">"GetProductBasePrice"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line"><span class="built_in">panic</span>(err)</span><br><span class="line">&#125;</span><br><span class="line">res := m.(<span class="function"><span class="keyword">func</span><span class="params">(<span class="keyword">int</span>)</span> <span class="title">int</span>)<span class="params">(30)</span></span></span><br><span class="line"><span class="function"><span class="title">fmt</span>.<span class="title">Println</span><span class="params">(res)</span></span></span><br><span class="line"><span class="function">&#125;</span></span><br></pre></td></tr></table></figure></p><p>plugin.go<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">"fmt"</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line"></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">GetProductBasePrice</span><span class="params">(basePrice <span class="keyword">int</span>)</span> <span class="title">int</span></span> &#123;</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> basePrice + <span class="number">200</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>注意几点问题：</p><ol><li>插件中定义的 struct 无法暴露出来，可以让主程序和插件程序import公共的 package 来解决</li><li>私有方法、变量不会被暴露出来</li></ol>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;自从 golang 1.8以后，提供了一个 Plugin 的机制使得 golang能够加载 so 动态链接库文件。&lt;br&gt;这样可以对外发布动态链接库而不需要把源码共享给对方去进行编译,还可以做到按需加载相应的插件来实现动态开启插件相应的功能。&lt;/p&gt;
&lt;p&gt;官方文档在此：&lt;a href=&quot;https://golang.org/pkg/plugin/&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;https://golang.org/pkg/plugin/&lt;/a&gt; &lt;/p&gt;
    
    </summary>
    
      <category term="go" scheme="https://javasgl.github.io/categories/go/"/>
    
    
      <category term="go" scheme="https://javasgl.github.io/tags/go/"/>
    
      <category term="plugin" scheme="https://javasgl.github.io/tags/plugin/"/>
    
  </entry>
  
  <entry>
    <title>sync.Pool缓存池中对象回收时机</title>
    <link href="https://javasgl.github.io/go-sync-pool/"/>
    <id>https://javasgl.github.io/go-sync-pool/</id>
    <published>2018-06-27T03:48:04.000Z</published>
    <updated>2018-06-27T04:04:09.000Z</updated>
    
    <content type="html"><![CDATA[<p>为了缓解GC压力，go标准库在sync包中提供了一个Pool，但是这个Pool和我们一般意义上的Pool不太一样，主要有以下几点区别:<br>1.Pool无法设置大小，所以理论上只受限于系统内存大小。<br>2.Pool中的对象不支持自定义过期时间及策略，究其原因，Pool并不是一个Cache.<br>3.Pool的设计初衷是为了缓解GC压力，所以Pool中的对象会在GC开始前全部清除。</p><a id="more"></a><p>下面这段注释来源于<code>pool.go</code>:<br><figure class="highlight php"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// A Pool is a set of temporary objects that may be individually saved and</span></span><br><span class="line"><span class="comment">// retrieved.</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// Any item stored in the Pool may be removed automatically at any time without</span></span><br><span class="line"><span class="comment">// notification. If the Pool holds the only reference when this happens, the</span></span><br><span class="line"><span class="comment">// item might be deallocated.</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// A Pool is safe for use by multiple goroutines simultaneously.</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// Pool's purpose is to cache allocated but unused items for later reuse,</span></span><br><span class="line"><span class="comment">// relieving pressure on the garbage collector. That is, it makes it easy to</span></span><br><span class="line"><span class="comment">// build efficient, thread-safe free lists. However, it is not suitable for all</span></span><br><span class="line"><span class="comment">// free lists.</span></span><br></pre></td></tr></table></figure></p><p>让我们用代码验证一下 sync.Pool中对象的回收时机:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"runtime"</span></span><br><span class="line"><span class="string">"sync"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line"></span><br><span class="line">p := &amp;sync.Pool&#123;</span><br><span class="line">New: <span class="function"><span class="keyword">func</span><span class="params">()</span> <span class="title">interface</span></span>&#123;&#125; &#123;</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span></span><br><span class="line">&#125;,</span><br><span class="line">&#125;</span><br><span class="line">a := p.Get().(<span class="keyword">int</span>)</span><br><span class="line">p.Put(<span class="number">1</span>)</span><br><span class="line">b := p.Get().(<span class="keyword">int</span>)</span><br><span class="line">fmt.Println(a, b)</span><br><span class="line"></span><br><span class="line">a = p.Get().(<span class="keyword">int</span>)</span><br><span class="line">p.Put(<span class="number">1</span>)</span><br><span class="line">runtime.GC() <span class="comment">//手动调用GC</span></span><br><span class="line">b = p.Get().(<span class="keyword">int</span>)</span><br><span class="line">fmt.Println(a, b)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>执行结果分别打印出了:</p><pre><code>0 10 0</code></pre><p>可见，在手动调用GC之后，Pool中的对象被全部清除了，在Get的时候重新调用定义的New方法创建了新的对象</p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;为了缓解GC压力，go标准库在sync包中提供了一个Pool，但是这个Pool和我们一般意义上的Pool不太一样，主要有以下几点区别:&lt;br&gt;1.Pool无法设置大小，所以理论上只受限于系统内存大小。&lt;br&gt;2.Pool中的对象不支持自定义过期时间及策略，究其原因，Pool并不是一个Cache.&lt;br&gt;3.Pool的设计初衷是为了缓解GC压力，所以Pool中的对象会在GC开始前全部清除。&lt;/p&gt;
    
    </summary>
    
      <category term="go" scheme="https://javasgl.github.io/categories/go/"/>
    
    
      <category term="go" scheme="https://javasgl.github.io/tags/go/"/>
    
      <category term="sync.Pool" scheme="https://javasgl.github.io/tags/sync-Pool/"/>
    
  </entry>
  
  <entry>
    <title>使用go-svc安全退出程序</title>
    <link href="https://javasgl.github.io/go-svc/"/>
    <id>https://javasgl.github.io/go-svc/</id>
    <published>2018-06-26T07:24:02.000Z</published>
    <updated>2018-06-26T08:17:43.000Z</updated>
    
    <content type="html"><![CDATA[<p>阅读 <a href="https://github.com/nsqio/nsq" target="_blank" rel="noopener">NSQ</a> 源码时，看到NSQ使用了 go-svc 来启动nsq的相关程序,使得在程序退出的时候可以做一些释放资源等收尾工作。</p><p>使用起来非常简单,定义一个Service,实现其 svc.Service中的接口即可。</p><a id="more"></a><h3 id="简单使用"><a href="#简单使用" class="headerlink" title="简单使用"></a>简单使用</h3><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"github.com/judwhite/go-svc/svc"</span></span><br><span class="line"><span class="string">"log"</span></span><br><span class="line"><span class="string">"syscall"</span></span><br><span class="line"><span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> program <span class="keyword">struct</span> &#123;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line">prg := &amp;program&#123;&#125;</span><br><span class="line"><span class="keyword">if</span> err := svc.Run(prg, syscall.SIGUSR1, syscall.SIGINT, syscall.SIGTERM); err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatal(err)</span><br><span class="line">&#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(program)</span> <span class="title">Init</span><span class="params">(env svc.Environment)</span> <span class="title">error</span></span> &#123;</span><br><span class="line"></span><br><span class="line">fmt.Println(<span class="string">"init....."</span>)</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(program)</span> <span class="title">Start</span><span class="params">()</span> <span class="title">error</span></span> &#123;</span><br><span class="line"></span><br><span class="line">fmt.Println(<span class="string">"start....."</span>)</span><br><span class="line">fmt.Println(syscall.Getpid())</span><br><span class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> &#123;</span><br><span class="line">ticker := time.NewTicker(<span class="number">2</span> * time.Second)</span><br><span class="line"><span class="keyword">for</span> t := <span class="keyword">range</span> ticker.C &#123;</span><br><span class="line">fmt.Println(<span class="string">"tick at"</span>, t)</span><br><span class="line">&#125;</span><br><span class="line">&#125;()</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(program)</span> <span class="title">Stop</span><span class="params">()</span> <span class="title">error</span></span> &#123;</span><br><span class="line"></span><br><span class="line">fmt.Println(<span class="string">"stop....."</span>)</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="注意事项"><a href="#注意事项" class="headerlink" title="注意事项"></a>注意事项</h3><p>1.Start方法中不能只直接阻塞，需要在Start方法中新开goroutine去写需要阻塞的代码。<br>2.svc.Run()方法的第二个参数可以指定需要程序监听的信号，默认情况下不指定的话，默认会监听 SIGINT和 SIGTERM两个。根据具体需要进行指定，比如本例中还监听了SIGUSR1.</p><h3 id="实现原理"><a href="#实现原理" class="headerlink" title="实现原理"></a>实现原理</h3><p>其实go-svc代码实现很简单，使用了标准库中的<code>os/signal</code>的<code>Notify</code>方法。<br>下面的代码截取于<code>judwhite/go-svc/svc/svc-other.go</code>：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Run runs your Service.</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line"><span class="comment">// Run will block until one of the signals specified in sig is received.</span></span><br><span class="line"><span class="comment">// If sig is empty syscall.SIGINT and syscall.SIGTERM are used by default.</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">Run</span><span class="params">(service Service, sig ...os.Signal)</span> <span class="title">error</span></span> &#123;</span><br><span class="line">env := environment&#123;&#125;</span><br><span class="line"><span class="keyword">if</span> err := service.Init(env); err != <span class="literal">nil</span> &#123;</span><br><span class="line"><span class="keyword">return</span> err</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> err := service.Start(); err != <span class="literal">nil</span> &#123;</span><br><span class="line"><span class="keyword">return</span> err</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> <span class="built_in">len</span>(sig) == <span class="number">0</span> &#123;</span><br><span class="line">sig = []os.Signal&#123;syscall.SIGINT, syscall.SIGTERM&#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">signalChan := <span class="built_in">make</span>(<span class="keyword">chan</span> os.Signal, <span class="number">1</span>)</span><br><span class="line">signalNotify(signalChan, sig...)</span><br><span class="line"><span class="comment">//signalNotify方法其实就是 signal.Notify 方法</span></span><br><span class="line"><span class="comment">//var signalNotify = signal.Notify</span></span><br><span class="line">&lt;-signalChan</span><br><span class="line"></span><br><span class="line"><span class="keyword">return</span> service.Stop()</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>标准库里面的 <code>os/signal</code>中的 <code>Notfiy</code>方法签名如下：<br><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">func Notify(c chan&lt;- os<span class="selector-class">.Signal</span>, sig ..<span class="selector-class">.os</span><span class="selector-class">.Signal</span>) &#123;&#125;</span><br></pre></td></tr></table></figure></p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;阅读 &lt;a href=&quot;https://github.com/nsqio/nsq&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;NSQ&lt;/a&gt; 源码时，看到NSQ使用了 go-svc 来启动nsq的相关程序,使得在程序退出的时候可以做一些释放资源等收尾工作。&lt;/p&gt;
&lt;p&gt;使用起来非常简单,定义一个Service,实现其 svc.Service中的接口即可。&lt;/p&gt;
    
    </summary>
    
      <category term="go" scheme="https://javasgl.github.io/categories/go/"/>
    
    
      <category term="go" scheme="https://javasgl.github.io/tags/go/"/>
    
      <category term="go-svc" scheme="https://javasgl.github.io/tags/go-svc/"/>
    
      <category term="signal" scheme="https://javasgl.github.io/tags/signal/"/>
    
  </entry>
  
  <entry>
    <title>一个时区引发的`血案`</title>
    <link href="https://javasgl.github.io/go-time-zone/"/>
    <id>https://javasgl.github.io/go-time-zone/</id>
    <published>2018-06-06T06:10:10.000Z</published>
    <updated>2018-06-06T07:09:44.000Z</updated>
    
    <content type="html"><![CDATA[<p>项目中的一个单元测试，其中一个判断是比较两个时间是否相同。代码逻辑简化抽象如下：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">time1:=time.Now()</span><br><span class="line">xorm.engine.NewSession().Update(time1)</span><br><span class="line"><span class="keyword">var</span> time2 *time.Time</span><br><span class="line">xorm.engine.NewSession().Get(&amp;time2)</span><br><span class="line"><span class="keyword">if</span> time2.UTC().Format(<span class="string">"2018-06-06"</span>) != time2.UTC().Format(<span class="string">"2018-06-06"</span>)&#123;</span><br><span class="line"><span class="built_in">panic</span> error</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>介绍一下各组件时区设置：</p><ul><li>mysql 默认配置，<code>SYSTEM</code></li><li>xorm dsn 配置为 <code>parseTime=true&amp;loc=Asia%2FShanghai</code></li></ul><p>以上代码，在本地机器上跑单元测试时无异常，能正常通过。但是，上了CI环境后，单元测试却失败了。<br>通过观察CI的日志和自己打印的日志，发现入库前后，这两个时间相差了8个小时，很明显，这个时间受到时区的影响了。</p><a id="more"></a><p>后来调试发现，本地 mysql time_zone SYSTEM的实际值是 CST (中国标准时间),所以 dsn中指定loc为 <code>Asia/Shanghai</code> 并不会有问题。<br>而CI环境中，mysql的 time_zone SYSTEM的实际值是 UTC.而此时代码中使用dsn却是<code>Asia/Shanghai</code>,这样就会导致 <code>xorm.Get()</code>的时候，将时间按照<code>Asia/Shanghai</code>来解析，所以，解析出来的结果和UTC时间相差了8个小时。</p><p>解决方案：<br>将代码中使用的dsn改为 parseTime=true&amp;loc=Local 。</p><p>最终总结，其实这个问题是可以避免的，起初在设计数据库的时候，时间字段如果没有特殊需求，不应该使用数据库的<code>date</code>,<code>timestamp</code>等类型，而是统一都使用 <code>INT UNSIGNED NOT NULL</code>。这样数字类型的时间戳，即可以避免时区问题，还可以用来直接比较大小和进行算数运算。</p><p>附上mysql查看时区配置的几种方式:<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">select</span> @@global.time_zone,@@session.time_zone;</span><br></pre></td></tr></table></figure></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">show</span> <span class="keyword">variables</span> <span class="keyword">like</span> %<span class="keyword">time_zone</span>%;'</span><br></pre></td></tr></table></figure>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;项目中的一个单元测试，其中一个判断是比较两个时间是否相同。代码逻辑简化抽象如下：&lt;br&gt;&lt;figure class=&quot;highlight go&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;6&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;7&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;time1:=time.Now()&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;xorm.engine.NewSession().Update(time1)&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;keyword&quot;&gt;var&lt;/span&gt; time2 *time.Time&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;xorm.engine.NewSession().Get(&amp;amp;time2)&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;keyword&quot;&gt;if&lt;/span&gt; time2.UTC().Format(&lt;span class=&quot;string&quot;&gt;&quot;2018-06-06&quot;&lt;/span&gt;) != time2.UTC().Format(&lt;span class=&quot;string&quot;&gt;&quot;2018-06-06&quot;&lt;/span&gt;)&amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	&lt;span class=&quot;built_in&quot;&gt;panic&lt;/span&gt; error&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;介绍一下各组件时区设置：&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;mysql 默认配置，&lt;code&gt;SYSTEM&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;xorm dsn 配置为 &lt;code&gt;parseTime=true&amp;amp;loc=Asia%2FShanghai&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;以上代码，在本地机器上跑单元测试时无异常，能正常通过。但是，上了CI环境后，单元测试却失败了。&lt;br&gt;通过观察CI的日志和自己打印的日志，发现入库前后，这两个时间相差了8个小时，很明显，这个时间受到时区的影响了。&lt;/p&gt;
    
    </summary>
    
      <category term="go" scheme="https://javasgl.github.io/categories/go/"/>
    
    
      <category term="go" scheme="https://javasgl.github.io/tags/go/"/>
    
      <category term="time" scheme="https://javasgl.github.io/tags/time/"/>
    
      <category term="timezone" scheme="https://javasgl.github.io/tags/timezone/"/>
    
      <category term="orm" scheme="https://javasgl.github.io/tags/orm/"/>
    
      <category term="sql" scheme="https://javasgl.github.io/tags/sql/"/>
    
  </entry>
  
  <entry>
    <title>go覆盖测试率图形化分析</title>
    <link href="https://javasgl.github.io/go-cover-ui/"/>
    <id>https://javasgl.github.io/go-cover-ui/</id>
    <published>2018-05-23T03:36:53.000Z</published>
    <updated>2018-05-23T04:07:28.000Z</updated>
    
    <content type="html"><![CDATA[<p>go test可以用来做单元测试，-cover参数可以显示测试覆盖率。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go <span class="built_in">test</span> -v ./... -cover</span><br></pre></td></tr></table></figure></p><p>结果如下：<br><figure class="highlight clean"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line">PASS</span><br><span class="line">coverage: <span class="number">34.4</span>% <span class="keyword">of</span> statements</span><br></pre></td></tr></table></figure></p><p>但是有时候需要提高覆盖测试率，就需要知道哪些代码被覆盖到了，而哪些代码没有被覆盖到。这时候可以借助go提供的一个工具来实现。</p><a id="more"></a><ul><li>首先，执行单元测试时，添加 <code>-coverprofile=profileName</code>,生成 profile文件。</li></ul><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go test -v ./<span class="built_in">..</span>. <span class="attribute">-coverprofile</span>=aaaa</span><br></pre></td></tr></table></figure><ul><li>之后，使用<code>go tool cover -html=profileName -o coverage.html</code>来生成html文件后用浏览器打开。</li></ul><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">go<span class="built_in"> tool </span>cover <span class="attribute">-html</span>=aaaa -o coverage.html</span><br><span class="line">open -a /Applications/Google\ Chrome.app coverage.html</span><br></pre></td></tr></table></figure><p>最终看到的效果是：<br><img src="/images/coverage.png" alt="coverage"><br>红色表示没有覆盖到的代码，绿色表示覆盖到的代码，这样就能针对性的编写单元测试，提高测试覆盖率了。</p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;go test可以用来做单元测试，-cover参数可以显示测试覆盖率。&lt;br&gt;&lt;figure class=&quot;highlight bash&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;go &lt;span class=&quot;built_in&quot;&gt;test&lt;/span&gt; -v ./... -cover&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;结果如下：&lt;br&gt;&lt;figure class=&quot;highlight clean&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;...&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;PASS&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;coverage: &lt;span class=&quot;number&quot;&gt;34.4&lt;/span&gt;% &lt;span class=&quot;keyword&quot;&gt;of&lt;/span&gt; statements&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;但是有时候需要提高覆盖测试率，就需要知道哪些代码被覆盖到了，而哪些代码没有被覆盖到。这时候可以借助go提供的一个工具来实现。&lt;/p&gt;
    
    </summary>
    
      <category term="go" scheme="https://javasgl.github.io/categories/go/"/>
    
    
      <category term="go" scheme="https://javasgl.github.io/tags/go/"/>
    
      <category term="UnitTest" scheme="https://javasgl.github.io/tags/UnitTest/"/>
    
  </entry>
  
  <entry>
    <title>使用 reindex 来修改 elasticsearch 索引mapping</title>
    <link href="https://javasgl.github.io/elastic-search-reindex/"/>
    <id>https://javasgl.github.io/elastic-search-reindex/</id>
    <published>2018-04-02T00:32:00.000Z</published>
    <updated>2018-05-23T03:36:23.000Z</updated>
    
    <content type="html"><![CDATA[<p>elasticsearch索引一旦建立，就无法动态修改其字段的映射类型，有时候因为人为原因污染了索引的mapping,这个时候就只能通过重建索引来修改索引的mapping设置了。</p><p>在一次项目中，有一个字段结构如下：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">&#123;</span><br><span class="line"><span class="attr">"logistics"</span>:&#123;</span><br><span class="line"><span class="attr">"company"</span>:<span class="string">"string"</span>,</span><br><span class="line"><span class="attr">"no"</span>:<span class="string">"string"</span></span><br><span class="line">&#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>由于当初创建索引的时候，既没有给这个索引中的这个字段指定合适的类型，也没有通过动态模板来为这个字段指定类型，导致 elasticsearch 默认将这种结构的数据设置成了 Object 类型，而我们真正想要设置的类型却是 Nested 类型。</p><p>这种情况下，一般有两种解决方案。<br><a id="more"></a></p><p>第一种方案，给这个索引追加一个新的字段，同时给这个字段指定类型。<br><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">PUT test/_mapping/test</span><br><span class="line">&#123;</span><br><span class="line">  <span class="string">"properties"</span>: &#123;</span><br><span class="line">    <span class="string">"logisticsV2"</span>:&#123;</span><br><span class="line">      <span class="string">"type"</span>: <span class="string">"nested"</span>,</span><br><span class="line">      <span class="string">"properties"</span>: &#123;</span><br><span class="line">        <span class="string">"company"</span>:&#123;</span><br><span class="line">          <span class="string">"type"</span>:<span class="string">"keyword"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="string">"no"</span>:&#123;</span><br><span class="line">          <span class="string">"type"</span>:<span class="string">"keyword"</span></span><br><span class="line">        &#125;</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>执行之后，index的mapping中就是这样的了：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line">&#123;</span><br><span class="line">　　<span class="attr">"test"</span>:&#123;</span><br><span class="line">　　　　<span class="attr">"mappings"</span>:&#123;</span><br><span class="line">　　　　　　<span class="attr">"test"</span>:&#123;</span><br><span class="line">　　　　　　　　<span class="attr">"properties"</span>:&#123;</span><br><span class="line">　　　　　　　　　　<span class="attr">"logistics"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　<span class="attr">"properties"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　<span class="attr">"company"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　<span class="attr">"type"</span>:<span class="string">"text"</span>,</span><br><span class="line">　　　　　　　　　　　　　　　　<span class="attr">"fields"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　　　<span class="attr">"keyword"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　　　　　<span class="attr">"type"</span>:<span class="string">"keyword"</span>,</span><br><span class="line">　　　　　　　　　　　　　　　　　　　　<span class="attr">"ignore_above"</span>:<span class="number">256</span></span><br><span class="line">　　　　　　　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　　　　　&#125;,</span><br><span class="line">　　　　　　　　　　　　　　<span class="attr">"no"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　<span class="attr">"type"</span>:<span class="string">"text"</span>,</span><br><span class="line">　　　　　　　　　　　　　　　　<span class="attr">"fields"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　　　<span class="attr">"keyword"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　　　　　<span class="attr">"type"</span>:<span class="string">"keyword"</span>,</span><br><span class="line">　　　　　　　　　　　　　　　　　　　　<span class="attr">"ignore_above"</span>:<span class="number">256</span></span><br><span class="line">　　　　　　　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　&#125;,</span><br><span class="line">　　　　　　　　　　<span class="attr">"logisticsV2"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　<span class="attr">"type"</span>:<span class="string">"nested"</span>,</span><br><span class="line">　　　　　　　　　　　　<span class="attr">"properties"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　<span class="attr">"company"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　<span class="attr">"type"</span>:<span class="string">"keyword"</span></span><br><span class="line">　　　　　　　　　　　　　　&#125;,</span><br><span class="line">　　　　　　　　　　　　　　<span class="attr">"no"</span>:&#123;</span><br><span class="line">　　　　　　　　　　　　　　　　<span class="attr">"type"</span>:<span class="string">"keyword"</span></span><br><span class="line">　　　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　　　&#125;</span><br><span class="line">　　　　　　　　&#125;</span><br><span class="line">　　　　　　&#125;</span><br><span class="line">　　　　&#125;</span><br><span class="line">　　&#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>其中 <code>logisticsV2</code>的类型是 nested 类型，之后依赖于 nested 类型的相关功能使用 logisticsV2字段来实现即可。<br>这个中方式有一些弊端，比如数据冗余问题、数据一致性问题等</p><p>第二个方案，使用 elasticsearch 提供的 reindex api 来迁移数据，创建新的索引。<br>首先创建好目标索引,并设置好mapping:<br><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">PUT test_new</span><br><span class="line">PUT test_new/_mapping/test</span><br><span class="line">&#123;</span><br><span class="line">  <span class="string">"properties"</span>: &#123;</span><br><span class="line">    <span class="string">"logistics"</span>:&#123;</span><br><span class="line">      <span class="string">"type"</span>: <span class="string">"nested"</span>,</span><br><span class="line">      <span class="string">"properties"</span>: &#123;</span><br><span class="line">        <span class="string">"company"</span>:&#123;</span><br><span class="line">          <span class="string">"type"</span>:<span class="string">"keyword"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="string">"no"</span>:&#123;</span><br><span class="line">          <span class="string">"type"</span>:<span class="string">"keyword"</span></span><br><span class="line">        &#125;</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>之后，使用 reindex 将原来的索引重建到新的索引上：<br><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">POST _reindex</span><br><span class="line">&#123;</span><br><span class="line">  <span class="string">"source"</span>: &#123;</span><br><span class="line">    <span class="string">"index"</span>: <span class="string">"test"</span></span><br><span class="line">  &#125;,</span><br><span class="line">  <span class="string">"dest"</span>: &#123;</span><br><span class="line">    <span class="string">"index"</span>: <span class="string">"test_new"</span></span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>等待 reindex 完成后， test_new 就结构就就是你想要的mapping了。</p><p>为了对线上业务做到无感，可以使用 alias 别名功能来实现，具体操作可以参考 <a href="/use-alias-migrate-index">使用 Elasticsearch alias 功能切换 indexl</a>. 这里不再赘述.</p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;elasticsearch索引一旦建立，就无法动态修改其字段的映射类型，有时候因为人为原因污染了索引的mapping,这个时候就只能通过重建索引来修改索引的mapping设置了。&lt;/p&gt;
&lt;p&gt;在一次项目中，有一个字段结构如下：&lt;br&gt;&lt;figure class=&quot;highlight json&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;6&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	&lt;span class=&quot;attr&quot;&gt;&quot;logistics&quot;&lt;/span&gt;:&amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;		&lt;span class=&quot;attr&quot;&gt;&quot;company&quot;&lt;/span&gt;:&lt;span class=&quot;string&quot;&gt;&quot;string&quot;&lt;/span&gt;,&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;		&lt;span class=&quot;attr&quot;&gt;&quot;no&quot;&lt;/span&gt;:&lt;span class=&quot;string&quot;&gt;&quot;string&quot;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;由于当初创建索引的时候，既没有给这个索引中的这个字段指定合适的类型，也没有通过动态模板来为这个字段指定类型，导致 elasticsearch 默认将这种结构的数据设置成了 Object 类型，而我们真正想要设置的类型却是 Nested 类型。&lt;/p&gt;
&lt;p&gt;这种情况下，一般有两种解决方案。&lt;br&gt;
    
    </summary>
    
      <category term="elasticsearch" scheme="https://javasgl.github.io/categories/elasticsearch/"/>
    
    
      <category term="elasticsearch" scheme="https://javasgl.github.io/tags/elasticsearch/"/>
    
      <category term="reindex" scheme="https://javasgl.github.io/tags/reindex/"/>
    
  </entry>
  
  <entry>
    <title>python的浅复制和深复制</title>
    <link href="https://javasgl.github.io/python-reference-value-md/"/>
    <id>https://javasgl.github.io/python-reference-value-md/</id>
    <published>2018-03-30T10:54:49.000Z</published>
    <updated>2018-04-02T00:28:10.000Z</updated>
    
    <content type="html"><![CDATA[<p>最近使用python过程中遇到了一个看似诡异的问题,由于惯性思维，这个问题花了一点时间一步步调试才找到。<br>先来看看下面的代码的结果：<br><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-tag">a</span> = [<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>]</span><br><span class="line"><span class="selector-tag">b</span> = a</span><br><span class="line"><span class="selector-tag">b</span>.append(<span class="number">4</span>)</span><br><span class="line"><span class="function"><span class="title">print</span><span class="params">(b)</span></span></span><br><span class="line"><span class="function"><span class="title">print</span><span class="params">(a)</span></span></span><br></pre></td></tr></table></figure></p><p>如果这段代码放在其他语言,Java 或者 PHP 中，结果大多是这样的：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>]</span><br><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>]</span><br></pre></td></tr></table></figure></p><p>但是在 python 中，实际执行的结果却是这样的：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>]</span><br><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>]</span><br></pre></td></tr></table></figure></p><p>修改 b 的值，却影响了 a 的值。这个就是 python 和其他语言有一点不一样的地方了，简单可以理解为：python 没有赋值，只有引用。<br>如果以其他语言的方式来理解，那就是 a,b 都是指向同一块内存数据的地址，所以修改其中任何一个，都会引起彼此的变化，因为底层数据都是同一份。</p><p>现在，如果需要这段代码表现的和其他语言一致，该怎么写呢？</p><a id="more"></a><h3 id="浅复制-shallow-copy"><a href="#浅复制-shallow-copy" class="headerlink" title="浅复制(shallow copy)"></a>浅复制(shallow copy)</h3><p>对于list 列表，python 有 copy 方法：<br><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-tag">a</span> = [<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>]</span><br><span class="line"><span class="selector-tag">b</span> = <span class="selector-tag">a</span>.copy()</span><br><span class="line"><span class="selector-tag">b</span>.append(<span class="number">4</span>)</span><br><span class="line"><span class="function"><span class="title">print</span><span class="params">(b)</span></span></span><br><span class="line"><span class="function"><span class="title">print</span><span class="params">(a)</span></span></span><br></pre></td></tr></table></figure></p><p>代码执行结果就基本能达到预期了：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>]</span><br><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>]</span><br></pre></td></tr></table></figure></p><h3 id="深复制-deep-coopy"><a href="#深复制-deep-coopy" class="headerlink" title="深复制(deep coopy)"></a>深复制(deep coopy)</h3><p>但是需要注意的是，这个 copy 方法是浅复制(shallow copy),那么它对于嵌套对象就无能为力了。<br>如果需要复制嵌套对象，可以使用 import copy ,然后使用 <code>copy.deepcoopy()</code>来实现对嵌套对象的复制<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> copy</span><br><span class="line">a = [<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,[<span class="number">4</span>,<span class="number">5</span>]]</span><br><span class="line"><span class="comment"># b = a.copy()</span></span><br><span class="line">b = copy.deepcopy(a)</span><br><span class="line">b.append(<span class="number">7</span>)</span><br><span class="line">b[<span class="number">3</span>][<span class="number">1</span>]=<span class="number">6</span></span><br><span class="line">print(b)</span><br><span class="line">print(a)</span><br></pre></td></tr></table></figure></p><p>代码执行结果是这样的：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, [<span class="number">4</span>, <span class="number">6</span>],<span class="number">7</span>]</span><br><span class="line">[<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, [<span class="number">4</span>, <span class="number">5</span>]]</span><br></pre></td></tr></table></figure></p><p>可以看到 a 并没有因为 b 对嵌套内容的修改而改变。这里需要注意的一点是，如果修改的不是嵌套的内容，那么普通的copy也还是可以的，<br>此例中，append(7) 时，使用 <code>copy</code> 或者 <code>copy.deepcopy</code> 的效果是一样的，a的值都不会受到影响。</p><p>deepcopy的本质是递归复制。</p><p>对于 list 数据结构而言，可以用 list(x) 来代替 copy, 不过它依然是浅复制,这点需要注意。</p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;最近使用python过程中遇到了一个看似诡异的问题,由于惯性思维，这个问题花了一点时间一步步调试才找到。&lt;br&gt;先来看看下面的代码的结果：&lt;br&gt;&lt;figure class=&quot;highlight stylus&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;selector-tag&quot;&gt;a&lt;/span&gt; = [&lt;span class=&quot;number&quot;&gt;1&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;2&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;3&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;selector-tag&quot;&gt;b&lt;/span&gt; = a&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;selector-tag&quot;&gt;b&lt;/span&gt;.append(&lt;span class=&quot;number&quot;&gt;4&lt;/span&gt;)&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;function&quot;&gt;&lt;span class=&quot;title&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;params&quot;&gt;(b)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;function&quot;&gt;&lt;span class=&quot;title&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;params&quot;&gt;(a)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;如果这段代码放在其他语言,Java 或者 PHP 中，结果大多是这样的：&lt;br&gt;&lt;figure class=&quot;highlight json&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;[&lt;span class=&quot;number&quot;&gt;1&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;2&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;3&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;4&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;[&lt;span class=&quot;number&quot;&gt;1&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;2&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;3&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;但是在 python 中，实际执行的结果却是这样的：&lt;br&gt;&lt;figure class=&quot;highlight json&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;[&lt;span class=&quot;number&quot;&gt;1&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;2&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;3&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;4&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;[&lt;span class=&quot;number&quot;&gt;1&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;2&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;3&lt;/span&gt;,&lt;span class=&quot;number&quot;&gt;4&lt;/span&gt;]&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;修改 b 的值，却影响了 a 的值。这个就是 python 和其他语言有一点不一样的地方了，简单可以理解为：python 没有赋值，只有引用。&lt;br&gt;如果以其他语言的方式来理解，那就是 a,b 都是指向同一块内存数据的地址，所以修改其中任何一个，都会引起彼此的变化，因为底层数据都是同一份。&lt;/p&gt;
&lt;p&gt;现在，如果需要这段代码表现的和其他语言一致，该怎么写呢？&lt;/p&gt;
    
    </summary>
    
      <category term="python" scheme="https://javasgl.github.io/categories/python/"/>
    
    
      <category term="python" scheme="https://javasgl.github.io/tags/python/"/>
    
      <category term="shallowcopy" scheme="https://javasgl.github.io/tags/shallowcopy/"/>
    
      <category term="deepcopy" scheme="https://javasgl.github.io/tags/deepcopy/"/>
    
  </entry>
  
  <entry>
    <title>python monkey patch解决gevent block forever问题</title>
    <link href="https://javasgl.github.io/python-monkey-patch/"/>
    <id>https://javasgl.github.io/python-monkey-patch/</id>
    <published>2018-03-12T03:31:53.000Z</published>
    <updated>2018-04-01T23:40:00.000Z</updated>
    
    <content type="html"><![CDATA[<p>在 python 中使用 multiprocess 进行多线程或者多进程处理的时候，遇到了 gevent 问题：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">gevent.hub.LoopExit:(<span class="string">'This operation would block forever....'</span>)</span><br></pre></td></tr></table></figure></p><p>这个问题可以通过打上 monkey patch(猴子补丁)来规避。</p><p>在导入 multiprocess 库的文件头部加入以下代码：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> gevent <span class="keyword">import</span> monkey</span><br><span class="line"></span><br><span class="line">monkey.patch_all()</span><br></pre></td></tr></table></figure></p><p>注意，这两行代码需要放置在文件导入库的最前面.</p><p>我遇到的场景下， 导致 LoopExit 的原因是 multiprocess 的job中远程网络请求超时导致的，如果相应的job任务中没有远程请求的话，即使不用 pathch，代码也是能够正常运行的。由于避免加上超时处理的复杂逻辑，就简单的使用monkey patch来解决这个问题。</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;在 python 中使用 multiprocess 进行多线程或者多进程处理的时候，遇到了 gevent 问题：&lt;br&gt;&lt;figure class=&quot;highlight python&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span cla
      
    
    </summary>
    
      <category term="python" scheme="https://javasgl.github.io/categories/python/"/>
    
    
      <category term="python" scheme="https://javasgl.github.io/tags/python/"/>
    
      <category term="monkey" scheme="https://javasgl.github.io/tags/monkey/"/>
    
      <category term="patch" scheme="https://javasgl.github.io/tags/patch/"/>
    
  </entry>
  
  <entry>
    <title>python asyncio 简单使用</title>
    <link href="https://javasgl.github.io/use-asyncio-in-python/"/>
    <id>https://javasgl.github.io/use-asyncio-in-python/</id>
    <published>2018-03-07T07:22:37.000Z</published>
    <updated>2018-04-01T23:40:00.000Z</updated>
    
    <content type="html"><![CDATA[<p>在python 3.4及以上的版本中，可以通过 asyncio 来实现协程(coroutine)。在 python 3.5以后可以使用 async/await 来简化代码，如果python版本是3.4的话，需要通过 <code>@asyncio.coroutine</code>注解 和 <code>yield from</code> 来实现。</p><p>一下代码是3.5+以后的写法，和3.4版本的写法主要区别在于:</p><ol><li>用 async 代替了 @asyncio.coroutine </li><li>用 await 代替了 yield from</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">task</span><span class="params">(n)</span>:</span></span><br><span class="line">print(<span class="string">"do task:"</span>,n)</span><br><span class="line"><span class="keyword">await</span> asyncio.sleep(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line">tasks = [ task(_) <span class="keyword">for</span> _ <span class="keyword">in</span> range(<span class="number">10</span>)]</span><br><span class="line"></span><br><span class="line">loop = asyncio.get_event_loop()</span><br><span class="line">res, _ = loop.run_until_complete(asyncio.wait(tasks))</span><br><span class="line">loop.close()</span><br><span class="line"><span class="keyword">for</span> _ <span class="keyword">in</span> res:</span><br><span class="line">print(_.result())</span><br></pre></td></tr></table></figure><p>参考：<br>    廖雪峰的Python3.x教程-<a href="https://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432090954004980bd351f2cd4cc18c9e6c06d855c498000" target="_blank" rel="noopener">异步io</a></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;在python 3.4及以上的版本中，可以通过 asyncio 来实现协程(coroutine)。在 python 3.5以后可以使用 async/await 来简化代码，如果python版本是3.4的话，需要通过 &lt;code&gt;@asyncio.coroutine&lt;/cod
      
    
    </summary>
    
      <category term="python" scheme="https://javasgl.github.io/categories/python/"/>
    
    
      <category term="python" scheme="https://javasgl.github.io/tags/python/"/>
    
      <category term="coroutine" scheme="https://javasgl.github.io/tags/coroutine/"/>
    
      <category term="asyncio" scheme="https://javasgl.github.io/tags/asyncio/"/>
    
  </entry>
  
  <entry>
    <title>python 多进程、多线程简单使用</title>
    <link href="https://javasgl.github.io/use-theading-in-python/"/>
    <id>https://javasgl.github.io/use-theading-in-python/</id>
    <published>2018-03-07T06:42:01.000Z</published>
    <updated>2018-04-01T23:40:00.000Z</updated>
    
    <content type="html"><![CDATA[<p>python 中可以使用 multiprocessing (多进程) 和 multiprocessing.dummy(多线程) 来实现并发操作。<br>两者在使用方式上一样，只不过实现并发的方式不同。</p><p>以下简单使用案例以 multiprocessing.dummy 多线程为例：<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">from</span> multiprocessing.dummy <span class="keyword">import</span> Pool</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">compute</span><span class="params">(param)</span>:</span></span><br><span class="line"><span class="keyword">return</span> param*<span class="number">100</span></span><br><span class="line"></span><br><span class="line">params = [ _ <span class="keyword">for</span> _ <span class="keyword">in</span> range(<span class="number">10</span>) ]</span><br><span class="line"></span><br><span class="line">pool = Pool()</span><br><span class="line"></span><br><span class="line">results = pool.map(compute, params)</span><br><span class="line"></span><br><span class="line">pool.close()</span><br><span class="line"></span><br><span class="line">pool.join()</span><br><span class="line"></span><br><span class="line">print(results)</span><br></pre></td></tr></table></figure></p><p>注意：多进程(multiprocessing) 无法在 celery 等后台进程中使用，因为 celery 等后台进程不再允许生成子进程,这时候就只能使用多线程或者协程了。</p><figure class="highlight applescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">daemonic processes are <span class="keyword">not</span> allowed <span class="keyword">to</span> have children</span><br></pre></td></tr></table></figure>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;python 中可以使用 multiprocessing (多进程) 和 multiprocessing.dummy(多线程) 来实现并发操作。&lt;br&gt;两者在使用方式上一样，只不过实现并发的方式不同。&lt;/p&gt;
&lt;p&gt;以下简单使用案例以 multiprocessing.dum
      
    
    </summary>
    
      <category term="python" scheme="https://javasgl.github.io/categories/python/"/>
    
    
      <category term="python" scheme="https://javasgl.github.io/tags/python/"/>
    
      <category term="thread" scheme="https://javasgl.github.io/tags/thread/"/>
    
  </entry>
  
  <entry>
    <title>go build 常见编译优化</title>
    <link href="https://javasgl.github.io/go-build-args/"/>
    <id>https://javasgl.github.io/go-build-args/</id>
    <published>2018-02-27T08:09:14.000Z</published>
    <updated>2018-04-01T23:40:00.000Z</updated>
    
    <content type="html"><![CDATA[<p>一般情况下，go build 可以直接编译程序，无需额外的参数设定。<br>但在编译生产环境下使用的可执行程序时候，go build 的一些参数还是很有用的。</p><h2 id="减小编译后可执行程序的大小"><a href="#减小编译后可执行程序的大小" class="headerlink" title="减小编译后可执行程序的大小"></a>减小编译后可执行程序的大小</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">go</span> build -ldflags <span class="string">'-w -s'</span></span><br></pre></td></tr></table></figure><p>说明：<br>    -w 禁止生成debug信息,注意使用该选项后，无法使用 gdb 进行调试<br>    -s 禁用符号表<br>可以使用 <code>go tool link --help</code> 查看 ldflags 各参数含义</p><h2 id="禁止gc优化和内联"><a href="#禁止gc优化和内联" class="headerlink" title="禁止gc优化和内联"></a>禁止gc优化和内联</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">go</span> build -gcflags <span class="string">'-N -l'</span></span><br></pre></td></tr></table></figure><p>说明:<br>    -N 禁止编译优化<br>    -l 禁止内联,禁止内联也可以一定程度上减小可执行程序大小<br>可以使用 <code>go tool compile --help</code> 查看 gcflags 各参数含义</p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;一般情况下，go build 可以直接编译程序，无需额外的参数设定。&lt;br&gt;但在编译生产环境下使用的可执行程序时候，go build 的一些参数还是很有用的。&lt;/p&gt;
&lt;h2 id=&quot;减小编译后可执行程序的大小&quot;&gt;&lt;a href=&quot;#减小编译后可执行程序的大小&quot; class
      
    
    </summary>
    
      <category term="go" scheme="https://javasgl.github.io/categories/go/"/>
    
    
      <category term="go" scheme="https://javasgl.github.io/tags/go/"/>
    
  </entry>
  
  <entry>
    <title>golang同时监听TCP、HTTP端口提供服务</title>
    <link href="https://javasgl.github.io/golang-listen-tcp-and-http-port/"/>
    <id>https://javasgl.github.io/golang-listen-tcp-and-http-port/</id>
    <published>2018-02-24T03:45:30.000Z</published>
    <updated>2018-04-01T23:40:00.000Z</updated>
    
    <content type="html"><![CDATA[<p>之前一个golang写的服务，是使用 TCP 方式进行通信的，但是并没很好的处理粘包问题(其实是根本就没有处理粘包问题)。项目需要添加新功能后迅速上线,所以准备先采用http来通信，避免粘包问题。</p><p>项目入口 main.go 中，之前是监听 TCP 连接，代码大致如下：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span>&#123;</span><br><span class="line">server,err:= net.Listen(<span class="string">"tcp"</span>,<span class="string">"host:port"</span>)</span><br><span class="line"><span class="keyword">if</span> err!=<span class="literal">nil</span>&#123;</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">defer</span> server.Close()</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span>&#123;</span><br><span class="line">conn,err:= server.Accept() </span><br><span class="line"><span class="keyword">if</span> err!=<span class="literal">nil</span>&#123;</span><br><span class="line"><span class="keyword">continue</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">go</span> handleConn(conn)</span><br><span class="line">&#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">handleConn</span><span class="params">(conn net.Conn)</span></span>&#123;</span><br><span class="line"><span class="comment">//do somethings</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>现在需要在此基础之上监听 http 连接,由于 http.ListenAndServe() 方法是阻塞的，所以需要新开goroutine进行监听，代码示意如下：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span>&#123;</span><br><span class="line"></span><br><span class="line">http.HandleFunc(<span class="string">"/hi"</span>, Router)</span><br><span class="line"><span class="comment">//因为会阻塞，所以需要新开goroutine进行监听</span></span><br><span class="line"><span class="keyword">go</span> http.ListenAndServe(<span class="string">"host:http_port"</span>, <span class="literal">nil</span>)</span><br><span class="line"></span><br><span class="line">server,err:= net.Listen(<span class="string">"tcp"</span>,<span class="string">"host:tcp_port"</span>)</span><br><span class="line"><span class="keyword">if</span> err!=<span class="literal">nil</span>&#123;</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">defer</span> server.Close()</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span>&#123;</span><br><span class="line">conn,err:= server.Accept() </span><br><span class="line"><span class="keyword">if</span> err!=<span class="literal">nil</span>&#123;</span><br><span class="line"><span class="keyword">continue</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">go</span> handleConn(conn)</span><br><span class="line">&#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">handleConn</span><span class="params">(conn net.Conn)</span></span>&#123;</span><br><span class="line"><span class="comment">//do somethings</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>]]></content>
    
    <summary type="html">
    
      
      
        &lt;p&gt;之前一个golang写的服务，是使用 TCP 方式进行通信的，但是并没很好的处理粘包问题(其实是根本就没有处理粘包问题)。项目需要添加新功能后迅速上线,所以准备先采用http来通信，避免粘包问题。&lt;/p&gt;
&lt;p&gt;项目入口 main.go 中，之前是监听 TCP 连接，代码大
      
    
    </summary>
    
      <category term="go" scheme="https://javasgl.github.io/categories/go/"/>
    
    
      <category term="go" scheme="https://javasgl.github.io/tags/go/"/>
    
      <category term="http" scheme="https://javasgl.github.io/tags/http/"/>
    
  </entry>
  
  <entry>
    <title>使用 Elasticsearch alias 功能切换 index</title>
    <link href="https://javasgl.github.io/use-alias-migrate-index/"/>
    <id>https://javasgl.github.io/use-alias-migrate-index/</id>
    <published>2017-12-20T09:20:11.000Z</published>
    <updated>2018-04-01T23:40:00.000Z</updated>
    
    <content type="html"><![CDATA[<p>有时候，需要将已有的索引做些调整重建为另外一个索引，但是为了不影响线上的访问，需要无缝切换到新的索引上。</p><p>基本上有两种方案，一是改代码，使线上代码访问新的索引，二是使用 alias 别名功能。</p><p>第一种方案有弊端，在代码进行发布到具体机器之前的并不能保证所有的访问都是访问的新的索引。</p><p>这里选择通过 alias 来切换索引，因为 alias 中的多条命令是原子性。</p><a id="more"></a><p>1.创建索引 test 和 test_v2<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">PUT <span class="built_in">test</span></span><br><span class="line">PUT test_v2</span><br><span class="line">GET <span class="built_in">test</span>*</span><br></pre></td></tr></table></figure></p><p>结果如下:<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">&#123;</span><br><span class="line">  <span class="attr">"test"</span>: &#123;</span><br><span class="line">    <span class="attr">"aliases"</span>: &#123;&#125;,</span><br><span class="line">    <span class="attr">"mappings"</span>: &#123;&#125;,</span><br><span class="line">    <span class="attr">"settings"</span>: &#123;</span><br><span class="line">      <span class="attr">"index"</span>: &#123;</span><br><span class="line">        <span class="attr">"creation_date"</span>: <span class="string">"1514183636616"</span>,</span><br><span class="line">        <span class="attr">"number_of_shards"</span>: <span class="string">"5"</span>,</span><br><span class="line">        <span class="attr">"number_of_replicas"</span>: <span class="string">"1"</span>,</span><br><span class="line">        <span class="attr">"uuid"</span>: <span class="string">"oOgEhLzqS_usf7floIhNig"</span>,</span><br><span class="line">        <span class="attr">"version"</span>: &#123;</span><br><span class="line">          <span class="attr">"created"</span>: <span class="string">"5030099"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="attr">"provided_name"</span>: <span class="string">"test"</span></span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;,</span><br><span class="line">  <span class="attr">"test_v2"</span>: &#123;</span><br><span class="line">    <span class="attr">"aliases"</span>: &#123;&#125;,</span><br><span class="line">    <span class="attr">"mappings"</span>: &#123;&#125;,</span><br><span class="line">    <span class="attr">"settings"</span>: &#123;</span><br><span class="line">      <span class="attr">"index"</span>: &#123;</span><br><span class="line">        <span class="attr">"creation_date"</span>: <span class="string">"1514183638407"</span>,</span><br><span class="line">        <span class="attr">"number_of_shards"</span>: <span class="string">"5"</span>,</span><br><span class="line">        <span class="attr">"number_of_replicas"</span>: <span class="string">"1"</span>,</span><br><span class="line">        <span class="attr">"uuid"</span>: <span class="string">"o6x6VtmQSFaOpOB-OpIwBg"</span>,</span><br><span class="line">        <span class="attr">"version"</span>: &#123;</span><br><span class="line">          <span class="attr">"created"</span>: <span class="string">"5030099"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="attr">"provided_name"</span>: <span class="string">"test_v2"</span></span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>2.现在需要将对 test 的访问无缝切换到 test_v2,使用 alias 功能即可做到，对线上服务、对用户无感<br><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">POST _aliases</span><br><span class="line">&#123;</span><br><span class="line">  <span class="string">"actions"</span>: [</span><br><span class="line">    &#123;</span><br><span class="line">      <span class="string">"add"</span>: &#123;</span><br><span class="line">        <span class="string">"index"</span>: <span class="string">"test_v2"</span>,</span><br><span class="line">        <span class="string">"alias"</span>: <span class="string">"test"</span></span><br><span class="line">      &#125;</span><br><span class="line">    &#125;,</span><br><span class="line">    &#123;</span><br><span class="line">      <span class="string">"remove_index"</span>: &#123;</span><br><span class="line">        <span class="string">"index"</span>: <span class="string">"test"</span></span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  ]</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p><p>actions 中的多条命令是原子性的，所以可以做到无缝切换，不必担心切换过程中会存在访问不到 test 时候。</p><p>3.命令解释<br>actions 中第一条命令：add 命令，给索引 test_v2 添加了一个别名 test。<br>actions 中第二条命令：remove_index， 删除原来的 test 索引。<br>执行之后，原来的 test索引会被删除，而对于访问方而言,访问的索引依然是 test,不过访问的是一个别名，底层实际访问的索引已经是 test_v2了</p><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">GET test*</span><br><span class="line"></span><br><span class="line">&#123;</span><br><span class="line">  <span class="string">"test_v2"</span>: &#123;</span><br><span class="line">    <span class="string">"aliases"</span>: &#123;</span><br><span class="line">      <span class="string">"test"</span>: &#123;&#125;</span><br><span class="line">    &#125;,</span><br><span class="line">    <span class="string">"mappings"</span>: &#123;&#125;,</span><br><span class="line">    <span class="string">"settings"</span>: &#123;</span><br><span class="line">      <span class="string">"index"</span>: &#123;</span><br><span class="line">        <span class="string">"creation_date"</span>: <span class="string">"1514183638407"</span>,</span><br><span class="line">        <span class="string">"number_of_shards"</span>: <span class="string">"5"</span>,</span><br><span class="line">        <span class="string">"number_of_replicas"</span>: <span class="string">"1"</span>,</span><br><span class="line">        <span class="string">"uuid"</span>: <span class="string">"o6x6VtmQSFaOpOB-OpIwBg"</span>,</span><br><span class="line">        <span class="string">"version"</span>: &#123;</span><br><span class="line">          <span class="string">"created"</span>: <span class="string">"5030099"</span></span><br><span class="line">        &#125;,</span><br><span class="line">        <span class="string">"provided_name"</span>: <span class="string">"test_v2"</span></span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>从结果中可见，test索引已经不存在了，但是通过 GET test 还是能访问到,而现在 test_v2 多了一个 alias test。</p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;有时候，需要将已有的索引做些调整重建为另外一个索引，但是为了不影响线上的访问，需要无缝切换到新的索引上。&lt;/p&gt;
&lt;p&gt;基本上有两种方案，一是改代码，使线上代码访问新的索引，二是使用 alias 别名功能。&lt;/p&gt;
&lt;p&gt;第一种方案有弊端，在代码进行发布到具体机器之前的并不能保证所有的访问都是访问的新的索引。&lt;/p&gt;
&lt;p&gt;这里选择通过 alias 来切换索引，因为 alias 中的多条命令是原子性。&lt;/p&gt;
    
    </summary>
    
      <category term="elasticsearch" scheme="https://javasgl.github.io/categories/elasticsearch/"/>
    
    
      <category term="elasticsearch" scheme="https://javasgl.github.io/tags/elasticsearch/"/>
    
      <category term="es" scheme="https://javasgl.github.io/tags/es/"/>
    
      <category term="alias" scheme="https://javasgl.github.io/tags/alias/"/>
    
  </entry>
  
  <entry>
    <title>python celery 强制使用root用户执行</title>
    <link href="https://javasgl.github.io/python-celery-force-root/"/>
    <id>https://javasgl.github.io/python-celery-force-root/</id>
    <published>2017-12-19T07:20:22.000Z</published>
    <updated>2018-04-01T23:40:00.000Z</updated>
    
    <content type="html"><![CDATA[<p>当 celery 使用  root 用户执行时，会报出如下警告:<br><figure class="highlight ada"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">RuntimeWarning: You<span class="symbol">'re</span> running the worker <span class="keyword">with</span> superuser privileges: this <span class="keyword">is</span> absolutely <span class="keyword">not</span> recommended!</span><br></pre></td></tr></table></figure></p><p>低版本的 celery 情况不清楚，但是 celery 4.1.0 版本中，这个警告是无法去除的。因为源码中是这样写的:</p><a id="more"></a><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">check_privileges</span><span class="params">(accept_content)</span>:</span></span><br><span class="line">    uid = os.getuid() <span class="keyword">if</span> hasattr(os, <span class="string">'getuid'</span>) <span class="keyword">else</span> <span class="number">65535</span></span><br><span class="line">    gid = os.getgid() <span class="keyword">if</span> hasattr(os, <span class="string">'getgid'</span>) <span class="keyword">else</span> <span class="number">65535</span></span><br><span class="line">    euid = os.geteuid() <span class="keyword">if</span> hasattr(os, <span class="string">'geteuid'</span>) <span class="keyword">else</span> <span class="number">65535</span></span><br><span class="line">    egid = os.getegid() <span class="keyword">if</span> hasattr(os, <span class="string">'getegid'</span>) <span class="keyword">else</span> <span class="number">65535</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> hasattr(os, <span class="string">'fchown'</span>):</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">not</span> all(hasattr(os, attr)</span><br><span class="line">                   <span class="keyword">for</span> attr <span class="keyword">in</span> [<span class="string">'getuid'</span>, <span class="string">'getgid'</span>, <span class="string">'geteuid'</span>, <span class="string">'getegid'</span>]):</span><br><span class="line">            <span class="keyword">raise</span> SecurityError(<span class="string">'suspicious platform, contact support'</span>)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> <span class="keyword">not</span> uid <span class="keyword">or</span> <span class="keyword">not</span> gid <span class="keyword">or</span> <span class="keyword">not</span> euid <span class="keyword">or</span> <span class="keyword">not</span> egid:</span><br><span class="line">        <span class="keyword">if</span> (<span class="string">'pickle'</span> <span class="keyword">in</span> accept_content <span class="keyword">or</span></span><br><span class="line">                <span class="string">'application/x-python-serialize'</span> <span class="keyword">in</span> accept_content):</span><br><span class="line">            <span class="keyword">if</span> <span class="keyword">not</span> C_FORCE_ROOT:</span><br><span class="line">                <span class="keyword">try</span>:</span><br><span class="line">                    print(ROOT_DISALLOWED.format(</span><br><span class="line">                        uid=uid, euid=euid, gid=gid, egid=egid,</span><br><span class="line">                    ), file=sys.stderr)</span><br><span class="line">                <span class="keyword">finally</span>:</span><br><span class="line">                    os._exit(<span class="number">1</span>)</span><br><span class="line">        warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(</span><br><span class="line">            uid=uid, euid=euid, gid=gid, egid=egid,</span><br><span class="line">        )))</span><br></pre></td></tr></table></figure><p>很明显，这个 RuntimeWarning 是无法通过 C_FORCE_ROOT 设置来屏蔽的。当然了，这个只是一个警告，celery的运行并不受影响。</p><p>如果遇到某些版本的 celery 因为 root 用户而无法启动、无法执行的情况，可以通过改变 C_FORCE_ROOT=True 来规避这个问题。</p><p>主要有四种方式:</p><p>1.手动设置环境变量<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="builtin-name">export</span> <span class="attribute">C_FORCE_ROOT</span>=<span class="string">"true"</span></span><br></pre></td></tr></table></figure></p><p>2.使用 supervisor 管理 celery 时，可以用 environment 来设置环境变量.<br><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">environment</span>=PYTHONPATH=<span class="string">"xxx"</span>,C_FORCE_ROOT=<span class="string">"true"</span></span><br></pre></td></tr></table></figure></p><p>3.代码中硬编码指定.<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> celery <span class="keyword">import</span> platforms</span><br><span class="line"></span><br><span class="line">platforms.C_FORCE_ROOT=<span class="keyword">True</span></span><br></pre></td></tr></table></figure></p><p>4.写入 bashrc 或者其他系统启动脚本之中<br><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">file:.bashrc</span><br><span class="line"><span class="builtin-name">export</span> <span class="attribute">C_FORCE_ROOT</span>=<span class="string">"true"</span></span><br></pre></td></tr></table></figure></p>]]></content>
    
    <summary type="html">
    
      &lt;p&gt;当 celery 使用  root 用户执行时，会报出如下警告:&lt;br&gt;&lt;figure class=&quot;highlight ada&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;RuntimeWarning: You&lt;span class=&quot;symbol&quot;&gt;&#39;re&lt;/span&gt; running the worker &lt;span class=&quot;keyword&quot;&gt;with&lt;/span&gt; superuser privileges: this &lt;span class=&quot;keyword&quot;&gt;is&lt;/span&gt; absolutely &lt;span class=&quot;keyword&quot;&gt;not&lt;/span&gt; recommended!&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;低版本的 celery 情况不清楚，但是 celery 4.1.0 版本中，这个警告是无法去除的。因为源码中是这样写的:&lt;/p&gt;
    
    </summary>
    
      <category term="python" scheme="https://javasgl.github.io/categories/python/"/>
    
    
      <category term="python" scheme="https://javasgl.github.io/tags/python/"/>
    
      <category term="celery" scheme="https://javasgl.github.io/tags/celery/"/>
    
      <category term="supervisor" scheme="https://javasgl.github.io/tags/supervisor/"/>
    
  </entry>
  
</feed>