Python

批量注释和解注释:control+/

批量缩进: tab and tab+shift

Matlab

Tex studio

批量注释和解注释:command+T + command+U

Switching from one backend to another

If you have run Keras at least once, you will find the Keras configuration file at:

1

$HOME/.keras/keras.json

If it isn’t there, you can create it.

NOTE for Windows Users: Please change $HOME with %USERPROFILE%.

The default configuration file looks like this:

1
2
3
4
5
6

{
"image_data_format": "channels_last",
"epsilon": 1e-07,
"floatx": "float32",
"backend": "tensorflow"
}

Simply change the field backend to either “theano” or “tensorflow”, and Keras will use the new configuration next time you run any Keras code.

You can also define the environment variable KERAS_BACKEND and this will override what is defined in your config file :

1
2

KERAS_BACKEND=tensorflow python -c "from keras import backend"
Using TensorFlow backend.

virtual function

在某基类中声明为 virtual 并在一个或多个派生类中被重新定义的成员函数，用法格式为：virtual 函数返回类型 函数名（参数表） {函数体}；实现多态性，通过指向派生类的基类指针或引用，访问派生类中同名覆盖成员函数

multiple thread

deadlock and starvation

UDP

UDP is Connectionless Protocal

TCP

To establish a connection, TCP uses a three-way handshake. Before a client attempts to connect with a server, the server must first bind to and listen at a port to open it up for connections: this is called a passive open. Once the passive open is established, a client may initiate an active open. To establish a connection, the three-way (or 3-step) handshake occurs:

SYN: The active open is performed by the client sending a SYN to the server. The client sets the segment’s sequence number to a random value A.
SYN-ACK: In response, the server replies with a SYN-ACK. The acknowledgment number is set to one more than the received sequence number i.e. A+1, and the sequence number that the server chooses for the packet is another random number, B.
ACK: Finally, the client sends an ACK back to the server. The sequence number is set to the received acknowledgement value i.e. A+1, and the acknowledgement number is set to one more than the received sequence number i.e. B+1.
At this point, both the client and server have received an acknowledgment of the connection. The steps 1, 2 establish the connection parameter (sequence number) for one direction and it is acknowledged. The steps 2, 3 establish the connection parameter (sequence number) for the other direction and it is acknowledged. With these, a full-duplex communication is established.

some tricks

• What is the difference between using ++i and i++?

++i will increment the value of i, and then return the incremented value.

1
2
3

 i = 1;
 j = ++i;
(i is 2, j is 2)

i++ will increment the value of i, but return the original value that i held before being incremented.

1
2
3

i = 1;
j = i++;
(i is 2, j is 1)

Minimum Depth of Binary Tree

Given a binary tree, find its minimum depth.The minimum depth is the number of nodes along the shortest path from the root node down to the nearest leaf node.

java:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

/**
 * Definition for binary tree
 * public class TreeNode {
 *     int val;
 *     TreeNode left;
 *     TreeNode right;
 *     TreeNode(int x) { val = x; }
 * }
 */
public class Solution {
    public int run(TreeNode root) {
        
        if (root == null){
            return 0;
        } else if (root.left == null){
            return run(root.right) + 1;
        } else if (root.right == null){
            return run(root.left) + 1;
        } else {
            return Math.min(run(root.left), run(root.right))+1;
        }
    }
}

c++

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

class Solution {
public:
    int run(TreeNode *root) {
        
        if(root == nullptr) {
            return 0;
        } else if (root->left == nullptr ){
            
            return run(root ->right) + 1;
            
        } else if (root-> right == nullptr){
            return run(root ->left) +1;
        } else{
            return 1 + min(run(root->left),run(root->right));
        }
    }
};

Reverse Polish Notation

Evaluate the value of an arithmetic expression in Reverse Polish Notation.
Valid operators are+,-,*,/. Each operand may be an integer or another expression.
Some examples:

1
2

["2", "1", "+", "3", "*"] -> ((2 + 1) * 3) -> 9
["4", "13", "5", "/", "+"] -> (4 + (13 / 5)) -> 6

max-points-on-a-line

Given n points on a 2D plane, find the maximum number of points that lie on the same straight line.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

#include <vector>
#include <iostream>
 
 
using namespace std;
 
/**
 * Definition for a point.
 * struct Point {
 *     int x;
 *     int y;
 *     Point() : x(0), y(0) {}
 *     Point(int a, int b) : x(a), y(b) {}
 * };
 */
class Solution {
public:
     
    int maxPoints(vector<Point> &points) {
         
        int size = points.size();
         
        if( size == 0){
            return 0;
        } else if (size == 1){
            return 1;
        } else {
             
            int result = 0;
             
             
            for (int i = 0; i < size; ++i){
                int curmax = 1;
                int same = 0;
                int verticle = 1;
                 
                map < double, int > m;
                 
                for (int j = 0; j < size; ++j){
                     
                    if(i == j) continue;
                     
                    int dx = points[i].x - points[j].x;
                    int dy = points[i].y - points[j].y;
                     
                
                    if(dx == 0){
                       if (dy == 0) {same ++;}
                       else{
                           verticle ++;
                           curmax = max( curmax, verticle);
                       }
                        
                          
                    }else{
                        double k = (1.0*dy)/dx;
                        if(m[k] == 0){
                            m[k] = 2;
                        }else{
                            ++m[k];
                        }
                        curmax = max(m[k],curmax);
                         
                    }
                     
                }
                result = max(result, curmax + same);
                 
            }
            return result;
        }
                 
         
         
    }
};

sort-list

Sort a linked list in O(n log n) time using constant space complexity.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

/**
 * Definition for singly-linked list.
 * struct ListNode {
 *     int val;
 *     ListNode *next;
 *     ListNode(int x) : val(x), next(NULL) {}
 * };
 */
class Solution {
public:
    
    ListNode *merge(ListNode * left, ListNode * right){
        ListNode tmp(0);
        ListNode *p = &tmp;
        
        while (left && right){
            if (left -> val < right -> val){
                p -> next = left;
                left = left -> next;
            } else{
                p -> next = right;
                right = right -> next;
            }
            
            p = p ->next;
        }
        
        if(left){
            p -> next = left;
        }
        
        if(right){
            p -> next = right;
        }
        
        
        
        return tmp.next;
        
        
        
    }
    
    ListNode *sortList(ListNode *head) {
        if(!head || !head->next){
            return head;
        }
        
        ListNode *p = head, *q = head->next;
        
        while(q && q->next){
            p = p->next;
            q = q->next->next;
        }
        
        ListNode * right = sortList( p ->next);
        p ->next = NULL;
        ListNode * left = sortList(head);
        
        return merge(left, right);
        
    }
};

two-sum

Given an array of integers, find two numbers such that they add up to a specific target number.
The function twoSum should return indices of the two numbers such that they add up to the target, where index1 must be less than index2. Please note that your returned answers (both index1 and index2) are not zero-based.
You may assume that each input would have exactly one solution.
Input: numbers={2, 7, 11, 15}, target=9
Output: index1=1, index2=2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

class Solution {
public:
    vector<int> twoSum(vector<int> &numbers, int target) {
        vector <int> ans;
        int index1 = 0;
        int index2 = 0;
        for (int i = 0; i < numbers.size() - 1; ++i){
            int tmp = target - numbers[i];
            for(int j = i+1; j <numbers.size(); ++j ){
                if (tmp == numbers[j]){
                    ans.push_back(i+1);
                    ans.push_back(j+1);
                    
                    
                }
            }
        }
        
        return ans;
    }
};

using hash unordered_map

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

#include <unordered_map>
using namespace std;
class Solution {
public:
    vector<int> twoSum(vector<int> &numbers, int target) {
        
        unordered_map <int, int> hash;
        vector <int> res;
        
        for (int i = 0; i < numbers.size(); ++i){
            hash[numbers[i]] = i;
        }
        
        for (int i = 0; i < numbers.size(); ++i){
            int tmp = target - numbers[i];
            if(hash.find(tmp)!= hash.end() && hash[tmp] > i){
                res.push_back(i+1);
                res.push_back(hash[tmp] + 1);
                
            }
        }
        return res;
        /*
        vector <int> ans;
        int index1 = 0;
        int index2 = 0;
        for (int i = 0; i < numbers.size() - 1; ++i){
            int tmp = target - numbers[i];
            for(int j = i+1; j <numbers.size(); ++j ){
                if (tmp == numbers[j]){
                    ans.push_back(i+1);
                    ans.push_back(j+1);
                    
                    
                }
            }
        }
        
        return ans;*/
    }
};

balanced-binary-tree

Given a binary tree, determine if it is height-balanced.
For this problem, a height-balanced binary tree is defined as a binary tree in which the depth of the two subtrees of every node never differ by more than 1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

/**
 * Definition for binary tree
 * struct TreeNode {
 *     int val;
 *     TreeNode *left;
 *     TreeNode *right;
 *     TreeNode(int x) : val(x), left(NULL), right(NULL) {}
 * };
 */
class Solution {
public:
    
    int getDepth(TreeNode * root){
        if (root == NULL) return 0;
        if (root -> left == NULL) return 1 + getDepth(root ->right);
        if (root -> right == NULL) return 1 + getDepth(root ->left);
        return 1 + max(getDepth(root-> left), getDepth(root-> right));
        
    }
    bool isBalanced(TreeNode *root) {
        
        if (root == NULL) return true;
        
        
        int left = getDepth(root -> left);
        int right = getDepth(root -> right);
        
        if (abs(left - right) > 1 ) return false;
        
        return (isBalanced(root -> left) && isBalanced(root -> right));
        
    }
};

binary-preorder-traversal

Given a binary tree, return the preorder traversal of its nodes’ values.
For example:
Given binary tree{1,#,2,3},

1
2
3
4
5

1
 \
  2
 /
3

return[1,2,3].
Note: Recursive solution is trivial, could you do it iteratively?

###Recursive

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

/**
 * Definition for binary tree
 * struct TreeNode {
 *     int val;
 *     TreeNode *left;
 *     TreeNode *right;
 *     TreeNode(int x) : val(x), left(NULL), right(NULL) {}
 * };
 */
class Solution {
public:
    vector<int> preorderTraversal(TreeNode *root) {
        vector <int> result;
        
        if (root == NULL) return result;
        
        helper(root,result);
        
        return result;
    }
    
private:
    void helper(TreeNode * root, vector <int> &vec){
        if (root == NULL) return;
        vec.push_back(root -> val);
        helper(root -> left, vec);
        helper(root -> right, vec);
    }
};

iterative

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

/**
 * Definition for binary tree
 * struct TreeNode {
 *     int val;
 *     TreeNode *left;
 *     TreeNode *right;
 *     TreeNode(int x) : val(x), left(NULL), right(NULL) {}
 * };
 */
class Solution {
public:
    vector<int> preorderTraversal(TreeNode *root) {
        vector <int> result;
        stack <TreeNode *> st;
        
        if (root == NULL) return result;
        
        st.push(root);
        
        while(!st.empty()){
            TreeNode * current = st.top();
            st.pop();
            
            if (current -> right != NULL) st.push(current->right);
            if (current -> left != NULL) st.push(current->left);
            result.push_back(current -> val);
            
            
        }
        
        
        
        
        //helper(root,result);
        
        return result;
    }
    
private:
    
    
    
    void helper(TreeNode * root, vector <int> &vec){
        if (root == NULL) return;
        vec.push_back(root -> val);
        helper(root -> left, vec);
        helper(root -> right, vec);
    }
};

string to integer atoi

Implement atoi to convert a string to an integer.
Hint: Carefully consider all possible input cases. If you want a challenge, please do not see below and ask yourself what are the possible input cases.
Notes: It is intended for this problem to be specified vaguely (ie, no given input specs). You are responsible to gather all the input requirements up front.
spoilers alert… click to show requirements for atoi.
Requirements for atoi:
The function first discards as many whitespace characters as necessary until the first non-whitespace character is found. Then, starting from this character, takes an optional initial plus or minus sign followed by as many numerical digits as possible, and interprets them as a numerical value.
The string can contain additional characters after those that form the integral number, which are ignored and have no effect on the behavior of this function.
If the first sequence of non-whitespace characters in str is not a valid integral number, or if no such sequence exists because either str is empty or it contains only whitespace characters, no conversion is performed.
If no valid conversion could be performed, a zero value is returned. If the correct value is out of the range of representable values, INT_MAX (2147483647) or INT_MIN (-2147483648) is returned.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

class Solution {
public:
    int atoi(const char *str) {
        long long res = 0;
        if (str == NULL){
            return res;
        }
        
        int len = strlen(str);
        int i = 0;
        bool neg = false;
        
        while (str[i] == ' '|| str[i] == '0'){
            ++i;
        }
        
        if (str[i] == '-'){
            
            neg = true;
            ++i;
        } else if (str[i] == '+'){
            neg = false;
            ++i;
        }
        
        for (;i < len;++i){
            char ch = str[i];
            if (ch >='0' && ch <='9'){
                res = res*10 + ch - '0';
                
                if (res > INT_MAX){
                    return neg? INT_MIN:INT_MAX;
                }
                
            }else{
                if (neg) return -res;
        		return res;   
            }
            
        }
        
        if (neg) return -res;
        return res;
        
    }
};

merge two sorted lists

Merge two sorted linked lists and return it as a new list. The new list should be made by splicing together the nodes of the first two lists.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

/**
 * Definition for singly-linked list.
 * struct ListNode {
 *     int val;
 *     ListNode *next;
 *     ListNode(int x) : val(x), next(NULL) {}
 * };
 */
class Solution {
public:
    ListNode *mergeTwoLists(ListNode *l1, ListNode *l2) {
        if (l1 == NULL) return l2;
        if (l2 == NULL) return l1;
        
        ListNode * dummy = new ListNode(0);
        ListNode * p = dummy;
        
        while(l1!=NULL && l2!=NULL){
            if (l1->val < l2->val){
                p->next = l1;
                l1 = l1->next;
            }else {
                p -> next = l2;
                l2 = l2 -> next;
            }
            
            p = p ->next;
        }
        
        if(l1!= NULL){
            p -> next = l1;
        }
        if(l2!= NULL){
            p -> next = l2;
        }
        
        return dummy -> next;
    }
};

##

1
2

Hexo is a fast, simple and powerful blog framework. And we can use Markdown to write it, so it is very efficient. I migrate the old blogs to this theme.

Install bazel

Install java jdk

Required Java version:

• Java JDK 8 or later (JDK 7 is still supported but deprecated).

In centos we can simply install this by command:

1

yum install -y java-1.8.0-openjdk

set JAVA_HOME environment variables in .bashrc:

1

export JAVA_HOME=/usr/lib/jvm/java-openjdk

Download bazel source code

Download from github and compile:

1
2
3

git clone https://github.com/bazelbuild/bazel.git
cd bazel
./compile.sh

Set bazel to PATH

ADD bazel-bin/src/bazel to .bashrc

In my server as:

1

export PATH=$PATH:/root/bazel/output

Now, we have install the bazel in centos.

If you use Ubuntu OS, you can install it form apt repository. In this blog, I will not tell the detailed, which you can reference the link:

1

http://bazel.io/docs/install.html#compiling-from-source

Install Tensorflow

Download source code

Clone the tensorflow from github:

1

git clone https://github.com/tensorflow/tensorflow

Configure for installation

1
2

cd tensorflow/
./configure

Then, it will tell you which environment to set. such as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

Please specify the location of python. [Default is /usr/local/anaconda2/bin/python]: 
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] n
No Google Cloud Platform support will be enabled for TensorFlow
Found possible Python library paths:
  /usr/local/anaconda2/lib/python2.7/site-packages
Please input the desired Python library path to use.  Default is [/usr/local/anaconda2/lib/python2.7/site-packages]
/usr/local/anaconda2/lib/python2.7/site-packages
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: 
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 
Please specify the location where CUDA  toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 
Please specify the location where cuDNN  library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 
Setting up Cuda include
Setting up Cuda lib64
Setting up Cuda bin
Setting up Cuda nvvm
Setting up CUPTI include
Setting up CUPTI lib64
Configuration finished

Create the pip package and install

When building from source, you will still build a pip package and install that.

1
2
3
4

# To build with GPU support:
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-0.10.0rc0-py2-none-any.whl

Setting up TensorFlow for Development

If you’re working on TensorFlow itself, it is useful to be able to test your changes in an interactive python shell without having to reinstall TensorFlow.

To set up TensorFlow such that all files are linked (instead of copied) from the system directories, run the following commands inside the TensorFlow root directory:

1
2
3
4
5
6
7
8
9

bazel build -c opt //tensorflow/tools/pip_package:build_pip_package
# To build with GPU support:
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
mkdir _python_build
cd _python_build
ln -s ../bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/org_tensorflow/* .
ln -s ../tensorflow/tools/pip_package/* .

Such as change the version from 0.10.0 to 0.11.0 filetensorflow/tools/pip_package/setup.py:

1

_VERSION = '0.11.0rc0'

Then, do the following:

1

python setup.py develop

Now, we can build our develop tensorflow:

1
2

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-0.11.0rc0-py2-none-any.whl

Verify the installation

1
2
3
4
5
6
7
8
9
10
11
12
13

[root@node-4 site-packages]# pip show tensorflow
Metadata-Version: 2.0
Name: tensorflow
Version: 0.11.0rc0
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
Installer: pip
License: Apache 2.0
Location: /usr/local/anaconda2/lib/python2.7/site-packages
Requires: numpy, mock, protobuf, wheel, six

We can find the version changed from the 0.10.0tc0 to 0.11.0rc0.

Reference

official link:this.

Introduction

TensorFlow™ is an open source software library for numerical computation using data flow graphs.

TensorFlow is for everyone. It’s for students, researchers, hobbyists, hackers, engineers, developers, inventors and innovators and is being open sourced under the Apache 2.0 open source license.

Distributed version

In April 13,2006.Distributed version of Tensorflow was published.

Announcing TensorFlow 0.8 – now with distributed computing support!

This blog is also reposted by Jeff Dean in G+ who is the Google Senior Fellow in the Systems Infrastructure Group.

Installation

I try the distributed version in 3 virtual machines on OpenStack platform which is a famous open source cloud OS.

Environment

• 1 master node
• 2 worker nodes
• Configuration:4GB RAM, 2 VCPU, 40.0GB Disk
• Operating System:ubuntu-14.04-server-cloudimg-amd64
• Python version:2.7
• Tensorflow version:r0.8
• vCPU only.

Install pip

1. Download get-pip.py file.

1

wget https://bootstrap.pypa.io/get-pip.py

1. Install the pip behind a proxy

1

python get-pip.py --proxy="[user:passwd@]proxy.server:port"

1. Upgrading pip

1

pip install -U pip

Install Tensorflow by pip

1. Ubuntu/Linux 64-bit, GPU enabled. Requires CUDA toolkit 7.5 and CuDNN v4.

1

sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl

1. Ubuntu/Linux 64-bit, CPU only:

1

sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl

Demo of Distributed TensorFlow

Now we have setup the preparation for a demo. This demo is according to the tutorial of Tensorflow.

https://www.tensorflow.org/versions/r0.8/how_tos/distributed/index.html

Describe of a cluster

A TensorFlow “cluster” is a set of “tasks” that participate in the distributed execution of a TensorFlow graph. Each task is associated with a TensorFlow “server”, which contains a “master” that can be used to create sessions, and a “worker” that executes operations in the graph. A cluster can also be divided into one or more “jobs”, where each job contains one or more tasks.

Two important instrucions

Create a tf.train.ClusterSpec that describes all of the tasks in the cluster. This should be the same for each task.

Create a tf.train.Server, passing the tf.train.ClusterSpec to the constructor, and identifying the local task with a job name and task index.

Specific a cluster

We define 2 worker node and 1 parameter server node as the following:

1
2
3

cluster = tf.train.ClusterSpec({"worker": ["tensorflow-worker0:2222",
                                           "tensorflow-worker1:2222"],
                                "ps": ["tensorflow-master0:2222"]})

master node

1. Write ps.py as the following for parameter server:

1
2
3
4
5
6
7
8
9

import tensorflow as tf
cluster = tf.train.ClusterSpec({"worker": ["tensorflow-worker0:2222",
                                           "tensorflow-worker1:2222"],
                                "ps": ["tensorflow-master0:2222"]})
server = tf.train.Server(cluster,
                          job_name="ps",
                          task_index=0)
server.join()

1. Run the script

1
2
3
4

# python ps.py 
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job ps -> {localhost:2222}
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job worker -> {tensorflow-worker0:2222, tensorflow-worker1:2222}
I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:202] Started server with target: grpc://localhost:2222

It will listen on the 2222 port of master0 server.

worker node

In the two worker nodes, do the following steps:

1. Write worker0.py as the following:

1
2
3
4
5
6
7
8
9

import tensorflow as tf
cluster = tf.train.ClusterSpec({"worker": ["tensorflow-worker0:2222",
                                           "tensorflow-worker1:2222"],
                                "ps": ["tensorflow-master0:2222"]})
server = tf.train.Server(cluster,
                          job_name="worker",
                          task_index=0)
server.join()

1. Run the script

1
2
3
4

# python worker0.py 
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job ps -> {tensorflow-master0:2222}
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job worker -> {localhost:2222, tensorflow-worker1:2222}
I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:202] Started server with target: grpc://localhost:2222

It will listen on the 2222 port of worker0 server.

1. Write worker1.py as the following:

1
2
3
4
5
6
7
8
9

import tensorflow as tf
cluster = tf.train.ClusterSpec({"worker": ["tensorflow-worker0:2222",
                                           "tensorflow-worker1:2222"],
                                "ps": ["tensorflow-master0:2222"]})
server = tf.train.Server(cluster,
                          job_name="worker",
                          task_index=1)
server.join()

1. Run the script

1
2
3
4

# python worker1.py 
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job ps -> {tensorflow-master0:2222}
I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job worker -> {tensorflow-worker0:2222, localhost:2222}
I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:202] Started server with target: grpc://localhost:2222

It will listen on the 2222 port of worker1 server.

Test Job Task

1. The test job script is the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

import tensorflow as tf
with tf.device("/job:ps/task:0"):
    weights0 = tf.Variable(tf.random_normal(shape=[1024, 512]))
    bias0 = tf.Variable(tf.zeros(shape=[512]))
with tf.device("/job:worker/task:1"):
    inputs = tf.random_normal(shape=[10, 1024])
    l0 = tf.nn.relu(tf.matmul(inputs, weights0) + bias0)
    l1 = tf.nn.relu(tf.matmul(l0, tf.transpose(weights0)))
    loss = tf.nn.l2_loss(l1-inputs)
    train_op = tf.train.AdamOptimizer().minimize(loss)
with tf.Session("grpc://tensorflow-master0:2222") as sess:
   for _ in range(1000):
       sess.run(tf.initialize_all_variables())
       _, l = sess.run([train_op, loss])
       print l

• ps node is to generate normal random variables.
• worker node is to training the model.

1. Results

When run the script, it will generate data in device: "/job:ps/task:0" and train the data in device=/job:worker/task:1 as the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# python job.py 
name: "Adam"
op: "NoOp"
input: "^Adam/update_Variable/ApplyAdam"
input: "^Adam/update_Variable_1/ApplyAdam"
input: "^Adam/Assign"
input: "^Adam/Assign_1"
device: "/job:ps/task:0"
 Tensor("L2Loss:0", shape=(), dtype=float32, device=/job:worker/task:1)
name: "Adam"
op: "NoOp"
input: "^Adam/update_Variable/ApplyAdam"
input: "^Adam/update_Variable_1/ApplyAdam"
input: "^Adam/Assign"
input: "^Adam/Assign_1"
device: "/job:ps/task:0"
 Tensor("L2Loss:0", shape=(), dtype=float32, device=/job:worker/task:1)
name: "Adam"
op: "NoOp"
input: "^Adam/update_Variable/ApplyAdam"
input: "^Adam/update_Variable_1/ApplyAdam"
input: "^Adam/Assign"
input: "^Adam/Assign_1"
device: "/job:ps/task:0"
 Tensor("L2Loss:0", shape=(), dtype=float32, device=/job:worker/task:1)
name: "Adam"
op: "NoOp"
input: "^Adam/update_Variable/ApplyAdam"
input: "^Adam/update_Variable_1/ApplyAdam"
input: "^Adam/Assign"
input: "^Adam/Assign_1"
device: "/job:ps/task:0"
...

Further Work

This blog is only a try of distributed tensorflow, there are many works need to do. such as how to combined working with GPU, how to benchmark and so on.

Reference

https://www.tensorflow.org

https://github.com/tensorflow/tensorflow

http://googleresearch.blogspot.com/2016/04/announcing-tensorflow-08-now-with.html

Playing Tricircle with Virtualbox

1 Preparation

1 server with Linux kernel (demo is Ubuntu 14.04 LTS)

2 Install softwares

2.1 Install virtualbox

Follow the steps in virtualbox downloads

First, Add the following line to your /etc/apt/sources.list:

1

deb http://download.virtualbox.org/virtualbox/debian trusty contrib

According to your distribution, replace ‘vivid’ by ‘utopic’, ‘trusty’, ‘raring’, ‘quantal’, ‘precise’, ‘lucid’, ‘jessie’, ‘wheezy’, or ‘squeeze’.

Then, add The Oracle public key for apt-secure:

1

wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -

Next, install with apt-get method:

1
2

sudo apt-get update
sudo apt-get install virtualbox-5.0

2.2 Connect to the virtualbox with x11

First, copy your public key to the ~/authorized_keys

Then, connect with -X command

1

ssh -X root@HostIP

Next, input the virtualbox to start install virtual machine

1

virtualbox

Then you can see the virtualbox graph interface:

3 Install Virtual Machines

For playing Tricircle, we need to install 3 nodes for devstack.One for the Top OpenStack, Two for cross pod bottom OpenStacks.

3.1 Configuration of VMs

The most important is to set up networks for use

In order to make the VMs with multiple VLAN networks, then add 2 network devices for bridge use.

eth0

The eth0 is the default network with NAT methods.

eth1

The eth1 is the VLAN external network,and using bridge method, in my environment, I attached to the eth1.

[attention] The Promiscuous Mode must to set “Allow All”.And be in use after reboot.

Otherwise, The Ping test with VLAN tag from Node1 to Node2 will be blocked.

eth2

The same setting as the above, the Promiscuous Mode must be set to “Allow All”. And be in use after reboot.

3.2 Installation the VMs

Download a ios for installation

There are many mirror sites in the world. I download a ubuntu-14.04-LTS in the Ali-OSM(Alibaba Open Source Mirror Site). Because it’s very fast in China.

Install the Operating System

Follow the steps while installing, because it’s easy, so it will be ignored.
After installation, you will see the console like this:

3.3 Running in the backgroud

After installation of the VMs, we can login from SSH. So we need to let’s these VMs running in the background.
And so when we close the VMs, we need to choose the option:

4 Playing tricircle with devstack

The detailed methods can be seen in the OpenStack/Tricircle.

4.1 In Top OpenStack

Configure the network

Install the openvswitch for creating bridges.

1

apt-get install openvswitch-switch

Create the stack user

1
2
3
4

adduser stack
``` 
#### Give the stack user sudo privileges:

apt-get install sudo -y
echo “stack ALL=(ALL) NOPASSWD: ALL” >> /etc/sudoers

1
2

#### Download the devstack

sudo apt-get install git -y
git clone https://git.openstack.org/openstack-dev/devstack
cd devstack

1
2
3
4
5

#### Configure the local.conf
Change the local.conf to fit your environment.
The example is in the Tricircle Project. such as [local.conf.sample](https://github.com/openstack/tricircle/blob/master/devstack/local.conf.sample).
#### Install the devstack with Tricircle project

./stack.sh

1
2
3
4
5
6

After installing the devstack with tricircle, the next step is verifying the installation.
#### Verifying the installation
Before verifying, It should create the client environment variables to import, such as :
admin-openrc.sh

export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password #change password as you set in your own environment
export OS_AUTH_URL=http://127.0.0.1:5000
export OS_IDENTITY_API_VERSION=3

#It’s very important to set region name to the top openstack, because tricircle has different API urls.
export OS_REGION_NAME=RegionOne

1
2
3

Modify the verify install scripts as your own environment.
And run:

cd tricircle/devstack
chmod +x verify_top_install.sh
./verify_top_install.sh 2>&1 | tee logs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

It will save the outputs in the logs, you can check if it's installed correct.
I pasted one copy in my environments as [this](https://docs.google.com/document/d/1NQvRJo5bnnvYMUCLar-rRRm65KHOMmNpZasQlZCBgXs/edit?usp=sharing).
### 4.2 Cross pods OpenStack
#### Installing
First, in the node1 install the tricircle, and then in the node2 install the tricircle.As the above:
<li> Modify the networks;</li>
<li> Create the stack user;</li>
<li> Install git;</li>
<li> Download the devstack;</li>
<li> Modify the local.conf;</li>
<li> Install the devstack with tricircle;</li>
<br>
#### Verifying the tricircle
Before verifying, It should create the client environment variables to import, such as :admin-openrc.sh

export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password #change password as you set in your own environment
export OS_AUTH_URL=http://127.0.0.1:5000
export OS_IDENTITY_API_VERSION=3

#It’s very important to set region name to the top openstack, because tricircle has different API urls.
export OS_REGION_NAME=RegionOne

1
2
3

Modify the verify install scripts as your own environment.
And run:

cd tricircle/devstack
chmod +x verify_top_install.sh
./verify_cross_pod_install.sh 2>&1 | tee logs

1
2
3
4
5
6
7
8
9

One copy logs like [this](https://docs.google.com/document/d/1srGuk2XY8VIbmhGKEVBVbQ_MW5sUByGBDZcJVVS_2Cg/edit?usp=sharing).
#### Ping test
Using the VNC to login the instances in Node1 and Node2.
And Ping with each other.
VM1: IP 10.0.1.3/24

ping -c 4 10.0.2.3

1
2
3
4

![vm1](http://img.blog.csdn.net/20160308143200217)
VM2: IP 10.0.2.3/24

ping -c 4 10.0.1.3
```

So the cross pod networking has been verified.

SODA比赛(Shanghai Open Data Apps)一共有超过400个参赛选手，对于比赛大数据的处理，需要强力的技术支持，我们为进入复赛的参赛队伍免费提供了两个数据平台，并分别导入大赛数据，供选手使用。它们分别是：

• Spark
• Vertica

本文介绍 Gitlab 环境的部署过程、使用方法以及如何通过后台自动调用Spark集群进行数据计算。

SODA瓶之Spark环境请戳

• SODA瓶之Ganglia监控
• SODA瓶之Vertica on OpenStack

Gitlab介绍

Gitlab

GitLab，是一个利用 Ruby on Rails 开发的开源应用程序，实现一个自托管的Git项目仓库，可通过Web界面进行访问公开的或者私人项目。

它拥有与GitHub类似的功能，能够浏览源代码，管理缺陷和注释。可以管理团队对仓库的访问，它非常易于浏览提交过的版本并提供一个文件历史库。团队成员可以利用内置的简单聊天程序（Wall）进行交流。它还提供一个代码片段收集功能可以轻松实现代码复用，便于日后有需要的时候进行查找。

比赛代码托管平台搭建

本次比赛的所有代码考虑到隐私性、安全性等因素，采用了完全由OMNILab搭建的Gitlab私有代码仓库。参赛选手可以将编写好的代码上传至比赛专用的Gitlab，然后可以通过Gitlab调用后台为比赛专门准备的Spark计算集群，进行数据计算。

具体代码仓库搭建过程如下：

gitlab预装环境搭建

安装包括ssh、email等服务

1
2

sudo apt-get install curl openssh-server ca-certificates postfix
gitlab仓库搭建以及配置

增加gitlab的package并且安装：

1
2

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
sudo apt-get install gitlab-ce

配置并启动gitlab服务

1
2

curl -LJO https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/trusty/gitlab-ce-XXX.deb/download
dpkg -i gitlab-ce-XXX.deb

批量创建用户名、密码

首先，通过正则表达式匹配，将KESCI提供的邮箱中的用户名进行提取：

1

cat list.txt | sed 's/\(.*\)@\(.*\)/\1/g' >>user.list

分别对应生成密码，这里采用了linux下的makepassword工具包批量生成密码

1

makepasswd --count=404

采用gitlab的命令行工具进行批量的用户创建

gitlab 提供了一个可以通过private token的方式进行命令行创建用户等操作，这样省去了手动创建的操作。

我们通过以下两个脚本进行用户的批量创建：

• 创建用户脚本

1
2
3
4
5
6
7
8

#!/bin/bash
#MAIL=$1
USER=$1
PASS=$2
curl -H "Content-Type:application/json" http://IP/api/v3/users?private_token=XXXXXXX -d \
"{ \"email\":\"$1@IP\",\"password\":\"$2\",\"name\":\"$1\",\"username\":\"$1\",\"confirm\":\"false\" }"

• 批量创建脚本

1
2
3
4
5
6
7
8

#!/bin/bash
cat final.txt | while read name password
do
  echo $name
  echo $password
  ./create-git-account.sh $name $password
done

Spark介绍

Spark

Apache Spark是一个开源簇运算框架，最初是由加州大学柏克莱分校AMPLab所开发。相对于Hadoop的MapReduce会在运行完工作后将中介数据存放到磁盘中，Spark使用了存储器内运算技术，能在数据尚未写入硬盘时即在存储器内分析运算。Spark在存储器内运行程序的运算速度能做到比Hadoop MapReduce的运算速度快上100倍，即便是运行程序于硬盘时，Spark也能快上10倍速度。

Spark集群介绍

本次比赛中OMNILab为参赛选手提供了强大的Spark集群：

• 2个控制节点，8个计算节点，1个客户端节点
• 共计256核和2TB物理内存可用于计算

使用方法

• 上传代码到git,参照模版来提交你的代码，代码准备完毕后创建release branch
  后台调度器会自动调度代码到Spark上运行
• 从release branch上拿到结果
• 搞定！