Backend on Time Friend

cheeriojs Chinese Parsed to Unicode Issue When Loading HTML

Thu, 29 Nov 2018 20:32:05 +0000

cheeriojs Parsing Chinese to Unicode Issue

When using cheeriojs, I found that whenever using the html method, Chinese will be parsed to unicode by default.

Usage is as follows:

var cheerio = require('cheerio');
var $ = cheerio.load('<title>我是中文,我将会被解析成unicode</title>');
console.log($('title').html());

When using the text method, the above problem doesn’t occur

var cheerio = require('cheerio');
var $ = cheerio.load('<title>我是中文,我将会被解析成unicode</title>');
$('title').text()

Solution

Default Configuration

When we load html content, cheerio actually has default configuration. HTML parsing uses the htmlparser2 library, so htmlparser2 configuration is also applicable in cheerio.

Build Simpler Private npm Server with verdaccio

Tue, 10 Apr 2018 22:30:05 +0000

Building private npm servers like cnpm.org is already quite simple, but compared to verdaccio’s simplicity, there’s no comparison. Because verdaccio is too simple.

Introduction

verdaccio is a lightweight private NPM Registry (forked from Sinopia, Sinopia’s last update was years ago). Initially planned to use cnpmjs to build private npm repository but after building there were some problems, so used Sinopia;

Next I’ll briefly introduce verdaccio usage.

Install verdaccio on Your Server

$ npm i verdaccio -g

Start

# Directly enter verdaccio command
$ verdaccio

// If startup successful, will display the following information
 Verdaccio doesn't need superuser privileges. Don't run it under
 warn --- config file - /root/.config/verdaccio/config.yaml
 warn --- http address - http://localhost:4873/ - verdaccio/

We can also start with pm2 (also the method I chose to use)

Multi-Server Automated Deployment Solution Based on shipit-deploy

Sun, 17 Dec 2017 20:32:05 +0000

Why automate deployment, I won’t elaborate here.

Automated deployment based on shipit-deploy can achieve the following effects

One-click deployment to multiple servers.
One-click rollback of multiple servers.
Local operation, no need to log into servers.
Easy to customize, easy to extend, achieve automation.

Usage

1. Download and Install

npm install --save-dev shipit-cli
npm install --save-dev shipit-deploy

2. Establish Trust Relationship with Servers

ssh-copy-id USER@REMOTE_HOST

3. Configuration

module.exports = function (shipit) {
 require('shipit-deploy')(shipit);
 require('shipit-pm2')(shipit);
 require('shipit-cnpmjs')(shipit);

 shipit.initConfig({
 default: {
 workspace: '/tmp/github-monitor',
 deployTo: '/tmp/deploy_to', // Server target path
 repositoryUrl: 'https://github.com/user/repo.git', // Git repository address
 ignores: ['.git', 'node_modules'], // Excluded files
 keepReleases: 2, // Number of release versions to keep
 deleteOnRollback: false,
 key: '/path/to/key',
 shallowClone: true,
 cnpm: {
 remote: false
 }
 },
 dev: { // Development server deployment
 servers: ['user@devServer1', 'user@devServer1'],
	 branch: 'dev' // Git branch to publish,
 pm2: {
 json: 'pm2-dev-app.json' // Development environment pm2 startup configuration
 }
 },
 prod: { // Production server deployment
 servers: ['user@prodServer1', 'user@prodServer2'],
	 branch: 'master' // Git branch to publish,
 pm2: {
 json: 'pm2-prod-app.json' // Production environment pm2 startup configuration
 }
 }
 });
};

pm2-prod-app.json example:

Enable Gzip Mode for Express, Egg.js, Koa.js

Wed, 13 Dec 2017 20:32:05 +0000

To reduce download size of APIs and static files, we can enable Gzip when server resources are sufficient.

Express

Express version below 4.0

var express = require('express');
var app = express();
app.use(express.compress()); //Mainly this line

Express version 4.0 and above (including 4.0)

var compression = require('compression');
var express = require('express');
var app = express();
app.use(compression());

Egg.js

// app/middleware/compress.js
// koa-compress exposed interface (`(options) => middleware`) matches framework's middleware requirements
module.exports = require('koa-compress')

// config/config.default.js
module.exports = {
 middleware: [ 'compress' ],
 compress: {
 threshold: 2048,
 },
};

Koa.js

const koa = require('koa');
const compress = require('koa-compress');

const app = koa();

const options = { threshold: 2048 };
app.use(compress(options));

Apache, Nginx, Express, Egg.js Support Frontend HTML5 History Mode

Sat, 02 Dec 2017 22:30:05 +0000

When you use history mode, URLs look like normal URLs, e.g. http://xxx.com/user/id.

To perfectly support this mode, backend configuration support is also needed. When the application is a single-page client application, if the backend is not correctly configured, when users directly access http://xxx.com/user/id in the browser, it will return 404.

Backend support examples:

Apache

<IfModule mod_rewrite.c>
 RewriteEngine On
 RewriteBase /
 RewriteRule ^index\.html$ - [L]
 RewriteCond %{REQUEST_FILENAME} !-f
 RewriteCond %{REQUEST_FILENAME} !-d
 RewriteRule . /index.html [L]
</IfModule>

nginx

location / {
 try_files $uri $uri/ /index.html;
}

Node.js (Express)

If using Express, you can use connect-history-api-fallback https://github.com/bripkens/connect-history-api-fallback

How Egg.js Supports Single Page Application

Fri, 01 Dec 2017 22:30:05 +0000

Enable ejs in config/plugin.js // Other template engines also work

exports.ejs = {
 enable: true,
 package: 'egg-view-ejs',
};

Configure static directory

//config/config.{env}.js

 config.static = {
 prefix: '/',
 dir: path.join(appInfo.baseDir, 'app/view/')
 }

Template configuration

//config/config.{env}.js

 config.view = {
 defaultExt: '.html',
 mapping: {
 '.ejs': 'ejs',
 '.html': 'ejs',
 }
 }

Configure root directory route mapping

// app/router.js

module.exports = app => {
 app.router.get('/', app.controller.home.index);
}

Configure route corresponding controller

// app/controller/home.js

const { Controller } = require('egg');
class HomeController extends Controller {
 async index() {
 const { ctx } = this;
 // render index.html
 await ctx.render('index');
 }
}
module.exports = HomeController;

After completing the above configuration, you can combine with single page application

Publish Company Private Code to Self-Hosted cnpm

Sat, 25 Nov 2017 22:30:05 +0000

Configuration

First need to add some configuration properties to the original config/config.js:

 enablePrivate: false, // Anyone can publish packages
 admins: {
 admin: 'test@company.com' // Admin permissions
 },
 scopes: ['@company'], // Private packages must be under scope

Restart cnpm

Enter cnpm directory

npm stop // Stop service
npm start // Start service

Add code to package.json file:

 "name": "@company/testjs", // Package name, must add scope name before

 npm login --registry=http://192.168.0.100:7001 // Register previous user
 Username: admin // Admin name
 Password: 1234 // Password you want

 npm publish --registry=http://192.168.80.130:7001

Publish successful

Build Private cnpm Server in Company

Fri, 24 Nov 2017 22:30:05 +0000

There are more and more js projects now. If we open source our code, we’ll publish these codes to npm. But npm is public, company code after all won’t be publicly available online. So we need to build our own npm server. Publish company private code on the company’s npm server. This way we can conveniently download the code we want, and also not put this code on open source platforms

Visual Studio Code Hide Extra js and map Files Generated from ts

Wed, 22 Nov 2017 20:32:05 +0000

Open 【File】>【Preferences】>【Workspace Settings】, put in the following code:

// Put settings in this file to override defaults and user settings.
{
 "files.exclude": {
 // exclude .js and .js.map files, when in a TypeScript project
 "node_modules": true,
 "**/*.js": { "when": "$(basename).ts"},
 "**/*.js.map": true
 }
}

Debug Node.js with Chrome DevTools

Tue, 08 Aug 2017 00:04:05 +0000

Prerequisites

Node.js 6.3+;
Chrome 55+;

Configure Chrome (Latest versions no longer need this step);

Enter URL: chrome://flags/#enable-devtools-experiments to enter Developer Lab
Enable highlighted option
Restart Chrome
Open DevTools Setting -> Experiments;
Press Shift 6 times consecutively to show hidden options
Find Node debugging and check it. (New versions no longer have this option, it’s enabled by default. So Chrome doesn’t need configuration)

Run Node.js

Just add –inspect to the command, followed by the file you want to execute;

Publish Aliyun API Signature Generation Tool Based on Node.js -- AliToSing

Sun, 16 Jul 2017 00:04:05 +0000

Aliyun API signature generation tool based on Node.js

Github AliToSign

Aliyun Signature Official Documentation

Installation method:

npm install alitosign --save

Usage method:

Example: Add Domain

const alitosign = require("alitosign");
const querystring = require("querystring");
const moment = require("moment");
const http = require("http");

// Basically common parameters needed for calling interfaces
let originParams = {
 AccessKeyId: "xxxxxxx", // AccessKeyId acquisition method please refer to official documentation
 Format: "JSON",
 Version: "2015-01-09",
 SignatureMethod: "HMAC-SHA1",
 SignatureVersion: "1.0",
};

function AddDomain() {
 // Copy common parameters
 let params = Object.assign({}, originParams);

 // Add necessary parameters for the interface you want to call on the new object,
 // Each interface is different, please check official documentation for details
 params.Action = "AddDomain";
 params.DomainName = "time-friend.com";
 params.GroupId = "2223";

 // Add timestamp
 params.Timestamp = moment.utc().format(); // Use Greenwich Mean Time;
 params.SignatureNonce = new Date().getTime();

 // Generate signature
 let signParams = alitosign(params);

 // Convert object containing signature property to query format string
 let queryParams = querystring.stringify(signParams);

 // Concatenate url
 let url = `http://alidns.aliyuncs.com/?` + queryParams;

 // Call interface
 http.get(url, (res) => {
 res.on("data", function (data) {
 // After serialization, you can get the data you want
 let oData = JSON.parse(data);
 });
 });
}

npm, yarn Package Manager Switch to Taobao Registry

Sat, 01 Jul 2017 20:32:05 +0000

Due to some special reasons, package management software like npm, Yarn always downloads a bit slow in China. Can switch to Taobao registry, directly pull from domestic servers. Here’s a note:

NPM

npm config set registry https://registry.npm.taobao.org --global
npm config set disturl https://npm.taobao.org/dist --global

If the above configuration doesn’t work happily, can try the following method:

npm config set registry http://registry.cnpmjs.org
npm info underscore 
//(If above configuration is correct, this command will have string response)

yarn

yarn config set registry https://registry.npm.taobao.org --global
yarn config set disturl https://npm.taobao.org/dist --global

If EACCES: permission denied permission error appears, Fix /usr/local directory ownership:

Express Set CORS, Let Browser Cross-Origin Access Your Server

Sat, 18 Feb 2017 21:32:05 +0000

Speaking of frontend cross-origin, it really makes many frontend developers sweat.

Cross-origin solutions are like eight immortals crossing the sea. Various black tech, various black magic. Makes people amazed. (If you don’t understand, you can search)

But none are perfect solutions, all can do this but not that.

Today I’ll talk about the ultimate cross-origin solution (IE8 and lower versions don’t support)

var express = require('express'); 
var app = express(); 
//Set CORS
app.all('*', function(req, res, next) { 
 res.header("Access-Control-Allow-Origin", "*"); 
 //Second parameter is an asterisk, means pages from any domain can request this server;
 //Set specified domain:
 //res.header("Access-Control-Allow-Origin", "http://baidu.com");
 //This way, web pages under baidu.com can ajax request your server


 res.header("Access-Control-Allow-Headers", "X-Requested-With"); 


 res.header("Access-Control-Allow-Methods","PUT,POST,GET,DELETE,OPTIONS"); 
 //Second parameter is which HTTP request methods the other party can use to request your server, set according to your situation


 res.header("X-Powered-By",' 3.2.1') 
 res.header("Content-Type", "application/json;charset=utf-8"); 
 next(); 
}); 
 
app.get('/', function(req, res) { 
 res.send("You have successfully accessed this server"); 
}); 
 
app.listen(3000);

Make Node.js Fetch Like Browser

Fri, 17 Feb 2017 20:32:05 +0000

Previous article briefly introduced Node.js http module’s get and post methods, but because post configuration is still quite cumbersome, today recommend a module:

node-fetch

Installation

npm install node-fetch --save

Usage

var fetch = require('node-fetch');

// HTML 
fetch('https://github.com/')
 .then(function(res) {
 return res.text();
 }).then(function(body) {
 console.log(body);
 });
 


// JSON 
fetch('https://api.github.com/users/github')
 .then(function(res) {
 return res.json();
 }).then(function(json) {
 console.log(json);
 });
 


// When network error occurs
fetch('http://domain.invalid/')
 .catch(function(err) {
 console.log(err);
 });
 


// stream
var fs = require("fs")
fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
 .then(function(res) {
 var dest = fs.createWriteStream('./octocat.png');
 res.body.pipe(dest);
 });
 


// buffer 
var fileType = require('file-type');
fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
 .then(function(res) {
 return res.buffer();
 }).then(function(buffer) {
 fileType(buffer);
 });
 

// meta 
fetch('https://github.com/')
 .then(function(res) {
 console.log(res.ok);
 console.log(res.status);
 console.log(res.statusText);
 console.log(res.headers.raw());
 console.log(res.headers.get('content-type'));
 });
 


// POST request 
fetch('http://httpbin.org/post', { method: 'POST', body: 'a=1' })
 .then(function(res) {
 return res.json();
 }).then(function(json) {
 console.log(json);
 });
 


// post with stream from resumer 
var resumer = require('resumer');
var stream = resumer().queue('a=1').end();
fetch('http://httpbin.org/post', { method: 'POST', body: stream })
 .then(function(res) {
 return res.json();
 }).then(function(json) {
 console.log(json);
 });
 



// post form data
 
var FormData = require('form-data');
var form = new FormData();
form.append('a', 1);
fetch('http://httpbin.org/post', { method: 'POST', body: form })
 .then(function(res) {
 return res.json();
 }).then(function(json) {
 console.log(json);
 });
 




// post form data and modify header 
var FormData = require('form-data');
var form = new FormData();
form.append('a', 1);
fetch('http://httpbin.org/post', { method: 'POST', body: form, headers: form.getHeaders() })
 .then(function(res) {
 return res.json();
 }).then(function(json) {
 console.log(json);
 });
 


// node 0.12+, yield with co 
var co = require('co');
co(function *() {
 var res = yield fetch('https://api.github.com/users/github');
 var json = yield res.json();
 console.log(res);
});

API

fetch(url, options)

Returns a Promise object

Node.js http Module GET and POST Requests

Thu, 16 Feb 2017 20:32:05 +0000

In front-end, we often need to use ajax to get and submit data to the server. The server can also initiate the same request to another server.

Today, let’s talk about how Node.js initiates requests to another server.

GET Request

Writing is quite simple, a bit similar to angular’s $http.get.

 var http = require('http');
 http.get("http://baidu.com", function(response) {
 console.log(response)
 });
})

POST Request

Compared to get, post writing is much more cumbersome.

Get Public IP with Node.js

Wed, 15 Feb 2017 21:30:05 +0000

Because Raspberry Pi is placed at home, due to ISP reasons, public IP keeps changing. How can we quickly get the current network’s public IP?

Today introduce a small tool that can simply get public IP.

public-ip

Installation

npm install --save public-ip

Usage

const publicIp = require('public-ip');
 
publicIp.v4().then(ip => { 
 console.log(ip);
 //=> '46.5.21.123' 
});
 
publicIp.v6().then(ip => {
 console.log(ip);
 //=> 'fe80::200:f8ff:fe21:67cf' 
});

Set HTTPS

If using https, it will query based on icanhazip.com service.

Automated Deployment Based on Github and Coding Code Hosting Platforms

Tue, 14 Feb 2017 22:30:05 +0000

Since playing with Raspberry Pi, I need to frequently update code to Raspberry Pi every day.

But it’s too frequent, repetitive things need to be done over and over. Very time-consuming.

Found Github and Coding both have webhook functionality,

So I had an idea:

Want every time git push updates code, Use hosting platform’s webhook to notify my Raspberry Pi, After receiving notification, automatically update hosting platform’s latest code, and automatically restart.

Generate Aliyun API Signature with Node.js

Mon, 13 Feb 2017 21:32:05 +0000

To call Aliyun’s DNS API, you must generate the signature required by Aliyun. Signature Mechanism

I couldn’t find Node.js related code for getting signatures online. According to Aliyun’s requirements, I wrote a program myself to easily generate signatures.

const querystring = require('querystring');
const crypto = require("crypto");

module.exports=function getSignatureParams(params) {
 StringToSign = "GET" + "&" + percentEncode("/") + "&" + percentEncode(uriSort(params))
 var Signture = getSignture(StringToSign);
 params.Signature = Signture;
 return params; // Return complete uri with signature
}


 function percentEncode(str) { // Percent encode uri
 var s = encodeURI(str);
 return s
 .replace(/ /g, "%20")
 .replace(/\//g, "%2F")
 .replace(/\+/g, "%20")
 .replace(/\*/g, "%2A")
 .replace(/\%7E/g, "~")
 .replace(/\=/g, "%3D")
 .replace(/\&/g, "%26")
 .replace(/\:/g, "%253A")
 }
 
 function getSignture(Signature) { // Calculate HMAC
 // 47awTgVxfVEBL8hewkBgYD6kEvuJn0 
 return crypto
 .createHmac('sha1', "testsecret&") // Your secret
 .update(Signature)
 .digest()
 .toString('base64');
 }

 function uriSort(uri) { // Sort uri parameters
 var arr = querystring.stringify(uri).split("&");
 arr = arr.sort();
 var str = "";
 [].forEach.call(arr, function (s, i) {
 if (i == (arr.length - 1)) {
 str = str + s
 } else {
 str = str + s + "&"
 }
 });
 return str;
 }

Backend on Time Friend

cheeriojs Chinese Parsed to Unicode Issue When Loading HTML

cheeriojs Parsing Chinese to Unicode Issue

Solution

Default Configuration

Build Simpler Private npm Server with verdaccio

Introduction

Install verdaccio on Your Server

Start

Multi-Server Automated Deployment Solution Based on shipit-deploy

Usage

1. Download and Install

2. Establish Trust Relationship with Servers

3. Configuration

Enable Gzip Mode for Express, Egg.js, Koa.js

Express

Egg.js

Koa.js

Apache, Nginx, Express, Egg.js Support Frontend HTML5 History Mode

Apache

nginx

Node.js (Express)

How Egg.js Supports Single Page Application

Publish Company Private Code to Self-Hosted cnpm

Configuration

Restart cnpm

npm Login

Build Private cnpm Server in Company

Visual Studio Code Hide Extra js and map Files Generated from ts

Debug Node.js with Chrome DevTools

Prerequisites

Configure Chrome (Latest versions no longer need this step);

Run Node.js

Publish Aliyun API Signature Generation Tool Based on Node.js -- AliToSing

npm, yarn Package Manager Switch to Taobao Registry

NPM

yarn

Express Set CORS, Let Browser Cross-Origin Access Your Server

Cross-Origin Resource Sharing (CORS);

Make Node.js Fetch Like Browser

Installation

Usage

API

fetch(url, options)

Node.js http Module GET and POST Requests

GET Request

POST Request

Get Public IP with Node.js

Installation

Usage

Set HTTPS

Automated Deployment Based on Github and Coding Code Hosting Platforms

Generate Aliyun API Signature with Node.js