Linux笔记

字数 4161 · 2018-05-23

#linux

Shortcuts

Ctrl+A - Move cursor to the beginning of line
Ctrl+E - Move cursor to the end of line
Ctrl+U - Delete from cursor to the beginning of line
Ctrl+K - Delete from cursor to the end of line

ls



1
2
3
4
5



# one entry per line
ls -1
# -A: all except . and ..
# -a: include hidden
# -h: human read

exa

man

display manual pages.

Most underlined words in man pages are for emphasis. Its is not a hyperlink .

MANUAL SECTIONS
The standard sections of the manual include:



1
2
3
4
5
6
7
8



1      User Commands
2      System Calls
3      C Library Functions
4      Devices and Special Files
5      File Formats and Conventions
6      Games et. al.
7      Miscellanea
8      System Administration tools and Daemons

set



1
2



# 指令返回不等于0，则退出shell
set -e

xargs

construct argument list(s) and execute utility

-I



1



find -type f emotion | xargs -I{} cp {} dest1

find

-type

  • f regular file
  • d directory

-maxdepth



1



find . -type d -maxdepth 4 | grep tencent

-E -regex



1



find -E emotion/28 -regex ".*[0-9]"





1
2



# 删除空文件夹
find -maxdepth 1 -type d -empty -delete

wc

-l lines
-m characters
-w words

uniq

uniq 命令只会去除相邻的重复行

-c --count 在每行前添加计数信息

sort

-r --reverse 倒序
-n numeric-sort
-t 设置分隔符
-k 设置key



1
2



# 多列排序
sort -t$'\t' -k1,1nr -k2,2nr file

cut

对每一行进行剪切

-d 指定分隔符，默认为 \t
-f 选择分割片段



1
2



# 取出每行的第一列和第二列
cut -f 1,2 file1

head



1



head -n 20

tail

crontab



1



# min	hour	day	month	weekday	command

awk

An AWK pogram is a sequence of pattern {action} pairs and user function definitions.



1
2
3
4



# NF 是 number of fields
awk "{print NF}" file
# 取第一个和第二个变量
awk '{print $1 $2}' file

关于 awk 脚本，我们需要注意两个关键词 BEGIN 和 END。

BEGIN{ 这里面放的是执行前的语句 }
END {这里面放的是处理完所有的行后要执行的语句 }
{这里面放的是处理每一行时要执行的语句}

内建变量

变量 描述
$n 当前行的第n个字段
$0 当前行
NF number of fields
NR number of the current record
FS field separator
OFS output field separator

数据结构

list map



1



awk '{my_dict[$1] = $2} END { for (key in my_dict) { print key ": " my_dict[key] } }' file1.txt

控制语句

if



1



awk 'if(表达式){语句}else{语句}' file

while

for

do

函数

system

调用系统命令



1



find src -type f | grep -e "\.js$" | awk '{print $0}{ system("cat " $0 " | grep live")}'

sed

stream editor

-E extended (modern) regular expressions

使用 man re_format 查看两种的区别

sed 命令的格式 [address[,address]]function[arguments]

通常 sed 会把输入的每一行复制到一个 pattern space 中（不包括换行符），接着执行所有的命令，然后将 pattern space 复制到输出并加上换行符，最后删除 pattern space。

一些函数使用 hold space 来保存部分或所有 pattern space 用于之后的查找。

Sed Addresses

数字 - 输入的行号
$ - 输入的最后一行
context address - 前后有分隔符的正则表达式

不指定address选中所有行。
使用一个address选中匹配的行。
使用两个address选中所有匹配的范围，范围是从匹配第一个address的行到匹配第二个address的行。

Sed Functions

i\ 在选中行前插入文本
d 删除选中行

[2addr]s/regular expression/replacement/flags 替换选中行
flags可以为:
N - 替换第N个匹配的字符串
g - 替换全部匹配的字符串
Any character other than backslash or newline can be used instead of a slash to delimit the RE and the replacement.

[2addr]!function
[2addr]!function-list
Apply the function or function-list only to the lines that are not selected by the address(es).

多条命令使用; 分割

demo

html正文提取
test.html



1
2
3
4
5
6
7
8
9



<html>
<head>
    <title>This is the page title</title>
</head>
<body>
    <p> This is the <b>first</b> line in the Web page.
    This should provide some <i>useful</i> information to use in our sed script.
</body>
</html>





1



cat test.html | sed -E 's/<[^>]+>//g;s/ {2,}//g;/^$/d'

执行结果：



1
2
3



This is the page title
This is the first line in the Web page.
This should provide some useful information to use in our sed script.

grep

print lines matching a pattern

-i ignore-case, grep 默认区分大小写

-E 扩展正则

In basic regular expressions the meta-characters ?, +, {, |, (, and ) lose their special meaning.

查找 jpg



1



ls | grep .*\.jpg

多关键字匹配



1
2



grep 'warning\|error\|critical' /var/log/messages

实例

数据



1
2
3
4
5
6



$ cat score.txt
Marry   2143 78 84 77
Jack    2321 66 78 45
Tom     2122 48 77 71
Mike    2537 87 97 95
Bob     2415 40 57 62

脚本



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24



$ cat cal.awk
#!/bin/awk -f
#运行前
BEGIN {
    math = 0
    english = 0
    computer = 0
 
    printf "NAME    NO.   MATH  ENGLISH  COMPUTER   TOTAL\n"
    printf "---------------------------------------------\n"
}
#运行中
{
    math+=$3
    english+=$4
    computer+=$5
    printf "%-6s %-6s %4d %8d %8d %8d\n", $1, $2, $3,$4,$5, $3+$4+$5
}
#运行后
END {
    printf "---------------------------------------------\n"
    printf "  TOTAL:%10d %8d %8d \n", math, english, computer
    printf "AVERAGE:%10.2f %8.2f %8.2f\n", math/NR, english/NR, computer/NR
}

输出



1
2
3
4
5
6
7
8
9
10
11



$ awk -f cal.awk score.txt
NAME    NO.   MATH  ENGLISH  COMPUTER   TOTAL
---------------------------------------------
Marry  2143     78       84       77      239
Jack   2321     66       78       45      189
Tom    2122     48       77       71      196
Mike   2537     87       97       95      279
Bob    2415     40       57       62      159
---------------------------------------------
  TOTAL:       319      393      350
AVERAGE:     63.80    78.60    70.00

Bash 基础

参数

参数 描述
$1,$2,$3 位置参数
$# 参数个数
$0 文件名
$* 所有参数
$? 上个命令的退出状态

Array

定义



1
2



# 使用空格分隔
arr=(1 2 3 4 5)





1
2
3
4



arr
arr[0]="a"
arr[1]="2"
arr[3]="3"

打印



1
2
3
4



# 全部
echo ${arr[*]}
# 第一个
echo ${arr[1]}

获取长度



1
2



${#arr[@]}
${#arr[*]}

删除



1
2
3
4



# 删除指定元素
unset arr[1]
# 删除整个数组
unset arr

遍历



1
2
3
4



for var in ${ arr[@] };
do
    echo $var
done

循环

函数



1
2
3



function_name () {
<commands>
}





1
2
3



function function_name {
<commands>
}

上述两种方法是等价的
返回值是 void
() 仅仅是装饰
定义的语句要先于调用的语句

参数传递

传参的方式和调用命令相似，直接写在函数名后面即可，函数内使用 $1 $2 这样的调用方式。



1
2
3
4
5
6
7



#!/bin/bash
# Passing arguments to a function
print_something () {
echo Hello $1
}
print_something Mars
print_something Jupiter

返回值

Bash函数不支持返回值，仅仅支持返回状态值，使用关键字 return。
通常返回0代表执行正常，非0则代表出现了异常



1
2
3
4



print_something () {
echo Hello $1
return 5
}

bash-scripting-tutorial

命令查找

which
whereis
type

系统信息

du

du – display disk usage statistics



1



du -hd1 | sort -h

df

df – display free disk space



1



df -h .

free

free – display amount of free and used memory in the system

uptime

uptime – show how long system has been running



1
2



# 当前时间 | 已运行时间 | 用户连接数 | 最近 1，5，15 分钟的系统平均负载
# 10:52  up 13 days, 18:06, 9 users, load averages: 2.58 2.70 4.38

文件压缩

zip

  • -r 递归目录
  • -x 排除


1



zip -r app.zip app -x "*/.git/*" "*.DS_Store*" "*/node_modules/*"

gzip

*.gz

bzip2

*.bz2

tar

-c Create a new archive containing the specified items.
-t List archive contents to stdout.
-x Extract to disk from the archive.

-j (c mode only) Compress the resulting archive with bzip2
-z (c mode only) Compress the resulting archive with gzip
-Z (c mode only) Compress the resulting archive with compress



1
2
3



tar -zcvf app.tar.gz app
tar -jcvf app.tar.bz2 app
tar -Jcvf app.tar.xz app

网络命令

curl



1



curl <url>

wget

The non-interactive network downloader.



1



wget <url>

--header=header-line



1
2
3



wget --header='Accept-Charset: iso-8859-2' \
     --header='Accept-Language: hr'        \
       http://fly.srk.fer.hr/

wget ftp://ftp.sample.test/file.txt

aria

BitTorrent:



1



aria2c http://example.org/mylinux.torrent

BitTorrent Magnet URI:



1



aria2c 'magnet:?xt=urn:btih:248D0A1CD08284299DE78D5C1ED359BB46717D8C'

https://aria2.github.io/

ftp



1
2



# connect to the remote machine
ftp ftp.demo.org

通常可以使用匿名账户 anonymous 登录

登录成功后使用 FTP 命令进行操作

Command Desc
?/help require help
bye/quit/exit exit the FTP
cd change directory
ls list files
pwd find current directory pathname
get ABC copy one file from remote with same name to curent local directory

traceroute



1
2



# use ICMP
traceroute -q 1 -I baidu.com

ssh

Alias



1
2
3
4
5
6



# ssh ubuntu
# ~/.ssh/config
Host ubuntu
    HostName 192.168.0.101
    User ubuntu
    Port 8022

iptables

chain - set of rules

四表五链

  • raw
    • PREROUTING
    • OUTPUT
  • mangle
    • PREROUTING
    • POSTROUTING
    • INPUT
    • OUTPUT
    • FORAWRD
  • nat
    • PREROUTING
    • POSTROUTING
    • OUTPUT
  • filter
    • INPUT
    • FORAWRD
    • OUTPUT

表的优先级：

raw > mangle > nat > filter

链内的规则匹配：

  • 从上到下依次检查，找到匹配的规则即停止
  • 没有找到匹配的规则，则使用默认策略 （默认为允许）

配置文件：/etc/sysconfig/iptables

  • INPUT
  • OUTPUT
  • NAT
  • POSTROUTING
  • PREROUTING

配置完即刻生效

  • Terminating Targets
    • Accept - Allow
    • Drop - act like it nver happened
    • Reject - Don’t allow, and send back an error
      • connection reset for TCP
      • host unreachable for UDP and ICMP

non-terminating targets, which keep matching other rules:

  • LOG

多媒体

file

determine file type

ffmpeg



1
2
3
4



# 生成测试视频
ffmpeg -f lavfi -i testsrc=duration=10:size=1280x720:rate=30 testsrc.mpg
# 转换格式
ffmpeg -i testsrc.mpg testsrc.mp4





1
2



# aiff to mp3
ffmpeg -i hello.aiff -f mp3 -acodec libmp3lame -ab 19200 -ar 44100 hello.mp3





1
2



# 2 hours blank video
ffmpeg -t 7200 -f lavfi -i color=c=black:s=640x480 -c:v libx264 -tune stillimage -pix_fmt yuv420p output.mp4





1
2
3
4



# cut video
ffmpeg -i movie.mp4 -ss 00:00:03 -t 00:00:08 -async 1 cut.mp4
# alternative
ffmpeg -ss 00:01:12 -i movie.mp4 -ss 00:00:00 -t 00:00:03 -c copy cut.mp4





1
2



# merge video and audio
ffmpeg -i video.mp4 -i audio.wav -c:v copy -c:a aac output.mp4





1
2
3
4



# extract audio from mp4
ffmpeg.exe -i in.mp4 -vn -c:a copy -ac 2 out.m4a
# extract audio from webm
ffmpeg -i "source.webm" -vn -acodec copy "output.opus"

Synopsis



1



ffmpeg [global_options] {[input_file_options] -i input_url} ... {[output_file_options] output_url} ...

filter

-vf -af -lavfi

The -lavfi option is equivalent to -filter_complex

main options

  • -f - fmt
  • -i - input
  • -t - duration
  • -to - Stop writing the output or reading the input at position.
  • -ss - When used as an input option (before -i), seeks in this input file to position.

video options

  • -r - fps
  • -s - frame size

advanced video options

  • -pix_fmt - pixel format

Stream specifiers

Some options are applied per-stream.
A stream specifier is a string generally appended to the option name and separated from it by a colon. E.g. -codec:a:1 ac3.

A stream specifier can match several streams.
An empty stream specifier matches all streams.

Possible forms of stream specifiers are:

  • stream_index
  • stream_type[:additional_stream_specifier]
    • v or V for video, a for audio, s for subtitle, d for data, and t for attachments
  • p:program_id[:additional_stream_specifier]
  • #stream_id or i:stream_id
  • m:key[:value]
  • u

lame

lame - create mp3 audio files



1
2



# Encoding Audio.aiff to Audio.mp3
lame Audio.aiff

imagemagick

ImageMagick - is a free software suite for the creation, modification and display of bitmap images.



1



magick image.jpg image.png

command-line tools:

  • magick
  • mogrify
  • identify
  • composite
  • compare
  • …


1
2
3
4
5
6



# show info
identify test.jpg # test.jpg JPEG 1536x960 1536x960+0+0 8-bit sRGB 267493B 0.030u 0:00.039
# resize picture, keep ratio
magick test.jpg -resize 1280x720 resized.png
# crop resize picture
magick big.png  -resize 1080x1920\^ -gravity center  -extent 1080x1920 sm.png

webp

Convert Images to the WebP Format:



1



cwebp -q 80 image.png -o image.webp

Convert Images from the WebP Format:



1
2



dwebp image.webp -o image.png

签名

gpg



1



gpg --verify doc.sig doc

https://www.gnupg.org/gph/en/manual/x135.html

其他

htop

htop - interactive process viewer



1
2
3



CPU usage bar: [low-priority/normal/kernel/virtualiz               used%]
Memory bar:    [used/buffers/cache                            used/total]
Swap bar:      [used                                          used/total]

P M T: sort by CPU%, MEM% or TIME
I: invert sort order



1
2
3
4
5
6
7
8
9
10
11
12
13



## xxd 
make a hexdump

## hexdump

## base64

```sh
# decode
cat f1 | base64 -d
# encode
echo 'aaa' | base64

jq

Command-line JSON processor



1
2
3
4



# pretty print
jq . test.json
# path
jq .[0].a test.json

fx

Command-line JSON processor

  • Interactive mode

https://github.com/antonmedv/fx

trans

Command-line translator powered by Google Translate



1
2



trans --brief '单元测试'
# > unit test

https://github.com/soimort/translate-shell

/etc

passwd



1
2



# list all user
cat /etc/passwd





1
2
3



# username:password:UID:GID:Comment:HomeDirectory:ShellUsed
root:x:0:0:root:/root:/bin/zsh
# ...

group



1
2



# list all group
cat /etc/group





1
2
3
4



# group_name:password:GID:Users_in_the_group
root:x:0:root
bin:x:1:root,bin,daemon
# ...





1
2
3
4
5



# list groups is in
groups
# root bin daemon sys adm disk wheel floppy dialout tape video
groups bin
# bin : bin daemon sys

重定向



1



command1 > everything.txt 2>&1

modern unix

rg



1
2
3
4
5
6
7



# 计算文件数
find -type f | wc -l
# 更快
dust -f | tail -n1 | cut -d' ' -f1
# 更更快
rg --files --hidden --no-ignore | wc -l
# fd | wc -l





1
2



# 寻找文件
rg --files | rg pdf$

https://github.com/BurntSushi/ripgrep

dust



1
2



# 列出文件数
dust -f





1
2
3
4
5



# 递归删除空文件夹
# dust: -d depth, -f file count, -b no percent bar, -c, no color
# rm: -d directory, -r recursive, -v verbose 
dust -rfbcd1 | rg '^0.*── (.*)' -r '$1' | xargs -d '\n' rm -drv
# Mac: ... | tr '\n' '\0' | xargs -0 | ...





1
2



# 递归清除小文件夹
dust -rbcd1 | rg '[0-9\.]+K.*── (.*)' -r '$1 $2' | awk '{if ($1 < 12) print $2}' | xargs -d '\n' rm -drv

exa

tldr

bat

A cat(1) clone with syntax highlighting and Git integration.

https://github.com/sharkdp/bat

delta

https://github.com/dandavison/delta

https://github.com/ibraheemdev/modern-unix

zoxide

https://github.com/ajeetdsouza/zoxide

volta

alternative nvm



1
2



# node path
~/.volta/tools/image/node/14.18.3/bin/node





1
2
3
4



# install node
volta install node@17
# install tools
volta install yarn





1
2



# set your default version of a tool, fetch that tool if it isn’t cached
volta install





1
2



# update a project’s package.json file to use the selected version of a tool
volta pin





1



volta list

https://docs.volta.sh/

ni

ni nr nx

https://github.com/antfu/ni

hexyl

a command-line hex viewer

  • NULL bytes 0 (00)
  • printable ASCII characters (21-7e)
  • ASCII whitespace characters _ (09, 0a, 0c, 0d, 20)
  • other ASCII characters • (01-08, 0b, 0e-1f, 7f)
  • non-ASCII x (80-ff)

https://github.com/sharkdp/hexyl

Misc

创建 Swap



1
2
3
4
5
6



sudo -i
dd if=/dev/zero of=/data/swap bs=512 count=8388616
mkswap /data/swap
swapon /data/swap
# permanent
echo "/data/swap swap swap defaults 0 0" >> /etc/fstab

pretty print PATH



1



echo $PATH | tr : \\n