mycli是印度程序员开发的,用在linux操作系统的mysql客户端
yum install python-pip
pip install mycli
mycli –version
mycli –help
mycli my_database
mycli -u my_user -h my_host.com my_database
mycli是印度程序员开发的,用在linux操作系统的mysql客户端
mycli是印度程序员开发的,用在linux操作系统的mysql客户端
yum install python-pip
pip install mycli
mycli –version
mycli –help
mycli my_database
mycli -u my_user -h my_host.com my_database
kafka集群(3节点)+zk集群(3节点)接入项目中,待观察性能
kafka集群(3节点)+zk集群(3节点)接入项目中,待观察性能
环境配置:
操作系统 windows7旗舰版 16G内存 64位
2.kafka集群节点分别安装在三台win7系统上
3.zk集群安装在一台win7系统上,是伪集群
kafka配置文件:
server.properties
#此Broker的ID,集群中每个Broker的ID不可相同
broker.id=0
#监听器,端口号与port一致即可
listeners=PLAINTEXT://:9092
#Broker的端口
port=9092
#Broker的Hostname,填主机IP即可
host.name=192.168.1.102
#向Producer和Consumer建议连接的Hostname和port
advertised.host.name=192.168.1.102
advertised.port=9092
#进行IO的线程数,应大于主机磁盘数
num.network.threads=8
# The number of threads doing disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=1048576
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=1048576
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
queued.max.requests=16
fetch.purgatory.purge.interval.requests=100
producer.purgatory.purge.interval.requests=100
#自动进行重新选举leader
auto.leader.rebalance.enable=true
# Replication configurations
num.replica.fetchers=4
replica.fetch.max.bytes=1048576
replica.fetch.wait.max.ms=500
replica.high.watermark.checkpoint.interval.ms=5000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.lag.time.max.ms=10000
############################# Log Basics #############################
#消息文件存储的路径
log.dirs=D:/kafka_2.11-0.9.0.0/logs
#每个Topic默认的分区数,一般在创建Topic时都会指定分区数,所以这个配成1就行了
num.partitions=1
message.max.bytes=1000000
auto.create.topics.enable=true
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.flush.scheduler.interval.ms=2000
log.roll.hours=168
controller.socket.timeout.ms=30000
controller.message.queue.size=10
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Log Flush Policy #############################
# The number of messages to accept before forcing a flush of data to disk
log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
log.flush.interval.ms=1000
############################# Log Retention Policy #############################
#消息文件清理周期,即清理x小时前的消息记录
log.retention.minutes=2
# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
# By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires.
# If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction.
log.cleaner.enable=false
############################# Zookeeper #############################
#Zookeeper连接串,此处填写上一节中安装的三个zk节点的ip和端口即可
zookeeper.connect=192.168.1.102:2181,192.168.1.102:2182,192.168.1.102:2183
#zookeeper.connect=192.168.1.102:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
zookeeper.sync.time.ms=2000
zk配置文件:
zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=D:/software/zookeeper-2181/data
dataLogDir=D:/software/zookeeper-2181/logs
clientPort=2181
server.0=192.168.1.102:2777:3777
server.1=192.168.1.102:2888:3888
server.2=192.168.1.102:2999:3999
在生成的build.gradle文件中配置: plugins { id 'java' id 'application' id 'war' id 'org.gretty' version '2.2.0' id 'com.gradle.build-scan' version '1.16' id 'org.springframework.boot' version '2.1.2.RELEASE' id "io.spring.dependency-management" version "1.0.6.RELEASE" } //基本配置 group 'com.df' version '1.0-SNAPSHOT' mainClassName = 'com.df.run.JobApplication' sourceCompatibility = 1.8 targetCompatibility = 1.8 //Java compiler options compileJava { options.incremental = true options.fork = true options.failOnError = false options.encoding = "UTF-8" } repositories { jcenter() mavenLocal() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } } configurations { developmentOnly runtimeClasspath { extendsFrom developmentOnly } } dependencies { //spring boot implementation 'org.springframework.boot:spring-boot-parent:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-aop:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-data-jdbc:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-data-jpa:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-dependencies:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-test-autoconfigure:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-tools:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-cloud-connectors:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-websocket:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-reactor-netty:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-cache:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-groovy-templates:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-docs:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-actuator:2.1.2.RELEASE' //spring boot监控工具 annotationProcessor "org.springframework.boot:spring-boot-configuration-processor" //配置文件处理器 developmentOnly("org.springframework.boot:spring-boot-devtools") //授权Oauth2模块 //implementation 'org.springframework.boot:spring-boot-starter-security:2.1.2.RELEASE' //implementation 'org.springframework.security.oauth:spring-security-oauth2:2.1.2.RELEASE' //implementation 'org.springframework.security.oauth.boot:spring-security-oauth2-autoconfigure:2.1.2.RELEASE' //接口授权oauth2 //implementation 'org.springframework.boot:spring-boot-starter-oauth2-client:2.1.2.RELEASE' //接口授权oauth2 //implementation 'org.springframework.boot:spring-boot-starter-data-redis:2.1.2.RELEASE' //将token存储在redis中 //配置shiro认证 implementation 'com.github.theborakompanioni:thymeleaf-extras-shiro:2.0.0' implementation 'org.apache.shiro:shiro-core:1.4.0' implementation 'org.apache.shiro:shiro-spring:1.4.0' implementation 'org.apache.shiro:shiro-ehcache:1.4.0' implementation 'org.apache.shiro:shiro-web:1.4.0' implementation 'org.apache.shiro:shiro-quartz:1.4.0' //配置hibernate /* implementation ('org.hibernate:hibernate-core:5.2.17.Final') { exclude group: 'org.slf4j',module: 'slf4j-api' //排除某一个库(slf4j)依赖 } implementation 'org.hibernate:hibernate-entitymanager:4.3.11.Final'*/ //配置mysql数据库 implementation 'mysql:mysql-connector-java:8.0.13' //mysql 6.0.5 implementation 'com.zaxxer:HikariCP:3.3.0' //HikariCP高效数据库连接池 implementation 'org.mybatis.spring.boot:mybatis-spring-boot-starter:1.3.2' //mybatis implementation 'com.github.pagehelper:pagehelper-spring-boot-starter:1.2.10' //分页查询pagehelper版本1.2.10对应springboot2.1.0.RELEASE implementation 'com.alibaba:druid-spring-boot-starter:1.1.10' //Web相关 implementation 'org.springframework.boot:spring-boot-starter-web:2.1.2.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-thymeleaf:2.1.2.RELEASE' implementation 'net.sourceforge.nekohtml:nekohtml:1.9.22' providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat' providedCompile 'javax.servlet:javax.servlet-api:3.1.0' //quartz定时器 //implementation 'org.springframework.boot:spring-boot-starter-quartz:2.1.2.RELEASE' //缺点:同一个task,如果前一个还没跑完后面一个就不会触发,不同的task也不能同时运行 implementation 'org.quartz-scheduler:quartz:2.3.0' implementation 'org.quartz-scheduler:quartz-jobs:2.3.0' implementation 'org.springframework:spring-tx:5.1.4.RELEASE' implementation 'org.springframework:spring-context-support:5.1.4.RELEASE' implementation 'org.springframework.boot:spring-boot-starter-amqp:2.1.2.RELEASE' //RabbitMQ compile group: 'c3p0', name: 'c3p0', version: '0.9.1.2' //在quartz定时器默认使用c3p0连接池 compile group: 'org.springframework.boot', name: 'spring-boot-starter-velocity', version: '1.4.7.RELEASE' //velocity compile group: 'org.springframework', name: 'spring-context-support', version: '5.1.4.RELEASE' //compile group: 'org.springframework', name: 'spring-tx', version: '5.1.4.RELEASE' //配置H2数据库 runtime group: 'com.h2database', name: 'h2', version: '1.4.197' //H2内嵌数据库 //其它 implementation 'com.alibaba:fastjson:1.2.54' //阿里fastjson implementation 'commons-io:commons-io:2.6' //commons-io implementation 'com.google.guava:guava:23.0' //guava implementation 'redis.clients:jedis:2.9.0' //implementation 'org.slf4j:slf4j-api:1.7.25' //slf4j implementation 'org.jsoup:jsoup:1.11.3' implementation 'commons-fileupload:commons-fileupload:1.2' implementation 'org.apache.poi:poi-ooxml:3.17' implementation 'com.github.penggle:kaptcha:2.3.2' implementation 'io.springfox:springfox-swagger2:2.7.0' implementation 'io.springfox:springfox-swagger-ui:2.7.0' implementation 'eu.bitwalker:UserAgentUtils:1.19' implementation 'com.github.oshi:oshi-core:3.9.1' components { withModule('org.springframework:spring-beans') { allVariants { withDependencyConstraints { // Need to patch constraints because snakeyaml is an optional dependency it.findAll { it.name == 'snakeyaml' }.each { it.version { strictly '1.19' } } } } } } //hadoop //implementation 'org.apache.hadoop:hadoop-client:2.7.3' //implementation 'org.apache.hadoop:hadoop-common:2.7.3' //implementation 'org.apache.hadoop:hadoop-hdfs:2.7.3' //scala implementation 'org.scala-lang:scala-library:2.11.12' //spark implementation 'org.apache.spark:spark-mllib_2.11:2.4.0' implementation 'org.apache.spark:spark-sql_2.11:2.4.0' implementation 'org.apache.spark:spark-graphx_2.11:2.4.0' implementation 'org.apache.spark:spark-launcher_2.11:2.4.0' implementation 'org.apache.spark:spark-catalyst_2.11:2.4.0' implementation 'org.apache.spark:spark-streaming_2.11:2.4.0' implementation 'org.apache.spark:spark-core_2.11:2.4.0' //单元测试 testCompile 'org.mockito:mockito-core:2.7.19' testCompile group: 'junit', name: 'junit', version: '4.12' testCompile 'org.seleniumhq.selenium:selenium-java:3.3.1' testRuntime 'org.scala-lang.modules:scala-xml_2.11:1.1.0' testImplementation 'org.springframework.boot:spring-boot-starter-test' } //打包 jar { enabled = true version = '0.0.1' manifest { attributes("Implementation-Title": "sparkjob", "Implementation-Version": version) } } //设置启动类 bootJar { mainClassName = 'com.df.run.JobApplication' } // buildScan buildScan { // always accept the terms of service termsOfServiceUrl = 'https://gradle.com/terms-of-service' termsOfServiceAgree = 'yes' // always publish a build scan publishAlways() }
1.修改 php-fpm.conf监听参数
vim /usr/local/php/etc/php-fpm.conf
1.1 代理接口服务器47.110.157.1上
listen = 172.16.231.137:9000
1.2代理接口服务器47.110.158.10上
listen = 172.16.231.136:9000
1.3检测php-fpm.conf文件配置准确性
/usr/local/php/sbin/php-fpm -t
1.4查看修改后的php-fpm.conf配置
grep -v ‘^$’ /usr/local/php/etc/php-fpm.conf | grep -v ‘^;’
1.5重启php-fpm服务
systemctl restart php-fpm
systemctl status php-fpm
2.配置nginx.conf文件
2.1在http节点内
vim /etc/nginx/nginx.conf
#代理接口服务器47.110.157.1和47.110.158.10上都增加
#设置置php负载均衡
upstream fastcgiserver {
server 172.16.231.137:9000;
server 172.16.231.136:9000;
}
2.2.修改具体项目修改fastcgi_pass地址
#下面以agyl项目为例
vim /etc/nginx/conf.d/home_agyl.conf
#修改内容如下所示
#astcgi_pass 127.0.0.1:9000;
fastcgi_pass fastcgiserver;
2.3重载nginx配置文件
/usr/sbin/nginx -t
/usr/sbin/nginx -s reload
3.修改php.ini参数
3.1创建共享session文件夹
mkdir -pv /data/php_session
cp -rf /var/lib/php/session/* /data/php_session/
chown -R nginx.nginx /data/php_session
chmod 777 -R /data/php_session
3.4#修改php.ini文件
vim /usr/local/php/etc/php.ini
#Redis共享方式,修改内容如下所示
session.save_handler = Redis
session.save_path = “tcp://192.168.5.114:6379” #Redis不需要密码验证
session.save_path = “tcp://192.168.5.114:6379?auth=password” #Redis需要密码验证
#NFS共享磁盘方式,修改内容如下所示
session.save_handler = files
session.save_path = “/data/php_session”
3.3检测php-fpm.conf文件配置准确性
/usr/local/php/sbin/php-fpm -t
3.4重启php-fpm服务
systemctl restart php-fpm
systemctl status php-fpm
3.5查看修改后的php-fpm.conf配置
grep -v ‘^$’ /usr/local/php/etc/php.ini | grep -v ‘^;’
Mycat官网:http://mycat.io/
官网下载地址:http://dl.mycat.io/
源码git地址:https://github.com/MyCATApache/Mycat-download
1.环境准备
需要安装jdk7+以上,本人安装的是jdk8;怎样安装jdk,在这里就不再详说了,大家可以网上查下度娘。
2.安装
mkdir -pv /home/sofrt
wget http://dl.mycat.io/1.6-RELEASE/Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz
tar zxvf Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz
mv mycat /usr/local/
3.配置
配置文件在/home/mycat/conf下
–bin 启动目录
–conf 配置文件存放配置文件:
–server.xml:是Mycat服务器参数调整和用户授权的配置文件。
–schema.xml:是逻辑库定义和表以及分片定义的配置文件。
–rule.xml: 是分片规则的配置文件,分片规则的具体一些参数信息单独存放为文件,也在这个目录下,配置文件修改需要重启MyCAT。
–log4j.xml: 日志存放在logs/log中,每天一个文件,日志的配置是在conf/log4j.xml中,根据自己的需要可以调整输出级别为debug debug级别下,会输出更多的信息,方便排查问题。
–autopartition-long.txt,partition-hash-int.txt,sequence_conf.properties, sequence_db_conf.properties 分片相关的id分片规则配置文件
–lib MyCAT自身的jar包或依赖的jar包的存放目录。
–logs MyCAT日志的存放目录。日志存放在logs/log中,每天一个文件
1.主机规划
192.168.1.52 gfs1
192.168.1.71 gfs2
2.配置ssh免密码登录
在两个节点上分别执行
1).生成密钥文件
ssh-keygen -t rsa
2).将公钥拷贝到对方
#在192.168.1.52 gfs1上执行
ssh-copy-id root@192.168.1.71
#在192.168.1.71 gfs2上执行
ssh-copy-id root@192.168.1.52
3).测试
#在192.168.1.52 gfs1上执行
ssh root@192.168.1.71
#在192.168.1.71 gfs2上执行
ssh root@192.168.1.52
3.安装
1).在node{1-2}上安装glusterfs服务器和客户端
yum install -y centos-release-gluster38 glusterfs glusterfs-server glusterfs-fuse
2).设置开机自启动并启动
systemctl enable glusterd
systemctl start glusterd
3).在gfs1-gfs2节点上配置整个GlusterFS集群,把各个节点加入到集群
vim /etc/hosts
修改内容如下:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.52 gfs1
192.168.1.71 gfs2
4).在任意一个节点上执行(添加对方为到集群)
如gfs1节点上执行
gluster peer probe gfs2
如果gfs2节点上即执行
gluster peer probe gfs1
5).在gfs{1-2}设置防火墙
systemctl start firewalld
systemctl enable firewalld
firewall-cmd –add-service=glusterfs –permanent
firewall-cmd –zone=public –add-port=24007-24008/udp –permanent
firewall-cmd –zone=public –add-port=24007-24008/tcp –permanent
firewall-cmd –zone=public –add-port=49152/tcp –permanent
firewall-cmd –reload
4.查看节点状态
gluster peer status
5.在gfs{1-2}上创建数据存储目录
mkdir -p /home/share/models
6.在gfs1上创建GlusterFS磁盘
注意:加上replica 2就是2个节点中,每个节点都要把数据存储一次,就是一个数据存储2份,每个节点一份
如果不加replica 2,就是2个节点的磁盘空间整合成一个硬盘
gluster volume create models replica 2 transport tcp gfs1:/home/share/models gfs2:/home/share/models force
7.创建/启动/停止/删除卷
gluster volume start models
gluster volume stop models
gluster volume delete models
注意,删除卷的前提是先停止卷
8.设置卷参数,如关闭只读,启动写,删除操作
gluster volume set models features.read-only Off
gluster volume set models nfs.trusted-write On
9.客户端以glusterfs方式挂载(此步重要,数据同步)
部署GlusterFS客户端并mount GlusterFS文件系统
注:ro:只读模式 rw:读写模式
1). 在gfs{1-2}上创建数据存储目录
mkdir -p /mnt/models
2). 在192.168.1.52 gfs1上执行
mount -t glusterfs -o rw gfs1:models /mnt/models/
3). 在192.168.1.71 gfs2上执行
mount -t glusterfs -o rw gfs2:models /mnt/models/
4).在gfs{1-2}上设置开机自动挂载卷
vim /etc/fstab
在192.168.1.52 gfs1上执行添加内容:
gfs1:models /mnt/models glusterfs defaults,_netdev 0 0
在72.168.30.71 gfs2上执行添加内容:
gfs2:models /mnt/models glusterfs defaults,_netdev 0 0
10.修改客户端只读为读写方式
1).检查是否有ro只读模式
mount
如果发现有ro,先umount后重新mount
umount /mnt/models
2).如果发现有提示“device is busy”,执行以下操作
在192.168.1.52 gfs1上执行
gluster volume stop models
umount /mnt/models
gluster volume start models
mount -t glusterfs -o rw gfs1:models /mnt/models/
在192.168.1.71 gfs2上执行
gluster volume stop models
umount /mnt/models
gluster volume start models
mount -t glusterfs -o rw gfs2:models /mnt/models/