kafka笔记

kafka集群(3节点)+zk集群(3节点)接入项目中,待观察性能

kafka集群(3节点)+zk集群(3节点)接入项目中,待观察性能

环境配置:

  1. 操作系统 windows7旗舰版 16G内存 64位

      2.kafka集群节点分别安装在三台win7系统上

      3.zk集群安装在一台win7系统上,是伪集群

kafka配置文件:

server.properties

#此Broker的ID,集群中每个Broker的ID不可相同

broker.id=0

#监听器,端口号与port一致即可

listeners=PLAINTEXT://:9092

#Broker的端口 

port=9092

#Broker的Hostname,填主机IP即可

host.name=192.168.1.102

#向Producer和Consumer建议连接的Hostname和port

advertised.host.name=192.168.1.102

advertised.port=9092

#进行IO的线程数,应大于主机磁盘数 

num.network.threads=8

# The number of threads doing disk I/O

num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server

socket.send.buffer.bytes=1048576

# The receive buffer (SO_RCVBUF) used by the socket server

socket.receive.buffer.bytes=1048576

# The maximum size of a request that the socket server will accept (protection against OOM)

socket.request.max.bytes=104857600

queued.max.requests=16

fetch.purgatory.purge.interval.requests=100

producer.purgatory.purge.interval.requests=100

#自动进行重新选举leader

auto.leader.rebalance.enable=true

# Replication configurations

num.replica.fetchers=4

replica.fetch.max.bytes=1048576

replica.fetch.wait.max.ms=500

replica.high.watermark.checkpoint.interval.ms=5000

replica.socket.timeout.ms=30000

replica.socket.receive.buffer.bytes=65536

replica.lag.time.max.ms=10000

############################# Log Basics #############################

#消息文件存储的路径 

log.dirs=D:/kafka_2.11-0.9.0.0/logs

#每个Topic默认的分区数,一般在创建Topic时都会指定分区数,所以这个配成1就行了 

num.partitions=1

message.max.bytes=1000000

auto.create.topics.enable=true

log.index.interval.bytes=4096

log.index.size.max.bytes=10485760

log.flush.scheduler.interval.ms=2000

log.roll.hours=168

controller.socket.timeout.ms=30000

controller.message.queue.size=10

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.

# This value is recommended to be increased for installations with data dirs located in RAID array.

num.recovery.threads.per.data.dir=1

############################# Log Flush Policy #############################

# The number of messages to accept before forcing a flush of data to disk

log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush

log.flush.interval.ms=1000

############################# Log Retention Policy #############################

#消息文件清理周期,即清理x小时前的消息记录 

log.retention.minutes=2

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining

# segments don't drop below log.retention.bytes.

#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.

log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according 

# to the retention policies

log.retention.check.interval.ms=300000

# By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires.

# If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction.

log.cleaner.enable=false

############################# Zookeeper #############################

#Zookeeper连接串,此处填写上一节中安装的三个zk节点的ip和端口即可 

zookeeper.connect=192.168.1.102:2181,192.168.1.102:2182,192.168.1.102:2183

#zookeeper.connect=192.168.1.102:2181

# Timeout in ms for connecting to zookeeper

zookeeper.connection.timeout.ms=6000

zookeeper.sync.time.ms=2000

zk配置文件:

zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir=D:/software/zookeeper-2181/data

dataLogDir=D:/software/zookeeper-2181/logs

clientPort=2181

server.0=192.168.1.102:2777:3777

server.1=192.168.1.102:2888:3888

server.2=192.168.1.102:2999:3999

用gradle替代maven常用配置

在生成的build.gradle文件中配置:
plugins {
    id 'java'
    id 'application'
    id 'war'
    id 'org.gretty' version '2.2.0'
    id 'com.gradle.build-scan' version '1.16'
    id 'org.springframework.boot' version '2.1.2.RELEASE'
    id "io.spring.dependency-management" version "1.0.6.RELEASE"
}

//基本配置
group 'com.df'
version '1.0-SNAPSHOT'
mainClassName = 'com.df.run.JobApplication'
sourceCompatibility = 1.8
targetCompatibility = 1.8

//Java compiler options
compileJava {
    options.incremental = true
    options.fork = true
    options.failOnError = false
    options.encoding = "UTF-8"
}

repositories {
    jcenter()
    mavenLocal()
    mavenCentral()
    maven {
        url "https://plugins.gradle.org/m2/"
    }
}

configurations {
    developmentOnly
    runtimeClasspath {
        extendsFrom developmentOnly
    }
}

dependencies {
    //spring boot
    implementation 'org.springframework.boot:spring-boot-parent:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-aop:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-data-jdbc:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-dependencies:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-test-autoconfigure:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-tools:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-cloud-connectors:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-websocket:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-reactor-netty:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-cache:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-groovy-templates:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-docs:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-actuator:2.1.2.RELEASE'  //spring boot监控工具
    annotationProcessor "org.springframework.boot:spring-boot-configuration-processor" //配置文件处理器
    developmentOnly("org.springframework.boot:spring-boot-devtools")

    //授权Oauth2模块
    //implementation 'org.springframework.boot:spring-boot-starter-security:2.1.2.RELEASE'
    //implementation 'org.springframework.security.oauth:spring-security-oauth2:2.1.2.RELEASE'
    //implementation 'org.springframework.security.oauth.boot:spring-security-oauth2-autoconfigure:2.1.2.RELEASE' //接口授权oauth2
    //implementation 'org.springframework.boot:spring-boot-starter-oauth2-client:2.1.2.RELEASE' //接口授权oauth2
    //implementation 'org.springframework.boot:spring-boot-starter-data-redis:2.1.2.RELEASE' //将token存储在redis中

    //配置shiro认证
    implementation 'com.github.theborakompanioni:thymeleaf-extras-shiro:2.0.0'
    implementation 'org.apache.shiro:shiro-core:1.4.0'
    implementation 'org.apache.shiro:shiro-spring:1.4.0'
    implementation 'org.apache.shiro:shiro-ehcache:1.4.0'
    implementation 'org.apache.shiro:shiro-web:1.4.0'
    implementation 'org.apache.shiro:shiro-quartz:1.4.0'

    //配置hibernate
  /*  implementation ('org.hibernate:hibernate-core:5.2.17.Final') {
        exclude group: 'org.slf4j',module: 'slf4j-api' //排除某一个库(slf4j)依赖
    }
    implementation 'org.hibernate:hibernate-entitymanager:4.3.11.Final'*/

    //配置mysql数据库
    implementation 'mysql:mysql-connector-java:8.0.13'  //mysql 6.0.5
    implementation 'com.zaxxer:HikariCP:3.3.0' //HikariCP高效数据库连接池
    implementation 'org.mybatis.spring.boot:mybatis-spring-boot-starter:1.3.2' //mybatis
    implementation 'com.github.pagehelper:pagehelper-spring-boot-starter:1.2.10'  //分页查询pagehelper版本1.2.10对应springboot2.1.0.RELEASE
    implementation 'com.alibaba:druid-spring-boot-starter:1.1.10'

    //Web相关
    implementation 'org.springframework.boot:spring-boot-starter-web:2.1.2.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-thymeleaf:2.1.2.RELEASE'
    implementation 'net.sourceforge.nekohtml:nekohtml:1.9.22'
    providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
    providedCompile 'javax.servlet:javax.servlet-api:3.1.0'

    //quartz定时器
    //implementation 'org.springframework.boot:spring-boot-starter-quartz:2.1.2.RELEASE'  //缺点:同一个task,如果前一个还没跑完后面一个就不会触发,不同的task也不能同时运行
    implementation 'org.quartz-scheduler:quartz:2.3.0'
    implementation 'org.quartz-scheduler:quartz-jobs:2.3.0'
    implementation 'org.springframework:spring-tx:5.1.4.RELEASE'
    implementation 'org.springframework:spring-context-support:5.1.4.RELEASE'
    implementation 'org.springframework.boot:spring-boot-starter-amqp:2.1.2.RELEASE' //RabbitMQ
    compile group: 'c3p0', name: 'c3p0', version: '0.9.1.2'  //在quartz定时器默认使用c3p0连接池
    compile group: 'org.springframework.boot', name: 'spring-boot-starter-velocity', version: '1.4.7.RELEASE' //velocity
    compile group: 'org.springframework', name: 'spring-context-support', version: '5.1.4.RELEASE'
    //compile group: 'org.springframework', name: 'spring-tx', version: '5.1.4.RELEASE'

    //配置H2数据库
    runtime group: 'com.h2database', name: 'h2', version: '1.4.197' //H2内嵌数据库
    

    //其它
    implementation 'com.alibaba:fastjson:1.2.54' //阿里fastjson
    implementation 'commons-io:commons-io:2.6' //commons-io
    implementation 'com.google.guava:guava:23.0' //guava
    implementation 'redis.clients:jedis:2.9.0'
    //implementation 'org.slf4j:slf4j-api:1.7.25' //slf4j
    implementation 'org.jsoup:jsoup:1.11.3'
    implementation 'commons-fileupload:commons-fileupload:1.2'
    implementation 'org.apache.poi:poi-ooxml:3.17'
    implementation 'com.github.penggle:kaptcha:2.3.2'
    implementation 'io.springfox:springfox-swagger2:2.7.0'
    implementation 'io.springfox:springfox-swagger-ui:2.7.0'
    implementation 'eu.bitwalker:UserAgentUtils:1.19'
    implementation 'com.github.oshi:oshi-core:3.9.1'

    components {
        withModule('org.springframework:spring-beans') {
            allVariants {
                withDependencyConstraints {
                    // Need to patch constraints because snakeyaml is an optional dependency
                    it.findAll { it.name == 'snakeyaml' }.each { it.version { strictly '1.19' } }
                }
            }
        }
    }

    //hadoop
    //implementation 'org.apache.hadoop:hadoop-client:2.7.3'
    //implementation 'org.apache.hadoop:hadoop-common:2.7.3'
    //implementation 'org.apache.hadoop:hadoop-hdfs:2.7.3'

    //scala
    implementation 'org.scala-lang:scala-library:2.11.12'
    
    //spark
    implementation 'org.apache.spark:spark-mllib_2.11:2.4.0'
    implementation 'org.apache.spark:spark-sql_2.11:2.4.0'
    implementation 'org.apache.spark:spark-graphx_2.11:2.4.0'
    implementation 'org.apache.spark:spark-launcher_2.11:2.4.0'
    implementation 'org.apache.spark:spark-catalyst_2.11:2.4.0'
    implementation 'org.apache.spark:spark-streaming_2.11:2.4.0'
    implementation 'org.apache.spark:spark-core_2.11:2.4.0'

    //单元测试
    testCompile 'org.mockito:mockito-core:2.7.19'
    testCompile group: 'junit', name: 'junit', version: '4.12'
    testCompile 'org.seleniumhq.selenium:selenium-java:3.3.1'
    testRuntime 'org.scala-lang.modules:scala-xml_2.11:1.1.0'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'

}

//打包
jar {
    enabled = true
    version = '0.0.1'
    manifest {
        attributes("Implementation-Title": "sparkjob",
                "Implementation-Version": version)
    }
}
//设置启动类
bootJar {
    mainClassName = 'com.df.run.JobApplication'
}

// buildScan
buildScan {
    // always accept the terms of service
    termsOfServiceUrl = 'https://gradle.com/terms-of-service'
    termsOfServiceAgree = 'yes'

    // always publish a build scan
    publishAlways()
}

Centos7上PHP集群session共享

1.修改 php-fpm.conf监听参数
vim /usr/local/php/etc/php-fpm.conf
1.1 代理接口服务器47.110.157.1上
listen = 172.16.231.137:9000

1.2代理接口服务器47.110.158.10上
listen = 172.16.231.136:9000

1.3检测php-fpm.conf文件配置准确性
/usr/local/php/sbin/php-fpm -t

1.4查看修改后的php-fpm.conf配置
grep -v ‘^$’ /usr/local/php/etc/php-fpm.conf | grep -v ‘^;’

1.5重启php-fpm服务
systemctl restart php-fpm
systemctl status php-fpm

2.配置nginx.conf文件
2.1在http节点内
vim /etc/nginx/nginx.conf
#代理接口服务器47.110.157.1和47.110.158.10上都增加
#设置置php负载均衡
upstream fastcgiserver {
server 172.16.231.137:9000;
server 172.16.231.136:9000;
}

2.2.修改具体项目修改fastcgi_pass地址
#下面以agyl项目为例
vim /etc/nginx/conf.d/home_agyl.conf
#修改内容如下所示
#astcgi_pass 127.0.0.1:9000;
fastcgi_pass fastcgiserver;

2.3重载nginx配置文件
/usr/sbin/nginx -t
/usr/sbin/nginx -s reload

3.修改php.ini参数
3.1创建共享session文件夹
mkdir -pv /data/php_session
cp -rf /var/lib/php/session/* /data/php_session/
chown -R nginx.nginx /data/php_session
chmod 777 -R /data/php_session

3.4#修改php.ini文件
vim /usr/local/php/etc/php.ini

#Redis共享方式,修改内容如下所示
session.save_handler = Redis
session.save_path = “tcp://192.168.5.114:6379” #Redis不需要密码验证
session.save_path = “tcp://192.168.5.114:6379?auth=password” #Redis需要密码验证

#NFS共享磁盘方式,修改内容如下所示
session.save_handler = files
session.save_path = “/data/php_session”

3.3检测php-fpm.conf文件配置准确性
/usr/local/php/sbin/php-fpm -t

3.4重启php-fpm服务
systemctl restart php-fpm
systemctl status php-fpm

3.5查看修改后的php-fpm.conf配置
grep -v ‘^$’ /usr/local/php/etc/php.ini | grep -v ‘^;’

如何在Centos7安装MyCat数据库中间件

Mycat官网:http://mycat.io/
官网下载地址:http://dl.mycat.io/
源码git地址:https://github.com/MyCATApache/Mycat-download

1.环境准备
需要安装jdk7+以上,本人安装的是jdk8;怎样安装jdk,在这里就不再详说了,大家可以网上查下度娘。

2.安装
mkdir -pv /home/sofrt
wget http://dl.mycat.io/1.6-RELEASE/Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz
tar zxvf Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz
mv mycat /usr/local/

3.配置
配置文件在/home/mycat/conf下
–bin 启动目录
–conf 配置文件存放配置文件:
–server.xml:是Mycat服务器参数调整和用户授权的配置文件。
–schema.xml:是逻辑库定义和表以及分片定义的配置文件。
–rule.xml: 是分片规则的配置文件,分片规则的具体一些参数信息单独存放为文件,也在这个目录下,配置文件修改需要重启MyCAT。
–log4j.xml: 日志存放在logs/log中,每天一个文件,日志的配置是在conf/log4j.xml中,根据自己的需要可以调整输出级别为debug debug级别下,会输出更多的信息,方便排查问题。
–autopartition-long.txt,partition-hash-int.txt,sequence_conf.properties, sequence_db_conf.properties 分片相关的id分片规则配置文件
–lib MyCAT自身的jar包或依赖的jar包的存放目录。
–logs MyCAT日志的存放目录。日志存放在logs/log中,每天一个文件

CentOS7下分布式系统GlusterFS安装配置

1.主机规划
192.168.1.52 gfs1
192.168.1.71 gfs2

2.配置ssh免密码登录
在两个节点上分别执行
1).生成密钥文件
ssh-keygen -t rsa
2).将公钥拷贝到对方
#在192.168.1.52 gfs1上执行
ssh-copy-id root@192.168.1.71
#在192.168.1.71 gfs2上执行
ssh-copy-id root@192.168.1.52
3).测试
#在192.168.1.52 gfs1上执行
ssh root@192.168.1.71
#在192.168.1.71 gfs2上执行
ssh root@192.168.1.52

3.安装
1).在node{1-2}上安装glusterfs服务器和客户端
yum install -y centos-release-gluster38 glusterfs glusterfs-server glusterfs-fuse
2).设置开机自启动并启动
systemctl enable glusterd
systemctl start glusterd
3).在gfs1-gfs2节点上配置整个GlusterFS集群,把各个节点加入到集群
vim /etc/hosts
修改内容如下:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.52 gfs1
192.168.1.71 gfs2
4).在任意一个节点上执行(添加对方为到集群)
如gfs1节点上执行
gluster peer probe gfs2
如果gfs2节点上即执行
gluster peer probe gfs1
5).在gfs{1-2}设置防火墙
systemctl start firewalld
systemctl enable firewalld
firewall-cmd –add-service=glusterfs –permanent
firewall-cmd –zone=public –add-port=24007-24008/udp –permanent
firewall-cmd –zone=public –add-port=24007-24008/tcp –permanent
firewall-cmd –zone=public –add-port=49152/tcp –permanent
firewall-cmd –reload

4.查看节点状态
gluster peer status

5.在gfs{1-2}上创建数据存储目录
mkdir -p /home/share/models

6.在gfs1上创建GlusterFS磁盘
注意:加上replica 2就是2个节点中,每个节点都要把数据存储一次,就是一个数据存储2份,每个节点一份
如果不加replica 2,就是2个节点的磁盘空间整合成一个硬盘
gluster volume create models replica 2 transport tcp gfs1:/home/share/models gfs2:/home/share/models force

7.创建/启动/停止/删除卷
gluster volume start models
gluster volume stop models
gluster volume delete models
注意,删除卷的前提是先停止卷

8.设置卷参数,如关闭只读,启动写,删除操作
gluster volume set models features.read-only Off
gluster volume set models nfs.trusted-write On

9.客户端以glusterfs方式挂载(此步重要,数据同步)
部署GlusterFS客户端并mount GlusterFS文件系统
注:ro:只读模式 rw:读写模式
1). 在gfs{1-2}上创建数据存储目录
mkdir -p /mnt/models
2). 在192.168.1.52 gfs1上执行
mount -t glusterfs -o rw gfs1:models /mnt/models/
3). 在192.168.1.71 gfs2上执行
mount -t glusterfs -o rw gfs2:models /mnt/models/
4).在gfs{1-2}上设置开机自动挂载卷
vim /etc/fstab
在192.168.1.52 gfs1上执行添加内容:
gfs1:models /mnt/models glusterfs defaults,_netdev 0 0
在72.168.30.71 gfs2上执行添加内容:
gfs2:models /mnt/models glusterfs defaults,_netdev 0 0

10.修改客户端只读为读写方式
1).检查是否有ro只读模式
mount
如果发现有ro,先umount后重新mount
umount /mnt/models
2).如果发现有提示“device is busy”,执行以下操作
在192.168.1.52 gfs1上执行
gluster volume stop models
umount /mnt/models
gluster volume start models
mount -t glusterfs -o rw gfs1:models /mnt/models/
在192.168.1.71 gfs2上执行
gluster volume stop models
umount /mnt/models
gluster volume start models
mount -t glusterfs -o rw gfs2:models /mnt/models/

Tomcat 9.0版本文件上传后无权限访问的问题

项目部署时用了Nginx+Tomcat9.0版本
      有同事反映图片上传后有时无法访问,报403 Forbidden 错误 ,这分明是权限问题导致的。上到图片物理路径,果然凡是通过Tomcat上传的图片,权限都比较低,上传时用户是Tomcat
前端用户访问时用的用户是nginx;但为什么用Tomcat7.0版本就没正常?后来查度娘和查看Tomcat官网发现,原来Tomcat在8.x以后的权限 UMASK=’0027′ 但在Tomcat7.x中是UMASK=’0022′
修改Tomcat的bin/catalina.sh
查找到UMASK=”0027″ 修改为 UMASK=”0022″ ;然后重启Tomcat 服务即可 systemctl restart tomcat