Loomaaiatalitaja

Zookeeperi server ja klient laiendavad sõlme andmete 1M suuruse piirangut

Zookeeper Server Client Expand Node Data 1m Size Limit



ZooKeeperi iga sõlme andmete vaikimisi piirang on 1M. Kui peate salvestama rohkem kui 1 miljonit andmeid, peate muutma parameetrit jute.maxbuffer. Kõigepealt vaatame selle parameetri tähendust zookeeperis.

jute.maxbuffer ： Vaikeväärtus on baitides 1048575, mida kasutatakse ühe andmesõlme (ZNode) salvestatava maksimaalse andmesuuruse konfigureerimiseks. Tuleb märkida, et selle parameetri muutmisel peate selle jõustumiseks määrama kõigile zookeeperi klastri serveritele ja klientidele.



Järgmisena vaatame, kuidas seda parameetrit seada.



Teenindusterminal:
Kui server seda parameetrit ei muuda, loobub liiga suurte andmete otse kirjutamine vea: org.apache.zookeeper.KeeperException $ ConnectionLossException: KeeperErrorCode = ConnectionLoss for / test
pilt



  • Lahendus:

    zkServer.sh Lisatud -Djute.maxbuffer konfiguratsioon, siin on näitena 10M, konkreetset suurust tuleb muuta vastavalt tegelikule olukorrale (ZOO_USER_CFG on märksõna muudetud osa):

    #!/usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the 'License') you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an 'AS IS' BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # If this scripted is run out of /usr/bin or some other system bin directory # it should be linked to and not copied. Things like java jar files are found # relative to the canonical path of this script. # # use POSTIX interface, symlink is followed automatically ZOOBIN='${BASH_SOURCE-$0}' ZOOBIN='$(dirname '${ZOOBIN}')' ZOOBINDIR='$(cd '${ZOOBIN}' pwd)' if [ -e '$ZOOBIN/../libexec/zkEnv.sh' ] then . '$ZOOBINDIR/../libexec/zkEnv.sh' else . '$ZOOBINDIR/zkEnv.sh' fi # See the following page for extensive details on setting # up the JVM to accept JMX remote management: # http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html # by default we allow local JMX connections ZOO_USER_CFG='-Djute.maxbuffer=10240000' if [ 'x$JMXLOCALONLY' = 'x' ] then JMXLOCALONLY=false fi if [ 'x$JMXDISABLE' = 'x' ] || [ '$JMXDISABLE' = 'false' ] then echo 'ZooKeeper JMX enabled by default' >&2 if [ 'x$JMXPORT' = 'x' ] then # for some reason these two options are necessary on jdk6 on Ubuntu # accord to the docs they are not necessary, but otw jconsole cannot # do a local attach ZOOMAIN='-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=$JMXLOCALONLY org.apache.zookeeper.server.quorum.QuorumPeerMain' else if [ 'x$JMXAUTH' = 'x' ] then JMXAUTH=false fi if [ 'x$JMXSSL' = 'x' ] then JMXSSL=false fi if [ 'x$JMXLOG4J' = 'x' ] then JMXLOG4J=true fi echo 'ZooKeeper remote JMX Port set to $JMXPORT' >&2 echo 'ZooKeeper remote JMX authenticate set to $JMXAUTH' >&2 echo 'ZooKeeper remote JMX ssl set to $JMXSSL' >&2 echo 'ZooKeeper remote JMX log4j set to $JMXLOG4J' >&2 ZOOMAIN='-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=$JMXPORT -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH -Dcom.sun.management.jmxremote.ssl=$JMXSSL -Dzookeeper.jmx.log4j.disable=$JMXLOG4J org.apache.zookeeper.server.quorum.QuorumPeerMain' fi else echo 'JMX disabled by user request' >&2 ZOOMAIN='org.apache.zookeeper.server.quorum.QuorumPeerMain' fi if [ 'x$SERVER_JVMFLAGS' != 'x' ] then JVMFLAGS='$SERVER_JVMFLAGS $JVMFLAGS' fi if [ 'x$2' != 'x' ] then ZOOCFG='$ZOOCFGDIR/$2' fi # if we give a more complicated path to the config, don't screw around in $ZOOCFGDIR if [ 'x$(dirname '$ZOOCFG')' != 'x$ZOOCFGDIR' ] then ZOOCFG='$2' fi if $cygwin then ZOOCFG=`cygpath -wp '$ZOOCFG'` # cygwin has a 'kill' in the shell itself, gets confused KILL=/bin/kill else KILL=kill fi echo 'Using config: $ZOOCFG' >&2 case '$OSTYPE' in *solaris*) GREP=/usr/xpg4/bin/grep *) GREP=grep esac if [ -z '$ZOOPIDFILE' ] then ZOO_DATADIR='$($GREP '^[[:space:]]*dataDir' '$ZOOCFG' | sed -e 's/.*=//')' if [ ! -d '$ZOO_DATADIR' ] then mkdir -p '$ZOO_DATADIR' fi ZOOPIDFILE='$ZOO_DATADIR/zookeeper_server.pid' else # ensure it exists, otw stop will fail mkdir -p '$(dirname '$ZOOPIDFILE')' fi if [ ! -w '$ZOO_LOG_DIR' ] then mkdir -p '$ZOO_LOG_DIR' fi _ZOO_DAEMON_OUT='$ZOO_LOG_DIR/zookeeper.out' case $1 in start) echo -n 'Starting zookeeper ... ' if [ -f '$ZOOPIDFILE' ] then if kill -0 `cat '$ZOOPIDFILE'` > /dev/null 2>&1 then echo $command already running as process `cat '$ZOOPIDFILE'`. exit 0 fi fi nohup '$JAVA' '$ZOO_USER_CFG' '-Dzookeeper.log.dir=${ZOO_LOG_DIR}' '-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}' -cp '$CLASSPATH' $JVMFLAGS $ZOOMAIN '$ZOOCFG' > '$_ZOO_DAEMON_OUT' 2>&1 '$ZOOPIDFILE' *) /bin/echo -n $! > '$ZOOPIDFILE' esac if [ $? -eq 0 ] then sleep 1 echo STARTED else echo FAILED TO WRITE PID exit 1 fi else echo SERVER DID NOT START exit 1 fi start-foreground) ZOO_CMD=(exec '$JAVA') if [ '${ZOO_NOEXEC}' != '' ] then ZOO_CMD=('$JAVA') fi '${ZOO_CMD[@]}' '$ZOO_USER_CFG' '-Dzookeeper.log.dir=${ZOO_LOG_DIR}' '-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}' -cp '$CLASSPATH' $JVMFLAGS $ZOOMAIN '$ZOOCFG' print-cmd) echo ''$JAVA' -Dzookeeper.log.dir='${ZOO_LOG_DIR}' -Dzookeeper.root.logger='${ZOO_LOG4J_PROP}' -cp '$CLASSPATH' $JVMFLAGS $ZOOMAIN '$ZOOCFG' > '$_ZOO_DAEMON_OUT' 2>&1 /dev/null | $GREP Mode` if [ 'x$STAT' = 'x' ] then echo 'Error contacting service. It is probably not running.' exit 1 else echo $STAT exit 0 fi *) echo 'Usage: $0 restart' >&2 esac

Klient:
Pärast zkServer.sh muutmist zookeeperi taaskäivitamiseks saab kirjutada rohkem kui 1 miljonit andmeid, kuid kui lugesin kliendilt, sisestasin valesti ka java.io.IOException: ebamõistlik pikkus = 1560001
pilt



  • zkCli.sh

    Järgmisena peame muutma kliendi konfiguratsiooni. Kui kasutate andmete lugemiseks otse zookeeperiga kaasas olevat zkCli.sh-d, peate zkCli.sh-i lisama konfiguratsiooni -Djute.maxbuffer. Siin kasutatakse näitena 10M. Muudetud vastavalt tegelikule olukorrale (ZOO_USER_CFG on märksõna muudetud osa):

    #!/usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the 'License') you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an 'AS IS' BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This script cleans up old transaction logs and snapshots # # # If this scripted is run out of /usr/bin or some other system bin directory # it should be linked to and not copied. Things like java jar files are found # relative to the canonical path of this script. # # use POSTIX interface, symlink is followed automatically ZOOBIN='${BASH_SOURCE-$0}' ZOOBIN='$(dirname '${ZOOBIN}')' ZOOBINDIR='$(cd '${ZOOBIN}' pwd)' ZOO_USER_CFG='-Djute.maxbuffer=10240000' if [ -e '$ZOOBIN/../libexec/zkEnv.sh' ] then . '$ZOOBINDIR'/../libexec/zkEnv.sh else . '$ZOOBINDIR'/zkEnv.sh fi '$JAVA' '$ZOO_USER_CFG' '-Dzookeeper.log.dir=${ZOO_LOG_DIR}' '-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}' -cp '$CLASSPATH' $CLIENT_JVMFLAGS $JVMFLAGS org.apache.zookeeper.ZooKeeperMain 'root@xxxxx'

  • Koodiplokis

Kui seda loetakse koodist, siis tuleks modifitseeritud atribuudi jute.maxbuffer väärtus ümber kirjutada, ma kasutan Java-koodis paketti zookeeper-3.4.10.jar, vaadake kõigepealt, kuidas see pakett on Initialize the jute.maxbuffer value
pilt
maxbuffer loeb kõigepealt süsteemi omaduselt jute.maxbuffer väärtuse (System.getProperty ()) või määrab 1048575, kui see pole saadaval.
Pärast maxbufferi väärtusloogika tundmist saame välja kirjutada õige ravimi. Enne zookeeperi kliendi initsialiseerimist lisage süsteemi atribuudile muudetud jute.maxbufferi võtmeväärtus.
pilt
Lõpuks ei taha ma loomaaiapidajasse liiga palju andmeid salvestada, mistõttu jute.maxbuffer väärtus tuleb kindlaks määrata vastavalt konkreetsele olukorrale.

viide:
Paxosest kuni ZooKeeperini
https://blog.csdn.net/hopingwhite/article/details/8255979