René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"ERROR Failed to get topics from kafka zk: {} (com.uber.stream.kafka.mirrormaker.common.core.KafkaBrokerTopicObserver)","articleBody":"Hi there,\r\n\r\nI am doing a little poc to learn more about uReplicator and how it replicates the topics and messages\r\n\r\nI am new to kafka and uber uReplicator, i am trying to deploy federated uReplicator. I am not able to configure it properly i guess. There are certain things which aren't making sense. \r\n1) i have cloned all the git repos (master, 0.1.0, 1.0 ) but after i use mvn clean package, the jar files its creating have some different versions than i see in the quick start guide commands, especially the start commands for federated one.\r\n2) if i am correct, in the federated mode we can have more than one src clusters and more than one dest clusters right?\r\n3) i have learned that one uReplicator can replicate between one src and one dest cluster. But this is for non-federated. In the federated mode we can have multiple src clusters and multiple target clusters. Is that correct?\r\n4) If i am correct, uReplicator does not replicate topic configurations unlike mm2 from kafka, it does not replicate topic offsets?\r\n\r\nIn my case, i want to start uReplicator in federated mode so that i can add more clusters to it later, i want all the topics to replicate from source machine (single node kafka\u0026zk) to target (single node kafka\u0026zk). I started federated uReplicator on my target node. So i am starting manager, controller and worker on the same machine (target), so the machine i am starting uRep-federated is the destination/target also for replication, but i am seeing this error when i start manager. \r\n![image](https://user-images.githubusercontent.com/60720179/129999445-66dc8e9d-b165-4ce7-8557-eaa3f46593bf.png)\r\n\r\nThe manager is running, its not stopping but throwing this error too and when i create a topic and produce some messages in the source machine, i am not able to consume those messages from the target machine.\r\n\r\nHere are my logs:\r\n\r\nmanager:\r\n![image](https://user-images.githubusercontent.com/60720179/130000306-7f28aed5-3e07-404c-8502-b0cac9bcf410.png)\r\n\r\ncontroller:\r\n![image](https://user-images.githubusercontent.com/60720179/130000528-3bf08f2f-d05f-4ebb-9529-54488ab143d5.png)\r\n\r\nWorker:\r\n![image](https://user-images.githubusercontent.com/60720179/130000600-43a2e09e-728d-4818-942c-168728e16025.png)\r\n\r\nHere are my configuration files:\r\nclusters.properties\r\n![image](https://user-images.githubusercontent.com/60720179/130000828-9522ce6c-c785-4662-a9f6-5373a6f3d4b5.png)\r\n\r\nconsumer.properties:\r\n![image](https://user-images.githubusercontent.com/60720179/130001036-4fa69708-9036-4748-937b-479511c0bb4d.png)\r\n\r\nproducer.properties:\r\n![image](https://user-images.githubusercontent.com/60720179/130001144-f2d4762b-150c-4d74-ac9c-f6982dacfe07.png)\r\n\r\nhelix.properties:\r\n![image](https://user-images.githubusercontent.com/60720179/130001232-a1bfc108-38a6-4357-b089-0c871851f192.png)\r\n\r\nmy start commands:\r\n\r\nmanager:\r\njava -Dlog4j.configuration=file:config/tools-log4j.properties -Xms3g -Xmx3g -Xmn512m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+PrintCommandLineFlags -XX:CMSInitiatingOccupancyFraction=80 -XX:SurvivorRatio=2 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -Xloggc:./logs/gc-ureplicator-manager.log -server -cp uReplicator-Manager/target/uReplicator-Manager-2.0.2-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.kafka.mirrormaker.manager.ManagerStarter -config config/clusters.properties -srcClusters cluster1 -destClusters cluster3 -enableRebalance false -zookeeper localhost:2181 -managerPort 9000 -deployment c1-c2 -env dc1.c1-c2 -instanceId 100 -graphiteHost 127.0.0.1 -graphitePort 4756 -workloadRefreshPeriodInSeconds 300 -initMaxNumPartitionsPerRoute 1500 -maxNumPartitionsPerRoute 2000 -initMaxNumWorkersPerRoute 10 -maxNumWorkersPerRoute 80\r\n\r\ncontroller:\r\njava -Dlog4j.configuration=file:config/tools-log4j.properties -Xms3g -Xmx3g -Xmn512m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+PrintCommandLineFlags -XX:CMSInitiatingOccupancyFraction=80 -XX:SurvivorRatio=2 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -Xloggc:./logs/gc-ureplicator-controller.log -server -cp uReplicator-Controller/target/uReplicator-Controller-2.0.2-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.kafka.mirrormaker.controller.ControllerStarter -config config/clusters.properties -srcClusters cluster1 -destClusters cluster3 -enableFederated true -deploymentName c1-c2 -env dc1.c1-c2 -mode customized -zookeeper localhost:2181 -port 9100 -instanceId 1 -hostname swarm1 -enableAutoWhitelist true -enableAutoTopicExpansion true -autoRebalanceDelayInSeconds 120 -autoRebalancePeriodInSeconds 120 -autoRebalanceMinIntervalInSeconds 600 -autoRebalanceMinLagTimeInSeconds 900 -autoRebalanceMinLagOffset 100000 -autoRebalanceMaxOffsetInfoValidInSeconds 1800 -autoRebalanceWorkloadRatioThreshold 1.5 -maxDedicatedLaggingInstancesRatio 0.2 -maxStuckPartitionMovements 3 -moveStuckPartitionAfterMinutes 20 -workloadRefreshPeriodInSeconds 300 -patternToExcludeTopics ^__.*  -enableSrcKafkaValidation true -consumerCommitZkPath \"\" -maxWorkingInstances 0 -autoRebalanceDelayInSeconds 120 -refreshTimeInSeconds 600 -initWaitTimeInSeconds 120 -numOffsetThread 10 -blockingQueueSize 30000 -offsetRefreshIntervalInSec 300 -backUpToGit false -localBackupFilePath ./logs/ureplicator-controller -localGitRepoClonePath ./logs/ureplicator-controller-bkp\r\n\r\nworker:\r\njava -Dlog4j.configuration=file:config/test-log4j.properties -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=45 -verbose:gc -Xmx1g -Xms1g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:gc-ureplicator-worker.log -server -cp uReplicator-Worker/target/uReplicator-Worker-2.0.2-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.ureplicator.worker.WorkerStarter -federated_enabled true -cluster_config config/clusters.properties -consumer_config config/consumer.properties -producer_config config/producer.properties -helix_config config/helix.properties\r\n\r\nI am really sorry for my novice questions. But i would really appreciate if someone can explain me what's wrong with my setup.\r\n\r\nregards,\r\nAbdul Hai Mohammed\r\n ","author":{"url":"https://github.com/amohammed3","@type":"Person","name":"amohammed3"},"datePublished":"2021-08-19T03:15:49.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/348/uReplicator/issues/348"}