RabbitMQ for PCF - Pivotal · Documentation for RabbitMQ for PCF v1.7 and earlier only describes a pre-provisioned service. What are On-Demand Instances In RabbitMQ for PCF versions

RabbitMQforPCF®

Documentation

Version1.12

Published:5March2019

©2019PivotalSoftware,Inc.AllRightsReserved.

236

16212428415657646869737678808297

100105107

TableofContents

TableofContentsRabbitMQforPCFRabbitMQ®forPCFReleaseNotesUnlockingthePowerofOn-DemandRabbitMQforPCFOn-DemandServiceArchitectureDeployingtheRabbitMQPre-ProvisionedServiceInstallingandConfiguringtheOn-DemandServiceInstallingandConfiguringthePre-ProvisionedServiceSmokeTestsMonitoringandKPIsforPre-ProvisionedRabbitMQforPCFSettingLimitsforOn-DemandServiceInstancesControllingAccesstoServicePlansbyOrgIsolatingClusterswiththeRabbitMQforPCFReplicatorSettingDefaultPoliciesfortheRabbitMQServiceUsingtheRabbitMQManagementDashboardClusteringandNetworkPartitionsUpgradingRabbitMQforPCFTroubleshootingandFAQsforOn-DemandRabbitMQforPCFFrequentlyAskedQuestionsforPre-ProvisionedRabbitMQforPCFUsingOn-DemandRabbitMQforPCFRabbitMQEnvironmentVariablesTroubleshootingInstances

©CopyrightPivotalSoftwareInc,2013-2019 2 1.12

RabbitMQforPCF

AboutRabbitMQforPCFRabbitMQforPivotalCloudFoundry(PCF)enablesPCFappdeveloperstoprovisionandusetheRabbitMQmessagebrokerwithasinglecommand.

RabbitMQforPCFv1.8andlatersupporttwotypesofservice,anon-demandserviceandapre-provisionedservice.Thistablesummarizesthemaindifferencesbetweenthetwo:

AvailableSince VMsitRunsOn HowVMsareCreated MetricsName

Prefix

On-DemandService v1.8

DedicatedVMthatservesasingleserviceinstance.Seethistopicfordetails.

PCFcreateseachVMon-demandwhenappdevelopercreatesserviceinstance

p.rabbitmq

(withadot)

Pre-ProvisionedService

v1.2Multi-tenantVMssharedbyappsacrossPCFdeployment

PCFcreatesallVMswhenoperatordeploysorupdatesservice

p-rabbitmq

(withadash)

ThisRabbitMQforPCFdocumentationdescribesbothservicetypes.DocumentationforRabbitMQforPCFv1.7andearlieronlydescribesapre-provisionedservice.

WhatareOn-DemandInstancesInRabbitMQforPCFversionsbeforev1.8.0,theRabbitMQserviceinstancescorrespondtoauniqueRabbitMQvhostonthemulti-tenantRabbitMQcluster.RabbitMQforPCFv1.8.0introducedOn-DemandBroker(ODB) support.Thatmeansthatanew,single-tenant,clustercanbecreatedanddedicatedtoasingleapp.

Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCFandOn-DemandServiceArchitecture.

AboutRabbitMQRabbitMQisafastanddependableopen-sourcemessageserver,whichsupportsawiderangeofusecasesincludingreliableintegration,content-basedroutingandglobaldatadelivery,andhigh-volumemonitoringanddataingestion.

Emergingasthedefactostandardforcloudmessaging,RabbitMQisusedforefficientcommunicationbetweenservers,appsanddevices,andcreateslastingvaluebyenablingrapiddevelopmentofmoderndecentralizedappanddataarchitecturesthatcanscalewithyourbusinessneeds.

ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutRabbitMQforPCF.

Element Details

Version v1.12.15

Releasedate December28,2018

Softwarecomponentversions RabbitMQOSSv3.6.16andRabbitMQOSSv3.7.9

CompatibleOpsManagerversions v2.1.xandv2.2.x

CompatiblePivotalApplicationServiceversions v2.1.xandv2.2.x

IaaSsupport AWS,Azure,GCP,OpenStack,andvSphere

IPsecsupport No

Note:RabbitMQforPCFv1.12isnolongersupported.Thesupportperiodforv1.12hasexpired.Tostayup-to-datewiththelatestsoftwareandsecurityupdates,upgradetoasupportedversion.


https://docs.pivotal.io/svc-sdk/odb/index.html

Features

On-DemandCreateupto5differenton-demandRabbitMQplanswhichcanbeprovisionedthroughthemarketplace

Choosewhetheraplanhasone,three,five,orsevennodes

Defaultresourcesizesinplanstoguideselection

Morecontroloverwhichorgsandspaceshavevisibilityofeachconfiguredplan

Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding

Managementdashboardaccesstoappdevelopers

Deploymentintoanavailabilityzonespecifiedbytheplan

AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seereleasenotesfordowntimerequirements)

RabbitMQSyslogforwardingconfigurationinheritedfromthepre-provisionedconfiguration

RabbitMQmetricsareexposedontheFirehose

Runsmoketestsforon-demandplansonplan1

ErrandsarerunoncolocatedVMstodecreasedeploymenttimes

Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCF.

Pre-ProvisionedProvisionaninstanceoftheRabbitMQservice,whichcorrespondstoauniqueRabbitMQvhost(virtualhost)

Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding

ManagementdashboardaccesstoPCFOperatorsandappdevelopers

Deploymentacrossmultipleavailabilityzones,withnodesstripedacrosstheAZsautomatically

EnableSSL(SecureSocketsLayer)fortheAMQP,MQTT,STOMPprotocols

HAProxyloadbalanceracrossallnodestobalanceconnections

Pluginconfigurationcanbeeasilychangedatanytimeandtheclusterredeployedandupdated

Theclustertopologycanbechangedandeasilyscaledout

AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seeDowntimeWhenUpgradingfordowntimerequirements)

ConfiguretheendpointfortheRabbitMQSyslog

RabbitMQandHAProxymetricsareexposedonthefirehose

Syslogforwardingonbydefault

ErrandsarerunoncolocatedVMstodecreasedeploymenttimes

ReleaseNotesandKnownIssuesCheckthereleasenotesforyourreleaseversionforimportantinformationandknownissues.Toseereleasenotesforanotherversion,selecttheversionfromthedropdownlistatthetopofthepage.

RabbitMQforPCFandOtherPCFServicesSomePCFservicesofferon-demandserviceplans.Theseplansletdevelopersprovisionserviceinstanceswhentheywant.

Thesecontrastwiththemorecommonpre-provisionedserviceplans,whichrequireoperatorstoprovisiontheserviceinstancesduringinstallationandconfigurationthroughtheservicetileUI.

ThefollowingPCFservicesofferon-demandserviceplans:

MySQLforPCFv2.0andlater

RabbitMQforPCF


RedisforPCF

PivotalCloudCache(PCC)

Theseservicespackageanddelivertheiron-demandserviceofferingsdifferently.Forexample,someservices,likeRedisforPCF,haveonetile,andyouconfigurethetiledifferentlydependingonwhetheryouwanton-demandserviceplansorpre-provisionedserviceplans.

Forotherservices,likePCCandMySQLforPCF,onlyon-demandserviceplansareavailable.

ThefollowingtablelistsandcontraststhedifferentwaysthatPCFservicespackageon-demandandpre-provisionedserviceofferings.

PCFservicetile Standaloneproductrelatedtotheservice Versionssupportingondemand Versionssupportingpre-provisioned

RabbitMQforPCF PivotalRabbitMQ v1.8andlater Allversions

RedisforPCF Redis v1.8andlater Allversions

MySQLforPCF MySQL v2.x NA

PCC PivotalGemFire Allversions NA

Pleaseprovideanybugs,featurerequests,orquestionstothePCFFeedbacklist.


mailto:[email protected]

RabbitMQ®forPCFReleaseNotes

UpgradetotheLatestVersionPivotalrecommendsthatyouupgradetothelatestversionofyourcurrentminorline,thenupgradetothelatestavailableversionofthenewminorline.Forexample,ifyouuseanolderv1.11.xversion,upgradetothelatestv1.11.xversionbeforeupgradingtothelatestv1.12.xversion.

Forproductversionsandupgradepaths,seetheProductCompatibilityMatrix .

UseRabbitMQv3.7On-DemandPlansRabbitMQv3.7plansareavailable.EnsurethatallappsusetheRabbitMQv3.7on-demandplansinsteadoftheRabbitMQv3.6on-demandplans.

Formoreinformation,seethefollowing:

AboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7

TheANNRabbitMQ3.6.xsupporttimeline postontheRabbitMQusersforum

v1.12.15ReleaseDate:December28,2018

FeaturesNewfeaturesandchangesinthisrelease:

Requiresstemcell 3586.65

ScriptstodropandrestoreAMQP(S)traffic.Formoreinformation,seeDropandRestoreAMQP(S)TraffictoaRabbitMQInstance.

FixedIssuesThisreleasefixesthefollowingissue:

InRabbitMQforPCFv1.12.14andv1.12.13,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.

KnownIssuesThisreleasehasthefollowingissues:

[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .

IfyourelyonthetiletogeneratethecookiebyleavingtheErlangcookiefieldblank,seeSecurityIssuewiththeTileGeneratedErlangCookie.IfyouhavesettheErlangCookieexplicitlyinthetileconfiguration,youarenotaffectedbythisissue.

ThereisanissuewithupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.2.5andearlier.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1,oronPCFv2.2.6orlater.

Note:PivotalnolongerrecommendswaitingforSpringCloudServices(SCS)v3.0,wherethedependencyonpre-provisionedRabbitMQforPCFwillberemoved,beforeupgradingfromRabbitMQforPCFv1.12tov1.13.ThisisduetotheriskofgoingoutofsupportonyourRabbitMQforPCFtile,PivotalApplicationService(PAS),andothercomponents.

WARNING:PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1oronPCFv2.2.6orlater.


http://docs.pivotal.io/compatibility-matrix.pdf

https://groups.google.com/forum/#!msg/rabbitmq-users/kXkI-f3pgEw/UFowJIK4BQAJ

https://network.pivotal.io/products/stemcells

https://pivotal.io/security/cve-2018-1279

ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Changingnetworksand/orIPaddressesforthe RabbitMQServer jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.

PackagesOSSRabbitMQv3.6.16

OSSRabbitMQv3.7.9

Erlangv20.3.8.15

HAProxyv1.6.13

v1.12.14ReleaseDate:December20,2018

FeaturesRequiresstemcell 3586.60

Setswapto1GBforRabbitMQnodes

SupportstheLogCachecfCLIpluginthatenablesdeveloperstoaccesslogsforanon-demandserviceinstanceusingthecommand cf tail .

Formoreinformationaboutthisfeature,seeAccessRabbitMQMetricsforOn-DemandServiceInstances.

KnownIssues[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .


IfyouupgradetoRabbitMQforPCFtothisversion,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.Thisissuedoesnotaffectnewinstallations.






OSSRabbitMQv3.7.9

Erlangv20.3.8.15

HAProxyv1.6.13


https://docs.pivotal.io/rabbitmq-cf/1-12/changing-ips.html

https://docs.pivotal.io/pivotalcf/customizing/managing_errands.html





v1.12.13ReleaseDate:November22,2018

SecurityFixesThisreleaseincludesthefollowingsecurityfix:

CriticalCVE-2018-15759:OnDemandServicesSDKTimingAttackVulnerability

FeaturesAddsmutualTLSbetweenservicemetricsandtheloggregatorsystem

RemovescredentialsfromthedashboardURLinthepre-provisionedservice



IfyouupgradetoRabbitMQforPCFtothisversion,youcannotseeRabbitMQmetricsontheLoggregatorFirehose.Thisissuedoesnotaffectnewinstallations.






OSSRabbitMQv3.7.8

Erlangv20.3.8.10

HAProxyv1.6.13

v1.12.12ReleaseDate:November1,2018


Smoketestsarenowmoreresilientwhenusingexternalloadbalancers.

KnownIssues







[SecurityIssue]ThemethodforgeneratingtheErlangCookieisnotsecure.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .







OSSRabbitMQv3.7.8

Erlangv20.3.8.9

HAProxyv1.6.13

v1.12.10ReleaseDate:October5,2018


RabbitMQ3.7updatedto3.7.8

Erlangupdatedtov20.3.8.9








OSSRabbitMQv3.7.8

Erlangv20.3.8.9









HAProxyv1.6.13

v1.12.9ReleaseDate:September13,2018


Smoketestsnowwaitupto5minutesforatestapptodeploy.








OSSRabbitMQv3.7.7

Erlangv20.3.8.6

HAProxyv1.6.13

v1.12.8ReleaseDate:August20,2018

FeaturesRequiresstemcell3468.64



Thereisanissuewithupgradingfromv1.12.xtov1.13.xonPCFv2.2.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1.






https://github.com/cloudfoundry/bosh-linux-stemcell-builder/releases/tag/stable-3468.64


YoucannotupgradefromRabbitMQforPCFv.12toRabbitMQforPCFv1.13onPCFv2.2.0-2.2.5.YoumustupgradeonPCFv2.1oronPCFv2.2.6orlater.





OSSRabbitMQv3.7.7

Erlangv20.3.8.1

HAProxyv1.6.13

v1.12.7ReleaseDate:July13,2018


FixesbuginthesmoketestswhenCredHubisenabled



Thereisanissuewithupgradingfromv1.12.xtov1.13.xonPCFv2.2.PivotalrecommendsupgradingfromRabbitMQforPCFv1.12.xtov1.13.xonPCFv2.1.

YoucannotupgradefromRabbitMQforPCFv.12toRabbitMQforPCFv1.13onPCFv2.2.0-2.2.5.YoumustupgradeonPCFv2.1oronPCFv2.2.6orlater.





OSSRabbitMQv3.7.7

Erlangv20.3.8.1

HAProxyv1.6.13

v1.12.6







ReleaseDate:June28,2018


Requiresstemcell3468.51 .

UpdatesOSSRabbitMQServerandErlangpackages







OSSRabbitMQv3.7.6

Erlangv20.3.8.1

HAProxyv1.6.13

v1.12.5ReleaseDate:June5,2018



cf-cligolangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .

FixedIssuesThisreleasefixesthefollowingissue:

Smoketestssometimesfailedbecauseoflowtime-outvalues.








https://github.com/golang/go/issues/24590







OSSRabbitMQv3.7.4

Erlangv19.3.6.4

HAProxyv1.6.13

v1.12.4ReleaseDate:May16,2018



Timestampsaddedtologentriesthatdidnothavethem.

Golangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .



Smoketestsmightfailbecausetime-outvaluesinthesmoketestsaretoolow.Formoreinformationabouttroubleshootingsmoketests,seeSmokeTests.





OSSRabbitMQv3.7.4

Erlangv19.3.6.4

HAProxyv1.6.13





https://github.com/golang/go/issues/24590




v1.12.3ReleaseDate:April19,2018


On-Demand

ThefollowingerrandsarecolocatedwiththeirrespectivebrokertodecreaseerrandruntimeandVMfootprint:

Register/DeregisterOnDemandServiceBrokerOnDemandInstanceSmokeTestsUpgradeAllServiceInstancesDeleteAllServiceInstances

Formoreinformationabouterrands,seeErrands .

[Beta]ServiceSharingallowsRabbitMQdeploymentstobesharedacrossCloudFoundryspacesandorgs.Formoreinformationaboutsharingserviceinstances,seeCloudFoundryDocumentation .

[Beta]Supportforsecurecredentials(CredHub).Credentialsarestoredinacentralrepositoryandarerestrictedtocomponentsthatactuallyneedthem.

FormoreinformationaboutCredHub,seeCloudFoundryDocumentation .

Pre-Provisioned

ThefollowingerrandsarecolocatedwiththeirrespectivebrokertodecreaseerrandruntimeandVMfootprint:

BrokerRegistrar/DeregistrarSmokeTestserrands

Formoreinformationabouterrands,seeErrands .







OSSRabbitMQv3.7.4

Erlangv19.3.6.4

HAProxyv1.6.13

ViewReleaseNotesforAnotherVersionToviewthereleasenotesforanotherproductversion,selecttheversionfromthedrop-downlistatthetopofthispage.


http://docs.pivotal.io/tiledev/tile-errands.html

https://docs.cloudfoundry.org/devguide/services/sharing-instances.html

https://docs.cloudfoundry.org/credhub/index.html






UnlockingthePowerofOn-DemandRabbitMQforPCF

IntroductionRabbitMQforPivotalCloudFoundry(PCF)respondstothedemandsofPCFoperatorstoofferaRabbitMQon-demandclusterfortheirapplicationteams,inadditiontotheexistingsingle-nodeon-demandplan.Theon-demandclusterplanisaimedatworkloadsthatrequirethesameresiliencerequirementsasthePre-Provisionedoffering,butalsorequiretheirworkloadsbeisolated.

TheplatformoperationsteamcannowconfigureaRabbitMQforPCFclustertomeettheirbusinessrequirementsandempowerappdevelopmentteamstoself-servetheirownRabbitMQcluster.

RabbitMQforPCFalsoprovidessmoketestsfortheon-demandplanssothatoperationsteamscanvalidatetheappdeveloperworkflowforon-demandservices.SeeDedicatedInstanceSmokeTestProcess.

PlatformoperatorscannowoffertheirappdevelopersthreetypesofRabbitMQforPCFserviceplans:

Pre-provisioned—Forlighttomoderatemessagingneeds,thisserviceisfullyoperatedandmanagedbyplatformoperatorsasaservice.

On-demandsinglenode—Forapplicationteamsrequiringgreaterisolationthanprovidedbythepre-provisionedapproach.Appdevelopmentteamscanhavefullaccesstotheirownmessagebrokertoadapttheruntimeparameterstotheirrequirements.Formoreinformationontheseparameters,seeParametersandPolicies intheRabbitMQdocumentation.

On-demandcluster—Foranincreasedlevelofmessageresilienceandclusteravailability,aswellasthebenefitsofworkloadisolationmentionedabove.

Thistopicexplainshowtobenefitfromthetwoon-demandplansabove.

Forinformationaboutthepre-provisionedplan,seeDeployingtheRabbitMQPre-ProvisionedService.Forinformationonusingpre-provisionedplanstoisolateworkloads,seeCreatingIsolationwiththeTileReplicator.

On-DemandSingleNodePlanUsingRabbitMQ3.7Thisplanisdesignedtobesimpletoconfigure,deploy,anduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingopensourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.

Thisplancansuithigh-performanceworkloadsrequiringmessagingresilienceandasynchronousmessagingreplication.RabbitMQcopiesmessagestodiskforresilienceandallowsasynchronousmessagingreplicationthroughtheRabbitMQFederationplug-in.

Thisplanoffers:

FastaccesstoanisolatedinstanceofRabbitMQscopedfortheapplicationteams

OrgandSpaceAdministratoraccesstotheRabbitMQManagementUIsoapplicationteamscanhavefullcontroloverthenode

Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes

MessageresilienceprovidedthroughRabbitMQexchange,queueFederation,andShovelplugins.

ToBeDeprecated:On-DemandSingleNodePlanUsingRabbitMQ3.6Thisplanprovidesthesamebenefitsasdescribedabove,butusesRabbitMQ3.6forbackwardscompatibility.

RabbitMQ3.6isreachingendofsupport.ConsidermovingtoRabbitMQ3.7assoonaspossible.FormoreinformationaboutsupportforRabbitMQ3.6,seetheRabbitMQ3.6.xsupporttimelineannouncement .

IfyouarealreadyusingRabbitMQforPCF,youcanchoosetoprovideserviceplanswithRabbitMQ3.6forbackwardscompatibility.However,youshouldalsoenableRabbitMQ3.7planstoallowdeveloperstomigratetothenewversion.

Fornewdeployments,PivotalrecommendsdisablingallRabbitMQ3.6plansandusingRabbitMQ3.7toavoidtheneedtoupgradeinthenearfuture.

On-DemandClusterPlanUsingRabbitMQ3.7


https://www.rabbitmq.com/parameters.html


Likethesinglenodeplan,thisplanisdesignedtobesimpletoconfigure,deployanduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingOpenSourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.

Thisplancansuithighperformanceworkloadsrequiringmessagingresilience(copiedtodisk)andasynchronousmessagingreplicationthroughtheRabbitMQFederationplugin.Withthisplan,however,youalsoscaleoutRabbitMQforPCFtomultiplenodes.

Thisplanoffers:

Fastaccesstoanisolated,clusteredinstanceofRabbitMQscopedtotheapplicationteamOrgsandSpaces

AdministratoraccesstotheRabbitMQManagementUItogiveapplicationteamsfullcontroloverthecluster

Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes.

MessageresilienceprovidedbymirroringqueuesacrossRabbitMQnodes,andtheoptiontousetheFederationandShovelplugins.

ToBeDeprecated:On-DemandClusterPlanUsingRabbbitMQ3.6Thisplanprovidesthesamebenefitsasdescribedabove,butusesRabbitMQ3.6forbackwardscompatibility.

RabbitMQ3.6isreachingendofsupport.ConsidermovingtoRabbitMQ3.7assoonaspossible.FormoreinformationaboutsupportforRabbitMQ3.6,seetheRabbitMQ3.6.xsupporttimelineannouncement .

IfyouarealreadyusingRabbitMQforPCF,youcanchoosetoprovideserviceplanswithRabbitMQ3.6forbackwardscompatibility.However,youshouldalsoenableRabbitMQ3.7planstoallowdeveloperstomigratetothenewversion.

Fornewdeployments,PivotalrecommendsdisablingallRabbitMQ3.6plansandusingRabbitMQ3.7toavoidtheneedtoupgradeinthenearfuture.

GeneralPrinciplesoftheClusterPlanThefollowingaresomegeneralprinciplestobeawareofwhenconfiguringtheclusterplan:

DesignedforConsistency

RabbitMQclusteringisnotprimarilyasolutionforincreasedavailability.Instead,itisdesignedforconsistencyandpartitiontolerance,asdescribedintheCAPtheorem .RabbitMQclusteringprovidesincreasedmessageconsistencythroughqueuemirroring.Thismeansthatmessagesaccessedinonequeueareexactlythesameasinanotherqueue.Formoreinformation,seeConsistencyorAvailabilityTradeoff.

Otheroptionscanbeusedforavailabilityrequirements,suchastheuseoffederationbetweenexchangesorqueues.

ForadetaileddescriptionofdistributedRabbitMQbrokers,seetheRabbitMQdocumentation .

NumberofNodes

Everynodeintheon-demandclustermaintainsacompletedatabaseofallmetadata,andallchangestothemetadataareconfirmedbyeverynodeinthecluster.Therefore,goingbeyondsevennodescanhaveasignificantnegativeimpactonperformance.Foroptimumresilienceandperformance,Pivotalrecommendsthreenodesformostworkloads.

NetworkLatency

RabbitMQclustersareonlyrecommendedfordeploymentinlowlatencynetworks,whichnormallymeansthatitisnotadvisabletodeploytheseclustersacrossavailabilityzones(AZs).ThestabilityandperformanceoftheRabbitMQclusterisheavilyinfluencedbytheworkloadonthenodes,replicationchoices,andnetworklatency.

Forthisreason,PivotalrecommendsthatyoudeployRabbitMQclustersintoasingleOpsManagerAZ.However,wheredifferentAZsareinthesamedatacenter,withreliablelowlatencylinks,spanningAZscanbeused.

ForcloudIaaSdeployments,Pivotaldoesnotrecommendthatdeploymentsspanregions.Forexample,inAmazonWebServices(AWS)terms,deployingaRabbitMQclusteracrossAZswithinaregionshouldprovidehighenoughnetworkperformancetopreventimpactingclusterstability.However,deployingacrossAWSregionsislikelytoleadtoclusterinstability.Formoreinformation,seetheAWSdocumentation .



https://en.wikipedia.org/wiki/CAP_theorem

http://www.rabbitmq.com/distributed.html

http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/RegionsAndAZs.html

ConsistencyorAvailabilityTradeoffInadistributedmessagingsystem,atradeoffmustbemadebetweenavailabilityorconsistencywhenanetworkpartitioneventoccursandoneormorenodesarenotabletocommunicatewitheachother.TheclusterplanletsoperatorsdecidehowtheywanttheRabbitMQclustertoreactintheeventofanetworkpartition.

Pivotalrecommendskeepingthedefaultclusterpartitionoptionof pause_minority becausethissatisfiesmostusecases.Choosingthe pause_minority

partition-handlingstrategyfavorsmessageconsistencyoveravailability.Formoreinformationabouttheoptionsforhandlingpartitions,seetheRabbitMQdocumentation .ForadetaileddescriptionoftheoptionsavailableinRabbitMQforPCF,seeClusteringandNetworkPartitions.

Hereisanexampleofhow pause_minority works.IfyoucreateaRabbitMQclusterwiththreenodesandonenodebecomesunabletocommunicatewiththeothertwo,thisnodeisintheminority.Thenodethatisintheminorityispaused,andtheothertwonodescontinueservingtraffic.Ifeachofthenodeslosesconnectivitywiththeothertwo,thentheentireclusterispausedtopreservedatasincenomajoritycanbeestablished.Theclusterhealswhentwoormorenodesareabletocommunicatewitheachother.

RabbitMQQueueAvailabilityItisimportanttobeawarethatmessagequeueavailabilityisdifferentfromclusteravailability.So,havingclusteravailabilitydoesnotmeanthatallofthemessageswithinthequeuesarealsoavailable.

Bydefault,queueswithinaRabbitMQclusterarelocatedonasinglenode—thenodeonwhichtheywerefirstdeclared.However,queuescanbeconfiguredtomirroracrossmultiplenodes,sothatanymessagepublishedtothequeueisreplicatedtoallmirrors.Enablingmirroringcanhaveanegativeimpactonqueueperformancebecausemessagesmustbecopiedtoallmirrorsbeforebeingacknowledged.

Eachmirroredqueueconsistsofonemasterandoneormoremirrors,withtheoldestmirrorbeingpromotedtothenewmasteriftheoldmasterdisappearsforanyreason.Consumersareconnectedtothemasterregardlessofwhichnodetheyconnectto,andmirrorsdropmessagesthathavebeenacknowledgedatthemaster.Queuemirroringenhancesqueueavailability,butdoesnotdistributeloadacrossnodesbecauseeachoftheparticipatingnodesmuststilldoallthework.

Appdevelopersmustdecideiftheywanttousequeuemirroringanddeterminethepolicytheywanttoapplytotheirqueues.Thesechoiceshavesignificantimpactontheavailabilityoftheirqueues.Formoreinformation,seetheRabbitMQdocumentation .

Unlikethepre-provisionedplan,theclusterplandoesnotshipwithadefaultloadbalancer.Therefore,developersmustconfiguretheirapptousethearrayofhostsprovidedin VCAP_SERVICES .Ifdevelopersenablequeuemirroring,theymustalsoensuretheirappshavere-trylogicandreconnectionlogicthatiteratesovertherangeofhostsprovided.MostcommonRabbitMQclientshavethislogicbuiltintothem.Formoreinformation,seetheSpringAMQPdocumentation .

Becausetheclusterplanisdesignedtoenableapplicationteamstoself-serve,nothavingaloadbalancerinfrontoftheRabbitMQclusterhasthesebenefits:

Manageresourcesbetter,asfewerVMsareneeded.

Helpwithtroubleshooting.ClientIPisnowtheIPofthesourcecontainerandnottheHAProxy.

Reducethenumberofhopsbetweenappsandbroker.Thishelpswithlatency.

Determinequeueplacement.Thismakessenseforlargerscaledeployments.

Empowerapplicationteamstomanagetheirclusterinthebestwayfortheirapp.

Requirere-trylogicinanappifitneedsHAaccesstoaqueue.Thus,allnodescanroutetoaqueueifitisavailable.

ManagingOn-DemandResourcesThroughPlansInconfiguringeachplan,thereareanumberofoperationalcontrolsthatplatformoperationsteamscanusetomanagetheresourcesconsumedbyon-demandRabbitMQ:

ControlAccess—Operatorscanchoosetheappdevelopmentorgsandspacesforwhichtheplansareavailableandvisible.Eachplancanbeenabledordisabled,andserviceaccessandvisibilitycaneitherbeglobal,orenabledperorgandspacethroughthecommandline.

Forexample,youmaydecidetoenablethesinglenodeon-demandplanacrossallapplicationteamstomeettheirdemandtoisolatetheirworkload.Youmaythenchoosetooffertheon-demandclusterplanonlytoasubsetofapplicationteamswhorequiretheextraresources.

SetQuotas—Youcansetaglobalquotaforallon-demandinstancesthattakesprecedenceovereachplanquota.Thisletsyouguardagainsttheriskofover-committingresources,butallowstheflexibilityofover-committingeachplan,soyoucanmeetthefluctuatingdemandsofyourappdevelopers.

ControlResourceConsumption—Eachplanoffersmorefine-grainedcontroloverindividualplanresourceconsumption.Atthehighestlevel,youcanusetheplanquotatocontrolthenumberofinstancesthatcanbedeployedwithinafoundation.Foreachplan,youcanalsoconfigurethenumberof


https://www.rabbitmq.com/partitions.html

https://www.rabbitmq.com/ha.html

https://docs.spring.io/spring-amqp/reference/htmlsingle/#auto-recovery

nodesthatconstituteacluster(3,5,or7),theinstancetype,andpersistentdiskstoragesizetobestsuityourrequirements.

Monitor—Youcanmonitorthenumberofinstancesthathavebeendeployedagainstthequotayouhavesetsothatyoucanplanfutureresourcerequirements.

CustomizingPlanOptionsTheRabbitMQforPCFon-demandplansexposeanumberofconfigurationoptions.Inmostcases,thedefaultconfigurationsmeetmostappdemands.However,itisimportantforanoperationsteamtoconsidertheoptionstoensurethattheyprovidethebestservicetotheirappdevelopers.Thissectionexplainstheseoptions.

ConfigurationOptions

SingleNodeandClusterPlansEnable/Disableplan

Determinewhichorgsandspacescanseeandaccesstheplan

SetServiceInstanceQuota

SelectAZplacement(whereapplicable)

SetRabbitMQinstancesize(CPUandMemory)

Setpersistentdisksize(PersistedMessageStore)fortheRabbitMQinstance.Ensurethesizeofthepersistentdiskisatleasttwiceaslargeastheinstancememory.

ClusterPlanOnlySetnumberofnodesto3,5,or7

Determinenetworkpartitionbehavior.SeeConsistencyorAvailabilityTradeoffabove.

ThingsThatArePreconfiguredThefollowingarepreconfiguredforboththesinglenodeandtheclusterplans:

RabbitMQVMType—WheninstallingonPCFv2.0orlater,eachRabbitMQnodeisconfiguredtohavethefollowingproperties:

CPUs:2RAM:8GBEphemeraldisk:16GB

YoucanchangethesesettingsintheServicePlanConfigurationpage.Changingthesesettingsaffectsallnodes.

PersistentDiskType—WheninstallingonPCFv2.0orlater,eachRabbitMQnodeisconfiguredtohave30GBofpersistentdiskspace.

YoucanchangethissettingintheServicePlanConfigurationpage.PivotalrecommendsyousetthisvaluetobetwicetheamountofRAMoftheselectedRabbitMQVMType.

Metrics—EmittedtotheLoggregatorFirehoseforallon-demandinstances.ThepollingintervalissetintheOpsManager,intheMetricspollingintervalfield,inthePre-ProvisionedRabbitMQtaboftheRabbitMQforPCFtile.Duetotheimpactofsomeoftheclustersettingsdetailedbelow,PivotalstronglyrecommendsthatyoumonitortheexposedmetricsandconfigurealarmsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF .SeealsoMonitoringOn-DemandRabbitMQClustersbelow.

Logs—RabbitMQon-demandinstancelogsareforwardedusingthesameconfigurationascontainedintheSyslogtaboftheRabbitMQforPCFtile.

Diskfreespacelimit—Thediskfreespacelimitissetto150%ofRAMoftheinstancetypeyouselect.Forexample,ifyouselectaninstancetypewith10GBofRAM,thediskfreespacelimitissetto15GB.Acluster-widealarmistriggerediftheamountoffreediskspacedropsbelowthis,andallpublishersareblocked.InstancesmustbeconfiguredtohavepersistentdisksthatareatleasttwicethesizeofinstanceRAM.Formoreinformation,seetheRabbitMQdocumentation .

Memorythresholdfortriggeringflowcontrol—Thresholdatwhichflowcontrolistriggeredissetto40%oftheinstanceRAM.Thismeansthatwhen

Note:Aloadbalancer,suchasHAProxy,isnotdeployedwithon-demandclusterplans.


https://docs.pivotal.io/rabbitmq-cf/1-12/monitor.html#kpi

https://www.rabbitmq.com/disk-alarms.html

thealarmistriggered,allconnectionspublishingmessagesareblockedcluster-wideuntilthealarmiscleared.

Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan4GBofmemoryisused,allpublishingconnectionsareblocked.Formoreinformation,seeMemoryAlarms intheRabbitMQdocumentation.

Memorypagingthreshold—ThisisthelevelatwhichRabbitMQtriestofreeupmemorybyinstructingqueuestopagetheircontentsouttodisk.Thisisdonetotrytoavoidreachingthehighwatermarkandblockingpublishers.Thisthresholdissetto50%oftheconfiguredhighwatermark,whichis20%ofconfiguredmemory.Formoreinformationonmemorycalculation,seeChangestoMemoryAllocationwhenUpgrading .

Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan2GBofmemoryisused,allqueuesstartwritingallqueuecontentstodisk.Formoreinformation,seetheRabbitMQdocumentation .

MonitoringOn-DemandRabbitMQClustersItisimportanttomonitorandcomparethenumberofinstancesthathavebeendeployedagainstthequotayousetviathemetricexposedontheFirehose.

Eachinstanceispre-configuredtoemitmetricstotheFirehoseandcanbeidentifiedbythe deployment tag,whichhastheserviceinstanceID.ItisimportanttomonitorthesemetricsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF .


https://www.rabbitmq.com/memory.html

https://docs.pivotal.io/rabbitmq-cf/1-12/rmq-3611-upgrade.html

https://www.rabbitmq.com/memory.html

https://docs.pivotal.io/rabbitmq-cf/1-12/monitor.html#kpi

On-DemandServiceArchitectureThistopicdescribesthearchitectureforon-demandRabbitMQ®forPivotalCloudFoundry(PCF).

Forinformationaboutarchitectureoftheolder,pre-provisionedservice,seeDeployingtheRabbitMQ®Service.

ServiceNetworkRequirementWhenyoudeployPCF,youmustcreateastaticallydefinednetworktohostthecomponentvirtualmachinesthatconstitutethePCFinfrastructure.

PCFcomponents,liketheCloudControllerandUAA,runonthisinfrastructurenetwork.On-demandPCFservicesmayrequirethatyouhostthemonanetworkthatrunsseparatelyfromthisnetwork.Youcanalsodeploytilesonseparateservicenetworkstomeetyourownsecurityrequirement.

PCFv2.0andEarlierInPCFv2.0andearlier,cloudoperatorspre-provisionserviceinstancesfromOpsManager.Foreachservice,OpsManagerallocatesandrecoversstaticIPaddressesfromapre-definedblockofaddresses.

Toenableon-demandservicesinPCFv2.0andearlier,operatorsmustcreateaservicenetworksinBOSHDirectorandselecttheServiceNetworkcheckbox.Operatorsthencanselecttheservicenetworktohoston-demandserviceinstanceswhentheyconfigurethetileforthatservice.

PCFv2.1andLaterPCFv2.1andlaterincludedynamicnetworking.InPCFv2.1andlater,operatorscanusedynamicnetworkingwithasynchronousserviceprovisioningtodefinedynamically-provisionedservicenetworks.Formoreinformation,seeDefaultNetworkandServiceNetwork.

InPCFv2.1andlater,on-demandservicesareenabledbydefaultonallnetworks.OperatorscancreateseparatenetworkstohostservicesinBOSHDirector,butdoingsoisoptional.Operatorsselectwhichnetworkhostson-demandserviceinstanceswhentheyconfigurethetileforthatservice.

DefaultNetworkandServiceNetworkOn-demandPCFservicesrelyontheBOSH2.0abilitytodynamicallydeployVMsinadedicatednetwork.Theon-demandservicebrokerusesthiscapabilitytocreatesingle-tenantserviceinstancesinadedicatedservicenetwork.

On-demandservicesusethedynamically-provisionedservicenetworktohostthesingle-tenantworkerVMsthatrunasserviceinstanceswithindevelopmentspaces.ThisarchitectureletsdevelopersprovisionIaaSresourcesfortheirserviceinstancesatcreationtime,ratherthantheoperatorpre-provisioningafixedquantityofIaaSresourceswhentheydeploytheservicebroker.

Bymakingservicessingle-tenant,whereeachinstancerunsonadedicatedVMratherthansharingVMswithunrelatedprocesses,on-demandserviceseliminatethe“noisyneighbor”problemwhenoneapphogsresourcesonasharedcluster.Single-tenantservicescanalsosupportregulatorycompliancewheresensitivedatamustbecompartmentalizedacrossseparatemachines.

Anon-demandservicesplitsitsoperationsbetweenthedefaultnetworkandtheservicenetwork.Sharedcomponentsoftheservice,suchasexecutivecontrollersanddatabases,runcentrallyonthedefaultnetworkalongwiththeCloudController,UAA,andotherPCFcomponents.Theworkerpooldeployedtospecificspacesrunsontheservicenetwork.

ThediagrambelowshowsworkerVMsinanon-demandserviceinstancerunningonaseparateservicesnetwork,whileothercomponentsrunonthedefaultnetwork.


RequiredNetworkingRulesforOn-DemandServicesBeforedeployingaservicetilethatusestheon-demandservicebroker(ODB),requesttheneedednetworkconnectionstoallowcomponentsofPivotalCloudFoundry(PCF)tocommunicatewithODB.

ThespecificsofhowtoopenthoseconnectionsvariesforeachIaaS.

Seethefollowingtableforkeycomponentsandtheirresponsibilitiesinanon-demandarchitecture.

KeyComponents TheirResponsibilities

BOSHDirector

CreatesandupdatesserviceinstancesasinstructedbyODB.

BOSHAgentIncludesanagentoneveryVMthatitdeploys.TheagentlistensforinstructionsfromtheBOSHDirectorandcarriesoutthoseinstructions.TheagentreceivesjobspecificationsfromtheBOSHDirectorandusesthemtoassignarole,orjob,totheVM.

BOSHUAA IssuesOAuth2tokensforclientstousewhentheyactonbehalfofBOSHusers.

PAS Containstheappsthatareconsumingservices

ODB InstructsBOSHtocreateandupdateservices,andconnectstoservicestocreatebindings.

Deployedserviceinstance

Runsthegivendataservice.Forexample,thedeployedRedisforPCFserviceinstancerunstheRedisforPCFdataservice.

Regardlessofthespecificnetworklayout,theoperatormustensurenetworkrulesaresetupsothatconnectionsareopenasdescribedinthetablebelow.

Thiscomponent…Mustcommunicatewith…

DefaultTCPPort Communicationdirection(s) Notes

ODB

BOSHDirector

BOSHUAA

25555

8443 One-way Thedefaultportsarenotconfigurable.

ODBDeployedserviceinstances

15672(RabbitMQManagementUI)

One-wayThisconnectionisforadministrativetasks.Avoidopeninggeneraluse,app-specificportsforthisconnection.

ODBPAS(orElastic 8443 One-way Thedefaultportisnotconfigurable.


Runtime)

ErrandVMs

PAS(orElasticRuntime)

ODB

DeployedServiceInstances

8443

8080


5671-2(AMQP/AMQPS)

One-way Thedefaultportisnotconfigurable.

BOSHAgentBOSHDirector

4222 Two-way

TheBOSHAgentrunsoneveryVMinthesystem,includingtheBOSHDirectorVM.TheBOSHAgentinitiatestheconnectionwiththeBOSHDirector.Thedefaultportisnotconfigurable.

DeployedappsonPAS(orElasticRuntime)

Deployedserviceinstances


5671-2(AMQP/AMQPS)

61613-4(STOMP/STOMPS)

1883,8883(MQTT/MQTTS)

One-wayThisconnectionisforgeneraluse,app-specifictasks.Avoidopeningadministrativeportsforthisconnection.

PAS(orElasticRuntime)

ODB 8080 One-wayThisportmaybedifferentforindividualservices.Thisportmayalsobeconfigurablebytheoperatorifallowedbythetiledeveloper.

DeployedappsonPAS

RuntimeCredHub

8844 One-wayThisportisneededifsecureserviceinstancecredentialsareenabled.Forinformation,see(Beta)OnDemand-SecureServiceInstanceCredentialswithRuntimeCredHub.


DeployingtheRabbitMQPre-ProvisionedService

DefaultDeploymentDeployingRabbitMQforPivotalCloudFoundry (PCF)throughOpsManagerdeploysaRabbitMQclusterof3nodesbydefault.

Thedeploymentincludesasingleloadbalancer haproxy whichspreadsconnectionsonallofthedefaultports,foralloftheshippedpluginsacrossallofthemachineswithinthecluster.

Thedeploymentoccursinasingleavailabilityzone(AZ).

ThedefaultconfigurationisfortestingpurposesonlyandPivotalrecommendsthatcustomershaveaminimumof3RabbitMQnodesand2HAProxynodes

ConsiderationsforthisdeploymentProvideshighavailabilityfortheRabbitMQcluster

Queuesmustbeconfiguredtobehighavailabilityastheyareplacedononenodebydefault

Customersshoulddecidewhichpartitionbehaviorisbestsuitedtotheirusecase.Fortwonodes‘automatic’ispreferred.

HAProxyisasinglepointoffailure(SPOF)

TheentiredeploymentisinasingleAZ,whichdoesnotprotectagainstexternalfailuresfromfailuresinhardware,networking,etc.

RecommendedDeployment


https://network.pivotal.io/products/pivotal-cf

PivotalrecommendsthatRabbitMQforPCFisdeployedacrossatleasttwoAZs.ScaleRabbitMQservernodestoanoddnumberthatisgreaterthanorequaltothree.

Onlyusereplicationofqueueswhererequiredasitcanhaveabigimpactonsystemperformance.

TheHAProxyjobinstancecountshouldalsobeincreasedtomatchthenumberofAZstoensurethereisaHAProxylocatedineachAZ.ThisremovestheHAProxySPOFandprovidesfurtherredundancy.

ThediagramaboveshowsthatyoucannowsufferthefailureofasingleHAProxyandsingleRabbitMQnodeandstillkeepyourclusteronlineandapplicationsconnected.

UpgradingtothisdeploymentfromasingleAZdeploymentItisnotpossibletoupgradetothissetupfromthedefaultdeploymentacrossasingleAZ.

ThisisbecausetheAZsetupcannotbechangedafterthetilehasbeendeployedforthefirsttime.ThisistoprotectagainstdatalosswhenmovingjobsbetweenAZs.

UpgradingtothisdeploymentfromamultiAZdeploymentIfyouhavedeployedthetileacrosstwoAZs,butwithasingleHAProxyinstance,youcanmigratetothissetupbydeployinganadditionalHAProxyinstancethroughOpsManager.Neworre-boundapplicationstotheRabbitMQserviceseetheIPsofbothHAProxiesimmediately.Existingboundapplicationswillcontinuetowork,butonlyusingthepreviouslydeployedHAProxyIPAddress.Theycanbere-boundasrequiredatyourdiscretion.

ConsiderationsforthisdeploymentRequiresIaaSconfigurationforAZsaheadofdeployingtheRabbitMQtile

ApplicationdevelopersarehandedtheIPsofeachdeployedHAProxyintheirenvironmentvariables

Queuesmustbeconfiguredtobehighavailabilityastheyareplacedononenodebydefault


Customersshoulddecideonwhichpartitionbehaviorisbestsuitedtotheirusecase.Forthreeormorenodes'pause_minority’ispreferred.

AdvancedDeploymentThisdeploymentbuildsupontheaboverecommendeddeploymentandsofollowsthesameupgradepaths.

ThisallowsyoutoreplacetheuseofHAProxywithyourownexternalloadbalancer.

YoumightchoosetodothistoremoveanyknowledgeofthetopologyoftheRabbitMQsetupfromapplicationdevelopers.

Advantages

ApplicationdevelopersdonotneedtohandlemultipleIPsfortheHAProxyjobsintheirapplications

Disadvantages

TheloadbalancerneedstobeconfiguredwiththeIPsoftheRabbitMQNodes.Theseareonlybeknownafterthedeploymentisfinished.TheIPsshouldremainthesameduringsubsequentdeploymentsbutthereisarisktheymightchange.

UpgradingtothisdeploymentfromtherecommendeddeploymentItispossibletofirstdeploywithmultipleHAProxyjobs,aspertherecommendeddeployment,andlateruseyourownexternalloadbalancer.

Thiscanbeachievedwithoutdowntimetoyourapplications.Followthesestepstodoso:

1. ConfigureyourexternalloadbalancertopointtotheRabbitMQNodeIPs.

2. ConfiguretheDNSnameorIPaddressfortheexternalloadbalancer(ELB)ontheRabbitMQtileinOpsManager.

3. Deploythechanges.AnynewinstancesoftheRabbitMQserviceoranyre-boundconnectionswillusetheDNSnameorIPaddressoftheELBintheirVCAP_SERVICES .AnyexistinginstanceswillcontinuetousetheHAProxyIPaddressesintheir VCAP_SERVICES

4. Phasethere-bindingofexistingapplicationstoupdatetheirenvironmentvariables.

5. Afterallapplicationsareupdated,reducetheinstancecountofthe HAProxy jobinOpsManagerto1.

6. Deploythechanges.

Thisapproachworksasanyexistingboundapplicationshavetheir VCAP_SERVICES informationcachedinCloudControllerandareonlyupdatedbyare-


bindrequest.

DowngradingfromthisdeploymenttotherecommendeddeploymentIfyouarecurrentlyusinganexternalloadbalancer,thenyoucanmovebacktousingHAProxiesinstead.

Youcanachievethisbyfollowingtheabovestepsinreverseorderandre-instatingtheHAProxyjobs.

ResourcerequirementsThefollowingtableshowsthedefaultresourceandIPrequirementsforinstallingthetile:

Product Resource Instances CPU Ram Ephemeral Persistent StaticIP DynamicIP

RabbitMQ RabbitMQnode 3 2 8192 16384 30720 1 0

RabbitMQ HAProxyforRabbitMQ 1 1 2048 4096 0 1 0

RabbitMQ RabbitMQservicebroker 1 1 2048 4096 0 1 0

RabbitMQ BrokerRegistrar 1 1 1024 2048 0 0 1

RabbitMQ BrokerDeregistrar 1 1 1024 2048 0 0 1

RabbitMQ SmokeTests 1 1 1024 2048 0 0 1

RabbitMQ RabbitMQon-demandbroker 1 1 1024 8192 1024 0 1

RabbitMQ RegisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1

RabbitMQ DeregisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1

RabbitMQ DeleteAllServiceInstances 1 1 1024 2048 0 0 1

RabbitMQ UpgradeAllServiceInstances 1 1 1024 2048 0 0 1

Notes:Thenumberof RabbitMQ Node canbeincreasedifrequired.

ChangingthenumberofRabbitMQnodeswhentheerlangcookieisnotdefinedwillrestartthecluster.Checkhereformoreinformation.


InstallingandConfiguringtheOn-DemandServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideon-demandservice.

TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .

Role-BasedAccessinOpsManagerOpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.

FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .

PrerequisitesforDeployingtheOn-DemandServiceBeforedeployingRabbitMQforPCFasanon-demandservice,youmustensurethattherequirednetworkrulesareinplacetoallowvariouscomponentstocommunicate.

SeeRequiredNetworkingRulesforOn-DemandServicesfordetailsonthenetworkconnectionsthatmustbeopentoenabletheon-demandservice.

Forinformationabouttheon-demandservicearchitecture,seeOn-DemandServiceArchitecture.

DownloadandInstallRabbitMQforPCF1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.

3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.

4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.

ConfigureOn-DemandRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.

Note:Forinstructionsonhowtoinstall,configure,anddeploytheRabbitMQforPCFtileasapre-provisionedservice,seeInstallingandConfiguringthePre-ProvisionedService.


https://www.rabbitmq.com/production-checklist.html

https://docs.pivotal.io/pivotalcf/opsguide/config-rbac.html#about

https://network.pivotal.io/products/pivotal-rabbitmq-service/

WhichSettingsTabstoConfigurefortheOn-DemandServiceConfigurethefollowingtabsfortheon-demandservice:

RabbitMQSettingsTab Instructions

AssignAZsandNetworks ConfigureAZsandNetworks

Pre-ProvisionedRabbitMQ ConfigureAdminCredentialsandMetricsPollingInterval

Networking Networking


Syslog SetupSyslogForwarding

GlobalSettingsforOn-DemandPlans ConfigureGlobalSettings

OnDemandInstance:Plann ConfiguretheServicePlan

Errands Errands

Stemcell VerifytheStemcell

ConfigureAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.

1. ClickAssignAZsandNetworks.

2. ConfigurethefieldsontheAssignAZsandNetworksasfollows:

Field Instructions

Placesingletonjobsin

SelecttheregionthatyouwantforsingletonVMs.PCFcreatestheRabbitMQbrokerinthisAZ.

Balanceotherjobsin

Selectadditionalregion.Thisselectiondoesnotaffecttheon-demandRabbitMQforPCFservice.

Network

SelectanetworkfortheRabbitMQOn-DemandBroker.

ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.FormoreinformationabouttheDefaultNetwork,seeDefaultNetworkandServiceNetwork.

Typically,youselectthenetworkusedforthePivotalApplicationService(PAS)orElasticRuntimecomponents.

ServiceNetwork

Selectaseparatenetworkthattheon-demandserviceinstancesrunon.

Atypicalpracticeistoputallon-demandservicesonasinglenetwork,separatefromthenetworkthatPivotalApplicationService(PAS)orElasticRuntimeandtheOn-DemandBrokerrunon.ForinformationabouttheServiceNetwork,seeDefaultNetworkandServiceNetwork.

Thisfieldisalsorequiredforthepre-provisionedservice,thoughinthatcase,itdoesn’tmatterwhichnetworkyouselect.

3. ClickSave.

ConfigureAdminCredentialsandMetricsPollingIntervalOnthePre-ProvisionedRabbitMQtab,youmustconfiguretwoitemsonlyfortheondemandservice:

SpecifyadmincredentialsforusingtheRabbitMQManagementDashboard.

Enterametricspollinginterval.

SpecifyAdminUserCredentials

IntheRabbitMQadminusercredentialsfieldofthePre-ProvisionedRabbitMQtab,enteranadminusernameandpassword:

Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesintheApplyChangesfromYourConfigurationbelow.

WARNING:ChangingtheNetworkorServiceNetworkafteryouhaveconfiguredthem,orchangingtheirIPconfigurations,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .


https://docs.pivotal.io/rabbitmq-cf/1-12/monitor.html#config


Youcanuseacombinationofupperorlowercasealphanumericsandsupportedspecialcharacters: []^_!"#$%&()*+,-./\:;<=>? .

ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.

EnteraMetricsPollingInterval

IntheMetricspollingintervalfield,setametricspollinginterval:

Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.

Changingthissettingaffectsalldeployedinstances.

Formoreinformation,seeWhatareMetrics .

ConfigureLoggingtoMonitorRabbitMQforPCFPivotalrecommendsthatyouconfigureloggingtomonitorthehealthofRabbitMQforPCF.FollowSetUpSyslogForwarding toconfigurelogging.

NetworkingTospecifyastaticIPaddressfortheOnDemandServiceBroker,dothefollowing:

1. SelectNetworking.

2. EnteranIPaddressintheOnDemandServiceBrokerStaticIPfield.ThisIPaddressisassignedtoyourOnDemandServiceBrokernode.BOSHallocatesanIPaddressifthefieldisleftblank.

3. ClickSave.

ConfigureGlobalSettingsFollowthestepsbelowtoconfigureglobalsettings.

1. ClickGlobalSettingsforOn-DemandPlans.

Note:Youcannotusebackquote ` orsinglequote ' .

Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.


https://docs.pivotal.io/rabbitmq-cf/1-12/monitor.html#metrics

https://docs.pivotal.io/rabbitmq-cf/1-12/monitor.html#config

2. Configurethefollowing:

Serviceinstancequotamin:0,max:50setthetotalnumberofon-demandserviceinstanceswhichcanbedeployed.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.

VMoptions:

Allowoutboundinternetaccess(IaaS-dependent).Thischeckboxmustbetickedtoallowexternallogforwarding,sendingbackupartifactstoexternaldestinations,andcommunicatingwithanexternalBOSHblobstore.

(Beta)ShareableInstances:ClickYestoenablethebetafeatureforsharinginstances.

Note:OutboundnetworktrafficrulesalsodependonyourIaaSsettings.ConsultyournetworkorIaaSadministratortoensurethatyourIaaSallowsoutboundtraffictotheexternalnetworksyouneed.

Note:ThisisabetafeaturebasedonanexperimentalfeatureinCloudFoundry.Usethefeatureatyourownriskinnon-productionenvironments.Ifyoutrythisfeature,pleasesendyourcommentsandfeedbacktoPCFFeedbackList.



Sharingaserviceinstancebetweenspaces,allowsappsindifferentspacestosharedatabases,messagingqueues,andmanyothertypesofservices.Formoreinformation,seeSharingServiceInstances(Beta) .

*(Beta)OnDemand-SecureServiceInstanceCredentialswithRuntimeCredHub:Foron-demandservicesinstances,clickYestosecurecredentialswithCredHub.

3. ClickSave.

ConfiguretheServicePlanToenabletheon-demandservice,youmustconfigureatleastoneon-demandplan.

Youcanconfigureuptofiveon-demandplans:OnDemandInstance:Plan1–OnDemandInstance:Plan5.

Allon-demandplanscanbeconfiguredtohave1,3,5,or7RabbitMQnodes.

Iftheon-demandserviceisnotenabled,theon-demandbrokerisdeployedalongsidetheRabbitMQinstallation,butitisnotavailableintheMarketplace.

1. Choosetheon-demandserviceinstanceyouwanttoconfigure:

OnDemandInstance:Plan1(completerequiredfieldsevenifyoudisablethisplan):

Note1:ForthisfeaturetoworkyoumustalsoenableitinPAS.Forinstructions,seeStep1:ConfigurethePASTile .Afterdeployingthetile,notifydevelopersthattheymustunbindandrebindexistingserviceinstancestosecuretheircredentialswithCredHub.

Note2:Thisisabetafeature.Useitatyourownriskinnon-productionenvironments.Ifyoutrythisfeature,pleasesendyourcommentsandfeedbacktoPCFFeedbackList.

Note:YoumustfullyconfigureOnDemandInstance:Plan1evenifyoudisableaccesstothisplan(seeCFServiceAccessinthetablebelow).


https://docs.pivotal.io/pivotalcf/devguide/services/sharing-instances.html

https://docs.pivotal.io/pivotalcf/2-0/opsguide/secure-si-creds.html#pas-config


OnDemandInstance:Plan2,3,4,and5:



2. Configurethefieldsasfollows:

Field Instructions

EnableThisPlan

(Plans2-5only)Toenable,selectPlanEnable.

CFServiceAccess

Enableordisableaccesstothisplan,orleaveaccessunchanged.

IfyouenablePlan1,thedefaultsettingforPlans2-5isEnableServiceAccess.Ifyouchangethisdefaultsetting,thesmoketestsfail.Therefore,ifyouenablePlan1andwanttochangethisdefault,beforedoingso,settheOn-DemandInstanceSmokeTestserrandtoOff.Formoreinformation,seeErrands.

EnableServiceAccess—Givesaccesstoallorgs,anddisplaystheserviceplantoalldevelopersintheMarketplace.DisableServiceAccess—Disablesaccesstoallorgs,andhidestheserviceplantoalldevelopersintheMarketplace.ThissettingcannotbeselectedatalatertimeintheUI.LeaveServiceAccessUnchanged—Keepsanyexistingaccesssettings,andonlydisplaystheserviceplanintheMarketplacetomembersoforgsthathaveaccesstotheplan.YoucanchangetheaccesssettingslaterusingthecfCLI.Forinstructions,seeControllingAccesstoServicePlansbyOrg.

PlanName Acceptthedefaultorenteraname.ThisisthenamethatappearsintheMarketplace.

PlanDescription

Acceptthedefaultorenteradescription.ThisdescriptionappearsintheMarketplace.

PlanFeatures

Acceptthedefaultorenteradescription.ThisdescriptionappearsinAppsManager.

ServiceInstanceQuota

Enterthemaximumnumberofon-demandserviceinstancesthatcanbeavailableatonetime.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.

NumberofNodes

Enter1,3,5or7.Thissettingonlyaffectsnewserviceinstances.Previouslydeployedserviceinstancesarenotupdated.

NetworkPartitionBehaviour

Select pause_minority or autoheal .Pivotalrecommendsusingpauseminority.Formoreinformation,seeConsistencyorAvailabilityTradeoff.

RabbitMQVMType

SelectalargeVMtype.Theplancreatesaserviceinstanceofthissize.Formoreinformation,seeUnderstandingRabbitMQVMTypesandPersistentDiskSizebelow.

PersistentDiskType

ThisiswhereRabbitMQpagesmessagestodisk.ServiceinstancedeploymentsfailifthisvalueislessthantwicethevolumeofRAMoftheselectedRabbitMQVMType.Formoreinformation,seeUnderstandingRabbitMQVMTypesandPersistentDiskSizebelow.

AZPlacement

ThisfieldisavailableafteryoucompletetheAssignAZsandNetworkspage.

Forasingle-nodeplan,selectoneormoreAZs.Foraplancontainingmultiplenodes,selectonlyoneAZ.Pivotalrecommendsthisformulti-nodeplanstominimizerisksduetonetworklatencyandpartitions.SeeNetworkLatencyandConsistencyorAvailabilityTradeofffordetails.

Ifyouchangethisselectionafterdeployment,existinginstancesarenotaffectedbythechange.SeeDeterminewhichAZsaServiceInstanceUses.

3. ClickSave.

DeterminewhichAZsaServiceInstanceUses

TodeterminewhichAZsaserviceinstanceisplacedin,dooneofthefollowing:

RetrievetheserviceGUIDusingthe cf service SERVICE_INSTANCE --guid commandandthenruntheBOSH instances commandfortheservice-instance_GUID deployment.

Withsyslogforwardingenabled,inspecttheservicebrokerlogswhenrunningtheUpgradeAllServiceInstanceserrand.Foreachexistingservice

Important:IfyouchangethisconfigurationafteryouhaveselectedAZsanddeployedserviceinstances,existinginstancesarenotplacedinthenewlyconfiguredAZswhentheUpgradeAllServiceInstanceserrandisrun.Thispreventsre-creationoftheVMsindifferentAZs,whichcanleadtodataloss.Allnewserviceinstances,however,willbecreatedinthenewlyconfiguredAZs.


instance,thelogmessageincludestheserviceinstanceGUIDandtheAZstheserviceinstanceisrunningin.

UnderstandingRabbitMQVMTypesandPersistentDiskSize

TheRabbitMQVMTypeandPersistentdisktypearerequiredfieldsontheserviceplanconfigurationpages.IfyouareinstallingonPCFv2.0orlater,thesepropertiesarepre-configuredbydefault.

PivotalrecommendsthatthevalueofPersistentdisktypebetwicetheamountofRAMoftheselectedRabbitMQVMType.

YoucanchangetheRabbitMQVMtypeandthesizeofthepersistentdiskthatisattachedtotheRabbitMQinstances.Forexample,ifyouarerunningoutofdiskspaceyoumightwanttoincreasethepersistentdisksizebychangingthePersistentdisktypefield.Ifyoumakechanges,ensurethatthepersistentdisksizeisstilltwicethesizeoftheRAMoftheRabbitMQVMtype.

RabbitMQraisesalarmswhendiskspacedropsbelowtheconfiguredlimit.Incorrectdisksizesmightcausethedeployedinstancenottostart.RabbitMQdeclinestostartifthereisnotenoughspaceavailableaccordingtothethreshold.

On-Demandinstancesareconfiguredwithathresholdsettothe150%ofthememory(RAM)oftheVM.Usethefollowingtableasaguidewhenselectingthesizeofthepersistentdisk.

ThefollowingtableshowsanexampleofpossibleRAMvalues,absoluteminimalvaluebelowwhichRabbitMQdeclinestostart,andthedisksizesuggestedforanaverageusecase.

RAM Freediskalarmthreshold(1.5xRAM) Suggesteddisksize(2xRAM)

10GB 15GB 20GB

16GB 24GB 32GB

32GB 48GB 64GB

MinimumresourcesrequiredforeachRabbitMQVM:

CPU:2

RAM:1GB

Ephemeraldisk:2GB

Persistentdisk:4GB

Formoreinformation,seethefollowing:

MemoryandDiskAlarms

DiskAlarms

Forinformationonallpreconfiguredsettings,seeThingsthatarePreconfigured.

VerifytheStemcellForOpsManagerv2.0:

1. ClickStemcell.


http://www.rabbitmq.com/alarms.html

http://www.rabbitmq.com/disk-alarms.html

2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .

ForOpsManagerv2.1:

1. FollowtheprocedureoutlinedinImportingandManagingStemcells .

ApplyChangesfromYourConfigurationYourinstallationisnotcompleteuntilyouapplyyourconfigurationchanges.Followthestepsbelow:

1. ReturntotheOpsManagerInstallationDashboard.

2. ClickApplyChanges.

ErrandsWhendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofPost-DeployErrands.Anexampleisthe Smoke

Testserrand,

whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.

Youcandecidewhethertorunerrandsbytogglingthemonoroffbeforeanupdate.Thisisaone-timesettingontheinstallationdashboard:

However,ifnecessary,youcanchangethesedefaultsbyclickingErrandsintheRabbitMQforPCFSettingstab.

Formoreinformationonerrandrunrules,seeErrandRunRules .

RabbitMQforPCFerrandsarecolocatedwiththeirbrokerstodecreaseerrandruntimeandVMfootprint.Inearlierreleases,anewVMwasdeployedforeacherrand.Formoreinformationabouterrands,seeErrands .

Important:AsofRabbitMQforPCFv1.9.0,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,becausethesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmightfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.


https://docs.pivotal.io/pivotalcf/customizing/pcf-aws-manual-er-config.html#stemcell

https://docs.pivotal.io/pivotalcf/customizing/azure-er-config.html#stemcell

https://docs.pivotal.io/pivotalcf/customizing/gcp-er-config.html#stemcell

https://docs.pivotal.io/pivotalcf/customizing/config-er-vmware.html#stemcell

https://docs-pcf-staging.cfapps.io/pivotalcf/2-1/opsguide/managing-stemcells.html

https://docs.pivotal.io/pivotalcf/customizing/managing_errands.html#rules


Post-DeployErrands

Errand Description

BrokerRegistrar

Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace

SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTests.

RegisterOn-DemandServiceBroker

Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.

On-DemandInstanceSmokeTests

Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTestsbelow.

UpgradeAllServiceInstances

On-Demandinstancesareupdatedandredeployediftherearechangestoon-demandplansettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoon-demandplansettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.

Pre-DeleteErrands

Errand Description

BrokerDeregistrarRemovesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings

DeleteAllServiceInstancesUnbindsanddeletesexistingon-demandserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.

DeregisterOn-DemandServiceBroker

Removestheon-demandRabbitMQservicefromtheMarketplace

On-DemandInstanceSmokeTestsSmoketestsonlyrunagainstPlan1.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.

CreateanAdminUserforaServiceInstanceIfyouwanttogiveappdevelopersadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstanceandsharetheusercredentialswithappdevelopers.

Bothoperatorsandappdeveloperscanusethisprocedure.

TocreateanadminuseronaRabbitMQinstancedothefollowing:

1. Runthiscommandtocreateaservicekey:

cfcreate-service-keySERVICE_INSTANCESERVICE_KEY-c'{"tags":"administrator"}'

where:

SERVICE_INSTANCE isthenameyousuppliedwhenyouran cfcreate-service .SERVICE_KEY isanameyouchoosetoidentifytheservicekey.

Forexample:

$cfcreate-service-keymy-instancemy-admin-key-c'{"tags":"administrator"}'

Creatingservicekeymy-admin-keyforserviceinstancemy-instanceasuser@example.com...OK

2. Runthiscommandtogettheadminusercredentials:


cfservice-keySERVICE_INSTANCESERVICE_KEY wherethevariablesarethesameasabove.

ThisreturnsaDashboardURLcontainingtheadmincredentials,whichcanbeusedtoaccessthemanagementUI.Forexample:

$cfservice-keymy-instancemy-admin-key

Gettingkeymy-admin-keyforserviceinstancemy-instanceasuser@example.com...{"dashboard_url":"https://my-instance.bosh-lite.com/#/login/admin-username/admin-password","username":"admin-username","password":"admin-password",...}

RabbitMQServerSettingsThatCannotBeDisabledThefollowingpluginsareenabledbydefaultandcannotbedisabled:

rabbitmq_management

rabbitmq_federation

rabbitmq_federation_management

rabbitmq_shovel

rabbitmq_shovel_management


InstallingandConfiguringthePre-ProvisionedServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideapre-provisionedservice.

TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .

Role-BasedAccessinOpsManagerOpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.

FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .

DownloadandInstalltheTile1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.

3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.

4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.

ConfigurePre-ProvisionedRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.

Note:Forinstructionsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtileasanon-demandservice,seeInstallingandConfiguringRabbitMQforPCFasanOn-DemandService.


https://www.rabbitmq.com/production-checklist.html

https://docs.pivotal.io/pivotalcf/opsguide/config-rbac.html#about

https://network.pivotal.io/products/pivotal-rabbitmq-service/

WhichSettingsTabstoConfigureforthePre-ProvisionedServiceConfigurethefollowingtabsforthepre-provisionedservice:

RabbitMQSettingsTab Instructions

AssignAZsandNetworks AssignAZsandNetworks

Pre-ProvisionedRabbitMQ RabbitMQ

Networking Networking


Syslog SyslogGlobalSettingsforOn-DemandPlans GlobalSettings

Errands Errands

Stemcell Stemcell

AssignAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.

1. IntheSettingsscreen,clickAssignAZsandNetworks.

2. ConfigurethefieldsontheAssignAZsandNetworksasfollows.Allfieldsarerequired,thoughsomedonotapplytothepre-provisionedservice.

RequiredFields Instructions

Placesingletonjobsin

Selectaregion.Thisselectiononlyaffectstheon-demandservice.

Balanceotherjobsin

Selectadditionalregion(s).Thisselectiononlyaffectsthepre-provisionedservice.

Network

SelectanetworkfortheRabbitMQBroker.

ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.ThisnetworkisrepresentedbytheDefaultNetwork,describedinDefaultNetworkandServiceNetwork.Typically,youselectthenetworkusedforthePivotalApplicationService(PAS)orElasticRuntimecomponents.

ServiceNetwork

Selectanetwork.Thisselectiononlyaffectstheon-demandservice.

3. ClickSave.

RabbitMQToconfigurethefollowingsectionsinthePre-ProvisionedRabbitMQtab,intheSettingsscreen,clickPre-ProvisionedRabbitMQ.

RabbitMQAdminUserCredentialsIntheRabbitMQadminusercredentialsfieldoftheRabbitMQpane,enteranadminusernameandpassword:

Youcanuseacombinationofupperorlowercasealphanumericsandsupportedspecialcharacters: -=_+":;/?.><,~ .

ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.

Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesinthefinalstepbelow.

WARNING:ChangingtheNetworkafteryouhaveconfiguredit,orchangingtheIPconfiguration,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

Note:Youcannotusebackquote ` orsinglequote ' .



PluginsChoosewhichpluginsyouwanttoenableinthissectionofthePre-ProvisionedRabbitMQtab.

Youmustleavetherabbitmq_managementpluginenabledforthisproducttowork.

FormoreinformationaboutRabbitMQplugins,seetheRabbitMQdocumentation .

HAProxyPortsEntertheportsHAProxyshouldloadbalancetotheRabbitMQnodesinthissectionofthePre-ProvisionedRabbitMQtab:

Allthedefaultportsofalltheavailablepluginsareload-balancedbydefault.However,ifyouinstallextraprotocolplugins,orprovideacustomconfigurationthatchangestheportsRabbitMQlistenson,thenyoumustupdatethelistofload-balancedports.

Youmustleavethemanagementpluginlisteningonport 15672 andloadbalancethatport.

IfyouchangethetopologyofyourRabbitMQcluster,theHAProxyisautomaticallyreconfiguredduringthedeployment.

SSL(Optional)ProvideSSLcertificatesandkeysforusebytheRabbitMQclusterinthissectionofthePre-ProvisionedRabbitMQtab:

Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.


https://www.rabbitmq.com/plugins.html

SSLissimultaneouslyprovidedforAMQPS,STOMPandMQTT.NootherpluginsareautomaticallyconfiguredforusewithSSL.

IfyouprovideSSLkeysandcertificates,non-SSLsupportisdisabled.IfyoupreviouslydeployedthisservicewithoutSSLsupportandhaveappsconnectedtotheservice,theseappslosetheirconnectionsandmustreconnectusingSSL.

SSLsettingsareappliedequallyacrossallVMsinthecluster.

YoucanprovidemorethenoneCAcertificate.

FormoreinformationaboutSSLsupport,seetheRabbitMQdocumentation .

ErlangCookie(Optional)ProvideanErlangcookietobeusedbytheclusterinthissectionofthePre-ProvisionedRabbitMQtab.ThisisusefulifyouwanttoconnectdirectlytotheRabbitMQcluster,forexamplewith rabbitmqctl ,ortoconnectotherVMsrunningErlang.

KnownIssueswithErlangCookie

TherearethreeknownissuesassociatedwiththeErlangcookie:

SecurityIssuewiththeTileGeneratedErlangCookie

ClusterScalingKnownIssue

ChangingtheErlangCookieValueKnownIssue

Note:Leavingthisfieldblankisasecurityrisk.SeeSecurityIssuewiththeTileGeneratedErlangCookiebelow.


https://www.rabbitmq.com/ssl.html

SecurityIssuewiththeTileGeneratedErlangCookie

IfyouleavetheErlangcookiefieldblank,thetilegeneratesthecookieinawaythatcanbereverse-engineered.Formoreinformationaboutthissecurityissue,seeCVE-2018-1279:RabbitMQclustercompromiseduetodeterministicallygeneratedcookie .

Toavoidthisissue,settheErlangcookietoasecurepasswordvalue.Thisrequiresclusterdowntime,seeChangingtheErlangCookieValueKnownIssuebelow.

ClusterScalingKnownIssue

IfyouhavenotsettheErlangcookieandyouwanttoscaleoutyourclustersizewithoutdowntime,followthesesteps:

1. Followthestepsinthelinkbelow,uptoandincludingthesectionLogintotheBOSHDirector:AdvancedTroubleshootingwiththeBOSHCLI

2. Runthefollowingcommand:boshsshrabbitmq-server/0

3. Runthefollowingcommands:

sudo-iecho$(cat/var/vcap/store/rabbitmq/.erlang.cookie)

4. PastethevaluereturnedfromthelastcommandintotheErlangcookiefieldinthePre-ProvisionedRabbitMQtab.Thisfieldisshownabove.

5. Toincreasethesizeofyourcluster,navigatetotheResourceConfigtaband,inthefirstrow,raisethevalueforthenumberofInstancesoftheRabbitMQnode.

6. ReturntotheOpsManagerInstallationDashboard,andclickApplyChanges.

ChangingtheErlangCookieValueKnownIssue

ChangingtheErlangcookievaluerequiresclusterdowntime.Pivotalstronglyrecommendsthatyoudonotchangeanythingelseduringthistime,becauseitispossiblefortheconfigurationtobeinconsistentlyappliedduringthisprocess.

Thedeploymentmightfailafterthisprocess.Ifso,redeployingfixestheissue.

RabbitMQConfiguration(Optional)Provideafull rabbitmq.config filebypastingitscontentsintheRabbitMQconfigurationfieldinthePre-ProvisionedRabbitMQtab.Thisrabbitmq.config fileisthenprovidedtoallthenodesinthecluster.

TheinputinthisfieldmustbeBase64encoded.

Forexample,supposeyouwanttoconfigurethe rates_mode ofthe rabbitmq_management statsbelow:

[{rabbitmq_management,[{rates_mode,detailed}]}].

Note:BOSHtellsyouthatthecookiehaschanged—thisisbecausethedefaultvalueinthemanifestisempty,whichresultsinanauto-generatedcookie.However,thevalueofthecookieontheserverremainsthesame,sotheknownissuebelowdoesnotapply.



https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#gather

1. EncodethefileintoBase64:

WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K

2. PastetheaboveintotheRabbitMQconfigurationfield:

Youcanseeanexample rabbitmq.config filehere .FormoreinformationabouttheRabbitMQconfiguration,seetheRabbitMQdocumentation .

TLSSupportConfigureTLSinthissectionofthePre-ProvisionedRabbitMQtab:

TLSv1.0isdisabledbydefault,duetosecurityissues.

TLSv1.1andv1.2areenabledbydefaultandcanbeturnedonandoff.

ExternalLoadBalancer(Optional)EnteraDNSnameorIPaddressofanexternalloadbalancertobereturnedinthebindingcredentials( VCAP_SERVICES )toappdevelopers.EnterthisinthissectionofthePre-ProvisionedRabbitMQtab:

Ifyouconfigureanexternalloadbalancer,toavoidanunnecessaryVMdeployment,intheResourceConfigtabsettheHAProxyforRabbitMQinstancecountto0.

MetricsPollingIntervalThemetricspollingintervalissetinthissectionofthePre-ProvisionedRabbitMQtab:

Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.


https://github.com/rabbitmq/rabbitmq-server/blob/stable/docs/rabbitmq.config.example

https://www.rabbitmq.com/configure.html

Changingthissettingaffectsalldeployedinstances.

DiskFreeAlarmLimitChoosehowmuchdiskspaceRabbitMQattemptstokeepfreeatanygiventimeinthissectionofthePre-ProvisionedRabbitMQtab:

RabbitMQperiodicallychecksifthereissufficientfreespaceondisk.Ifthereisnot,RabbitMQtemporarilystopsacceptingnewmessages.Thisgivesyourappstimetoconsumeexistingmessages,andthusfreeupsomediskspace.TheRabbitMQtileprovidesfouroptionsforthisvalue:

50MBistheminimumvalue.(NotRecommended)Selecting50MBisnotrecommendedandcancausedataloss.Formoreinformation,seeDangersofSettingThisValueTooLowandWhentoUsethe50MBValuebelow.

100%MemoryensuresthatatthetimewhenRabbitMQcheckstheavailabledisk,theremustbeenoughspaceforRabbitMQtopageallmemory-basedmessagesouttodisk.

150%Memoryisrecommended.Thisisbecauseitispossiblethatinbetweendisk-spacechecks,RabbitMQmay:

Writepersistentmessagestodisk(usingupsomediskspace).Acceptmorememory-basedmessagesintovariousqueues.Pageallmemory-basedmessagestodisk.

Intheabovesituations,RabbitMQmightrequiremorefreediskthanithasmemory.

200%MemoryisaconservativevalueusedwhentheoperatorwantshigherconfidencethatRabbitMQneverrunsoutofdiskspace.

Formoreinformationaboutdiskalarms,seetheRabbitMQdocumentation .

DangersofSettingThisValueTooLow

IfthediskofagivenRabbitMQnodecompletelyfillswhileRabbitMQisrunning,thatnodecrashes.Thiscanleadtodataloss,andlossofavailability.

RabbitMQreservestherighttopageanyandallmessagesinmemory(eventransientmessages)todiskatanytime.YoumustsetyourDiskfreealarmlimithighenoughtoensurethatRabbitMQalwayshasatleastenoughspacetodothis.

DisadvantagesofSettingThisValueTooHigh

IfyousetyourDiskfreealarmlimittoavaluelargerthanthesizeofyourpersistentdisk,thenRabbitMQisnotabletofreeupenoughdiskspacetoacceptnewmessages.EnsurethatyouhavealargeenoughdisktopersistallthemessagesyouintendtopersistwhilealsoleavingenoughspacefreetosatisfytheDiskfreealarmlimitthatyouchoose.

WhentoUsethe50MBValue

Pivotaldoesnotrecommendusingthisvalueinproduction.However,ifyouareexperimentingwithadevelopmentenvironmentyoumightwanttouseasmalldisktokeepdowncosts,thoughthisincreasesthepossibilitythatRabbitMQcrashesandlosesdata.

NetworkingTospecifystaticIPaddresses,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.

2. IntheSettingstab,selectNetworking.


https://www.rabbitmq.com/disk-alarms.html

3. Configurethefieldsasfollows:

Pre-ProvisionedHAProxyStaticIPs:Enteracomma-delimitedlistofIPaddressesgroupedintheorderofAZsconfigured.TheseIPaddressesareassignedtoyourPre-ProvisionedHAproxynodes.

Pre-ProvisionedRabbitMQServerStaticIPs:Enteracomma-delimitedlistofIPaddressesgroupedintheorderofAZsconfigured.TheseIPaddressesareassignedtoyourPre-ProvisionedRabbitMQServernodes.

Pre-ProvisionedServiceBrokerStaticIP:EnteranIPaddress.ThisaddressmustbeinthenetworkrangeofthenetworknamespecifiedintheNetworkdrop-downlistintheAssignAZsandNetworkstabandmustnotbeintheServiceNetwork.ThisIPaddressisassignedtoyourPre-ProvisionedServiceBrokernode.

4. ClickSave.

SyslogToenablemonitoringforRabbitMQforPCF,operatorsforwardthesyslogbydesignatinganexternalsyslogendpointforRabbitMQcomponentlogmessages.ThisendpointservesastheinputtoamonitoringplatformsuchasDatadog,Papertrail,orSumoLogic.

TospecifythedestinationforRabbitMQforPCFlogmessages,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.

2. IntheRabbitMQtile,clicktheSettingstab.

Note:Ifanyoftheabovefieldsareleftblank,BOSHallocatesanIPaddress.


3. ClickSyslog.

4. ConfigurethefieldsontheSyslogpaneasfollows:

Option Description

Syslogaddress IPorDNSaddressofthesyslogserver

Syslogport Portofthesyslogserver

Transportprotocol

Transportprotocolofthesyslogserver.Oneof udp , tcp , relp .

Formatforlogs Formatforlogs.Pivotalrecommends RFC 5424 ,but Legacy Format canbeusedforcompatibilityreasons.

EnableTLS EnableTLStothesyslogserver.

PermittedPeerIfthereareseveralpeerserversthatcanrespondtoremotesyslogconnections,thenyoumayprovideawildcardinthedomain,suchas *.example.com .

CustomCACertificate

Iftheservercertificateisnotsignedbyaknownauthority,forexample,aninternalsyslogserver,providetheCAcertificateofthelogmanagementserviceendpoint.


5. ClickSave.

6. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoredeploywiththechanges.

GlobalSettingsFollowthestepsbelowtoenabletheshareableinstancesbetafeature.Sharingaserviceinstancebetweenspaces,allowsappsindifferentspacestosharedatabases,messagingqueues,andmanyothertypesofservices.Formoreinformation,seeSharingServiceInstances(Experimental) .

1. ClickGlobalSettings.

2. Select(Beta)ShareableInstancestoenablethebetafeatureforsharinginstances.

3. ClickSave.

Errands(Optional)IntheErrandstab,choosethedefaultsforwhenerrandsrun.

Errandscanbethoughtofastasks.Forexample,whendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofpost-deployerrands.AnexampleistheSmokeTestserrand,whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.

Youcandecidewhethertorunpost-deployerrandsbytogglingthemonoroffbeforeyouclickApplyChangestoupdateaconfigurationintheOpsManagerInstallationDashboard:

Thisisaone-timeactionbeforeanupdate.YoucanchangetheabovedefaultsintheErrandstab,aswellasthedefaultsforpre-deleteerrands.

WARNING:ThisisabetafeaturebasedonanexperimentalfeatureinCloudFoundry.Usethefeatureatyourownriskinnon-productionenvironments.

Important:InRabbitMQforPCFv1.9.0andlater,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,because


https://docs.pivotal.io/pivotalcf/devguide/services/sharing-instances.html

Formoreinformationonerrandrunrules,seeErrandRunRules .

Post-DeployErrands

Errand Description

BrokerRegistrar

Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace

SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTestsbelow.

RegisterOn-DemandServiceBroker

Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.

On-DemandInstanceSmokeTests

Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTests.


On-demandinstancesareupdatedandredeployediftherearechangestotheDedicatedInstancesettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoDedicatedInstancesettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.

Pre-ProvisionedInstanceSmokeTestsSmoketestsrunasapost-deploymenterrand.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.

Pre-DeleteErrandsPre-deleteerrandsrunafteranoperatorchoosestodeleteaproductintheOpsManagerInstallationDashboard,butbeforeOpsManagerfinishesdeletingtheproduct.

Errand Description

BrokerDeregistrar Removesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings

DeleteAllServiceInstancesUnbindsanddeletesexistingdedicatedserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.

DeregisterOn-DemandServiceBroker

Removestheon-demandRabbitMQservicefromtheMarketplace

StemcellForOpsManagerv2.0:

1. ClickStemcell.

2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .

ForOpsManagerv2.1:

1. FollowtheprocedureoutlinedinImportingandManagingStemcells .

ApplyConfigurationandCompletetheInstallation

thesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmightfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.


https://docs.pivotal.io/tiledev/tile-errands.html#run-rules

https://docs.pivotal.io/pivotalcf/customizing/pcf-aws-manual-er-config.html#stemcell

https://docs.pivotal.io/pivotalcf/customizing/azure-er-config.html#stemcell

https://docs.pivotal.io/pivotalcf/customizing/gcp-er-config.html#stemcell

https://docs.pivotal.io/pivotalcf/customizing/config-er-vmware.html#stemcell

https://docs-pcf-staging.cfapps.io/pivotalcf/2-1/opsguide/managing-stemcells.html

ReturntotheOpsManagerInstallationDashboardandclickApplyChangestocompletetheinstallationofRabbitMQforPCF.

OtherConfigurationTopics

ConnectingtoaHighlyAvailableRabbitMQClusterTheRabbitMQtile,allowsforahighlyavailableclusterthroughmultipleHAProxynodes.The hostnames , uris and hosts propertieshavebeenaddedandshouldbeusedinpreferenceovertheequivalentsingularproperties.Thesingularpropertiesaremaintainedforbackwardscompatibilityandalwayscontainthefirstvaluefromtheequivalentpluralproperty.Thesingularpropertieswilleventuallybedeprecated.

Forexample,withtwoHAProxyjobsdeployed,thefollowingpropertieswillbepresent:

"hostname":"10.0.0.41","hostnames":["10.0.0.41","10.0.0.51"]

Porttoprotocolmappings15672=Managementdashboard

5672=RabbitMQ

5671=RabbitMQSSL

1883=MQTT

8883=MQTTSSL

61613=STOMP

61614=STOMPSSL

15674=WebSTOMP

4567=RabbitMQServiceBroker

3457-3459=CFLoggregator

SecurityGroupsToenableaccesstotheRabbitMQtileservice,youmustensureyoursecuritygroupallowsaccesstotheHAProxyandRabbitMQServiceBrokerVMsconfiguredinyourdeployment.YoucanobtaintheIPaddressesforthesefromtheOpsManagerStatuspagefortheRabbitMQtile.EnsurethefollowingportsareenabledforthoseVMs:

15672

5672

5671

1883

8883

61613

61614

15674

4567

3457-3459

ThefollowingisatemplateforconfiguringyourCloudFoundrysecuritygroups:[{"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"},{"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"}]


ApplicationSecurityGroupsToallowthisservicetohavenetworkaccess,youmustcreateApplicationSecurityGroups (ASGs).

ApplicationContainerNetworkConnectionsApplicationcontainersthatuseinstancesoftheRabbitMQservicerequirethefollowingoutboundnetworkconnections:

Destination Ports Protocol Reason

HAProxy IPs 5672 tcp ApplicationcontainersusingAMQP

HAProxy IPs 5671 tcp ApplicationcontainersusingAMQPoverSSL

HAProxy IPs 1883 tcp ApplicationcontainersusingMQTT

HAProxy IPs 8883 tcp ApplicationcontainersusingMQTToverSSL

HAProxy IPs 61613 tcp ApplicationcontainersusingSTOMP

HAProxy IPs 61614 tcp ApplicationcontainersusingSTOMPoverSSL

HAProxy IPs 61613 tcp ApplicationcontainersusingWebSTOMP

CreateanASGnamed rabbitmq-app-containers withtheaboveconfigurationandbindittoeither:

Theappropriatespace

The default-running ASGsetifyouwanttoprovideaccesstoallstartedapps.Thenrestartyourapps.

Ifyouareusinganexternalloadbalancer,orhavemorethanoneIPaddressforHAProxy,youmustalsocreateegressrulesforthese.Forexample:

[{"ports":"5671-5672","protocol":"tcp","destination":"10.10.10.10/32"}]

AssignedIPsRabbitMQforPCFdoesnotsupportchangingtheIPaddresseswhichhavebeenassignedtotheRabbitMQdeployments.Forexample,youcannotchangethesubnetintowhichtheRabbitMQclusterwasoriginallyprovisioned.Doingsocausesthedeploymenttofail.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

PreservingDynamicallyAssignedIPsYoucannotswitchfromdynamicallyassignedIPaddressestoadifferentsetofstaticIPaddresses.However,youcanconfigureOpsManagersothecurrentsetofdynamicallyassignedIPaddressesalwayscontinuetobeused.Thismightbeusefulwhenupgrading.

Todothis,followthesesteps:

1. GototheStatuspageintheRabbitMQtile.

2. TakenoteoftheIPaddressesfortheRabbitMQServerandHAProxyforRabbitMQjobs,intheordernodesappearintheUI.

3. GototheSettingspage,andclickNetworking.

4. EntertheIPaddressesyougotfromtheStatuspageasacomma-separatedlist.

Note:TheserviceisunusablewithoutApplicationSecurityGroups.


http://docs.pivotal.io/pivotalcf/adminguide/app-sec-groups.html


5. ClickSave.

RabbitMQServerSettingsthatCannotbeOverwrittenInallcases:

rabbit halt_on_upgrade_failure false

rabbitmq_mqtt subscription_ttl 1800000

log_levels [{connection,info}]

halt_on_upgrade_failure false

{rabbit, [ {collect_statistics_interval, 60000}] }

{rabbitmq_management, [ {rates_mode, none}] }

WhenSSLisenabled:

rabbit tcp_listeners []

rabbit ssl_listeners [5671]

rabbitmq_management listener [{port,15672},{ssl,false}]

rabbitmq_mqtt ssl_listeners [8883]

rabbitmq_stomp ssl_listeners [61614]


SmokeTestsRabbitMQforPCFrunsasetofsmoketestsduringinstallationtoconfirmsystemhealth.

SmokeTestStepsThesmoketestsperformthefollowingforeachavailableserviceplan:

1. Targetstheorg system andcreatesaspacetorunthetests.

2. DeploysaninstanceoftheCFRabbitMQExampleApp tothisspace

3. CreatesaRabbitMQserviceinstanceandbindsittotheCFRabbitMQExampleApp

4. ChecksthattheCFRabbitMQExampleAppcanwritetoandreadfromtheRabbitMQserviceinstance

5. Cleansupalldeployedapplicationandallitsservicebindings.Finally,thecfspaceisdeleted.

TroubleshootingIferrorsoccurwhilethesmoketestsrun,theyaresummarizedattheendoftheerrandlogoutput.Detailedlogscanbefoundwherethefailureoccurs.

Whenencounteringanerrorwhenrunningsmoketests,itcanbehelpfultosearchthelogforotherinstancesoftheerrorsummaryprintedattheendofthetests,forexample, FailedtotargetCloudFoundry .Lookoutfor TIP:... inthelogsnexttoanyerroroutputforfurthertroubleshootinghints.

Note:Smoketestsfailunlessyouenableglobaldefaultapplicationsecuritygroups(ASGs).YoucanenableglobaldefaultASGsbybindingtheASGtothe system orgwithoutspecifyingaspace.ToenableglobaldefaultASGs,use cfbind-running-security-

group.


https://github.com/pivotal-cf/rabbit-example-app

MonitoringandKPIsforPre-ProvisionedRabbitMQforPCFThistopicexplainshowtomonitorthehealthofthepre-provisionedversionoftheRabbitMQforPivotalCloudFoundry(PCF)serviceusingthelogs,metrics,andKeyPerformanceIndicators(KPIs)generatedbyRabbitMQforPCFcomponentVMs.

Pre-provisionedRabbitMQforPCFcomponentsgeneratemanyofthesamemetrics astheon-demandRabbitMQservicecomponents.

SeeLoggingandMetrics forgeneralinformationaboutloggingandmetricsinPCF.

SettingupSyslogForwardingOperatorscanenablelogforwardingbyconfiguringanexternalsyslogendpointforRabbitMQcomponentlogmessages.Forinstructionsonsettingupsyslogforwarding,seeSyslog.

Ifsyslogforwardingisenabled,logentrieswithtimestampsareavailablelocallyin /var/log/messages .Logsareavailableunder /var/vcap/sys/log/ whethersyslogforwardingisenabledornot.

LoggingFormatsWithpre-provisionedRabbitMQforPCFloggingconfigured,threetypesofcomponentgeneratelogs:theRabbitMQmessageservernodes,theservicebroker,andHAProxy.IfyouhavemultipleserverorHAProxynodes,youcanidentifylogsfromindividualnodesbytheirindex,whichcorrespondstotheindexoftheRabbitMQVMinstancesdisplayedinOpsManager:

ThelogsforRabbitMQservernodesfollowtheformat [job=rabbitmq-server-partition-GUID index=X]

ThelogsforHAProxynodesfollowtheformat [job=rabbitmq-haproxy-partition-GUID index=X]

ThelogsfortheRabbitMQservicebrokerfollowtheformat [job=rabbitmq-broker-partition-GUID index=X]

RabbitMQandHAProxyserverslogatthe info levelandcaptureerrors,warnings,andinformationalmessages.

Forusersfamiliarwithdocumentationforpreviousversionsofthetile,thetagweusedtocallthe app_name isnowcalledthe program_name .

Thegenericlogformatisasfollows:

<PRI>TIMESTAMPIP_ADDRESSPROGRAM_NAME[job=NAMEindex=JOB_INDEXid=JOB_ID]MESSAGE

Therawlogslooksimilartothefollowing:

<7>2017-06-28T16:06:10.733560+00:0010.244.16.133vcap.agent[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]2017/06/2816:06:10CEF:0|CloudFoundry|BOSH|1|agent_api|ssh|1|duser=director.be5a66bb-a9b4-459f-a0d3-1fc5c9c3ed79.be148cc6-91ef-4eed-a788-237b0b8c63b7src=10.254.50.4spt=4222shost=5ae233e0-ecc5-4868-9ae0-f9767571251b<86>2017-06-28T16:06:16.704572+00:0010.244.16.133useradd[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]newgroup:name=bosh_ly0d2rbjr,GID=1003<86>2017-06-28T16:06:16.704663+00:0010.244.16.133useradd[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]newuser:name=bosh_ly0d2rbjr,UID=1001,GID=1003,home=/var/vcap/bosh_ssh/bosh_ly0d2rbjr,shell=/bin/bash<86>2017-06-28T16:06:16.736932+00:0010.244.16.133usermod[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]add'bosh_ly0d2rbjr'togroup'admin'<86>2017-06-28T16:06:16.736964+00:0010.244.16.133usermod[job=rmqindex=0id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5]add'bosh_ly0d2rbjr'togroup'vcap'

LogssenttoexternalloggingtoolssuchasPapertrailmaybepresentedinadifferentformat.

Thefollowingtabledescribestheloggingtagsusedinthistemplate:

Tag Description

PRI Thisisavaluewhichinfuturewillbeusedtodescribetheseverityofthelogmessageandwhichfacilityitcamefrom.

TIMESTAMPThisisthetimestampofwhenthelogisforwarded,forexample, 2016-08-24T05:14:15.000003Z .Thetimestampvalueistypicallyslightlyafterwhenthelogmessagewasgenerated.

IP_ADDRESS TheinternalIPaddressofserveronwhichthelogmessageoriginated

PROGRAM_NAMEProcessnameoftheprogramthegeneratedthemessage.Sameas app_name beforev1.9.0.Formoreinformationaboutprogramname,seeRabbitMQProgramNamesbelow.

NAME TheBOSHinstancegroupname(forexample, rabbitmq_server )

JOB_INDEX BOSHjobindex.Usedtodistinguishbetweenmultipleinstancesofthesamejob.

JOB_IDBOSHVMGUID.ThisisdistinctfromtheCIDdisplayedintheOpsManagerStatustab,whichcorrespondstotheVMIDassignedbytheinfrastructureprovider.


https://docs.pivotal.io/rabbitmq-cf/1-12/monitor.html

http://docs.pivotal.io/pivotalcf/loggregator/index.html

MESSAGE Thelogmessagethatappears

RabbitMQProgramNames

ProgramName Description

rabbitmq_server_cluster_check ChecksthattheRabbitMQclusterishealthy.Runsaftereverydeploy.

rabbitmq_server_node_check ChecksthattheRabbitMQnodeishealthy.Runsaftereverydeploy.

rabbitmq_route_registrar_stderrRegisterstherouteforthemanagementAPIwiththeGorouterinyourPivotalApplicationService(PAS)orElasticRuntimedeployment.

rabbitmq_route_registrar_stdout RegisterstherouteforthemanagementAPIwiththeGorouterinyourPASorElasticRuntimedeployment.

rabbitmq_server TheErlangVMandRabbitMQapps.Logsmayspanmultiplelines.

rabbitmq_server_drain ShutsdowntheErlangVMandRabbitMQapps.RunsaspartoftheBOSHlifecycle.

rabbitmq_server_http_api_access AccesstotheRabbitMQmanagementUI.

rabbitmq_server_init StartstheErlangVMandRabbitMQ.

rabbitmq_server_post_deploy_stderr Runsthenodecheckandclustercheck.Runsaftereverydeploy.

rabbitmq_server_post_deploy_stdout Runsthenodecheckandclustercheck.Runsaftereverydeploy.

rabbitmq_server_pre_start Runsbeforetherabbitmq-serverjobisstarted.

rabbitmq_server_sasl Supervisor,progress,andcrashreportingfortheErlangVMandRabbitMQapps.

rabbitmq_server_shutdown_stderr StopstheRabbitMQappandErlangVM.

rabbitmq_server_shutdown_stdout StopstheRabbitMQappandErlangVM.

rabbitmq_server_startup_stderr StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.

rabbitmq_server_startup_stdout StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.

rabbitmq_server_upgrade ShutsdownErlangVMandRabbitMQappifrequiredduringanupgrade.

MetricsMetricsareregularly-generatedlogmessagesthatreportmeasuredcomponentstates.Themetricspollingintervaldefaultsto30seconds.ThemetricspollingintervalisaconfigurationoptionontheRabbitMQtile(Settings>RabbitMQ).Theintervalsettingappliestoallcomponentsdeployedbythetile.

Metricsarelong,singlelinesoftextthatfollowtheformat:

origin:"p-rabbitmq"eventType:ValueMetrictimestamp:1441188462382091652deployment:"cf-rabbitmq"job:"cf-rabbitmq-node"index:"0"ip:"10.244.3.46"valueMetric:<name:"/p-rabbitmq/rabbitmq/system/memory"value:1024unit:"MB">

PartitionIndicatorAnewmetrichasbeenintroducedtohelptoidentifynetworkpartitions.Essentiallyitexposeshowmanynodeseachnodeknows.Whenanodeisinpartitiontheonlynodethatitrecognizesisitselfandthatisagoodindicationthatthatnodemightbeinapartition.

Anexampleofthatmetricsis:

origin:"p-rabbitmq"eventType:ValueMetrictimestamp:1441188462382091652deployment:"cf-rabbitmq"job:"cf-rabbitmq-node"index:"0"ip:"10.244.3.46"valueMetric:<name:"/p-rabbitmq/rabbitmq/erlang/reachable_nodes"value:3unit:"count">

Monitorscanbecreatedtoemitalertsincaseaclusterseemstobeinapartition.Ametricsisemittedforeachnodeinthecluster.Forexample:inathree-nodeclusteramonitorcanexpecttohaveatotalof9(nine)sinceeachnodeisexpectedtoemit3(2reachablenodesanditself).Otherwise,analertcanbesenttotheteam.

RecoveringfromanetworkpartitionPleaserefertotheoficialRabbitMQguidetounderstandhowtorecoverfromanetworkpartition:https://www.rabbitmq.com/partitions.html



KeyPerformanceIndicatorsKeyPerformanceIndicators(KPIs)forRabbitMQforPCFaremetricsthatoperatorsfindmostusefulformonitoringtheirRabbitMQservicetoensuresmoothoperation.KPIsarehigh-signal-valuemetricsthatcanindicateemergingissues.KPIscanberawcomponentmetricsorderivedmetricsgeneratedbyapplyingformulastorawmetrics.

PivotalprovidesthefollowingKPIsasgeneralalertingandresponseguidancefortypicalRabbitMQforPCFinstallations.Pivotalrecommendsthatoperatorscontinuetofine-tunethealertmeasurestotheirinstallationbyobservinghistoricaltrends.Pivotalalsorecommendsthatoperatorsexpandbeyondthisguidanceandcreatenew,installation-specificmonitoringmetrics,thresholds,andalertsbasedonlearningfromtheirowninstallations.

ForalistofallRabbitMQforPCFrawcomponentmetrics,seeComponentMetricsReference.

ComponentHeartbeatsKeyRabbitMQforPCFcomponentsperiodicallyemitheartbeatmetrics:theRabbitMQservernodes,HAProxynodes,andtheServiceBroker.TheheartbeatsareBooleanmetrics,where 1 meansthesystemisavailable,and 0 ortheabsenceofaheartbeatmetricmeanstheserviceisnotrespondingandshouldbeinvestigated.

ServiceBrokerHeartbeat

p-rabbitmq.service_broker.heartbeat

Description

RabbitMQServiceBroker is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:IftheServiceBrokerdoesnotemitheartbeats,thisindicatesthatitisoffline.TheServiceBrokerisrequiredtocreate,update,anddeleteserviceinstances,whicharecriticalfordependenttilessuchasSpringCloudServicesandSpringCloudDataFlow.

Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)

Recommendedmeasurement Averageoverlast5minutes

RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1

RecommendedresponseChecktheRabbitMQServiceBrokerlogsforerrors.YoucanfindthisVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningthefollowingcommand:bosh -d service-instance_GUID vms

HAProxyHeartbeat

p-rabbitmq.haproxy.heartbeat

Description

RabbitMQHAProxy is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:IftheHAProxydoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireHAProxy.




ChecktheRabbitMQHAProxylogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwith


Recommendedresponse BOSHandrunningthefollowingcommand,whichlists HAProxy_GUID :bosh -d service-instance_GUID vms

ServerHeartbeat

p-rabbitmq.rabbitmq.heartbeat

Description

RabbitMQServer is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:Iftheserverdoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireRabbitMQServer.




RecommendedresponseChecktheRabbitMQServerlogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommands,whichlists rabbitmq :bosh -d service-instance_GUID vms

RabbitMQServerKPIsThefollowingKPIsfromtheRabbitMQservercomponent:

FileDescriptors

p-rabbitmq.rabbitmq.system.file_descriptors

Description

Filedescriptorsconsumed.

Use:Ifthenumberoffiledescriptorsconsumedbecomestoolarge,theVMmaylosetheabilitytoperformdiskIO,whichcancausedataloss.

Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)


RecommendedalertthresholdsYellowwarning:>250000Redcritical:>280000

Recommendedresponse

Thedefault ulimit forRabbitMQforPCFis300000.Ifthismetricismetorexceededforanextendedperiodoftime,consideroneofthefollowingactions:

ScalingtherabbitnodesinthetileResourceConfigpane.

Reducetheloadontheserver

ErlangProcesses

p-rabbitmq.rabbitmq.erlang.erlang_processes

Erlang processesconsumedbyRabbitMQ,whichrunsonanErlangVM.

Note:Thisassumesnon-persistentmessagesarehandledbyretriesorsomeotherlogicbytheproducers.


https://www.erlang.org/docs

DescriptionUse:Thisisthekeyindicatoroftheprocessingcapabilityofanode.

Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)



RecommendedresponseThedefaultErlangprocesslimitinRabbitMQforPCFv1.6andlateris1,048,816.Ifthismetricmeetsorexceedstherecommendedthresholdsforextendedperiodsoftime,considerscalingtheRabbitMQnodesinthetileResourceConfigpane.

BOSHSystemHealthMetricsTheBOSHlayerthatunderliesPCFgenerates healthmonitor metricsforallVMsinthedeployment.AsofPCFv2.0,thesemetricsareincludedintheLoggregatorFirehosebydefault.Formoreinformation,seeBOSHSystemMetricsAvailableinLoggregatorFirehose inPivotalApplicationService(PAS)ReleaseNotes.

AllBOSH-deployedcomponentsgeneratethesystemhealthmetricsbelow.ThesecomponentmetricsarefromRabbitMQforPCFcomponents,andserveasKPIsfortheRabbitMQforPCFservice.

RAM

system.mem.percent

Description

RAMbeingconsumedbythe p-rabbitmq VM.

Use:RabbitMQisconsideredtobeinagoodstatewhenithaslittleornomessages.Inotherwords,“anemptyrabbitisahappyrabbit.”Alertingonthismetriccanindicatethattherearetoofewconsumersorappsthatreadmessagesfromthequeue.

HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsRAMforthepasttenminutes.

Origin:BOSHHMType:percentFrequency:30s(default),10s(configurableminimum)



Recommendedresponse Addmoreconsumerstodrainthequeueasfastaspossible.

CPU

system.cpu.percent

Description

CPUbeingconsumedbythe p-rabbitmq VM.

Use:AnodethatexperiencescontextswitchingorhighCPUusagewillbecomeunresponsive.Thisalsoaffectstheabilityofthenodetoreportmetrics.

HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.




https://docs.pivotal.io/pivotalcf/2-0/pcf-release-notes/runtime-rn.html#bosh-metrics

Recommendedalertthresholds Yellowwarning:>60Redcritical:>75

Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.

EphemeralDisk

system.disk.percent

Description

EphemeralDiskbeingconsumedbythe p-rabbitmq VM.

Use:Ifsystemdiskfillsup,therearetoofewconsumers.






PersistentDisk

persistent.disk.percent

Description

PersistentDiskbeingconsumedbythe p-rabbitmq VM.

Use:Ifsystemdiskfillsup,therearetoofewconsumers.






ComponentMetricReferenceRabbitMQforPCFcomponentVMsemitthefollowingrawmetrics.Thefullnameofthemetricfollowstheformat: /p-rabbitmq/COMPONENT/METRIC-NAME

RabbitMQServerMetricsRabbitMQforPCFmessageservercomponentsemitthefollowingmetrics.

FullName Unit Description

/p-rabbitmq.rabbitmq.heartbeat boolean IndicateswhethertheRabbitMQserverisavailableandabletorespondtorequests

/p-rabbitmq/rabbitmq/erlang/erlang_processes

count ThenumberofErlangprocesses

/p-rabbitmq/rabbitmq/system/memory MB ThememoryinMBusedbythenode


/p-rabbitmq/rabbitmq/system/mem_alarm boolean Indicatesifthememoryalarmwentoff

/p-rabbitmq/rabbitmq/system/disk_free_alarm boolean Indicatesifthediskfreealarmwentoff

/p-rabbitmq/rabbitmq/system/disk_free MB Thediskspaceavailableonthenode

/p-rabbitmq/rabbitmq/connections/count count Thetotalnumberofconnectionstothenode

/p-rabbitmq/rabbitmq/consumers/count count Thetotalnumberofconsumersregisteredinthenode

/p-rabbitmq/rabbitmq/messages/delivered count Thetotalnumberofmessageswiththestatus deliver_get onthenode

/p-rabbitmq/rabbitmq/messages/delivered_noack

count Thenumberofmessageswiththestatus deliver_noack onthenode

/p-rabbitmq/rabbitmq/messages/delivered_rate

rateTheratepersecondatwhichmessagesarebeingdeliveredtoconsumersorclientsonthenode

/p-rabbitmq/rabbitmq/messages/published count Thetotalnumberofmessageswiththestatus publish onthenode

/p-rabbitmq/rabbitmq/messages/published_rate

rate Theratepersecondatwhichmessagesarebeingpublishedbythenode

/p-rabbitmq/rabbitmq/messages/redelivered count Thetotalnumberofmessageswiththestatus redeliver onthenode

/p-rabbitmq/rabbitmq/messages/redelivered_rate rate

Theratepersecondatwhichmessagesaregettingthestatus redeliver onthenode

/p-rabbitmq/rabbitmq/messages/get_no_ack count Thenumberofmessageswiththestatus get_no_ack onthenode

/p-rabbitmq/rabbitmq/messages/get_no_ack_rate

rate Theratepersecondatwhichmessagesgetthestatus get_no_ack onthenode

/p-rabbitmq/rabbitmq/messages/pending count Thenumberofmessageswiththestatus messages_unacknowledged onthenode

/p-rabbitmq/rabbitmq/messages/depth countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready onthenode

/p-rabbitmq/rabbitmq/system/file_descriptors

count Thenumberofopenfiledescriptorsonthenode

/p-rabbitmq/rabbitmq/exchanges/count count Thetotalnumberofexchangesonthenode

/p-rabbitmq/rabbitmq/messages/available count Thetotalnumberofmessageswiththestatus messages_ready onthenode

/p-rabbitmq/rabbitmq/queues/count count Thenumberofqueuesonthenode

/p-rabbitmq/rabbitmq/channels/count count Thenumberofchannelsonthenode

/p-rabbitmq/rabbitmq/vhosts/count count Thenumberofvhosts

/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/consumers

count Thenumberofconsumerspervirtualhostperqueue

/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/depth

countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready pervirtualhostperqueue

HAProxyMetricsRabbitMQforPCFHAProxycomponentsemitthefollowingmetrics.

NameSpace Unit Description

/p-rabbitmq.haproxy.heartbeat booleanIndicateswhethertheRabbitMQHAProxycomponentisavailableandabletorespondtorequests

/p-rabbitmq/haproxy/health/connections

count Thetotalnumberofconcurrentfront-endconnectionstotheserver

/p-rabbitmq/haproxy/backend/qsize/amqp

size ThetotalsizeoftheAMQPqueueontheserver

/p-rabbitmq/haproxy/backend/retries/amqp count ThenumberofAMQPretriestotheserver

/p-rabbitmq/haproxy/backend/ctime/amqp

time ThetotaltimetoestablishtheTCPAMQPconnectiontotheserver



SettingLimitsforOn-DemandServiceInstancesOn-demandprovisioningisintendedtoaccelerateappdevelopmentbyeliminatingtheneedfordevelopmentteamstorequestandwaitforoperatorstocreateaserviceinstance.However,tocontrolcosts,operationsteamsandadministratorsmustensureresponsibleuseofresources.

Thereareseveralwaystocontroltheprovisioningofon-demandserviceinstancesbysettingvariousquotasattheselevels:

Global

Plan

Org

Space

Afteryousetquotas,youcan:

ViewCurrentOrgandSpace-levelQuotas

MonitorQuotaUseandServiceInstanceCount

CalculateResourceCostsforOn-DemandPlans

CreateGlobal-levelQuotasEachPivotalCloudFoundry(PCF)servicehasaseparateservicebroker.Aglobalquotaattheservicelevelsetsthemaximumnumberofserviceinstancesthatcanbecreatedbyagivenservicebroker.Ifaservicehasmorethanoneplan,thenthenumberofserviceinstancesforallplanscombinedcannotexceedtheglobalquotafortheservice.

TheoperatorsetsaglobalquotaforeachPCFserviceindependently.Forexample,ifyouhaveRedisforPCFandRabbitMQforPCF,youmustsetaseparateglobalservicequotaforeachofthem.

Whentheglobalquotaisreachedforaservice,nomoreinstancesofthatservicecanbecreatedunlessthequotaisincreased,orsomeinstancesofthatservicearedeleted.

CreatePlan-levelQuotasAservicemayofferoneormoreplans.Youcansetaseparatequotaperplansothatinstancesofthatplancannotexceedtheplanquota.Foraservicewithmultipleplans,thetotalnumberofinstancescreatedforallplanscombinedcannotexceedtheglobalquotafortheservice.

Whentheplanquotaisreached,nomoreinstancesofthatplancanbecreatedunlesstheplanquotaisincreasedorsomeinstancesofthatplanaredeleted.

CreateandSetOrg-levelQuotasAnorg-levelquotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesanorganizationcancreatewithinPCF.Forexample,ifyousetyourorg-levelquotato100,developerscancreateupto100serviceinstancesinthatorgusinganycombinationofPCFservices.

Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedintheorgunlessthequotaisincreasedorsomeserviceinstancesaredeleted.

Tocreateandsetanorg-levelquota,dothefollowing:

1. Runthiscommandtocreateaquotaforserviceinstancesattheorglevel:

cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans

Where:

QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg


Forexample:

cfcreate-quotamyquota-m1024mb-i16gb-r30-s50--allow-paid-service-plans

2. Associatethequotayoucreatedabovewithaspecificorgbyrunningthefollowingcommand:

cf set-quota ORG-NAME QUOTA-NAME

Forexample:

cfset-quotadev_orgmyquota

Formoreinformationonmanagingorg-levelquotas,seeCreatingandModifyingQuotaPlans .

CreateandSetSpace-levelQuotasAspace-levelservicequotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesthatcanbecreatedwithinagivenspaceinPCF.Forexample,ifyousetyourspace-levelquotato100,developerscancreateupto100serviceinstancesinthatspaceusinganycombinationofPCFservices.

Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedinthespaceunlessthequotaisupdatedorsomeserviceinstancesaredeleted.

Tocreateandsetaspace-levelquota,dothefollowing:

1. Runthefollowingcommandtocreatethequota:

cf create-space-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans

Where:

QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg

Forexample:

cfcreate-space-quotamyspacequota-m1024mb-i16gb-r30-s50--allow-paid-service-plans

2. Associatethequotayoucreatedabovewithaspecificspacebyrunningthefollowingcommand:

cf set-space-quota SPACE-NAME QUOTA-NAME

Forexample:

cfset-space-quotamyspacemyspacequota

Formoreinformationonmanagingspace-levelquotas,seeCreatingandModifyingQuotaPlans .

ViewCurrentOrgandSpace-levelQuotasTovieworgquotas,runthefollowingcommand.

cforgORG-NAME

Toviewspacequotas,runthefollowingcommand:

cfspaceSPACE-NAME


https://docs.pivotal.io/pivotalcf/1-12/adminguide/quota-plans.html


Formoreinformationonmanagingorgandspace-levelquotas,seetheCreatingandModifyingQuotaPlans .

MonitorQuotaUseandServiceInstanceCountService-levelandplan-levelquotause,andtotalnumberofserviceinstances,areavailablethroughtheon-demandbrokermetricsemittedtoLoggregator.Thesemetricsarelistedbelow:

MetricName Description

on-demand-broker/SERVICE-NAME/quota_remaining Quotaremainingforallinstancesacrossallplans

on-demand-broker/SERVICE-NAME/PLAN-NAME/quota_remaining

Quotaremainingforaspecificplan

on-demand-broker/SERVICE-NAME/total_instances Totalinstancescreatedacrossallplans

on-demand-broker/SERVICE-NAME/PLAN-NAME/total_instances

Totalinstancescreatedforaspecificplan

CalculateResourceCostsforOn-DemandPlansOn-demandplansusededicatedVMs,disks,andvariousotherresourcesfromanIaaS,suchasAWS.Tocalculatemaximumresourcecostforplansindividuallyorcombined,youmultiplythequotabythecostoftheresourcesselectedintheplanconfiguration(s).ThespecificcostsdependonyourIaaS.

TheimagebelowshowsanexamplethatincludestheVMtypeandpersistentdiskselectedfortheserverVMs,aswellasthequotaforthisplan.

Note:Quotametricsarenotemittedifnoquotahasbeenset.

Important:Althoughoperatorscanlimiton-demandinstanceswithplanquotasandaglobalquota,asdescribedintheabovetopics,IaaSresourceusagestillvariesbasedonthenumberofon-demandinstancesprovisioned.



CalculateMaximumResourceCostPerOn-DemandPlanTocalculatethemaximumcostofVMsandpersistentdiskforeachplan,dothefollowingcalculation:

planquotaxcostofselectedresources

Forexample,ifyouselectedtheoptionsintheaboveimage,youhaveselectedaVMtypemicroandapersistentdisktype20GB,andtheplanquotais15.TheVMandpersistentdisktypeshaveanassociatedcostfortheIaaSyouareusing.Therefore,tocalculatethemaximumcostofresourcesforthisplan,multiplythecostoftheresourcesselectedbytheplanquota:

(15xcostofmicroVMtype)+(15xcostof20GBpersistentdisk)=maxcostperplan

CalculateMaximumResourceCostforAllOn-DemandPlansTocalculatethemaximumcostforallplanscombined,addtogetherthemaximumcostsforeachplan.Thisassumesthatthesumofyourindividualplanquotasislessthantheglobalquota.

Hereisanexample:

(plan1quotaxplan1resourcecost)+(plan2quotaxplan2resourcecost)=maxcostforallplans

CalculateActualResourceCostofallOn-DemandPlansTocalculatethecurrentactualresourcecostacrossallyouron-demandplans:

1. Findthenumberofinstancescurrentlyprovisionedforeachactiveplanbylookingatthe total_instance metricforthatplan.

2. Multiplythe total_instance countforeachplanbythatplan’sresourcecosts.Recordthecostsforeachplan.

3. AddupthecostsnotedinStep2togetyourtotalcurrentresourcecosts.

Forexample:

(plan1total_instancesxplan1resourcecost)+(plan2total_instancesxplan2resourcecost)=currentcostforallplans


ControllingAccesstoServicePlansbyOrgIfyouwanttorestrictaccesstoaserviceplantoaspecificorg,followtheinstructionsbelow.

Youcanalsolimitthenumberofserviceinstancesbysettingquotas—forinstructions,seeSettingLimitsforOn-DemandInstances.

ChangeAccesstoServicePlans

Torestrictaccesstoaplanforaspecificorg,runthiscommand:

cfenable-service-accessp.rabbitmq-pPLAN_NAME-oORG_NAME

Forexample:

$cfenable-service-accessp.rabbitmq-pmy-cluster-plan-omy-dev-org

Formoreinformationabouttheabovecommand,seeAccessControl .

Note:Iftheplanyouarerestrictingiscurrentlyenabledforallorgs,youmustfirstDisableServiceAccessfortheplan,thengrantaccesstotheplantospecificorgs.UsetheCFServiceAccessfieldtodisableaccessintheserviceplanconfiguration.


https://docs.pivotal.io/pivotalcf/services/access-control.html

IsolatingClusterswiththeRabbitMQforPCFReplicator

OverviewRabbitMQforPCFReplicatorisatoolthatallowsyoutoinstallmultipleRabbitMQforPivotalCloudFoundry(PCF)tilesinasingleOpsManagerenvironment.Thisletsyourunmultiplepre-provisionedRabbitMQclustersthatareisolatedfromeachother.

Forexample,youmaywanttoisolatetheclusterservingSpringCloudServices(SCS)fromtheclusterservingappsintheMarketplace.Oryoumaywanttogiveacertainteamtheirowndedicated,pre-provisionedclusterthatyoumanageforthem.Forinformationonhowtoaccomplishthesescenarios,seeCommonUseCases.

CommonUseCasesTheimagebelowillustrateshowtoisolateSCSonRabbitMQforPCFfromclusteredandsinglenodeserviceinstancesdedicatedtodifferentteams.Inthisusecase:

TheunreplicatedRabbitMQforPCFtiledeploystwotypesofservices:

Apre-provisionedserviceisusedasabackingserviceforSCSAnon-demandserviceisusedtocreatethreecompletelyisolatedsinglenodeserviceinstances

Tworeplicatilesareusedtocreatetwodedicatedpre-provisionedclusters,witheachonededicatedtoaspecificteam.

ThesescenariosareexplainedbelowinRunningSCSonaDedicatedRabbitMQClusterandProvidingaPre-ProvisionedDedicatedCluster.


RunningSCSonaDedicatedRabbitMQClusterReplicaRabbitMQtilescannotbeusedtoprovideabackingserviceforSpringCloudServices(SCS)becauseSCSexpectsthattheserviceiscalledp-rabbitmq .Therefore,ifyouwanttoisolatetheRabbitMQclusterthatisusedbySCSfromothertenants,youcanreservetheunreplicatedRabbitMQforPCFtileforSCS,asshowninthediagrambelow.YoucanthenaddreplicaRabbitMQclustersforusebyappsintheMarketplace.

PivotalrecommendsthatyouusetheunreplicatedRabbitMQforPCFtilesolelyforSCStoavoidcontentionbetweenappsusingSCS,andappsusingRabbitMQforPCFintheMarketplace.

ToreservetheunreplicatedtileforSCS,turnofftheBrokerRegistrarerrandtopreventthebrokerfrombeingexposedintheMarketplace.Formoreinformation,seeErrands.

ToofferRabbitMQasacloudmessagingserviceintheMarketplace,createoneorseveralreplicas,installtheminOpsManager,andeitherallowthebrokerregistrarerrandtorun,orregistertheservicemanuallyusingCF .

ProvidingaPre-ProvisionedDedicatedClusterToreserveaRabbitMQclusterforusebyaspecificteam,disabletheBrokerRegistrarerrandinOpsManager.ThispreventsserviceregistrationintheMarketplace.Formoreinformation,seeErrands.

Afteryoudeploythetile,manuallyexposetheservicebrokertoyourdesiredorgsandspaces.Forinstructions,seeRegisteraBroker .

UsingReplicasWhileOfferingtheOn-DemandRabbitMQServiceTheOn-Demandserviceisnotofferedinreplicatiles,sincethepurposeofthereplicatoristocreateadditionalpre-provisionedclusters.Ifyouwishtoofferon-demandserviceplans,usetheunreplicatedRabbitMQforPCFtileasshowninthediagramabove.

Blue-GreenUpgrades(Advanced)Inordertodoblue-greenstyleupgradestominimizedowntime,youcanstandupanewclusterandmigratedataandusersovertothenewclusteroveraperiodoftime.SpeakwithyourPlatformArchitectabouthowtoenablethisworkflow.

GeneratingReplicaTilesThistopicdescribeshowtoinstallthereplicatorandgeneratereplicatilesofRabbitMQforPCF.

PrerequisitesRabbitMQforPCFv1.8.xorv1.9.x

2.5GBoffreediskspace

DownloadtheReplicatorTheRabbitMQforPCFReplicatoriscurrentlyavailablefromPivotalNetwork .Searchforanddownloadthisarchive,andthenruntheenclosedbinary.

GenerateReplicaTilesThefollowingisthesyntaxforgeneratingareplicatile:

./rabbitmq-replicator-darwin\--nameYOUR_DESIRED_TILE_NAME\--pathPATH_TO_TILE\--outputDESIRED_FILE_NAME.pivotal

Thefollowingaretheparametersexpectedintheabovesyntax:


https://docs.pivotal.io/pivotalcf/services/managing-service-brokers.html#register-broker

https://docs.pivotal.io/pivotalcf/services/managing-service-brokers.html#register-broker

https://network.pivotal.io

Parameter Description

nameThedesireduniqueidentifierforthereplicatile,whichisusedtogeneratemanifests,deploymentnames,andMarketplacenamefortheservice.Onlyalphanumericcharactersand - areacceptedbythetool.

path Thelocationoftheoriginal,unreplicated,RabbitMQforPCFsourcetilethatyoudownloaded

output Thedesiredfilenameandpathforthereplicatile

NamingConventionsinOriginalandReplicaTilesThetablebelowshowsthenamingconventionsforvariouscomponentsrelatedtotheoriginalRabbitMQforPCFtileandtothereplicatile.

Forthepurposesofthisexample,assumethatwhenyougenerateareplicatileasshownabove,inthe name fieldyouprovidethestring finance\ .Thentheattributesfortheoriginalandreplicatilesareasfollows:

Component NamewithOriginalTile NamewithReplicaTile

Brokername p-rabbitmq p-rabbitmq-finance

BrokerURL pivotal-rabbitmq-broker.YOUR_CF_DOMAIN pivotal-rabbitmq-broker-finance.YOUR_CF_DOMAIN

Servicename p-rabbitmq p-rabbitmq-finance

URLfortheRabbitMQManagementUIDashboard pivotal-rabbitmq.YOUR_CF_DOMAIN pivotal-rabbitmq-finance.YOUR_CF_DOMAIN

TiledisplaynameinOpsManager RabbitMQ RabbitMQ(finance)

TilenameusedinternallybyOpsManager p-rabbitmq p-rabbitmq-finance

Metrics/LoggingOrigin p-rabbitmq p-rabbitmq-finance

InstallingReplicaTilesAfteryouhavegeneratedareplicatile,youcanuploadittoOpsManagerasyouwouldanyothertile.Afteryouhaveuploadedit,followtheinstructionsforInstallingandConfiguringRabbitMQforPCFasaPre-ProvisionedService.TheOn-Demandserviceisnotofferedonreplicatiles.

LimitingAccesstoReplicaTilestoSpecificOrgsWhenyoureplicateRabbitMQforPCF,thereplicatilehastheBrokerRegistrarerrandsettoOnbydefault.ThisfieldappearsintheErrandstabinthetile:

Withanytile,iftheBrokerRegistrarerrandissettoOn,itrunsautomaticallywhenyoufinishinstallingthetileandcausesthetiletobeavailabletoallCForgs.

Ifyouwanttolimitaccesstothetiletoaspecificorg,followthesesteps:


1. SetthebrokerregistrarerrandtoOff,andapplyyourchanges.

2. ManuallyregisterthetilewithaspecificCForgusingthefollowingcommand.Seetheabovetablefor BROKER_NAME , BROKER_URL ,andSERVICE_NAME :

cfcreate-service-brokerBROKER_NAMEBROKER_USERNAMEBROKER_PASSWORDBROKER_URL

3. Togiveaccesstotheorg,usethefollowingcommandandrepeatforeachadditionalorg:

cfenable-service-accessSERVICE_NAME-oORG_NAME

UpgradingReplicaTilesYoucanupgradereplicatileslikeregulartileswithoneimportantdifference.YoumustgenerateareplicaofthenewerversionoftheRabbitMQforPCFtile,usingthereplicator,andgivethenewreplicathesame name astheexistingreplica.Thisisshownintheexampleworkflowbelow.

ExampleofanIn-PlaceUpgradeofaReplicaSupposeyouusedthereplicatortogenerateareplicaofv1oftheRabbitMQforPCFtile,withthe name trading-team,andyouinstalleditinOpsManager.Hereisthesamplereplicatorcommandyouusedfortheinitialinstallation:

./rabbitmq-replicator-darwin\--nametrading-team\--path/download/p-rabbitmq-v1.pivotal\--output/output/p-rabbitmq-v1-trading-team.pivotal

Toupgradetov2,followthesesteps:

1. DownloadthenewRabbitMQforPCFv2.

2. Runthereplicatorcommand,usingthepathtothenewRabbitMQforPCFv2tile,andsupplythesame name ,trading-team,asshownbelow.

./rabbitmq-replicator-darwin\--nametrading-team\--path/download/p-rabbitmq-v2.pivotal\--output/output/p-rabbitmq-v2-trading-team.pivotal

3. Afteryouhavethereplicatilep-rabbitmq-v2-trading-team.pivotal,uploadittoOpsManager.Thisupgradesthev1replicatileinplace.

Youcanthenproceedwithupgradingthecluster.

Ifyouwanttodoablue-greenstyleupgrade,seeBlue-GreenUpgrades.

LimitationsTheOn-Demandserviceisnotofferedonreplicatiles.


SettingDefaultPoliciesfortheRabbitMQService

UnderstandingaRabbitMQPolicyYoucansetadefaultqueueandanexchangepolicyintheRabbitMQforPivotalCloudFoundry(PCF)tiletobeappliedtotheRabbitMQcluster.Afteryoudeploythetile,PivotalrecommendsthatyouusetheRabbitMQManagementInterfacetomakeconfigurationchanges.

FormoreinformationaboutRabbitMQpolicies,seetheRabbitMQdocumentation .

RulesforPoliciesSetintheTileThefollowingrulesapplytopolicessetthroughtheRabbitMQforPCFtile:

Anewpolicy,oranupdatetoapolicy,onlyappliestonewinstances(vhosts).Existinginstancesarenotaffectedbythepolicy.

ThepolicycanonlybedeletedmanuallyfromtheRabbitMQnodes.

PoliciescanbeaddeddynamicallyusingtheRabbitMQManagementInterface.

Itisnotpossibletousepatternmatchingwithpolicies.Policieswillbeappliedtoallqueuesandexchanges.Forgranularpolicysettings,PivotalrecommendsusingtheRabbitMQManagementUI.Seta prioritynumber lowerthan 50 ,thedefault prioritynumber

appliedthroughtheOpsManagerconfiguration.

AnExamplePolicy:MirroronTwoNodesHereisanexamplepolicythatensuresmessagesaremirroredontwonodes:

{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}

Operatorsshouldconsidersomeoftheperformanceimplicationsofmakingqueuesandexchangeshighlyavailable.Formoreinformationabouthighlyavailablequeues,seetheRabbitMQdocumentation .

BestPracticeforSyncingQueuesWhenaqueuesyncsallitsmessages,theyareloadedintomemory.Whenqueuesaresyncing,theycanuseasmuchmemoryasthetotalsizeofallmessages.Thisappliestobothnodes—thenodewherethequeueleaderruns(fromnode)andthenodewherethequeuefollowerruns(tonode),butonlyappliestonewlycreatedqueuefollowers.

Thisbehaviorisespeciallyrelevantwhenanychangeaffectsthedeployment,forexample:stemcellupdates,deploymentconfigurationchanges,andnetworkchanges.Verifythatyouhaveenoughmemoryanddiskavailabletosupportallmessages.

Forexample:

Thereare5GBofmessagesinamirroredqueuethatissettoautomaticsync.Whenthisqueueneedstosync,thenodewherethequeueleaderrunscanuseupto5GBofextramemory.Thesameappliestothenodewherethenewqueuefolloweriscreated.

SettingorChangingthePolicyTosettheRabbitMQpolicy,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQforPCFtileandthenclickPre-ProvisionedRabbitMQPolicy.


http://www.rabbitmq.com/parameters.html#policies

https://www.rabbitmq.com/ha.html

2. SelectEnablecustompolicyonnewinstances.

3. InthePolicyfornewinstancesfield,pastethepolicy.ThepolicymustbevalidJSONandshouldmeetvalidRabbitMQpolicycriteria.

4. IntheSelectthenetworkpartitionbehavioroftheRabbitMQcluster,choosethedesiredbehavior:pause_minorityorautoheal.

FormoreinformationabouttheseoptionsandonRabbitMQclustersandnetworkpartitions,seetheRabbitMQdocumentation .

Forproductionpurposes,PivotalrecommendsthatcustomershaveatleastthreeRabbitMQservernodesandtwoHAProxiesspreadacrosslowlatencyavailabilityzones.

ViewingPoliciesintheRabbitMQManagementDashboardYoucanviewRabbitMQpoliciesintheRabbitMQManagementDashboard,shownbelow.TheexamplepolicyenteredintheRabbitMQforPCFtileaboveisappliedtoallqueuesandgivenaPriorityof50.Thisallowsyoutooverrideitbydefininganotherpolicywithahigherpriority.

IntheQueuessectionshownbelow,youcanseethatanynewqueuescreatedhavethepolicyautomaticallyapplied.

Note:Nopolicyvalidationoccursduringthedeployment,anderrorscancausethedeploymenttofailorpoliciestobeappliedincorrectly.



Note:DeveloperscanobtaintheURLofthepolicyfrom VCAP_SERVICES forappdevelopers.


UsingtheRabbitMQManagementDashboard

RabbitMQManagementDashboard

AdminUserTogainaccesstothemanagementdashboardasthe admin user,visit http://pivotal-rabbitmq.SYS-DOMAIN .Toretrieveyoursystemdomain,navigatetoyourPivotalApplicationService(PAS)orElasticRuntimetileandlocatetheSystemDomainfieldoftheDomainssection.

TheusernameandpasswordistheusernameandpasswordyouprovidedintheRabbitMQconfigurationinOpsManager,whichisalsoshownintheCredentialstab.

ApplicationDeveloperUsersofCloudFoundrywhocreateinstancesviatheAppsManagerorthecfCLIalsogetaccesstotheManagementUI.Thisisdoneusingcredentialsthatprovideaccessonlytotheirparticularvhost.

TheappropriateURLisaccessibleviatheManagebuttonwithintheAppsManager.


Oritisalsoinjectedintothe VCAP_SERVICES environmentvariableprovidedtoappsrunningonCloudFoundry.ThiscanalsobefoundviatheCLIusingcfenv<yourappname>

.

LoggingATCPSyslogendpointcanbeconfiguredinOpsManager.LogsarecurrentlyonlyforwardedfortheRabbitMQcluster.

RabbitMQCLIIfyouwishtoruncommandssuchas rabbitmqctl thenyouhavetwooptions:

SSHintooneofthemachinesrunningtherabbitmq-server.IPscanbefoundfromtheStatustabandaccesscredentialsfromtheCredentialstabwithintheRabbitMQcomponentoftheinstaller.FromthereyouneedtobringRabbitMQandErlangintoyourenvironmentandfromthereyoucanuserabbitmqctl :

bash-4.1#exportPATH=$PATH:/var/vcap/packages/rabbitmq-server/binbash-4.1#exportPATH=$PATH:/var/vcap/packages/erlang/binbash-4.1#rabbitmqctlcluster_statusClusterstatusofnoderabbit@node0...[{nodes,[{disc,[rabbit@node0,rabbit@node1,rabbit@node2,rabbit@node3]}]},{running_nodes,[rabbit@node3,rabbit@node2,rabbit@node1,rabbit@node0]},{partitions,[]}]...done.

Alternatively,installRabbitMQandErlangonamachineofyourchoice.Besuretomatchversionsofbothtothecluster:theManagementUIshowsboththeversionofRabbitMQandErlang.

Thensetyour ~/.erlang.cookie tomatchthecookieusedinthecluster(youmayhavesuppliedthisaspartoftheinstallation;seeabove).

Youwillneedtosetupyour /etc/hosts filetomatchtheRabbitMQnodes.


ClusteringandNetworkPartitions

ClusteringinRabbitMQforPCFInRabbitMQforPCF,theRabbitMQ®brokerisalwaysdeployedasaclusterofoneormorevirtualmachines(nodes).ARabbitMQbrokerisalogicalgroupingofoneorseveralErlangnodes,eachrunningtheRabbitMQapplicationandsharingusers,virtualhosts,queues,exchanges,bindings,andruntimeparameters.

WhatisReplicatedbetweennodesinaRabbitMQcluster?

Alldata/staterequiredfortheoperationofaRabbitMQbrokerisreplicatedacrossallnodes.Anexceptiontothisaremessagequeues,whichbydefaultresideononenode,thoughtheyarevisibleandreachablefromallnodes.ThismeansthattheRabbitMQclustermaybeavailableandservingrequests,whileanindividualqueueresidingonasinglenodeisoffline.

Replicatingmessagequeuesacrossnodesisanexpensiveoperationandshouldonlybedonetotheextentneededbytheapplication.Tounderstandmoreaboutreplicatingqueuesacrossnodesinacluster,seethedocumentation onhighavailability.

AutomaticNetworkPartitionBehaviorsinRabbitMQClustersTheRabbitMQ®tileusesthe pause_minority optionforhandlingclusterpartitionsbydefault.Thisensuresdataintegritybypausingthepartitionoftheclusterintheminority,andresumesitwiththedatafromthemajoritypartition.Youmustmaintainmorethantwonodes.Ifthereisapartitionwhenyouonlyhavetwonodes,bothnodesimmediatelypause.

Youcanalsochoosethe autoheal optioninthePre-ProvisionedRabbitMQPolicytab.Inthismode,ifapartitionoccurs,RabbitMQautomaticallydecidesonawinningpartition,andrestartsallnodesthatarenotinthewinningpartition.Thisoptionallowsyoutocontinuetoreceiveconnectionstobothpartsofpartitions.

DetectingaNetworkPartitionWhenanetworkpartitionoccurs,alogmessageiswrittentotheRabbitMQnodelog:

=ERRORREPORT====15-Oct-2012::18:02:30===Mnesia(rabbit@da3be74c053640fe92c6a39e2d7a5e46):**ERROR**mnesia_eventgot{inconsistent_database,running_partitioned_network,rabbit@21b6557b73f343201277dbf290ae8b79}

Youcanalsorunthe rabbitmqctlcluster_status commandonanyoftheRabbitMQnodestoseethenetworkpartition.Torun rabbitmqctlcluster_status ,dothefollowing:

1. $ sudo su -

2. $ cd /var/vcap/packages

3. $ export ERL_DIR=$PWD/erlang/bin/

4. $ cd rabbitmq-server/bin/

5. $./rabbitmqctlcluster_status

[...{partitions,[{rabbit@da3be74c053640fe92c6a39e2d7a5e46,[rabbit@21b6557b73f343201277dbf290ae8b79]}]}]

RecoveringBecausetheRabbitMQtileusesthe pause_minority option,minoritynodesrecoverautomaticallyafterthepartitionisresolved.Afteranoderecovers,itresumesaccessingthequeuealongwithdatafromthequeuesontheothernodes.However,ifyourqueuesuse ha-mode:all ,theyonlysynchronizefully


http://www.rabbitmq.com/ha.html

afterconsumingallthemessagescreatedwhilethenodewasdown.Thisissimilartohowmessagessynchronizewhenyoucreateanewqueue.

ManuallySynchronizingafteraPartitionAfteranetworkpartition,aqueueonaminoritynodesynchronizesafterconsumingallthemessagescreatedwhileitwasdown.Youcanalsorunthesync_queue commandtosynchronizeaqueuemanually.Torun sync_queue ,dothefollowingoneachnode:

1. $ sudo su -

2. $ cd /var/vcap/packages

3. $ export ERL_DIR=$PWD/erlang/bin/

4. $ cd rabbitmq-server/bin/

5. $ ./rabbitmqctl list_queues

6. $ ./rabbitmqctl sync_queue name


UpgradingRabbitMQforPCFThisproductenablesautomatedupgradesbetweenversionsoftheproductandisdeployedthroughOpsManager.Insomecases,youmightberequiredtotaketheclusteroffline.Whenthisisnecessary,itisclearlynotedinthereleasenotesforthatversion.

TheupgradepathsforeachversionaredetailedatPivotalNetwork-RabbitMQforPCFpage .

Toupgradeon-demandinstancesfromRabbitMQv3.6tov3.7,seeAboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7.

DowntimeWhenUpgradingAguidefordowntimeduringupgradedeploymentsisshowninthetablebelow.Insomecases,theclusterremainsavailableduringatileupgrade,butindividualqueuesonclusternodesmaybetakenoffline.

Thisisonlyaguide,sobeforeupgrading,checkthereleasenotesfortheversionyouareupgradingto.

TheRabbitMQclusterbecomesunavailableonlywhenupgradingbetweenspecificversionsofErlangorRabbitMQ.Thisisstatedinthereleasenotesforthoseversions.

UpgradeType WillDowntimeBeRequiredForThisUpgrade/Update

MajorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.

MinorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.

PatchTileVersion

Normallythesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.Therearespecificmigrationpathsthatrequiredowntime,whichareidentifiedinthereleasenotesforthatversion.

Stemcell-OnlyPatchTileVersion

Wherethepatchupdateisonlyanewstemcellversionthesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.

NotesontheUpgradeProcessReviewthefollowingbeforestartinganupgradeofRabbitMQforPCF:

Upgradingtoanewerversionoftheproductdoesnotcauseanylossofdataorconfiguration.

ItmaytakebusyRabbitMQnodesalongtimetoshutdownduringtheupgradeandyoumustnotinterruptthisprocess.

ThebenefityougetfromstemcellrollingupgradesdependsonhowyouhaveconfigurednetworkpartitionhandlingandtheResourceConfigtab.AnHAProxyinstancecountof2andaRabbitMQnodecountof3arerequiredforrollingstemcellupgrades.Asofv1.7.7,thesecountsarethedefault.Formoreinformation,seeClusteringandNetworkPartitions.

ThelengthofthedowntimedependsonwhetherthereisastemcellupdatetoreplacetheoperatingsystemimageoriftheexistingVMcanjusthavetheRabbitMQsoftwareupdated.StemcellupdatesincuradditionaldowntimewhiletheIaaScreatesthenewVM.

OpsManagerensurestheinstancesareupdatedwiththenewpackagesandanyconfigurationchangesareappliedautomatically.

ForissueswithupgradingRabbitMQforPCF,seeTroubleshootingOn-DemandRabbitMQforPCF.

BeforeUpgradingRabbitMQorErlangEnsuretheclusterishealthyusingtheRabbitMQManagementUI.YoucannotrelyontheBOSH instances outputbecausethatreflectsthestateofErlangVM,notRabbitMQ.

UpgradeRabbitMQforPCFToupgradetheproduct,followthesesteps:

1. DownloadthelatestversionoftheproductfromPivotalNetwork .


https://network.pivotal.io/products/p-rabbitmq

https://network.pivotal.io/products/pivotal-rabbitmq-service

2. Uploadthenew.pivotalfiletoOpsManager.

3. Uploadthestemcellassociatedwiththeupdate(ifrequired).

4. Updateanynewmandatoryconfigurationparameters(ifrequired).

5. ClickApplychangesintheOpsManagerInstallationDashboard.Therestoftheprocessisautomated.

AboutUpgradingOn-DemandInstancesfromRabbitMQv3.6tov3.7BeforeRabbitMQforPCFv1.12,on-demandserviceinstancesweredeployedusingRabbitMQv3.6.RabbitMQforPCFv1.12provideson-demandserviceplanswithRabbitMQv3.6andv3.7.

Appdeveloperscaneitherstartfreshwithav3.7on-demandinstanceortheycanmigratetheirRabbitMQv3.6instancestoRabbitMQv3.7usingblue-greenappdeploymentswithoutdowntime.

Toenablethemigrationfromon-demandv3.6instancestov3.7instances,theon-demandv3.7plansshouldbeconfiguredtomirrortheexistingv3.6plans.Formoreinformation,seetheblogBlue-GreenApplicationDeploymentswithRabbitMQ andthevideoBlue-GreenDeploymentofApplicationsleveragingRabbitMQ .

RabbitMQv3.6on-demandplanswillberemovedinanupcomingRabbitMQforPCFtilerelease.Encouragealldeveloperstomovetheirappsawayfromtheseplans.

ReleasePolicyWhenanewversionofRabbitMQisreleased,anewversionofRabbitMQforPCFisreleasedsoonafter.

FormoreinformationaboutthePCFreleasepolicy,seeReleasePolicy .


https://content.pivotal.io/blog/blue-green-application-deployments-with-rabbitmq

https://www.youtube.com/watch?v=S2oO-t-E38c

https://docs.pivotal.io/pivotalcf/security/process/pcf-security.html#release-policy

TroubleshootingandFAQsforOn-DemandRabbitMQforPCFThistopicprovidesoperatorswithbasictroubleshootingtechniquesandFAQsforon-demandRabbitMQforPivotalCloudFoundry(PCF).

HowtoRetrieveaServiceinstanceGUIDYouneedtheGUIDofyourserviceinstancetorunsomeBOSHcommands.ToretrievetheGUID,runthecommand:

cfserviceSERVICE-INSTANCE-NAME--guid

Ifyoudonotknowthenameoftheserviceinstance,run cfservices toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthenamecolumn.

TroubleshootingErrorsStarthereifyou’rerespondingtoaspecificerrororerrormessages.

FailedInstall1. Certificateissues:Theon-demandbroker(ODB)requiresvalidcertificates.Ensurethatyourcertificatesarevalidandgeneratenewones if

necessary.

2. Deployfails:Deployscanfailforavarietyofreasons.ViewthelogsusingOpsManagertodeterminewhythedeployisfailing.

3. Networkingproblems:

CloudFoundrycannotreachtheRabbitMQforPCFservicebrokerCloudFoundrycannotreachtheserviceinstancesTheservicenetworkcannotaccesstheBOSHdirector

4. Registerbrokererrandfails.

5. Thesmoketesterrandfails.

6. Resourcesizingissues:TheseoccurwhentheresourcesizesselectedforagivenplanarelessthantheRabbitMQforPCFservicerequirestofunction.CheckyourresourceconfigurationinOpsManagerandensurethattheconfigurationmatchesthatrecommendedbytheservice.

7. Otherservice-specificissues.

CannotCreateorDeleteServiceInstancesIfdevelopersreporterrorssuchas:

Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:create

Followthesesteps:

1. IftheBOSHerrorshowsaproblemwiththedeploymentmanifest,openthemanifestinatexteditortoinspectit.

2. Tocontinuetroubleshooting,LogintoBOSH andtargettheRabbitMQforPCFserviceinstanceusingtheinstructionsonparsingaCloudFoundryerrormessage.


http://docs.pivotal.io/p-mysql/credential-rotation.html

https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#log-in

3. RetrievetheBOSHtaskIDfromtheerrormessageandrunthefollowingcommand:

boshtaskTASK-ID

4. Ifyouneedmoreinformation,accessthebrokerlogsandusethe broker-request-id fromtheerrormessageabovetosearchthelogsformoreinformation.Checkfor:

AuthenticationerrorsNetworkerrorsQuotaerrors

BrokerRequestTimeoutsIfdevelopersreporterrorssuchas:

Servererror,statuscode:504,errorcode:10001,message:Therequesttotheservicebrokertimedout:https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b

Followthesesteps:

1. ConfirmthatCloudFoundry(CF)isconnectedtotheservicebroker.

2. ChecktheBOSHqueuesize:

a. LogintoBOSHasanadmin.b. Run bosh tasks .

3. Iftherearealargenumberofqueuedtasks,thesystemmaybeundertoomuchload.BOSHisconfiguredwithtwoworkersandonestatusworker,whichmaynotbesufficientresourcesforthelevelofload.Adviseappdeveloperstotryagainoncethesystemisunderlessload.

CannotBindtoorUnbindfromServiceInstances

InstanceDoesNotExist

Ifdevelopersreporterrorssuchas:

Servererror,statuscode:502,errorcode:10001,message:Servicebrokererror:instancedoesnotexist`

Followthesesteps:

1. ConfirmthattheRabbitMQforPCFserviceinstanceexistsinBOSHandobtaintheGUIDCFbyrunning:

cfserviceMY-INSTANCE--guid

2. UsingtheGUIDobtainedabove,thefollowingBOSHCLIcommand:

bosh-dservice-instance_GUIDvms

IftheBOSHdeploymentisnotfound,ithasbeendeletedfromBOSH.ContactPivotalsupportforfurtherassistance.

OtherErrors



Servererror,statuscode:502,errorcode:10001,message:Servicebrokererror:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:example-service,service-instance-guid:8d69de6c-88c6-4283-b8bc-1c46103714e2,broker-request-id:15f4f87e-200a-4b1a-b76c-1c4b6597c2e1,operation:bind

Tofindouttheexactissuewiththebindingprocess:

1. Accesstheservicebrokerlogs.

2. Searchthelogsforthe broker-request-id stringlistedintheerrormessageabove.

3. ContactPivotalsupportforfurtherassistanceifyouareunabletoresolvetheproblem.

4. Checkfor:

AuthenticationerrorsNetworkerrors

CannotConnecttoaServiceInstanceIfdevelopersreportthattheirappcannotuseserviceinstancesthattheyhavesuccessfullycreatedandbound:

Asktheusertosendapplicationlogsthatshowtheconnectionerror.Iftheerrorisoriginatingfromtheservice,thenfollowRabbitMQforPCF-specificinstructions.Iftheissueappearstobenetwork-related,then:

1. Checkthatapplicationsecuritygroups areconfiguredcorrectly.Accessshouldbeconfiguredfortheservicenetworkthatthetileisdeployedto.

2. EnsurethatthenetworkthePivotalApplicationService(PAS)tileisdeployedtohasnetworkaccesstotheservicenetwork.YoucanfindthenetworkdefinitionforthisservicenetworkintheBOSHDirectortile.

3. InOpsManagergointotheservicetileandseetheservicenetworkthatisconfiguredinthenetworkstab.

4. InOpsManagergointothePAStileandseethenetworkitisassignedto.Makesurethatthesenetworkscanaccesseachother.

UpgradeAllServiceInstancesFailsIfthe upgrade-all-service-instances errandfails,lookattheerrandoutputintheOpsManagerlog.

Ifaninstancefailstoupgrade,debugandfixitbeforerunningtheerrandagaintopreventanyfailureissuesfromspreadingtootheron-demandinstances.

OncetheOpsManagerlognolongerliststhedeploymentas failing ,re-runtheerrandtoupgradetherestoftheinstances.

MissingLogsandMetricsIfnologsarebeingemittedbytheon-demandbroker,checkthatyoursyslogforwardingaddressiscorrectinOpsManager.

1. Ensureyouhaveconfiguredsyslogforthetile.

2. Ensurethatyouhavenetworkconnectivitybetweenthenetworksthatthetileisusingandthesyslogdestination.Ifthedestinationisexternal,youneedtousethepublicip VMextensionfeatureavailableinyourOpsManagertileconfigurationsettings.

3. VerifythattheFirehoseisemittingmetrics:

a. Installthe cf nozzle plugin.Forinstructions,seethefirehoseplugin GitHubrepository.b. Tofindlogsfromyourserviceinthe cfnozzle output,runthefollowing:

cfnozzle-fValueMetric|grep--line-buffered"on-demand-broker/MY-SERVICE"


https://docs.pivotal.io/pivotalcf/adminguide/app-sec-groups.html

https://docs.pivotal.io/svc-sdk/odb/tile.html#public-ip

https://github.com/cloudfoundry-community/firehose-plugin

Ifnometricsappearwithinfiveminutes,verifythatthebrokernetworkhasaccesstotheLoggregatorsystemonallrequiredports.

ContactPivotalsupportifyouareunabletoresolvetheissue.

FailedDeploymentonUpgradeorafterApplyChangesIfthedeploymentfailsaftereditingtheAssignAZsandNetworkspaneoftheRabbitMQforPCFtile,itmightbeduetoachangetotheIPaddressesassignedtothe RabbitMQ Server job.RabbitMQforPCFrequiresthattheseIPaddressesdonotchangeonceassigned.Ifyouchangethem,thedeploymentfails.Thisincludeschangesmadetoyourcurrentinstallationorduringanupgrade.Todiagnoseandsolvethisissue,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

TroubleshootingComponentsGuidanceoncheckingforandfixingissuesinon-demandservicecomponents.

BOSHproblems

LargeBOSHQueue

On-demandservicebrokersaddtaskstotheBOSHrequestqueue,whichcanbackupandcausedelayunderheavyloads.AnappdeveloperwhorequestsanewRabbitMQforPCFinstancesees createinprogress intheCloudFoundryCommandLineInterface(cfCLI)untilBOSHprocessesthequeuedrequest.

OpsManagercurrentlydeploystwoBOSHworkerstoprocessitsqueue.FutureversionsofOpsManagerwillletusersconfigurethenumberofBOSHworkers.

Configuration

Serviceinstancesinfailingstate

YoumayhaveconfiguredaVM/DisktypeintileplanpageinOpsManagerthatisinsufficientlylargefortheRabbitMQforPCFserviceinstancetostart.Seetile-specificguidanceonresourcerequirements.

Authentication

UAAChanges

IfyouhaverotatedanyUAAusercredentialsthenyoumayseeauthenticationissuesintheservicebrokerlogs.

Toresolvethis,redeploytheRabbitMQforPCFtileinOpsManager.Thisprovidesthebrokerwiththelatestconfiguration.

Note:YoumustensurethatanychangestoUAAcredentialsarereflectedintheOpsManager credentials tabofthePivotalApplicationService(PAS)tile.



NetworkingCommonissueswithnetworkinginclude:

Issue Solution

LatencywhenconnectingtotheRabbitMQforPCFserviceinstancetocreateordeleteabinding.

Tryagainorimprovenetworkperformance.

FirewallrulesareblockingconnectionsfromtheRabbitMQforPCFservicebrokertotheserviceinstance.

OpentheRabbitMQforPCFtileinOpsManagerandcheckthetwonetworksconfiguredintheNetworkspane.Ensurethatthesenetworksallowaccesstoeachother.

FirewallrulesareblockingconnectionsfromtheservicenetworktotheBOSHdirectornetwork.

EnsurethatserviceinstancescanaccesstheDirectorsothattheBOSHagentscanreportin.

Appscannotaccesstheservicenetwork.ConfigureCloudFoundryapplicationsecuritygroupstoallowruntimeaccesstotheservicenetwork.

ProblemsaccessingBOSH’sUAAortheBOSHdirector.FollownetworktroubleshootingandcheckthattheBOSHdirectorisonline

ValidateServiceBrokerConnectivitytoServiceInstances

Tovalidateconnectivity,dothefollowing:

1. ToSSHintotheRabbitMQforPCFservicebroker,runthefollowingcommnand:

bosh-dservice-instance_GUIDssh

2. IfnoBOSH task-id appearsintheerrormessage,lookinthebrokerlogusingthe broker-request-id fromthetask.

ValidateAppAccesstoServiceInstance

Use cfssh toaccesstotheappcontainer,thentryconnectingtotheRabbitMQforPCFserviceinstanceusingthebindingincludedintheVCAP_SERVICES environmentvariable.

Quotas

PlanQuotaissues


Message:Servicebrokererror:Thequotaforthisserviceplanhasbeenexceeded.PleasecontactyourOperatorforhelp.

1. Checkyourcurrentplanquota.

2. Increasetheplanquota.

3. LogintoOpsManager.


4. Reconfigurethequotaontheplanpage.

5. Deploythetile.

6. Findwhoisusingtheplanquotaandtaketheappropriateaction.

GlobalQuotaIssues


Message:Servicebrokererror:Thequotaforthisservicehasbeenexceeded.PleasecontactyourOperatorforhelp.

1. Checkyourcurrentglobalquota.

2. Increasetheglobalquota.

3. LogintoOpsManager.

4. Reconfigurethequotaontheon-demandsettingspage.

5. Deploythetile.

6. Findoutwhoisusingthequotaandtaketheappropriateaction.

FailingjobsandunhealthyinstancesTodeterminewhetherthereisanissuewiththeRabbitMQforPCFservicedeployment,inspecttheVMs.Todoso,runthefollowingcommand:

bosh-dservice-instance_GUIDvms--vitals

Foradditionalinformation,runthefollowingcommand:

boshinstances--ps--vitals

IftheVMisfailing,followtheservice-specificinformation.Anyunadvisedcorrectiveactions(suchasrunningBOSH restart onaVM)cancauseissuesintheserviceinstance.

TechniquesforTroubleshootingThissectioncontainsinstructionsoninteractingwiththeon-demandservicebrokerandon-demandserviceinstanceBOSHdeployments,andonperforminggeneralmaintenanceandhousekeepingtasks.

ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cfserviceINSTANCE-NAME .


$cfservicemyservice

Serviceinstance:myserviceService:super-dbBoundapps:Tags:Plan:dedicated-vmDescription:DedicatedInstanceDocumentationurl:Dashboard:

LastOperationStatus:createfailedMessage:Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:createStarted:2017-03-13T10:16:55ZUpdated:2017-03-13T10:17:58Z

Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.

The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe boshtaskTASK-ID .

The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping boshlogsbroker0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.

AccessBrokerandInstanceLogsandVMsBeforefollowingtheproceduresbelow,logintothecfCLI andtheBOSHCLI .

AccessBrokerLogsandVM(s)

YoucanaccesslogsusingOpsManager byclickingontheLogstabinthetileanddownloadingthebrokerlogs.

ToaccesslogsusingtheBOSHCLI,dothefollowing:

1. Identifytheon-demandbroker(ODB)deploymentbyrunningthefollowingcommand:

boshdeployments

2. ViewVMsinthedeploymentbyrunningthefollowingcommand:

bosh-dDEPLOYMENT-NAMEinstances

3. SSHontotheVMbyrunningthefollowingcommand:


4. Downloadthebrokerlogsbyrunningthefollowingcommand:

bosh-dservice-instance_GUIDlogs

ThearchivegeneratedbyBOSHorOpsManagerincludesthefollowinglogs:

LogName Description

broker.logRequeststotheon-demandbrokerandtheactionsthebrokerperformswhileorchestratingtherequest(e.g.generatingamanifest


https://docs.pivotal.io/pivotalcf/cf-cli/getting-started.html

https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#prepare

https://docs.pivotal.io/pivotalcf/customizing/troubleshooting.html#component_logs

andcallingBOSH).Startherewhentroubleshooting.broker_ctl.log Controlscriptlogsforstartingandstoppingtheon-demandbroker.

post-start.stderr.log

Errorsthatoccurduringpost-startverification.

post-start.stdout.log

Post-startverification.

drain.stderr.log Errorsthatoccurwhilerunningthedrainscript.

AccessServiceInstanceLogsandVMs1. Totargetanindividualserviceinstancedeployment,retrievetheGUIDofyourserviceinstancewiththefollowingcfCLIcommand:

cfserviceMY-SERVICE--guid

2. ToviewVMsinthedeployment,runthefollowingcommand:

bosh-dDEPLOYMENT-NAMEinstances

3. ToSSHintoaVM,runthefollowingcommand:


4. Todownloadtheinstancelogs,runthefollowingcommand:

bosh-dservice-instance_GUIDlogs

RunServiceBrokerErrandstoManageBrokersandInstancesFromtheBOSHCLI,youcanrunservicebrokererrandsthatmanagetheservicebrokersandperformmassoperationsontheserviceinstancesthatthebrokerscreated.Theseservicebrokererrandsinclude:

register-broker registersabrokerwiththeCloudControllerandlistsitintheMarketplace.

deregister-broker deregistersabrokerwiththeCloudControllerandremovesitfromtheMarketplace.

upgrade-all-service-instances upgradesexistinginstancesofaservicetoitslatestinstalledversion.

delete-all-service-instances deletesallinstancesofservice.

orphan-deployments detects“orphan”instancesthatarerunningonBOSHbutnotregisteredwiththeCloudController.

Torunanerrand,runthefollowingcommand:

bosh-dDEPLOYMENT-NAMErun-errandERRAND-NAME

Forexample:

bosh-dmy-deploymentrun-errandderegister-broker

RegisterBroker

The register-broker errandregistersthebrokerwithCloudFoundryandenablesaccesstoplansintheservicecatalog.Runthiserrandwheneverthebrokerisre-deployedwithnewcatalogmetadatatoupdatetheCloudFoundrycatalog.

Planswithdisabledserviceaccessarenotvisibletonon-adminCloudFoundryusers,includingOrgManagersandSpaceManagers.AdminCloudFoundry


userscanseeallplansincludingthosewithdisabledserviceaccess.

Theerranddoesthefollowing:

RegisterstheservicebrokerwithCloudController.

Enablesserviceaccessforanyplansthathavetheradiobuttonsetto enabled inthetileplanpage.

Disablesserviceaccessforanyplansthathavetheradiobuttonsetto disabled inthetileplanpage.

Doesnothingforanyforanyplansthathavetheradiobuttonsetto manual .

Toruntheerrand,runthefollowingcommand:

bosh-dDEPLOYMENT-NAMErun-errandregister-broker

DeregisterBroker

ThiserrandderegistersabrokerfromCloudFoundry.

Theerranddoesthefollowing:

DeletestheservicebrokerfromCloudController

Failsifthereareanyserviceinstances,withorwithoutbindings

UsetheDeleteAllServiceInstanceserrandtodeleteanyexistingserviceinstances.


bosh-dDEPLOYMENT-NAMErun-errandderegister-broker


IfyouhavemadechangestotheplandefinitionoruploadedanewtileintoOpsManager,youmightwanttoupgradealltheRabbitMQforPCFserviceinstancestothelatestsoftwareorplandefinition.

The upgrade-all-service-instances erranddoesthefollowing:

Collectsalloftheserviceinstancestheon-demandbrokerhasregistered

Foreachinstancetheerranddoesthefollowingserially

Issuesanupgradecommandtotheon-demandbrokerRegeneratestheserviceinstancemanifestbasedonitslatestconfigurationfromthetileDeploysthenewmanifestfortheserviceinstanceWaitsforthisoperationtocomplete,thenproceedstothenextinstance

AddstoaretrylistanyinstancesthathaveongoingBOSHtasksatthetimeofupgrade

Retriesanyinstancesintheretrylistuntilallareupgraded

Ifanyinstancefailstoupgrade,theerrandfailsimmediately.Thispreventssystemicproblemsfromspreadingtotherestofyourserviceinstances.

Toruntheerrand,dooneofthefollowing:

SelecttheerrandthroughtheOpsManagerUIandhaveitrunwhenyouclickApplyChanges.

Runthefollowingcommand.


bosh-dDEPLOYMENT-NAMErun-errandupgrade-all-service-instances

DeleteAllServiceInstances

ThiserrandusestheCloudControllerAPItodeleteallinstancesofyourbroker’sserviceofferingineveryCloudFoundryorgandspace.ItonlydeletesinstancestheCloudControllerknowsabout.ItdoesnotdeleteorphanBOSHdeployments.

The delete-all-service-instances erranddoesthefollowing:

1. Unbindsallappsfromtheserviceinstances.

2. Deletesallserviceinstancessequentially.Eachserviceinstancedeletionincludes:

a. Runninganypre-deleteerrandsb. DeletingtheBOSHdeploymentoftheserviceinstancec. RemovinganyODB-managedsecretsfromCredhubd. Checkingforinstancedeletionfailure,whichresultsintheerrandfailingimmediately

3. Determineswhetheranyinstanceshavebeencreatedwhiletheerrandwasrunning.Ifnewinstancesaredetected,theerrandreturnsanerror.Inthiscase,Pivotalrecommendsrunningtheerrandagain.


bosh-dservice-instance_GUIDdelete-deployment

DetectOrphanedInstancesServiceInstances

Aserviceinstanceisdefinedas“orphaned”whentheBOSHdeploymentfortheinstanceisstillrunning,buttheserviceisnolongerregisteredinCloudFoundry.

The orphan-deployments errandcollatesalistofservicedeploymentsthathavenomatchingserviceinstancesinCloudFoundryandreturnthelisttotheoperator.ItisthenuptotheoperatortoremovetheorphanedBOSHdeployments.


bosh-dDEPLOYMENT-NAMErun-errandorphan-deployments

Iforphandeploymentsexist—Theerrandscriptdoesthefollowing:

Exitwithexitcode10

Outputalistofdeploymentnamesundera [stdout] header

Provideadetailederrormessageundera [stderr] header

Forexample:

Note:OrphanBOSHdeploymentsdonotcorrespondtoaknownserviceinstance.Whilerare,orphandeploymentscanoccur.Usetheorphan-deployments errandtoidentifythem.

WARNING:Useextremecautionwhenrunningthiserrand.Youshouldonlyuseitwhenyouwanttototallydestroyalloftheon-demandserviceinstancesinanenvironment.


[stdout][{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]

[stderr]OrphanBOSHdeploymentsdetectedwithnocorrespondingserviceinstanceinCloudFoundry.BeforedeletinganydeploymentitisrecommendedtoverifytheserviceinstancenolongerexistsinCloudFoundryandanydataissafetodelete.

Errand'orphan-deployments'completedwitherror(exitcode10)

ThesedetailswillalsobeavailablethroughtheBOSH /tasks/ APIendpointforuseinscripting:

$curl'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result'|jq.{"exit_code":10,"stdout":"[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n","stderr":"OrphanBOSHdeploymentsdetectedwithnocorrespondingserviceinstanceinCloudFoundry.BeforedeletinganydeploymentitisrecommendedtoverifytheserviceinstancenolongerexistsinCloudFoundryandanydataissafetodelete.\n","logs":{"blobstore_id":"d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d"}}

Ifnoorphandeploymentsexist—Theerrandscriptdoesthefollowing:

Exitwithexitcode0

Stdoutwillbeanemptylistofdeployments

Stderrwillbe None

[stdout][]

[stderr]None

Errand'orphan-deployments'completedsuccessfully(exitcode0)

Iftheerrandencountersanerrorduringrunning—Theerrandscriptdoesthefollowing:

Exitwithexit1

Stdoutwillbeempty

Anyerrormessageswillbeunderstderr

Tocleanuporphanedinstances,runthefollowingcommandoneachinstance:

boshdelete-deploymentservice-instance_SERVICE-INSTANCE-GUID

GetAdminCredentialsforaServiceInstance1. IdentifytheservicedeploymentbyGUID.

2. LogintoBOSH .

3. Openthemanifestinatexteditor.

4. Lookinthemanifestforthecredentials.

WARNING:RunningthiscommandmayleaveIaaSresourcesinanunusablestate.


https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html#prepare

ReinstallaTileToreinstallatileinthesameenvironmentwhereitwaspreviouslyuninstalled:

1. Ensurethattheprevioustilewascorrectlyuninstalledasfollows:

a. Loginasanadminbyrunning:

cf login

b. ConfirmthattheMarketplacedoesnotlistRabbitMQforPCFbyrunning:

cf m

c. LogintoBOSHasanadminbyrunning:

bosh log-in

d. DisplayyourBOSHdeploymentstoconfirmthattheoutputdoesnotshowRabbitMQforPCFdeploymentbyrunning:

bosh deployments

e. Runthe“delete-all-service-instances”errandtodeleteeveryinstanceoftheservice.f. Runthe“deregister-broker”errandtodeletetheservicebroker.g. DeletetheservicebrokerBOSHdeploymentbyrunning:

bosh delete-deployment BROKER-DEPLOYMENT-NAME

h. Reinstallthetile.

ViewResourceSaturationandScalingToviewusagestatisticsforanyservice,dothefollowing:

1. Runthefollowingcommand:

bosh-dDEPLOYMENT-NAMEvms--vitals

2. Toviewprocess-levelinformation,run:

bosh-dDEPLOYMENT-NAMEinstances--ps

IdentifyServiceInstanceOwnerIfyouwanttoidentifywhichappsareusingaspecificserviceinstancefromtheBOSHdeploymentsname,youcanrunthefollowingsteps:

1. Takethedeploymentnameandstripthe service-instance_ leavingyouwiththeGUID.

2. LogintoCFasanadmin.

3. Obtainalistofallservicebindingsbyrunningthefollowing:

cfcurl/v2/service_instances/GUID/service_bindings


4. Theoutputfromtheabovecurlgivesyoualistof resources ,witheachitemreferencingaservicebinding,whichcontainsthe APP-URL .Tofindthename,org,andspacefortheapp,runthefollowing:

a. cf curl APP-URL andrecordtheappnameunder entity.name .b. cf curl SPACE-URL toobtainthespace,usingthe entity.space_url fromtheabovecurl.Recordthespacenameunder entity.name .c. cf curl ORGANIZATION-URL toobtaintheorg,usingthe entity.organization_url fromtheabovecurl.Recordtheorganizationname

under entity.name .

MonitorQuotaSaturationandServiceInstanceCountQuotasaturationandtotalnumberofserviceinstancesareavailablethroughODBmetricsemittedtoLoggregator.Themetricnamesareshownbelow:

MetricName Description

on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining globalquotaremainingforallinstancesacrossallplans

on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/quota_remaining quotaremainingforaparticularplan

on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances totalinstancescreatedacrossallplans

on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/total_instances totalinstancescreatedforagivenplan

DropandRestoreAMQP(S)TraffictoaRabbitMQInstanceWhiledebuggingaRabbitMQinstance,youcanpreventappsfromsendingandreceivingmessages,forexample,todecreasetheserverload.Youcanuse drop-amqp-traffic and restore-amqp-traffic scripts,whichrunthenecessary iptables commandstoachievethat.

TostopandthenrestoretraffictoaRabbitMQinstance,dothefollowing:1. TostopallAMQP(S)traffictoaRabbitMQinstance,runthefollowingcommand:

bosh -d service-instance_GUID ssh rabbitmq-server "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/drop-amqp-traffic"

2. Afterperformingthetroubleshootingsteps,restorethetraffic.Todothis,runthefollowingcommand:

bosh -d service-instance_GUID ssh rabbitmq-server "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/restore-amqp-traffic"

Alternatively,youcanrunthesescriptsonindividualnodes:1. bosh ssh toarabbitmq-serverinstance.

2. sudo -s togainrootprivileges.

3. Execute drop-amqp-traffic todropallAMQP(S)traffictothisinstance,or restore-amqp-traffic tostartacceptingtrafficagain.

FrequentlyAskedQuestions

Note:Whenrunning cfcurl ensurethatyouqueryallpages,becausetheresponsesarelimitedtoacertainnumberofbindingsperpage.Thedefaultis50.Tofindthenextpagecurlthevalueunder next_url .

Note:Quotametricsarenotemittedifnoquotahasbeenset.


WhatshouldIcheckbeforedeployinganewversionofthetile?EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.YoucannotrelysolelyontheBOSH instances outputasthatreflectsthestateoftheErlangVMusedbyRabbitMQandnottheRabbitMQapp.

WhatisthecorrectwaytostopandstartRabbitMQinPCF?OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapp.

Forexample:

bosh stop rabbitmq-server and bosh start rabbitmq-server .

ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningthestopcommandandspecifying JOB [index] .

WhathappenswhenIrunboshstoprabbitmq-server?BOSHstartstheshutdownsequencefromthebootstrapinstance.

WestartbytellingtheRabbitMQapptoshutdownandthenshutdowntheErlangVMwithinwhichitisrunning.Ifthissucceeds,werunthefollowingcheckstoensurethattheRabbitMQappandErlangVMhavestopped:

1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.NoticethatwearetrackingtheErlangPIDandnottheRabbitMQPID.

2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID.

Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.Allremainingrabbitmq-serverinstanceswillbestoppedonebyone.

Whathappenswhenboshstoprabbitmq-serverfails?IftheBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:

result:1of1drainscriptsfailed.FailedJobs:rabbitmq-server.

WhatdoIdowhenboshstoprabbitmq-serverfails?

Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.

First,BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monitstartrabbitmq-server

.Youwillnotbeableto

startthejobviaBOSH start asthisalwaysrunsthedrainscriptfirstandwillfailsincethedrainscriptisfailing.

Oncerabbitmq-serverjobisrunning(confirmthisvia monitstatus ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.

HowcanImanuallybackupthestateoftheRabbitMQcluster?

Note:Donotuse monitstoprabbitmq-server

asthisdoesnotcallthedrainscripts


ItispossibletobackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.

BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.

2. Selectexportdefinitionsfromthemainpage.

BackupandRestorewithaScript

UsetheAPItorunscriptswithcodesimilartothefollowing:

Forthebackup:

curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"-o "$BACKUP-FOLDER/rabbit-backup.json"

Fortherestore:

curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"-X POST -H "Content-Type: application/json" -d"@$BACKUP-FOLDER/rabbit-backup.json"

Whatpre-upgradechecksshouldIdo?

BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:

1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.

2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.

3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.

KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .

FileaSupportTicketYoucanfileaticketwithPivotalSupport .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .

Toexpeditetroubleshooting,provideyourservicebrokerlogsandyourserviceinstancelogs.Ifyour cfserviceYOUR-SERVICE-INSTANCE outputincludesatask-id ,providetheBOSHtaskoutput.


http://discuss.pivotal.io

https://support.pivotal.io/

FrequentlyAskedQuestionsforPre-ProvisionedRabbitMQforPCFThistopiclistsfrequentlyaskedquestionsthatapplytotheRabbitMQforPivotalCloudFoundry(PCF)pre-provisionedservice.

FrequentlyAskedQuestions

WhatshouldIcheckbeforedeployinganewversionofthetile?EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.YoucannotrelysolelyontheBOSH instances outputasthatreflectsthestateoftheErlangVMusedbyRabbitMQandnottheRabbitMQapplication.

WhatisthecorrectwaytostopandstartRabbitMQinPCF?OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapp.

Forexample:

boshstoprabbitmq-server

and boshstartrabbitmq-server

.

ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningthestopcommandandspecifying JOB[index]

WhathappenswhenIrunboshstoprabbitmq-server?BOSHstartstheshutdownsequencefromthebootstrapinstance.

WestartbytellingtheRabbitMQapplicationtoshutdownandthenshutdowntheErlangVMwithinwhichitisrunning.Ifthissucceeds,werunthefollowingcheckstoensurethattheRabbitMQapplicationandErlangVMhavestopped:

1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.NoticethatwearetrackingtheErlangPIDandnottheRabbitMQPID.

2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID

Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.Allremainingrabbitmq-serverinstanceswillbestoppedonebyone.

Whathappenswhenboshstoprabbitmq-serverfails?IftheBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:

result:1of1drainscriptsfailed.FailedJobs:rabbitmq-server.

WhatdoIdowhenboshstoprabbitmq-serverfails?Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.

Note:Donotuse monitstoprabbitmq-server

asthisdoesnotcallthedrainscripts


First,BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monitstartrabbitmq-server

.Youwillnotbeableto

startthejobviaBOSH start asthisalwaysrunsthedrainscriptfirstandwillfailsincethedrainscriptisfailing.

Oncerabbitmq-serverjobisrunning(confirmthisvia monitstatus ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.

HowcanImanuallybackupthestateoftheRabbitMQcluster?ItispossibletobackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.

BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.

2. Selectexportdefinitionsfromthemainpage.

BackupandRestorewithaScript

UsetheAPItorunscriptswithcodesimilartothefollowing:

1. Forthebackup:

curl-u"$USERNAME:$PASSWORD""http://$RABBIT_ADDRESS:15672/api/definitions"-o"$BACKUP_FOLDER/rabbit-backup.json"

2. Fortherestore:

curl-u"$USERNAME:$PASSWORD""http://$RABBIT_ADDRESS:15672/api/definitions"-XPOST-H"Content-Type:application/json"-d"@$BACKUP_FOLDER/rabbit-backup.json"

Whatpre-upgradechecksshouldIdo?BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:

1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.

2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.

3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.


FileaSupportTicketYoucanfileaticketwithPivotalSupport .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .

Toexpeditetroubleshooting,provideyourservicebrokerlogsandyourserviceinstancelogs.Ifyour cfserviceYOUR-SERVICE-INSTANCE outputincludesatask-id ,providetheBOSHtaskoutput.





UsingOn-DemandRabbitMQforPCFThistopicprovidesinstructionsfordevelopersusingtheon-demandRabbitMQservicefortheirPivotalCloudFoundry(PCF)apps.RabbitMQenablesmessagingbetweencloud-basedservers,appsanddevices.

TheseproceduresusetheCloudFoundryCommand-LineInterface(cfCLI).YoucanalsouseAppsManager toperformthesametasksusingagraphicalUI.

Forgeneralinformation,seeManagingServiceInstanceswiththecfCLI .

PrerequisitesTouseon-demandRabbitMQforPCFwithyourPCFapps,youneed:

APCFinstallationwithRabbitMQforPCF installedandlistedintheMarketplace

ASpaceDeveloper orAdminaccountonthePCFinstallation

Alocalmachinewiththefollowinginstalled:

abrowserashelltheCloudFoundryCommand-LineInterface (cfCLI)theLinuxwatch command

Tologinto theorgandspacecontainingyourapp

DeveloperGuide

EntriesintheVCAP_SERVICESEnvironmentVariableAppsrunninginCloudFoundrygainaccesstotheboundserviceinstancesviaanenvironmentvariablecredentialshashcalledVCAP_SERVICES .Anexamplehashisshownbelow:

{ "p-rabbitmq": [{ "credentials": { "dashboard_url": "http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "password": "tavk86pnnns1ddiqpsdtbchurn", "protocols": { "amqp": { "password": "tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7" ] } } } }]}

Formoreinformationabouttheenvironmentvariable VCAP_SERVICES ,seeRabbitMQEnvironmentVariables.

TheCreate-BindProcessBecauseeveryappandserviceinPCFisscopedtoaspace ,anappcanonlyuseaserviceifaninstanceoftheserviceexistsinthesamespace.

TouseRabbitMQinaPCFapp:

1. UsethecfCLI orAppsManager tologintotheorgandspacethatcontainstheapp.


http://docs.pivotal.io/pivotalcf/console/dev-console.html

http://docs.pivotal.io/devguide/services/managing-services.html

https://network.pivotal.io/products/pivotal-rabbitmq-service

http://docs.pivotal.io/devguide/services/#instances

http://docs.pivotal.io/pivotalcf/concepts/roles.html#roles

http://docs.pivotal.io/pivotalcf/cf-cli/install-go-cli.html

http://www.linfo.org/watch.html

http://docs.pivotal.io/pivotalcf/cf-cli/getting-started.html#login

http://docs.pivotal.io/pivotalcf/concepts/roles.html#spaces

http://docs.pivotal.io/pivotalcf/cf-cli/getting-started.html#login

http://docs.pivotal.io/pivotalcf/customizing/console-login.html

2. MakesureaninstanceoftheRabbitMQforPCFserviceexistsinthesamespaceastheapp.

IfthespacedoesnotalreadyhaveaRabbitMQforPCFinstance,createone.IfthespacealreadyhasaRabbitforPCFinstance,youcanbindyourapptotheexistinginstanceorcreateanewinstancetobindtoyourapp.

3. BindtheapptotheRabbitMQforPCFserviceinstance,toenabletheapptouseRabbitMQ.

ConfirmServiceAvailabilityForanapptouseaservice,1)theservicemustbeavailableintheMarketplaceforitsspaceand2)aninstanceoftheservicemustexistinitsspace.

YoucanconfirmbothoftheseusingthecfCLIasfollows.

1. TofindoutifOn-DemandRabbitMQforPCFserviceisavailableintheMarketplace:

a. Enter cf marketplaceb. Iftheoutputlists ondemand-rabbitmq inthe service column,on-demandRabbitMQforPCFisavailable.Ifitisnotavailable,askyour

operatortoinstallit.

$cfmarketplaceGettingservicesfrommarketplaceinorgmy-org/[email protected][...]ondemand-rabbitmqSoloRabbitMQService[...]

2. ToconfirmthatanOn-DemandRabbitMQforPCFinstanceisrunninginthespace

a. Enter cf servicesb. Any ondemand-rabbitmq listingsinthe service columnareserviceinstancesofon-demandRabbitMQinthespace.

$cfservicesGettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded

Youcanbindyourapptoanexistinginstanceorcreateanewinstancetobindtoyourapp.

CreateaServiceInstanceUnlikepre-provisionedservices,on-demandservicesarecreatedasynchronously,notimmediately.The watch commandshowsyouwhenyourserviceinstanceisreadytobindanduse.

Tocreateaninstanceoftheon-demandRabbitMQforPCFservice,run cfcreate-service

:

1. Enter cfcreate-serviceondemand-rabbitmqSoloSERVICE_INSTANCE

Where SERVICE_INSTANCE isanameyouchoosetoidentifytheserviceinstance.Thisnamewillappearunder service [sic]inoutputfrom cf

services .

2. Enter watchcfservices andwaitforthe lastoperation foryourinstancetoshowas createsucceeded .

$cfcreate-serviceondemand-rabbitmqSolomy-instance

Creatingservicemy-instanceinorgmy-org/[email protected]

$watchcfservices

Gettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded

Ifyougetanerror,seeTroubleshootingInstances.


BindaServiceInstancetoYourAppForanapptouseaservice,youmustbindittoaserviceinstance.Dothisafteryoupushorre-pushtheappusing cfpush .

TobindanapptoaRabbitMQinstancerun $cfbind-service

.

1. Enter cfbind-serviceAPPSERVICE_INSTANCE

Where APP istheappyouwanttousetheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cfcreate-

service .

$cfbind-servicemy-appmy-instance

Bindingservicemydbtomy-appinorgmy-org/[email protected]:Use'cfpush'toensureyourenvvariablechangestakeeffect

UsetheRabbitMQServiceinYourAppToaccesstheRabbitMQservicefromyourapp:

1. Run cfenvAPP_NAME withthenameoftheappboundtotheRabbitMQforPCFinstance.

2. Intheoutput,notetheconnectionstringslistedinthe VCAP_SERVICES > credentials objectfortheapp.

3. Inyourappcode,calltheRabbitMQserviceusingtheconnectionstrings.

ForhowtocodeyourapptouseRabbitMQmessaging,seeAboutUsingPivotalRabbitMQ>ClientDocumentationintheRabbitMQdocumentation .

UpdatingaServiceInstanceIfyoubindanewserviceorchangetheservicebindings,youneedtorun cf

restarttoupdatethe VCAP_SERVICES environmentvariableintheapplication

container.

1. Enter cfrestart-appAPP

Where APP istheappyouwanttousetheupdatedserviceinstance.

$cfrestartmy-app

Pushingnewversionofanappautomaticallyrestagesandrestartstheapponanyserviceinstancesitisboundto.

UnbindaServiceInstancetoYourAppTostopanappfromusingaserviceitnolongerneeds,unbinditfromtheserviceinstanceusing cfunbind-

service.

1. Enter cfunbind-serviceAPPSERVICE_INSTANCE

Where APP istheappyouwanttostopusingtheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf

create-service .

$cfunbind-servicemy-appmy-instance

Unbindingappmy-appfromservicemy-instanceinorgmy-org/[email protected]

DeleteaServiceInstanceTodeleteaserviceinstance,run cfdelete-

service.


https://rabbitmq.docs.pivotal.io

1. Enter cfdelete-serviceSERVICE_INSTANCE

Where SERVICE_INSTANCE isthenameoftheservicetodelete.

$cfdelete-servicemy-instance

Areyousureyouwanttodeletetheservicemy-instance?yDeletingservicemy-serviceinorgmy-org/[email protected]

2. Enter watchcfserviceSERVICE_INSTANCE andwaitfora Serviceinstancenotfound errorindicatingthattheinstancenolongerexists.

Youcannotdeleteaserviceinstancethatanappisboundto.

CreateanAdminUserforaServiceInstanceIfyouwanttogetadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstance,andobtainusercredentialsthatyoucansharewithotherappdevelopers.

Bothoperatorsandappdeveloperscanusethisprocedure.Forinstructions,seeCreateanAdminUserforaServiceInstance.

SharingServiceInstancesInordertoshareserviceinstances thefeature-flag service_instance_sharing mustbeenabledbyyourOperator.YoucanthenfollowthedocumentationtoshareyourserviceinstancesacrossCloudFoundryOrganizationsandSpaces.

FederateExchangesandQueuesYoucanfederateexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.

Tofederateexchangesandqueues,dothefollowing:

1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.

Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.

2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:

{..."uri":"amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris":["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}

Forexample:

{..."uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}

3. Setupfederationasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.

Forinstructionsonfederation,seetheRabbitMQdocumentation .




https://www.rabbitmq.com/federation.html

ShovelExchangesandQueuesYoucanshovelexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.

Toshovelexchangesandqueues,dothefollowing:

1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.

Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.

2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:

{..."uri":"amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris":["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}

Forexample:

{..."uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}

3. Setupshovelasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.

Forshovelinstructions,seetheRabbitMQdocumentation .Atthemoment,RabbitMQforPCFonlysupportsDynamicShovels.


https://www.rabbitmq.com/shovel.html

RabbitMQEnvironmentVariablesThistopicprovidesareferencefortheenvironmentvariablesthatPivotalCloudFoundry(PCF)storesforRabbitMQforPCFserviceinstances.Thesevariablesincludethecredentialsthatappsusetoaccesstheserviceinstances.

VCAP_SERVICESAppsrunninginPCFgainaccesstotheboundserviceinstancesthroughanenvironmentvariablecredentialshashcalled VCAP_SERVICES .Anexamplehashisshowbelow:

{"p-rabbitmq":[{"label":"p-rabbitmq","name":"my-rabbit-service-instance","plan":"standard","tags":["rabbitmq","messaging","message-queue","amqp","pivotal"],"credentials":{"dashboard_url":"http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn","username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","vhost":"62e5ab21-7b38-44ac-b139-6aa97af01cd7","password":"#passwordexample123456789","ssl":false,"hostname":"10.0.0.41","hostnames":["10.0.0.41","10.0.0.51"],"uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7"],"http_api_uri":"http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","http_api_uris":["http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api"],"protocols":{"amqp":{"password":"passwordexample123456789","port":5672,"ssl":false,"username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","vhost":"62e5ab21-7b38-44ac-b139-6aa97af01cd7","host":"10.0.0.41","hosts":["10.0.0.41","10.0.0.51"],"uri":"amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris":["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},"management":{"username":"b5d0ad14-4352-48e8-8982-d5b1d257029f","password":"passwordexample123456789","path":"/api","port":15672,"ssl":false,"host":"10.0.0.41","hosts":["10.0.0.41","10.0.0.51"],"uri":"http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","uris":["http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api","http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api"]}}}}]}

Youcansearchforyourservicebyits name ,givenwhencreatingtheserviceinstance,ordynamicallythroughthe tags or label properties.Thecredentials propertycanbeusedasfollows:

Thetoplevelproperties uri , uris , vhost , username , password , hostname ,and hostnames provideaccesstotheAMQP0.9.1protocol.

Amoreflexibleapproachisprovidedbythe credentials.protocols property,whichhasakeyperenabledprotocol.Thepossiblekeysare amqp ,management , mqtt ,and stomp .IfSSLisenabled,thenthekeysare amqp+ssl , management+ssl , mqtt+ssl ,and stomp+ssl respectively.

Thevaluesassociatedwitheachofthesekeysgivesaccesscredentialsspecifictoeachprotocol.Inallcases,URIsareprovided,alongwiththeindividualcomponents.


ChangingEnabledPluginsandProtocols

IfyouadjustthepluginsandprotocolsenabledforRabbitMQ,youmightneedtoforceallapp’sVCAP_SERVICES environmentvariabletoberegenerated.Addingandremovingthefollowingpluginsrequireboundappstoberestaged:

rabbitmq_management

rabbitmq_stomp

rabbitmq_mqtt

rabbitmq_amqp1_0

IncommonwithallservicesinPCF,the VCAP_SERVICES environmentvariableforanappisonlymodifiedwhentheappisboundtoaserviceinstance.Usersneedto cfunbind-

service, cfbind-

service,and cfrestage theirappinthisscenario.

Note:Removingoraddingplugins/protocolsmightcauseappsboundwithRabbitMQtobreak.


TroubleshootingInstancesThistopicprovidesbasicinstructionsforappdeveloperstroubleshootingOn-DemandRabbitMQ®forPCF.

ErrorsYoumayseeanerrorwhenusingtheCloudFoundryCommand-LineInterface(cfCLI)toperformbasicoperationsonaRabbitMQforPCFserviceinstance:

cf create

cf update

cf bind

cf unbind

cf delete

ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cfserviceINSTANCE-NAME .

$cfservicemyservice

Serviceinstance:myserviceService:super-dbBoundapps:Tags:Plan:dedicated-vmDescription:DedicatedInstanceDocumentationurl:Dashboard:

LastOperationStatus:createfailedMessage:Instanceprovisioningfailed:Therewasaproblemcompletingyourrequest.Pleasecontactyouroperationsteamprovidingthefollowinginformation:service:redis-acceptance,service-instance-guid:ae9e232c-0bd5-4684-af27-1b08b0c70089,broker-request-id:63da3a35-24aa-4183-aec6-db8294506bac,task-id:442,operation:createStarted:2017-03-13T10:16:55ZUpdated:2017-03-13T10:17:58Z

Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.

The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe boshtaskTASK-ID .

The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping boshlogsbroker0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.

RetrieveServiceInstanceInformation1. Logintothespacecontainingtheinstanceorfailedinstance.

$cflogin

2. Ifyoudonotknowthenameoftheserviceinstance,run cfservices toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthe name column.


$cfservicesGettingservicesinorgmy-org/spacemy-spaceasuser@example.com...OKnameserviceplanboundappslastoperationmy-instanceondemand-rabbitmqSolocreatesucceeded

3. Run cfserviceSERVICE-INSTANCE-NAME toretrievemoreinformationaboutaspecificinstance.

4. Run cfserviceSERVICE-INSTANCE-NAME--guid toretrievetheGUIDoftheinstance,whichisusefulfordebugging.

RetrieveRabbitMQInstanceCredentialsIfyouwanttoaccesstheManagementDashboardortheRabbitMQserverfortroubleshooting,youcancreateanewservice-keytoretrieveRabbitMQinstancecredentials.Pivotalrecommendsthatyouusethiskeyfortroubleshootingonly,andthatyoudeletethekeyaftertroubleshooting.Toretrievethecredentials,dothefollowing:

1. Createaservice-keyforyourRabbitMQinstanceusingthecommand cfcreate-service-keyINSTANCE-NAMESERVICE-KEY-NAME .

2. Retrievethecredentialsusingthecommand cfservice-keyINSTANCE-NAMESERVICE-KEY-NAME .

Forexample:

$cfcreate-service-keymy-rmq-instancemy-keyCreatingservicekeymy-keyforserviceinstancemy-rmq-instanceasadmin...OK$cfservice-keymy-rmq-instancemy-keyGettingkeymy-keyforserviceinstancemy-rmq-instanceasadmin...{"host":"10.0.8.4","password":"","port":6379}


FileaSupportTicketYoucanfileasupporttickethere .Besuretoprovidetheerrormessagefrom cfserviceYOUR-SERVICE-INSTANCE .

Toexpeditetroubleshooting,ifpossible,provideyourservicebrokerlogs,serviceinstancelogs,andBOSHtaskoutput.Yourcloudoperatorshouldbeabletoobtainthesefromyourerrormessage.

DeleteRabbitMQInstancesOn-DemandBrokerprovidesaBOSHcommandtodeletealltheOn-DemandBrokerdeployedinstances.Todeletetheinstances,dothefollowingprocedure:

1. RunthefollowingcommandtodeleteallinstancesoftheOn-DemandBroker:

boshrun-erranddelete-sub-deployments

WARNING:Thiscommanddeletesdeploymentinstancesserially.Itisverydestructiveandcannotbeundone.




Documents

RabbitMQ for PCF - Pivotal · Documentation for RabbitMQ for PCF v1.7 and earlier only describes a pre-provisioned service. What are On-Demand Instances In RabbitMQ for PCF versions