51 thoughts on “Sizing Limits to Web Intelligence Report Server Maximum Simultaneous Connections”

1. Hi Julian,

   Quite interesting article. This remind me about one of the error we faced regularly. The Error was

   “Could not initialize the Report Engine Server”. When we check CCM, we can see Report Server status as Green, but in fact service would have hanged or not responding. This happens due to overloading of requests in Webi Report Server. Easy solution is just restart the service from CCM,though not a permanent solution, its quite handy. Permanent solution as explained above to add one more Webi Report Server. But as suggested by you we never edited “Maximum Simultaneous Connection”. This is indeed quite making lots of sense. Ideally how do we do the sizing, i don’t have much idea. Why don’t you start a new thread on Sizing? This would be really worth a try

   Thanks and Regards

   Aravind

2. Hi Aravind,

   Yes, I was thinking a sizing topic is long overdue. I have a few guidelines from BO consulting engagements, BO documentation, and experience. I think if I put this all down as a new topic others might add their own experience and we could end up with a decent collection of guidelines.

   This also makes me think that if I can find the time I should go back to some of the more commented topics and add some of the content in the main article to be sure that the search engines index it for the benefit of others (based on the possibly erroneous assumption that some search engines ignore comments).

   Thanks, Julian

3. Our connection limit is set at 50 across 4 WebI servers. How can you monitor how many are being used? I see in the metrics that 300+ requests are being served, but not how many connections are used.

4. The only precise method of measuring connections is to look at the target databases. I have created reports in the past that returned the number of sessions based on connecting application from a database. If you have a lot of data sources or if you cannot get an account with privileges to query sessions than this won’t work for you.

   On a side note, most BO consultants recommend one WebI server per CPU and that the connection limit be between 25 and 40 (depending on the RAM available, for example 8GB RAM would be good enough for 40). I need to write some sizing articles.

5. Thanks Julian, but we have 900+ Universes and over 600 Databases, so what you suggest is not possible. I was hoping there was somthing on the BO Report Server side to monitor at least a current connection count and connection queue. The Metrics in the CMC only show #Requests on each WebI server. I’m guessing there is one or more connections used per request…is that true? If I have 300 requests, and the server only handles 40 simultaneous connections, what’s happening to the other 250 requests? I’m assuming there’s a erquest queue somewhere.

6. It seems that the issue at hand is the definition of “requests” and of “connections”. It seems to me that “connections” refers the number of WebI client users. A “user” then is anyone or thing that needs the WebI Processing Server to do something. So if I open a WebI report for viewing, that is a connection. If I also have a WebI report open for editing, another connection. If a job server is refreshing a WebI report, another connection.

   Now “requests”, that is a strange beast. I wonder if you always observe that requests is higher than connections. It can’t be queued requests for connections, because then users would be sitting there looking at hourglasses or errors. I need to experiment some more with this. I tried some tests in a new BOE XI 3.1 SP2 FP 2.4 environment I just built with no users and I found no surprises or revelations yet.

7. I’m glad I found this thread. Very useful. We are in a XI R2 (SP4) two-node clustered environment. Each node has one WIRS with a Maximum Simultaneous Connection setting of 50. We have 2 Quad Core CPU’s – but half of the cores are disabled in the bios for licensing purposes. We have approximately 700 users and a typical maximum user concurrency of less than 10%. Despite this, we’ve intermittently seen the Web Intelligence Report Server reaching its threshold of 50 Maximum Simultaneous Connections and then crashing, all while the other node’s WIRS is up and running and ready to accept connections. we do have a Cisco CSS load balancer on top of that architecture that routes the HTTP requests. It’s set to “Sticky Source IP”, which the vendor says overrides the default load balancing and forces the processing to one node only, which they say, is why the other node’s WIRS could not process requests once the first node’s WIRS crashed. Do you have any experience with CSS load balancers? Does this make sense to you? regardless, if that is indeed the case, would adding more WIRSs to each node resolve the issue?

   Thanks,
   Sleuth

8. The Sleuth is back! I don’t see how a load balancer comes in to play here. The CMS should be doing its own load balancing and routing requested WebI resources in an even distribution on average. The load balancer does share the load between the web/http servers, but nothing else. Beyond determining which web server a user receives it is out of their hands which CMS you are routed to (unless you did not setup your CMS cluster in correctly int he web app). So still, I can’t see how a load balancer would impact any WebI server assignments. CMS is the traffic cop for the cluster, nothing else.

   But obviously, something is off. Do you have a balanced number of WebI Processing Servers on each machine? Do you have any server groups that are not evenly balanced across the cluster? What happens if you shutdown the WIRSs on the node that crashed, does the load shift to the other one?

9. Thanks for the feedback! I’ve never worked with load balancers, so when the SAP rep told me this “Sticky Source IP” setting is forcing traffic and processing to one server, I counted the load balancer among the possible causes for the problem. Regarding setting up the CMS cluster incorrectly in the web app, an outside consulting firm created a document instructing how to set this up. I will review that document with an SAP rep.

   We only have one WIRS per machine, and we only have two machines in the cluster. We have no server groups set up. We haven’t tried shutting doen the WIRS on the node that crashed because it’s a pretty-much 24/7 environment with a small weekend maintenance window. I can try that this weekend.

   Thanks,
   Sleuth

10. Hi Sleuth, A guideline I have used on dedicated WIRS boxes is to run one WIRS per CPU (or core). So if your machines’ cores were all enabled you could run 8 WIRS on each box. Now, since your box is not dedicated, I would at least subtract from this the number of CMS running on the machine. So if you were running 1 CMS and all cores were enabled, then you could do 7 WIRS.

    I know that you do not have all cores running (too bad) and that you are probably running EVERYTHING on the same box, (don’t worry I have done this). So if you are running just 4 cores on a machine then I would try running 2 or 3 WIRS on each machine. I would also try disabling all WIRS on one machine and seeing how the CMS handles sending all future requests to the other WIRS.

11. Hello,
    I have a sort of side question. What do the ‘system connections’ represent in the CMC settings -> Metrics Screen. Logged on represent the concurrent CMC sessions, what about system connections ?

    Thanks in advance.

12. Hi all,

    We experiencing a problem with a xcelsius dashboard that loads data from 25 webi reports using LiveOffice connection and from what I’ve read here it seems that it might have to do with load balancing issues. The problem arises randomly when users are using the dashboard and suddenly they get a RWI error saying that the document couldn’t be obtained because of obsolete token.
    The server has 8 CPUs and 16GB of RAM. At the beginning we set up 2 webi servers with 65000 max connections. Now I think it is not the best configuration.

    Do you recommend me to set up 4 webi servers with a number of connections of 40? Would this be enough?

    All comments are welcome. Thanks in advance.

    cheers,

    Christian

13. Hi Christian, 65000 max connections would not in itself harm the system (I think) until there is a load on the system. Once a load is received, such a high number would likely result in faulty and uneven load balancing by WebI process. In fact, you might be able to see this by monitoring your processes and notice that one is getting abused while the others are kicking back having an nice nap.

    Depending on what else you are running and how heavily you are using it I would strongly consider running 1 WebI processing server per CPU and set each to 40-50 connections. You will know that your connections are too low when you start getting RWI errors (telling you know WebI processing connections are available for the requested action). 8 x 50 = 400. Do you think that your system has 400 concurrent WebI processes (interactive refreshes/views + schedule refreshes + report edits)? If so, you might push this number higher, but keep all servers at the same level and I would not go much higher than 100 connections each.

14. We are Could not initialize the Report Engine Server”. with Communication Exception..

    Can someone plz advise on this.

    What is the exact thing we need to look in..?

    Any help would be appreciated

15. Since want to understand the terms, what is the difference between webi session and CMS Connection.?

16. Hi John,
    For “Could not initialize the Report Engine Server” I would look very closely at CMS database connectivity, the database where your CMS repository / InfoStore is located. Make sure that you can reach this database from the server using the same account that you use to run the CMS.

    WebI sessions are sessions assigned by the CMS internal load balancer to the WebI Processing Servers. They can natively open connections to data sources (sources of data in the reports) without the need for a Connection Server.

    CMS Database Connections are the number of connections that the CMS attempts to open on its InfoStore / metadata repository. The number of connections are defined by the parameter “ndbqthreads” in XI R2 (10 maximum) and in XI 3.X these are a text box on the CMS editable in the CMC (default 14, max 99?).

17. Hey Julian,

    Thanks. But these errors are coming up sporadically/intermittently during our performance Testing. When we run reports with 10 users concurrently no issues. And when we gradually increase the user load, the errors comes in…
    After sometime, automatically the error disappears, I noticed in CMC, sessions reduced gradually by that time..

18. Hi Julian,

    We’ve been experiencing some issues with our Webi report servers lately. The dreaded RWI 000236 error. I think I’ve nailed down what the issue is and it’s down to a report exceeding the 2GB limit. The system in place here is a standalone BO install that had 1 webi report server trying to handle everything – I’ve now added an additional 4 webi report servers, and want to make sure I can make the right decision on the Max Sim Connections. It was initially set as default, 50, we have 8GB ram on a 32 bit server, with I believe 4 CPU’s. What should I set these to? I know our userbase doesn’t exceed 50 at any point in time.

    Any help would be great – thanks.

    Lloyd.

19. Hi Lloyd, please keep in mind that some users may use more than one WebI server connection (such as users who open Java Report Panel in a new window) and that scheduled jobs also consume WebI connections. Having said that, I think that 100 total would probably meet your needs. Since you are running everything on one server I think you might want to try running less than one WebI server per CPU, perhaps just 2 WebI servers would be good with 50 connections each. Or you could try 3 with 33 connections each, or even 4 with 25 connections each. Load test if you can. Please share your results if you can. Thanks, Julian

20. Hello,

    We have 1 boxi server with 2 cpu and are running 2 WebI reportservers [boxi r2]. The WebI jobserver still has the default 5 jobs. However, in the morning we schedule a lot of reports in a short time. When 5 jobs are running, the rest is queued. I read it is best to run only one WebI jobserver per server.
    Is it possible to increase the number of jobs in the WebI jobserver? Should the setting be 5 * number of WebI reportservers? What is the best solution to increase scheduling capacity for WebI (besides adding a second server)?

    Any help or thoughts would be appreciated.

    Kind regards, Marcel

21. Hi Marcel, The job servers use very little overhead themselves. Their impact comes through the jobs they spawn on the corresponding processing servers. Therefore, you could increase the number of jobs processed if you think that your WebI processing servers can handle it (keeping in mind the interactive sessions they will be handling). I think experimentation is a good idea here. I would stick with just one job server and just bump up its active job limit.

22. Hi Julian,

    Does BO have any restrictions on the number of Schedules and the number of adhoc user’s access concurrently?

    Actually we were facing the issue with the server as “Web intelligence server is busy and cannot be reached”

    We found that the data is too much, which is causing the error, we increased the maximum upper threshold to 2000 (MB) in WebI processing server which resolved the issue.

    What if 2000 (MB) is also not sufficient to server the request and how to manage the scheduling and adhoc reporting

    Regards.

    Srini

23. Hi Srini, I have load tested 1500 concurrent users on a Business Objects system, but it was appropriately sized to handle this many (such as 1 CMS for every 500 concurrent and at least 1 WebI processing session for every expected concurrent user and job (some users can tie up multiple WebI sessions). Which setting exactly did you push to 2000 MB? How many concurrent do you expect? How many CMS? How many WebI process servers? How many sessions configured for each?

24. Hi Julian,

    Thank you so much for you quick reply.

    I have changed the these values in Webi Peocessing server properties,
    Memory Upper Threshold (MB): 2000
    Memory Maximum Threshold (MB): 2000

    There could be only 5 users and 11 schedules run at a time and only 1 cms and 1 WebI process server..

    Data could be around 7 to 8 million.

    Regards,
    Srini

25. Hi Srini, 7 to 8 million rows of data is intense. No wonder you had to bump up memory. To be honest, most Business Intelligence experts would tell you that even 100,000 rows of data is too much. Your users must be using your system for ETL not BI.

    BTW, when you get a chance, please answer my questions from earlier.

26. Hi Julian,

    “such as 1 CMS for every 500 concurrent and at least 1 WebI processing session for every expected concurrent user and job”

    Does this mean we need to have diffrent WebI processing server for each user?

    I suspect, they dont have ant ETL…

    Regards,
    Srini

27. Hi Srini,
    1) when I said ETL, I meant to say that your users are using your BO system to Extract data from the data source, in order to Transform it for Loading somewhere else. This use of a Business Intelligence system is frowned on by all BI experts and BI admins (like yourself) should strive to discourage it where feasible.
    2) By default I think all WebI processing servers have 50 sessions configured. You can increase this, but not too far. So if I had 40 concurrent users with a job server permitting only 5 scheduled jobs at a time, I could possibly get away with a single WebI Processing server set to 50 sessions. Usually, you would add more processing servers if you need more than 50 and usually you have one processing server for each CPU/core as a maximum configuration.
    BTW, you still have not shared any information about your configuration or requirements. 🙂

28. Hi All,

    We have scheduled WebI reports that are pending in queue and also number of jobs running in queue are low compared to jobs pending in queue.

    We are planning to monitor the processes and use Probes. Can you please specify which processes to be monitiored for scheduled webI reports?

    Also can you please provide me how to design the requiremnts for monitoring the processes?

    Currently we have 2 CPU’s and 3 WIRS.

    Regards,
    Nickie

29. Hi Nickie, If you want to process more jobs in your job server’s queue you need to increase the number the job server can process. By default this is set to “5”. Try pushing this up, but not too much of course.

    Monitoring is a very different topic. The best you can do is to monitor the executables for the job server and the 3 WebI server. Assuming you are running on Windows then you would merely need to login to the server and look at Task Manager to get the exact process names. It would be best if you could monitor that all 3 WIRS and the 1 job server are always up. If you wanted to monitor functionality then you would have to find a way to combine your jobs with an event or program job that would write out a file for your monitor to check on. I am interested to see if anyone else has any recommendations on monitoring WebI and Job servers.

30. Hi Julian,

    Thank you for quick reply,
    Currently we have maximum jobs set to 15.
    we usually monitor them through task manager as you specified but what i guess is they seem to be dead inside and queued up.
    so to overcome this situation we would like to set up Probes to poll on specific criteria.

    Just wanted to know if any one has done monitoring other than through task manager?

31. Hi Julian,

    Your article is really interesting & seems very useful. Let me just mention my query here, I have my client with total 75 USERS out of them 25 would be Advanced users with Ad-Hoc data analysis capabilities & rest 50 would just be responsible to view those designed reports. All these 75 users would be concurrently accessing the server & will also include designing of reports & refresh.

    I would really appreciate if you could kindly provide me insights on server sizing.

    Awaiting response,
    Chaitanya

32. Hi Chaitanya, thanks for the feedback. For those users you only need the following (assuming only WebI/Designer is used):
    > 1 Tomcat Instance (or WebLogic/WebSphere)
    > 1 CMS
    > 2 WebI servers (set to handle 50 requests each, 3 WebI would be better)
    > 1 Input FRS
    > 1 Output FRS
    > 1 Destination Job Server
    > 1 WebI/Adaptive Job Server
    > 1 Event Server (if they use events only)
    > 1 Program Job Server (for you to be able to schedule scripts and executables, if you want)

    A server with 4 core (1 quad core CPU, or 2 dual-core CPUs) could probably handle this, but having 6 or more cores would be better. 4 GB of RAM would be good. You will need at least 40 GB of storage, but more would be nice for times when extra logging is needed. Of course, it would be better if you could have two boxes and have each house at least 1 of each kind of server (for high-availability).

33. Hi Julian,

    I’ve done benchmark tests between XI 3.1 SP3 and XI R2 thanks to JMeter.
    Because I want to upgrade my BO.

    And the results are alarming.
    I worked with two Windows platform (one R2, one 3.1) that have the same physical configuration :
    4CPU, 1CMS, 1WebIReportServer

    Let’s focus on the following scenario :
    Login
    Loop 15 times => Open a report
    Logoff

    Test properties :
    Ramp-Up Period (in seconds) = 150
    Duration = 600
    LoopCount = Forever

    Results Average (seconds)
    For 1 thread :
    Login R2 = 0,260
    Login 3.1 = 2,731
    OPEN R2 = 1,014
    OPEN 3.1 = 2,061
    Logoff R2 = 0,127
    Logoff 3.1 = 0,363

    For 20 threads :
    Login R2 = 0,146
    Login 3.1 = 2,702
    OPEN R2 = 1,101
    OPEN 3.1 = 5,245
    Logoff R2 = 0,107
    Logoff 3.1 = 0,498

    For 50 threads :
    Login R2 = 0,266
    Login 3.1 = 2,681
    OPEN R2 = 3,720
    OPEN 3.1 = 14,179
    Logoff R2 = 0,227
    Logoff 3.1 = 0,563

    Are you surprised by that ?
    Or can you confirm these figures by your experience ?
    The 3.1 WIRS is slower than R2 WIRS ??

34. Great article Julian.

    I was wondering if you or anyone else has recommendations on how many WebI Servers and connections per Webi Server is best for the following on R2:

    4 quad core CPU’s
    12 cores total (we disabled 4 cores because of licensing)
    64RAM
    2 instances

    I appreciate your help.

35. Hi Theoren, I’ve seen a lot of evidence to support your findings. In some cases a little bit of tuning on the web app server and the database has resulted in improvements, but regardless, Business Objects Enterprise XI R2 is faster at 97% of all functions.

36. Hi Deno, you provided a lot of information, but I’d like to know a little more.

    1) Regarding “2 instances”, do you mean to separate Business Objects environments? If so is this possible with XI R2? I don’t recall, its been a while.

    2) What is your expected peak concurrent Web Intelligence use level? In other words, how many WebI users do you expect at any peak time?

    3) Is this server running anything else?

37. Hi Julian,

    Not sure if you give answers to this post but anyway, I’ll try my luck. 🙂

    I have single installation BI 4.0 server with 4 CPU’s, memory is currently set at 16.
    Also this server has 4 webi processsing servers- each has connection limit set to 70.
    It seems that the user activity has increased during the last few weeks as I am getting errors about not enough webi connections available. What are my options:
    a) Request another CPU(as this is virtual server) and set up additional webi server.
    b) Increase maximum number of connections and increase server’s memory?

    Thank you,

    Rain

38. Hi Rain,

    Before we conclude on increasing the servers and Memory, we would need to do lot of analysis on logs to figure out the exact cause for the Issue.
    You may enable the trace logs and see what is the causing this.
    If it’s a memory, you would see the memory levels being consumed by your server.

    If, it’s connection unavailability, you would see the error stating the reason for the same.

    Regards,
    Srini

39. Hi Srini,
    we have seen both memory increasing and also user activity. So my initial assumption is that we run out of connections because increased user activity. Although number of sessions is increasing the memory usage doesn’t grow so fast.

    Regards,

    Rain

40. Hi Rain,

    Here you go, if the memory consumption is not increasing with increase in User, you can try to increase the connection under Web Intelligence Processing Server and see if it serve the purpose 🙂

    Right Julian?

    Regards,
    Srini

41. Hi Julian!

    Thanx alot for your article. We are just trying to fix some problems with our BO System (webiprocesses hang up often).
    I just changed the setting of the maximum numbers of connections from 100 to 25 (approximatly a maximum of 150Concurrent Users).
    Actually we have 3 CPU’s each side on a Cluster BO 3.1 SP5 FP4 System 32GB RAM.
    8 WebiProcesses – 4 on each site.
    But now i am also thinking about what do the other paramters do? Do you have any idea or an description how to Change the settings to get the best result?

    Cache In Use (KB) 686979
    Virtual Memory Size (MB) 409
    Maximum Idle Time (seconds) 1200
    Idle Document Timeout (seconds) 300
    Polling Interval (seconds) 120
    Maximum Documents Per User 5
    Opened Documents Before Server Recycle 50
    Errors On Maximum Map Size Enabled 1
    Cache Timeout (seconds) 4370
    Document Cache Cleanup Interval (minutes) 120
    Idle Connection Timeout (minutes) 60
    Maximum Connections 100
    Cache Enabling State 1
    Real-time Caching Enabled 1
    Cache Timeout (seconds) 4370
    Maximum Cache Size (KB) 1000000
    Maximum Document Cache Reduction Space (%) 70
    List Of Values Caching Enabled 1
    List Of Values Batch Size (entries) 1000
    Maximum Entries for Custom Sort 100
    Maximum Character Stream Size (MB) 200
    Maximum Binary Stream Size (MB) 200
    Maximum Universes in Universe Cache 20
    Maximum Documents in Document Cache 0
    Memory Watcher Enabled 1
    Lower Memory Threshold (MB) 1000
    Upper Memory Threshold (MB) 1500
    Maximum Memory Threshold (MB) 2000
    Secured Cache enabled 0
    Maximum List Of Values Size (entries) 10000000

    Does anyone have an experience how to change the settings!?
    Regards
    Tom

42. Hi Tom M., I am not sure you could overcome hanging WebI servers with tweaks to these parameters. I can tell you that I have heard that BO messed up some of the WebI Processing Server code sometime in SP5 and it stayed in SP6 too. I’ve heard of large customers on SP6 getting LA Fixes to correct the issue, but I don’t know any specifics. I would ask SAP Support if I were you.

43. Hi Julian!

    Thx for your answer but SAP is not the best case to figure this out. I think we know more about this product then they do :-).
    Thx anyway
    Tom

44. Hi Julian,

    We have two 4 core servers setting up a BOE 4.1 SP1 environment. We have some 450 concurrent users, 8 WebI Processing servers. Not sure about the Adaptive Job Servers though. How many AJS can we have considering we have 8 cores and 15-20 concurrent jobs (scheduled reports) running at the same time?

    Thanks,
    Aqueel

45. Hi!

    We use 1 AdaptiveJob Server on each Side
    AIX 7.1
    5 Power7+ Cpus / Side

    We have 80000 reports with 3000Users in our System and it works fine but it depends on the reports… How Long do they run, how much data do they fetch, how many variables are inside the Report.
    If the reports are Webis the WebiIntelligenceProcessingServer is important for you. If you have configured 4 Webi’s on each side it should do´…

    Tom M.

46. Hi,
    Can anyone tell me what is the maximum possible limit of the field in webi . To be more precise what the maximum text length of the field.

47. Hi Krishna,
    In Business Objects Designer, Universe Design Tool, or Information Design Tool (name depends on your version of BO) you can configure this. I believe the default setting is 1024 characters.

48. Hi All,

    I would like to know if i cancel the long running webi report it will cancel the report but what will happen at server and database side. I would like to know at backend side.

49. I would like to know if i cancel the long running webi report it will cancel the report but what will happen at server and database side. I would like to know at backend side.

50. I’m trying to find out how a subtotal can be created on a report which add column D every time a change occurs in Column A.
    I have a few reports where they column A total is obtained by summing the values of Column
    Thanks,

    Mel