Breaking News
I have some breaking news, a scoop perhaps, on a new bug that may only impact some (not all) innocent Business Objects XIR2 SP4 shops. Apparently, my contacts tell me that within the last week, Business Objects has internally identified and even resolved a bug that can wreak havoc on the unknowing BO XIR2 users out there who make the leap to Service Pack 4 (SP4).
Symptoms of the Bug: All Schedules Pending
It seems that the bug causes a loss of communication between the BO XIR2 CMS server(s) and the job server(s). The strange part is that the only detectable error or problem is that suddenly no scheduled jobs process; they all go to an eternal “Pending” status. Every Job Server’s scheduling function ceases to function. Even stranger is that fact that this bug does not effect all Business Objects XIR2 SP4 installations. Some experience the issue and others do not. Some assume that the SP4 install may have changed their configuration or their security and caused the issue; however, it seems that all settings stayed unchanged (good!), but the job servers just ceased to do their job (very bad!).
No Fix Until Fix Pack 4.5
BO just released Fix Pack 4.3 for SP4 and the fix to this issue was not included. My resources tell me that it is planned for inclusion in BO XIR2 Fix Pack 4.5. I am not certain what the due date is for FP4.5, but I would imagine that is not going to be available until after Columbus Day and probably not until after Halloween.
Rethink Your Planned BO XIR2 SP4 Upgrade
If you are thinking about installing SP4 on your Business Objects XIR2 system I would encourage you to rethink this plan. Could your users accept the new functionality of completely disabled scheduling? I doubt it. Of course, there may be fixes to other functionality in the SP4 upgrade for which you cannot wait. If this is true then you might want to test out the fixed DLL on a test environment. Yes, BusinessObjects has already corrected the issue and released a Limited Availability patch. I have managed to get my hands on the patch info and the patch that BO’s product development group has developed.
Download the BO XIR2 SP4 XIR2.LAFix4.3.3 patch directly from Business Objects
Get the Fix for the BO XIR2 SP4 Bug
On September 24th, BO released a Limited Availability patch. In truth the fix is still considered a “beta” fix and so it comes with the standard disclaimer. Basically the LA Fix hasn’t been through full regression testing and it may inadvertently introduce other issues. BO Customer Assurance still needs to confirm the issue is fixed by this patch.” That is a pretty big disclaimer, but it is the standard one on such “hot-off-the-press” fixes. If you need this fix then test it yourself.
Regardless, I recommend that you immediately contact your BusinessObjects Account Rep and/or BO Support. The more attention this issue gets the more likely it will be officially fixed sooner.
Download the BO XIR2 SP4 XIR2.LAFix4.3.3 patch directly from Business Objects
My disclaimer: I provide no guarantees or warranties with this patch, just take it as it is. I tested it and it seems to be fine. Others have claimed that it corrected their issue. That is all I can tell you. Good luck. Oh and one last thing… PLEASE backup your SchedulerSubsystem.dll before you install the patch!
Thanks for the info. We are just planning to upgrade to SP4. As we are not using scheduling for the moment, do you think it is save to upgrade ?
Hi Rik,
SP4 should be safe for most customers; however, I highly recommend testing out on a test or development environment first. I have heard reports of some customers discovering other problems that seem to have begun shortly after their SP4 upgrade. However, if you give it a good test then you should be OK.
Before you do the upgrade I would backup your repository and File Repository System. This will make reverting backwards possible.
Good Luck,
Julian
Hi
Just to let you know that this issue is present in BOXI-R3. I am waiting to hear from a BO developer who is currently writing an ad hoc fix for us. Whether this is resolved in 3.1 is still not a definite yes and will know shortly.
It appears that the codebase for 3.0 was possibly taken from 2.0 after SP4 thus introducting the problem to 3.0.
Hey Andrew,
I am very sorry to hear this, but thank you for sharing this breaking news with us. Wow! I guess Business Objects R3 is not going to bring about world peace after all.
Since BO sounds like they are working on a LA fix for this issue I wonder if that means that the issue is not definitely fixed in R3 SP1. Hmmm.. actually I imagine that it most definitely is not. The issue was recognized by BO (for R2 SP4) around September 20th so there is doubt that the fix went into R3’s SP1. Big doubt.
If you get the link to that LA fix please share it. I hope you get it soon.
Hang in There,
Julian
Hi,
Yes this is an issue in SP4, however we have LA fix available to combat this.
Try adding this on top of SP4.
Good Luck 🙂
Shivaprasad
Hi Shiva,
Are you speaking of a different LA fix than the one that is discussed and linked in the last paragraphs of this article?
Just curious, I’d like to be sure the info I provide here is accurate and up-to-date.
Thanks,
Julian
Hi,
I am speaking about the same which was updated on 9th of this month. Well being a BO Tech support guy, I have seen many issues regarding memory leakage after LA fix. I am still dealing with that.
Do you guys have any idea about this.?
Thanks
Shiva
Hi Folks,
One more tip!!
Installing SP4 also dismantels Webi report formating.
Symtops :-
>> Collapse in right hand margin of Webi reports.
>> Right hand side of the webi report spanned and moves on the other page.
>>Bottom bar disapearing.
This usually happens with reports having large number of columns.
Workaround :- Manually edit the page size to be A2 from A4 in the page layout section of Properties and save the report.
Thanks
Shiva
We are facing problem of CMS. CMS (Central Management Server) services restart automatically. I don’t know why it is restart automatically.
Also I want my Pending Job list.
Pl. reply.
I have heard of CMS restarting issues, but the ones I heard of were due to loss of communication with the repository database. Have you checked your connection there from the server? How is the ping? Also have you checked your network connection (ping from BO server to DB server)? As far as I know there have not been any patches for random CMS restarting. BTW, are you sure it is CMS and not your web server? Also, try to see if any errors were logged with the restart (on windows server, look your the Event Viewer, for example). If none of this gets you anywhere, you are going to have to engage BO support and they will probably have you activate CMS tracing. Yikes! That is a barrel of monkeys!
Pending Job List: Have you looked at Instance Manager? It is available at the URL: http:///yourserver/businessobjects/enterprise115/adminlaunch/launchpad.html
Let me know if that meets your needs, if not then I can dig up a Query Builder query for that, I think.
Good luck!
Julian
I’ve just come across this problem with a customer running Crystal Reports Server XI R2 SP4. They could run 2 schedules, then everything went into pending.
Worryingly, BO are now distributing CRS XI R2 WITH SP4 slipstreamed so you don’t have the option of staying with SP2.
Fortunately, it looks like the interim patch worked, but as they don’t have BO support, can’t get hold of Fix Pack 4.5. Annoyingly, only Service Packs are available for CRS.
Hi,
Do you know if this problem is fixed in SP5?
Thank you.
I believe is fixed as of SP5 Fix Pack 2.
Hi Julian,
We’re starting to go bonkers here and would greatly appreciate any help on our issue.
We’ve recently upgraded to SP4 from SP2, and we’re getting problems with our Crystal reports intermittently failing in Infoview.
We took backups of our SP2 structure and configuration and followed this to a tie when doing the install of SP4. I have checked the Event Viewer/BO logs, and have found nothing glaring that could be causing this error.
I’ve checked all the timeout parameters in the CMC on both of our live servers, checked the configuration again and again in the CCM and all seems to be as they were in SP2, yet we’re getting report failures.
The business are going ape to put it mildy and I’d like to sort this out as quickly as I can, but I’m getting a touch stuck on where to go next.
The reason we upgraded to SP4 was because a different Project couldn’t have its reports on an SP2 platform (due to the development being completed on SP3 :o)) so they decided to update our current SP2 platform with SP4 to cater for these new reports.
Any help would be great. I can post more detail on our system if needed.
Cheers,
Lloyd.
Hi Lloyd,
Sorry for the delay, thanks for the question. The first thing that comes to mind is whether any nodes (machines) are not on the same subnet (including the database). I am curious if SP4 introduces some kind of problems with latency and or firewalls between nodes (and their BO servers/services).
Could you post what the intermittent error is exactly? Also please describe the conditions of the failure (on open, after a few minutes, something else).
Thanks, Julian
Cheers Julian, appreciate the response.
The error is:
“Failed to retrieve data from the database” – There is no Oracle code, so it’s extremely hard to pin down this generic message.
We seem to have tied the issue down to larger data sets. We can split a “failing” report into say 3 smaller reports and each one is successful, yet running the report as a whole fails. We have retrieved reports on SP2 in excess of 40k rows, now a 9k row reports fails on SP4.
The schedule normally runs for approx 2-3 min, then brings back that failure message.
I have succesfully ran the failing reports locally via Crystal Designer also.
I built a standalone version of SP2 on our Dev enviroment – everything worked. I then rebuilt the environment with SP4 – the failures began.
Thanks for your help, appreciated.
Lloyd.
Hey Lloyd,
Here are some thoughts. First take a look at these two postings on Experts Exchange:
Automated reports randomly fail in Crystal Reports Server: “failed to retrieve data from the database”
Failed to retrieve data from database Crystal Reports 11
After reading those I was beginning to suspect that you may need to update the Oracle client/middleware on your server. Perhaps SP2 was more forgiving of or compatible with an older or mismatched (to the database) client version. SP4 might be less compatible or forgiving of such an issue. If nothing else, it is worth a try. It is the same dart that consultants will throw for you if you pay them thousands of Euros.
Please remember that I am not as fluent or experienced with Crystal Reports as I would like to be, so I may not be able to help much here.
Just curious, do scheduled WebI reports with multiple queries have the same issue?
Good Luck, Julian
Hey Julian,
Once again, thanks for your reponse.
I will certainly look at those links now and see if we can progress with them, thanks.
I have ran a few tests with Webi reports, and things were fine. However, due to the nature of our reports which uses an XML data provider combined with an Oracle data provider, the result set only gets into Oracle ‘on the fly’. So basically, a user will schedule a report and the XML data provider will populate Oracle with the result set. Our 2nd data provider will then retreive the data supplied via the XML source.
Unfortunatley, it’s not a straightforward solution as we use ETL to initially get the data. So during a WEBI test, we are just pulling the data ALREADY stored in the Oracle table. Hope that makes sense? Our existing solution populates and retreives practically at the same time.
Cheers,
Lloyd.
Julian,
How are things?
Just thought I’d pop by and send you an update to our issue. It’s pretty much resolved now, thankfully.
During deployment of SP4, the CRDBXMLServer.jar file sends a duplicate request for any report that runs over 1 minute (for our setup anyway! Not sure if this is applicable SP4 wide!). The intial request (during the 1st minute) goes away, gets the data and sends the data to Oracle. The duplicate request (after the 1 min elapse) then does the EXACT same thing, but doesn’t find the data as the initial request has already retrieved it. This then produces the above error.
As a workaround, we have the SP2 version of this .jar file (not ideal) and everything works ok. BO themselves are looking to see if this can be fixed correctly with SP4.
Thanks for all your help on this.
Cheers,
Lloyd.
Hey Lloyd,
Great news! Thanks for updating us on the work-around. Please let us know if BO releases an LA Fix for this bug.
Best Regards, Julian
I have done a scheduling of report.Next day I have seen it has not been saved automatically to the destination path.Process FAILED with the following error……
destination DLL disabled. []: [CrystalEnterprise.Ftp]
Kindly assist me what to do in this issue….
It sounds like the scheduled report is configured to deliver to an FTP site. The error also seems to indicate that you have not properly enabled/configured FTP as a destination on your job server.
Hi Julian,
I have three reports in test environment, Recently we moved them to production. We tried to schedule them. All are scheduled well except one report. We are getting an error like “30270: An internal error occurred while calling the ‘processDPCommands’ API. (Error: WIS 30270) BOException caught:RESULT=80004005;WHAT=RefreshBatch failed “. The report is running fine, Only the problem is getting faild statuis when we schedule it. And one main thing, In that report we use prompts. When i select different values in those prompts and save it as different report, Scheduled well.But the original report moved from test to production is not scheduling. Can you help in this issue? Thanks in advance.
Hi Julian,
We are running on XI 3.1 with SP4. Since the install of SP4, our heavy publications(900 Enterprise recipients and around 5,000 page webi report in PDF format) increased from 30 minutes run time to 6 hours. We also noticed that when generating a webi report, the queries finish in time, but it takes extra time to show the results, especially if there are grouping sections or breaks involved. Also, it takes much longer to convert a webi report to PDF format. Have you seen this behavior before? Any suggestions on how to get around this issue?
Thanks,
Anton
Hi Anton, I’m sorry for the trouble Service Pack 4 is causing you. Are you certain that you upgraded all servers in the cluster and also that you redeployed the web apps? Probably you did, but just thinking of areas of possible issue causes. You might try to install the latest Fix Pack for SP4, if you haven’t. Sorry my mind I’d on vacation now and I’m not army fullest.
Hi Julian,
We are having BOXI 3.1 with Fp 3.4 on Linux machine(weblogic application server+apache webserver+oracle db as repository) .Scheduling works perfectly fine. Only refreshing long running reports throws the WIS 30270 Dp process Command Error. and some sesiion related errors. Error is intermittent. some time appears and some time not. That too more in peak usage time. small queries run perfectly fine no issues. But not the same case with Long running queries. we have enabled tracing.. In tracing ICDZ_i:>>:Synchro=true
2012/04/17 15:33:15.909|>=| | |11059|1626463120|{|||||||||||||||CCDZMgr:canCreateDoc
2012/04/17 15:33:15.909|==| | |11059|1626463120| |||||||||||||||ctProcessInfo:VM size (Mb):226
2012/04/17 15:33:15.909|==| | |11059|1626463120| |||||||||||||||ctProcessInfo:Allocated size (Mb):211
2012/04/17 15:33:15.909|>=| | |11059|1626463120|}|||||||||||||||CCDZMgr:canCreateDoc: 0
2012/04/17 15:33:15.909|>=| | |11059|1626463120|{|||||||||||||||WICDZ_i:StateManagementWelcome
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:[Cdztools Error Trace:BEGIN]
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:PREFIX:ERR
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:COMPONENT:WIS
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:ERRORCODE:30111
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:ERRORTYPE:SUPERVISOR
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:MESSAGE:DocToken invalid.Token value: dx5dx5dx37630_D_X_1277756363
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:BORESULT:5
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:MODULENAME:WICDZServer
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:FILENAME:kdgWICDZ_i_member.cpp
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:LINEPOSITION:2736
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:DPNAME:
2012/04/17 15:33:15.910|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:DPLONGNAME:
2012/04/17 15:33:15.911|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:DPID:
2012/04/17 15:33:15.911|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:COMMANDNAME:
2012/04/17 15:33:15.911|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:COMMANDID:
2012/04/17 15:33:15.911|==| | |11059|1626463120| |||||||||||||||ct_cdz_error_context:[Cdztools Error Trace:END]
kdgWICDZ_i_member.cpp:2221:void dgWICDZ_i::StateManagementWelcome(ibo_ptr&, const bo_wstring&): TraceLog message 306165
2012/04/17 15:33:15.911|>>|E| |11059|1626463120| |||||||||||||||**ERROR:WICDZ_i:StateMgmt: : 30111 : Invalid Token; found:dx5dx5dx37630_D_X_1277756363 expected: P29PI11059KT28 [kdgWICDZ_i_member.cpp;2221]
kdgWICDZ_i_member.cpp:2209:void dgWICDZ_i::StateManagementWelcome(ibo_ptr&, const bo_wstring&): TraceLog message 306166
2012/04/17 15:33:15.911|>=|E|X|11059|1626463120|}|||||||||||||||WICDZ_i:StateManagementWelcome: 0.002
kdgWICDZProc.cpp:432:void WICDZProcBasic::BOExceptionHandler(BOException&, ASFTypes_UTF8String*&): TraceLog message 306167
2012/04/17 15:33:15.911|>>|E| |11059|1626463120| |||||||||||||||**ERROR:WICDZ_i:BOException : BOException caught:RESULT=80004005;WHAT=30111 : Invalid Token; found:dx5dx5dx37630_D_X_1277756363 expected: P29PI11059KT28 [kdgWICDZProc.cpp;43
from the error code not able undersatnd what BO is saying..? Your help would be highly appreciated.. thanks.