Showing posts with label Alfresco. Show all posts
Showing posts with label Alfresco. Show all posts

Enable JMX access in Alfresco to monitor JVM memory usage through VisualVM or Jconsle

One small post but useful for those who wants to monitor Alfresco's JVM memory usage through VisualVM.

To enable connection through JMX, first you need to enable it in setenv.sh file. Simply add below line and restart Alfresco. You should be able to connect to JMX.

Linux:
export JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote -Djava.net.preferIPv4Stack=true"

Winodws:
set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote -Djava.net.preferIPv4Stack=true

In VisulVM, you can connect using below string. Please select "Add JMX connection".

Alfresco Fetal Error: libdcpr.so+0x1330b Java Crashed

You would have seen java crash error log. This message does not show any information like due to which class or process java crashed. But finally we found that one culprit document we were uploading in Alfresco, which was having "Hindi" characters. So if you come across this kind of error, check your recently uploaded document if it is having language other than English.

Hope this will help!

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fbcd266430b, pid=8028, tid=140448967481088
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libdcpr.so+0x1330b]  signed char+0xbb
#
# An error report file with more information is saved as:
# /opt/alfresco-4.0.1/hs_err_pid8028.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Alfresco Performance tuning by setting JVM memory parameters

Since last couple of days I was working on Alfresco's JVM tuning. I faced below issues when it was configured previously.
  • OldGen space was utilized completely
  • Per Survivor Space was too utilized completely
  • Full GC was taking time which leads to stop application responding for time being
My previous JVM settings were as given below:

export JAVA_OPTS="-Xms2048m -Xmx4096m -Xss1024k -XX:MaxPermSize=512m -XX:NewSize=1024m -XX:+UseConcMarkSweepGC -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=64m -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco4 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dsun.security.ssl.allowUnsafeRenegotiation=true -Djava.net.preferIPv4Stack=true"

After making changes in few parameters, I got below settings which works perfectly fine. I had to increase memory allocation and tune other parameters.

export JAVA_OPTS="-Xms4096m -Xmx8192m -XX:MaxPermSize=512m -XX:NewSize=3072m -XX:+UseConcMarkSweepGC -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=64m -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco4 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dsun.security.ssl.allowUnsafeRenegotiation=true -Djava.net.preferIPv4Stack=true"

I have taken screenshot from VisualVM.

Heap Memory and CPU Usage:

OldGen and Eden Space usage:


Alfresco Backup and Restore Process

This post describes backup and restore process of Alfresco. Here I have mentioned steps to backup alfresco's content store(alf_data), lucene index data and Database.

Alfresco Backup process:
There are two methods by which we can do alfresco backup. First one is Hot backup, when alfresco is running and you want to do backup without shutting down alfresco. Second is is Cold backup, when you can shutdown alfresco and do backup. Most of people prefer doing Hot backup, as no one will prefer to shutdown alfresco everytime before doing backup and turn back on when finished. Also hot backup also gives surety of data consistency except latest lucene index. But yes, when you restore backup from hot backup set, Alfresco will rebuild Lucene indexes. So follow below steps for Alfresco backup.

Cold Backup Process:
  1. Stop Alfresco.
  2. Backup the database Alfresco is configured to use, using your database vendor's backup tools.
  3. Backup the Alfresco “dir.root” and “dir.indexes” directory. You can get location of these two paths from alfresco-global.properties file.
  4. Store both the database and Alfresco “dir.root/dir.indexes” backups together as a single unit. For example, store the backups in the same directory or compressed file.
  5. Start Alfresco.
Hot Backup Process:

It is absolutely critical that hot backups are performed in the following order:
  1. Make sure you have a "backup-lucene-indexes" folder under 'dir.root' if not specified separate path. Backup-lucene-indexes can be useful when you want to restore indexes.
  2. Backup the database Alfresco is configured to use, using your database vendor's backup tools.
  3. As soon as the database backup completes, backup specific subdirectories in the Alfresco dir.root.
Sub-directories would be as listed below.
  • contentstore
  • contentstore.deleted
  • audit.contentstore
  • backup-lucene-indexes
Finally, store both the database and Alfresco dir.root backups together as a single unit. For example, store the backups in the same directory or in a single compressed file.
Do not store the database and dir.root backups independently, as that makes it unnecessarily difficult to reconstruct a valid backup set for restoration.


'Note': Alfresco generates "backup-lucene-indexes" everyday at 3:00 am, so make sure that database backup time does not overlap with this schedule. The "backup-lucene-indexes" generation should be finished before you start the SQL backup. So if backup-lucene-indexes is running at 3:00 am then you can set database backup at 4:00 am considering that your alfresco system is not being used during this time.

Restore Process

1. Stop Alfresco.
2. Copy the existing dir.root to a temporary location.
3. Restore dir.root and dir.indexes from backup.
4. If you are restoring from a hot backup, rename <dir.indexes>/backup-lucene-indexes to <dir.root>/lucene-indexes.
5. Restore the database from the database backups.
6. Start Alfresco.

You would see in logs that alfresco is rebuilding indexing. But index rebuilding will be incremental so it wont take time.

Alfresco start error : internal error: ObjID already in use

You may come across below error. That happens due to hostname which does not resolved through any IP.




Solution: Make hosts file entry so that your hostname resolves IP address of your IP address.
filename: /etc/hosts


Alfresco Certificate Error javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException

This post applies to Alfresco 4. When we install Alfresco and start it, it shows error in catalina.out as shown below.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1699)
        at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
        at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
...
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:289)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:263)

Alfresco ships with certificates which are already created, but when you install it has already reached expiration limit.

So below steps will help you to regenerate SSL certificate for Alfresco.

1. First download script to generate SSL certificate from here.
2. Keep this file in alfresco-4.0.1/alf_data/keystore folder.
3. Make it executable using below command.
chmod a+x generate_keystores.sh
4. You may need to change "ALFRESCO_HOME" and other parameters in that script based on your alfresco directory structure.
5. Now run that script file using below command.
sh generate_keystores.sh

Once you run that script, it will restart Alfresco then will ask you to provide details to generate new SSL certificate. Once done with certificate creation, you can start alfresco with valid certificate.

Log rotation / retention in Alfresco

Alfresco creates daily log files in ALFRESCO_HOME folder with alfresco.log.{date} and another log file location is ALFRESCO_HOME/tomcat/logs/catalina.{date}.log.

These files are created everyday, but bydefault there is no configuration for removing old log files which are not needed. So I have jotted down steps to configure Alfresco Log rotation and retention.

By default Alfresco uses Log4j library for logging. While using Log4j, by default it uses "org.apache.log4j.DailyRollingFileAppender" class. Which helps to create log files daily by appending date at end of log file in ALFRESCO_HOME folder. But this class does not provide facility to remove old files.

So we can use "org.apache.log4j.RollingFileAppender" class from log4j, which provides facility of rotating log files and removing old log files. But there is a con, that it does rotates log files based on Size we provide, not daily bases. We can give value of "MaxFileSize" in log4j.properties file, so whenever it cross the size limit it will rename alfresco log file with alfresco.log.1 then alfresco.log.2 etc.
Here you can give limit, how many numbers of such log files you want to retain. That can be done through "MaxBackupIndex" parameter. Lets say if we keep "MaxBackupIndex" to 10. Then Alfresco will create file from alfresco.log.1 to alfresco.log.10. When it reaches to alfresco.log.10 and again if alfresco.log file exceeds size of "MaxFileSize" then it will rename alfresco.log file to alfresco.log.1 and alfresco.log.1 file will be renamed to alfresco.log.2...and so on. The older alfresco.log.10 file will be deleted in that case, as we have set "MaxBackupIndex" to 10.

So before we start lets remove tomcat's default java.util.logging configuration and add log4j.

You will have to download log4j jar file from here. But if you want, you can get it from tomcat/webapps/alfresco/WEB-INF/lib/log4j-1.2.15.jar location too. Apart from log4j jar file you need to get tomcat-juli.jar and tomcat-juli-adapters.jar files from this location.

So after downloading log4j jar, tomcat-juli.jar and tomcat-juli-adapters.jar files follow below steps.
1. Copy tomcat-juli.jar and tomcat-juli-adapters.jar files into tomcat/lib folder.
2. Create log4j.properties file in tomcat/lib folder with below content.

3. Now  copy downloaded tomcat-juli.jar into tomcat/bin directory. You need to replace existing tomcat-juli.jar file.

4. Then remove tomcat/conf/logging.properties file.

Now lets make changes at Alfresco side.There are two location where Alfresco has configured log4j.properties file for Alfresco and Share.

tomcat/webapps/alfresco/WEB-INF/classes/log4j.properties
tomcat/webapps/share/WEB-INF/classes/log4j.properties

You can add below mentioned snippet in log4j.properties files at both location to apply log rotation and retention. Here we have used "File" appender for alfresco.log file and "CONSOLE" appender for catalina.out file.

note: you can add below two appenders in log4j.properties file, but you will also have to remove words/line which are strikethrough.


# Set root logger level to error
log4j.rootLogger=error, Console, File, CONSOLE

###### Console appender definition #######

# All outputs currently set to be a ConsoleAppender.
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{ISO8601} [%x] [%p] [%c{3}] [%t] [%r] %m%n

###### File appender definition #######
log4j.appender.File=org.apache.log4j.RollingFileAppender
log4j.appender.File.File=alfresco.log
log4j.appender.File.Append=true
log4j.appender.File.MaxFileSize=10MB
log4j.appender.File.MaxBackupIndex=10
log4j.appender.File.layout=org.apache.log4j.PatternLayout
log4j.appender.File.layout.ConversionPattern=%d{ABSOLUTE} %-5p [%c] %m%n

log4j.appender.CONSOLE=org.apache.log4j.RollingFileAppender
log4j.appender.CONSOLE.File=${catalina.base}/logs/catalina.out
log4j.appender.CONSOLE.MaxFileSize=10MB
log4j.appender.CONSOLE.MaxBackupIndex=10
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{DATE} %x %-5p [%c{3}] %m%n




Once above changes done you can restart Alfresco and check the logs.

Hope this post helps to rotate log files in Alfresco.





Alfresco Cluster Validation


This post would be helpful, if you have setup the Alfresco cluster and found issues(L2 cache replication, Startup error, failed cluster message transmission,etc). Below mentioned points should be taken care before you start testing Cluster in Alfresco.
(1) Make sure the hostname of both Alfresco server are reachable on Resolved IP address.
Ex: you can check with "hostname" command. This command will give you hostname. After that you can try to ping hostname. If you are getting output like "Reply from...", that means hostname is rechable. If you are not getting any other message, then contact system Admin.
(2) Validate Max Open File limit is configured. By default, it is 1024. You can check if Max Open File limit using below command.
First login with the user, by which you are going to start Alfresco. Then run below command.
ulimit -Hn
ulimit -Sn
If you are getting output of above command as "1024" or less. Then modify  /etc/security/limits.conf file and add below lines. Here we are considering that Alfresco is running as "alfresco" user.
alfresco soft nofile 1024
alfresco hard nofile 10240
After you give this command, restart the server. Then check again with ulimit command.
(3) Check if firewall is on or off. If firewall is on in Server, then you may need to open Open ports. Port depends on the setup of Alfresco Custer.
(4) Validate the Java version is Sun JDK.
(5) Validate that the directory in which Alfresco is installed does not contain spaces.
(6) Validate that the directory in which the JVM is installed does not contain spaces.
(7) Validate that the directory Alfresco will use for the repository (typically called alf_data) is both readable and writeable by the operating system user that the Alfresco process will run as.
(8) Validate that you can connect to the database as the Alfresco database user, from the Alfresco server. And you have provided proper library file to connect.
(9) Validate that the character encoding for the Alfresco database is UTF-8. (MySQL only) Validate that the storage engine for the Alfresco database is InnoDB.
(10) If you are using Openoffice then make sure that OpenOffice is able to run in headless mode

Register Alfresco FSR as Windows Service running on 64 bit using YAJSW

(1) First download the latest stable build of YAJSW from here.
(2) Unzip yajsw-stable-11.0.zip in D:\apps\yajsw folder.
(3) Install Alfresco FSR into D:\apps\FSR folder. (Ex. D:\apps\FSR)
(4) Copy all *.jar files from D:\apps\FSR folder to D:\apps\yajsw\lib folder.
(5) Copy application-context.xml and shutdown-context.xml files from D:\apps\FSR to D:\apps\yajsw\bat folder.
(6) Now open coammand prompt. Goto D:\apps\FSR directory and run below command.

jar cf deployment-config.jar deployment.properties deployment/*

Above command will create “deployment-config.jar” jar file in D:\apps\FSR directory.
(7) Now Start FSR manually and check its PID in “Task Manager”. Note down the PID of Alfresco FSR.
(8) Now goto D:\apps\yajsw\bat folder and run below command.

genConfig.bat “PID of Alfresco FSR”

Ex: genConfig.bat 1492

(9) This will generate the wrapper.conf file under D:\apps\yajsw\conf folder. Now Open wrapper.conf file and set below path.
Note: please make changes of path as per your configuration.

wrapper.working.dir=D:\\apps\\FSR\\
wrapper.java.app.mainclass=org.alfresco.deployment.Main
wrapper.console.title=Alfresco Deployment Server
wrapper.ntservice.name=Alfresco Deployment Server
wrapper.ntservice.displayname=Alfresco Deployment Server
wrapper.ntservice.description=Alfresco Deployment Server
wrapper.java.command = C:\\Program Files\\Java\\jdk1.6.0_22\\bin\\java
wrapper.app.parameter.1 = application-context.xml
wrapper.java.additional.1=-server
wrapper.java.additional.2 = -Djava.ext.dirs=.;C:\\Program Files\\Java\\jdk1.6.0_22\\jre\\lib\\ext


(10) Execute Alfresco FSR as console application by calling and confirm that Alfresco FSR is running correctly.

D:\apps\yajsw\bat\runConsole.bat

(11) If everything is ok then install Alfresco FSR as Windows Service call using below command.

D:\apps\yajsw\bat\installService.bat

Done!!!

Google