Thursday, February 6, 2014

multithreading MessagePack usage without java PermGen memory error


If you use new MessagePack() too many time, perm gem memory error occurs soon.

MessagePack msgpack = new MessagePack();
byte[] bytes = msgpack.write(object);

You can test this with
-XX:+TraceClassLoading -XX:+TraceClassUnloading
VM parameters or JConsole.

[Loaded learningtest.msgpack.MessagePackTest$MyMessage_$$_Template_1527741453_0 from __JVM_DefineClass__]
MyMessage [name=msgpack, version=0.6]
[Loaded learningtest.msgpack.MessagePackTest$MyMessage_$$_Template_831556519_0 from __JVM_DefineClass__]
MyMessage [name=msgpack, version=0.6]
[Loaded learningtest.msgpack.MessagePackTest$MyMessage_$$_Template_1872305359_0 from __JVM_DefineClass__]
MyMessage [name=msgpack, version=0.6]

MessagePack.write()
TemplateRegistry.lookup()
TemplateRegistry.lookupAfterBuilding()
TemplateRegistry.buildAndRegister()
AbstractTemplateBuilder.buildTemplate()
DefaultBuildContext.buildTemplate()
BuildContext.build()
BuildContext.createClass()
CtClass.toClass()


But, writing with the MessagePack instance is not thread-safe.
msgpack.setOutputStream(outputStream);
msgpack.write(object);

Insteads, you can should the packer and the unpacker from a singleton MessagePack instance

MessagePack msgpack = new MessagePack(); // singleton
Packer packer = msgpack.createPacker(outputStream); // createPacker every time
packer.write(object);

for (int i = 0; i < 100; i++) {
final int iTask = i;
tasks.add(new Thread() {

@Override
public void run() {
try {
System.out.println(iTask + " start ");
File file = new File("eraseme" + iTask);
FileOutputStream outputStream = null;

outputStream = new FileOutputStream(file);
SomeClass info = new SomeClass();
try {
packer.write(info);
catch (IOException e) {
System.out.println(iTask + " write failed " + e.getMessage());
assertTrue(false);
}
outputStream.close();
FileInputStream inputStream = new FileInputStream(file);
Unpacker unpacker = msgpack.createUnpacker(inputStream);
// Deserialize directly using a template
SomeClass reply = null;
try {
reply = (ReplyMessage.ImageInfoMessage) unpacker.read(SomeClass.class);
catch (IOException e) {
System.out.println(iTask + " read failed " + e.getMessage());
assertTrue(false);
}
System.out.println(iTask + " finished " + reply + " " + file.getAbsolutePath());
catch (Exception exc) {
System.out.println("exc " + exc.getMessage());
assertTrue(false);
}
};
});
}
for (Thread task : tasks) {
task.start();
}
for (Thread task : tasks) {
task.join();
}

assertTrue(true);


Or, we should use pool of MessagePack instances.
I suggest the apache commons pool.
http://commons.apache.org/proper/commons-pool/



No comments:

Post a Comment