try { // CompareRecord record = new CompareRecord(); Connection conn = new CompareRecord().getConection("eliteddaprd","eliteddaprd","192.168.14.104","1521"); ResultSet res = null; if (conn != null){ Statement stmt = conn.createStatement(); res = stmt.executeQuery("select rowindx,ADDRLINE1 from dedupinitial order by rowindx"); } Map<Integer,String> adddressMap = new LinkedHashMap<Integer, String>(); if (res != null){ System.out.println("result set is not null "); while(res.next()){ adddressMap.put(res.getInt(1),res.getString(2)); } } System.out.println("address Map size =========> "+adddressMap.size()); Iterator it = adddressMap.entrySet().iterator(); int count = 0; int min = 0; while (it.hasNext()){ Map.Entry pairs = (Map.Entry)it.next(); Pattern p = Pattern.compile("[,\\s]+"); Integer outerkey = (Integer)pairs.getKey(); String outerValue = (String)pairs.getValue(); //System.out.println("outer Value ======> "+outerValue); String[] outerresult = p.split(outerValue); Map.Entry pairs2 = null; count++; List<Integer> dupList = new ArrayList<Integer>(); Iterator innerit = adddressMap.entrySet().iterator(); boolean first = true; while (innerit.hasNext()){ //System.out.println("count value ===> "+count); int totmatch = 0; if(first){ if(count == adddressMap.size()){ break; } for(int i=0;i<=count;i++){ pairs2 = (Map.Entry)innerit.next(); } first = false; } else{ pairs2 = (Map.Entry)innerit.next(); } Integer innterKey = (Integer)pairs2.getKey(); String innerValue = (String)pairs2.getValue(); //System.out.println("innrer value "+innerValue); String[] innerresult = p.split(innerValue); for(int j=0;j<outerresult.length;j++){ for(int k=0;k<innerresult.length;k++){ if(outerresult[j].equalsIgnoreCase(innerresult[k])){ //System.out.println(outerresult[j]+" Match With "+innerresult[k]); totmatch++; break; } } } min = Math.min(outerresult.length, innerresult.length); if(min != 0 && ((totmatch*100)/min) > 50) { //System.out.println("maching inner key =========> "+innterKey); dupList.add(innterKey); } } //System.out.println("Duplilcate List Sisze ===================> "+dupList.size()+" "+outerkey); } System.out.println("End =========> "+new Date()); } catch (Exception e) { e.printStackTrace(); }
Here ResultSet have processed around 500000 records, but it will give me error like:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.resize(HashMap.java:508) at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:406) at java.util.HashMap.put(HashMap.java:431) at spite.CompareRecord.main(CompareRecord.java:91)
I know this error comes because of VM memory, but don't know how to increase it in Eclipse?
What do I do if I have to process even more than 500,000 records?
OutOfMemoryError: Java heap space. 1) An easy way to solve OutOfMemoryError in java is to increase the maximum heap size by using JVM options "-Xmx512M", this will immediately solve your OutOfMemoryError.
lang. OutOfMemoryError exception. Usually, this error is thrown when there is insufficient space to allocate an object in the Java heap. In this case, The garbage collector cannot make space available to accommodate a new object, and the heap cannot be expanded further.
In Run->Run Configuration find the Name of the class you have been running, select it, click the Arguments tab then add:
-Xms512M -Xmx1524M
to the VM Arguments section
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With