How can I significantly reduce the size of an encoded string (hash) while keeping it reversible?

Question

I'm working on a Java (JDK 21) application where I need to transmit a very large string (around 20,000 characters). This string is an encoded structure (like a hash or serialized data), and I need:

The encoded result to be as short as possible, ideally under 4,000 characters.

The process to be fully reversible (I must recover the original string exactly).

To avoid using any external storage, such as Redis, a database, or cache.

A clean Java-only solution, no dependencies on external tools or services.

What I've tried so far:

Compressing the string using Deflater, GZIP, and encoding the result using Base64 (URL-safe).

The result is still too long — for example, around 13,000+ characters with Base64URL-encoded Deflate.

So my question is:

Is there any more efficient way — algorithmic, custom encoding, or otherwise — to reduce the size of a large string, while keeping it reversible and implemented in plain Java?

Any suggestions, techniques, or insights would be very much appreciated!

public class EncryptionUtil {

    private static final String AES = "AES/ECB/PKCS5Padding";
    private static final String SECRET = "secret";
    private final SecretKey secretKey;
    private final RefinementEncryptionConfig encryptionConfig;

    public FilterEncryptionUtil(RefinementEncryptionConfig encryptionConfig) {
        this.encryptionConfig = encryptionConfig;
        this.secretKey = new SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "AES");
    }

    public String generateEncryptedUrl(List<Refinement> refinements) {
        String compactString = generateCompactRefinementString(refinements);
        return encryptCompactString(compactString);
    }

    public String encryptCompactString(String compactString) {
        try {
            Cipher cipher = Cipher.getInstance(AES);
            cipher.init(Cipher.ENCRYPT_MODE, secretKey);
            byte[] encrypted = cipher.doFinal(compactString.getBytes(StandardCharsets.UTF_8));
            return Base64.getUrlEncoder().withoutPadding().encodeToString(encrypted);
        } catch (Exception e) {
            throw new RuntimeException("Encryption failed", e);
        }
    }

    private String generateCompactRefinementString(List<Refinement> refinements) {
        Map<String, String> navigationAliases = encryptionConfig.getEncryptIdentifiers();

        StringBuilder buffer = new StringBuilder();
        Map<String, List<String>> valueRefinements = new LinkedHashMap<>();
        List<String> rangeRefinements = new ArrayList<>();

        for (Refinement r : refinements) {
            String alias = navigationAliases.getOrDefault(r.getNavigationName(), r.getNavigationName());

            if ("Range".equalsIgnoreCase(r.getType())) {
                String rangeStr = alias + ":R:" + r.getLow() + "-" + r.getHigh();
                rangeRefinements.add(rangeStr);
            } else {
                valueRefinements.computeIfAbsent(alias, k -> new ArrayList<>()).add(r.getValue());
            }
        }

        if (!rangeRefinements.isEmpty()) {
            buffer.append(String.join("#", rangeRefinements));
        }

        for (Map.Entry<String, List<String>> entry : valueRefinements.entrySet()) {
            if (!buffer.isEmpty()) buffer.append("#");
            String part = entry.getKey() + ":V:" + String.join("|", entry.getValue());
            buffer.append(part);
        }

        return buffer.toString();
    }

    public List<Refinement> decodeEncryptedUrl(String encryptedBase64) {
        try {
            byte[] decoded = Base64.getUrlDecoder().decode(encryptedBase64);

            Cipher cipher = Cipher.getInstance(AES);
            cipher.init(Cipher.DECRYPT_MODE, secretKey);
            byte[] decryptedBytes = cipher.doFinal(decoded);
            String compactString = new String(decryptedBytes, StandardCharsets.UTF_8);

            return parseCompactRefinementString(compactString);
        } catch (Exception e) {
            throw new RuntimeException("Decryption failed", e);
        }
    }

    private List<Refinement> parseCompactRefinementString(String compactString) {
        List<Refinement> refinements = new ArrayList<>();
        Map<String, String> aliasToNav = encryptionConfig.getDecryptIdentifiers();

        String[] parts = compactString.split("#");
        for (String part : parts) {
            String[] section = part.split(":");
            if (section.length < 3) continue;

            String alias = section[0];
            String type = section[1];
            String data = section[2];
            String navName = aliasToNav.getOrDefault(alias, alias);

            if ("R".equals(type)) {
                String[] range = data.split("-");
                if (range.length == 2) {
                    Refinement r = new Refinement();
                    r.setType("Range");
                    r.setNavigationName(navName);
                    r.setLow(range[0]);
                    r.setHigh(range[1]);
                    refinements.add(r);
                }
            } else if ("V".equals(type)) {
                String[] values = data.split("\\|");
                for (String value : values) {
                    Refinement r = new Refinement();
                    r.setType("Value");
                    r.setNavigationName(navName);
                    r.setValue(value);
                    refinements.add(r);
                }
            }
        }

        return refinements;
    }

}

Additional Context:

To clarify some constraints:

I cannot use a POST request — the encoded data must be transmitted as a URL parameter in a GET request.

This is why minimizing the resulting string size is critical (ideally < 4000 characters to stay within typical URL limits).

I'm encoding a large list of integers (e.g., [6957109, 11370445, ...], potentially up to 20,000 unique values).

These numbers are not repeated and must be preserved exactly on decode.

The class I shared is responsible for encoding this data, and I must keep the entire process self-contained in Java, with no external services, no database (e.g., Mongo), and no temporary cache (e.g., Redis).

I’m not allowed to change the architectural decision — including mapping the data to a UUID or external lookup.

So, my current idea is to compress or represent the list of integers in a more compact form before encoding.

I've tried Deflate + Base64 (URL-safe), but with 20,000 numbers, the final string still ends up being too large. I’m looking for ideas such as:

Better compression strategies for lists of integers in Java.

Custom number encoding (e.g., delta encoding, variable-length encoding).

Any creative reversible transformation that produces a smaller string.

Any insights or Java-only techniques would be incredibly helpful. Thank you!

public static byte[] compress(List<String> numberStrings) throws IOException {
        if (numberStrings == null || numberStrings.isEmpty()) return new byte[0];

        // Convert strings to longs and sort
        List<Long> numbers = numberStrings.stream()
                .map(Long::parseLong)
                .sorted()
                .toList();

        // Delta encoding with variable-length bytes
        List<Byte> deltas = new ArrayList<>();
        long prev = numbers.get(0);
        deltas.add((byte) (prev & 0xFF)); // Store first number's least significant byte

        for (int i = 1; i < numbers.size(); i++) {
            long delta = numbers.get(i) - prev;
            prev = numbers.get(i);
            while (delta > 0) {
                byte b = (byte) (delta & 0x7F);
                delta >>>= 7; // Use unsigned right shift
                if (delta > 0) b |= 0x80;
                deltas.add(b);
            }
        }

        // Deflate compression
        Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION, true);
        deflater.setInput(toByteArray(deltas));
        try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
            byte[] buffer = new byte[1024];
            deflater.finish();
            while (!deflater.finished()) {
                int count = deflater.deflate(buffer);
                baos.write(buffer, 0, count);
            }
            return baos.toByteArray();
        } finally {
            deflater.end();
        }
    }

    private static byte[] toByteArray(List<Byte> bytes) {
        byte[] result = new byte[bytes.size()];
        for (int i = 0; i < bytes.size(); i++) {
            result[i] = bytes.get(i);
        }
        return result;
    }

    public static List<Integer> decompress(byte[] compressed) {
        if (compressed == null || compressed.length == 0) return new ArrayList<>();

        // Inflate decompression
        Inflater inflater = new Inflater(true);
        inflater.setInput(compressed);
        try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
            byte[] buffer = new byte[1024];
            while (!inflater.finished()) {
                int count = inflater.inflate(buffer);
                baos.write(buffer, 0, count);
            }
            return decodeDeltas(baos.toByteArray());
        } catch (Exception e) {
            e.printStackTrace();
            return new ArrayList<>();
        } finally {
            inflater.end();
        }
    }

    private static List<Integer> decodeDeltas(byte[] decompressedBytes) {
        List<Integer> numbers = new ArrayList<>();
        int value = decompressedBytes[0] & 0xFF;
        numbers.add(value);

        for (int i = 1; i < decompressedBytes.length; ) {
            int delta = 0;
            int shift = 0;
            while (i < decompressedBytes.length && (decompressedBytes[i] & 0x80) != 0) {
                delta |= (decompressedBytes[i] & 0x7F) << shift;
                shift += 7;
                i++;
            }
            if (i < decompressedBytes.length) {
                delta |= (decompressedBytes[i] & 0x7F) << shift;
                i++;
            }
            value += delta;
            numbers.add(value);
        }
        return numbers;
    }

    public static String bytesToHex(byte[] bytes) {
        StringBuilder sb = new StringBuilder();
        for (byte b : bytes) sb.append(String.format("%02X", b));
        return sb.toString();
    }

    public static void main(String[] args) throws IOException {
        Set<String> uniqueNumbers = new HashSet<>();
        Random random = new Random();
        long min = 1_000_000_000L;
        long max = 9_999_999_999L;

        while (uniqueNumbers.size() < 2000) {
            long number = min + (long) (random.nextDouble() * (max - min));
            uniqueNumbers.add(String.valueOf(number));
        }
        List<String> listSkus = new ArrayList<>(uniqueNumbers);

        byte[] compressed = compress(listSkus);
        String compressedString = bytesToHex(compressed);
        System.out.println("Compressed size (bytes): " + compressed.length);
        System.out.println("Compressed string: " + compressedString);

        listSkus.subList(0, 10).forEach(System.out::println);

        List<Integer> decompressed = decompress(compressed);
        System.out.println("Decompressed count: " + decompressed.size());
        decompressed.subList(0, 10).forEach(System.out::println);
    }
   

Answer 1

Hashes are not reversible. They do allow collisions.

So using compression is the best idea. If it does not work, you have to accept that sometimes it's not possible.

What may be possible is to make the string generation shorter ; or split the string and transmit parts one at a time.