How can I smooth my JFrame shape

Question

I gave my JFrame window a custom shape with rounded corners but how can I smooth it(anti-aliasing)

Answer 1

A lot will come down to how you are rendering your content, but the basic concept is to supply rendering hints to the Graphics context you are drawing to...

For example, if I was painting to a component, I might use something like...

// Create a "copy" of the graphics context so we don't modify any of it's existing
// settings.  This makes it easier to manage the graphics context as we 
// may not want to effect anything else that might be using this graphics context
// into the future
Graphics2D g2d = (Graphics2D)g.create();
RenderingHints hints = new RenderingHints(
    RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON
);
g2d.setRenderingHints(hints);
//... continue drawing.
// Dispose of our "copy" of the graphics context
g2d.dispose();

Check out Controlling Rendering Quality for more details

Updated with example

public class AATest {

    public static void main(String[] args) {
        new AATest();
    }

    public AATest() {
        EventQueue.invokeLater(new Runnable() {
            @Override
            public void run() {
                try {
                    UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
                } catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
                }

                JFrame frame = new JFrame();
                frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
                frame.setLayout(new BorderLayout());
                frame.add(new PaintPane());
                frame.pack();
                frame.setLocationRelativeTo(null);
                frame.setVisible(true);
            }
        });
    }

    public class PaintPane extends JPanel {

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(215, 110);
        }

        @Override
        protected void paintComponent(Graphics g) {
            super.paintComponent(g);
            Graphics2D g2d = (Graphics2D) g.create();
            RenderingHints hints = new RenderingHints(
                    RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
            g2d.setRenderingHints(hints);
            g2d.setColor(Color.RED);
            drawShape(g2d, 5, 5);
            g2d.dispose();
            g2d = (Graphics2D) g.create();
            g2d.setColor(Color.BLUE);
            drawShape(g2d, 110, 5);
            g2d.dispose();
        }

        protected void drawShape(Graphics2D g2d, int x, int y) {
            g2d.draw(new Ellipse2D.Float(x, y, 100, 100));
        }
    }
}

Updated with new example

One of the tricks I use is, instead of using "setShape", I simply make a transparent window and use a custom panel to provide me with the shape I want to use.

The major issue with this, is you now become responsible for ensuring that the content is painted inside the shape...

public class ShapedWindow {

    public static void main(String[] args) {
        new ShapedWindow();
    }

    public ShapedWindow() {
        EventQueue.invokeLater(new Runnable() {
            @Override
            public void run() {
                try {
                    UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
                } catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
                }

                JWindow frame = new JWindow();
                frame.setBackground(new Color(0, 0, 0, 0));
                frame.setContentPane(new ShapedPane());
                frame.pack();
                frame.setLocationRelativeTo(null);
                frame.setVisible(true);
                frame.setAlwaysOnTop(true);
            }
        });
    }

    public class ShapedPane extends JPanel {

        public ShapedPane() {

            setOpaque(false);
            setLayout(new GridBagLayout());

            JButton button = new JButton("Close");
            button.addActionListener(new ActionListener() {
                @Override
                public void actionPerformed(ActionEvent e) {
                    System.exit(0);
                }
            });

            add(button);

        }

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(200, 200);
        }

        @Override
        protected void paintComponent(Graphics g) {
            super.paintComponent(g); //To change body of generated methods, choose Tools | Templates.
            Graphics2D g2d = (Graphics2D) g.create();
            RenderingHints hints = new RenderingHints(
                    RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
            g2d.setRenderingHints(hints);
            g2d.setColor(getBackground());
            g2d.fill(new Ellipse2D.Float(0, 0, getWidth(), getHeight()));
            g2d.dispose();
        }
    }

}