How to implement distributed transaction with hystrix fallback based on Spring Cloud architect

Question

I am using spring cloud to implement my micro services system, a ticket sale platform. The scenario is, there is a zuul proxy, a eureka registry, and 3 service: user service, order service and ticket service. Services use feign declarative REST Client to communicate with each other.

Now there is a function to buy tickets, the main process is as below:
1. order service accept request to create order
2. order service create Order entity with Pending status.
3. order service call user service to process user pay.
4. order service call ticket service to update user tickets.
5. order service update the order entity as FINISHED.

And I want to use Hystrix Fallback to implement transaction. For example, if the payment process is finished, but some error happened during ticket movement. How to revet user payment, and order status. Because user payment is in other service.

The following is my current solution, I am not sure whether it is proper. Or is there any other better way to do that.

At first, the OrderResource:

@RestController
@RequestMapping("/api/order")
public class OrderResource {

  @HystrixCommand(fallbackMethod = "createFallback")
  @PostMapping(value = "/")
  public Order create(@RequestBody Order order) {
    return orderService.create(order);
  }

  private Order createFallback(Order order) {
    return orderService.createFallback(order);
  }
}

Then the OrderService:

@Service
public class OrderService {

    @Transactional
    public Order create(Order order) {
        order.setStatus("PENDING");
        order = orderRepository.save(order);

        UserPayDTO payDTO = new UserPayDTO();
        userCompositeService.payForOrder(payDTO);

        order.setStatus("PAID");
        order = orderRepository.save(order);

        ticketCompositeService.moveTickets(ticketIds, currentUserId);

        order.setStatus("FINISHED");
        order = orderRepository.save(order);
        return order;
    }

    @Transactional
    public Order createFallback(Order order) {
        // order is the object processed in create(), there is Transaction in create(), so saving order will be rollback,
        // but the order instance still exist.
        if (order.getId() == null) { // order not saved even.
            return null;
        }
        UserPayDTO payDTO = new UserPayDTO();
        try {
            if (order.getStatus() == "FINISHED") { // order finished, must be paid and ticket moved
                userCompositeService.payForOrderFallback(payDTO);
                ticketCompositeService.moveTicketsFallback(getTicketIdList(order.getTicketIds()), currentUserId);
            } else if (order.getStatus() == "PAID") { // is paid, but not sure whether has error during ticket movement.
                userCompositeService.payForOrderFallback(payDTO);
                ticketCompositeService.moveTicketsFallback(getTicketIdList(order.getTicketIds()), currentUserId);
            } else if (order.getStatus() == "PENDING") { // maybe have error during payment.
                userCompositeService.payForOrderFallback(payDTO);
            }
        } catch (Exception e) {
            LOG.error(e.getMessage(), e);
        }

        order.setStatus("FAILED");
        orderRepository.save(order); // order saving is rollbacked during create(), I save it here to trace the failed orders.
        return order;
    }
}

Some key points here are:

1. Using @HystrixCommand in OrderResource.create(order) method, with fallback function.
2. If there is some error in creation, the order instance used in OrderResource.create(order) will be used again in fallback function. Although the persistence of this order will be roll-backed. But the data in this instance still can be used to check the running.
3. So I use a status: 'PENDING', 'PAID', 'FINISHED' to check whether some service call is made.
4. ticketCompositeService and userCompositeService is a feign client. For feign client method payForOrder(), there is another method payForOrderFallback() for fallback.
5. I need to make sure the fallback methods can be called multiple times.
6. I add try/catch for ticketCompositeService and userCompositeService call, to make sure the order will be save anyway with 'FAILED' status.

It seems that this solution can work at the most of the time. Except that, in fallback function, if there is some error in userCompositeService.payForOrderFallback(payDTO);, then the following composite service call will not be called.

And, another problem is, I think it is too complicated.

So, for this scenario, how should I implement dist transaction properly and effectively. Any suggestion or advice will help. Thanks.

Answer 1

Writing compensation logic within Hystrix fallback is dangerous because of no persistence involved.

This approach doesn't offer any resiliency. ACID guarantee from the database is not enough here because of external parties involved, and the Hystrix fallback will not guard you from anything that's not part of your code.

For example, if your solution experiences outage (say, power outage or a simple kill -9) after payment completion, you will lose both the order and the compensation logic, meaning order will be paid for, but not present in the database.

A more resilient approach would involve any popular message broker for event-driven delivery and some deduplication in processing logic to ensure exactly-once quality of service when the events get redelivered after an outage.