How to choose thread/process priority in real-time Linux?

Question

I am developing real-time software that will run on Linux with the PREEMPT_RT patch. While this will let me get realtime performance by using the Linux realtime schedulers (SCHED_RR or SCHED_FIFO), I am unsure what priority values I should use for my threads. On my system, when I look at the priorities I see (only showing processes with RTPRIO):

$ ps -e -o cmd,pri,rtprio
CMD                         PRI RTPRIO
[sirq-high/0]                89     49
[sirq-timer/0]               89     49
[sirq-net-tx/0]              89     49
[sirq-net-rx/0]              89     49
[sirq-block/0]               89     49
[sirq-block-iopo]            89     49
[sirq-tasklet/0]             89     49
[sirq-sched/0]               89     49
[sirq-hrtimer/0]             89     49
[sirq-rcu/0]                 89     49
[posixcputmr/0]             139     99
[events/0]                   41      1
[irq/17-systemac]            90     50
[irq/19-watchdog]            90     50
[irq/18-eth0]                90     50
[irq/16-serial]              90     50

I have two questions:

1. What are all these sirq, irq, posixcputmr tasks? Kernel threads?
2. At what priority should I make my software run? Should I be running at a higher priority than most of these system threads (>50 or even 99), or lower (<49)?

Answer 1

What are all these sirq, irq, posixcputmr tasks? Kernel threads?

Yes, all the tasks in brackets are kernel threads.

At what priority should I make my software run?

From 2-69 for normal RT, and 90-98 for your very high priority application threads. The latter will block all IRQ handlers, so try to do as little as possible at high priority. Source