Testable Annotation Rubrics: The Missing Layer Between Guidelines and Good Annotations

No dark-mode styles — this email renders the same in a dark client.

A graph-based method for evaluation design with Nikolai Liubimov, CTO of HumanSignal

humansignal-logo-full-dark (https://email.heartex.com/e3t/Ctc/I6+113/cVVKt04/VVzq8K4w458tW4DR-tR31N3rPW6mk3Cf5VhLFBN8DGmD43l5QzW6N1vHY6lZ3mlW6MlsQt6NmzwzW3p7Xc33N7nWkVDyxhG596ZPbW8Y78r16k1XtxV60lgf7W5yQyV8Wr6c9gtPsCW37jMcX1t5N-QW3db6l21h2XL0W4KBvRD1tBbBQW218tyZ56wg8ZW74rSQc4mFdBmW7zX6lF39rkCVW7C34Lc6Tm4k6VPjVZP811xGsV8WFsQ42kSBSW8WSxCk7tTnpPW2432f52k1sdtW14gcx73FPsGTN3_1Gh-X0dc0W2XcJ942rwSCtW2ML5GM1dPkqgW2vYnXw5Z6yjXf4wRV_K04 )

Turn evaluation guidelines into testable reasoning

When annotators disagree on evaluation projects, the problem is often the reasoning process we designed for them, not the people doing the annotation.

In a new technical post (https://email.heartex.com/e3t/Ctc/I6+113/cVVKt04/VVzq8K4w458tW4DR-tR31N3rPW6mk3Cf5VhLFBN8DGmCb5jGjMW50kH_H6lZ3kGW1SZKbB3Fkwt1W5FM0pC1x6x2tW4_zYWc2sNqddW98lsm45FDdQWW2yPdg44dQ4RFW7bDZ8C6LxZmfW39qp5Y13Z_wlV6697W1yRQ_mW5y6z1B3MJnkDW6TgQCc7DP3jVW2ZNyn25dvLvsW91LrXw6Jd7jvW4dzxpl1fbC0_W1fggzV1_ZYT7W66p6fq9cDqH9W8Zn5_N1t67_JVQ0VM81YxcR1W1W2-VS68XgQQW2ldWd626wQj0W8bK_Vp7_7j51W7P_sls2wYDCDW5nWqm63nqv9jW9gqnp-70L-XbVJq_2S8_LnQfW4Lr0v-5nFfZtV5ZRVx5xwXyWW153vbG8BQfnqW4NQ2V97pVL1bW6PjWVv4Y7zrcW781dhb1gr0tLW66p7398C5MLtW4LljQz1cV9Klf1xqbZK04 ) , HumanSignal CTO Nikolai Liubimov describes a rubric-design method that inserts an explicit reasoning model between project requirements and the annotation interface.

The approach:

- Represents evaluation logic as a directed graph
- Separates observable evidence from subjective judgments
- Makes rubric changes versionable, inspectable, and auditable
- Uses synthetic annotators to test before investing in a human pilot

Read the rubric-design post
(https://email.heartex.com/e3t/Ctc/I6+113/cVVKt04/VVzq8K4w458tW4DR-tR31N3rPW6mk3Cf5VhLFBN8DGmCb5jGjMW50kH_H6lZ3kGW1SZKbB3Fkwt1W5FM0pC1x6x2tW4_zYWc2sNqddW98lsm45FDdQWW2yPdg44dQ4RFW7bDZ8C6LxZmfW39qp5Y13Z_wlV6697W1yRQ_mW5y6z1B3MJnkDW6TgQCc7DP3jVW2ZNyn25dvLvsW91LrXw6Jd7jvW4dzxpl1fbC0_W1fggzV1_ZYT7W66p6fq9cDqH9W8Zn5_N1t67_JVQ0VM81YxcR1W1W2-VS68XgQQW2ldWd626wQj0W8bK_Vp7_7j51W7P_sls2wYDCDW5nWqm63nqv9jW9gqnp-70L-XbVJq_2S8_LnQfW4Lr0v-5nFfZtV5ZRVx5xwXyWW153vbG8BQfnqW4NQ2V97pVL1bW6PjWVv4Y7zrcW781dhb1gr0tLW66p7398C5MLtW4LljQz1cV9Klf1xqbZK04 )

If similar disagreement shows up in your own evaluation pipeline, we can help. Learn more about HumanSignal Data Services. (https://email.heartex.com/e3t/Ctc/I6+113/cVVKt04/VVzq8K4w458tW4DR-tR31N3rPW6mk3Cf5VhLFBN8DGmDn3l5QzW7lCdLW6lZ3mCN3VDkTtJ9yx8W5tWbYN2ZXmNrW6WyKlr78fD09W7W6zqN2DZyhpW7S9CW07M1PCNW22GSlB78KbjLW9cLjXl7s-tn0W8shTMW8tr26XV4xl381CWy5hW4_2C-p2yhY9hW3dCjKL7mxSm8W8QpRGy5S2XxCW4HR8T863jnRxW38KR6h1l2FkFW76Lh513MMtSzW4wpfl04sLhcmVJLzV668WpgrW6snDqF8VKF6nW8JSdP53WPYH7W57Jf4P3YDtnmW5J77lt7k954wW3Z0QSg7wBvc8W40S9-P7VDB6NMl94Dfc844cf2pmhbW04 )

HumanSignal, 548 Market St. PMB 34372, San Francisco, CA 94104

Unsubscribe (https://email.heartex.com/hs/preferences-center/en/direct?data=W2nXS-N30h-z5W1Bp_nG23nCDSW3QC_lJ1V1JYdW32xFDl3dbCVXW3VKWxF4mdvnrW3dgllZ3NN41FW4tDyCQ2FscNzW3jwjsZ3NWf-SW2-v_mF384qffW2Cvc863bvDQzW25llvH2KStybW3b1JZP36nNwQW4hd2lS2B2mPfW1S56Gl2RMLNPW1X0FM836gYx8W4kl0VR1L9Cb8W3j6rv84cgwLzW3GZm4n3P6jp-W2RRx8g3P7BwFW4mHB-Y3F9_k5W2KD21k45VGjlW36nRqr30n5RxW3XF0-F3JX13HW2PD3HJ3F0TgPW2HRsx03dwDh0W4tjPsL4fP_RlW2Kz69r3XFxw-W1L9ynG4pCGCsW1Sb89R36s_VwW3dghzv2nXts2W4rGNgD38lvTNW2Cvb5w4pmm25W1SBmYX2PmNb3W3Z_n1n41WvJJW2vZkSx2xKHKZW2zyMbZ2zKSpDW3DLtNC4pHrsxW2y3lR82t3LdwW4cjFWX30r0DPW3SRYqh34GyR9W3F5nqv2sVmmSW1Qg41T3bsyb1W2q_0_225hrl1W32svY-233XDpW2-BGks1Y_DmtW2RhRyp4t6rHqW1SqhbX1X5_HsW1QbPxz2WQF-RW2Pz76_3QKQ--W1Sl-xX3NBXk5W45qcWp1Lp_PmW2CYgtq3ZLF4tW3byy2D3QCBcgW328gfs3dz6_QW23n_J24kbDVGW1Bqrms4pxW-cW1VfM672MXxc-W1Btkrc41GYPkW2YtBVx3SyxNGW38lzhs2-p73fW2r87v93d7bZ-W3QztCR1XlTLzW2FKqy543Vl_Gf3Kb2Q904&_hsenc=p2ANqtz-88qApvjSpXiGH54N4J-YSPSWNJbmBIK5Mp4qwvmn34FkR21ZKAC5S2sMqZAZW5IAe_UB6PoSxbd3D54MihvRVzQYIn0bhbnukTGwNQJQg2AHkZfjOI5Vl0YupezoVB0021vL5P&_hsmi=440539138 )
Manage preferences (https://email.heartex.com/hs/preferences-center/en/page?data=W2nXS-N30h-z5W1Bp_nG23nCDSW3QC_lJ1V1JYdW32xFDl3dbCVXW3VKWxF4mdvnrW3dgllZ3NN41FW4tDyCQ2FscNzW3jwjsZ3NWf-SW2-v_mF384qffW2Cvc863bvDQzW25llvH2KStybW3b1JZP36nNwQW4hd2lS2B2mPfW1S56Gl2RMLNPW1X0FM836gYx8W4kl0VR1L9Cb8W3j6rv84cgwLzW3GZm4n3P6jp-W2RRx8g3P7BwFW4mHB-Y3F9_k5W2KD21k45VGjlW36nRqr30n5RxW3XF0-F3JX13HW2PD3HJ3F0TgPW2HRsx03dwDh0W4tjPsL4fP_RlW2Kz69r3XFxw-W1L9ynG4pCGCsW1Sb89R36s_VwW3dghzv2nXts2W4rGNgD38lvTNW2Cvb5w4pmm25W1SBmYX2PmNb3W3Z_n1n41WvJJW2vZkSx2xKHKZW2zyMbZ2zKSpDW3DLtNC4pHrsxW2y3lR82t3LdwW4cjFWX30r0DPW3SRYqh34GyR9W3F5nqv2sVmmSW1Qg41T3bsyb1W2q_0_225hrl1W32svY-233XDpW2-BGks1Y_DmtW2RhRyp4t6rHqW1SqhbX1X5_HsW1QbPxz2WQF-RW2Pz76_3QKQ--W1Sl-xX3NBXk5W45qcWp1Lp_PmW2CYgtq3ZLF4tW3byy2D3QCBcgW328gfs3dz6_QW23n_J24kbDVGW1Bqrms4pxW-cW1VfM672MXxc-W1Btkrc41GYPkW2YtBVx3SyxNGW38lzhs2-p73fW2r87v93d7bZ-W3QztCR1XlTLzW2FKqy543Vl_Gf3Kb2Q904&_hsenc=p2ANqtz-88qApvjSpXiGH54N4J-YSPSWNJbmBIK5Mp4qwvmn34FkR21ZKAC5S2sMqZAZW5IAe_UB6PoSxbd3D54MihvRVzQYIn0bhbnukTGwNQJQg2AHkZfjOI5Vl0YupezoVB0021vL5P&_hsmi=440539138 )

Accession

nse/heartex/2026-09-23/2299

From

HumanSignal · noreply@humansignal.com

Sent

23 September 2026 · 17:00 UTC