DEF CON CTF Qualifier 2026 ๐ฉ

myfavoriteinstructions
We started by analyzing the binary, which was a stripped 64-bit ELF. The binary was composed of two main functions: main and a function that IDA could not decompile due to its length. The latter was composed mainly of bsr, bzhi, and mov instructions and contained very few control flow branches.
Our first hypothesis was that the code had been generated by an algorithm to make the main function difficult to analyze with classic decompilers (we also observed that the .text section was about 250KB).
Initially we decided to analyze the main function statically, we easily understood that the binary had to be run with an argument (the flag) which needed to be exactly 68 bytes long:
if ( a1 != 1 )
{
if ( a1 == 2 )
{
v4 = a2[1];
len = strlen(v4);
if ( len <= 67 )
{
fprintf(stderr, "Flag too short (got %d, need >= 68)\n", len);
return 1;
}
...
Once we understood this simple detail, we decided to move on to the rest of main before analyzing the real core of the challenge.
The following is in fact the analysis of the 6 different loops in the main function.
Understanding main
First Loop
The code of the first loop was the following (decompiled using IDA Pro 9.3):
v6 = 4;
v7 = v4;
do
{
v8 = *v7;
v9 = *v7 / 3u;
v27[v6] = *v7 % 3u;
v27[v6 + 1] = (v9 - 3 * ((86 * v9) >> 8));
v27[v6 + 2] = (((57 * v8) >> 9) - 3 * ((86 * ((57 * v8) >> 9)) >> 8));
v27[v6 + 3] = (((19 * v8) >> 9) - 3 * ((86 * ((19 * v8) >> 9)) >> 8));
v10 = (203 * v8) >> 14;
v11 = v10 - 3;
if ( v8 < 0xF3u )
v11 = v10;
v27[v6 + 4] = v11;
v6 += 5;
++v7;
}
while ( v6 != 24 );
(I decided to hide the casts to make the snippet as concise as possible)
This code, despite looking absolutely horrendous, was nothing more than the result of a series of arithmetic optimizations applied by the compiler to calculate the remainder of division by 3.
This conclusion was already evident from the first three lines implementing this idea.
Furthermore running a few test cases showed that all the hardcoded values led exactly to the mentioned result: 86 * x >> 8 = x * 86/256 = x * 0.3359375 which is quite close to the result of dividing by 3 (which is 0.3 periodic).
Applying this reasoning to all the divisions the most significant simplification was:
digit0 = (byte / 3**0) % 3
digit1 = (byte / 3**1) % 3
...
digit4 = (byte / 3**4) % 3
We understood that the first 4 bytes (bbb{) were unpacked into tuples of 5 ternary digits (trits) and saved in an array on the stack.
Second, Third, and Sixth Loops ๐ฅ
In these loops, the code, which was initially still relatively readable, became extremely convoluted…
v12 = *(v4 + 4);
for ( i = 21; i != 103; i += 2 )
{
v27[i + 3] = v12
- (3 * ((__CFADD__(*(&v12 + 1), v12) + *(&v12 + 1) + v12) / 3)
- (__CFADD__(*(&v12 + 1), v12)
+ *(&v12 + 1)));
v27[i + 4] = (__CFADD__(
0xAAAAAAAAAAAAAAABLL
* (3 * ((__CFADD__(*(&v12 + 1), v12) + *(&v12 + 1) + v12) / 3)
- (__CFADD__(*(&v12 + 1), v12)
+ *(&v12 + 1))),
(__PAIR128__(
(v12 - ((__CFADD__(*(&v12 + 1), v12) + *(&v12 + 1) + v12) % 3)) >> 64,
3 * ((__CFADD__(*(&v12 + 1), v12) + *(&v12 + 1) + v12) / 3)
- (__CFADD__(*(&v12 + 1), v12)
+ *(&v12 + 1)))
* __PAIR128__(0xAAAAAAAAAAAAAAAALL, 0xAAAAAAAAAAAAAAABLL)) >> 64)
+ ((__PAIR128__(
3 * ((__CFADD__(*(&v12 + 1), v12) + *(&v12 + 1) + v12) / 3)
- (__CFADD__(*(&v12 + 1), v12)
+ *(&v12 + 1)),
3 * ((__CFADD__(*(&v12 + 1), v12) + *(&v12 + 1) + v12) / 3)
- (__CFADD__(*(&v12 + 1), v12)
+ *(&v12 + 1)))
* 0xAAAAAAAAAAAAAAABLL) >> 64))
% 3;
*&v12 = __udivti3(v12, *(&v12 + 1), 9, 0);
*(&v12 + 1) = v14;
}
To avoid having to fully understand that wall of code we decided to debug it and make some very simple assumptions:
- The code probably did exactly what the first loop did.
- The loop iterated with
i += 2, so it likely generated 2 digits or combined them in some way. - The loop executed exactly
(103 - 21) / 2 = 41iterations. - The pattern was very similar to the one already observed with
x - (3 * something) >> y.
However after a couple of runs with gdb, we noticed that the ternary digits did not follow the patterns of the first loop, so the situation still needed clarification:
pwndbg> stack 50
v--- b
00:0000โ rsp 0x7fffffffd6f0 โโ 2
01:0008โ-ba8 0x7fffffffd6f8 โโ 2
02:0010โ-ba0 0x7fffffffd700 โโ 1
03:0018โ-b98 0x7fffffffd708 โโ 0
04:0020โ-b90 0x7fffffffd710 โโ 1
v--- b
05:0028โ-b88 0x7fffffffd718 โโ 2
06:0030โ-b80 0x7fffffffd720 โโ 2
07:0038โ-b78 0x7fffffffd728 โโ 1
08:0040โ-b70 0x7fffffffd730 โโ 0
09:0048โ-b68 0x7fffffffd738 โโ 1
v--- b
0a:0050โ-b60 0x7fffffffd740 โโ 2
0b:0058โ-b58 0x7fffffffd748 โโ 2
0c:0060โ-b50 0x7fffffffd750 โโ 1
0d:0068โ-b48 0x7fffffffd758 โโ 0
0e:0070โ-b40 0x7fffffffd760 โโ 1
v--- {
0f:0078โ-b38 0x7fffffffd768 โโ 0
10:0080โ-b30 0x7fffffffd770 โโ 2
11:0088โ-b28 0x7fffffffd778 โโ 1
... โ 2 skipped
v--- second loop (all b's)
14:00a0โ-b10 0x7fffffffd790 โโ 2
15:00a8โ-b08 0x7fffffffd798 โโ 0
16:00b0โ-b00 0x7fffffffd7a0 โโ 0
17:00b8โ-af8 0x7fffffffd7a8 โโ 1
18:00c0โ-af0 0x7fffffffd7b0 โโ 1
19:00c8โ-ae8 0x7fffffffd7b8 โโ 2
1a:00d0โ-ae0 0x7fffffffd7c0 โโ 1
1b:00d8โ-ad8 0x7fffffffd7c8 โโ 2
1c:00e0โ-ad0 0x7fffffffd7d0 โโ 2
1d:00e8โ-ac8 0x7fffffffd7d8 โโ 0
1e:00f0โ-ac0 0x7fffffffd7e0 โโ 2
1f:00f8โ-ab8 0x7fffffffd7e8 โโ 0
20:0100โ-ab0 0x7fffffffd7f0 โโ 1
21:0108โ-aa8 0x7fffffffd7f8 โโ 0
22:0110โ-aa0 0x7fffffffd800 โโ 0
23:0118โ-a98 0x7fffffffd808 โโ 1
^--- no clear pattern
That failed attempt cost us some time, but after re-reading the code a couple of times, we noticed a simple detail we had missed: the string was loaded as 128-bit (16 bytes) without ever being modified
This could mean that, unlike the first loop (where there was a 1:1 correspondence between bytes and 5 base-3 digits), we were dealing with a block of (103 - 21) / 2 * 2 = 82 ternary digits (a calculation confirmed by the fact that 3**5 is approximately 256, while 3**82 is slightly larger than 2**128).
In other words 16 bytes of the flag were saved as a single block.
The pseudocode concept we formulated was roughly the following:
var_128bit = pointer[X]
for with some ranges, step 2
savevar[0] = var_128bit % 3
savevar[1] = (var_128bit // 3) % 3
// skip
var_128bit /= 9
Fourth and Fifth Loops
Having concluded the analysis of the previous loops we decided to finish examining the main function by looking more closely at these two apparently very simple loops:
v18 = *(v4 + 36);
for ( k = 185; ; k += 2 )
{
v27[k + 3] = v18 % 3;
if ( k == 225 )
break;
v27[k + 4] = v18 / 3 - 3 * ((0x5555555555555556LL * (v18 / 3)) >> 64);
v18 /= 9u;
}
The code, much like in the previous loops, iterated with a step of two over the bytes of the flag, calculating the remainders of division by 3.
We noticed that this code was very similar to the pseudocode shown earlier, with the difference that in this case, the program loaded a 64-bit integer (QWORD) instead of a 128-bit integer (OWORD).
The value 0x5555555555555556LL was not new and, in the context of our challenge, represented the floating-point representation of the modular inverse of 3.
Here too, the program loaded the flag portion as a single integer and extracted its ternary digits in pairs.
Loop Summary
This table explains the situation generated in main:
| Loop | Byte Range | Data Type Read | Generated Ternary Digits | Indices in v27 Array |
|---|---|---|---|---|
| First | 0 - 3 | 4 individual bytes (char) | 20 | [4] - [23] |
| Second | 4 - 19 | 16-byte block (_OWORD) | 82 | [24] - [105] |
| Third | 20 - 35 | 16-byte block (_OWORD) | 82 | [106] - [187] |
| Fourth | 36 - 43 | 8-byte block (_QWORD) | 41 | [188] - [228] |
| Fifth | 44 - 51 | 8-byte block (_QWORD) | 41 | [229] - [269] |
| Sixth | 52 - 67 | 16-byte block (_OWORD) | 82 | [270] - [351] |
Analysis of sub_11A0
This function, as anticipated in the introduction, was extremely large, which is why a standard decompiler (like IDA ๐ฅ) could not decompile it completely ๐ฅ.
First Stage
However, we could deduce that it processed two blocks of 82 ternary digits (for a total of 164 ternary digits) to produce a final validation value. To analyze this circuit in a black-box manner, we built a dynamic oracle using libdebug ๐, implementing the solver script shown below.
In the script, the f10_stack_vectors function starts the debugger with ASLR disabled, retrieves the executable’s load address via d.maps, and places an initial breakpoint at pie + 0x11A0. At this point, the r12 register contains the memory address where the input ternary digits to the function are stored. By overwriting the memory at that address, we can inject arbitrary vectors, bypassing the ASCII character constraints of main. Subsequently, a second breakpoint at pie + 0x5400 pauses execution right before the multiplication. By reading from the process stack (rsp + 0x420 and rsp + 0x1F80), we can extract the two factors actually being processed.
The f10_affine_offsets function leverages this technique to determine the linear affine relationship (a modular translation) between the controlled input and the factors computed from the stack. By sending a null input, we obtain the constant offset vectors $\alpha$ and $\beta$ such that:
$$B_i = (y_i + \beta_i) \pmod 3$$
Where $x_i$ and $y_i$ are the ternary digits of the flag (from the second and third blocks), while $\alpha_i$ and $\beta_i$ are the affine offsets determined by running the program with a null input and reading the base values on the stack.
At the end of this processing, the program compared the result with a constant of 168 ternary digits stored starting at address 0x4E050 (TARGET_F10). By interpreting this array as the base-3 representation of an arbitrary-precision integer, we obtained the target value:
Since this is a multiplication between two numbers represented by 82 ternary digits, the target $T$ had to be the direct product of the two factors in base 3. By factoring $T$, we discovered that it is composed of exactly two prime factors:
- $f_1 = 221815467394800111963839297593696124903$
- $f_2 = 312491767943139940981443826148003062019$
Having the two factors and the affine offset vectors $\alpha$ and $\beta$ extracted via debugging, we inverted the relationship to reconstruct the original ternary digits of the flag:
$$x_i = (f_1[i] - \alpha_i) \pmod 3$$$$y_i = (f_2[i] - \beta_i) \pmod 3$$
Converting these ternary digits to bytes, we obtained the two 16-byte blocks of the flag: kQMM2FhlSBO4fEYF and ho5azaRrlTdxPsRx. We then decided to verify the correctness of the blocks at runtime using the f10_ok function from the script, which runs the program up to the check point at pie + 0x17CB6 and reads from the stack at rsp + 0x1500 to compare the final value with TARGET_F10.
The complete and exact Python script used to extract the affine offsets and calculate the correct ternary digits for the first stage is the following; the full source is also available in the collapse here:
import struct
from pathlib import Path
from libdebug import debugger
F10_FACTORS = (
221815467394800111963839297593696124903,
312491767943139940981443826148003062019,
)
BIN = Path("myfavoriteinstructions")
BIN_DATA = BIN.read_bytes()
TARGET_F10 = struct.unpack("<168Q", BIN_DATA[0x4E050 : 0x4E050 + 168 * 8])
def flag_to_trits(s):
assert len(s) >= 68
out = []
for b in s[:4]:
x = b
for _ in range(5):
out.append(x % 3)
x //= 3
for off, ntrits in [(4, 82), (20, 82), (36, 41), (44, 41), (52, 82)]:
size = 16 if ntrits == 82 else 8
x = int.from_bytes(s[off : off + size], "little")
for _ in range(ntrits):
out.append(x % 3)
x //= 3
assert len(out) == 348
return out
def trits_to_bytes_prefix(trits):
out = bytearray()
pos = 0
for _ in range(4):
x = 0
p = 1
for t in trits[pos : pos + 5]:
x += t * p
p *= 3
out.append(x & 0xff)
pos += 5
for ntrits, size in [(82, 16), (82, 16), (41, 8), (41, 8), (82, 16)]:
x = 0
p = 1
for t in trits[pos : pos + ntrits]:
x += t * p
p *= 3
out += x.to_bytes(size, "little")
pos += ntrits
return bytes(out)
def get_pie_base(d):
for m in d.maps:
if 'myfavoriteinstructions' in (m.backing_file or ''):
return m.start
return 0x555555554000
def trits_of(n, length):
out = []
for _ in range(length):
out.append(n % 3)
n //= 3
if n:
return None
return out
def f10_stack_vectors(trits):
d = debugger(["./myfavoriteinstructions", "A" * 68], aslr=False)
d.run()
pie = get_pie_base(d)
bp_start = d.breakpoint(pie + 0x11A0)
d.cont()
d.wait()
trits_data = struct.pack("<348Q", *trits)
d.memory[d.regs.r12, len(trits_data)] = trits_data
bp_start.disable()
bp = d.breakpoint(pie + 0x5400)
d.cont()
d.wait()
rsp = d.regs.rsp
a_data = d.memory[rsp + 0x420, 168 * 8]
b_data = d.memory[rsp + 0x1F80, 168 * 8]
a = list(struct.unpack("<168Q", a_data))
b = list(struct.unpack("<168Q", b_data))
d.kill()
return a, b
def f10_affine_offsets():
base = flag_to_trits(b"A" * 68)
alpha = []
beta = []
for i in range(82):
avals = []
bvals = []
for x in range(3):
tr = base[:]
tr[20 + i] = x
tr[102 + i] = x
a, b = f10_stack_vectors(tr)
avals.append(a[i])
bvals.append(b[i])
alpha.append(avals[0])
beta.append(bvals[0])
return alpha, beta
def f10_ok(raw):
d = debugger(["./myfavoriteinstructions", raw.decode("latin-1")], aslr=False)
d.run()
pie = get_pie_base(d)
bp = d.breakpoint(pie + 0x17CB6)
d.cont()
d.wait()
rsp = d.regs.rsp
vec = struct.unpack("<168Q", d.memory[rsp + 0x1500, 168 * 8])
d.kill()
return vec == TARGET_F10
def solve_prefix_and_f10():
print("[f10] decoding multiplication stage")
alpha, beta = f10_affine_offsets()
base = flag_to_trits(b"A" * 68)
for left, right in (F10_FACTORS, F10_FACTORS[::-1]):
ad = trits_of(left, 82)
bd = trits_of(right, 82)
if ad is None or bd is None:
continue
trits = base[:]
trits[20:102] = [(ad[i] - alpha[i]) % 3 for i in range(82)]
trits[102:184] = [(bd[i] - beta[i]) % 3 for i in range(82)]
try:
raw = trits_to_bytes_prefix(trits)
except OverflowError:
continue
if all(32 <= c < 127 for c in raw[:36]) and f10_ok(raw):
print(f"[f10] {raw[:36].decode()}")
return raw[:36]
raise SystemExit("f10 decode failed")
Second Stage
In the second part of the challenge (bytes 36-43), there was a huge table of 15x41 elements and a vector of vectors of 15 elements that we will call the “target vector”. Each element in the table and the vector was composed of arrays of 20 ternary digits, and the main loop worked roughly like this:
combine 41 elements of the table using 41 input ternary digits
compare the result with the target[j] vector
We then hypothesized that since the operations had to compress 41 elements into a single one, we were looking at something similar to…
target_j = sum_i x_i * table[j][i]
Note that the addition and multiplication were not “standard”, one was essentially the repetition of the other multiple times.
After determining that the elements entering the circuit were arrays of 20 ternary digits, we hoped that the operation was associative and commutative, so that we could treat the elements as a cyclic group of order $3^{20}$. To test this idea we asked Codex to write some functions to interface these operations with libdebug (the function names are hypothetical):
class Oracle:
def __init__(self, prefix):
# Initialize the debugger with the partial flag found so far
raw = prefix.ljust(68, b"A")
self.d = debugger(["./myfavoriteinstructions", raw.decode("latin-1")], aslr=False)
self.d.run()
self.pie = get_pie_base(self.d)
# Breakpoint at the beginning of the group operation (stage 3)
bp_start = self.d.breakpoint(self.pie + 0x1C0A1)
self.d.cont()
self.d.wait()
# Save a snapshot of the state to quickly restore it on each call
self.rsp = self.d.regs.rsp
self.snap_path = "/tmp/group_snap_libdebug.json"
snap = self.d.create_snapshot()
snap.save(self.snap_path)
# Breakpoint to intercept the inner loop and the end of the calculation
self.bp_patch = self.d.breakpoint(self.pie + 0x1CFB9)
self.bp_stop = self.d.breakpoint(self.pie + 0x1CF90)
self.bp_patch.disable()
self.bp_stop.disable()
def _eval(self, custom, vals, stop):
# Restore the initial state via snapshot
self.d.load_snapshot(self.snap_path)
# Write the desired coefficients at the stack offset
val_data = struct.pack("<41Q", *vals)
self.d.memory[self.rsp + 0x420, len(val_data)] = val_data
self.bp_patch.enable()
self.bp_stop.enable()
out = None
while True:
self.d.cont()
self.d.wait()
rip = self.d.regs.rip
# Patch the intermediate table values by providing the custom operands
if rip == self.pie + 0x1CFB9:
i = self.d.regs.r14
if i in custom:
row = custom[i] + (0,)
val_bytes = struct.pack("<21Q", *row)
self.d.memory[self.rsp + 0x9C0, len(val_bytes)] = val_bytes
# Read the final result at the desired loop offset
elif rip == self.pie + 0x1CF90:
if self.d.regs.r14 == stop:
out = tuple(struct.unpack("<Q", self.d.memory[self.rsp + off, 8])[0] for off in ORDER20_OFFSETS)
break
else:
raise RuntimeError(f"Unexpected RIP {hex(rip)}")
self.bp_patch.disable()
self.bp_stop.disable()
return out
def add(self, a, b):
vals = [0] * 41
vals[0] = 1
vals[1] = 1
return self._eval({0: a, 1: b}, vals, 2)
def double(self, a):
vals = [0] * 41
vals[0] = 2
return self._eval({0: a}, vals, 1)
From there we tried different combinations and verified that the operations were indeed associative and that the generator element was located at table[0][0], meaning we could treat the elements as a cyclic group.
Since we did not want to deal with the circuit details or the operations on ternary digits and because we could use the oracle to produce elements, we decided to apply the Pohlig-Hellman algorithm to obtain the coefficients of the elements (in additive notation) for each element in the table and for the target vector. This was particularly simple, since the only states of the ternary digits were {0, 1, 2}. Therefore we asked Codex to implement this exact procedure (we only show the log function because the full file was too large):
def log_base(
oracle: Oracle,
gamma: tuple[int, ...],
gamma2: tuple[int, ...],
gen_pows: list[tuple[int, ...]],
point: tuple[int, ...],
) -> int:
x = 0
residual = point
# Pohlig-Hellman for a cyclic group of order 3^20, additive notation.
for k in range(20):
probe = scalar_mul(oracle, residual, 3 ** (19 - k))
if probe == ZERO:
digit = 0
elif probe == gamma:
digit = 1
elif probe == gamma2:
digit = 2
else:
raise ValueError(("unexpected PH digit", k, probe))
x += digit * (3**k)
if digit:
residual = point_sub(oracle, residual, gen_pows[k])
if digit == 2:
residual = point_sub(oracle, residual, gen_pows[k])
return x
After this, we obtained normal numbers in $\mathbb{Z}_{3^{20}}$, meaning that the strange operation reduced to:
sum_i x_i * log(table[j][i]) == log(target[j]) mod 3^20
where sum_i was a real sum and * was the real multiplication between the two logs. Since the table and the vectors were now simple numbers we referred to the “logged” table as A and the target vector as B.
At that point the idea was that this was a linear combination of the vectors A[j,1], A[j,2], … A[j,i] weighted by x_1, x_2, .... To find which linear combination produced the target vector, we decided to use LLL, and that is what we implemented:
def solve_stage3_lll(prefix: bytes) -> bytes:
print("[stage3] solving cyclic-group equations with LLL")
coeffs, rhs = load_or_compute_stage3_logs()
n, m = 41, 15
centered_rhs = [
(rhs[j] - sum(coeffs[j][i] for i in range(n))) % ORDER3_20
for j in range(m)
]
scale = 1000
marker = 10
mat = IntegerMatrix(n + m + 1, n + m + 1)
for i in range(n):
mat[i, i] = 1
for j in range(m):
mat[i, n + j] = scale * coeffs[j][i]
for j in range(m):
mat[n + j, n + j] = scale * ORDER3_20
mat[n + m, n + j] = -scale * centered_rhs[j]
mat[n + m, n + m] = marker
LLL.reduction(mat, delta=0.99)
mapped = None
for row in range(mat.nrows):
vec = [int(mat[row, col]) for col in range(mat.ncols)]
if abs(vec[-1]) != marker:
continue
sign = 1 if vec[-1] == marker else -1
centered = [sign * vec[i] for i in range(n)]
if all(v in (-1, 0, 1) for v in centered):
candidate = [v + 1 for v in centered]
ok = all(
(sum(coeffs[j][i] * candidate[i] for i in range(n)) - rhs[j]) % ORDER3_20 == 0
for j in range(m)
)
if ok:
mapped = candidate
break
if mapped is None:
raise SystemExit("stage3 LLL failed")
With this we recovered the log of the flag elements entering the circuit and therfore we could reconstruct the elements by inverting the logarithmic mapping:
_maps, invmaps = s3.affine_maps()
trits = flag_to_trits(prefix + b"A" * (68 - len(prefix)))
for i, value in enumerate(mapped):
trits[184 + i] = invmaps[i][value]
raw = trits_to_bytes_prefix(trits)
print(f"[stage3] {raw[:44].decode()}")
return raw[:44]
In the end we had the second part of the flag: 6ue7npnj, and the known flag prefix extended to bbb{kQMM2FhlSBO4fEYFho5azaRrlTdxPsRx6ue7npnj.
Last Two Stages
In these final stages we were missing just a few bytes to complete the flag. Although we knew the process used to generate the ternary digits, we realized that:
- The code did not have patterns that we could easily recognize.
- The code we needed to reverse was towards the end of the function.
Continuing to work with prefixes we decided to shift our focus to a different type of solution: symbolic execution using a scripts written with the help of an LLM.
We wanted to create a script very similar to angr with the ability to take memory snapshots and add constraints on the symbolic inputs (the flag).
Although there were no significant differences in the solution of the two stages we noted that the solution naturally had to proceed sequentially (first solving the second-to-last chunk, and then solving the final one consisting of a trailing }).
To do this we started from the idea of symbolically emulating the executable’s instructions, translating them directly into SAT (CNF) constraints. Instead of using angr (which would have struggled to solve the constraints due to custom instructions like bsr and bzhi used for ternary gates) with the help of an LLM we wrote a small emulator in Python using Capstone to disassemble the executable’s code and translate it into CNF clauses for pysat.
The snapshotting approach allowed us to avoid emulating the entire binary from the beginning: we executed the code concretely with Unicorn up to the start of the specific stage, saved the state of the stack and registers, and from there started the symbolic emulator, setting only the missing ternary digits as symbolic variables.
For example to enforce ASCII characters for the flag we used pseudo-Boolean constraints linking the input ternary digits to the corresponding bytes:
def add_byte_constraints(emu: SatEmu, trit_start: int, trit_count: int, allowed_per_byte: list[bytes]) -> None:
carry_max = trit_count
carries: list[list[int]] = []
for _ in range(len(allowed_per_byte) + 1):
vals = [emu.cnf.new_lit() for _ in range(carry_max + 1)]
exactly_one(emu.cnf, vals)
carries.append(vals)
emu.cnf.add([carries[0][0]])
emu.cnf.add([carries[-1][0]])
for byte_index, allowed in enumerate(allowed_per_byte):
byte_lits = [emu.cnf.new_lit() for _ in allowed]
exactly_one(emu.cnf, byte_lits)
lits: list[int] = []
weights: list[int] = []
for i in range(trit_count):
coeff = (3**i >> (8 * byte_index)) & 0xFF
if not coeff:
continue
val = emu.inputs[trit_start + i]
assert val.lits is not None
for digit in (1, 2):
lits.append(val.lits[digit])
weights.append(coeff * digit)
for carry_value in range(carry_max + 1):
lits.append(carries[byte_index][carry_value])
weights.append(carry_value)
for i, char in enumerate(allowed):
lits.append(byte_lits[i])
weights.append(255 - char)
for carry_value in range(carry_max + 1):
lits.append(carries[byte_index + 1][carry_value])
weights.append(256 * (carry_max - carry_value))
cnf = PBEnc.equals(
lits=lits,
weights=weights,
bound=255 + 256 * carry_max,
top_id=emu.cnf.next_var - 1,
encoding=PBEncType.adder,
)
emu.cnf.clauses.extend(cnf.clauses)
emu.cnf.next_var = max(emu.cnf.next_var, cnf.nv + 1)
For the second-to-last stage we started by loading the snapshot at address pie + 0x26400, running the symbolic emulator up to 0x325BC, and requiring that the stack value at address rsp + 0xF08 be equal to 2:
def solve_stage4_sat(prefix: bytes) -> bytes:
emu, _snap = make_symbolic_emu(prefix, list(range(225, 266)), 0x26400)
emu.run(0x325BC, start=0x26400)
emu.cnf.force(emu.qload(emu.regs["rsp"] + 0xF08), 2)
add_byte_constraints(emu, 225, 41, [BASE62] * 8)
model = solve_cnf(emu.cnf, "stage4")
raw = model_to_raw(prefix, emu, model)
return raw[:52]
This gave us the partial flag: bbb{kQMM2FhlSBO4fEYFho5azaRrlTdxPsRx6ue7npnjATDcm6d4.
At this point we repeated the exact same procedure for the final stage, starting from the snapshot at pie + 0x325BC and emulating up to 0x4D881, forcing the rax register to 2 to satisfy the final check of the validation function:
def solve_final_sat(prefix: bytes) -> bytes:
emu, _snap = make_symbolic_emu(prefix, list(range(266, 348)), 0x325BC)
emu.run(0x4D881, start=0x325BC)
emu.cnf.force(emu.regs["rax"], 2)
add_byte_constraints(emu, 266, 82, [BASE62] * 15 + [b"}"])
model = solve_cnf(emu.cnf, "final")
raw = model_to_raw(prefix, emu, model)
return raw[:68]
Solving this final SAT block, we obtained the last missing bytes and the closing } character, thus recovering the final flag:
bbb{kQMM2FhlSBO4fEYFho5azaRrlTdxPsRx6ue7npnjATDcm6d4hPe25PNGBdT9MK0}
Show
sat_stage.py#!/usr/bin/env python3
from __future__ import annotations
import argparse
import struct
from dataclasses import dataclass
from pathlib import Path
from capstone import Cs, CS_ARCH_X86, CS_MODE_64
from capstone.x86 import X86_OP_IMM, X86_OP_MEM, X86_OP_REG
from pysat.solvers import Solver
CODE = Path("myfavoriteinstructions").read_bytes()
TEXT_START = 0x11A0
INPUT = 0x100000
STACK = 0x800000
MASK64 = (1 << 64) - 1
MD = Cs(CS_ARCH_X86, CS_MODE_64)
MD.detail = True
INSNS = {ins.address: ins for ins in MD.disasm(CODE[TEXT_START:0x4D990], TEXT_START)}
REG64 = {}
for names, full in [
(["rax", "eax", "ax", "al", "ah"], "rax"),
(["rbx", "ebx", "bx", "bl", "bh"], "rbx"),
(["rcx", "ecx", "cx", "cl", "ch"], "rcx"),
(["rdx", "edx", "dx", "dl", "dh"], "rdx"),
(["rsi", "esi", "si", "sil"], "rsi"),
(["rdi", "edi", "di", "dil"], "rdi"),
(["rbp", "ebp", "bp", "bpl"], "rbp"),
(["rsp", "esp", "sp", "spl"], "rsp"),
]:
for name in names:
REG64[name] = full
for i in range(8, 16):
for suffix in ("", "d", "w", "b"):
REG64[f"r{i}{suffix}"] = f"r{i}"
def trits_to_bytes_prefix(trits: list[int]) -> bytes:
out = bytearray()
pos = 0
for _ in range(4):
x = 0
p = 1
for t in trits[pos : pos + 5]:
x += t * p
p *= 3
out.append(x & 0xff)
pos += 5
for ntrits, size in [(82, 16), (82, 16), (41, 8), (41, 8), (82, 16)]:
x = 0
p = 1
for t in trits[pos : pos + ntrits]:
x += t * p
p *= 3
out += x.to_bytes(size, "little")
pos += ntrits
return bytes(out)
def flag_to_trits(s: bytes) -> list[int]:
assert len(s) >= 68
out: list[int] = []
for b in s[:4]:
x = b
for _ in range(5):
out.append(x % 3)
x //= 3
for off, ntrits in [(4, 82), (20, 82), (36, 41), (44, 41), (52, 82)]:
size = 16 if ntrits == 82 else 8
x = int.from_bytes(s[off : off + size], "little")
for _ in range(ntrits):
out.append(x % 3)
x //= 3
assert len(out) == 348
return out
def canon(name: str) -> str:
return REG64.get(name, name)
@dataclass(frozen=True)
class TVal:
const: int | None = None
lits: tuple[int, int, int] | None = None
def is_const(self) -> bool:
return self.const is not None
class CNF:
def __init__(self):
self.next_var = 1
self.clauses: list[list[int]] = []
self.cache: dict[tuple, TVal] = {}
def new_lit(self) -> int:
lit = self.next_var
self.next_var += 1
return lit
def add(self, clause: list[int]) -> None:
self.clauses.append(clause)
def new_val(self) -> TVal:
lits = (self.new_lit(), self.new_lit(), self.new_lit())
self.add([*lits])
self.add([-lits[0], -lits[1]])
self.add([-lits[0], -lits[2]])
self.add([-lits[1], -lits[2]])
return TVal(lits=lits)
def const(self, value: int) -> TVal | int:
if 0 <= value <= 2:
return TVal(const=value)
return value
def force(self, val: TVal, value: int) -> None:
if val.is_const():
if val.const != value:
self.add([])
return
assert val.lits is not None
self.add([val.lits[value]])
def gate2(self, op: str, a: TVal, b: TVal, table) -> TVal:
if a.is_const() and b.is_const():
return TVal(const=table[a.const][b.const])
key = (op, a, b)
if key in self.cache:
return self.cache[key]
out = self.new_val()
assert out.lits is not None
avals = [a.const] if a.is_const() else [0, 1, 2]
bvals = [b.const] if b.is_const() else [0, 1, 2]
for av in avals:
for bv in bvals:
cond = []
if not a.is_const():
assert a.lits is not None
cond.append(-a.lits[av])
if not b.is_const():
assert b.lits is not None
cond.append(-b.lits[bv])
self.add([*cond, out.lits[table[av][bv]]])
self.cache[key] = out
return out
def bsr(self, old: TVal | int, src: TVal | int) -> TVal | int:
if isinstance(src, int):
if src == 0:
return old
return src.bit_length() - 1
if isinstance(old, int):
if 0 <= old <= 2:
old = TVal(const=old)
else:
raise NotImplementedError(("nontrit old bsr", old, src))
table = [[oldv, 0, 1] for oldv in range(3)]
return self.gate2("bsr", old, src, table)
def bzhi(self, src: TVal | int, idx: TVal | int) -> TVal | int:
if isinstance(idx, int):
if idx == 0:
return 0
if idx == 1:
if isinstance(src, int):
return src & 1
return self.gate2("mod2", src, TVal(const=0), [[0], [1], [0]]) # unreachable
return src
if isinstance(src, int):
if 0 <= src <= 2:
src = TVal(const=src)
else:
raise NotImplementedError(("nontrit src bzhi", src, idx))
table = [
[0, 0, 0],
[0, 1, 1],
[0, 0, 2],
]
return self.gate2("bzhi", src, idx, table)
def is_conc(x) -> bool:
return isinstance(x, int)
class SatEmu:
def __init__(self, fixed: list[int], symbolic: list[int]):
self.cnf = CNF()
self.regs = {r: 0 for r in ["rax", "rbx", "rcx", "rdx", "rsi", "rdi", "rbp", "rsp", "r8", "r9", "r10", "r11", "r12", "r13", "r14", "r15"]}
self.regs.update({
"rbx": 349,
"r9": 266,
"r12": INPUT,
"r13": 0x200000,
"r14": 0xAAAAAAAAAAAAAAAB,
"r15": 0xAAAAAAAAAAAAAAAA,
"rsp": STACK + 0x100008,
})
self.mem: dict[int, TVal | int] = {}
self.xmm: dict[str, tuple[TVal | int, TVal | int]] = {}
self.inputs: dict[int, TVal] = {}
symbolic_set = set(symbolic)
for i in range(348):
if i in symbolic_set:
val = self.cnf.new_val()
self.inputs[i] = val
self.mem[INPUT + i * 8] = val
else:
self.mem[INPUT + i * 8] = fixed[i]
self.last_cmp = (0, 0)
self.steps = 0
def reg(self, name: str):
return self.regs.get(canon(name), 0)
def write_reg(self, name: str, val):
self.regs[canon(name)] = val
def addr(self, ins, op) -> int:
m = op.mem
base = 0
if m.base:
bname = ins.reg_name(m.base)
base = ins.address + ins.size if bname == "rip" else self.reg(bname)
if not is_conc(base):
raise NotImplementedError(("symbolic base", hex(ins.address), ins.op_str))
idx = 0
if m.index:
idx = self.reg(ins.reg_name(m.index))
if not is_conc(idx):
raise NotImplementedError(("symbolic index", hex(ins.address), ins.op_str))
return (base + idx * m.scale + m.disp) & MASK64
def qload(self, addr: int):
if addr in self.mem:
return self.mem[addr]
if 0 <= addr <= len(CODE) - 8:
return int.from_bytes(CODE[addr:addr + 8], "little")
return 0
def qstore(self, addr: int, val):
self.mem[addr] = val
def read_op(self, ins, op):
if op.type == X86_OP_IMM:
return op.imm & MASK64
if op.type == X86_OP_REG:
return self.reg(ins.reg_name(op.reg))
if op.type == X86_OP_MEM:
return self.qload(self.addr(ins, op))
raise NotImplementedError(op.type)
def write_op(self, ins, op, val):
if op.type == X86_OP_REG:
self.write_reg(ins.reg_name(op.reg), val)
return
if op.type == X86_OP_MEM:
self.qstore(self.addr(ins, op), val)
return
raise NotImplementedError(op.type)
def copy(self, dst: int, src: int, nbytes: int):
vals = [self.qload(src + i * 8) for i in range(nbytes // 8)]
for i, val in enumerate(vals):
self.qstore(dst + i * 8, val)
def zero(self, dst: int, nbytes: int):
for i in range(nbytes // 8):
self.qstore(dst + i * 8, 0)
def run(self, stop: int, start: int = TEXT_START):
rip = start
while rip != stop:
ins = INSNS[rip]
nrip = rip + ins.size
self.steps += 1
if self.steps % 250000 == 0:
print("steps", self.steps, "rip", hex(rip), "vars", self.cnf.next_var - 1, "clauses", len(self.cnf.clauses), flush=True)
m = ins.mnemonic
ops = ins.operands
if m in {"nop", "endbr64"}:
pass
elif m == "push":
self.regs["rsp"] -= 8
self.qstore(self.regs["rsp"], self.read_op(ins, ops[0]))
elif m == "pop":
self.write_op(ins, ops[0], self.qload(self.regs["rsp"]))
self.regs["rsp"] += 8
elif m == "mov":
self.write_op(ins, ops[0], self.read_op(ins, ops[1]))
elif m in {"movups", "movaps"}:
dst, src = ops
if dst.type == X86_OP_REG:
addr = self.addr(ins, src)
self.xmm[ins.reg_name(dst.reg)] = (self.qload(addr), self.qload(addr + 8))
elif dst.type == X86_OP_MEM:
addr = self.addr(ins, dst)
lo, hi = self.xmm[ins.reg_name(src.reg)]
self.qstore(addr, lo)
self.qstore(addr + 8, hi)
else:
raise NotImplementedError((hex(rip), m, ins.op_str))
elif m == "xorps":
self.xmm[ins.reg_name(ops[0].reg)] = (0, 0)
elif m == "xor":
if ops[0].type == X86_OP_REG and ops[1].type == X86_OP_REG and ops[0].reg == ops[1].reg:
self.write_op(ins, ops[0], 0)
else:
a, b = self.read_op(ins, ops[0]), self.read_op(ins, ops[1])
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic xor", hex(rip), ins.op_str))
self.write_op(ins, ops[0], a ^ b)
elif m == "add":
a, b = self.read_op(ins, ops[0]), self.read_op(ins, ops[1])
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic add", hex(rip), ins.op_str))
self.write_op(ins, ops[0], (a + b) & MASK64)
elif m == "sub":
a, b = self.read_op(ins, ops[0]), self.read_op(ins, ops[1])
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic sub", hex(rip), ins.op_str))
self.write_op(ins, ops[0], (a - b) & MASK64)
elif m == "inc":
a = self.read_op(ins, ops[0])
if not is_conc(a):
raise NotImplementedError(("symbolic inc", hex(rip), ins.op_str))
self.write_op(ins, ops[0], a + 1)
elif m == "shr":
a, b = self.read_op(ins, ops[0]), self.read_op(ins, ops[1])
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic shr", hex(rip), ins.op_str))
self.write_op(ins, ops[0], (a >> (b & 0x3f)) & MASK64)
elif m == "shl":
a, b = self.read_op(ins, ops[0]), self.read_op(ins, ops[1])
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic shl", hex(rip), ins.op_str))
self.write_op(ins, ops[0], (a << (b & 0x3f)) & MASK64)
elif m == "addl":
a, b = self.read_op(ins, ops[0]), self.read_op(ins, ops[1])
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic addl", hex(rip), ins.op_str))
self.write_op(ins, ops[0], (a + b) & MASK64)
elif m == "lea":
self.write_op(ins, ops[0], self.addr(ins, ops[1]))
elif m == "imul":
if len(ops) == 3:
a, b = self.read_op(ins, ops[1]), self.read_op(ins, ops[2])
self.write_op(ins, ops[0], a * b)
elif len(ops) == 2:
a, b = self.read_op(ins, ops[0]), self.read_op(ins, ops[1])
self.write_op(ins, ops[0], a * b)
else:
raise NotImplementedError((hex(rip), m, ins.op_str))
elif m == "bsr":
self.write_op(ins, ops[0], self.cnf.bsr(self.read_op(ins, ops[0]), self.read_op(ins, ops[1])))
elif m == "bzhi":
self.write_op(ins, ops[0], self.cnf.bzhi(self.read_op(ins, ops[1]), self.read_op(ins, ops[2])))
elif m == "cmp":
self.last_cmp = (self.read_op(ins, ops[0]), self.read_op(ins, ops[1]))
elif m in {"jne", "je", "jb"}:
a, b = self.last_cmp
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic branch", hex(rip), ins.op_str))
take = (a != b) if m == "jne" else (a == b) if m == "je" else (a < b)
if take:
nrip = int(ops[0].imm)
elif m == "jmp":
nrip = int(ops[0].imm)
elif m == "cmovae":
a, b = self.last_cmp
if not (is_conc(a) and is_conc(b)):
raise NotImplementedError(("symbolic cmovae", hex(rip), ins.op_str))
if a >= b:
self.write_op(ins, ops[0], self.read_op(ins, ops[1]))
elif m == "call":
target = int(ops[0].imm)
if target == 0x1050:
self.zero(self.regs["rdi"], self.regs["rdx"])
self.regs["rax"] = self.regs["rdi"]
elif target in {0x1060, 0x1090}:
self.copy(self.regs["rdi"], self.regs["rsi"], self.regs["rdx"])
self.regs["rax"] = self.regs["rdi"]
else:
raise NotImplementedError(("call", hex(rip), hex(target)))
else:
raise NotImplementedError((hex(rip), m, ins.op_str))
rip = nrip
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--stage", choices=["f10"], default="f10")
ap.add_argument("--solver", default="kissat")
args = ap.parse_args()
fixed = flag_to_trits((b"bbb{" + b"A" * 64))
symbolic = list(range(20, 184))
emu = SatEmu(fixed, symbolic)
emu.run(0x17CB6)
target = struct.unpack("<168Q", CODE[0x4E050:0x4E050 + 168 * 8])
base = emu.regs["rsp"] + 0x1500
for i, value in enumerate(target):
val = emu.qload(base + i * 8)
if isinstance(val, TVal):
emu.cnf.force(val, value)
else:
if val != value:
emu.cnf.add([])
print("built", "steps", emu.steps, "vars", emu.cnf.next_var - 1, "clauses", len(emu.cnf.clauses), flush=True)
with Solver(name=args.solver, bootstrap_with=emu.cnf.clauses) as solver:
print("solving", flush=True)
ok = solver.solve()
print("sat", ok, flush=True)
if not ok:
return
model = set(solver.get_model())
trits = fixed[:]
for i, val in emu.inputs.items():
assert val.lits is not None
for digit, lit in enumerate(val.lits):
if lit in model:
trits[i] = digit
break
raw = trits_to_bytes_prefix(trits)
print(raw[:68])
print(trits[20:184])
if __name__ == "__main__":
main()
Show
solve.py#!/usr/bin/env python3
import multiprocessing as mp
import pickle
import struct
import time
from pathlib import Path
from fpylll import IntegerMatrix, LLL
from pysat.pb import PBEnc, EncType as PBEncType
from pysat.solvers import Solver
from sat_stage import INPUT, SatEmu
from libdebug import debugger
# --- Constants & Helpers ---
BASE62 = b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
F10_FACTORS = (
221815467394800111963839297593696124903,
312491767943139940981443826148003062019,
)
ORDER3_20 = 3**20
BIN = Path("myfavoriteinstructions")
BIN_DATA = BIN.read_bytes()
TARGET_F10 = struct.unpack("<168Q", BIN_DATA[0x4E050 : 0x4E050 + 168 * 8])
T_F10 = sum(v * (3**i) for i, v in enumerate(TARGET_F10))
def flag_to_trits(s):
assert len(s) >= 68
out = []
for b in s[:4]:
x = b
for _ in range(5):
out.append(x % 3)
x //= 3
for off, ntrits in [(4, 82), (20, 82), (36, 41), (44, 41), (52, 82)]:
size = 16 if ntrits == 82 else 8
x = int.from_bytes(s[off : off + size], "little")
for _ in range(ntrits):
out.append(x % 3)
x //= 3
assert len(out) == 348
return out
def trits_to_bytes_prefix(trits):
out = bytearray()
pos = 0
for _ in range(4):
x = 0
p = 1
for t in trits[pos : pos + 5]:
x += t * p
p *= 3
out.append(x & 0xff)
pos += 5
for ntrits, size in [(82, 16), (82, 16), (41, 8), (41, 8), (82, 16)]:
x = 0
p = 1
for t in trits[pos : pos + ntrits]:
x += t * p
p *= 3
out += x.to_bytes(size, "little")
pos += ntrits
return bytes(out)
def get_pie_base(d):
for m in d.maps:
if 'myfavoriteinstructions' in (m.backing_file or ''):
return m.start
return 0x555555554000
# --- Libdebug Oracle for F10 / Multiplication Stage ---
def trits_of(n, length):
out = []
for _ in range(length):
out.append(n % 3)
n //= 3
if n:
return None
return out
def f10_stack_vectors(trits):
d = debugger(["./myfavoriteinstructions", "A" * 68], aslr=False)
d.run()
pie = get_pie_base(d)
bp_start = d.breakpoint(pie + 0x11A0)
d.cont()
d.wait()
trits_data = struct.pack("<348Q", *trits)
d.memory[d.regs.r12, len(trits_data)] = trits_data
bp_start.disable()
bp = d.breakpoint(pie + 0x5400)
d.cont()
d.wait()
rsp = d.regs.rsp
a_data = d.memory[rsp + 0x420, 168 * 8]
b_data = d.memory[rsp + 0x1F80, 168 * 8]
a = list(struct.unpack("<168Q", a_data))
b = list(struct.unpack("<168Q", b_data))
d.kill()
return a, b
def f10_affine_offsets():
base = flag_to_trits(b"A" * 68)
alpha = []
beta = []
for i in range(82):
avals = []
bvals = []
for x in range(3):
tr = base[:]
tr[20 + i] = x
tr[102 + i] = x
a, b = f10_stack_vectors(tr)
avals.append(a[i])
bvals.append(b[i])
alpha.append(avals[0])
beta.append(bvals[0])
return alpha, beta
def f10_ok(raw):
d = debugger(["./myfavoriteinstructions", raw.decode("latin-1")], aslr=False)
d.run()
pie = get_pie_base(d)
bp = d.breakpoint(pie + 0x17CB6)
d.cont()
d.wait()
rsp = d.regs.rsp
vec = struct.unpack("<168Q", d.memory[rsp + 0x1500, 168 * 8])
d.kill()
return vec == TARGET_F10
def solve_prefix_and_f10():
print("[f10] decoding multiplication stage")
alpha, beta = f10_affine_offsets()
base = flag_to_trits(b"bbb{" + b"A" * 64)
for left, right in (F10_FACTORS, F10_FACTORS[::-1]):
ad = trits_of(left, 82)
bd = trits_of(right, 82)
if ad is None or bd is None:
continue
trits = base[:]
trits[20:102] = [(ad[i] - alpha[i]) % 3 for i in range(82)]
trits[102:184] = [(bd[i] - beta[i]) % 3 for i in range(82)]
try:
raw = trits_to_bytes_prefix(trits)
except OverflowError:
continue
if all(32 <= c < 127 for c in raw[:36]) and f10_ok(raw):
print(f"[f10] {raw[:36].decode()}")
return raw[:36]
raise SystemExit("f10 decode failed")
# --- Libdebug Oracle for Stage 3 (Cyclic Group) ---
ORDER20_OFFSETS = [
0x1F0, 0x1F8, 0x200, 0x208, 0x170, 0x158, 0x190, 0xD0, 0xD8, 0x198,
0x1A0, 0x1A8, 0x1B0, 0x1B8, 0x1C0, 0x1C8, 0x1D0, 0x178, 0x180, 0x188
]
ZERO = (0,) * 20
def load_table():
vals = struct.unpack("<%dQ" % (15 * 41 * 21), BIN_DATA[0x4E590 : 0x4E590 + 15 * 41 * 21 * 8])
return [[tuple(vals[(j * 41 + i) * 21 : (j * 41 + i) * 21 + 20]) for i in range(41)] for j in range(15)]
def load_targets():
vals = struct.unpack("<315Q", BIN_DATA[0x67930 : 0x67930 + 315 * 8])
return [tuple(vals[j * 21 : j * 21 + 20]) for j in range(15)]
class Oracle:
def __init__(self, prefix):
raw = prefix.ljust(68, b"A")
self.d = debugger(["./myfavoriteinstructions", raw.decode("latin-1")], aslr=False)
self.d.run()
self.pie = get_pie_base(self.d)
bp_start = self.d.breakpoint(self.pie + 0x1C0A1)
self.d.cont()
self.d.wait()
self.rsp = self.d.regs.rsp
self.snap_path = "/tmp/group_snap_libdebug.json"
snap = self.d.create_snapshot()
snap.save(self.snap_path)
self.bp_patch = self.d.breakpoint(self.pie + 0x1CFB9)
self.bp_stop = self.d.breakpoint(self.pie + 0x1CF90)
self.bp_patch.disable()
self.bp_stop.disable()
def _eval(self, custom, vals, stop):
self.d.load_snapshot(self.snap_path)
val_data = struct.pack("<41Q", *vals)
self.d.memory[self.rsp + 0x420, len(val_data)] = val_data
self.bp_patch.enable()
self.bp_stop.enable()
out = None
while True:
self.d.cont()
self.d.wait()
rip = self.d.regs.rip
if rip == self.pie + 0x1CFB9:
i = self.d.regs.r14
if i in custom:
row = custom[i] + (0,)
val_bytes = struct.pack("<21Q", *row)
self.d.memory[self.rsp + 0x9C0, len(val_bytes)] = val_bytes
elif rip == self.pie + 0x1CF90:
if self.d.regs.r14 == stop:
out = tuple(struct.unpack("<Q", self.d.memory[self.rsp + off, 8])[0] for off in ORDER20_OFFSETS)
break
else:
raise RuntimeError(f"Unexpected RIP {hex(rip)}")
self.bp_patch.disable()
self.bp_stop.disable()
return out
def add(self, a, b):
vals = [0] * 41
vals[0] = 1
vals[1] = 1
return self._eval({0: a, 1: b}, vals, 2)
def double(self, a):
vals = [0] * 41
vals[0] = 2
return self._eval({0: a}, vals, 1)
def stage3_affine_maps(prefix):
fixed = flag_to_trits(prefix.ljust(68, b"A"))
base = fixed[:]
maps = []
for i in range(41):
vals = []
for x in range(3):
tr = base[:]
tr[184 + i] = x
d = debugger(["./myfavoriteinstructions", "A" * 68], aslr=False)
d.run()
pie = get_pie_base(d)
bp_start = d.breakpoint(pie + 0x11A0)
d.cont()
d.wait()
trits_data = struct.pack("<348Q", *tr)
d.memory[d.regs.r12, len(trits_data)] = trits_data
bp_start.disable()
bp = d.breakpoint(pie + 0x1C0A1)
d.cont()
d.wait()
val = struct.unpack("<Q", d.memory[d.regs.rsp + 0x420 + 8 * i, 8])[0]
vals.append(val)
d.kill()
maps.append(vals)
invmaps = []
for vals in maps:
inv = [0, 0, 0]
for inp, out in enumerate(vals):
inv[out] = inp
invmaps.append(inv)
return maps, invmaps
def load_or_compute_stage3_logs():
cache_file = Path("stage3_logs.pkl")
if cache_file.exists():
data = pickle.loads(cache_file.read_bytes())
return data["coeffs"], data["rhs"]
raise SystemExit("stage3_logs.pkl not found! Computation skipped.")
def solve_stage3_lll(prefix):
print("[stage3] solving cyclic-group equations with LLL")
coeffs, rhs = load_or_compute_stage3_logs()
n, m = 41, 15
centered_rhs = [(rhs[j] - sum(coeffs[j][i] for i in range(n))) % ORDER3_20 for j in range(m)]
scale = 1000
marker = 10
mat = IntegerMatrix(n + m + 1, n + m + 1)
for i in range(n):
mat[i, i] = 1
for j in range(m):
mat[i, n + j] = scale * coeffs[j][i]
for j in range(m):
mat[n + j, n + j] = scale * ORDER3_20
mat[n + m, n + j] = -scale * centered_rhs[j]
mat[n + m, n + m] = marker
LLL.reduction(mat, delta=0.99)
mapped = None
for row in range(mat.nrows):
vec = [int(mat[row, col]) for col in range(mat.ncols)]
if abs(vec[-1]) != marker:
continue
sign = 1 if vec[-1] == marker else -1
centered = [sign * vec[i] for i in range(n)]
if all(v in (-1, 0, 1) for v in centered):
candidate = [v + 1 for v in centered]
ok = all((sum(coeffs[j][i] * candidate[i] for i in range(n)) - rhs[j]) % ORDER3_20 == 0 for j in range(m))
if ok:
mapped = candidate
break
if mapped is None:
raise SystemExit("stage3 LLL failed")
# Automatically resolve offsets with the dynamic prefix! No more hardcoded strings.
_maps, invmaps = stage3_affine_maps(prefix)
trits = flag_to_trits(prefix + b"A" * (68 - len(prefix)))
for i, value in enumerate(mapped):
trits[184 + i] = invmaps[i][value]
raw = trits_to_bytes_prefix(trits)
print(f"[stage3] {raw[:44].decode()}")
return raw[:44]
# --- Libdebug Snapshots for SAT stages ---
def libdebug_snapshot_at(raw, address):
d = debugger(["./myfavoriteinstructions", "A" * 68], aslr=False)
d.run()
pie = get_pie_base(d)
bp_start = d.breakpoint(pie + 0x11A0)
d.cont()
d.wait()
tr = flag_to_trits(raw.ljust(68, b"A"))
trits_data = struct.pack("<348Q", *tr)
d.memory[d.regs.r12, len(trits_data)] = trits_data
bp_start.disable()
bp = d.breakpoint(pie + address)
d.cont()
d.wait()
rsp = d.regs.rsp
snap = {
"regs": {
"rax": d.regs.rax, "rbx": d.regs.rbx, "rcx": d.regs.rcx, "rdx": d.regs.rdx,
"rsi": d.regs.rsi, "rdi": d.regs.rdi, "rbp": d.regs.rbp, "rsp": d.regs.rsp,
"r8": d.regs.r8, "r9": d.regs.r9, "r10": d.regs.r10, "r11": d.regs.r11,
"r12": d.regs.r12, "r13": d.regs.r13, "r14": d.regs.r14, "r15": d.regs.r15,
},
"stack_base": rsp - 0x3000,
}
snap["stack"] = bytes(d.memory[snap["stack_base"], 0x18000])
snap["trit_ptr"] = struct.unpack("<Q", d.memory[rsp + 0x358, 8])[0]
d.kill()
return snap
def make_symbolic_emu(prefix, symbolic, snap_addr):
raw = prefix + b"A" * (68 - len(prefix))
fixed = flag_to_trits(raw)
snap = libdebug_snapshot_at(raw, snap_addr)
emu = SatEmu(fixed, symbolic)
emu.regs.update(snap["regs"])
for off in range(0, len(snap["stack"]), 8):
emu.mem[snap["stack_base"] + off] = int.from_bytes(snap["stack"][off : off + 8], "little")
for i in range(348):
emu.mem[snap["trit_ptr"] + i * 8] = emu.mem[INPUT + i * 8]
return emu, snap
# --- SAT Helpers ---
def exactly_one(cnf, lits):
cnf.add(lits[:])
for i, a in enumerate(lits):
for b in lits[i + 1 :]:
cnf.add([-a, -b])
def add_byte_constraints(emu, trit_start, trit_count, allowed_per_byte):
carry_max = trit_count
carries = []
for _ in range(len(allowed_per_byte) + 1):
vals = [emu.cnf.new_lit() for _ in range(carry_max + 1)]
exactly_one(emu.cnf, vals)
carries.append(vals)
emu.cnf.add([carries[0][0]])
emu.cnf.add([carries[-1][0]])
for byte_index, allowed in enumerate(allowed_per_byte):
byte_lits = [emu.cnf.new_lit() for _ in allowed]
exactly_one(emu.cnf, byte_lits)
lits = []
weights = []
for i in range(trit_count):
coeff = (3**i >> (8 * byte_index)) & 0xFF
if not coeff:
continue
val = emu.inputs[trit_start + i]
assert val.lits is not None
for digit in (1, 2):
lits.append(val.lits[digit])
weights.append(coeff * digit)
for carry_value in range(carry_max + 1):
lits.append(carries[byte_index][carry_value])
weights.append(carry_value)
for i, char in enumerate(allowed):
lits.append(byte_lits[i])
weights.append(255 - char)
for carry_value in range(carry_max + 1):
lits.append(carries[byte_index + 1][carry_value])
weights.append(256 * (carry_max - carry_value))
cnf = PBEnc.equals(
lits=lits,
weights=weights,
bound=255 + 256 * carry_max,
top_id=emu.cnf.next_var - 1,
encoding=PBEncType.adder,
)
emu.cnf.clauses.extend(cnf.clauses)
emu.cnf.next_var = max(emu.cnf.next_var, cnf.nv + 1)
def solve_cnf(cnf, label):
for solver_name in ("kissat404",):
try:
with Solver(name=solver_name, bootstrap_with=cnf.clauses) as solver:
print(f"[{label}] solving with {solver_name}")
started = time.time()
ok = solver.solve()
print(f"[{label}] {solver_name}: {ok} in {time.time() - started:.1f}s")
if ok:
return set(solver.get_model())
except Exception as exc:
print(f"[{label}] {solver_name} failed: {exc}")
raise SystemExit(f"{label}: no SAT solution")
def model_to_raw(prefix, emu, model):
trits = flag_to_trits(prefix + b"A" * (68 - len(prefix)))
for index, val in emu.inputs.items():
assert val.lits is not None
for digit, lit in enumerate(val.lits):
if lit in model:
trits[index] = digit
break
return trits_to_bytes_prefix(trits)
def solve_stage4_sat(prefix):
print("[stage4] solving 41-trit SAT block")
emu, _snap = make_symbolic_emu(prefix, list(range(225, 266)), 0x26400)
emu.run(0x325BC, start=0x26400)
emu.cnf.force(emu.qload(emu.regs["rsp"] + 0xF08), 2)
add_byte_constraints(emu, 225, 41, [BASE62] * 8)
print(f"[stage4] vars={emu.cnf.next_var - 1} clauses={len(emu.cnf.clauses)}")
model = solve_cnf(emu.cnf, "stage4")
raw = model_to_raw(prefix, emu, model)
print(f"[stage4] {raw[:52].decode()}")
return raw[:52]
def solve_final_sat(prefix):
print("[final] solving 82-trit SAT block")
emu, _snap = make_symbolic_emu(prefix, list(range(266, 348)), 0x325BC)
emu.run(0x4D881, start=0x325BC)
emu.cnf.force(emu.regs["rax"], 2)
add_byte_constraints(emu, 266, 82, [BASE62] * 15 + [b"}"])
print(f"[final] vars={emu.cnf.next_var - 1} clauses={len(emu.cnf.clauses)}")
model = solve_cnf(emu.cnf, "final")
raw = model_to_raw(prefix, emu, model)
print(f"[final] {raw[:68].decode()}")
return raw[:68]
def main():
print("=== myfavoriteinstructions solver ===")
start_time = time.time()
# We start with a generic placeholder since the stages evaluate independently
flag = solve_prefix_and_f10()
flag = solve_stage3_lll(flag)
flag = solve_stage4_sat(flag)
flag = solve_final_sat(flag)
# We now have the correct suffix, brute-force the independent first 4 bytes!
print(f"\n[*] FLAG: {flag.decode()}")
print(f"[*] Solved in {time.time() - start_time:.2f} seconds")
if __name__ == "__main__":
mp.set_start_method("fork")
main()