lambda and beta are non-trivial cube roots of 1 modulo n and p.
λ
3 = 1 (mod n)
β
3 = 1 (mod p)
[λ](x,y) = (βx,y)
Since there are 3 of each you'd have to match them (2 are non-trivial, so there is just one equality check).
You could find how to get such root of 1
here.
So, let your curve is modulo q, and has order m, both prime numbers.
Compute a primitive root of 1 modulo q:
a≠1
a
q-1 = 1 (mod q)
Then the cube root of 1 would be:
k = a
(q-1)/3 = 1
1/3 (mod q)
Instead of finding a primitive root of 1, one could directly find a cube root of 1, this is faster:
k = (-1-sqrt(-3))/2 (mod q)
Do the same modulo the order of the curve m:
b≠1
b
m-1 = 1 (mod m)
λ = b
(m-1)/3 = 1
1/3 (mod m)
Then check which one matches:
[λ](x,y) = (kx,y)
β = k
or
[λ](x,y) = (k
2x,y)
β = k
2 (mod q)