Line a loaf pan with parchment paper and keep aside to be used later.
Pour the milk in a bowl, crush the saffron thread into it, give it a stir and keep aside for 5 minutes. The milk will get infused and turn into saffron color.
Warm the heavy cream slightly and add the orange juice along with sugar. Give it a stir and keep it aside. Do not heat the heavy cream too much or else, the orange juice will curdle the cream.
Take a heavy bottom pan and place it on medium heat. Pour the ghee and once the ghee melts, scatter the almond flour. Keep stirring constantly for 3 to 4 minutes and you will see the shade of the almond flour darkens slightly. Do not leave unattended as the flour can stick to the pan and burn in a second.
Then, pour the orange and cream mixture along with saffron milk into the pan. Keep stirring for at least 7 to 10 minutes or until most of the liquid is absorbed and you see the ghee separating at the sides of the pan. The consistency of the mixture will also thicken.
Put the entire mixture to the prepared loaf pan and press it down to form an even layer. Allow it cool down to the room temperature for a while and then, keep it in the refrigerator for a few hours or overnight to set.
The fudge comes out of the pan quite easily and the parchment paper also peels off without any effort. Using a serrated knife, slice into bite size fudge. You could either serve it right away or store in an airtight jar in the refrigerator. It stays good for a couple of weeks.