To avoide having a phalanx killing a tank there is a simple solution.

First of all you might give the tank a defense value incredibly high. This makes balanceing quite complicated.

I prefer another thought to solve this. The only problem behind the whole thing is, that the two units are fromdifferent tech levels. A stack of eight phalanx is not able to do any damage to a tank. At that time of technology they hardly knew a screwdriver.

So any unit should receive a tech level. If units of different tech levels fight each other, the unit with the higher tech level receives a malus on it's attack value. A malus is the opposite of a bonus. While 50% bonus means "add the half", 50% malus means "substract the half".

The more the two tech levels differ the progressively higher the malus on the attack value.

Guess the folowing figures:

Phalanx Attack 1, Defense 2, Tech level 1
Musketeer Attack 3, Defense 3, Tech Level 4
WWII Tank Attack 10, Defense 10, Tech level 5

Note that I speak of a WWII tank. It has no radar, no advanced communication etc. Therefore it only has a +1 higher tech level than the musketeer.

But between Phalanx and musketeer there is +3 difference, because between spear and rifle (fine mechanism and explosives) are 2 thousand years of development. Actually it is not about the distance of time. Development speeds up every day with additional knowledge of man.

In the implemented version it should be the following way:
If the maximum tech level is 6, the maximum tech level gap between two units is 5.
If the tech level gap between two units is X it should receive a Y% malus on it's attack value:
tech level gap malus
1 10%
2 20%
3 35%
4 55%
5 85%
Note the progressive way the malus gains.

If musketeer attacks tank, the musketeer should receive 10% malus on its attack value. The rifle is just not the effective against the more modern armor of the tank.

But if phalanx attacks tank, the phalanx will receive a 55% malus on it's attack value making it impossible to cause any damage. Tank defense will remain 10 but phalanx attack will decrease to 0.45.

Your point that they can get away in certain terrain is completely valid, however. They just couldn't do much damage to the armor unit. There should be a chance that the attacking armor might "miss" the phalanx.

Unarmed troops could actually slow or stop an armor unit temporarily by using obstacles or, in the case of lightly armed troops, harassing fire in an urban setting.

We don't Have any problems with stacks of phalanxes vs modern armor. That's all due to the bizarre way Civ-like games handle combat power.

You'd get further if you'd do a little reading first...

In the Clash Status thread (referred to in my post @ CTP) is a list of all the major game models. Under the military area you would have found my suggestion for how to handle different tech levels of units in the thread Military/Combat System - Whatcha Think?

I'll check it, but the thread is quit loooong